[SERVER-36601] WiredTigerLAS.wt does not shrink after major members are available Created: 12/Aug/18 Updated: 27/Oct/23 Resolved: 13/Aug/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Bruce Zu | Assignee: | Nick Brewer |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Participants: |
| Description |
|
This is a follow up of https://jira.mongodb.org/browse/SERVER-36512 but this topic focuses on the issue happens after retriggering the issue of unlimited grows of lookaside table: WiredTigerLAS.wt. Details see Issue of WiredTiger.wt grows unlimitedly is triggerable.pdf I want to ask: 1> Is it a bug that the WiredTigerLAS.wt cannot shrink to normal when the major data bearing member are available? 2> a question: If it is the designed behavior that the lookaside table: WiredTigerLAS.wt grows unlimitedly till it uses out all the diskspace if major data bearing members are not available. If true, I would like to know if there is configuration option or something else to avoid WiredTigerLAS.wt stop halfway before it uses out all the disk space? As a user, I dislike this kind of behavior, I expect mongod can be smart enough to stop before the disk space used by WiredTigerLAS.wt reaching to a configured percent of disk space. Is this available or possible? Thanks Bruce
|
| Comments |
| Comment by Bruce Zu [ 13/Aug/18 ] |
|
I got it. Nice! Thank you so much! Nick In my case: 1 primary, 2 secondary, 1 arbiter. The read concern majority number should be > 4 * (2 data bearning member)/( 4 member in replset) Suggest giving a precise description on how to calculate the majority in mongod document. Bruce |
| Comment by Nick Brewer [ 13/Aug/18 ] |
While only data-bearing members can satisfy a read concern majority, the majority is calculated against the total number of nodes in the replica set, regardless of whether or not they contain any data. Thus two data bearing nodes and one arbiter do not satisfy the majority in your four member replica set. See -Nick |
| Comment by Bruce Zu [ 13/Aug/18 ] |
|
Hi Nick How do you know " your replica set does not have a sufficient number of data bearing members to satisfy the read concern " Please do not judge it only from your picture. That is just the place there is a bug. I have shown you with the mongd.log in https://jira.mongodb.org/browse/SERVER-36512 and repeat more times in https://jira.mongodb.org/browse/SERVER-36597 lookaside table grows unlimited is the result, not the root reason for the issue. Please focus on why mongo calculate out the major data bearing members are down. I think there is a bug around there, in my case the calculated result is wrong. because only 1/3 data bearing member is unreachable. mongod.log can show it.
very regret to find lookaside table grows unlimited is designed behavior Thanks! Bruce
|
| Comment by Nick Brewer [ 13/Aug/18 ] |
|
brucezu This issue is occurring because your replica set does not have a sufficient number of data bearing members to satisfy the read concern - in the example from your previous ticket, lookaside table growth began accruing over a period of days once one of the secondaries went down, due to the amount of information that was required to be stored in cache: The red line in this graph represents the time (over a period of days) that your secondary was offline - you can see here that it directly lines up with the time when the lookaside table accumulated entries.
The content in the lookaside file is removed the next time that data is accessed, or when a table is dropped. However the recommended method for reclaiming the space is simply to reboot the mongod.
If the replica set is left without enough data-bearing nodes for a long enough time - in this case multiple days - this is in fact the expected behavior. Replica sets are not intended to be left without a data-bearing majority for such an extended period of time. The behavior of the lookaside table mirrors the behavior of mongod in general - it cannot continue to operate if it runs out of available disk space. -Nick |