[SERVER-27230] Poor compression of diagnostic data when replica set is unhealthy Created: 30/Nov/16 Updated: 22/May/18 Resolved: 27/Mar/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Diagnostics |
| Affects Version/s: | 3.2.11 |
| Fix Version/s: | 3.4.16, 3.6.6, 3.7.4 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Bruce Lucas (Inactive) | Assignee: | Vesselina Ratcheva (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | SWDI, neweng | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||
| Backport Requested: |
v3.6, v3.4, v3.2
|
||||||||||||||||||||
| Sprint: | Repl 2018-01-15, Repl 2018-01-29, Repl 2018-02-26, Repl 2018-03-12, Repl 2018-03-26, Repl 2018-04-09 | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
Under some circumstances involving network or other cluster health issues we can see frequent diagnostic data schema changes due to "lastHeartbeatMessage" and "syncingTo" fields that disappear and reappear frequently in replica set status. In one case this was observed to result in about 5x larger compressed ftdc data (~1300 bytes per sample vs the normal ~250 bytes per sample) and therefore about 5x shorter retention period, which is unfortunate because one use case for ftdc is diagnosing the cause of an unhealthy cluster. Since the fields in question have string values they don't contribute to ftdc data so shouldn't participate in determining whether there is a schema change. |
| Comments |
| Comment by Githook User [ 22/May/18 ] |
|
Author: {'username': 'vessy-mongodb', 'name': 'Vesselina Ratcheva', 'email': 'vesselina.ratcheva@10gen.com'}Message: (cherry picked from commit d01dde5b3cca80a5db8c07c6d3f029c60f3b332e) |
| Comment by Githook User [ 22/May/18 ] |
|
Author: {'username': 'smani87', 'name': 'Suganthi Mani', 'email': 'suganthi.mani@mongodb.com'}Message: (cherry picked from commit d01dde5b3cca80a5db8c07c6d3f029c60f3b332e) |
| Comment by Githook User [ 27/Mar/18 ] |
|
Author: {'email': 'vesselina.ratcheva@10gen.com', 'name': 'Vesselina Ratcheva', 'username': 'vessy-mongodb'}Message: |
| Comment by Bruce Lucas (Inactive) [ 23/Mar/18 ] |
|
Perfect, thanks. |
| Comment by Spencer Brody (Inactive) [ 23/Mar/18 ] |
|
bruce.lucas |
| Comment by Bruce Lucas (Inactive) [ 21/Mar/18 ] |
|
Thanks vesselina.ratcheva, spencer. Can we coordinate this with |
| Comment by Bruce Lucas (Inactive) [ 28/Feb/18 ] |
|
Would it be possible to implement |
| Comment by Bruce Lucas (Inactive) [ 28/Feb/18 ] |
|
The same problem is observed with respect to the infoMessage fields. There may be others. If it is preferred to keep these string fields optional, I think it's a simple fix for FTDC to ignore string-valued fields when deciding whether there has been a schema change (second option above). This would have the side benefit of fixing other places in serverStatus where we may have optional string-valued fields, if there are any other such. |
| Comment by Bruce Lucas (Inactive) [ 31/Jan/18 ] |
|
Two possible fixes have been discussed:
|