[SERVER-27230] Poor compression of diagnostic data when replica set is unhealthy Created: 30/Nov/16  Updated: 22/May/18  Resolved: 27/Mar/18

Status: Closed
Project: Core Server
Component/s: Diagnostics
Affects Version/s: 3.2.11
Fix Version/s: 3.4.16, 3.6.6, 3.7.4

Type: Bug Priority: Major - P3
Reporter: Bruce Lucas (Inactive) Assignee: Vesselina Ratcheva (Inactive)
Resolution: Done Votes: 0
Labels: SWDI, neweng
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
depends on SERVER-33546 Add numeric syncing field to replSetG... Closed
Related
related to SERVER-26464 Poor compression of diagnostic data w... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v3.6, v3.4, v3.2
Sprint: Repl 2018-01-15, Repl 2018-01-29, Repl 2018-02-26, Repl 2018-03-12, Repl 2018-03-26, Repl 2018-04-09
Participants:

 Description   

Under some circumstances involving network or other cluster health issues we can see frequent diagnostic data schema changes due to "lastHeartbeatMessage" and "syncingTo" fields that disappear and reappear frequently in replica set status. In one case this was observed to result in about 5x larger compressed ftdc data (~1300 bytes per sample vs the normal ~250 bytes per sample) and therefore about 5x shorter retention period, which is unfortunate because one use case for ftdc is diagnosing the cause of an unhealthy cluster.

Since the fields in question have string values they don't contribute to ftdc data so shouldn't participate in determining whether there is a schema change.



 Comments   
Comment by Githook User [ 22/May/18 ]

Author:

{'username': 'vessy-mongodb', 'name': 'Vesselina Ratcheva', 'email': 'vesselina.ratcheva@10gen.com'}

Message: SERVER-27230 Always emit string status fields in replica set status

(cherry picked from commit d01dde5b3cca80a5db8c07c6d3f029c60f3b332e)
Branch: v3.6
https://github.com/mongodb/mongo/commit/2fd20c7ffd8c99ce6c404ec24b8e302045a30071

Comment by Githook User [ 22/May/18 ]

Author:

{'username': 'smani87', 'name': 'Suganthi Mani', 'email': 'suganthi.mani@mongodb.com'}

Message: SERVER-27230 Always emit string status fields in replica set status

(cherry picked from commit d01dde5b3cca80a5db8c07c6d3f029c60f3b332e)
Branch: v3.4
https://github.com/mongodb/mongo/commit/a5dc50015eb113aecb62f22af67b40241da14e27

Comment by Githook User [ 27/Mar/18 ]

Author:

{'email': 'vesselina.ratcheva@10gen.com', 'name': 'Vesselina Ratcheva', 'username': 'vessy-mongodb'}

Message: SERVER-27230 Always emit string status fields in replica set status
Branch: master
https://github.com/mongodb/mongo/commit/d01dde5b3cca80a5db8c07c6d3f029c60f3b332e

Comment by Bruce Lucas (Inactive) [ 23/Mar/18 ]

Perfect, thanks.

Comment by Spencer Brody (Inactive) [ 23/Mar/18 ]

bruce.lucas SERVER-33546 will go in in a couple of weeks, but we'll make sure to backport them together.

Comment by Bruce Lucas (Inactive) [ 21/Mar/18 ]

Thanks vesselina.ratcheva, spencer. Can we coordinate this with SERVER-33546 (which should be an equally small code change) so that both go in (or at least both are backported) at the same time, for the reasons detailed on that ticket (basically to preserve our ability to reconstruct sync source information from ftdc data)?

Comment by Bruce Lucas (Inactive) [ 28/Feb/18 ]

Would it be possible to implement SERVER-33546 at the same time this is fixed? See that ticket for details about why this would be useful.

Comment by Bruce Lucas (Inactive) [ 28/Feb/18 ]

The same problem is observed with respect to the infoMessage fields. There may be others.

If it is preferred to keep these string fields optional, I think it's a simple fix for FTDC to ignore string-valued fields when deciding whether there has been a schema change (second option above). This would have the side benefit of fixing other places in serverStatus where we may have optional string-valued fields, if there are any other such.

Comment by Bruce Lucas (Inactive) [ 31/Jan/18 ]

Two possible fixes have been discussed:

  • Fix FTDC to ignore string-valued fields when deciding whether there has been a schema change, as string-valued fields are not emitted in FTDC data.
Generated at Thu Feb 08 04:14:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.