[SERVER-35652] Free Monitoring metric pings seem to double up, with different polling frequency Created: 18/Jun/18  Updated: 29/Oct/23  Resolved: 19/Jun/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.0.0-rc5
Fix Version/s: 4.0.0-rc6, 4.1.1

Type: Bug Priority: Critical - P2
Reporter: Duncan Armstrong Assignee: Mark Benvenuto
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0
Steps To Reproduce:

We haven't been able to reproduce it

Sprint: Platforms 2018-07-02
Participants:

 Description   

We (specifically, @john.morales) have observed an issue where the same metrics pings seem to be POSTed twice by the mongod, and at odd intervals. Specifically in this case, every minute at 45 seconds and 47 seconds past.

2018-06-17T15:25:45.636-0400	debug	metrics	/Users/johnmorales/projects/mms-free-monitoring-workspace/src/github.com/10gen/mms-free-monitoring/internal/metrics/service.go:33	Metrics request received from.	{"rid": "MSA4NjBlNz"}
2018-06-17T15:25:47.007-0400	debug	metrics	/Users/johnmorales/projects/mms-free-monitoring-workspace/src/github.com/10gen/mms-free-monitoring/internal/metrics/service.go:33	Metrics request received from.	{"rid": "MSA4NjBlNz"}
2018-06-17T15:26:45.652-0400	debug	metrics	/Users/johnmorales/projects/mms-free-monitoring-workspace/src/github.com/10gen/mms-free-monitoring/internal/metrics/service.go:33	Metrics request received from.	{"rid": "MSA4NjBlNz"}
2018-06-17T15:26:47.018-0400	debug	metrics	/Users/johnmorales/projects/mms-free-monitoring-workspace/src/github.com/10gen/mms-free-monitoring/internal/metrics/service.go:33	Metrics request received from.	{"rid": "MSA4NjBlNz"}
2018-06-17T15:26:50.780-0400	debug	metrics	/Users/johnmorales/projects/mms-free-monitoring-workspace/src/github.com/10gen/mms-free-monitoring/internal/metrics/service.go:33	Metrics request received from.	{"rid": "MSA4NjBlNz"}
2018-06-17T15:27:45.664-0400	debug	metrics	/Users/johnmorales/projects/mms-free-monitoring-workspace/src/github.com/10gen/mms-free-monitoring/internal/metrics/service.go:33	Metrics request received from.	{"rid": "MSA4NjBlNz"}

Our hunch is that either there are two POST threads running slightly out of sync, or two timers triggering the POSTs. This seems to happen indefinitely on the affected mongod, and never "fixes itself".

This issue has so far only happened once, and we haven't been able to reproduce it.



 Comments   
Comment by Githook User [ 19/Jun/18 ]

Author:

{'username': 'markbenvenuto', 'name': 'Mark Benvenuto', 'email': 'mark.benvenuto@mongodb.com'}

Message: SERVER-35652 Duplicate registrations should only trigger one metrics upload stream

(cherry picked from commit 99d48121c5bb18168eb08b20201065170a1d7213)
Branch: v4.0
https://github.com/mongodb/mongo/commit/28f341edb7ca112f80d2c6bf0bd23d17b4ed330a

Comment by Githook User [ 19/Jun/18 ]

Author:

{'username': 'markbenvenuto', 'name': 'Mark Benvenuto', 'email': 'mark.benvenuto@mongodb.com'}

Message: SERVER-35652 Duplicate registrations should only trigger one metrics upload stream
Branch: master
https://github.com/mongodb/mongo/commit/99d48121c5bb18168eb08b20201065170a1d7213

Comment by Mark Benvenuto [ 18/Jun/18 ]

This is an issue where we ignore the cancellation. If you enable, disable, and enable rapidly, we may ignore the cancellation, and start sending duplicates.

Comment by Mark Benvenuto [ 18/Jun/18 ]

Are you sure you were not just monitoring a replica set with two members? The first member would have sent metrics up at say ":45" seconds. And the secondary would have started sending metrics up at ":47" when it received the enable free monitoring request.

Comment by Duncan Armstrong [ 18/Jun/18 ]

CC jonathan.balsano, john.morales

Generated at Thu Feb 08 04:40:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.