[SERVER-17127] Make accounting for upserts consistent with oplog entries on Primaries Created: 30/Jan/15 Updated: 06/Dec/22 |
|
| Status: | Open |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | 2.6.6 |
| Fix Version/s: | Needs Further Definition |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Andre de Frere | Assignee: | Backlog - Replication Team |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Replication
|
||||||||
| Participants: | |||||||||
| Description |
|
There are three scenarios for an upsert. On the Primary all three will be reported as an update, but the actual oplog entry will not necessarily reflect this. When those oplog entries are replicated to Secondaries, the correct counters will be increased, but this makes the counts inconsistent with those on the Primary. The three scenarios are as follows
|
| Comments |
| Comment by Andre de Frere [ 16/Feb/15 ] | |||||||||||||||||||||||
|
I think that splitting the metrics would be good | |||||||||||||||||||||||
| Comment by Asya Kamsky [ 03/Feb/15 ] | |||||||||||||||||||||||
|
I think maybe it's more reasonable to split the metrics into "client requested" and "server executed". Client request 2 inserts, 5 updates and 2 deletes Of course then there are system triggered inserts, deletes, etc... | |||||||||||||||||||||||
| Comment by Andre de Frere [ 03/Feb/15 ] | |||||||||||||||||||||||
I think that the original request to make accounting for upserts to be consistent is to remove the uncertainty that comes from reading opcounters or metrics on the Primary, seeing the "update" counters and needing to differentiate between those numbers being updates or inserts (depending on how they are called). FWIW, that distinction does not need to be made on the Secondaries. While this distinction exists, there is no metric that you can rely on to tell you the number of updates, and conversely the number of inserts, that are actually occurring on a Primary node.
Certainly, other values could be used. It does not need to be opcounters.
Ignoring the oplog for one second, I think the idea is that an update that causes an insert to occur should be counted as an insert rather than an update. I think the difference is recording what underlying action the server actually took, rather than what the client issued. Given that the two operations could have different performance profiles, operationally speaking should the numbers that we gather agree with the actual operation that is occurring on the server, or what is being issued at the client? Now when including the OpLog, we can tell that the Primary records the op as an insert in the oplog, while the metrics (and opcounters) on the Primary have the op counted as an update (I've used this as shorthand for the actual underlying operation that occurs on the Primary in the description of the ticket). I understand that this comparison fails in the case where there are multiple documents being updated (since this will be recorded once in the opcounters, but will be recorded correctly in metrics) but I don't think you can generate multiple new documents from a single upsert.
Ideally those would not be counted, but I think this is a secondary concern as (I don't believe) this is measured in metrics (only in opcounters). I will update the description of the ticket to explicitly call out metrics. | |||||||||||||||||||||||
| Comment by Scott Hernandez (Inactive) [ 02/Feb/15 ] | |||||||||||||||||||||||
|
So you want the opcounter to record only what type the oplog entry is? And if there is no oplog entry created, due to a failure of the update, no documents found by the query or a noop for more than one of the documents, then it should still be an "update"? For example, this update is invalid but will be counted as an update operation the server receives, and produces a error for the client with no oplog entry produced:
Also, as you know, an update which effects multiple documents is still only countered once in the "opcounters", since it was just one operation issued from the client, but it may produce many updates to the actual collection and oplog entries. I'm not sure how this figures into your improvement request, but you might want to keep it in mind as an important detail. Now, as I've mentioned above, the "document" metrics will include the number of document operations done in the user's collections. As you can tell from my questions I'm still unclear exactly what the improvement is that you are requesting and how it would work. Maybe we can come at this from a different angle, what is the underlying reason you want "accounting for upserts consistent" across replicas? Also, why are the "opcounter" metrics the ones which must be changed – could other values be used to achieve the goals? | |||||||||||||||||||||||
| Comment by Andre de Frere [ 02/Feb/15 ] | |||||||||||||||||||||||
|
I think the issue with updates/inserts to the local.slaves collection will go away with Currently, we increment the "update" counter of opcounters when there is an update of any type - for example even if the update actually means that an insert operation occurs (and is therefore an insert operation as per the entry in the oplog). I realise that this is due to us counting the "update" operation from the client, rather than what the actual operation on the server ends up being. The question is philosophical - if an "update" from the client really means an insert on the server, should we increment the update opcounter, or the insert opcounter? I'm only using the oplog entry as a source of truth here, under the assumption that the oplog entry indicating that the op is an insert actually means the server itself performed an insert. This may be true for insert and delete as well, but I haven't come across that case. | |||||||||||||||||||||||
| Comment by Scott Hernandez (Inactive) [ 30/Jan/15 ] | |||||||||||||||||||||||
|
So the real issue here is that we do operations in the background/on-behalf-of-the-system on local.slaves/slavetracking and the oplog (that are not directly user-initiated), and you don't want those reflected in the serverStatus opcounter/metrics.document stats? Seems like this really has nothing to do with update + upsert, since insert and delete can result in different counts for opcounter/metrics.document stats as well. Could you try to refine the title and description to more accurately convey what the problem and improved behavior could be? | |||||||||||||||||||||||
| Comment by Andre de Frere [ 30/Jan/15 ] | |||||||||||||||||||||||
|
I don't think that the issue is that we are not recording an upsert, I think that the problem is we are recording everything as an update. I see that the difficulty is that the client is issuing an update (so that is what we record), but the OpLog entries do give different values. The problem is the same in either serverStatus().opcounters and serverStatus().metrics.document:
| |||||||||||||||||||||||
| Comment by Scott Hernandez (Inactive) [ 30/Jan/15 ] | |||||||||||||||||||||||
|
There is no "upsert" operation, just an option to update We have insert, update and delete operations issued from the clients and that is what the opcounters record – operations issued from the client. There are separate document/storage related metrics which I think is what you are really interested in:
I'm not sure that matching opcounters on the primary and secondary is ideal as they are separate things, and an indication of work being sent to server not what is actually done. It is arguable that all of these stats need better definition, and replacement but changing their meaning seems like a questionable choice due to compatibility and existing meaning. | |||||||||||||||||||||||
| Comment by Andre de Frere [ 30/Jan/15 ] | |||||||||||||||||||||||
|
If the accounting on the Primaries matched the Secondaries that would be ideal. I think that there is a small problem in that the operations on the Secondaries are actually what is being reported in the OpLog (since that is their source), while the Primaries write the op to the counter and then write the OpLog entry (which is then reported differently as in the description of the ticket). | |||||||||||||||||||||||
| Comment by Andy Schwerin [ 30/Jan/15 ] | |||||||||||||||||||||||
|
Seems to me that the secondaries have the more useful counters. If the primary used the accounting currently used on secondaries, would that be an acceptable outcome, andre@mongodb.com? |