[SERVER-17127] Make accounting for upserts consistent with oplog entries on Primaries Created: 30/Jan/15  Updated: 06/Dec/22

Status: Open
Project: Core Server
Component/s: Replication
Affects Version/s: 2.6.6
Fix Version/s: Needs Further Definition

Type: Improvement Priority: Major - P3
Reporter: Andre de Frere Assignee: Backlog - Replication Team
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-15825 Add stats for updates (w/upsert) whic... Open
Assigned Teams:
Replication
Participants:

 Description   

There are three scenarios for an upsert. On the Primary all three will be reported as an update, but the actual oplog entry will not necessarily reflect this. When those oplog entries are replicated to Secondaries, the correct counters will be increased, but this makes the counts inconsistent with those on the Primary.

The three scenarios are as follows

  1. The document with a given key does not exist. Insert occurs on the Primary and an "i" record written to oplog. This is still recorded in serverStatus().opcounters as an update, and in db.serverStatus().metrics.document as an update
  2. The document with a given key exists and the new document is different from the old one. The old document is replaced with the new one on the Primary and a "u' record written to oplog. Correctly recorded as an update in serverStatus().opcounters and db.serverStatus().metrics.document
  3. The document with a given key exists and the new document is identical to the old one. No changes on the Primary and no record goes to oplog. This is still recorded as an updated in serverStatus().opcounters, but correctly not recorded in db.serverStatus().metrics.document


 Comments   
Comment by Andre de Frere [ 16/Feb/15 ]

I think that splitting the metrics would be good

Comment by Asya Kamsky [ 03/Feb/15 ]

I think maybe it's more reasonable to split the metrics into "client requested" and "server executed".

Client request 2 inserts, 5 updates and 2 deletes
Server executed 4 inserts, 1 update, and 1 delete (or some such).
We already count documents - so add to the above document metrics - 4 inserted, 5 updated (maybe it was multi) and 234 deleted.

Of course then there are system triggered inserts, deletes, etc...

Comment by Andre de Frere [ 03/Feb/15 ]

what is the underlying reason you want "accounting for upserts consistent" across replicas?

I think that the original request to make accounting for upserts to be consistent is to remove the uncertainty that comes from reading opcounters or metrics on the Primary, seeing the "update" counters and needing to differentiate between those numbers being updates or inserts (depending on how they are called). FWIW, that distinction does not need to be made on the Secondaries. While this distinction exists, there is no metric that you can rely on to tell you the number of updates, and conversely the number of inserts, that are actually occurring on a Primary node.

Also, why are the "opcounter" metrics the ones which must be changed – could other values be used to achieve the goals?

Certainly, other values could be used. It does not need to be opcounters.

So you want the opcounter to record only what type the oplog entry is?

Ignoring the oplog for one second, I think the idea is that an update that causes an insert to occur should be counted as an insert rather than an update. I think the difference is recording what underlying action the server actually took, rather than what the client issued. Given that the two operations could have different performance profiles, operationally speaking should the numbers that we gather agree with the actual operation that is occurring on the server, or what is being issued at the client?

Now when including the OpLog, we can tell that the Primary records the op as an insert in the oplog, while the metrics (and opcounters) on the Primary have the op counted as an update (I've used this as shorthand for the actual underlying operation that occurs on the Primary in the description of the ticket). I understand that this comparison fails in the case where there are multiple documents being updated (since this will be recorded once in the opcounters, but will be recorded correctly in metrics) but I don't think you can generate multiple new documents from a single upsert.

For example, this update is invalid but will be counted as an update operation the server receives, and produces a error for the client with no oplog entry produced:

Ideally those would not be counted, but I think this is a secondary concern as (I don't believe) this is measured in metrics (only in opcounters).

I will update the description of the ticket to explicitly call out metrics.

Comment by Scott Hernandez (Inactive) [ 02/Feb/15 ]

So you want the opcounter to record only what type the oplog entry is? And if there is no oplog entry created, due to a failure of the update, no documents found by the query or a noop for more than one of the documents, then it should still be an "update"?

For example, this update is invalid but will be counted as an update operation the server receives, and produces a error for the client with no oplog entry produced:

db.a.update({}, {$set : {a : {"b.c": 1}}}, {multi:true, upsert:true}) // Cannot store a field named "b.c"

Also, as you know, an update which effects multiple documents is still only countered once in the "opcounters", since it was just one operation issued from the client, but it may produce many updates to the actual collection and oplog entries. I'm not sure how this figures into your improvement request, but you might want to keep it in mind as an important detail. Now, as I've mentioned above, the "document" metrics will include the number of document operations done in the user's collections.

As you can tell from my questions I'm still unclear exactly what the improvement is that you are requesting and how it would work. Maybe we can come at this from a different angle, what is the underlying reason you want "accounting for upserts consistent" across replicas? Also, why are the "opcounter" metrics the ones which must be changed – could other values be used to achieve the goals?

Comment by Andre de Frere [ 02/Feb/15 ]

I think the issue with updates/inserts to the local.slaves collection will go away with SERVER-13248. While there could still be updates to the local database (or indeed other "housekeeping" operations) that should not be counted, I think that the topic of this ticket is perhaps a little more nuanced.

Currently, we increment the "update" counter of opcounters when there is an update of any type - for example even if the update actually means that an insert operation occurs (and is therefore an insert operation as per the entry in the oplog). I realise that this is due to us counting the "update" operation from the client, rather than what the actual operation on the server ends up being. The question is philosophical - if an "update" from the client really means an insert on the server, should we increment the update opcounter, or the insert opcounter? I'm only using the oplog entry as a source of truth here, under the assumption that the oplog entry indicating that the op is an insert actually means the server itself performed an insert.

This may be true for insert and delete as well, but I haven't come across that case.

Comment by Scott Hernandez (Inactive) [ 30/Jan/15 ]

So the real issue here is that we do operations in the background/on-behalf-of-the-system on local.slaves/slavetracking and the oplog (that are not directly user-initiated), and you don't want those reflected in the serverStatus opcounter/metrics.document stats? Seems like this really has nothing to do with update + upsert, since insert and delete can result in different counts for opcounter/metrics.document stats as well.

Could you try to refine the title and description to more accurately convey what the problem and improved behavior could be?

Comment by Andre de Frere [ 30/Jan/15 ]

I don't think that the issue is that we are not recording an upsert, I think that the problem is we are recording everything as an update. I see that the difficulty is that the client is issuing an update (so that is what we record), but the OpLog entries do give different values. The problem is the same in either serverStatus().opcounters and serverStatus().metrics.document:

replset:PRIMARY> db.serverStatus().metrics.document
{
	"deleted" : NumberLong(0),
	"inserted" : NumberLong(1),
	"returned" : NumberLong(6),
	"updated" : NumberLong(2)
}
replset:PRIMARY> db.test.update( { x : 3 }, { x: 2 }, { upsert : true } ) // upserting a document, does not exist
WriteResult({
	"nMatched" : 0,
	"nUpserted" : 1,
	"nModified" : 0,
	"_id" : ObjectId("54cb0e220b61eac08ad58c6a")
})
replset:PRIMARY> db.serverStatus().metrics.document
{
	"deleted" : NumberLong(0),
	"inserted" : NumberLong(1),
	"returned" : NumberLong(8),
	"updated" : NumberLong(5) //updated counter increases by 3 (also counts the updates to local.slaves : SERVER-13248)
}
replset:PRIMARY> db.getSiblingDB("local").oplog.rs.find().sort( { ts : -1 } ).limit(1)
{ "ts" : Timestamp(1422593570, 1), "h" : NumberLong("-6381369349631795657"), "v" : 2, "op" : "i", "ns" : "test.test", "o" : { "_id" : ObjectId("54cb0e220b61eac08ad58c6a"), "x" : 2 } } // recorded in the oplog as an insert (and therefore on Secondaries as an insert)

Comment by Scott Hernandez (Inactive) [ 30/Jan/15 ]

There is no "upsert" operation, just an option to update We have insert, update and delete operations issued from the clients and that is what the opcounters record – operations issued from the client. There are separate document/storage related metrics which I think is what you are really interested in:

"document" : {
	"deleted" : NumberLong(0),
	"inserted" : NumberLong(1),
	"returned" : NumberLong(0),
	"updated" : NumberLong(0)
},

I'm not sure that matching opcounters on the primary and secondary is ideal as they are separate things, and an indication of work being sent to server not what is actually done. It is arguable that all of these stats need better definition, and replacement but changing their meaning seems like a questionable choice due to compatibility and existing meaning.

Comment by Andre de Frere [ 30/Jan/15 ]

If the accounting on the Primaries matched the Secondaries that would be ideal. I think that there is a small problem in that the operations on the Secondaries are actually what is being reported in the OpLog (since that is their source), while the Primaries write the op to the counter and then write the OpLog entry (which is then reported differently as in the description of the ticket).

Comment by Andy Schwerin [ 30/Jan/15 ]

Seems to me that the secondaries have the more useful counters. If the primary used the accounting currently used on secondaries, would that be an acceptable outcome, andre@mongodb.com?

Generated at Thu Feb 08 03:43:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.