[SERVER-35896] Support "replaceDocuments" mode in $out Created: 28/Jun/18  Updated: 29/Oct/23  Resolved: 06/Aug/18

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: 4.1.2

Type: Task Priority: Major - P3
Reporter: Kyle Suarez Assignee: Nicholas Zolnierz
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-35893 Update $out to accept new syntax Closed
is depended on by SERVER-36100 Allow missing "_id" from extracted un... Closed
is depended on by SERVER-36318 Add ability to extract dotted path un... Closed
is depended on by SERVER-36376 Don't extract uniqueKey from document... Closed
is depended on by SERVER-12280 allow $out to append to/merge with ex... Closed
Documented
is documented by DOCS-11948 Docs for SERVER-35896: Support "repla... Closed
Problem/Incident
causes SERVER-36598 aggregate $out fails when the _id is ... Closed
Related
related to SERVER-35895 Add ability for $out to write to remo... Closed
is related to SERVER-36047 (Sharded target collection) Enforce e... Closed
Backwards Compatibility: Fully Compatible
Sprint: Query 2018-07-16, Query 2018-07-30, Query 2018-08-13
Participants:

 Description   

If an existing document already exists with the same unique key, the old document is replaced with the new one.



 Comments   
Comment by Githook User [ 02/Aug/18 ]

Author:

{'username': 'nzolnierzmdb', 'name': 'Nick Zolnierz', 'email': 'nicholas.zolnierz@mongodb.com'}

Message: SERVER-35896: Support 'replaceDocuments' mode in $out
Branch: master
https://github.com/mongodb/mongo/commit/afbe688f0f18c5cb474fb1bcd933d6e06c0c5291

Comment by Kyle Suarez [ 10/Jul/18 ]

I filed SERVER-36047 to track the enforcement of a unique index.

Comment by Nicholas Zolnierz [ 10/Jul/18 ]

Makes sense to me, though I'd prefer to keep the uniqueKey work separate from this ticket. Also kyle.suarez, one other subtlety that we discussed in addition to what you mentioned above is allowing a missing _id in the pipeline document since the update will autogenerate one.

Comment by Kyle Suarez [ 10/Jul/18 ]

We might be able to do the same trick that we do for existing $out, where before we finish up we assert that the same indices and options still exist, thereby checking that it's still unique.

Yeah, we definitely want to keep that with dropTarget: true. When dropTarget: false, I think it's sufficient to check once at aggregation planning time, and not before every call to spill().

Comment by Charlie Swanson [ 10/Jul/18 ]

Oh, yea this totally makes sense. I forgot in the scope, we are allowed to require the 'uniqueKey' be provably unique via a unique index. We might be able to do the same trick that we do for existing $out, where before we finish up we assert that the same indices and options still exist, thereby checking that it's still unique.

Comment by Kyle Suarez [ 10/Jul/18 ]

Per in-person discussion, we think a reasonable approach is:

  • upsert: true
  • require that there exists a unique index containing the fields of the uniqueKey
    • If the user drops the unique index during the aggregation, they're on their own
  • multi: false
    • Assume that at most one document will be updated

I think it would be useful to push forward with this to have something working and see if the semantics are what we want. I'll also schedule time to discuss this more broadly with the team/product.

Comment by Kyle Suarez [ 28/Jun/18 ]

Let's use the update system to do this; ideally with upsert: "true".

Generated at Thu Feb 08 04:41:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.