[SERVER-17478] Remove or modify copyTo() shell helper Created: 05/Mar/15 Updated: 06/Dec/22 Resolved: 19/Oct/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Shell |
| Affects Version/s: | None |
| Fix Version/s: | 4.1.5 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | J Rassi | Assignee: | Backlog - Security Team |
| Resolution: | Done | Votes: | 0 |
| Labels: | platforms-re-triaged | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||
| Assigned Teams: |
Server Security
|
||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Major Change | ||||||||||||||||||||||||||||||||||||
| Sprint: | Security 2018-09-10 | ||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||
| Description |
|
To remove the dependency of the copyTo() shell helper on the eval command, we will be switching to using aggregation+$out to simulate the existing behavior. |
| Comments |
| Comment by Githook User [ 19/Oct/18 ] | |||||||||||||
|
Author: {'name': 'Sara Golemon', 'email': 'sara.golemon@mongodb.com', 'username': 'sgolemon'}Message: | |||||||||||||
| Comment by Gregory McKeon (Inactive) [ 23/Jul/18 ] | |||||||||||||
|
sara.golemon sending to you | |||||||||||||
| Comment by Ian Whalen (Inactive) [ 13/Jul/18 ] | |||||||||||||
|
Discussed at Query triage and agreed that we do not plan to modify $out. Better option is definitely just to remove copyTo. asya confirmed this removal is safe from a Product perspective. Passing back to Platforms to do the removal. | |||||||||||||
| Comment by Asya Kamsky [ 10/Jul/18 ] | |||||||||||||
|
I just revisted copyTo helper (including here for reference) it actually locks the database, and also it returns the number of records that were copied into/onto the target collection:
Aggregation with $out stage cannot emulate that. We can print count of number of records in source collection - that's always equal to number of records copied, but unlike db.eval we won't be locking out every other process which is the other thing that copyTo helper does. | |||||||||||||
| Comment by Gregory McKeon (Inactive) [ 09/Jul/18 ] | |||||||||||||
|
Sending to Query to consider modifying $out to act like copyTo. | |||||||||||||
| Comment by Sara Golemon [ 02/Jul/18 ] | |||||||||||||
|
If we can make aggregate $out behave like the current copyTo helper, then yes, I agree we should make use of it. The issue of supporting older MongoDB instances from newer shells could be resolved with wire protocol version or possibly by trying agg-out, then upon failure, trying eval. Neither version sounds incredibly arduous. | |||||||||||||
| Comment by Asya Kamsky [ 02/Jul/18 ] | |||||||||||||
|
Transactions don't support DDL (currently) so 3. shouldn't be an issue. $out does not propagate indexes from source collection, it preserves the indexes that existing target collection has when it's going to replace it. We are currently enhancing $out to allow appending to existing collection. I believe there may have been a disconnect and the helper for 4.2 should use $out in mode that's equivalent to old copyTo - adding documents to existing collection. So I agree that pre-4.2 we should not have changed copyTo helper, but we can change it for 4.2. Given the complexity of handling newer shell being used with older server, I guess this could get a bit more complex than anticipated? | |||||||||||||
| Comment by Sara Golemon [ 02/Jul/18 ] | |||||||||||||
|
Correct. collection.copyTo() (when called on MongoDB 4.2 or later) will now result in the current deprecation warning issues by the shell about db.eval(): "WARNING: db.eval is deprecated" followed by a CommandNotFound error returned by the server. Calling it against earlier MongoDB instances will obviously work as expected (including the deprecation warning which has been there for many versions). Differences between copyTo and aggregate-$out:
| |||||||||||||
| Comment by Asya Kamsky [ 02/Jul/18 ] | |||||||||||||
|
sara.golemon copyTo uses db.eval which is now removed. Is copyTo going to give an error now? Also, what’s the $out and copyTo behavior that concerned you when considering making the change? | |||||||||||||
| Comment by Sara Golemon [ 30/Jun/18 ] | |||||||||||||
|
spencer.jackson and I decided it'd be best to leave the copyTo helper alone since the semantics of an aggregation $out are subtly different from what copyTo does, and it should continue to work when connecting to older MongoDB instances. | |||||||||||||
| Comment by Gregory McKeon (Inactive) [ 28/Jun/18 ] | |||||||||||||
|
sara.golemon I've updated this ticket and assigned to you to track that work. | |||||||||||||
| Comment by Sara Golemon [ 28/Jun/18 ] | |||||||||||||
|
greg.mckeon FYI; As part of | |||||||||||||
| Comment by Gregory McKeon (Inactive) [ 06/Mar/18 ] | |||||||||||||
|
If we're removing db.eval, we also have to remove this. | |||||||||||||
| Comment by Asya Kamsky [ 23/Mar/17 ] | |||||||||||||
|
Is there a reason not to change the helper to use aggregation command with {$out:"newName"} stage? |