[SERVER-17478] Remove or modify copyTo() shell helper Created: 05/Mar/15  Updated: 06/Dec/22  Resolved: 19/Oct/18

Status: Closed
Project: Core Server
Component/s: Shell
Affects Version/s: None
Fix Version/s: 4.1.5

Type: Improvement Priority: Major - P3
Reporter: J Rassi Assignee: Backlog - Security Team
Resolution: Done Votes: 0
Labels: platforms-re-triaged
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-33662 Remove deprecated db.eval command Closed
Documented
is documented by DOCS-4929 follow Server-17478 about copyTo Closed
Duplicate
is duplicated by SERVER-7489 db.foo.copyTo() uses db.eval() which ... Closed
Related
related to SERVER-15571 Make a command to copy a collection l... Backlog
is related to SERVER-12742 remove collection.copyTo test shell h... Closed
Assigned Teams:
Server Security
Backwards Compatibility: Major Change
Sprint: Security 2018-09-10
Participants:

 Description   

To remove the dependency of the copyTo() shell helper on the eval command, we will be switching to using aggregation+$out to simulate the existing behavior.



 Comments   
Comment by Githook User [ 19/Oct/18 ]

Author:

{'name': 'Sara Golemon', 'email': 'sara.golemon@mongodb.com', 'username': 'sgolemon'}

Message: SERVER-17478 Remove coll.copyTo() shell helper
Branch: master
https://github.com/mongodb/mongo/commit/b6369bcb52320d8f73e0d654cbec952180dd269a

Comment by Gregory McKeon (Inactive) [ 23/Jul/18 ]

sara.golemon sending to you

Comment by Ian Whalen (Inactive) [ 13/Jul/18 ]

Discussed at Query triage and agreed that we do not plan to modify $out. Better option is definitely just to remove copyTo. asya confirmed this removal is safe from a Product perspective. Passing back to Platforms to do the removal.

Comment by Asya Kamsky [ 10/Jul/18 ]

I just revisted copyTo helper (including here for reference) it actually locks the database, and also it returns the number of records that were copied into/onto the target collection:

function (newName) {
    return this.getDB().eval(function(collName, newName) {
        var from = db[collName];
        var to = db[newName];
        to.ensureIndex({_id: 1});
        var count = 0;        var cursor = from.find();
        while (cursor.hasNext()) {
            var o = cursor.next();
            count++;
            to.save(o);
        }        return count;
    }, this.getName(), newName);
} 

Aggregation with $out stage cannot emulate that. We can print count of number of records in source collection - that's always equal to number of records copied, but unlike db.eval we won't be locking out every other process which is the other thing that copyTo helper does.

Comment by Gregory McKeon (Inactive) [ 09/Jul/18 ]

Sending to Query to consider modifying $out to act like copyTo.

Comment by Sara Golemon [ 02/Jul/18 ]

If we can make aggregate $out behave like the current copyTo helper, then yes, I agree we should make use of it.

The issue of supporting older MongoDB instances from newer shells could be resolved with wire protocol version or possibly by trying agg-out, then upon failure, trying eval.  Neither version sounds incredibly arduous.

Comment by Asya Kamsky [ 02/Jul/18 ]

Transactions don't support DDL (currently) so 3. shouldn't be an issue.

$out does not propagate indexes from source collection, it preserves the indexes that existing target collection has when it's going to replace it.
copyTo appends to existing collection so that's equivalent.

We are currently enhancing $out to allow appending to existing collection. I believe there may have been a disconnect and the helper for 4.2 should use $out in mode that's equivalent to old copyTo - adding documents to existing collection.

So I agree that pre-4.2 we should not have changed copyTo helper, but we can change it for 4.2. Given the complexity of handling newer shell being used with older server, I guess this could get a bit more complex than anticipated?

Comment by Sara Golemon [ 02/Jul/18 ]

Correct.  collection.copyTo() (when called on MongoDB 4.2 or later) will now result in the current deprecation warning issues by the shell about db.eval(): "WARNING: db.eval is deprecated" followed by a CommandNotFound error returned by the server.  Calling it against earlier MongoDB instances will obviously work as expected (including the deprecation warning which has been there for many versions).

Differences between copyTo and aggregate-$out:

  1. copyTo does NOT replace the target collection.  $out will replace it.
  2. $out propagates indexes from source collection (if applicable).
  3. $out and copyTo may have differing transaction semantics (we're honestly not sure).
Comment by Asya Kamsky [ 02/Jul/18 ]

sara.golemon copyTo uses db.eval which is now removed. Is copyTo going to give an error now?

Also, what’s the $out and copyTo behavior that concerned you when considering making the change?

Comment by Sara Golemon [ 30/Jun/18 ]

spencer.jackson and I decided it'd be best to leave the copyTo helper alone since the semantics of an aggregation $out are subtly different from what copyTo does, and it should continue to work when connecting to older MongoDB instances.

Comment by Gregory McKeon (Inactive) [ 28/Jun/18 ]

sara.golemon I've updated this ticket and assigned to you to track that work.

Comment by Sara Golemon [ 28/Jun/18 ]

greg.mckeon FYI; As part of SERVER-35847, the copyTo helper is going to switch to using aggregation+$out to simulate the existing behavior.

Comment by Gregory McKeon (Inactive) [ 06/Mar/18 ]

If we're removing db.eval, we also have to remove this.

Comment by Asya Kamsky [ 23/Mar/17 ]

Is there a reason not to change the helper to use aggregation command with {$out:"newName"} stage?

Generated at Thu Feb 08 03:44:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.