[SERVER-5731] db.eval() for sharding collections Created: 01/May/12  Updated: 13/Mar/15  Resolved: 08/Mar/15

Status: Closed
Project: Core Server
Component/s: JavaScript, Sharding
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Azat Khuzhin Assignee: Unassigned
Resolution: Won't Fix Votes: 9
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-11076 we have an monogdb sharded cluster.we... Closed
Backwards Compatibility: Fully Compatible
Participants:

 Description   

I see http://www.mongodb.org/display/DOCS/Sharding+Limits#ShardingLimits-db.evalDOCS%3AServersideCodeExecution
But is it too complicated?

I can`t use MR for it



 Comments   
Comment by Ramon Fernandez Marina [ 13/Mar/15 ]

db.eval is deprecated in 3.0

Comment by Azat Khuzhin [ 14/Dec/12 ]

Hi,

When you run something in "db.eval()" this javascript will be executed on server.
And as Scott commented:
> There are many technical issues around using javascript in a distributed cluster, yes.

But when you run something in mongo shell this will be executed in that shell, as client javascript.
It's like you run this code from you application server.

I hope that this explains why you can't run some javascript in "db.eval()", that you can run using mongo shell.

Comment by Caleb Jones [ 13/Dec/12 ]

Ugh!! Bit by this today.

My use case: Need to change the shard key for a sharded collection.

My plan?

  1. Copy sharded collection to tmp non-sharded collection (DB can handle the size)
  2. migrate the shard key there (i need to do this in my application code since there's business logic I don't want to port to javascript)
  3. delete data from original sharded collection
  4. Copy from tmp non-sharded collection back into original sharded collection
  5. Drop tmp non-sharded collection

I wrote it all up only to hit this brick wall.

It seems very awkward that I can do the following in the mongo shell on a sharded collection (2.2.1):

mongos> db.getCollection("myShardedCollection").find().foreach(function(doc) { db.getCollection("newNonShardedCollection").insert(doc)  } )

But the following in client drivers (e.g. Java driver):

db.eval("db.getCollection(\"myShardedCollection\").find().forEach( function(doc) { db.getCollection(\"newNonShardedCollection\").insert(doc); })");

...results in the following:

Error: command failed [$eval]: { "serverUsed" : "host/<IP>:27017" , "errno" : -3.0 , "errmsg" : "invoke failed: JS Error: Error: can't use sharded collection from db.eval src/mongo/shell/db.js:30" , "ok" : 0.0}

My expectation is that if I can do something in the Mongo shell, that I should be able to do it in a client driver. Am I doing something wrong here?

Comment by Pavel Chertorogov [ 07/Jun/12 ]

I need analog of MySQL syntax "INSERT ... SELECT ..." (http://dev.mysql.com/doc/refman/5.0/en/insert-select.html) in Mongo with sharded collections.
This avoid bottleneck in network transfer of the huge amount data between server and application/code.

My use-case detailed described here https://groups.google.com/group/mongodb-user/browse_thread/thread/1d9da32836f0347e/52508dea8b7c0122

Comment by Azat Khuzhin [ 02/May/12 ]

I understand that it can be done using application code.
But if we are using JS for it, it will decrease IO between mongos, mongod and application.

BTW how db.eval() works with "abilitiy to distribute collections in a single db" #SERVER-939 ?

Comment by Scott Hernandez (Inactive) [ 02/May/12 ]

This is something best done by the application/code not via javascript. The only advantage javascript has is that it runs closer to the data but that can be done with any application running close to the data. In fact javascript may be slower for this than running similar code from another language. There are also many issues with using javascript like in terms of data type support.

Eval should mostly be used for administrative or one-off scripts where you need things done in the lock, which doesn't make sense in a sharded system so much.

I'll leave this open since many people will ask for it and might have better use-cases – which I'd guess will require other features.

Comment by Azat Khuzhin [ 02/May/12 ]

I have two collections, for example "foo" and "bar"
They have common field: "key"

And I need update some fields from "bar" to "foo" (using "key"), for sorting by it

Comment by Scott Hernandez (Inactive) [ 01/May/12 ]

What do you need db.eval() for on a sharded cluster?

There are many technical issues around using javascript in a distributed cluster, yes.

Generated at Thu Feb 08 03:09:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.