I'm using mongodb as a distributed, query-able job queue. When a worker queries the database for a payload, it usually runs a query something like

job = find({"$and":[_myfilter, {"locked_by": null}]}).limit(1).

_myfilter is usually business requirements, and the locked_by part is to assure that no job is taken by two workers at the same time.
After a worker get a potential job, it tries to lock it:

update_one({"_id": job["_id"]}, {"$set":{"locked_by": workerX}}

This works pretty well for our case, but whenever we have a big load, two things happen:

• the workers compete for the same job, they find it and they try to lock it (only one succeedes) - this can be mitigated by inserting a random field in the document, which is the queried. {"seed" = random(0, 256)}

  and query by {"seed": {"$gt": random(0, 256)}}.

• the sharded DB queries ALL the available shards (we now have 8); they all respond, but will only use one result 7/8 of the effort is wasted.

The collection is sharded by _id: "hashed"
I'm trying to mitigate this by forcing the first find query to target only one of the shards, chosed by random, and so on. This should reduce my workload, because the next update_one operation is targeted specifically on a shard. I'm seeing three solutions for this:

• reshard the database by seed, not by hashed _id
• figure out if i can force only results from a a shard on find
• create 8 more connections to the shards, and use them randomly on the first find, then use the mongos connection on the update.