[CDRIVER-838] Support find, getMore and killCursor commands Created: 11/Sep/15  Updated: 16/Feb/16  Resolved: 16/Nov/15

Status: Closed
Project: C Driver
Component/s: None
Affects Version/s: None
Fix Version/s: 1.3.0-beta0

Type: New Feature Priority: Major - P3
Reporter: A. Jesse Jiryu Davis Assignee: A. Jesse Jiryu Davis
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on CDRIVER-906 Separate command and query code Closed
is depended on by CDRIVER-837 Add support for the readConcern option Closed
is depended on by CXX-672 Support new commands for find, getMor... Closed
is depended on by PHPC-429 Support new commands for find, getMor... Closed
is depended on by DRIVERS-211 New commands for find, getMore, and k... Closed
is depended on by PHPC-482 Update cursor iteration for libmongoc... Closed
is depended on by CDRIVER-988 Test limit and batchSize with mock se... Closed
Duplicate
is duplicated by CDRIVER-705 Use new getMore server command to rea... Closed
Related
related to CDRIVER-1079 Close socket after timeout from taila... Closed
related to CDRIVER-1023 Support reading 16mb using the new fi... Closed
related to CDRIVER-1064 find / getMore commands' batchSize mu... Closed
related to CDRIVER-705 Use new getMore server command to rea... Closed
is related to CDRIVER-987 Link to server "find" command manual ... Closed
Epic Link: MongoDB 3.2

 Comments   
Comment by Githook User [ 16/Feb/16 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-1117 delete mongoc_cluster_run_command_rpc

I introduced the function for CDRIVER-838 to implement "find"/"getMore"
commands – the cursor was built to send and receive RPCs, so it was
easier to stick a find or getMore command into an RPC and send it,
instead of using the standard command path from the cursor code.

This causes some duplication, however, and it makes it very hard to
consistently record command events originating from cursors and those
that don't (CDRIVER-769).

Remove mongoc_cluster_run_command_rpc and use the standard command path.
When a cursor runs a command, it no longer stores the reply in its rpc
struct; instead it stores the reply in a bson_t on its iface_data.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/dd115e8e2c22eb68dc1b47bbc57740520ee42244

Comment by Githook User [ 16/Nov/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-838 simplify killCursors test
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/ea68fc78b7a15206717176a8ee3ae8c61f2129c6

Comment by Githook User [ 16/Nov/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-838 killCursors: "collection", not full ns
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/fcb6e62be031cbb1e1b9a58245933c46372e04d9

Comment by Githook User [ 16/Nov/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-838 test killCursors cmd with mock server
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/5d1bd94570b307068ecc05eb8e5b75761a3ac8a2

Comment by Githook User [ 16/Nov/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-838 send killCursors cmd to MongoDB 3.2+
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/7d8679314f5ee7540a05a4c9e2a950ed9176ad69

Comment by Githook User [ 16/Nov/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-838 document MongoDB 3.2 query syntax
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/e643b54c199eddcfcc7cff190f501e2be63bc9a8

Comment by Githook User [ 11/Nov/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-838 use getMore cmd with MongoDB 3.2+
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/af4ae26c26aeb5ba6b15686c070e8cd299eb6bf8

Comment by Githook User [ 11/Nov/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-838 batchSize is int32, not int64

Also test find command with batchSize.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/3a0e53b021103e3e6811bf18405981a6322c949d

Comment by Githook User [ 11/Nov/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-838 factor out OP_GETMORE code
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/0337b9fb9392a9673bb054cda7f44e0dceb2d9c4

Comment by Githook User [ 10/Nov/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-838 remove field cursor->failed.

Redundant with cursor->error.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/a3d0fe2fab7abfb1bd343ab51fd254f09b6af083

Comment by Githook User [ 10/Nov/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-838 use "find" command with mongodb 3.2+
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/254938e2a3523f4f1e9a76509a5de95a91519f20

Comment by Githook User [ 05/Nov/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-838 fix aggregation cursor iteration
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/9810666cb1c21b6d471c67c07e632be2c82d9fab

Comment by Githook User [ 05/Nov/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-838 don't set aggregate cursor's is_command
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/8f0d9265fbfa4c1f955a88ddf0c63dc91ef5f1ab

Comment by A. Jesse Jiryu Davis [ 04/Nov/15 ]

You're right about $showRecordId, I'll update my comment. And yes, I'll need to deal with flags. Also skip, limit, batch size.

Comment by Githook User [ 03/Nov/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-838 refactor command cursors

Preparation for implementing the "find" command.

Cursors from commands (aggregate, listCollections, listDatabases) had
been implemented as regular cursors with a "query" that is the command.
mongoc_cursor_cursorid_prime() runs the query with _mongoc_cursor_next,
then transforms the cursor into a command cursor.

This will get very tangled if "find" is a command, so I've created a
specialized function _mongoc_cursor_run_command that cursors use to
run aggregate, listCollections, and listDatabases. This function calls
a new function mongoc_cluster_run_command_rpc which fills out the
cursor's buffer with the command reply. Command cursors then proceed
as before: iterate the initial batch from the command reply, then send
OP_GETMORE.

The main advantage of this refactoring is that _mongoc_cursor_next is
now purely for regular query cursors.

The next step will be to call the "find" command with this new function
_mongoc_cursor_run_command.

A subtle behavior change: the driver used to send the initial aggregate
command lazily from mongoc_cursor_next if connected to a legacy server,
but sent the command eagerly from mongoc_collection_aggregate when
connected to a modern server. Now, mongoc_collection_aggregate never
sends the command eagerly. The "aggregate" command is always sent from
mongoc_cursor_next.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/bb42cec93feda9463e47bc7c3e0ec1d7c39ce7f3

Comment by Hannes Magnusson [ 03/Nov/15 ]

I don't think we need to support

            if key == '$showRecordId': cmd['showRecordId'] = value

since that is a MongoDB 3.2 (e.g. find() command) feature, so legacy query documents wouldn't have it.

The mongoc_query_flags_t flags will have to be rolled out too.. Does that need to happen irregardless $query was found in the query document?

Comment by A. Jesse Jiryu Davis [ 02/Nov/15 ]

Quick spec on how to upgrade from old-style query documents like this:

{ $query: { key: value }, $orderby: { key: value } }

to this:

{ find: collection, filter: { key: value }, sort: { key: value } }

Of course that's not an exhaustive list of the transformations.

mongoc_collection_find doesn't take any separate arguments for "query" or "sort", much less for rarer options like maxTimeMS.

The driver should upgrade from old- to new-style with logic like:

if max wire version >= 4:
    cmd = { find: collection }
    if "$query" in cursor->query:
        for key, value in cursor->query:
            if key == '$query': cmd['filter'] = value
            if key == '$orderby': cmd['sort'] = value
            if key == '$hint': cmd['hint'] = value
            if key == '$comment': cmd['comment'] = value
            if key == '$maxScan': cmd['maxScan'] = value
            if key == '$maxTimeMS': cmd['maxTimeMS'] = value
            if key == '$max': cmd['max'] = value
            if key == '$min': cmd['min'] = value
            if key == '$returnKey': cmd['returnKey'] = value
            if key == '$showDiskLoc': cmd['showRecordId'] = value  # <= MongoDb 3.0
            if key == '$snapshot': cmd['snapshot'] = value
    else:
        append "{filter": cursor->query} to cmd
    
    if flags & tailable: cmd['tailable'] = true
    if flags & oplogreplay: cmd['oplogReplay'] = true
    if flags & nocursortimeout: cmd['noCursorTimeout'] = true
    if flags & awaitdata: cmd['awaitData'] = true
    if flags & partial: cmd['allowPartialResults'] = true
 
    if limit: cmd['limit'] = limit
    if batchsize: cmd['batchSize'] = batchsize
    if skip: cmd['skip'] = skip
 
    cursor_run_command (cmd)
else:
    cursor_send_op_query (cursor->query)

This logic is adapted from PyMongo's:

https://github.com/mongodb/mongo-python-driver/blob/28885b25e85940bf4d9c1e87638a861fe5a89b71/pymongo/message.py#L160-L160

We're not going to upgrade the "explain" command.

This logic provides a clear upgrade path: old code still works in the great majority of common cases. Future code that knows it's talking to MongoDB 3.2+ can just send new-style find-command options to mongoc_collection_find. No ABI changes are required.

Not yet shown : readConcern, that'll be another ticket.

Generated at Wed Feb 07 21:10:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.