[CDRIVER-837] Add support for the readConcern option Created: 11/Sep/15  Updated: 03/Dec/15  Resolved: 18/Nov/15

Status: Closed
Project: C Driver
Component/s: libmongoc
Affects Version/s: None
Fix Version/s: 1.3.0-beta0

Type: New Feature Priority: Major - P3
Reporter: A. Jesse Jiryu Davis Assignee: Hannes Magnusson
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on CDRIVER-838 Support find, getMore and killCursor ... Closed
depends on CDRIVER-1006 Document readConcernLevel MongoDB URI... Closed
depends on CDRIVER-1007 Document new read_concern functions Closed
is depended on by CXX-695 Support "read committed" isolation le... Closed
is depended on by DRIVERS-254 Add support for the readConcern option. Closed
Related
related to CDRIVER-1031 ReadConcern should be inherited from ... Closed
related to CDRIVER-1010 Document helpers that use readConcern Closed
Epic Link: MongoDB 3.2

 Description   

https://github.com/mongodb/specifications/blob/master/source/read-write-concern/read-write-concern.rst#location-specification

ReadConcern SHOULD be specifiable at the Client, Database, and Collection levels. Unless specified, the value MUST be inherited from its parent and SHOULD NOT be modifiable on an existing Client, Database, and Collection. In addition, a driver MAY allow it to be specified on a per-operation basis in accordance with the CRUD specification.

mongoc will support setting readConcern on:

  • mongoc_collection_t
  • mongoc_database_t
  • mongoc_client_t

mongoc will not support setting readConcern on per-operation basis, and will therefore not be adding extended versions of helpers that could otherwise take readConcern.

The following helpers (that support readConcern in MongoDB 3.2+) are currently implemented in mongoc:

  • mongoc_collection_count
  • mongoc_collection_find
  • mongoc_collection_aggregate

When using these helpers, readConcern must be set on the collection level, e.g.:

mongoc_collection_set_read_concern (collection, read_concern);
mongoc_collection_find (collection, MONGOC_QUERY_NONE, 0, 0, 0, NULL, NULL, read_prefs);

Other currently known commands that support readConcern, but don't have helpers in mongoc are:

  • mapReduce when the result is inline
  • distinct
  • parallelCollectionScan
  • geoNear
  • geoSearch

The following are new (public API) functions handling read_concern:

#define MONGOC_READ_CONCERN_LEVEL_LOCAL    "local"
#define MONGOC_READ_CONCERN_LEVEL_MAJORITY "majority"
 
typedef struct _mongoc_read_concern_t mongoc_read_concern_t;

mongoc_read_concern_t        *mongoc_read_concern_new            (void);
mongoc_read_concern_t        *mongoc_read_concern_copy           (const mongoc_read_concern_t  *read_concern);
void                          mongoc_read_concern_destroy        (mongoc_read_concern_t        *read_concern);
const char                   *mongoc_read_concern_get_level      (const mongoc_read_concern_t  *read_concern);
bool                          mongoc_read_concern_set_level      (mongoc_read_concern_t        *read_concern, const char *level);

const mongoc_read_concern_t  *mongoc_client_get_read_concern     (const mongoc_client_t        *client);
void                          mongoc_client_set_read_concern     (mongoc_client_t              *client,       const mongoc_read_concern_t *read_concern);

const mongoc_read_concern_t  *mongoc_database_get_read_concern   (const mongoc_database_t      *database);
void                          mongoc_database_set_read_concern   (mongoc_database_t            *database,     const mongoc_read_concern_t *read_concern);

const mongoc_read_concern_t  *mongoc_collection_get_read_concern (const mongoc_collection_t    *collection);
void                          mongoc_collection_set_read_concern (mongoc_collection_t          *collection,   const mongoc_read_concern_t *read_concern);

const mongoc_read_concern_t  *mongoc_uri_get_read_concern        (const mongoc_uri_t           *uri);


Note: To use this feature in MongoDB 3.2 you must start mongod with --enableMajorityReadConcern

Note that the comments on this ticket assumed that we would have to support readConcern as an option to the find helper, and therefore requiring a new extended version of that helper.
This is however not correct and was based on misunderstanding of the specification.



 Comments   
Comment by Githook User [ 18/Nov/15 ]

Author:

{u'username': u'bjori', u'name': u'Hannes Magnusson', u'email': u'bjori@php.net'}

Message: CDRIVER-837: Add the new symbols here too!
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/80b9d97128e6f84952e163b449c2417644419138

Comment by Githook User [ 18/Nov/15 ]

Author:

{u'username': u'bjori', u'name': u'Hannes Magnusson', u'email': u'bjori@php.net'}

Message: CDRIVER-837: add readConcern support for aggregate
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/15fd0799697916fbcd5aaf9249a91fbe1c9fb979

Comment by Githook User [ 18/Nov/15 ]

Author:

{u'username': u'bjori', u'name': u'Hannes Magnusson', u'email': u'bjori@php.net'}

Message: CDRIVER-837: Add test for mongoc_collection_find and readConcern
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/67c1691b3a7a56ff267effdc3eab3613b5f8633f

Comment by Githook User [ 18/Nov/15 ]

Author:

{u'username': u'bjori', u'name': u'Hannes Magnusson', u'email': u'bjori@php.net'}

Message: CDRIVER-837: apply readConcern to mongoc_collection_find() calls
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/578b378989a90d905ffaecab45400f5566fbe958

Comment by Githook User [ 18/Nov/15 ]

Author:

{u'username': u'bjori', u'name': u'Hannes Magnusson', u'email': u'bjori@php.net'}

Message: CDRIVER-837: Export new read_concern related symbols
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/1d740a86d0c0b495ba5095f727c07b4c51a440ba

Comment by Githook User [ 18/Nov/15 ]

Author:

{u'username': u'bjori', u'name': u'Hannes Magnusson', u'email': u'bjori@php.net'}

Message: CDRIVER-837: Add support for readConcern in mongoc_collection_count
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/0266789945f73019a31b273600fd911b4522f86f

Comment by Githook User [ 18/Nov/15 ]

Author:

{u'username': u'bjori', u'name': u'Hannes Magnusson', u'email': u'bjori@php.net'}

Message: CDRIVER-837: Define WIRE_VERSION_READ_CONCERN
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/001a305012a441a99233977d2c2f712acdb7d90a

Comment by Githook User [ 18/Nov/15 ]

Author:

{u'username': u'bjori', u'name': u'Hannes Magnusson', u'email': u'bjori@php.net'}

Message: CDRIVER-837: Generate mongoc_collection_find futures
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/ec1c7d77867ecbd39e4fc9376fba25fa09230c01

Comment by Githook User [ 18/Nov/15 ]

Author:

{u'username': u'bjori', u'name': u'Hannes Magnusson', u'email': u'bjori@php.net'}

Message: CDRIVER-837: Parse readConcernLevel from the connection string
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/49f956d9cad60a0e5529b16bf590248243ffceca

Comment by Githook User [ 18/Nov/15 ]

Author:

{u'username': u'bjori', u'name': u'Hannes Magnusson', u'email': u'bjori@php.net'}

Message: CDRIVER-837: Weave mongoc_read_concern_t through everywhere
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/e12e3bdb2b99509230f0a7a7b93f7e575b00aba6

Comment by Githook User [ 18/Nov/15 ]

Author:

{u'username': u'bjori', u'name': u'Hannes Magnusson', u'email': u'bjori@php.net'}

Message: CDRIVER-837: Add mongoc_read_concern_t
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/2c90394d9e9e41635d3e7ea2239ffd828e8cb501

Comment by A. Jesse Jiryu Davis [ 17/Nov/15 ]

Yeah, the new API looks great to me.

Comment by Matt Cotter [ 17/Nov/15 ]

Looks great, thanks!

Comment by Hannes Magnusson [ 17/Nov/15 ]

More or less identical to mongoc_write_concern_t(), with mongoc_read_concern_new() and mongoc_read_concern_set_level(rc, char *level) and mongoc_read_concern_destroy() and coupe of defines for the default levels: MONGOC_READ_CONCERN_LEVEL_MAJORITY and MONGOC_READ_CONCERN_LEVEL_LOCAL.
Ill share the exact protos shortly

Comment by Matt Cotter [ 17/Nov/15 ]

Do you know what mongoc_read_concern_t will look like? Will there be a mongoc_read_concern_new() and mongoc_read_concern_destroy()?

Comment by A. Jesse Jiryu Davis [ 12/Nov/15 ]

In other words I agree with you completely.

Comment by A. Jesse Jiryu Davis [ 12/Nov/15 ]

No, applications should be able to call mongoc_collection_find or mongoc_collection_find_with_opts correctly without knowing the server version.

There is a caveat if you actually issue a non-"local" readConcern:

https://github.com/mongodb/specifications/blob/master/source/read-write-concern/read-write-concern.rst#errors-1

Other than that, this API will work equally for OP_QUERY and "find" commands.

Comment by Jeremy Mikola [ 12/Nov/15 ]

Yesterday, bjori mentioned that this new find API is only intended to create find commands. The older BSON-based API (query document as a bson_t) will need to be used for servers before 3.2, although there will be up-convert logic, which I believe will not support things like readConcern. If I understand correctly, this will require an application or driver to issue version checks before deciding the API to use in a mixed-version cluster. I'd strongly suggest having the new API down-convert where possible and raise errors in the event that a user attempts to use 3.2+ features (e.g. readConcern) but the driver decides it must down-convert to the legacy API.

Comment by A. Jesse Jiryu Davis [ 12/Nov/15 ]

About "query" and "filter": Your proposed mongoc_collection_find_with_opts takes a "query" parameter, and there's a "filter" field in mongoc_find_opts_t. Let's get rid of "filter" but keep "query".

About "max_await_time_ms": Yes, that's right. We want a separate option in mongoc_find_opts_t called max_await_time_ms, we'll use it as the "maxTimeMS" argument to the "getMore" command if this is a tailable cursor with awaitData. Find, getMore And killCursors Commands Spec:

In the case of a tailable cursor with awaitData == true the driver MUST provide a Cursor level option named maxAwaitTimeMS (See CRUD specification for details). The maxTimeMS option on the getMore command MUST be set to the value of the option maxAwaitTimeMS. If no maxAwaitTimeMS is specified, the driver SHOULD not set maxTimeMS on the getMore command.

Comment by Hannes Magnusson [ 12/Nov/15 ]
  • Let's rename _mongoc_find_opts_t's batchSize to batch_size.

Copypasta fixed

  • Let's change the names like mongoc_find_set_opts_sort to names like mongoc_find_opts_set_sort (swap "set" and "opts").

Err. I thought I was being consistent with find_and_modify. I was wrong.

  • "filter" conflicts with the "query" parameter for mongoc_collection_find_with_opts. We should remove one of them, and I think the one we should remove is "filter". The "query" parameter is very convenient, I want to keep it.

Not sure what you are talking about here?
We just call the "query" argument "filter" in the struct as its the name used by the command.

  • mongoc_collection_find_with_opts returns mongoc_cursor_t. Remove "reply".

Copypasta fixed

  • Can we change the bson_t pointers in _mongoc_find_opts_t to embedded bson_t's? That way there's never a NULL, only empty and non-empty. Fewer allocations, too.
  • Rename mongoc_find_set_opts_maxScan to mongoc_find_opts_set_max_scan.

Copypasta fixed

  • Rename mongoc_find_set_opts_maxTimeMS to mongoc_find_opts_set_max_time_ms.

Copypasta fixed

  • Add a separate max_await_time_ms, it's distinct from max_time_ms.

There is no such option for the find command.. You want to set it there and not using until getmore?

Comment by A. Jesse Jiryu Davis [ 11/Nov/15 ]

Great!

  • Let's rename _mongoc_find_opts_t's batchSize to batch_size.
  • Let's change the names like mongoc_find_set_opts_sort to names like mongoc_find_opts_set_sort (swap "set" and "opts").
  • "filter" conflicts with the "query" parameter for mongoc_collection_find_with_opts. We should remove one of them, and I think the one we should remove is "filter". The "query" parameter is very convenient, I want to keep it.
  • mongoc_collection_find_with_opts returns mongoc_cursor_t. Remove "reply".
  • Can we change the bson_t pointers in _mongoc_find_opts_t to embedded bson_t's? That way there's never a NULL, only empty and non-empty. Fewer allocations, too.
  • Rename mongoc_find_set_opts_maxScan to mongoc_find_opts_set_max_scan.
  • Rename mongoc_find_set_opts_maxTimeMS to mongoc_find_opts_set_max_time_ms.
  • Add a separate max_await_time_ms, it's distinct from max_time_ms.
Comment by Hannes Magnusson [ 11/Nov/15 ]

"Expandable mongoc_find_flags_t for find boolean arguments"

typedef enum
{
   MONGOC_FIND_SINGLE_BATCH      = 1 << 0,
   MONGOC_FIND_RETURN_KEY        = 1 << 1,
   MONGOC_FIND_SHOW_RECORD_ID    = 1 << 2,
   MONGOC_FIND_SNAPSHOT          = 1 << 3,
   MONGOC_FIND_TAILABLE          = 1 << 4,
   MONGOC_FIND_OPLOG_REPLAY      = 1 << 5,
   MONGOC_FIND_NO_CURSOR_TIMEOUT = 1 << 6,
   MONGOC_FIND_AWAIT_DATA        = 1 << 7,
   MONGOC_FIND_ALLOW_PARTIAL_RESULTS  = 1 << 8
} mongoc_find_flags_t;

"Private mongoc_find_opts_t mapping find arguments"

struct _mongoc_find_opts_t
{
  bson_t                 filter;
  bson_t                 sort;
  bson_t                 projection;
  char                  *hint_index;
  bson_t                 hint_spec;
  int64_t                skip;
  int64_t                limit;
  int64_t                batch_size;
  char                  *comment;
  int32_t                max_scan;
  int32_t                max_time_ms;
  bson_t                 max;
  bson_t                 min;
  mongoc_find_flags_t    flags;
  mongoc_read_prefs_t   *read_prefs;
  mongoc_read_concern_t *read_concern;
};

typedef struct _mongoc_find_opts_t mongoc_find_opts_t;

"Creating mongoc_find_opts_t"

mongoc_find_opts_t* mongoc_find_opts_new (void);

"mongoc_find_ setters"

bool mongoc_find_opts_set_sort         (mongoc_find_opts_t *opts, const bson_t          *sort);
bool mongoc_find_opts_set_projection   (mongoc_find_opts_t *opts, const bson_t          *projection);
bool mongoc_find_opts_set_hint_index   (mongoc_find_opts_t *opts, const char            *hint_index);
bool mongoc_find_opts_set_hint_spec    (mongoc_find_opts_t *opts, const bson_t          *hint_spec);
bool mongoc_find_opts_set_skip         (mongoc_find_opts_t *opts, int64_t                skip);
bool mongoc_find_opts_set_limit        (mongoc_find_opts_t *opts, int64_t                limit);
bool mongoc_find_opts_set_batch_size   (mongoc_find_opts_t *opts, int64_t                batch_size);
bool mongoc_find_opts_set_comment      (mongoc_find_opts_t *opts, const char            *comment);
bool mongoc_find_opts_set_max_scan     (mongoc_find_opts_t *opts, int32_t                max_scan);
bool mongoc_find_opts_set_max_time_ms  (mongoc_find_opts_t *opts, int32_t                max_time_ms);
bool mongoc_find_opts_set_max          (mongoc_find_opts_t *opts, const bson_t          *max);
bool mongoc_find_opts_set_min          (mongoc_find_opts_t *opts, const bson_t          *min);
bool mongoc_find_opts_set_read_concern (mongoc_find_opts_t *opts, const mongoc_read_concern_t *read_concern);
bool mongoc_find_opts_set_read_prefs   (mongoc_find_opts_t *opts, const mongoc_read_prefs_t   *read_prefs);
bool mongoc_find_set_flags     (mongoc_find_opts_t *opts, mongoc_find_flags_t    flags, bool on_or_off);

"New find helper on mongoc_collection"

mongoc_cursor_t*
mongoc_collection_find_with_opts (mongoc_collection_t      *collection,
                                  const bson_t             *query,
                                  const mongoc_find_opts_t *opts_or_null,
                                  bson_error_t             *error);

"Couple of examples"

query = tmp_bson("{ 'score': { $gt: 100 } }");
mongoc_collection_find_with_opts (collection, query, NULL, &reply, &error);
 
 
mongoc_find_opts_t opts = mongoc_find_opts_new ();
mongoc_find_opts_set_skip (opts, 10);
mongoc_find_opts_set_comment (opts, "Track this query");
mongoc_find_set_flags (opts, MONGOC_FIND_RETURN_KEY|MONGOC_FIND_SNAPSHOT, true);
 
mongoc_collection_find_with_opts (collection, query, opts, &reply, &error);

Comment by A. Jesse Jiryu Davis [ 10/Nov/15 ]

bjori could you write down a proposed API here for setting readConcern? Including a new mongoc_collection_find_with_opts or whatever? Once we agree on the interface you can start implementing it from the top down while I approach from the bottom up.

Generated at Wed Feb 07 21:10:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.