[CDRIVER-2543] mongoc_collection_find_with_opts() does not work with multiple filters Created: 10/Mar/18  Updated: 27/Oct/23  Resolved: 12/Mar/18

Status: Closed
Project: C Driver
Component/s: None
Affects Version/s: 1.5.0, 1.6.0, 1.7.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Liviu Chircu Assignee: A. Jesse Jiryu Davis
Resolution: Works as Designed Votes: 0
Labels: filter
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

MongoDB 2.6 / MongoDB 3.4



 Description   

As soon as I moved to 1.7.0 and rewrote the code to use `mongoc_collection_find_with_opts()`, my filtering tests are starting to fail. Assuming we're working with this collection:

{_id: ..., "foo": "A"}
{_id: ..., "foo": "B"}
{_id: ..., "foo": "C"}
{_id: ..., "foo": "D"}

My pre-libmongoc-1.7.0 testing code used to construct the following filters (printed with `bson_as_json()` here) then pass them to `mongoc_collection_find()`:

`{ "$query" : { "opensips" : { "$lt" : "D" } } }`, gives 3 results. Correct.
`{ "$query" : { "$and" : [ { "foo" : { "$lt" : "D" } }, { "foo" : { "$gte" : "A" } } ] } }`, gives 3 results. Correct.

Now, I have updated the code to match the 1.7.0 API. Thanks to the new function, we can simplify the filters – here are the JSON views of the new filters, along with their results after passing them to the new `mongoc_collection_find_with_opts()`:

`{ "foo" : { "$lt" : "D" } }`, gives 3 results. Correct.
`{ "$and" : [ { "foo" : { "$lt" : "D" } }, { "foo" : { "$gte" : "A" } } ] }`, gives 0 results. Wrong!

Moreover, the 2nd query literally works if I hit it in the console, so either I'm doing something terribly wrong (forgetting some flag / API call / shouldn't be doing string comparisons anymore, etc.), or there is a good chance there is a bug in the 1.7.0 driver.

Any advice would be appreciated! Thanks!



 Comments   
Comment by A. Jesse Jiryu Davis [ 12/Mar/18 ]

No, there's been no change in the BSON-building interface, the functions you're using aren't deprecated. They would be marked deprecated in the documentation if they were.

Appending to an array requires numeric keys: "0", "1", "2", and so on. You can't append to an array with an empty key, that's what the "Invalid filter: empty key" error indicates. Try using numeric keys, and also make sure that you correctly specify the length of "$gte" as 4, not 3:

    bson_t and_bson, arr_item, subdoc;
    bson_append_array_begin(&query, "$and", 4, &and_bson);
 
    bson_append_document_begin(&and_bson, "0", 1, &arr_item);
    bson_append_document_begin(&arr_item, "foo", 3, &subdoc);
    bson_append_utf8(&subdoc, "$lt", 3, "D", 1);
    bson_append_document_end(&arr_item, &subdoc);
    bson_append_document_end(&and_bson, &arr_item);
 
    bson_append_document_begin(&and_bson, "1", 1, &arr_item);
    bson_append_document_begin(&arr_item, "foo", 3, &subdoc);
    bson_append_utf8(&subdoc, "$gte", 4, "A", 1);
    bson_append_document_end(&arr_item, &subdoc);
    bson_append_document_end(&and_bson, &arr_item);
 
    bson_append_array_end(&query, &and_bson);

I get the same behavior with 1.5.0 as I do with the latest code: you just need to create the filter array correctly for the query to succeed.

For convenience you can try the BSON_APPEND_UTF8 macro which doesn't require a key or value length, so you don't have to count characters yourself. And see the code example in this tutorial for using bson_uint32_to_string to create array keys.

Comment by Liviu Chircu [ 12/Mar/18 ]

Thank you for the nice example, Jesse! I can, indeed, confirm that the BCON API works as you posted. However, I was using the former API, which still does not work. Sample program below:

#include <mongoc.h>
#include <stdio.h>
#include <stdlib.h>
 
#define dbg_bson(_prepend_txt, __bson_ptr__) \
	do { \
		char *__bson_str__; \
		__bson_str__ = bson_as_json(__bson_ptr__, NULL); \
		fprintf(stdout, "%s%s\n", _prepend_txt, __bson_str__); \
		bson_free(__bson_str__); \
	} while (0)
 
int
main (int argc, char *argv[])
{
   mongoc_client_t *client;
   mongoc_collection_t *collection;
   mongoc_cursor_t *cursor;
   bson_error_t error;
   const bson_t *doc;
   bson_t query = BSON_INITIALIZER;
   char *str;
 
   mongoc_init ();
 
   client = mongoc_client_new ("mongodb://192.168.0.103:27017/test");
 
   bson_init (&query);
   // this works, indeed! :)
   //BCON_APPEND (&query, "$and", "[",
   //             "{", "foo", "{", "$lt", BCON_UTF8 ("D"), "}", "}",
   //             "{", "foo", "{", "$gte", BCON_UTF8 ("A"), "}", "}",
   //             "]");
 
   bson_t and_bson, arr_item, subdoc;
   bson_append_array_begin(&query, "$and", 4, &and_bson);
 
   bson_append_document_begin(&and_bson, "", 0, &arr_item);
   bson_append_document_begin(&arr_item, "foo", 3, &subdoc);
   bson_append_utf8(&subdoc, "$lt", 3, "D", 1);
   bson_append_document_end(&arr_item, &subdoc);
   bson_append_document_end(&and_bson, &arr_item);
 
   bson_append_document_begin(&and_bson, "", 0, &arr_item);
   bson_append_document_begin(&arr_item, "foo", 3, &subdoc);
   bson_append_utf8(&subdoc, "$gte", 3, "A", 1);
   bson_append_document_end(&arr_item, &subdoc);
   bson_append_document_end(&and_bson, &arr_item);
 
   bson_append_array_end(&query, &and_bson);
 
   dbg_bson("using filter: ", &query);
 
   collection = mongoc_client_get_collection (client, "test", "test");
   cursor = mongoc_collection_find_with_opts (collection, &query, NULL, NULL);
 
   while (mongoc_cursor_next (cursor, &doc)) {
      str = bson_as_canonical_extended_json (doc, NULL);
      fprintf (stdout, "%s\n", str);
      bson_free (str);
   }
 
   if (mongoc_cursor_error (cursor, &error)) {
      fprintf (stderr, "Cursor Failure: %s\n", error.message);
      return -1;
   }
 
   bson_destroy (&query);
   mongoc_cursor_destroy (cursor);
   mongoc_collection_destroy (collection);
   mongoc_client_destroy (client);
 
   mongoc_cleanup ();
 
   return 0;
}

This produces:

$ ./mongoc-test-1 
using filter: { "$and" : [ { "foo" : { "$lt" : "D" } }, { "foo" : { "$gt" : "A" } } ] }
Cursor Failure: Invalid filter: empty key

So, morale of the story: are the BSON functions I'm using completely deprecated? Again: this code works previously to 1.5.0! Thanks!

Comment by A. Jesse Jiryu Davis [ 11/Mar/18 ]

Thanks for the response. I see that my suggestion about "filter" was misguided, forget that.

However, I can't reproduce the bug you report. I've created a collection called "test.test" with the following data:

> db.test.find()
{ "_id" : ObjectId("5aa5bb86e69834d3ff77bd28"), "foo" : "A" }
{ "_id" : ObjectId("5aa5bb89e69834d3ff77bd29"), "foo" : "B" }
{ "_id" : ObjectId("5aa5bb8be69834d3ff77bd2a"), "foo" : "C" }
{ "_id" : ObjectId("5aa5bb8de69834d3ff77bd2b"), "foo" : "D" }

The following code uses a query like { "$and" : [ { "foo" : { "$lt" : "D" } }, { "foo" : { "$gte" : "A" } } ] } and it retrieves 3 results, using libmongoc 1.7.0:

#include <mongoc.h>
#include <stdio.h>
#include <stdlib.h>
 
int
main (int argc, char *argv[])
{
   mongoc_client_t *client;
   mongoc_collection_t *collection;
   mongoc_cursor_t *cursor;
   bson_error_t error;
   const bson_t *doc;
   bson_t query;
   char *str;
 
   mongoc_init ();
 
   client = mongoc_client_new (NULL);
 
   bson_init (&query);
   BCON_APPEND (&query, "$and", "[",
                "{", "foo", "{", "$lt", BCON_UTF8 ("D"), "}", "}",
                "{", "foo", "{", "$gte", BCON_UTF8 ("A"), "}", "}",
                "]");
 
   collection = mongoc_client_get_collection (client, "test", "test");
   cursor = mongoc_collection_find_with_opts (collection, &query, NULL, NULL);
 
   while (mongoc_cursor_next (cursor, &doc)) {
      str = bson_as_canonical_extended_json (doc, NULL);
      fprintf (stdout, "%s\n", str);
      bson_free (str);
   }
 
   if (mongoc_cursor_error (cursor, &error)) {
      fprintf (stderr, "Cursor Failure: %s\n", error.message);
      return -1;
   }
 
   bson_destroy (&query);
   mongoc_cursor_destroy (cursor);
   mongoc_collection_destroy (collection);
   mongoc_client_destroy (client);
 
   mongoc_cleanup ();
 
   return 0;
}

Comment by Liviu Chircu [ 10/Mar/18 ]

Prepending either "$filter" or "filter" seems to make things worse in 1.7.0. The following basic test does not work anymore:

{ "$filter" : { "foo" : { "$lt" : "D" } } }, gives 0 results.
{ "filter" : { "foo" : { "$lt" : "D" } } }, gives 0 results.

Comment by A. Jesse Jiryu Davis [ 10/Mar/18 ]

Thanks, I have a hunch you're correct, there's a bug and I think I know what it is.

Does mongoc_collection_find_with_opts give the correct results if you use a field named "filter" in the place you used to use "query"?:

{ "filter": { "$and" : [ { "foo" : { "$lt" : "D" } }, { "foo" : { "$gte" : "A" } } ] } }

Generated at Wed Feb 07 21:15:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.