Uploaded image for project: 'Node.js Driver'
  1. Node.js Driver
  2. NODE-5372

Using hasNext() and next() on mapped aggregation cursor causes map function to be called twice for every document

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 4.1.1, 5.6.0
    • Component/s: Aggregation
    • 1
    • Not Needed
    • Not Needed
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?

      What problem are you facing?

      There's a bug in the AggregationCursor that causes the `transform` function passed to `AggregationCursor.map()` to be called multiple times consecutively for the same document, when using `hasNext()` in combination with `next()`, which is a very common pattern in working with aggregation cursors. 

      When this map function transforms the type of the to-be-returned document (let's say from type A to type B) (note: that's what most people use `map` for), then calling `hasNext()` and then `next()` causes a TypeError to occur. From what I understand, it is because `hasNext()` saves the map function as a transform property on the aggregation cursor, which it then detects while executing `next()` to be an "old transform" which it applies first before it applies the map function. Obviously, `transform(transform(A))` is invalid when `transform` is a function `(arg: A) => B`

      See below for a simple reproducible example.

      I'm marking this as a critical issue because unless `transform` is a function (arg: A | B) => B`, this bug causes the entire NodeJS program to crash. Also, both the `if (cursor.hasNext()) doSomething(cursor.next());` pattern as well as having the map function transform the type of its input, are very common patterns.

      There is, however, a workaround: simply don't use `hasNext()` and just call `next()` until it returns `null`. But that means `hasNext()` is simply unusable.

      What driver and relevant dependency versions are you using?

      MongoDB Node driver: 5.6.0, though I originally found it on 4.1.1 and probably affects all versions in between and probably some versions prior as well. That's for you to test

      Steps to reproduce?

      To reproduce, I'll use an edited version of the MongoDB usage example:

      1. Create a folder
      2. `yarn init`
      3. `yarn add mongodb` This installed v5.6.0 at the time of writing.
      4. Add the code in the first code block below into `index.js`
      5. Ensure you have a collection `deb.Customer` on a locally running Mongo instance with a bunch of documents in it, or edit the `uri`, `dbname` and `collname` variables in the code snippet.
      6. `node index.js`
      7. Observe that the output is what is displayed in the second code block

       

      const { MongoClient } = require("mongodb");
      
      // Replace the uri, dbname and collname with a connection string, db name and collection name that contain some documents for you
      const uri = "mongodb://localhost";
      const dbname = "deb";
      const collname = "Customer";
      
      const client = new MongoClient(uri);
      
      async function run() {
        try {
          const database = client.db(dbname);
          const collection = database.collection(collname);
      
          // Do an aggregation (doesn't matter what it is), then perform a map that transforms the type.
          const cursor = collection.aggregate([
            { $match: { _id: { $exists: true } } },
          ]).map(doc => {
            const { _id, ...rest } = doc;
            console.log('map', _id);
            return { ...rest, id: doc._id.toHexString() };
          });
      
          let retrieved = 0;
          const limit = 10;
          while ((await cursor.hasNext()) && retrieved < limit) {
            console.log((await cursor.next()));
            retrieved++;
          }
        } finally {
          // Ensures that the client will close when you finish/error
          await client.close();
        }
      }
      run().catch(console.dir);
      
      map new ObjectId("59f86a80edaf5f0001b850bf")
      map undefined
      TypeError: Cannot read properties of undefined (reading 'toHexString')
          at AggregationCursor.<anonymous> (/mongo-cursor-map-problem/index.js:21:37)
          at nextDocument (/mongo-cursor-map-problem/node_modules/mongodb/lib/cursor/abstract_cursor.js:490:34)
          at next (/mongo-cursor-map-problem/node_modules/mongodb/lib/cursor/abstract_cursor.js:511:29)
          at node:internal/util:364:7
          at new Promise (<anonymous>)
          at next (node:internal/util:350:12)
          at AggregationCursor.next (/mongo-cursor-map-problem/node_modules/mongodb/lib/cursor/abstract_cursor.js:223:16)
          at run (/mongo-cursor-map-problem/index.js:27:33)
          at processTicksAndRejections (node:internal/process/task_queues:96:5) 

       

      P.S. this issue reporting system is a horrible user experience, please consider switching to GitHub Issues or Linear or anything else less horrible than Jira.

            Assignee:
            bailey.pearson@mongodb.com Bailey Pearson
            Reporter:
            bart@energiebespaarders.nl Bart van Oort
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: