Uploaded image for project: 'Node.js Driver'
  1. Node.js Driver
  2. NODE-1347

Unable to read through MongoDB 2.2.33 Cursor using AWS-SDK's ManagedUpload read loop

    • Type: Icon: Bug Bug
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.2.33
    • Component/s: Lambda
    • Labels:
    • Environment:
      AWS-Lambda for desired Runtime
      Problem can be reproduced locally with MAC-OSX running Node 6.10.3

      When working on a lambda for our team that dumps a MongoDB cursor's contents into an s3 bucket, we opted to use AWS-SDK ManagedUpload due to its ability to perform uploads from streams,buffers etc while specifying a part size in order to limit memory consumption.

      Unfortunately, I have not been able to get it to work with MongoDB cursor's stream. I have confirmed that the reading loop performed by ManagedUpload's send method will work for a normal stream, but not for Mongo's. I have attached a js file exporting a function that takes in a cursor, and runs a re-implimented version of ManagedUpload's read loop. I have pasted their code commented at the bottom of the file for reference. It seems to be a valid read-loop, and can empty out the normal readable just fine.

      Please let me know if you have any questions or issues reproducing this issue. Thanks

        1. cursorPlayground.js
          4 kB
          Maxwell Daum

            Assignee:
            Unassigned Unassigned
            Reporter:
            mdaum Maxwell Daum
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: