Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-2766

mongoimport does not import small amout of documents

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor - P4
    • Resolution: Works as Designed
    • Affects Version/s: 100.1.1
    • Fix Version/s: No versions
    • Component/s: mongoimport
    • Labels:
      None
    • Environment:

      Description

      I try to import document with `mongoimport` using STDIN

       

      It seems to work only for large amount of documents. Single documents do are not inserted:

       

      ````

      [mediation@talts01mipmed0 log]$ mongo --norc mip <<< 'db.getCollection("sessions.raw").deleteMany({})'
      MongoDB shell version v4.4.1
      connecting to: mongodb://127.0.0.1:27017/mip?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
      Implicit session: session { "id" : UUID("d9219b8e-1d1d-47ab-8955-6881bdb66519") }
      MongoDB server version: 4.4.1

      { "acknowledged" : true, "deletedCount" : 0 }

      bye

      [mediation@talts01mipmed0 log]$ mongoimport --db=mip --collection=sessions.raw
      2020-11-02T14:06:34.166+0100 connected to: mongodb://localhost/
      2020-11-02T14:06:37.166+0100 mip.sessions.raw 0B
      2020-11-02T14:06:40.166+0100 mip.sessions.raw 0B

      {"h":1}

      2020-11-02T14:06:43.166+0100 mip.sessions.raw 8B
      2020-11-02T14:06:46.166+0100 mip.sessions.raw 8B
      2020-11-02T14:06:49.166+0100 mip.sessions.raw 8B
      2020-11-02T14:06:52.166+0100 mip.sessions.raw 8B
      2020-11-02T14:06:55.166+0100 mip.sessions.raw 8B
      ^C2020-11-02T14:06:55.665+0100 signal 'interrupt' received; forcefully terminating

      [mediation@talts01mipmed0 log]$ mongo --norc mip <<< 'db.getCollection("sessions.raw").countDocuments({})'
      MongoDB shell version v4.4.1
      connecting to: mongodb://127.0.0.1:27017/mip?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
      Implicit session: session { "id" : UUID("03ae7a1c-03f1-4070-a0bd-9d8fd7f98b8d") }
      MongoDB server version: 4.4.1
      0
      bye

       

      ````

       

      I insert document from syslog-ng, see https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/44#TOPIC-1431069

       

      My syslog-ng code is like this:

      ```

      destination d_mongo_snat {
      program("/usr/bin/mongoimport --db=mip --collection=sessions.raw"
      flush-lines(1)
      persist-name("snat")
      template("{ \"v\":${VALID}, \"t0\":{\"$$date\":{\"$$numberLong\":\"${BEGINTIME}${.HW.BeginTime}000\"}} }")
      );
      };

      ```

      The number of documents in my collection is always a multiple of 1000, so it looks like mongoimport imports documents in batches of 1000 but as long as the batch is not full, the data is not written and gets lost.

      I was not able to find a way how to flush the buffer. "Ctrl-C" terminates the import without flushing the data.

       

      Any solution?

       

       

       

       

       

       

       

       

       

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              evgeni.dobranov Evgeni Dobranov
              Reporter:
              wernfried.domscheit@sunrise.net Wernfried Domscheit
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: