Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-3301

mongodump fails while exporting data

    • Type: Icon: Bug Bug
    • Resolution: Gone away
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: mongodump
    • Labels:
      None

      Problem Statement/Rationale

      When performing a `mongodump` over the internet, the socket closes with the error:  `socket was unexpectedly closed: EOF`

      Details

      • The error only occurs over the internet.  Connecting over a local network, or directly from the server does not cause it to fail.
      • This typically happens when there is a large amount of data to download
      • It does not appear to fail due to a time out.
        • The elapsed time in which the error occurs is variable
        • Specifying a longer timeout (i.e. 60 second) does not mitigate the problem
      • It is unclear whether or not specifying a collection will cause it not to fail
      • The FQDN nor use of TLS does not appear to be related, as I've used the exact same connection string on the local network.  A local DNS server resolves an address to the same server that the NAT forwards the connection to.
      • I've analyzed the `mongodump.go` file and it appears to be related to communication between the two streams while iterating through buffers of data:  There is a find() query (cursor) stream, and a file writer stream.  However, I was not able to determine the exact cause of the failure.
        • From what it seems like, the read stream can't keep up with the write stream, and eventually results in an empty buffer. (EOF)

      Steps to Reproduce

      Perform a `mongodump` command from a standalone MongoDB (CE or enterprise) connection that connects over the internet.

      Expected Results

      The operation should continue to stream data until it is complete.

      Actual Results

      The operation fails in the middle of the process (i.e. 37% of one of the collections)

      Additional Notes

      I've reproduced the problem, which recorded the following output:

      ```

      mongodump 'mongodb://****:***@****.com:27017/GTFS?tls=true&authSource=admin'            
      2023-05-16T15:54:51.091-0400    WARNING: On some systems, a password provided directly in a connection string or using --uri may be visible to system status programs such as `ps` that may be invoked by other users. Consider omitting the password to provide it via stdin, or using the --config option to specify a configuration file with the password.
      2023-05-16T15:54:52.445-0400    writing GTFS.stops to dump/GTFS/stops.bson
      2023-05-16T15:54:52.470-0400    writing GTFS.stop_times to dump/GTFS/stop_times.bson
      2023-05-16T15:54:52.485-0400    writing GTFS.routes to dump/GTFS/routes.bson
      2023-05-16T15:54:52.524-0400    writing GTFS.trips to dump/GTFS/trips.bson
      2023-05-16T15:54:52.658-0400    done dumping GTFS.routes (46 documents)
      2023-05-16T15:54:52.710-0400    writing GTFS.agency to dump/GTFS/agency.bson
      2023-05-16T15:54:52.791-0400    done dumping GTFS.stops (1795 documents)
      2023-05-16T15:54:52.824-0400    done dumping GTFS.agency (5 documents)
      2023-05-16T15:54:54.092-0400    [........................]  GTFS.stop_times  101/634063  (0.0%)
      2023-05-16T15:54:54.092-0400    [........................]       GTFS.trips   101/28445  (0.4%)
      2023-05-16T15:54:54.092-0400    
      2023-05-16T15:54:54.676-0400    #######################  GTFS.trips  28445/28445  (100.0%)
      2023-05-16T15:54:54.677-0400    done dumping GTFS.trips (28445 documents)
      2023-05-16T15:54:57.092-0400    #......................  GTFS.stop_times  59488/634063  (9.4%)
      2023-05-16T15:55:00.092-0400    ###....................  GTFS.stop_times  118875/634063  (18.7%)
      2023-05-16T15:55:03.092-0400    #####..................  GTFS.stop_times  178258/634063  (28.1%)
      2023-05-16T15:55:04.636-0400    #######................  GTFS.stop_times  237626/634063  (37.5%)
      2023-05-16T15:55:05.866-0400    Failed: error writing data for collection `GTFS.stop_times` to disk: error reading collection: connection pool for ****.com:27017 was cleared because another operation failed with: connection(****.com:27017[-9]) socket was unexpectedly closed: EOF

      ```

            Assignee:
            tim.fogarty@mongodb.com Tim Fogarty
            Reporter:
            mike.marcellaro@mongodb.com Mike Marcellaro
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: