Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-3598

Mongodump/Mongorestore support export from DocumentDB and Import into Mongo

    • Type: Icon: Task Task
    • Resolution: Won't Do
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Environment:
      DocumentDB to Mongo
    • Tools and Replicator
    • 0.25

      Problem Statement/Rationale

      Starting around early June 2024 Amazon added an option to collections in DocumentDB that is not supported by Mongorestore. Below is a code snippet we have tested locally and resolves the issue, however it is too generic for widespread deployment and ought to be restricted by an optional tag passed into Mongorestore.

      Likely Epic: Cross-version dump/restore

      Steps to Reproduce

      Run Mongodump on a DocumentDB database. (specifically

      mongodump --uri "mongodb://USERNAME:PASSWORD@HOST:PORT/TARGET_DB?ssl=true&ssl_ca_certs=global-bundle.pem&retryWrites=false&authSource=CREDENTIAL_DB"  --gzip --archive="FILE_LOCATION"
      

      )
      Attempt to restore with Mongorestore. (specifically

      mongorestore --drop --username USERNAME --nsInclude=DATABASE_NAME.* --authenticationDatabase CREDENTIAL_DB --archive=/home/path/backup.archive --gzip -p PASSWORD 

      )

      Expected Results

      Regardless of what storage engine the source had, restoration should be into a compatible engine.

      Actual Results

      Get error (retyped from snip): "error running create command: (InvalidOptions) documentDB is not a registered storage engine for this server"

      Additional Notes

      Cause: Around early June 2024 Amazon added the documentDB option to all collections. Although this option ought to be persisted on dump, it should be ignored when not restoring to DocumentDB. Specifcally the command db.getCollectionInfos() shows (retyped from snip) - 

          type: 'collection',
          options: {
              autoIndexId: true,
              capped: false,
              storageEngine: {documentDB: { compression: {enable: false }}}
          },
          info: { readonly: false}, 

      Partial Fix:
      It should be noted that I am not a GO developer and have only spent around 4 hours learning GO to create this fix. That said, this fix does compile and the result of the make command does resolve the issue locally. The down side is that it is an inelegant solution which ought to be limited to only running when a new option is passed into mongorestore such as "mongorestore --ignoreStorageEngine"
       

      cleanOptions := bson.D{}
      for index, option := range options{
          if(option.Key == "storageEngine"){
              log.Logv(log.Info, fmt.Sprintf("Skipping option 'storaged engine': %d", index))
          }else{
              cleanOptions = append(cleanOptions, option)
          }
      }
      options = cleanOptions 
      

      as of commit a9eeb2c20a0568ffec5534f94b7b6b47af651355 by John DuBois on Wed June 26, 2024 this code should be inserted in restore.go at line 328:

      325   // first create the collection with options from the metadata file
      326   uuid := intent.UUID
      327   options := intent.Options
      //snippet goes here to fix.
      328   if len(options) == 0 {
      329      logMessageSuffix = "with no metadata"
      330   }
      

        1. getCollectionInfos.png
          getCollectionInfos.png
          18 kB
        2. errorMessage.png
          errorMessage.png
          6 kB

            Assignee:
            Unassigned Unassigned
            Reporter:
            philip.lawrence.1992@gmail.com Philip Lawrence
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: