Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-748

mongodump i/o timeout error on listIndexes

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 3.0.3
    • Fix Version/s: 3.0.4, 3.1.4
    • Component/s: mongodump
    • Labels:
      None
    • Environment:
      RHEL7 with 3.0.3 rpm built from Fedora spec file with wiredTiger 2.5.3
    • Backport Requested:
      v3.0

      Description

      When trying to do a complete backup with mongodump, it always fails with an "i/o timeout", but not always on the exact same collection. Here's a typical output :

      [...]
      2015-05-19T13:45:19.202+0200	done dumping example12_analytics.tmp_d_carrier_country_date_device_error_flow_landing_op1_os_owner_provider_site
      2015-05-19T13:45:19.203+0200	writing example12_analytics.d_date_owner to /var/lib/mongodb-backup/mongodb_20150519-132021/example12_analytics/d_date_owner.bson
      2015-05-19T13:45:19.203+0200	done dumping example12_analytics.d_date_landing_owner
      2015-05-19T13:45:19.204+0200	writing example12_analytics.d_date_owner_provider to /var/lib/mongodb-backup/mongodb_20150519-132021/example12_analytics/d_date_owner_provider.bson
      2015-05-19T13:45:19.628+0200	writing example12_analytics.d_date_error_owner metadata to /var/lib/mongodb-backup/mongodb_20150519-132021/example12_analytics/d_date_error_owner.metadata.json
      2015-05-19T13:45:20.115+0200	done dumping example12_analytics.d_date_error_owner
      2015-05-19T13:45:20.115+0200	writing example12_analytics.client_context_mobile to /var/lib/mongodb-backup/mongodb_20150519-132021/example12_analytics/client_context_mobile.bson
      2015-05-19T13:45:21.919+0200	[###########.............]                                                  example12_analytics.client_context_example  5749116/12446330  (46.2%)
      2015-05-19T13:45:21.919+0200	[#######.................]                                             example12_analytics.billing_provider_repository  6231634/19349001  (32.2%)
      2015-05-19T13:45:21.919+0200	[######################..]  example12_analytics.d_carrier_country_date_device_error_op1_os_owner_provider_regdate_site   6487164/6773612  (95.8%)
      2015-05-19T13:45:21.919+0200	[####....................]            example12_analytics.d_carrier_country_date_device_flow_landing_op1_os_owner_site  8024276/40391310  (19.9%)
      2015-05-19T13:45:21.919+0200	[#############...........]                                                    example12_analytics.client_context_abcde      69672/126395  (55.1%)
      2015-05-19T13:45:21.919+0200	[#####...................]                                                            example12_analytics.d_date_owner       21491/91050  (23.6%)
      2015-05-19T13:45:21.919+0200	[#####...................]                                                   example12_analytics.d_date_owner_provider       18660/82934  (22.5%)
      2015-05-19T13:45:21.919+0200	[........................]                                                   example12_analytics.client_context_mobile         101/67141   (0.2%)
      2015-05-19T13:45:21.919+0200	
      2015-05-19T13:45:24.919+0200	[###########.............]                                                  example12_analytics.client_context_example  5769348/12446330  (46.4%)
      2015-05-19T13:45:24.919+0200	[#######.................]                                             example12_analytics.billing_provider_repository  6243221/19349001  (32.3%)
      2015-05-19T13:45:24.919+0200	[#######################.]  example12_analytics.d_carrier_country_date_device_error_op1_os_owner_provider_regdate_site   6500964/6773612  (96.0%)
      2015-05-19T13:45:24.919+0200	[####....................]            example12_analytics.d_carrier_country_date_device_flow_landing_op1_os_owner_site  8039685/40391310  (19.9%)
      2015-05-19T13:45:24.919+0200	[#################.......]                                                    example12_analytics.client_context_abcde      89666/126395  (70.9%)
      2015-05-19T13:45:24.919+0200	[################........]                                                            example12_analytics.d_date_owner       64371/91050  (70.7%)
      2015-05-19T13:45:24.919+0200	[##############..........]                                                   example12_analytics.d_date_owner_provider       48384/82934  (58.3%)
      2015-05-19T13:45:24.919+0200	[####....................]                                                   example12_analytics.client_context_mobile       12508/67141  (18.6%)
      2015-05-19T13:45:24.919+0200	
      2015-05-19T13:45:26.195+0200	writing example12_analytics.d_date_owner_provider metadata to /var/lib/mongodb-backup/mongodb_20150519-132021/example12_analytics/d_date_owner_provider.metadata.json
      2015-05-19T13:45:27.016+0200	done dumping example12_analytics.d_date_owner_provider
      2015-05-19T13:45:27.017+0200	writing example12_analytics.d_date_billing_owner to /var/lib/mongodb-backup/mongodb_20150519-132021/example12_analytics/d_date_billing_owner.bson
      2015-05-19T13:45:27.036+0200	writing example12_analytics.client_context_abcde metadata to /var/lib/mongodb-backup/mongodb_20150519-132021/example12_analytics/client_context_abcde.metadata.json
      2015-05-19T13:45:27.250+0200	writing example12_analytics.d_date_owner metadata to /var/lib/mongodb-backup/mongodb_20150519-132021/example12_analytics/d_date_owner.metadata.json
      2015-05-19T13:45:27.589+0200	done dumping example12_analytics.client_context_abcde
      2015-05-19T13:45:27.589+0200	writing example12_analytics.d_date_owner_product to /var/lib/mongodb-backup/mongodb_20150519-132021/example12_analytics/d_date_owner_product.bson
      2015-05-19T13:45:27.632+0200	writing example12_analytics.client_context_mobile metadata to /var/lib/mongodb-backup/mongodb_20150519-132021/example12_analytics/client_context_mobile.metadata.json
      2015-05-19T13:45:27.919+0200	[###########.............]                                                  example12_analytics.client_context_example  5799805/12446330  (46.6%)
      2015-05-19T13:45:27.919+0200	[#######.................]                                             example12_analytics.billing_provider_repository  6289711/19349001  (32.5%)
      2015-05-19T13:45:27.919+0200	[#######################.]  example12_analytics.d_carrier_country_date_device_error_op1_os_owner_provider_regdate_site   6528578/6773612  (96.4%)
      2015-05-19T13:45:27.919+0200	[####....................]            example12_analytics.d_carrier_country_date_device_flow_landing_op1_os_owner_site  8085866/40391310  (20.0%)
      2015-05-19T13:45:27.919+0200	[........................]                                                    example12_analytics.d_date_billing_owner         101/28106   (0.4%)
      2015-05-19T13:45:27.919+0200	
      2015-05-19T13:45:30.289+0200	Failed: error running `listIndexes`. Collection: `example12_analytics.d_date_owner` Err: read tcp 127.0.0.1:27017: i/o timeout
      

      It happens after running for a while. The /var/lib/mongodb directory is 21GB in size (wiredTiger+snappy, it used to be about 100GB with mmapv1).

      I am using the tools from the mongodb-linux-x86_64-rhel70-3.0.3.tgz binary download, since the tools are now written in Go and no longer included in the main sources. I have quickly tried to use a 2.6.10 mongodump, but it didn't work with :

      connected to: 127.0.0.1
      2015-05-20T09:45:29.900+0200 all dbs
      output of listDatabases isn't what we expected, no 'databases' field:
      { note: "from execCommand", ok: 0.0, errmsg: "not master" }
      

      (I need to run it against a SECONDARY)

      The command I have been using is this one, both with and without --oplog :

      mongodump -u admin -p xxxxxxxxxxxx --authenticationDatabase=admin --oplog
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              kyle.erf Kyle Erf
              Reporter:
              Thias Matthias Saou
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: