Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-2731

Mongorestore sometimes reports incorrect counts

    • Type: Icon: Investigation Investigation
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • 100.8.1
    • Affects Version/s: 100.1.1
    • Component/s: mongorestore
    • None
    • 1,490

      Successful reproduction depends on the data processed by the tool. I was able to make the wrong counts reported if I dump and restore the sample_airbnb database from my Atlas M0.

       

      Here is a sample output:

      villain@R7:/mnt/c/Users/Dmitry/Desktop/Tools$ ./dumpRestore.sh
      2020-09-09T16:25:37.250+1000 dumping up to 1 collections in parallel
      2020-09-09T16:25:37.995+1000 writing sample_airbnb.listingsAndReviews to mongodump/sample_airbnb/listingsAndReviews.bson
      2020-09-09T16:25:40.003+1000 [........................] sample_airbnb.listingsAndReviews 0/5555 (0.0%)
      2020-09-09T16:25:42.996+1000 [........................] sample_airbnb.listingsAndReviews 101/5555 (1.8%)
      2020-09-09T16:25:45.997+1000 [........................] sample_airbnb.listingsAndReviews 101/5555 (1.8%)
      2020-09-09T16:25:49.002+1000 [###.....................] sample_airbnb.listingsAndReviews 867/5555 (15.6%)
      2020-09-09T16:25:52.007+1000 [###.....................] sample_airbnb.listingsAndReviews 867/5555 (15.6%)
      2020-09-09T16:25:55.001+1000 [###.....................] sample_airbnb.listingsAndReviews 867/5555 (15.6%)
      2020-09-09T16:25:57.998+1000 [#######.................] sample_airbnb.listingsAndReviews 1687/5555 (30.4%)
      2020-09-09T16:26:00.998+1000 [#######.................] sample_airbnb.listingsAndReviews 1687/5555 (30.4%)
      2020-09-09T16:26:04.002+1000 [#######.................] sample_airbnb.listingsAndReviews 1687/5555 (30.4%)
      2020-09-09T16:26:07.002+1000 [############............] sample_airbnb.listingsAndReviews 2976/5555 (53.6%)
      2020-09-09T16:26:10.004+1000 [############............] sample_airbnb.listingsAndReviews 2976/5555 (53.6%)
      2020-09-09T16:26:13.001+1000 [############............] sample_airbnb.listingsAndReviews 2976/5555 (53.6%)
      2020-09-09T16:26:16.008+1000 [############............] sample_airbnb.listingsAndReviews 2976/5555 (53.6%)
      2020-09-09T16:26:18.999+1000 [############............] sample_airbnb.listingsAndReviews 2986/5555 (53.8%)
      2020-09-09T16:26:21.998+1000 [###################.....] sample_airbnb.listingsAndReviews 4584/5555 (82.5%)
      2020-09-09T16:26:25.007+1000 [###################.....] sample_airbnb.listingsAndReviews 4584/5555 (82.5%)
      2020-09-09T16:26:28.008+1000 [###################.....] sample_airbnb.listingsAndReviews 4584/5555 (82.5%)
      2020-09-09T16:26:31.006+1000 [######################..] sample_airbnb.listingsAndReviews 5195/5555 (93.5%)
      2020-09-09T16:26:33.999+1000 [######################..] sample_airbnb.listingsAndReviews 5195/5555 (93.5%)
      2020-09-09T16:26:36.700+1000 [########################] sample_airbnb.listingsAndReviews 5555/5555 (100.0%)
      2020-09-09T16:26:36.706+1000 done dumping sample_airbnb.listingsAndReviews (5555 documents)
      2020-09-09T16:26:36.999+1000 using write concern: &\{majority false 0}
      2020-09-09T16:26:40.677+1000 The --db and --collection flags are deprecated for this use-case; please use --nsInclude instead, i.e. with --nsInclude=${DATABASE}.${COLLECTION}
      2020-09-09T16:26:40.679+1000 building a list of collections to restore from mongodump/sample_airbnb dir
      2020-09-09T16:26:40.679+1000 found collection testdb.listingsAndReviews bson to restore to testdb.listingsAndReviews
      2020-09-09T16:26:40.680+1000 found collection metadata from testdb.listingsAndReviews to restore to testdb.listingsAndReviews
      2020-09-09T16:26:40.923+1000 restoring to existing collection testdb.listingsAndReviews without dropping
      2020-09-09T16:26:40.924+1000 reading metadata for testdb.listingsAndReviews from mongodump/sample_airbnb/listingsAndReviews.metadata.json
      2020-09-09T16:26:40.925+1000 collection testdb.listingsAndReviews already exists - skipping collection create
      2020-09-09T16:26:40.925+1000 restoring testdb.listingsAndReviews from mongodump/sample_airbnb/listingsAndReviews.bson
      2020-09-09T16:26:43.205+1000 [#####...................] testdb.listingsAndReviews 20.8MB/90.0MB (23.2%)
      2020-09-09T16:26:46.208+1000 [#####...................] testdb.listingsAndReviews 20.8MB/90.0MB (23.2%)
      2020-09-09T16:26:49.200+1000 [#####...................] testdb.listingsAndReviews 20.8MB/90.0MB (23.2%)
      2020-09-09T16:26:52.194+1000 [#####...................] testdb.listingsAndReviews 20.8MB/90.0MB (23.2%)
      2020-09-09T16:26:55.204+1000 [#####...................] testdb.listingsAndReviews 20.8MB/90.0MB (23.2%)
      2020-09-09T16:26:58.203+1000 [#####...................] testdb.listingsAndReviews 20.8MB/90.0MB (23.2%)
      2020-09-09T16:27:01.203+1000 [#####...................] testdb.listingsAndReviews 20.8MB/90.0MB (23.2%)
      2020-09-09T16:27:04.194+1000 [##########..............] testdb.listingsAndReviews 38.2MB/90.0MB (42.4%)
      2020-09-09T16:27:07.197+1000 [##########..............] testdb.listingsAndReviews 38.2MB/90.0MB (42.4%)
      2020-09-09T16:27:10.205+1000 [##########..............] testdb.listingsAndReviews 38.2MB/90.0MB (42.4%)
      2020-09-09T16:27:13.200+1000 [##########..............] testdb.listingsAndReviews 38.2MB/90.0MB (42.4%)
      2020-09-09T16:27:16.202+1000 [##########..............] testdb.listingsAndReviews 38.2MB/90.0MB (42.4%)
      2020-09-09T16:27:19.208+1000 [##########..............] testdb.listingsAndReviews 38.2MB/90.0MB (42.4%)
      2020-09-09T16:27:22.204+1000 [#############...........] testdb.listingsAndReviews 50.1MB/90.0MB (55.6%)
      2020-09-09T16:27:25.200+1000 [#############...........] testdb.listingsAndReviews 50.1MB/90.0MB (55.6%)
      2020-09-09T16:27:28.205+1000 [#############...........] testdb.listingsAndReviews 50.1MB/90.0MB (55.6%)
      2020-09-09T16:27:31.203+1000 [#############...........] testdb.listingsAndReviews 50.1MB/90.0MB (55.6%)
      2020-09-09T16:27:34.200+1000 [#############...........] testdb.listingsAndReviews 50.1MB/90.0MB (55.6%)
      2020-09-09T16:27:37.195+1000 [###############.........] testdb.listingsAndReviews 59.1MB/90.0MB (65.7%)
      2020-09-09T16:27:40.202+1000 [###############.........] testdb.listingsAndReviews 59.1MB/90.0MB (65.7%)
      2020-09-09T16:27:43.195+1000 [###############.........] testdb.listingsAndReviews 59.1MB/90.0MB (65.7%)
      2020-09-09T16:27:46.208+1000 [####################....] testdb.listingsAndReviews 77.6MB/90.0MB (86.2%)
      2020-09-09T16:27:49.201+1000 [####################....] testdb.listingsAndReviews 77.6MB/90.0MB (86.2%)
      2020-09-09T16:27:52.199+1000 [####################....] testdb.listingsAndReviews 77.6MB/90.0MB (86.2%)
      2020-09-09T16:27:55.201+1000 [####################....] testdb.listingsAndReviews 77.6MB/90.0MB (86.2%)
      2020-09-09T16:27:58.202+1000 [####################....] testdb.listingsAndReviews 77.6MB/90.0MB (86.2%)
      2020-09-09T16:28:01.204+1000 [####################....] testdb.listingsAndReviews 77.6MB/90.0MB (86.2%)
      2020-09-09T16:28:04.194+1000 [########################] testdb.listingsAndReviews 90.0MB/90.0MB (100.0%)
      2020-09-09T16:28:07.202+1000 [########################] testdb.listingsAndReviews 90.0MB/90.0MB (100.0%)
      2020-09-09T16:28:10.205+1000 [########################] testdb.listingsAndReviews 90.0MB/90.0MB (100.0%)
      2020-09-09T16:28:13.208+1000 [########################] testdb.listingsAndReviews 90.0MB/90.0MB (100.0%)
      2020-09-09T16:28:16.198+1000 [########################] testdb.listingsAndReviews 90.0MB/90.0MB (100.0%)
      2020-09-09T16:28:16.346+1000 [########################] testdb.listingsAndReviews 90.0MB/90.0MB (100.0%)
      2020-09-09T16:28:16.347+1000 restoring indexes for collection testdb.listingsAndReviews from metadata
      2020-09-09T16:28:16.347+1000 run create Index command for indexes: property_type_1_room_type_1_beds_1, name_1, address.location_2dsphere
      2020-09-09T16:28:16.708+1000 finished restoring testdb.listingsAndReviews (3018 documents, 0 failures)
      2020-09-09T16:28:16.709+1000 3018 document(s) restored successfully. 0 document(s) failed to restore.
      villain@R7:/mnt/c/Users/Dmitry/Desktop/Tools$
      

      Then, if I check the counts manually in mongo shell they match those reported by mongodump, not mongorestore:

      MongoDB Enterprise Cluster0-shard-0:PRIMARY> db.getSiblingDB("testdb").listingsAndReviews.find().itcount()
      5555
      

            Assignee:
            Unassigned Unassigned
            Reporter:
            dmitry.ryabtsev@mongodb.com Dmitry Ryabtsev
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: