[SERVER-5737] Mongorestore to mongos when using auth is slow. Created: 01/May/12  Updated: 15/Aug/12  Resolved: 01/May/12

Status: Closed
Project: Core Server
Component/s: Performance, Tools
Affects Version/s: 2.0.4
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Jonathan Schneider Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 6.2 x86_64 - Runnig on VMWare ESX


Issue Links:
Duplicate
duplicates SERVER-5405 mongos does not send reads to seconda... Closed
Operating System: ALL
Participants:

 Description   

I have observed and been able to duplicate mongorestore running very slowely when restoring data to a sharded cluster while using auth. I setup a simple test cluster for the below examples:
Before Auth: (fast, about 10MB/sec)
Restore to mongos:

[root@testserver scratch]# /opt/point2/mongodb/bin/mongorestore --host 
mongotest --port 27017 -d testdb properties.bson 
connected to: mongotest:27017 
Mon Apr 30 15:52:12 properties.bson 
Mon Apr 30 15:52:12      going into namespace [testdb.properties] 
                9168618/20325604307     0% 
                21463896/20325604307    0% 
                33556308/20325604307    0% 
                45867960/20325604307    0%

After Auth: (slow, about 3MB/sec)
Restore to mongos:

[root@testserver scratch]# /opt/point2/mongodb/bin/mongorestore --host 
mongotest --port 27017 -d testdb -u user -p password properties.bson 
connected to: mongotest:27017 
Tue May  1 08:37:05 properties.bson 
Tue May  1 08:37:05      going into namespace [testdb.properties] 
                2656954/20325604307     0% 
                7127948/20325604307     0% 
                10420282/20325604307    0% 
                15139072/20325604307    0%

Restore to replica set primary with auth: (fast, about 10MB/sec)

[root@testserver scratch]# /opt/point2/mongodb/bin/mongorestore --host 
mongotest --port 27018 -d testdb -u user -p password properties.bson 
connected to: mongotest:27018 
Tue May  1 08:39:32 properties.bson 
Tue May  1 08:39:32      going into namespace [testdb.properties] 
                9899255/20325604307     0% 
                23405742/20325604307    0% 
                36914260/20325604307    0% 
                50356611/20325604307    0%

This was a simple test setup that only contained 1 shard and 2 replica set members, the disparity in speed difference seems to be larger with a larger cluster. In our prod cluster mongorestore runs closer to 1MB/sec with auth against a mongos instance and 50MB/sec run directly against the primary server in one shard.



 Comments   
Comment by Jonathan Schneider [ 01/May/12 ]

Yes, sounds good, thanks.

Comment by Eric Milkie [ 01/May/12 ]

I marked SERVER-5405 as a candidate for backporting for 2.0.6. May I resolve this as a duplicate?

Comment by Jonathan Schneider [ 01/May/12 ]

This may have been fixed in https://jira.mongodb.org/browse/SERVER-5405

I tested with 2.1.1 latest nightly and the problem does not exist. Would love to see a 2.0 backport.

Comment by Jonathan Schneider [ 01/May/12 ]

I should add that during the slow restores nothing is pinned anywhere, iostat reveals quiet disks, top reveals quiet cpu's on both the mongos instance and the primary server. Nothing seems to be causing a bottleneck anywhere. When run without auth the mongod instance on the primary server becomes pinned clearly being the bottleneck to faster restore speed.

Comment by Jonathan Schneider [ 01/May/12 ]

I believe this is an important issue as it will make restoring a database in the event of a problem a much longer process than it should be. I needed to restore a 20GB database on prod and it took 12 hours, on my test setup without auth it did it in less than 30 minutes.

Generated at Thu Feb 08 03:09:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.