[SERVER-13712] Reduce peak disk usage of test suites Created: 24/Apr/14 Updated: 11/Jul/16 Resolved: 08/May/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | 2.6.2, 2.7.1 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Matt Kangas | Assignee: | Matt Kangas |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Backport Completed: | |||||||||
| Sprint: | Server 2.7.1 | ||||||||
| Participants: | |||||||||
| Linked BF Score: | 0 | ||||||||
| Description |
|
We need to ensure that our test suites never consume more than approximately 10 GB of /data on our MCI buildvariants. Currently MCI tasks are taking up to 33 GB of /data space. This prevents us from using ephemeral storage on some EC2 configurations Biggest offenders seen so far (see MCI-1449) Linux 64/Linux 64 debug:
Windows
The obvious offenders in sharding/replicasets/slow2 are suites that do not clean up test databases periodically. A simple fix may be to drop all databases every N tests, like smoke.py does. Related: MCI-1276, MCI-1449 |
| Comments |
| Comment by Githook User [ 08/May/14 ] | |||||||||||||||||||
|
Author: {u'username': u'kangas', u'name': u'Matt Kangas', u'email': u'matt.kangas@mongodb.com'}Message: (cherry picked from commit 630944421eebbbdbde600b6294e28ec992fc3fba) | |||||||||||||||||||
| Comment by Githook User [ 08/May/14 ] | |||||||||||||||||||
|
Author: {u'username': u'kangas', u'name': u'Matt Kangas', u'email': u'matt.kangas@mongodb.com'}Message: Invoke cleanbb via function call, not subprocess. clean_dbroot() now cleans entire /data/db if --with-cleanbb specified, (cherry picked from commit ab47b0b217ab40971a928bbe3d98bd315bbba716) | |||||||||||||||||||
| Comment by Githook User [ 08/May/14 ] | |||||||||||||||||||
|
Author: {u'username': u'kangas', u'name': u'Matt Kangas', u'email': u'matt.kangas@mongodb.com'}Message: (cherry picked from commit 8cd645a8fa0cce6313c4a15e4f3ec4fd8ac2787e) | |||||||||||||||||||
| Comment by Githook User [ 08/May/14 ] | |||||||||||||||||||
|
Author: {u'username': u'kangas', u'name': u'Matt Kangas', u'email': u'matt.kangas@mongodb.com'}Message: | |||||||||||||||||||
| Comment by Githook User [ 08/May/14 ] | |||||||||||||||||||
|
Author: {u'username': u'kangas', u'name': u'Matt Kangas', u'email': u'matt.kangas@mongodb.com'}Message: Invoke cleanbb via function call, not subprocess. clean_dbroot() now cleans entire /data/db if --with-cleanbb specified, | |||||||||||||||||||
| Comment by Githook User [ 08/May/14 ] | |||||||||||||||||||
|
Author: {u'username': u'kangas', u'name': u'Matt Kangas', u'email': u'matt.kangas@mongodb.com'}Message: | |||||||||||||||||||
| Comment by Matt Kangas [ 30/Apr/14 ] | |||||||||||||||||||
|
See attached file for data from my hacked patch build. Format is:
Suites on linux-64 that consumed more than a gigabyte after completion (values in MB)
| |||||||||||||||||||
| Comment by Randolph Tan [ 25/Apr/14 ] | |||||||||||||||||||
|
Can you try the dropdb.diff? This was actually the first one I tried, but dropDatabase actually deletes the datafiles, so it had an effect on the total runtime of the test, I believe this patch adds about 30sec more if I remember correctly. | |||||||||||||||||||
| Comment by Randolph Tan [ 24/Apr/14 ] | |||||||||||||||||||
|
mpobrien Can you try applying the patch? It passes on my machine locally runs almost the same time as the normal test run. | |||||||||||||||||||
| Comment by Michael O'Brien [ 24/Apr/14 ] | |||||||||||||||||||
|
renctan - if you want to test this with a patch, we can also extract disk usage info from your patch's system logs to assess how much of an effect it's made. |