Details
-
Type:
New Feature
-
Status: Open
-
Priority:
Major - P3
-
Resolution: Unresolved
-
Affects Version/s: None
-
Fix Version/s: 3.7 Desired
-
Component/s: Storage, WiredTiger
-
Labels:None
-
Epic Link:
-
Sprint:Storage 2017-07-31
Description
When the WT b-tree uses a file per collection or index then it uses a huge number of files for deployments that have a huge number of collections. I want an option to use a file per database for the WT-btree and an LSM tree per database for the WT LSM. For motivation see SERVER-17225
Issue Links
- is duplicated by
-
SERVER-24384
Extremely slow writes on OSX with WiredTiger and constant collection/index creation+drop
-
- Closed
-
-
SERVER-18812
Improve aggregation with $out performance on WiredTiger
-
- Closed
-
- is related to
-
SERVER-25025
Improve startup time when there are tens of thousands of collections/indexes on WiredTIger
-
- Open
-
-
SERVER-21067
Investigate having the storage size of a given collection/index/database cached within the WT/KV api
-
- Open
-
- related to
-
SERVER-22642
WiredTiger engine resync stalls with a lot of tables/indexes
-
- Closed
-
Activity
I'd be interested in simply allowing one-file per index. If we could get this configurable to do 1 file per collection and 1 file per collection's index, that would be excellent, and entirely remove any issues we currently have.
Since there appears to be motion on this. I'd like to add more detail. Our problem right now is almost entirely the number of files created to accommodate indexes. While one file per database might be nice for some, I'd imagine that some of the performance benefits of unique file per collection would be lost. However, I would absolutely love to be able to reduce the index files from 1 per index to 1 per collection. I'm not saying merge index file into collection file... simply allow the option to have 1 file for each collections entire set of indexes.
We average 30+ indexes per collection, so this would be huge for us.
We are in fact making progress towards having fewer files per database in Wired Tiger.
Asya
Asya, that would be great news for us too. Would you mind explaining a little about what model will you follow? file per DB? file per collection+index?
At the moment we are planning to have (as an option) a file per DB that would contain all collections for that DB and a second file for the indexes.
That would be amazing. Really.
Coming from
SERVER-24384, FWIW the big advantage of MongoDB is that it is (was) really cheap to bring up and tear down a database for every test, and that is what most Ruby developers using MongoDB do via libraries like database_cleaner. It's literally the default setup I've seen in just about any project. What we really want is that the default developer installation is fast for this kind of scenarios, this issue seems to possibly address that.