Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17675

Support file per database for WiredTiger

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major - P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 3.7 Desired
    • Component/s: Storage, WiredTiger
    • Labels:
      None

      Description

      When the WT b-tree uses a file per collection or index then it uses a huge number of files for deployments that have a huge number of collections. I want an option to use a file per database for the WT-btree and an LSM tree per database for the WT LSM. For motivation see SERVER-17225

        Issue Links

          Activity

          Hide
          dblock Daniel Doubrovkine added a comment - - edited

          Coming from SERVER-24384, FWIW the big advantage of MongoDB is that it is (was) really cheap to bring up and tear down a database for every test, and that is what most Ruby developers using MongoDB do via libraries like database_cleaner. It's literally the default setup I've seen in just about any project. What we really want is that the default developer installation is fast for this kind of scenarios, this issue seems to possibly address that.

          Show
          dblock Daniel Doubrovkine added a comment - - edited Coming from SERVER-24384 , FWIW the big advantage of MongoDB is that it is (was) really cheap to bring up and tear down a database for every test, and that is what most Ruby developers using MongoDB do via libraries like database_cleaner. It's literally the default setup I've seen in just about any project. What we really want is that the default developer installation is fast for this kind of scenarios, this issue seems to possibly address that.
          Hide
          sallgeud Chad Kreimendahl added a comment -

          I'd be interested in simply allowing one-file per index. If we could get this configurable to do 1 file per collection and 1 file per collection's index, that would be excellent, and entirely remove any issues we currently have.

          Show
          sallgeud Chad Kreimendahl added a comment - I'd be interested in simply allowing one-file per index. If we could get this configurable to do 1 file per collection and 1 file per collection's index, that would be excellent, and entirely remove any issues we currently have.
          Hide
          sallgeud Chad Kreimendahl added a comment -

          Since there appears to be motion on this. I'd like to add more detail. Our problem right now is almost entirely the number of files created to accommodate indexes. While one file per database might be nice for some, I'd imagine that some of the performance benefits of unique file per collection would be lost. However, I would absolutely love to be able to reduce the index files from 1 per index to 1 per collection. I'm not saying merge index file into collection file... simply allow the option to have 1 file for each collections entire set of indexes.

          We average 30+ indexes per collection, so this would be huge for us.

          Show
          sallgeud Chad Kreimendahl added a comment - Since there appears to be motion on this. I'd like to add more detail. Our problem right now is almost entirely the number of files created to accommodate indexes. While one file per database might be nice for some, I'd imagine that some of the performance benefits of unique file per collection would be lost. However, I would absolutely love to be able to reduce the index files from 1 per index to 1 per collection. I'm not saying merge index file into collection file... simply allow the option to have 1 file for each collections entire set of indexes. We average 30+ indexes per collection, so this would be huge for us.
          Hide
          asya Asya Kamsky added a comment -

          Chad Kreimendahl

          We are in fact making progress towards having fewer files per database in Wired Tiger.

          Asya

          Show
          asya Asya Kamsky added a comment - Chad Kreimendahl We are in fact making progress towards having fewer files per database in Wired Tiger. Asya
          Hide
          flozano Francisco Lozano added a comment -

          Asya, that would be great news for us too. Would you mind explaining a little about what model will you follow? file per DB? file per collection+index?

          Show
          flozano Francisco Lozano added a comment - Asya, that would be great news for us too. Would you mind explaining a little about what model will you follow? file per DB? file per collection+index?
          Hide
          asya Asya Kamsky added a comment - - edited

          At the moment we are planning to have (as an option) a file per DB that would contain all collections for that DB and a second file for the indexes.

          Show
          asya Asya Kamsky added a comment - - edited At the moment we are planning to have (as an option) a file per DB that would contain all collections for that DB and a second file for the indexes.
          Hide
          flozano Francisco Lozano added a comment -

          That would be amazing. Really.

          Show
          flozano Francisco Lozano added a comment - That would be amazing. Really.

            People

            • Votes:
              19 Vote for this issue
              Watchers:
              35 Start watching this issue

              Dates

              • Created:
                Updated: