[SERVER-1140] Support 64 (or more) indexes Created: 21/May/10 Updated: 07/Jun/18 Resolved: 26/May/10 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 1.5.2 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Doug Green | Assignee: | Dwight Merriman |
| Resolution: | Done | Votes: | 2 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Participants: | |||||||||||||
| Description |
|
The current code supports up to 40 indexes per collection. I think that we can support 64 without too much extra work, and much much more if you're willing to use some of the reserved bytes in Namespace. AFAICT, the 40 index restriction is twofold: 1. multiKeyIndexBits is a bitwise storage of the multi-key state for each index, and it uses an unsigned long long, limiting to 64 indexes 2. the first 10 indexes are stored in the Namespace, the next 30 are stored in an extra block, currently limiting to 40 total indexes, however forethought was given to this, and there exists the capability to chain additional extra blocks. The attached patch implements the extra block chaining. It accounts for the possibility of 10 more "extra" blocks by naming them $extra, $extr0, $extr1, $extr2 ... $extr9. You could actually get another 300 using this, for a hard limit of 340. However, because of the multiKeyIndexBits limitation, we're really limited to just 64 indexes. I implemented the full chaining for future expansion. If you're interested in 128 indexes, just take 8 or more bytes from the reserved array, and create a multiKeyIndexMoreBits[] and carry the logic into the multiKey functions. I did not do this because (a) the extra 24 indexes should be enough for our needs, (b) those reserved bytes are precious few and once you use them, they're gone, (c) I know we're crazy to want 64 indexes, and I think whoever wants 128 is even crazier. This patch is not yet complete. It dies in db/pdfile.h +284 on an assertion "ofs >= headerSize()". But I've only spent 2.5 hours on this so far, and it's late, and I wanted to get my work in progress in front of others to review and comment. |
| Comments |
| Comment by Ramon Fernandez Marina [ 07/Jun/18 ] |
|
cade@hyperblaster.org, the request to allow more than 64 indexes is being tracked in SERVER-12250 – please feel free to comment on that ticket and vote for it. Thanks, |
| Comment by Cade Embery [ 26/May/18 ] |
|
Any news on this? Is there anything that can be done to lift this limit? |
| Comment by Cade Embery [ 11/Feb/18 ] |
|
Is there any chance of the 64 index limit being upgraded? We really need the ability for more indexes |
| Comment by Benoit Jacquemont [X] [ 27/Oct/14 ] |
|
Hey people, I have several projects (around 20) using MongoDB with document having on average 200 fields and at max 2000 fields. We have between 500.000 and 1 million documents on average. We have a customisable grid where the user can choose which column to display and on which field to filter. With the 64 indexes limit, it's a real pain for the user to apply filter on field that cannot benefit from the indexes as the index stock is exhausted and the full collection scan is of course really slow. We are thinking about using ElasticSearch to handle this grid, but it's really a big addition that brings its own problem (index synchronization with MongoDB for example) and where we wont benefit from most of the features (no need for full text index for example). We cannot split our collection as well because our documents are really flat and there's no logical way to cut them into smaller pieces. Moreover all the structure is really flexible and defined by the user, so a post splitting is really difficult. Any idea on what could be done to overcome this problem ? I suppose there's nothing planned to support more than 64 indexes in a collection ? Thanks a lot for your answer ! |
| Comment by Wiliam [ 24/Jan/14 ] |
|
Thanks for your reply Dwight. One year ago we started our project using MongoDb, we namespaced at a key level our main collection (entity.NAMESPACE.key) and that was our fault. Easily we reached 64 indexes so I was analyzing the options to fix this, refactoring the project or changing the limit of indexes. |
| Comment by Dwight Merriman [ 24/Jan/14 ] |
|
@William it wouldn't be that hard to do, but there hasn't been a lot of interest especially given multikey and full text features, so it has not been a priority. in addition at higher #s i'd want to verify that the query optimizer still works as well as it should so that might be the biggest job actually. personally i'd consider different schemas if I were getting to these levels, I haven't myself had a situation where i needed that many indexes on one collection. |
| Comment by Wiliam [ 24/Jan/14 ] |
|
What is the reason for limiting to 64 indexes? It's technicaly possible to add more than 64 indexes? 128? 512? |
| Comment by Nik Kolev [ 23/Oct/12 ] |
|
A full text search won't help. For now we are using a number of arrays where each array represents the index for a bucket of the fields we need to index on. However as the nature of the documents we store evolves we would bump into the limit again. Hopefully you guys can put removing this limit on your roadmap. |
| Comment by Dwight Merriman [ 27/Sep/12 ] |
|
nothing at the moment. |
| Comment by Nik Kolev [ 26/Sep/12 ] |
|
Any plans on adding support for the (more) part? |
| Comment by Doug Green [ 15/Jun/10 ] |
|
Thanks Dwight! Here's the 1.4.3 backport, which we'll be testing heavily over the coming days. |
| Comment by Eliot Horowitz (Inactive) [ 26/May/10 ] |
|
seems to be working as far as i can tell |
| Comment by Dwight Merriman [ 23/May/10 ] |
|
code committed that hopefully works up to 64 indexes. give it a try and let me know. |
| Comment by auto [ 22/May/10 ] |
|
Author: {'login': 'dwight', 'name': 'Dwight Merriman', 'email': 'dwight@10gen.com'}Message: towards http://jira.mongodb.org/browse/SERVER-1140 |
| Comment by Doug Green [ 21/May/10 ] |
|
This fixes kill_ns leaks caused by passing the wrong input. I really need to learn how to submit this with git |
| Comment by Doug Green [ 21/May/10 ] |
|
Updated patch ... I think this works. |