[SERVER-4534] Index creation opens too many files for large array Created: 20/Dec/11 Updated: 11/Jul/16 Resolved: 10/Jul/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Index Maintenance |
| Affects Version/s: | None |
| Fix Version/s: | 2.5.1 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Mathias Stearn | Assignee: | Mathias Stearn |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
Background indexing is ok, forground inding crashes. Tue Dec 20 14:48:40 [conn2] build index test.huge_multikey_index { a: 1.0 }Tue Dec 20 14:48:45 [conn2] couldn't open /data/db/sconsTests/_tmp/esort.1324410520.1552787748//file.1010 errno:24 Too many open files slowNightly test will be committed shortly Fails with 1.8, 2.0, and master. |
| Comments |
| Comment by auto [ 10/Jul/13 ] |
|
Author: {u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}Message: Enable test for It was fixed when index building switched to using the new Sorter. |
| Comment by Mathias Stearn [ 27/Jun/13 ] |
|
Just need to reenable huge_multikey_index.js. Already passes after extsort refacter |
| Comment by Eliot Horowitz (Inactive) [ 21/Mar/12 ] |
|
Can you set ulimit to unlimited? |
| Comment by Paul Wang [ 21/Mar/12 ] |
|
Have the same problem, working on a collection size of 1 billion with 4 fields. Trying to create a single index on 3 fields, getting to 99% complete them the error message above hits. Some people suggested to increase the ulimit -n 10000, but still fails. Any luck on solving this problem yet, the index cannot be performed in the background since it will take too long. Thanks in advance, |
| Comment by Mathias Stearn [ 23/Dec/11 ] |
|
After some examination, this only effects collections with < 1,000,000 objects or an average of > 100 elements per array. This is because we will use the smaller of 1M or db.foo.count()+100 as the size of the InMemory array in ExternalSort. This has bad effects for small collections with huge arrays like this test. |
| Comment by auto [ 20/Dec/11 ] |
|
Author: {u'login': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}Message: Disabled test for |