[SERVER-6414] use regular file io, not mmap for external sort Created: 11/Jul/12 Updated: 11/Jul/16 Resolved: 11/Jul/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | None |
| Fix Version/s: | 2.0.7, 2.2.0-rc0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Eliot Horowitz (Inactive) | Assignee: | Eliot Horowitz (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
much better for memory consumption and performance |
| Comments |
| Comment by auto [ 24/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-23T11:42:15-07:00', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}Message: Align #ifdef with the code under #ifdef. | |||
| Comment by Andy Schwerin [ 12/Jul/12 ] | |||
|
Propopsed repro steps, in the mongo shell:
> for (i=0; i < 10000000; ++i) { arr = []; for (j = 0; j < 40; ++j) arr.push(i + j); db.foo.insert( {a: arr}); } ); The expected behavior is that the index gets built, and that phase 2 of index building (as observed by the mongod log) takes just as long to handle the first 10% of records as the last 10%. The failing behaviors are for the process to get OOM killed, for vmstat to report a much higher block-in and block-out rate late in phase 2 compared to early in phase 2 (10x or more), and/or for phase 2 progress to "slow down" (take more time per key) later in the phase. While running, it's a good idea to watch vmstat, and keep an eye on approximate heap size and private-dirty data size. The latter two can be observed with the following shell script, which takes the PID of mongod as its first parameter.
If you name the script heapwatch, and XXX is the pid of your mongod process, running "watch heapwatch XXX" is useful. | |||
| Comment by auto [ 12/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-12T10:38:00-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}Message: | |||
| Comment by auto [ 12/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-12T10:38:00-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}Message: | |||
| Comment by auto [ 11/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-11T16:12:26-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: | |||
| Comment by auto [ 11/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-11T16:12:26-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: | |||
| Comment by auto [ 11/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-11T16:08:37-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: | |||
| Comment by auto [ 11/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-11T16:08:37-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: | |||
| Comment by auto [ 11/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-11T15:31:14-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: | |||
| Comment by auto [ 11/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-11T15:31:14-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: | |||
| Comment by auto [ 11/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-11T15:24:51-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: | |||
| Comment by auto [ 11/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-11T15:24:51-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: | |||
| Comment by auto [ 11/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-11T15:20:13-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: | |||
| Comment by auto [ 11/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-11T15:20:13-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}Message: | |||
| Comment by auto [ 11/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-11T12:23:59-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: Conflicts: db/extsort.cpp | |||
| Comment by auto [ 11/Jul/12 ] | |||
|
Author: {u'date': u'2012-07-11T12:23:59-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}Message: |