[SERVER-6414] use regular file io, not mmap for external sort Created: 11/Jul/12  Updated: 11/Jul/16  Resolved: 11/Jul/12

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 2.0.7, 2.2.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Eliot Horowitz (Inactive) Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-6141 can't successfully replicate our shar... Closed
is duplicated by SERVER-5312 Resync fails due to out of memory Closed
Related
Participants:

 Description   

much better for memory consumption and performance



 Comments   
Comment by auto [ 24/Jul/12 ]

Author:

{u'date': u'2012-07-23T11:42:15-07:00', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-6414 make #ifdef test a symbol it is using

Align #ifdef with the code under #ifdef.
Branch: master
https://github.com/mongodb/mongo/commit/01389e7f3235a413002b57b727613532995bd489

Comment by Andy Schwerin [ 12/Jul/12 ]

Propopsed repro steps, in the mongo shell:

 

> for (i=0; i < 10000000; ++i) { arr = []; for (j = 0; j < 40; ++j) arr.push(i + j); db.foo.insert(

{a: arr}

); }
> db.foo.ensureIndex(

{a:1}

);

The expected behavior is that the index gets built, and that phase 2 of index building (as observed by the mongod log) takes just as long to handle the first 10% of records as the last 10%.

The failing behaviors are for the process to get OOM killed, for vmstat to report a much higher block-in and block-out rate late in phase 2 compared to early in phase 2 (10x or more), and/or for phase 2 progress to "slow down" (take more time per key) later in the phase.

While running, it's a good idea to watch vmstat, and keep an eye on approximate heap size and private-dirty data size. The latter two can be observed with the following shell script, which takes the PID of mongod as its first parameter.

(echo "import sys; sys.stdout.write('Heap %dMB; ' % (("; grep -E 'r[-w]-p [0-9]{8} 00' /proc/$1/maps | cut -d ' ' -f 1 | sed -E 's/^(.*)-(.*)$/(0x\2 - 0x\1) + /'; echo "0) / 1024 / 1024))") | python 
(echo "import sys; sys.stdout.write('Private_Dirty %dMB' % (("; grep Private_Dirty /proc/$1/smaps | sed -E 's/^.* ([0-9]+) .*$/\1 +/' ; echo '0) /1024))') | python

If you name the script heapwatch, and XXX is the pid of your mongod process, running "watch heapwatch XXX" is useful.

Comment by auto [ 12/Jul/12 ]

Author:

{u'date': u'2012-07-12T10:38:00-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}

Message: SERVER-6414 - fixed posix_fadvise call
Branch: v2.0
https://github.com/mongodb/mongo/commit/accb562a78780160e6844863193960b4c9ecc8ca

Comment by auto [ 12/Jul/12 ]

Author:

{u'date': u'2012-07-12T10:38:00-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}

Message: SERVER-6414 - fixed posix_fadvise call
Branch: master
https://github.com/mongodb/mongo/commit/ff94f30403ebfbd14c92a2ded908adb377a9afbd

Comment by auto [ 11/Jul/12 ]

Author:

{u'date': u'2012-07-11T16:12:26-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-6414 - windows compile part 5
Branch: v2.0
https://github.com/mongodb/mongo/commit/8e584c2fe8d29040692d0e44bda695df68e99838

Comment by auto [ 11/Jul/12 ]

Author:

{u'date': u'2012-07-11T16:12:26-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-6414 - windows compile part 5
Branch: master
https://github.com/mongodb/mongo/commit/2784aa1252157e2305539eec65d0fca3c9b803d1

Comment by auto [ 11/Jul/12 ]

Author:

{u'date': u'2012-07-11T16:08:37-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-6414 - windows compile part 4
Branch: v2.0
https://github.com/mongodb/mongo/commit/e67589e105d4ed85670e9dd264d0689223d0d324

Comment by auto [ 11/Jul/12 ]

Author:

{u'date': u'2012-07-11T16:08:37-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-6414 - windows compile part 4
Branch: master
https://github.com/mongodb/mongo/commit/d44927603e299acb940165d47954efc126774d4b

Comment by auto [ 11/Jul/12 ]

Author:

{u'date': u'2012-07-11T15:31:14-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-6414 - windows compile part 3
Branch: v2.0
https://github.com/mongodb/mongo/commit/c6a40c4423762f742bc5f643b169b1bbc612ee3b

Comment by auto [ 11/Jul/12 ]

Author:

{u'date': u'2012-07-11T15:31:14-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-6414 - windows compile part 3
Branch: master
https://github.com/mongodb/mongo/commit/cd76131c99e6b335b1cb305f190680debc50acac

Comment by auto [ 11/Jul/12 ]

Author:

{u'date': u'2012-07-11T15:24:51-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-6414 - windows compile part 2
Branch: v2.0
https://github.com/mongodb/mongo/commit/ea06d5d59af1b41f2f9d40a74dc77ea98aa4ecd2

Comment by auto [ 11/Jul/12 ]

Author:

{u'date': u'2012-07-11T15:24:51-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-6414 - windows compile part 2
Branch: master
https://github.com/mongodb/mongo/commit/0f29045081d94a36a2f6b494b2f2108c3f4ed4ce

Comment by auto [ 11/Jul/12 ]

Author:

{u'date': u'2012-07-11T15:20:13-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-6414 - windows compile
Branch: v2.0
https://github.com/mongodb/mongo/commit/87c31eb85c48d42e203cdcaf557c2e94c59d4e7d

Comment by auto [ 11/Jul/12 ]

Author:

{u'date': u'2012-07-11T15:20:13-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}

Message: SERVER-6414 - windows compile
Branch: master
https://github.com/mongodb/mongo/commit/7b3330420467f97d7be4602efdad875ce224745a

Comment by auto [ 11/Jul/12 ]

Author:

{u'date': u'2012-07-11T12:23:59-07:00', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-6414 - use regular file io, not mmap for external sort

Conflicts:

db/extsort.cpp
db/extsort.h
Branch: v2.0
https://github.com/mongodb/mongo/commit/6b24994f08a37739b359aed6950f87286aa562df

Comment by auto [ 11/Jul/12 ]

Author:

{u'date': u'2012-07-11T12:23:59-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}

Message: SERVER-6414 - use regular file io, not mmap for external sort
Branch: master
https://github.com/mongodb/mongo/commit/f74ba19a1e773bd2cdf578c78c316bb4418dafb6

Generated at Thu Feb 08 03:11:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.