[SERVER-5438] Segmentation fault. Invalid access at address Created: 29/Mar/12  Updated: 08/Mar/13  Resolved: 01/Oct/12

Status: Closed
Project: Core Server
Component/s: MapReduce
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Azat Khuzhin Assignee: Greg Studer
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Mongodb. 2.1.1-pre- (v8)
Linux 3.0.0-1-amd64 #1 SMP Sat Aug 27 16:21:11 UTC 2011 x86_64 GNU/Linux


Operating System: ALL
Participants:

 Description   

Wed Mar 28 04:52:12 [conn186070]                30437000/36561854       83%
Wed Mar 28 04:52:13 Invalid access at address: 0x178164c00058 from thread: conn186070
 
Wed Mar 28 04:52:13 Got signal: 11 (Segmentation fault).
 
Wed Mar 28 04:52:13 Backtrace:
0x585a7f 0x57c700 0x57ca0f 0x7f4f1580d020 0xa8c668 0xa8e8b7 0xa90121 0xa9019f 0xa216a8 0xa21bdc 0xa223fb 0x9e8443 0x2a0175c06302 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x1f) [0x585a7f]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x350) [0x57c700]
 /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x20f) [0x57ca0f]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0xf020) [0x7f4f1580d020]
 /usr/bin/mongod() [0xa8c668]
 /usr/bin/mongod() [0xa8e8b7]
 /usr/bin/mongod() [0xa90121]
 /usr/bin/mongod() [0xa9019f]
 /usr/bin/mongod() [0xa216a8]
 /usr/bin/mongod() [0xa21bdc]
 /usr/bin/mongod() [0xa223fb]
 /usr/bin/mongod() [0x9e8443]
 [0x2a0175c06302]
 
Logstream::get called in uninitialized state
Wed Mar 28 04:52:13 [conn186070] ERROR: Client::~Client _context should be null but is not; client:conn
Logstream::get called in uninitialized state
Wed Mar 28 04:52:13 [conn186070] ERROR: Client::shutdown not called: conn

Wed Mar 28 04:52:12 [conn186070] 30437000/36561854 83% - is my MR job, that run's about 1.5 days



 Comments   
Comment by Greg Studer [ 01/Oct/12 ]

Issue is from an rc - feel free to reopen if you see this in the 2.2 stable release.

Comment by Azat Khuzhin [ 03/May/12 ]

I understand for what we need sorting by emit key.

But I'v already tried sort by "key", but with sorting MR is slower than without.
I think that it because of concurrent reads/writes from/to collection (~60 q/s), it increase IO and sort is not help with this

Comment by Antoine Girbal [ 27/Apr/12 ]

The idea behind the sort is to try to fully reduce each key before writing to the temporary collection on disk, which reduces IO.
Right now you are emitting on "this.key + '_' + date" so in theory sorting by "key" or "key, date" should work well.
This is assuming that:

  • there is an index on it
  • there are many different "key" values with random distribution, but not too many different dates per key (fits in 100KB).
    Now if it does not help in your case, it would be interesting to understand why.
Comment by Azat Khuzhin [ 22/Apr/12 ]

I have concurrent read/write (find() / update()) to this database, so superfluous disk IO can`t be removed

Comment by Antoine Girbal [ 20/Apr/12 ]

A few things that could speed up MR:

  • try not to compute values within JS, it's better to precompute things and store in documents.
    For example hex_md5(' {' + this.userAgent + '_' + this.ip + '}

    ')

  • Just looking at map function there are some unnecessary objects (dateOb) and some code can definitely be optimized.
    If values get used several times, it's better to store in a local variable.
    Basically since this code is executed for every document, any optimization will help.
  • add a sort by "key" to MR, if you have an index on key.
    This will speed things up a lot by removing superfluous disk IO.
    It is especially useful when dealing with a large working set.
Comment by Azat Khuzhin [ 12/Apr/12 ]

Final reduce to collection (stage 3/3) is very SLOW too
~300 in 10 secs
80 000 000 = 3days

Comment by Azat Khuzhin [ 11/Apr/12 ]

My map() stage is not so fast, reduce() satisfies me

var map = function() {
        var
                time = this._id.getTimestamp(),
                date = time.getFullYear() + '-' + (time.getMonth() + 1) + '-' + time.getDate(),
                dateOb = new Date(time.getFullYear(), time.getMonth(), time.getDate());
 
        // compose emit value
        var emitValue = {
                timestamp: Math.floor(dateOb.getTime() / 1000), // milliseconds to seconds
                count: 1,
                uCount: 1,
                uKeys: [hex_md5('{' + this.userAgent + '_' + this.ip + '}')],
                countries: {
                }
        };
        // countries
        emitValue.countries[this.countryCode] = {
                name: this.countryName,
                count: 1,
                uCount: 1,
                uKeys: [hex_md5('{' + this.userAgent + '_' + this.ip + '}')]
        };
 
        emit(this.key + '_' + date, emitValue);
}

var reduce = function(k, vals) {
        // avoid references
        function clone(o) {
                if (!o || 'object' !== typeof o)  {
                        return o;
                }
                var c = 'function' === typeof o.pop ? [] : {};
                var p, v;
                for (p in o) {
                        if (o.hasOwnProperty(p)) {
                                v = o[p];
                                if (v && 'object' === typeof v) {
                                        c[p] = clone(v);
                                } else {
                                        c[p] = v;
                                }
                        }
                }
                return c;
        }
 
        // @see test/mapreduce/mongodb-bug-pop-at-mr.js - @TODO
        var result = clone(vals[0]);
        vals.shift();
 
        for (valIndex in vals) {
                var current = vals[valIndex];
 
                result.count += current.count;
                // unique counter
                for (keyIndex in current.uKeys) {
                        // not found -> increment unique values
                        if (result.uKeys.indexOf(current.uKeys[keyIndex]) == -1) {
                                ++result.uCount;
                                result.uKeys.push(current.uKeys[keyIndex]);
                        }
                }
 
                // countries
                for (countryCode in current.countries) {
                        // not found - just push current
                        if (!result.countries[countryCode]) {
                                result.countries[countryCode] = clone(current.countries[countryCode]);
                                continue;
                        }
 
                        result.countries[countryCode].count += current.countries[countryCode].count;
                        // unique per country counter
                        for (keyIndex in current.countries[countryCode].uKeys) {
                                // not found -> increment unique values
                                if (result.countries[countryCode].uKeys.indexOf(current.countries[countryCode].uKeys[keyIndex]) == -1) {
                                        ++result.countries[countryCode].uCount;
                                        result.countries[countryCode].uKeys.push(current.uKeys[keyIndex]);
                                }
                        }
                }
        }
        return result;
}

db.stat.insert({ "_id" : ObjectId("4f7edbce3554948f4300000e"), "countryCode" : "US", "countryName" : "USA", "ip" : "127.0.0.1", "key" : "a6f8a1050544ba2a651de174b4bf6ff0", "userAgent" : "Some browser" });

Comment by Azat Khuzhin [ 11/Apr/12 ]

I don`t have several MR jobs, I have one MR job & concurrent find/update/insert's - and it handled very slow, by process quick (nsecs from explain show this)
I already speed up my MR, drop `sort` flag and it runs 6-7 times faster (you probably want to update you doc about MR for this case)

I'l try to post here my MR job in a few hours

Comment by Antoine Girbal [ 11/Apr/12 ]

do you run several MR jobs concurrently in different connections?
There is currently a ticket that is being worked on, to fix certain native calls that are not thread safe.

Could you attach the map and reduce code, and possibly a sample document?
There may be ways to speed up the MR process.

Comment by Azat Khuzhin [ 10/Apr/12 ]

Trying to restart server

VIRT ~144G - was ~800M, after trying to restart

And next log: (it not stoped, until i run `pkill -9 mongod`)

Tue Apr 10 07:00:16 [conn49843] end connection 127.0.0.1:32969 (68 connections now open)
Tue Apr 10 07:00:16 [conn49880] end connection 127.0.0.1:33003 (67 connections now open)
Tue Apr 10 07:00:17 [interruptThread] shutdown: closing all files...
Tue Apr 10 07:00:17 [conn50195] end connection 127.0.0.1:33484 (66 connections now open)
Tue Apr 10 07:00:17 [interruptThread] closeAllFiles() finished
Tue Apr 10 07:00:17 [interruptThread] journalCleanup...
Tue Apr 10 07:00:17 [interruptThread] removeJournalFiles
Tue Apr 10 07:00:17 [conn49852] end connection 127.0.0.1:32977 (65 connections now open)
Tue Apr 10 07:00:17 [interruptThread] shutdown: removing fs lock...
Tue Apr 10 07:00:17 dbexit: really exiting now
Logstream::get called in uninitialized state
Tue Apr 10 07:00:17 [conn49840] end connection 127.0.0.1:32966 (64 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:17 [conn49840] boost assertion failure !pthread_mutex_lock(m) boost::pthread::pthread_mutex_scoped_lock::pthread_mutex_scoped_lock(pthread_mutex_t*) src/third_party/boost/boost/thread/pthread/pthread_mutex_scoped_lock.hpp:26
Logstream::get called in uninitialized state
Tue Apr 10 07:00:17 [conn49840]   Fatal Assertion 16108
0x8e5c78 0x9d311c 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x98) [0x8e5c78]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x5c) [0x9d311c]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
Logstream::get called in uninitialized state
Tue Apr 10 07:00:17 [conn49840] 
 
***aborting after fassert() failure
 
 
Tue Apr 10 07:00:17 Got signal: 6 (Aborted).
 
Tue Apr 10 07:00:17 Backtrace:
0x59383f 0x589e20 0x7fd30affa480 0x7fd30affa405 0x7fd30affd680 0x9d3181 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x1f) [0x59383f]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x350) [0x589e20]
 /lib/x86_64-linux-gnu/libc.so.6(+0x32480) [0x7fd30affa480]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fd30affa405]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7fd30affd680]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xc1) [0x9d3181]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
 
Tue Apr 10 07:00:17 Invalid access at address: 0x6912abf0 from thread: 
 
Tue Apr 10 07:00:17 Got signal: 11 (Segmentation fault).
 
*** glibc detected *** /usr/bin/mongod: corrupted double-linked list: 0x00000000032f2090 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x72606)[0x7fd30b03a606]
/lib/x86_64-linux-gnu/libc.so.6(+0x72a3d)[0x7fd30b03aa3d]
/lib/x86_64-linux-gnu/libc.so.6(+0x73f38)[0x7fd30b03bf38]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7fd30b03f33c]
/usr/bin/mongod(_ZN5boost6detail12shared_countD1Ev+0x49)[0x593559]
/usr/bin/mongod(_ZNSt8_Rb_treeISsSt4pairIKSsN5boost10shared_ptrIN5mongo25NamespaceDetailsTransientEEEESt10_Select1stIS7_ESt4lessISsESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E+0x214)[0x5e7074]
/lib/x86_64-linux-gnu/libc.so.6(+0x36d82)[0x7fd30affed82]
/lib/x86_64-linux-gnu/libc.so.6(+0x36dd5)[0x7fd30affedd5]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x389)[0x589e59]
/lib/x86_64-linux-gnu/libc.so.6(+0x32480)[0x7fd30affa480]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7fd30affa405]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x180)[0x7fd30affd680]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xc1)[0x9d3181]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b)[0x5aed7b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40)[0x7fd30baf4b40]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fd30b09d36d]
======= Memory map: ========
00400000-00bc9000 r-xp 00000000 08:03 797527                             /usr/bin/mongod
00dc9000-00df2000 rw-p 007c9000 08:03 797527                             /usr/bin/mongod
00df2000-01bc9000 rw-p 00000000 00:00 0 
032dd000-061bd000 rw-p 00000000 00:00 0                                  [heap]
7fb49f6b7000-7fb49f6b8000 ---p 00000000 00:00 0 
7fb49f6b8000-7fb49f7b8000 rw-p 00000000 00:00 0 
7fb4a4000000-7fb4a8000000 rw-p 00000000 00:00 0 
7fb4b4e13000-7fb4b4e14000 ---p 00000000 00:00 0 
7fb4b4e14000-7fb4b4f14000 rw-p 00000000 00:00 0 
7fb4d0000000-7fb4d221c000 rw-p 00000000 00:00 0 
7fb4d221c000-7fb4d4000000 ---p 00000000 00:00 0 
7fb4d58d9000-7fb4d58da000 ---p 00000000 00:00 0 
7fb4d58da000-7fb4d59da000 rw-p 00000000 00:00 0 
7fb4d65e6000-7fb4d65e7000 ---p 00000000 00:00 0 
7fb4d65e7000-7fb4d66e7000 rw-p 00000000 00:00 0 
7fb4d6ff0000-7fb4d6ff1000 ---p 00000000 00:00 0 
7fb4d6ff1000-7fb4d70f1000 rw-p 00000000 00:00 0 
7fb4d8000000-7fb4db80f000 rw-p 00000000 00:00 0 
7fb4db80f000-7fb4dc000000 ---p 00000000 00:00 0 
7fb4df4f5000-7fb4df4f6000 ---p 00000000 00:00 0 
7fb4df4f6000-7fb4df5f6000 rw-p 00000000 00:00 0 
7fb4e0000000-7fb4e227e000 rw-p 00000000 00:00 0 
7fb4e227e000-7fb4e4000000 ---p 00000000 00:00 0 
7fb4e4000000-7fb4e63e3000 rw-p 00000000 00:00 0 
7fb4e63e3000-7fb4e8000000 ---p 00000000 00:00 0 
7fb4e8000000-7fb4ebfcb000 rw-p 00000000 00:00 0 
7fb4ebfcb000-7fb4ec000000 ---p 00000000 00:00 0 
7fb4ec000000-7fb4ee56a000 rw-p 00000000 00:00 0 
7fb4ee56a000-7fb4f0000000 ---p 00000000 00:00 0 
7fb4f0000000-7fb4f3fff000 rw-p 00000000 00:00 0 
7fb4f3fff000-7fb4f4000000 ---p 00000000 00:00 0 
7fb4f6021000-7fb4f6022000 ---p 00000000 00:00 0 
7fb4f6022000-7fb4f6122000 rw-p 00000000 00:00 0 
7fb4f6728000-7fb4f6729000 ---p 00000000 00:00 0 
7fb4f6729000-7fb4f6829000 rw-p 00000000 00:00 0 
7fb4f7e46000-7fb4f7e47000 ---p 00000000 00:00 0 
7fb4f7e47000-7fb4f8647000 rw-p 00000000 00:00 0 
7fb4f8b4c000-7fb4f8b4d000 ---p 00000000 00:00 0 
7fb4f8b4d000-7fb4f8c4d000 rw-p 00000000 00:00 0 
7fb4f8d4e000-7fb4f8d4f000 ---p 00000000 00:00 0 
7fb4f8d4f000-7fb4f8e4f000 rw-p 00000000 00:00 0 
7fb4f9354000-7fb4f9355000 ---p 00000000 00:00 0 
7fb4f9355000-7fb4f9455000 rw-p 00000000 00:00 0 
7fb4f995a000-7fb4f995b000 ---p 00000000 00:00 0 
7fb4f995b000-7fb4f9a5b000 rw-p 00000000 00:00 0 
7fb4f9b5c000-7fb4f9b5d000 ---p 00000000 00:00 0 
7fb4f9b5d000-7fb4f9c5d000 rw-p 00000000 00:00 0 
7fb4f9d5e000-7fb4f9d5f000 ---p 00000000 00:00 0 
7fb4f9d5f000-7fb4f9e5f000 rw-p 00000000 00:00 0 
7fb4fa061000-7fb4fa062000 ---p 00000000 00:00 0 
7fb4fa062000-7fb4fa162000 rw-p 00000000 00:00 0 
7fb4fa263000-7fb4fa264000 ---p 00000000 00:00 0 
7fb4fa264000-7fb4fa364000 rw-p 00000000 00:00 0 
7fb4fa465000-7fb4fa466000 ---p 00000000 00:00 0 
7fb4fa466000-7fb4fa566000 rw-p 00000000 00:00 0 
7fb4fa566000-7fb4fa567000 ---p 00000000 00:00 0 
7fb4fa567000-7fb4fa667000 rw-p 00000000 00:00 0 
7fb4faa6b000-7fb4faa6c000 ---p 00000000 00:00 0 
7fb4faa6c000-7fb4fab6c000 rw-p 00000000 00:00 0 
7fb4fac6d000-7fb4fac6e000 ---p 00000000 00:00 0 
7fb4fac6e000-7fb4fad6e000 rw-p 00000000 00:00 0 
7fb4fb172000-7fb4fb173000 ---p 00000000 00:00 0 
7fb4fb173000-7fb4fb273000 rw-p 00000000 00:00 0 
7fb4fb576000-7fb4fb577000 ---p 00000000 00:00 0 
7fb4fb577000-7fb4fb677000 rw-p 00000000 00:00 0 
7fb4fb778000-7fb4fb779000 ---p 00000000 00:00 0 
7fb4fb779000-7fb4fb879000 rw-p 00000000 00:00 0 
7fb4fb97a000-7fb4fb97b000 ---p 00000000 00:00 0 
7fb4fb97b000-7fb4fba7b000 rw-p 00000000 00:00 0 
7fb4fba7b000-7fb4fba7c000 ---p 00000000 00:00 0 
7fb4fba7c000-7fb4fbb7c000 rw-p 00000000 00:00 0 
7fb4fbe7f000-7fb4fbe80000 ---p 00000000 00:00 0 
7fb4fbe80000-7fb4fbf80000 rw-p 00000000 00:00 0 
7fb4fc182000-7fb4fc183000 ---p 00000000 00:00 0 
7fb4fc183000-7fb4fc283000 rw-p 00000000 00:00 0 
7fb4fc889000-7fb4fc88a000 ---p 00000000 00:00 0 
7fb4fc88a000-7fb4fc98a000 rw-p 00000000 00:00 0 
7fb4fc98a000-7fb4fc98b000 ---p 00000000 00:00 0 
7fb4fc98b000-7fb4fca8b000 rw-p 00000000 00:00 0 
7fb4fcd8e000-7fb4fcd8f000 ---p 00000000 00:00 0 
7fb4fcd8f000-7fb4fce8f000 rw-p 00000000 00:00 0 
7fb4fd293000-7fb4fd294000 ---p 00000000 00:00 0 
7fb4fd294000-7fb4fd394000 rw-p 00000000 00:00 0 
7fb4fdd9e000-7fb4fdd9f000 ---p 00000000 00:00 0 
7fb4fdd9f000-7fb4fde9f000 rw-p 00000000 00:00 0 
7fb4fe1a2000-7fb4fe1a3000 ---p 00000000 00:00 0 
7fb4fe1a3000-7fb4fe2a3000 rw-p 00000000 00:00 0 
7fb4fe3a4000-7fb4fe3a5000 ---p 00000000 00:00 0 
7fb4fe3a5000-7fb4fe4a5000 rw-p 00000000 00:00 0 
7fb4fe4a5000-7fb4fe4a6000 ---p 00000000 00:00 0 
7fb4fe4a6000-7fb4fe5a6000 rw-p 00000000 00:00 0 
7fb4fe6a7000-7fb4fe6a8000 ---p 00000000 00:00 0 
7fb4fe6a8000-7fb4fe7a8000 rw-p 00000000 00:00 0 
7fb4fe7a8000-7fb4fe7a9000 ---p 00000000 00:00 0 
7fb4fe7a9000-7fb4fe8a9000 rw-p 00000000 00:00 0 
7fb4fe8a9000-7fb4fe8aa000 ---p 00000000 00:00 0 
7fb4fe8aa000-7fb4fe9aa000 rw-p 00000000 00:00 0 
7fb4fecad000-7fb4fecae000 ---p 00000000 00:00 0 
7fb4fecae000-7fb4fedae000 rw-p 00000000 00:00 0 
7fb4feeaf000-7fb4feeb0000 ---p 00000000 00:00 0 
7fb4feeb0000-7fb4fefb0000 rw-p 00000000 00:00 0 
7fb4ff1b2000-7fb4ff1b3000 ---p 00000000 00:00 0 
7fb4ff1b3000-7fb4ff2b3000 rw-p 00000000 00:00 0 
7fb4ff2b3000-7fb4ff2b4000 ---p 00000000 00:00 0 
7fb4ff2b4000-7fb4ff3b4000 rw-p 00000000 00:00 0 
7fb4ff7b8000-7fb4ff7b9000 ---p 00000000 00:00 0 
7fb4ff7b9000-7fb4ff8b9000 rw-p 00000000 00:00 0 
7fb4ffabb000-7fb4ffabc000 ---p 00000000 00:00 0 
7fb4ffabc000-7fb4ffbbc000 rw-p 00000000 00:00 0 
7fb5000c1000-7fb5000c2000 ---p 00000000 00:00 0 
7fb5000c2000-7fb5001c2000 rw-p 00000000 00:00 0 
7fb5002c3000-7fb5002c4000 ---p 00000000 00:00 0 
7fb5002c4000-7fb5003c4000 rw-p 00000000 00:00 0 
7fb5007c8000-7fb5007c9000 ---p 00000000 00:00 0 
7fb5007c9000-7fb5008c9000 rw-p 00000000 00:00 0 
7fb5008c9000-7fb5008ca000 ---p 00000000 00:00 0 
7fb5008ca000-7fb5009ca000 rw-p 00000000 00:00 0 
7fb500ccd000-7fb500cce000 ---p 00000000 00:00 0 
7fb500cce000-7fb500dce000 rw-p 00000000 00:00 0 
7fb500dce000-7fb500dcf000 ---p 00000000 00:00 0 
7fb500dcf000-7fb500ecf000 rw-p 00000000 00:00 0 
7fb5013d4000-7fb5013d5000 ---p 00000000 00:00 0 
7fb5013d5000-7fb5014d5000 rw-p 00000000 00:00 0 
7fb5014d5000-7fb5014d6000 ---p 00000000 00:00 0 
7fb5014d6000-7fb5015d6000 rw-p 00000000 00:00 0 
7fb5018d9000-7fb5018da000 ---p 00000000 00:00 0 
7fb5018da000-7fb5019da000 rw-p 00000000 00:00 0 
7fb501adb000-7fb501adc000 ---p 00000000 00:00 0 
7fb501adc000-7fb501bdc000 rw-p 00000000 00:00 0 
7fb501cdd000-7fb501cde000 ---p 00000000 00:00 0 
7fb501cde000-7fb501dde000 rw-p 00000000 00:00 0 
7fb501fe0000-7fb501fe1000 ---p 00000000 00:00 0 
7fb501fe1000-7fb5020e1000 rw-p 00000000 00:00 0 
7fb5021e2000-7fb5021e3000 ---p 00000000 00:00 0 
7fb5021e3000-7fb5022e3000 rw-p 00000000 00:00 0 
7fb5023e4000-7fb5023e5000 ---p 00000000 00:00 0 
7fb5023e5000-7fb5024e5000 rw-p 00000000 00:00 0 
7fb5026e7000-7fb5026e8000 ---p 00000000 00:00 0 
7fb5026e8000-7fb5027e8000 rw-p 00000000 00:00 0 
7fb5028e9000-7fb5028ea000 ---p 00000000 00:00 0 
7fb5028ea000-7fb5029ea000 rw-p 00000000 00:00 0 
7fb502ced000-7fb502cee000 ---p 00000000 00:00 0 
7fb502cee000-7fb502dee000 rw-p 00000000 00:00 0 
7fb502ff0000-7fb502ff1000 ---p 00000000 00:00 0 
7fb502ff1000-7fb5030f1000 rw-p 00000000 00:00 0 
7fb5032f3000-7fb5032f4000 ---p 00000000 00:00 0 
7fb5032f4000-7fb5033f4000 rw-p 00000000 00:00 0 
7fb5033f4000-7fb5033f5000 ---p 00000000 00:00 0 
7fb5033f5000-7fb5034f5000 rw-p 00000000 00:00 0 
7fb5034f5000-7fb5034f6000 ---p 00000000 00:00 0 
7fb5034f6000-7fb5035f6000 rw-p 00000000 00:00 0 
7fb5036f7000-7fb5036f8000 ---p 00000000 00:00 0 
7fb5036f8000-7fb5037f8000 rw-p 00000000 00:00 0 
7fb5039fa000-7fb5039fb000 ---p 00000000 00:00 0 
7fb5039fb000-7fb503afb000 rw-p 00000000 00:00 0 
7fb503dfe000-7fb503dff000 ---p 00000000 00:00 0 
7fb503dff000-7fb503eff000 rw-p 00000000 00:00 0 
7fb504000000-7fb508000000 rw-p 00000000 00:00 0 
7fb5082be000-7fb5082bf000 ---p 00000000 00:00 0 
7fb5082bf000-7fb5083bf000 rw-p 00000000 00:00 0 
7fb5087c3000-7fb5087c4000 ---p 00000000 00:00 0 
7fb5087c4000-7fb5088c4000 rw-p 00000000 00:00 0 
7fb508eca000-7fb508ecb000 ---p 00000000 00:00 0 
7fb508ecb000-7fb508fcb000 rw-p 00000000 00:00 0 
7fb5091cd000-7fb5091ce000 ---p 00000000 00:00 0 
7fb5091ce000-7fb5092ce000 rw-p 00000000 00:00 0 
7fb5095d1000-7fb5095d2000 ---p 00000000 00:00 0 
7fb5095d2000-7fb5096d2000 rw-p 00000000 00:00 0 
7fb5097d3000-7fb5097d4000 ---p 00000000 00:00 0 
7fb5097d4000-7fb5098d4000 rw-p 00000000 00:00 0 
7fb509ad6000-7fb509ad7000 ---p 00000000 00:00 0 
7fb509ad7000-7fb509bd7000 rw-p 00000000 00:00 0 
7fb509dd9000-7fb509dda000 ---p 00000000 00:00 0 
7fb509dda000-7fb509eda000 rw-p 00000000 00:00 0 
7fb50a4e0000-7fb50a4e1000 ---p 00000000 00:00 0 
7fb50a4e1000-7fb50a5e1000 rw-p 00000000 00:00 0 
7fb50a6e2000-7fb50a6e3000 ---p 00000000 00:00 0 
7fb50a6e3000-7fb50a7e3000 rw-p 00000000 00:00 0 
7fb50a9e5000-7fb50a9e6000 ---p 00000000 00:00 0 
7fb50a9e6000-7fb50aae6000 rw-p 00000000 00:00 0 
7fb50aae6000-7fb50aae7000 ---p 00000000 00:00 0 
7fb50aae7000-7fb50abe7000 rw-p 00000000 00:00 0 
7fb50ace8000-7fb50ace9000 ---p 00000000 00:00 0 
7fb50ace9000-7fb50ade9000 rw-p 00000000 00:00 0 
7fb50aeea000-7fb50aeeb000 ---p 00000000 00:00 0 
7fb50aeeb000-7fb50afeb000 rw-p 00000000 00:00 0 
7fb50b3ef000-7fb50b3f0000 ---p 00000000 00:00 0 
7fb50b3f0000-7fb50b4f0000 rw-p 00000000 00:00 0 
7fb50b4f0000-7fb50b4f1000 ---p 00000000 00:00 0 
7fb50b4f1000-7fb50b5f1000 rw-p 00000000 00:00 0 
7fb50b5f1000-7fb50b5f2000 ---p 00000000 00:00 0 
7fb50b5f2000-7fb50b6f2000 rw-p 00000000 00:00 0 
7fb50b6f2000-7fb50b6f3000 ---p 00000000 00:00 0 
7fb50b6f3000-7fb50b7f3000 rw-p 00000000 00:00 0 
7fb50b8f4000-7fb50b8f5000 ---p 00000000 00:00 0 
7fb50b8f5000-7fb50b9f5000 rw-p 00000000 00:00 0 
7fb50baf6000-7fb50baf7000 ---p 00000000 00:00 0 
7fb50baf7000-7fb50bbf7000 rw-p 00000000 00:00 0 
7fb50bcf8000-7fb50bcf9000 ---p 00000000 00:00 0 
7fb50bcf9000-7fb50bdf9000 rw-p 00000000 00:00 0 
7fb50c0fc000-7fb50c0fd000 ---p 00000000 00:00 0 
7fb50c0fd000-7fb50c1fd000 rw-p 00000000 00:00 0 
7fd304000000-7fd307f2b000 rw-p 00000000 00:00 0 
7fd307f2b000-7fd308000000 ---p 00000000 00:00 0 
7fd3080bd000-7fd3080be000 ---p 00000000 00:00 0 
7fd3080be000-7fd3081be000 rw-p 00000000 00:00 0 
7fd3082bf000-7fd3082c0000 ---p 00000000 00:00 0 
7fd3082c0000-7fd3083c0000 rw-p 00000000 00:00 0 
7fd308bc1000-7fd308bc2000 ---p 00000000 00:00 0 
7fd308bc2000-7fd3093c2000 rw-p 00000000 00:00 0 
7fd3093c2000-7fd3093c3000 ---p 00000000 00:00 0 
7fd3093c3000-7fd309bc3000 rw-p 00000000 00:00 0 
7fd309bc3000-7fd309bc4000 ---p 00000000 00:00 0 
7fd309bc4000-7fd30a3c4000 rw-p 00000000 00:00 0 
7fd30a3c4000-7fd30a3c5000 ---p 00000000 00:00 0 
7fd30a3c5000-7fd30abc5000 rw-p 00000000 00:00 0 
7fd30afc8000-7fd30b142000 r-xp 00000000 08:02 260613                     /lib/x86_64-linux-gnu/libc-2.13.so
7fd30b142000-7fd30b342000 ---p 0017a000 08:02 260613                     /lib/x86_64-linux-gnu/libc-2.13.so
7fd30b342000-7fd30b346000 r--p 0017a000 08:02 260613                     /lib/x86_64-linux-gnu/libc-2.13.so
7fd30b346000-7fd30b347000 rw-p 0017e000 08:02 260613                     /lib/x86_64-linux-gnu/libc-2.13.so
7fd30b347000-7fd30b34c000 rw-p 00000000 00:00 0 
7fd30b34c000-7fd30b361000 r-xp 00000000 08:02 260615                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd30b361000-7fd30b561000 ---p 00015000 08:02 260615                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd30b561000-7fd30b562000 rw-p 00015000 08:02 260615                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd30b562000-7fd30b5e3000 r-xp 00000000 08:02 266195                     /lib/x86_64-linux-gnu/libm-2.13.so
7fd30b5e3000-7fd30b7e2000 ---p 00081000 08:02 266195                     /lib/x86_64-linux-gnu/libm-2.13.so
7fd30b7e2000-7fd30b7e3000 r--p 00080000 08:02 266195                     /lib/x86_64-linux-gnu/libm-2.13.so
7fd30b7e3000-7fd30b7e4000 rw-p 00081000 08:02 266195                     /lib/x86_64-linux-gnu/libm-2.13.so
7fd30b7e4000-7fd30b8cf000 r-xp 00000000 08:03 525341                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd30b8cf000-7fd30bacf000 ---p 000eb000 08:03 525341                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd30bacf000-7fd30bad7000 r--p 000eb000 08:03 525341                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd30bad7000-7fd30bad9000 rw-p 000f3000 08:03 525341                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd30bad9000-7fd30baee000 rw-p 00000000 00:00 0 
7fd30baee000-7fd30bb05000 r-xp 00000000 08:02 266217                     /lib/x86_64-linux-gnu/libpthread-2.13.so
7fd30bb05000-7fd30bd04000 ---p 00017000 08:02 266217                     /lib/x86_64-linux-gnu/libpthread-2.13.so
7fd30bd04000-7fd30bd05000 r--p 00016000 08:02 266217                     /lib/x86_64-linux-gnu/libpthread-2.13.so
7fd30bd05000-7fd30bd06000 rw-p 00017000 08:02 266217                     /lib/x86_64-linux-gnu/libpthread-2.13.so
7fd30bd06000-7fd30bd0a000 rw-p 00000000 00:00 0 
7fd30bd0a000-7fd30bd29000 r-xp 00000000 08:02 266221                     /lib/x86_64-linux-gnu/ld-2.13.so
7fd30bf12000-7fd30bf1d000 rw-p 00000000 00:00 0 
7fd30bf26000-7fd30bf29000 rw-p 00000000 00:00 0 
7fd30bf29000-7fd30bf2a000 r--p 0001f000 08:02 266221                     /lib/x86_64-linux-gnu/ld-2.13.so
7fd30bf2a000-7fd30bf2b000 rw-p 00020000 08:02 266221                     /lib/x86_64-linux-gnu/ld-2.13.so
7fd30bf2b000-7fd30bf2c000 rw-p 00000000 00:00 0 
7fff6e381000-7fff6e3a2000 rw-p 00000000 00:00 0                          [stack]
7fff6e3ff000-7fff6e400000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Tue Apr 10 07:00:17 Got signal: 6 (Aborted).
 
Tue Apr 10 07:00:17 Backtrace:
0x59383f 0x589e20 0x7fd30affa480 0x7fd30affa405 0x7fd30affd680 0x7fd30b030dbb 0x7fd30b03a606 0x7fd30b03aa3d 0x7fd30b03bf38 0x7fd30b03f33c 0x593559 0x5e7074 0x7fd30affed82 0x7fd30affedd5 0x589e59 0x7fd30affa480 0x7fd30affa405 0x7fd30affd680 0x9d3181 0x5aed7b 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x1f) [0x59383f]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x350) [0x589e20]
 /lib/x86_64-linux-gnu/libc.so.6(+0x32480) [0x7fd30affa480]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fd30affa405]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7fd30affd680]
 /lib/x86_64-linux-gnu/libc.so.6(+0x68dbb) [0x7fd30b030dbb]
 /lib/x86_64-linux-gnu/libc.so.6(+0x72606) [0x7fd30b03a606]
 /lib/x86_64-linux-gnu/libc.so.6(+0x72a3d) [0x7fd30b03aa3d]
 /lib/x86_64-linux-gnu/libc.so.6(+0x73f38) [0x7fd30b03bf38]
 /lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c) [0x7fd30b03f33c]
 /usr/bin/mongod(_ZN5boost6detail12shared_countD1Ev+0x49) [0x593559]
 /usr/bin/mongod(_ZNSt8_Rb_treeISsSt4pairIKSsN5boost10shared_ptrIN5mongo25NamespaceDetailsTransientEEEESt10_Select1stIS7_ESt4lessISsESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E+0x214) [0x5e7074]
 /lib/x86_64-linux-gnu/libc.so.6(+0x36d82) [0x7fd30affed82]
 /lib/x86_64-linux-gnu/libc.so.6(+0x36dd5) [0x7fd30affedd5]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x389) [0x589e59]
 /lib/x86_64-linux-gnu/libc.so.6(+0x32480) [0x7fd30affa480]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fd30affa405]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7fd30affd680]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xc1) [0x9d3181]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 
Logstream::get called in uninitialized state
Tue Apr 10 07:00:17 [conn50128] end connection 127.0.0.1:33306 (63 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:17 [conn50128] boost assertion failure !pthread_mutex_lock(m) boost::pthread::pthread_mutex_scoped_lock::pthread_mutex_scoped_lock(pthread_mutex_t*) src/third_party/boost/boost/thread/pthread/pthread_mutex_scoped_lock.hpp:26
Logstream::get called in uninitialized state
Tue Apr 10 07:00:17 [conn50128]   Fatal Assertion 16108
0x8e5c78 0x9d311c 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x98) [0x8e5c78]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x5c) [0x9d311c]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
Logstream::get called in uninitialized state
Tue Apr 10 07:00:17 [conn50128] 
 
***aborting after fassert() failure
 
 
Tue Apr 10 07:00:17 Got signal: 6 (Aborted).
 
Tue Apr 10 07:00:17 Backtrace:
0x59383f 0x589e20 0x7fd30affa480 0x7fd30affa405 0x7fd30affd680 0x9d3181 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x1f) [0x59383f]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x350) [0x589e20]
 /lib/x86_64-linux-gnu/libc.so.6(+0x32480) [0x7fd30affa480]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fd30affa405]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7fd30affd680]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xc1) [0x9d3181]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
 
Logstream::get called in uninitialized state
Tue Apr 10 07:00:17 [conn50115] end connection 127.0.0.1:33290 (62 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:17 [conn49991] end connection 127.0.0.1:33106 (62 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:17 [conn50217] end connection 127.0.0.1:33508 (62 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:17 [conn50217] boost assertion failure !pthread_mutex_lock(m) boost::pthread::pthread_mutex_scoped_lock::pthread_mutex_scoped_lock(pthread_mutex_t*) src/third_party/boost/boost/thread/pthread/pthread_mutex_scoped_lock.hpp:26
Logstream::get called in uninitialized state
Tue Apr 10 07:00:17 [conn50217]   Fatal Assertion 16108
0x8e5c78 0x9d311c 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x98) [0x8e5c78]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x5c) [0x9d311c]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
Logstream::get called in uninitialized state
Tue Apr 10 07:00:17 [conn50217] 
 
***aborting after fassert() failure
 
 
Tue Apr 10 07:00:17 Got signal: 6 (Aborted).
 
Tue Apr 10 07:00:17 Backtrace:
0x59383f 0x589e20 0x7fd30affa480 0x7fd30affa405 0x7fd30affd680 0x9d3181 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x1f) [0x59383f]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x350) [0x589e20]
 /lib/x86_64-linux-gnu/libc.so.6(+0x32480) [0x7fd30affa480]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fd30affa405]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7fd30affd680]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xc1) [0x9d3181]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
 
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn49916] end connection 127.0.0.1:33042 (61 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn50016] end connection 127.0.0.1:33140 (61 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn50042] end connection 127.0.0.1:33167 (61 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn50047] end connection 127.0.0.1:33175 (61 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn49549] end connection 127.0.0.1:32810 (61 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn49989] end connection 127.0.0.1:33111 (61 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn49946] end connection 127.0.0.1:33072 (61 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn50108] end connection 127.0.0.1:33267 (61 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn50130] end connection 127.0.0.1:33309 (61 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn50130] boost assertion failure !pthread_mutex_lock(m) boost::pthread::pthread_mutex_scoped_lock::pthread_mutex_scoped_lock(pthread_mutex_t*) src/third_party/boost/boost/thread/pthread/pthread_mutex_scoped_lock.hpp:26
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn50130]   Fatal Assertion 16108
0x8e5c78 0x9d311c 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x98) [0x8e5c78]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x5c) [0x9d311c]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn50130] 
 
***aborting after fassert() failure
 
 
Tue Apr 10 07:00:18 Got signal: 6 (Aborted).
 
Tue Apr 10 07:00:18 Backtrace:
0x59383f 0x589e20 0x7fd30affa480 0x7fd30affa405 0x7fd30affd680 0x9d3181 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x1f) [0x59383f]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x350) [0x589e20]
 /lib/x86_64-linux-gnu/libc.so.6(+0x32480) [0x7fd30affa480]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fd30affa405]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7fd30affd680]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xc1) [0x9d3181]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
 
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn50244] end connection 127.0.0.1:33536 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn49995] end connection 127.0.0.1:33113 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:18 [conn50168] end connection 127.0.0.1:33397 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:19 [conn50201] end connection 127.0.0.1:33491 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:19 [conn50081] end connection 127.0.0.1:33206 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:19 [conn49862] end connection 127.0.0.1:32986 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:19 [conn49548] end connection 127.0.0.1:32814 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:19 [conn49992] end connection 127.0.0.1:33119 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:19 [conn50139] end connection 127.0.0.1:33338 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:19 [conn49850] end connection 127.0.0.1:32975 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:19 [conn50072] end connection 127.0.0.1:33195 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:19 [conn50049] end connection 127.0.0.1:33177 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:19 [conn50074] end connection 127.0.0.1:33202 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn49856] end connection 127.0.0.1:32982 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn50065] end connection 127.0.0.1:33182 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn50007] end connection 127.0.0.1:33132 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn50118] end connection 127.0.0.1:33294 (60 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn50118] boost assertion failure !pthread_mutex_lock(m) boost::pthread::pthread_mutex_scoped_lock::pthread_mutex_scoped_lock(pthread_mutex_t*) src/third_party/boost/boost/thread/pthread/pthread_mutex_scoped_lock.hpp:26
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn50118]   Fatal Assertion 16108
0x8e5c78 0x9d311c 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x98) [0x8e5c78]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x5c) [0x9d311c]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn50118] 
 
***aborting after fassert() failure
 
 
Tue Apr 10 07:00:20 Got signal: 6 (Aborted).
 
Tue Apr 10 07:00:20 Backtrace:
0x59383f 0x589e20 0x7fd30affa480 0x7fd30affa405 0x7fd30affd680 0x9d3181 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x1f) [0x59383f]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x350) [0x589e20]
 /lib/x86_64-linux-gnu/libc.so.6(+0x32480) [0x7fd30affa480]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fd30affa405]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7fd30affd680]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xc1) [0x9d3181]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
 
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn49704] end connection 127.0.0.1:32962 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn50197] end connection 127.0.0.1:33487 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn49918] end connection 127.0.0.1:33044 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn50060] end connection 127.0.0.1:33186 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn49851] end connection 127.0.0.1:32973 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn50219] end connection 127.0.0.1:33510 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn49975] end connection 127.0.0.1:33101 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:20 [conn49921] end connection 127.0.0.1:33047 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:21 [conn50131] end connection 127.0.0.1:33311 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:21 [conn49551] end connection 127.0.0.1:32816 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:21 [conn49868] end connection 127.0.0.1:32993 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:21 [conn49940] end connection 127.0.0.1:33065 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:21 [conn50013] end connection 127.0.0.1:33137 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:21 [conn49943] end connection 127.0.0.1:33066 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:22 [conn49962] end connection 127.0.0.1:33086 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:23 [conn50014] end connection 127.0.0.1:33138 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:23 [conn50015] end connection 127.0.0.1:33141 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:23 [conn50158] end connection 127.0.0.1:33378 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:24 [conn50117] end connection 127.0.0.1:33292 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:24 [conn49994] end connection 127.0.0.1:33121 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:24 [conn50256] end connection 127.0.0.1:33551 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:25 [conn50142] end connection 127.0.0.1:33340 (59 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:25 [conn50142] boost assertion failure !pthread_mutex_lock(m) boost::pthread::pthread_mutex_scoped_lock::pthread_mutex_scoped_lock(pthread_mutex_t*) src/third_party/boost/boost/thread/pthread/pthread_mutex_scoped_lock.hpp:26
Logstream::get called in uninitialized state
Tue Apr 10 07:00:25 [conn50142]   Fatal Assertion 16108
0x8e5c78 0x9d311c 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x98) [0x8e5c78]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x5c) [0x9d311c]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
Logstream::get called in uninitialized state
Tue Apr 10 07:00:25 [conn50142] 
 
***aborting after fassert() failure
 
 
Tue Apr 10 07:00:25 Got signal: 6 (Aborted).
 
Tue Apr 10 07:00:25 Backtrace:
0x59383f 0x589e20 0x7fd30affa480 0x7fd30affa405 0x7fd30affd680 0x9d3181 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x1f) [0x59383f]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x350) [0x589e20]
 /lib/x86_64-linux-gnu/libc.so.6(+0x32480) [0x7fd30affa480]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fd30affa405]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7fd30affd680]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xc1) [0x9d3181]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
 
Logstream::get called in uninitialized state
Tue Apr 10 07:00:25 [conn49894] end connection 127.0.0.1:33019 (58 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:25 [conn49894] boost assertion failure !pthread_mutex_lock(m) boost::pthread::pthread_mutex_scoped_lock::pthread_mutex_scoped_lock(pthread_mutex_t*) src/third_party/boost/boost/thread/pthread/pthread_mutex_scoped_lock.hpp:26
Logstream::get called in uninitialized state
Tue Apr 10 07:00:25 [conn49894]   Fatal Assertion 16108
0x8e5c78 0x9d311c 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x98) [0x8e5c78]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x5c) [0x9d311c]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
Logstream::get called in uninitialized state
Tue Apr 10 07:00:25 [conn49894] 
 
***aborting after fassert() failure
 
 
Tue Apr 10 07:00:25 Got signal: 6 (Aborted).
 
Tue Apr 10 07:00:25 Backtrace:
0x59383f 0x589e20 0x7fd30affa480 0x7fd30affa405 0x7fd30affd680 0x9d3181 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x1f) [0x59383f]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x350) [0x589e20]
 /lib/x86_64-linux-gnu/libc.so.6(+0x32480) [0x7fd30affa480]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fd30affa405]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7fd30affd680]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xc1) [0x9d3181]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
 
Logstream::get called in uninitialized state
Tue Apr 10 07:00:26 [conn50093] end connection 127.0.0.1:33246 (57 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:26 [conn50043] end connection 127.0.0.1:33168 (57 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:31 [conn50006] end connection 127.0.0.1:33130 (57 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:31 [conn49934] end connection 127.0.0.1:33054 (57 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:33 [conn50038] end connection 127.0.0.1:33163 (57 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:37 [conn50061] end connection 127.0.0.1:33190 (57 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:37 [conn50061] boost assertion failure !pthread_mutex_lock(m) boost::pthread::pthread_mutex_scoped_lock::pthread_mutex_scoped_lock(pthread_mutex_t*) src/third_party/boost/boost/thread/pthread/pthread_mutex_scoped_lock.hpp:26
Logstream::get called in uninitialized state
Tue Apr 10 07:00:37 [conn50061]   Fatal Assertion 16108
0x8e5c78 0x9d311c 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x98) [0x8e5c78]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x5c) [0x9d311c]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
Logstream::get called in uninitialized state
Tue Apr 10 07:00:37 [conn50061] 
 
***aborting after fassert() failure
 
 
Tue Apr 10 07:00:37 Got signal: 6 (Aborted).
 
Tue Apr 10 07:00:37 Backtrace:
0x59383f 0x589e20 0x7fd30affa480 0x7fd30affa405 0x7fd30affd680 0x9d3181 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x1f) [0x59383f]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x350) [0x589e20]
 /lib/x86_64-linux-gnu/libc.so.6(+0x32480) [0x7fd30affa480]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fd30affa405]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7fd30affd680]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xc1) [0x9d3181]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
 
Logstream::get called in uninitialized state
Tue Apr 10 07:00:38 [conn49920] end connection 127.0.0.1:33039 (56 connections now open)
Logstream::get called in uninitialized state
Tue Apr 10 07:00:38 [conn49920] boost assertion failure !pthread_mutex_lock(m) boost::pthread::pthread_mutex_scoped_lock::pthread_mutex_scoped_lock(pthread_mutex_t*) src/third_party/boost/boost/thread/pthread/pthread_mutex_scoped_lock.hpp:26
Logstream::get called in uninitialized state
Tue Apr 10 07:00:38 [conn49920]   Fatal Assertion 16108
0x8e5c78 0x9d311c 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x98) [0x8e5c78]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x5c) [0x9d311c]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
Logstream::get called in uninitialized state
Tue Apr 10 07:00:38 [conn49920] 
 
***aborting after fassert() failure
 
 
Tue Apr 10 07:00:38 Got signal: 6 (Aborted).
 
Tue Apr 10 07:00:38 Backtrace:
0x59383f 0x589e20 0x7fd30affa480 0x7fd30affa405 0x7fd30affd680 0x9d3181 0x5aed7b 0x7fd30baf4b40 0x7fd30b09d36d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x1f) [0x59383f]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x350) [0x589e20]
 /lib/x86_64-linux-gnu/libc.so.6(+0x32480) [0x7fd30affa480]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fd30affa405]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7fd30affd680]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xc1) [0x9d3181]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x76b) [0x5aed7b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40) [0x7fd30baf4b40]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd30b09d36d]
 
Logstream::get called in uninitialized state
Tue Apr 10 07:03:18 [conn6938] end connection 127.0.0.1:55564 (55 connections now open)
 
 
***** SERVER RESTARTED *****
 
 
Tue Apr 10 07:04:13 [initandlisten] MongoDB starting : pid=28791 port=27017 dbpath=/work1/mongodb 64-bit host=host9
Tue Apr 10 07:04:13 [initandlisten] 
Tue Apr 10 07:04:13 [initandlisten] ** NOTE: This is a development version (2.1.1-pre-) of MongoDB.
Tue Apr 10 07:04:13 [initandlisten] **       Not recommended for production.
Tue Apr 10 07:04:13 [initandlisten] 
Tue Apr 10 07:04:13 [initandlisten] db version v2.1.1-pre-, pdfile version 4.5
Tue Apr 10 07:04:13 [initandlisten] git version: a2d6f752d56aa446220b9f14c8ad3865c2fb5db8
Tue Apr 10 07:04:13 [initandlisten] build info: Linux azat 2.6.38custom1.0 #1 SMP Mon May 23 17:17:08 MSD 2011 x86_64 BOOST_LIB_VERSION=1_49
Tue Apr 10 07:04:13 [initandlisten] options: { bind_ip: "0.0.0.0", command: [ "run" ], config: "/etc/mongodb.conf", dbpath: "/work1/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log", unixSocketPrefix: "/var/run/mongodb" }
Tue Apr 10 07:04:13 [initandlisten] journal dir=/work1/mongodb/journal
Tue Apr 10 07:04:13 [initandlisten] recover : no journal files present, no recovery needed
Tue Apr 10 07:04:13 [initandlisten] waiting for connections on port 27017

Comment by Azat Khuzhin [ 10/Apr/12 ]

And also while this big MR job is running
Requests to mongod processes very slow

queries - are fast
but, mongod takes connection very slow

Because explain() shows that query runs about 5-50 ms

Comment by Azat Khuzhin [ 10/Apr/12 ]

v8 - 3.9.23

It was a single seg fault
But now I run such job one more, and it runs about 1.5 days, and only 24%, if it will crash - I paste trimmed MR code

Also I have a question about MR, first MR jobs (like this, but it collect another rows, and in output collection there is much less rows) and it runs about 4-6 hours
While this 1.5 days - 24%
Have you any advice how can I speedup or monitor why this MR job is so slow?

Comment by Antoine Girbal [ 05/Apr/12 ]

Azat,
was it a single seg fault occurrence, or is it reproducible?
If so could you provide with the MR code?
Also what is the v8 version/revision you are using?

Comment by Azat Khuzhin [ 29/Mar/12 ]

mongo$ git log -n1
commit a2d6f752d56aa446220b9f14c8ad3865c2fb5db8

Comment by Azat Khuzhin [ 29/Mar/12 ]

mongo$ git describe
r2.1.0-1667-ga2d6f75

Comment by Eliot Horowitz (Inactive) [ 29/Mar/12 ]

What git hash is with this?

Generated at Thu Feb 08 03:08:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.