[SERVER-18106] V8 crashes if mongod run as service on Azure Created: 17/Apr/15  Updated: 15/May/15  Resolved: 15/May/15

Status: Closed
Project: Core Server
Component/s: MapReduce
Affects Version/s: 3.0.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Victor Pozdniakov Assignee: Ramon Fernandez Marina
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 7, Azure VM


Attachments: Text File Backtrace.txt    
Operating System: Linux
Steps To Reproduce:

Following query crushes server:

db['2015-04-13'].group({
    key:{id:1},
    cond:{created_at:{$gte: new Date('2015-04-13')}},
    reduce:function (curr, resul) {},
    initial: {}
})

Participants:

 Description   

Query provided in "Steps to reproduce" crushes server.
There is output of server log with verbosity at level 5 is attached as file (starting only from a command execution).

Problem occur only if Mongod starts as service (service mongod start), if mongod started in following way:
sudo su -s /bin/bash mongod
mongod -f /etc/mongod.conf
Then query successfully runs; however service mongod status returns "inactive (dead)".

ulimits set according to documentation, mongod is an owner of Data folder, and mongod never runs on same data path as different users.

Another problem with running as service did come up during deployment, and may have something in common with described issue: if run as service, mongod is unable to create lock file and database in any other folder then default /var/lib/ even if mongod is assign owner recursively of new db path .



 Comments   
Comment by Ramon Fernandez Marina [ 15/May/15 ]

vpozdniakov, apologies for the long delay in getting back to you. While the output of paxctl doesn't show any useful information, I think something in the way mongod is run when launched as a service is preventing V8 from allocating memory. The second issue you mentioned related to permissions seems to indicate these limitations are not limited to memory usage. One common source of similar problems is running with SELinux enabled.

Given that mongod works when run from the command line I don't think there's evidence of a bug in the server, and since we keep the SERVER project is for reporting bugs or feature suggestions for the MongoDB server I'm going to resolve this ticket. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group.

Regards,
Ramón.

Comment by Victor Pozdniakov [ 20/Apr/15 ]

[vpozdniakov@MongoDB ~]$ sudo /sbin/paxctl -cm /usr/bin/mongod
file /usr/bin/mongod had a PT_GNU_STACK program header, converted
[vpozdniakov@MongoDB ~]$ sudo /sbin/paxctl -v /usr/bin/mongod
PaX control v0.9
Copyright 2004,2005,2006,2007,2009,2010,2011,2012,2014 PaX Team <pageexec@freemail.hu>
 
- PaX flags: -----m-x-e-- [/usr/bin/mongod]
        MPROTECT is disabled
        RANDEXEC is disabled
        EMUTRAMP is disabled

I also double checked log file for startup events, MongoDB specifies ulimits there as well, and i assume those should be actually correct ones. Those seems to be fine:

2015-04-20T12:39:09.092-0400 D SHARDING isInRangeTest passed
2015-04-20T12:39:09.092-0400 D NETWORK  [initandlisten] fd limit hard:64000 soft:64000 max conn: 51200


I was not able to find records regarding ulimit on memory allocation, but i will assume that if fd was changed from default, then memory allocation is changed as well.

Comment by Ramon Fernandez Marina [ 20/Apr/15 ]

Thanks for the additional information vpozdniakov. Can you please run

/sbin/paxctl -cm /usr/bin/mongod
/sbin/paxctl -v /usr/bin/mongod

and post the output? I'm not 100% sure the issue is with grsec, but since the stack trace you posted is the same as that in SERVER-12991 I think we should rule out grsec first.

The other hypothesis I have is different limits for services when started via initscripts than when started from the command line as the V8 failure you're seeing comes from attempting to allocate memory:

mongod(_ZN2v88internal2OS8AllocateEmPmb+0xD7) [0x11dbe57]

Comment by Victor Pozdniakov [ 20/Apr/15 ]

I believe Microsoft uses OpenLogic distribution of CentOS as a VM Image base.
Pax control returns:

PaX control v0.9
Copyright 2004,2005,2006,2007,2009,2010,2011,2012,2014 PaX Team <pageexec@freema il.hu>
 
file /usr/bin/mongo does not have a PT_PAX_FLAGS program header, try conversion

I also would like to notice that MongoDB works fine even if it runs as non-root user, including mongod (from bash console started up as mongod -f /etc/mongod.conf)

Comment by Ramon Fernandez Marina [ 17/Apr/15 ]

vpozdniakov, the crash you report happens inside V8, so you may be running into SERVER-12991. This could explain why things work when you run as root but not as a service (which uses a non-privileged user). Can you please check whether you have a grsecurity-enabled kernel? Also, can you please run:

paxctl -v /usr/bin/mongo

and post the output?

Thanks,
Ramón.

Generated at Thu Feb 08 03:46:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.