[SERVER-8446] MongoDB Segmentation Fault Created: 04/Feb/13  Updated: 10/Dec/14  Resolved: 02/Jun/13

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 2.2.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Adam Kusmierz Assignee: David Hows
Resolution: Done Votes: 0
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: Linux
Steps To Reproduce:

I can't.

Participants:

 Description   

Random segmentation fault.

Fri Jan  4 14:22:16 Invalid access at address: 0x1a3fffffffc from thread: conn56
 
Fri Jan  4 14:22:16 Got signal: 11 (Segmentation fault).
 
Fri Jan  4 14:22:16 Backtrace:
0xaffd31 0x558bb9 0x559142 0x7fcc3b822ff0 0x7fcc3b3a0ed6 0xab04bc 0xba94cd 0xb9125d 0xb6ccf6 0xabfb26 0xaa8abe 0xaa8bfb 0x71589c 0x716480 0x6e8d71 0x6ea5e2 0x6eb59c 0x832c68 0x8363b0 0x7b072d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x558bb9]
 /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x559142]
 /lib/libpthread.so.0(+0xeff0) [0x7fcc3b822ff0]
 /usr/lib/libstdc++.so.6(_ZNSsD1Ev+0x26) [0x7fcc3b3a0ed6]
 /usr/bin/mongod(_ZN5mongo13bson_finalizeEP9JSContextP8JSObject+0x5c) [0xab04bc]
 /usr/bin/mongod(js_FinalizeObject+0x5d) [0xba94cd]
 /usr/bin/mongod(js_GC+0xa6d) [0xb9125d]
 /usr/bin/mongod(js_DestroyContext+0x1e6) [0xb6ccf6]
 /usr/bin/mongod(_ZN5mongo7SMScopeD0Ev+0x96) [0xabfb26]
 /usr/bin/mongod(_ZN5mongo10ScopeCache4doneERKSsPNS_5ScopeE+0x7e) [0xaa8abe]
 /usr/bin/mongod(_ZN5mongo11PooledScopeD0Ev+0x4b) [0xaa8bfb]
 /usr/bin/mongod(_ZN5mongo6dbEvalERKSsRNS_7BSONObjERNS_14BSONObjBuilderERSs+0x37c) [0x71589c]
 /usr/bin/mongod(_ZN5mongo7CmdEval3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x260) [0x716480]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8d71]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xd52) [0x6ea5e2]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eb59c]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x832c68]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xc10) [0x8363b0]
 /usr/bin/mongod() [0x7b072d]



 Comments   
Comment by Adam Kusmierz [ 03/Jun/13 ]

Ok, you've right. But it's not a reason to segmentation fault the server! It's insane.

Comment by David Hows [ 28/Feb/13 ]

Hi Adam,

From what I can see of the error this related to the fact that you are looking to evaluate the obj.search.partner_id and while you check if search is true, you aren't checking if partner_id exists as part of the document you are evaluating against.

Is there a doc in your collection which has a search value, but no partner_id value?

As an aside, we normally recommend using standard query operators rather than server side javascript.

Regards,
David

Comment by Adam Kusmierz [ 27/Feb/13 ]

The beginning of the code is (called from php driver):

$code = new MongoCode('
		function () {
			var changedRecords = db.offers.find({
				$where: function() {
					if (!obj.search) {
						return true;
					}
					if (obj.partner.str != obj.search.partner_id.str) {
						return true;
					}
					if (obj.coupons != obj.search.coupons) {
						return true;
					}
				}
			});
// ...
');

Why this code kills mongo server? It's ("a little bit") annoying...

Comment by Adam Kusmierz [ 27/Feb/13 ]

Moar:

Wed Feb 27 10:50:07 [conn3] JS Error: TypeError: obj.search.partner_id has no properties nofile_b:4
Wed Feb 27 10:50:07 [conn3] assertion 10071 error on invocation of $where function:
JS Error: TypeError: obj.search.partner_id has no properties nofile_b:4 ns:hotprice.offers query:{ $where: function () {
    if (!obj.search) {
        return true;
    }
    if... }
Wed Feb 27 10:50:07 [conn3] problem detected during query over hotprice.offers : { $err: "error on invocation of $where function:
JS Error: TypeError: obj.search.partner_id has no properties nofile_b:4", code: 10071 }
Wed Feb 27 10:50:07 [conn3] JS Error: uncaught exception: error: {
        "$err" : "error on invocation of $where function:\nJS Error: TypeError: obj.search.partner_id has no properties nofile_b:4",
        "code" : 10071
}
Wed Feb 27 10:50:24 [conn19] JS Error: TypeError: obj.search.partner_id has no properties nofile_b:4
Wed Feb 27 10:50:24 [conn19] assertion 10071 error on invocation of $where function:
JS Error: TypeError: obj.search.partner_id has no properties nofile_b:4 ns:hotprice.offers query:{ $where: function () {
    if (!obj.search) {
        return true;
    }
    if... }
Wed Feb 27 10:50:24 [conn19] problem detected during query over hotprice.offers : { $err: "error on invocation of $where function:
JS Error: TypeError: obj.search.partner_id has no properties nofile_b:4", code: 10071 }
Wed Feb 27 10:50:24 [conn19] JS Error: uncaught exception: error: {
        "$err" : "error on invocation of $where function:\nJS Error: TypeError: obj.search.partner_id has no properties nofile_b:4",
        "code" : 10071
}
Wed Feb 27 10:50:24 Invalid access at address: 0 from thread: conn19
 
Wed Feb 27 10:50:24 Got signal: 11 (Segmentation fault).
 
Wed Feb 27 10:50:24 Backtrace:
0xb07561 0x5598c9 0x559e52 0x7fb9340e9ff0 0x7fb933c67fbd 0xab1eb6 0xb74115 0xb66c99 0xb92a1b 0xb67c50 0xaca554 0x7157f0 0x716560 0x6e8e51 0x6ea6c2 0x6eb67c 0x837468 0x83abb0 0x7b0afd 0x7b3e23
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5598c9]
 /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x559e52]
 /lib/libpthread.so.0(+0xeff0) [0x7fb9340e9ff0]
 /usr/lib/libstdc++.so.6(_ZNSs6assignERKSs+0x6d) [0x7fb933c67fbd]
 /usr/bin/mongod(_ZN5mongo13errorReporterEP9JSContextPKcP13JSErrorReport+0x106) [0xab1eb6]
 /usr/bin/mongod(js_ReportErrorNumberVA+0x1d5) [0xb74115]
 /usr/bin/mongod(JS_ReportErrorNumber+0x99) [0xb66c99]
 /usr/bin/mongod(js_ReportUncaughtException+0x1fb) [0xb92a1b]
 /usr/bin/mongod(JS_CallFunction+0x60) [0xb67c50]
 /usr/bin/mongod(_ZN5mongo7SMScope6invokeEP10JSFunctionPKNS_7BSONObjES5_ibbb+0x314) [0xaca554]
 /usr/bin/mongod(_ZN5mongo6dbEvalERKSsRNS_7BSONObjERNS_14BSONObjBuilderERSs+0x1f0) [0x7157f0]
 /usr/bin/mongod(_ZN5mongo7CmdEval3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x260) [0x716560]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8e51]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xd52) [0x6ea6c2]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eb67c]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x837468]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xc10) [0x83abb0]
 /usr/bin/mongod() [0x7b0afd]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a3) [0x7b3e23]

I'm looking into my code for the mongo code.

Comment by David Hows [ 25/Feb/13 ]

It may, it depends on where within the query the problem lies as if the error occurs before it actually begins processing the query then nothing would be logged to the profiler. If you can reproduce things that would be the biggest step forward, as it stands if you cannot reproduce there is not much we can do.

Comment by Adam Kusmierz [ 25/Feb/13 ]

What do you think. If I turn on debug log queries (profiler?) on the node, would it help you find what caused the issue? I haven't too much traffic on this machine, so I can turn debugging on for some time on production for you.

Comment by David Hows [ 24/Feb/13 ]

Hi Adam,

It likely is that execute, but given the stack trace i cannot say with 100% certainty. Nor can i say which part of that code could have caused an issue.

Here are the source code points which represent where this crash in mongo.

mongo/src/mongo/util/stacktrace.cpp:38
/include/c++/4.1.2/streambuf:515
/include/c++/4.1.2/sstream:444
??:0
??:0
/include/c++/4.1.2/ext/new_allocator.h:94
mongo/src/third_party/js-1.7/jsobj.c:2753
mongo/src/third_party/js-1.7/jsgc.c:3032
mongo/src/third_party/js-1.7/jscntxt.c:409
mongo/src/mongo/scripting/engine_spidermonkey.cpp:1472
mongo/src/mongo/scripting/engine.cpp:300
/include/c++/4.1.2/bits/basic_string.h:283
mongo/src/mongo/db/dbeval.cpp:106
mongo/src/mongo/db/dbeval.cpp:135
mongo/src/mongo/db/dbcommands.cpp:1859
mongo/src/mongo/db/dbcommands.cpp:1985
mongo/src/mongo/db/dbcommands.cpp:2078
mongo/src/mongo/db/ops/query.cpp:43
mongo/src/mongo/db/ops/query.cpp:920
mongo/src/mongo/db/instance.cpp:237

Without more details of what exactly was going on or a reproducible crash there is little we can do.

Regards,

David

Comment by Adam Kusmierz [ 24/Feb/13 ]

Probably this was one of this: exception 'MongoCursorException' with message 'couldn't send command' whith MongoCode (or it was just first code executed after mongo failed):

// $sort = array('_id' => 1); or $sort = array('name' => 1);
$code = new MongoCode('
			var roots = db.categories.find({"parent": null}).sort(sort);
			var tree = {};
 
			var getChildren = function(row) {
				var children = db.categories.find({"parent": row._id}).sort(sort);
				var tree2 = {};
 
				if (children && children.count() > 0) {
					children.forEach(function(child) {
						tree2[child._id.valueOf()] = child;
						tree2[child._id.valueOf()].children = getChildren(child);
					});
				}
 
				return tree2;
			};
 
			roots.forEach(function(row) {
				tree[row._id.valueOf()] = row;
				tree[row._id.valueOf()].children = getChildren(row);
			});
 
			return tree;', array('sort' => $sort));
 
		$response = self::getMongoDb()->execute($code);

But looking in my logs it's not only one place, or I'm looking in wrong. Other thing - I still can't reproduce it.

Comment by Adam Kusmierz [ 24/Feb/13 ]

I haven't any more logs from my application. I'll try log some things more, but have you any idea what can throw this exception? find? update? call user code? Thanks in advance.

Comment by David Hows [ 18/Feb/13 ]

Hi Adam,

I need the full logs for what your application was doing as a lead up to this. While a stacktrace itself can be useful in pointing out where in the code an issue was raised from it does not provide any details of the variables that were used in the call nor any context as to why it was within that particular codepath. As such we normally need the logfiles for the instance to determine what it was doing in the lead up to the crash and if possible details from the user about any actions that were being performed.

Regards,

David

Comment by Adam Kusmierz [ 18/Feb/13 ]

Hi David,

what do you mean "logs"? I've attached one. I've only this, how can I produce more detailed (the issue is repeatable - once a week or so)?

Comment by David Hows [ 18/Feb/13 ]

Hi Adam,

This looks like a crash within the javascript engine, what leads you to suspect this was corruption?

Can you attach logs for this node in the lead up to the crash?

Thanks,

David

Generated at Thu Feb 08 03:17:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.