[SERVER-17238] $group with invalid reduce step crashes server with segfault Created: 10/Feb/15  Updated: 10/Feb/15  Resolved: 10/Feb/15

Status: Closed
Project: Core Server
Component/s: MapReduce
Affects Version/s: 3.0.0-rc8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Bernie Hackett Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-13824 V8-3.12 Segfaults when compiled with ... Closed
Operating System: ALL
Participants:

 Description   

Found with a silly PyMongo test that just started failing. Reproduced in the shell:

> db.test.group({key: {}, condition: {}, initial: {}, reduce: "5 ++ 5"})
2015-02-09T16:35:10.995-0800 I NETWORK  DBClientCursor::init call() failed
2015-02-09T16:35:10.998-0800 E QUERY    Error: error doing query: failed
    at DBQuery._exec (src/mongo/shell/query.js:83:36)
    at DBQuery.hasNext (src/mongo/shell/query.js:240:10)
    at DBCollection.findOne (src/mongo/shell/collection.js:186:19)
    at DB.runCommand (src/mongo/shell/db.js:58:41)
    at DB.groupcmd (src/mongo/shell/db.js:543:20)
    at DBCollection.group (src/mongo/shell/collection.js:1284:21)
    at (shell):1:9 at src/mongo/shell/query.js:83
2015-02-09T16:35:10.999-0800 I NETWORK  trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2015-02-09T16:35:10.999-0800 W NETWORK  Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
2015-02-09T16:35:10.999-0800 I NETWORK  reconnect 127.0.0.1:27017 (127.0.0.1) failed failed couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed

mongod log at full verbosity:

2015-02-09T16:35:10.971-0800 D COMMAND  [conn1] run command test.$cmd { group: { key: {}, condition: {}, initial: {}, ns: "test", $reduce: "5 ++ 5" } }
2015-02-09T16:35:10.981-0800 D QUERY    [conn1] V8Scope 0x29b6000 registered for op 16
2015-02-09T16:35:10.981-0800 D QUERY    [conn1] [QLOG] Running query:
ns=test.system.js limit=0 skip=0
Tree: $and
Sort: {}
Proj: {}
2015-02-09T16:35:10.981-0800 D QUERY    [conn1] Running query: query: {} sort: {} projection: {} skip: 0 limit: 0
2015-02-09T16:35:10.981-0800 D QUERY    [conn1] Collection test.system.js does not exist. Using EOF plan: query: {} sort: {} projection: {} skip: 0 limit: 0
2015-02-09T16:35:10.981-0800 D QUERY    [conn1] [QLOG] Not caching executor but returning 0 results.
2015-02-09T16:35:10.981-0800 I QUERY    [conn1] query test.system.js planSummary: EOF ntoreturn:0 ntoskip:0 nscanned:0 nscannedObjects:0 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:0 reslen:20 0ms
2015-02-09T16:35:10.983-0800 E QUERY    [conn1] SyntaxError: Unexpected number at $group reduce setup
2015-02-09T16:35:10.983-0800 D -        [conn1] User Assertion: 16722:SyntaxError: Unexpected number at $group reduce setup
2015-02-09T16:35:10.983-0800 D QUERY    [conn1] V8Scope 0x29b6000 unregistered for op 16
2015-02-09T16:35:10.983-0800 F -        [conn1] Invalid access at address: 0
2015-02-09T16:35:10.991-0800 F -        [conn1] Got signal: 11 (Segmentation fault).
 
 0x1134d4e 0x11344f2 0x11348e6 0x7f3a107e5180 0x1252636 0x10a410c 0x10a458d 0x10990b7 0x1099129 0x1093cf4 0x10941e9 0xac3a0c 0xcfb32a 0x96576e 0x9ccc6d 0xa6f29a 0xa704be 0xa7134c 0xcbed85 0xb92994 0x7f9030 0x10effe3 0x7f3a107dc2d4 0x7f3a0f8efc0d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"D34D4E"},{"b":"400000","o":"D344F2"},{"b":"400000","o":"D348E6"},{"b":"7F3A107D5000","o":"10180"},{"b":"400000","o":"E52636"},{"b":"400000","o":"CA410C"},{"b":"400000","o":"CA458D"},{"b":"400000","o":"C990B7"},{"b":"400000","o":"C99129"},{"b":"400000","o":"C93CF4"},{"b":"400000","o":"C941E9"},{"b":"400000","o":"6C3A0C"},{"b":"400000","o":"8FB32A"},{"b":"400000","o":"56576E"},{"b":"400000","o":"5CCC6D"},{"b":"400000","o":"66F29A"},{"b":"400000","o":"6704BE"},{"b":"400000","o":"67134C"},{"b":"400000","o":"8BED85"},{"b":"400000","o":"792994"},{"b":"400000","o":"3F9030"},{"b":"400000","o":"CEFFE3"},{"b":"7F3A107D5000","o":"72D4"},{"b":"7F3A0F807000","o":"E8C0D"}],"processInfo":{ "mongodbVersion" : "3.0.0-rc9-pre-", "gitVersion" : "425e072cc40c4389d1761f43f2900c117c96909c", "uname" : { "sysname" : "Linux", "release" : "3.19.0-gentoo", "version" : "#1 SMP PREEMPT Mon Feb 9 09:28:22 PST 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF6B0CC000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "31327C2EC653AA31647B1433CA6DCF85E7EAF554" }, { "b" : "7F3A107D5000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F3A105CD000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F3A103C9000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7F3A100B9000", "path" : "/usr/lib/gcc/x86_64-pc-linux-gnu/4.9.2/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F3A0FDBE000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F3A0FBA7000", "path" : "/usr/lib/gcc/x86_64-pc-linux-gnu/4.9.2/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F3A0F807000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F3A109F0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x3E) [0x1134d4e]
 mongod(+0xD344F2) [0x11344f2]
 mongod(+0xD348E6) [0x11348e6]
 libpthread.so.0(+0x10180) [0x7f3a107e5180]
 mongod(_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x16) [0x1252636]
 mongod(_ZN5boost6detail17sp_counted_impl_pIN5mongo10BSONHolderEE7disposeEv+0x2C) [0x10a410c]
 mongod(_ZN5mongo10ObjTrackerINS_10BSONHolderEED1Ev+0x8D) [0x10a458d]
 mongod(_ZN5mongo7V8ScopeD1Ev+0x207) [0x10990b7]
 mongod(_ZN5mongo7V8ScopeD0Ev+0x9) [0x1099129]
 mongod(_ZN5mongo11PooledScopeD1Ev+0x954) [0x1093cf4]
 mongod(_ZN5mongo11PooledScopeD0Ev+0x9) [0x10941e9]
 mongod(_ZN5mongo10GroupStageD0Ev+0x6C) [0xac3a0c]
 mongod(_ZN5mongo12PlanExecutorD1Ev+0x6A) [0xcfb32a]
 mongod(_ZN5boost14checked_deleteIN5mongo12PlanExecutorEEEvPT_+0xE) [0x96576e]
 mongod(_ZN5mongo12GroupCommand3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xC3D) [0x9ccc6d]
 mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3A) [0xa6f29a]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xF8E) [0xa704be]
 mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x62C) [0xa7134c]
 mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5CurOpES3_b+0x21C5) [0xcbed85]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortEb+0xA34) [0xb92994]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xF0) [0x7f9030]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x2F3) [0x10effe3]
 libpthread.so.0(+0x72D4) [0x7f3a107dc2d4]
 libc.so.6(clone+0x6D) [0x7f3a0f8efc0d]
-----  END BACKTRACE  -----

Version:

$ ./mongod --version
db version v3.0.0-rc9-pre-
git version: 425e072cc40c4389d1761f43f2900c117c96909c



 Comments   
Comment by Ramon Fernandez Marina [ 10/Feb/15 ]

Closing as a dup of SERVER-13824.

Comment by Bernie Hackett [ 10/Feb/15 ]

OK, building with --opt=off fixes the problem. This is definitely a problem with gcc 4.9 and our vendored version V8 with any optimizations.

Comment by Bernie Hackett [ 10/Feb/15 ]

Huh - SERVER-13824 and SERVER-16985 also. That last one seems to have been reported by gentoo devs. Though note that I'm just using "scons -j16 core" to build. Not trying to use system libraries instead of vendored or custom -march, etc.

Comment by Bernie Hackett [ 10/Feb/15 ]

Indeed it was:

$ gcc --version
gcc (Gentoo 4.9.2 p1.0, pie-0.6.2) 4.9.2

So, V8 and gcc 4.9 don't get along?

Comment by Kamran K. [ 10/Feb/15 ]

Was the crash-prone server built with gcc 4.9? There have been a few similar crashes reported with 4.9: SERVER-17104 and SERVER-16394

Comment by Bernie Hackett [ 10/Feb/15 ]

Also can't reproduce using 3.0-rc8 from the downloads site. Nor can I reproduce it with the version in the description built on my mac. It seems like there must be something about builds done on this particular linux box.

Comment by Bernie Hackett [ 10/Feb/15 ]

Hmmm... I can't reproduce this with the nightly from the downloads site (git version: 8cf51d1d88c8301253ba42c44db0f2fcb0e9a62e). Maybe this is something weird with the tool chain on this machine. The "Invalid access at address: 0" is a little weird.

Generated at Thu Feb 08 03:43:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.