[SERVER-28817] Group command segfault Created: 17/Apr/17  Updated: 07/Sep/17  Resolved: 17/Aug/17

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Bernie Hackett Assignee: Andrew Morrow (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongo.log    
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

This started occurring sometime after I upgraded clang to 4.0. That could be the cause, or a coincidence. I can't seem to get mongo built with gcc anymore.

The cause of the segfault is in this test:

https://github.com/mongodb/mongo-python-driver/blob/3.4.0/test/test_collection.py#L1621

You can run it like this:

python -m unittest -v test.test_collection.TestCollection.test_group

Log output with increased verbosity:

2017-04-16T18:00:26.920-0700 D COMMAND  [conn8] run command pymongo_test.$cmd { drop: "test", writeConcern: {} }
2017-04-16T18:00:26.920-0700 I COMMAND  [conn8] CMD: drop pymongo_test.test
2017-04-16T18:00:26.920-0700 D STORAGE  [conn8] dropCollection: pymongo_test.test
2017-04-16T18:00:26.920-0700 D INDEX    [conn8] 	 dropAllIndexes dropping: { v: 2, key: { _id: 1 }, name: "_id_", ns: "pymongo_test.test" }
2017-04-16T18:00:26.920-0700 D STORAGE  [conn8] pymongo_test.test: clearing plan cache - collection info cache reset
2017-04-16T18:00:26.920-0700 D STORAGE  [conn8] 	 dropIndexes done
2017-04-16T18:00:26.920-0700 D STORAGE  [conn8] deleting metadata for pymongo_test.test @ RecordId(107)
2017-04-16T18:00:26.920-0700 D STORAGE  [conn8] WT drop of  table:index-79--369124514706915650 res 16
2017-04-16T18:00:26.920-0700 D STORAGE  [conn8] ~WiredTigerRecordStore for: pymongo_test.test
2017-04-16T18:00:26.921-0700 D STORAGE  [conn8] WT drop of  table:collection-78--369124514706915650 res 16
2017-04-16T18:00:26.921-0700 D REPL     [conn8] Waiting for write concern. OpTime: { ts: Timestamp 0|0, t: -1 }, write concern: { w: 1, wtimeout: 0 }
2017-04-16T18:00:26.921-0700 I COMMAND  [conn8] command pymongo_test.test command: drop { drop: "test", writeConcern: {} } numYields:0 reslen:80 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { W: 1 } } } protocol:op_query 0ms
2017-04-16T18:00:26.921-0700 D COMMAND  [conn8] run command pymongo_test.$cmd { group: { $reduce: function (obj, prev) { prev.count++; }, ns: "test", cond: {}, key: {}, initial: { count: 0 } } }
2017-04-16T18:00:26.931-0700 D QUERY    [js] SMScope 0x560845da8000 registered for op 747
2017-04-16T18:00:26.931-0700 D QUERY    [conn8] Running query: query: {} sort: {} projection: {}
2017-04-16T18:00:26.931-0700 D QUERY    [conn8] Collection pymongo_test.system.js does not exist. Using EOF plan: query: {} sort: {} projection: {}
2017-04-16T18:00:26.931-0700 I COMMAND  [conn8] query pymongo_test.system.js query: { find: "system.js" } planSummary: EOF ntoreturn:0 ntoskip:0 keysExamined:0 docsExamined:0 cursorExhausted:1 numYields:0 nreturned:0 reslen:20 locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, Collection: { acquireCount: { r: 2 } } } 0ms
2017-04-16T18:00:26.931-0700 D QUERY    [js] ImplScope 0x56084234e500 unregistered for op 747
2017-04-16T18:00:26.931-0700 I COMMAND  [conn8] command pymongo_test.test command: group { group: { $reduce: function (obj, prev) { prev.count++; }, ns: "test", cond: {}, key: {}, initial: { count: 0 } } } planSummary: EOF keysExamined:0 docsExamined:0 numYields:0 reslen:79 locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, Collection: { acquireCount: { r: 2 } } } protocol:op_query 10ms
2017-04-16T18:00:26.931-0700 D COMMAND  [conn8] run command pymongo_test.$cmd { insert: "test", ordered: true, documents: 3 }
2017-04-16T18:00:26.931-0700 D STORAGE  [conn8] create collection pymongo_test.test {}
2017-04-16T18:00:26.931-0700 D STORAGE  [conn8] stored meta data for pymongo_test.test @ RecordId(108)
2017-04-16T18:00:26.931-0700 D STORAGE  [conn8] WiredTigerKVEngine::createRecordStore uri: table:collection-80--369124514706915650 config: type=file,memory_page_max=10m,split_pct=90,leaf_value_max=64MB,checksum=on,block_compressor=snappy,,key_format=q,value_format=u,app_metadata=(formatVersion=1)
2017-04-16T18:00:26.960-0700 D STORAGE  [conn8] WiredTigerUtil::checkApplicationMetadataFormatVersion  uri: table:collection-80--369124514706915650 ok range 1 -> 1 current: 1
2017-04-16T18:00:26.960-0700 D STORAGE  [conn8] pymongo_test.test: clearing plan cache - collection info cache reset
2017-04-16T18:00:26.960-0700 D STORAGE  [conn8] WiredTigerKVEngine::createSortedDataInterface ident: index-81--369124514706915650 config: type=file,internal_page_max=16k,leaf_page_max=16k,checksum=on,prefix_compression=true,block_compressor=,,,,key_format=u,value_format=u,app_metadata=(formatVersion=8,infoObj={ "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_", "ns" : "pymongo_test.test" }),
2017-04-16T18:00:26.960-0700 D STORAGE  [conn8] create uri: table:index-81--369124514706915650 config: type=file,internal_page_max=16k,leaf_page_max=16k,checksum=on,prefix_compression=true,block_compressor=,,,,key_format=u,value_format=u,app_metadata=(formatVersion=8,infoObj={ "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_", "ns" : "pymongo_test.test" }),
2017-04-16T18:00:26.980-0700 D STORAGE  [conn8] WiredTigerUtil::checkApplicationMetadataFormatVersion  uri: table:index-81--369124514706915650 ok range 6 -> 8 current: 8
2017-04-16T18:00:26.980-0700 D STORAGE  [conn8] pymongo_test.test: clearing plan cache - collection info cache reset
2017-04-16T18:00:26.980-0700 D INDEX    [conn8] marking index _id_ as ready in snapshot id 2189
2017-04-16T18:00:26.980-0700 D REPL     [conn8] Waiting for write concern. OpTime: { ts: Timestamp 0|0, t: -1 }, write concern: { w: 1, wtimeout: 0 }
2017-04-16T18:00:26.980-0700 I COMMAND  [conn8] command pymongo_test.test command: insert { insert: "test", ordered: true, documents: 3 } ninserted:3 keysInserted:3 numYields:0 reslen:44 locks:{ Global: { acquireCount: { r: 3, w: 3 } }, Database: { acquireCount: { w: 2, W: 1 } }, Collection: { acquireCount: { w: 2 } } } protocol:op_query 48ms
2017-04-16T18:00:26.980-0700 D COMMAND  [conn8] run command pymongo_test.$cmd { group: { $reduce: function (obj, prev) { prev.count++; }, ns: "test", cond: {}, key: {}, initial: { count: 0 } } }
2017-04-16T18:00:26.980-0700 D QUERY    [conn8] Only one plan is available; it will be run but will not be cached. query: {} sort: {} projection: {}, planSummary: COLLSCAN
2017-04-16T18:00:26.980-0700 D QUERY    [js] SMScope 0x560845da8000 registered for op 749
2017-04-16T18:00:26.981-0700 D QUERY    [js] ImplScope 0x56084234e500 unregistered for op 749
2017-04-16T18:00:26.981-0700 I COMMAND  [conn8] command pymongo_test.test command: group { group: { $reduce: function (obj, prev) { prev.count++; }, ns: "test", cond: {}, key: {}, initial: { count: 0 } } } planSummary: COLLSCAN keysExamined:0 docsExamined:3 numYields:0 reslen:102 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 0ms
2017-04-16T18:00:26.981-0700 D COMMAND  [conn8] run command pymongo_test.$cmd { group: { $reduce: function (obj, prev) { prev.count++; }, ns: "test", cond: { a: { $gt: 1 } }, key: {}, initial: { count: 0 } } }
2017-04-16T18:00:26.981-0700 D QUERY    [conn8] Only one plan is available; it will be run but will not be cached. query: { a: { $gt: 1 } } sort: {} projection: {}, planSummary: COLLSCAN
2017-04-16T18:00:26.981-0700 D QUERY    [js] SMScope 0x560845da8000 registered for op 750
2017-04-16T18:00:26.981-0700 D QUERY    [js] ImplScope 0x56084234e500 unregistered for op 750
2017-04-16T18:00:26.981-0700 I COMMAND  [conn8] command pymongo_test.test command: group { group: { $reduce: function (obj, prev) { prev.count++; }, ns: "test", cond: { a: { $gt: 1 } }, key: {}, initial: { count: 0 } } } planSummary: COLLSCAN keysExamined:0 docsExamined:3 numYields:0 reslen:102 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 0ms
2017-04-16T18:00:26.981-0700 D COMMAND  [conn8] run command pymongo_test.$cmd { insert: "test", ordered: true, documents: [ { a: 2, b: 3, _id: ObjectId('58f413aafa5bd83bbc77c815') } ] }
2017-04-16T18:00:26.981-0700 D REPL     [conn8] Waiting for write concern. OpTime: { ts: Timestamp 0|0, t: -1 }, write concern: { w: 1, wtimeout: 0 }
2017-04-16T18:00:26.981-0700 I COMMAND  [conn8] command pymongo_test.test command: insert { insert: "test", ordered: true, documents: [ { a: 2, b: 3, _id: ObjectId('58f413aafa5bd83bbc77c815') } ] } ninserted:1 keysInserted:1 numYields:0 reslen:44 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { w: 1 } }, Collection: { acquireCount: { w: 1 } } } protocol:op_query 0ms
2017-04-16T18:00:26.982-0700 D COMMAND  [conn8] run command pymongo_test.$cmd { group: { $reduce: function (obj, prev) { prev.count++; }, ns: "test", cond: {}, key: { a: 1 }, initial: { count: 0 } } }
2017-04-16T18:00:26.982-0700 D QUERY    [conn8] Only one plan is available; it will be run but will not be cached. query: {} sort: {} projection: {}, planSummary: COLLSCAN
2017-04-16T18:00:26.982-0700 D QUERY    [js] SMScope 0x560845da8000 registered for op 752
2017-04-16T18:00:26.982-0700 D QUERY    [js] ImplScope 0x56084234e500 unregistered for op 752
2017-04-16T18:00:26.982-0700 I COMMAND  [conn8] command pymongo_test.test command: group { group: { $reduce: function (obj, prev) { prev.count++; }, ns: "test", cond: {}, key: { a: 1 }, initial: { count: 0 } } } planSummary: COLLSCAN keysExamined:0 docsExamined:4 numYields:0 reslen:173 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 0ms
2017-04-16T18:00:26.982-0700 D COMMAND  [conn8] run command pymongo_test.$cmd { group: { finalize: function (obj) { obj.count++; }, $reduce: function (obj, prev) { prev.count++; }, initial: { count: 0 }, cond: {}, key: { a: 1 }, ns: "test" } }
2017-04-16T18:00:26.982-0700 D QUERY    [conn8] Only one plan is available; it will be run but will not be cached. query: {} sort: {} projection: {}, planSummary: COLLSCAN
2017-04-16T18:00:26.982-0700 D QUERY    [js] SMScope 0x560845da8000 registered for op 753
2017-04-16T18:00:26.983-0700 D QUERY    [js] ImplScope 0x56084234e500 unregistered for op 753
2017-04-16T18:00:26.983-0700 I COMMAND  [conn8] command pymongo_test.test command: group { group: { finalize: function (obj) { obj.count++; }, $reduce: function (obj, prev) { prev.count++; }, initial: { count: 0 }, cond: {}, key: { a: 1 }, ns: "test" } } planSummary: COLLSCAN keysExamined:0 docsExamined:4 numYields:0 reslen:173 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 0ms
2017-04-16T18:00:26.983-0700 D COMMAND  [conn8] run command pymongo_test.$cmd { group: { finalize: function (obj) { return obj.count; }, $reduce: function (obj, prev) { prev.count++; }, initial: { count: 0 }, cond: {}, key: { a: 1 }, ns: "test" } }
2017-04-16T18:00:26.983-0700 D QUERY    [conn8] Only one plan is available; it will be run but will not be cached. query: {} sort: {} projection: {}, planSummary: COLLSCAN
2017-04-16T18:00:26.983-0700 D QUERY    [js] SMScope 0x560845da8000 registered for op 754
2017-04-16T18:00:26.984-0700 D QUERY    [js] ImplScope 0x56084234e500 unregistered for op 754
2017-04-16T18:00:26.984-0700 I COMMAND  [conn8] command pymongo_test.test command: group { group: { finalize: function (obj) { return obj.count; }, $reduce: function (obj, prev) { prev.count++; }, initial: { count: 0 }, cond: {}, key: { a: 1 }, ns: "test" } } planSummary: COLLSCAN keysExamined:0 docsExamined:4 numYields:0 reslen:112 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 0ms
2017-04-16T18:00:26.984-0700 D COMMAND  [conn8] run command pymongo_test.$cmd { group: { finalize: function (obj) { return obj.count; }, $reduce: function (obj, prev) { prev.count++; }, initial: { count: 0 }, cond: {}, $keyf: function (obj) { if (obj.a == 2) { return {a: true} }; return {b: true}; }, ns: "test" } }
2017-04-16T18:00:26.984-0700 D QUERY    [conn8] Only one plan is available; it will be run but will not be cached. query: {} sort: {} projection: {}, planSummary: COLLSCAN
2017-04-16T18:00:26.984-0700 D QUERY    [js] SMScope 0x560845da8000 registered for op 755
2017-04-16T18:00:26.984-0700 D QUERY    [js] ImplScope 0x56084234e500 unregistered for op 755
2017-04-16T18:00:26.985-0700 I COMMAND  [conn8] command pymongo_test.test command: group { group: { finalize: function (obj) { return obj.count; }, $reduce: function (obj, prev) { prev.count++; }, initial: { count: 0 }, cond: {}, $keyf: function (obj) { if (obj.a == 2) { return {a: true} }; return {b: true}; }, ns: "test" } } planSummary: COLLSCAN keysExamined:0 docsExamined:4 numYields:0 reslen:101 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 0ms
2017-04-16T18:00:26.985-0700 D COMMAND  [conn8] run command pymongo_test.$cmd { group: { $reduce: function (obj, prev) { prev.count++; }, ns: "test", cond: {}, initial: { count: 0 } } }
2017-04-16T18:00:26.985-0700 D QUERY    [conn8] Only one plan is available; it will be run but will not be cached. query: {} sort: {} projection: {}, planSummary: COLLSCAN
2017-04-16T18:00:26.985-0700 D QUERY    [js] SMScope 0x560845da8000 registered for op 756
2017-04-16T18:00:26.985-0700 D QUERY    [js] ImplScope 0x56084234e500 unregistered for op 756
2017-04-16T18:00:26.985-0700 I COMMAND  [conn8] command pymongo_test.test command: group { group: { $reduce: function (obj, prev) { prev.count++; }, ns: "test", cond: {}, initial: { count: 0 } } } planSummary: COLLSCAN keysExamined:0 docsExamined:4 numYields:0 reslen:102 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 0ms
2017-04-16T18:00:26.985-0700 D COMMAND  [conn8] run command pymongo_test.$cmd { group: { $reduce: 5 ++ 5, ns: "test", cond: {}, key: {}, initial: {} } }
2017-04-16T18:00:26.985-0700 D QUERY    [conn8] Only one plan is available; it will be run but will not be cached. query: {} sort: {} projection: {}, planSummary: COLLSCAN
2017-04-16T18:00:26.985-0700 D QUERY    [js] SMScope 0x560845da8000 registered for op 757
2017-04-16T18:00:26.985-0700 E QUERY    [js] SyntaxError: invalid increment operand @group reduce init:1:10
2017-04-16T18:00:26.985-0700 D -        [js] User Assertion: 139:SyntaxError: invalid increment operand @group reduce init:1:10
 src/mongo/scripting/mozjs/implscope.cpp 918
2017-04-16T18:00:26.986-0700 D -        [conn8] User Assertion: 139:SyntaxError: invalid increment operand @group reduce init:1:10
 src/mongo/scripting/mozjs/proxyscope.cpp 295
2017-04-16T18:00:26.986-0700 F -        [conn8] Invalid access at address: 0x560845af9780
2017-04-16T18:00:26.987-0700 F -        [conn8] Got signal: 11 (Segmentation fault).
 
 0x56083f78d507 0x56083f78d045 0x56083fae4da8 0x7f102b6ccf9f 0x560845af9780 0x56084590edaf
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"56083E406000","o":"1387507","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"},{"b":"56083E406000","o":"1387045"},{"b":"56083E406000","o":"16DEDA8"},{"b":"7F102B6BC000","o":"10F9F"},{"b":"0","o":"560845AF9780"},{"b":"0","o":"56084590EDAF"}],"processInfo":{ "mongodbVersion" : "3.5.5-206-g932c2f3455", "gitVersion" : "932c2f345598d8e1d283e8c2bb54fd8d0e11c853", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "4.10.9-gentoo", "version" : "#1 SMP PREEMPT Tue Apr 11 22:31:37 PDT 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "56083E406000", "elfType" : 3, "buildId" : "FDB31ABC184A9B9C9778496241C88683BA9DBB9A" }, { "b" : "7FFEB58DF000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "2B5CF8A7A3924038EDF8887C27AC2E1754D42DF0" }, { "b" : "7F102E00A000", "path" : "/usr/lib64/libnetsnmpmibs.so.30", "elfType" : 3 }, { "b" : "7F102DE06000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7F102DBA2000", "path" : "/usr/lib64/libnetsnmpagent.so.30", "elfType" : 3 }, { "b" : "7F102D997000", "path" : "/lib64/libwrap.so.0", "elfType" : 3 }, { "b" : "7F102D6B3000", "path" : "/usr/lib64/libnetsnmp.so.30", "elfType" : 3 }, { "b" : "7F102D276000", "path" : "/usr/lib64/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "7F102CF7B000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F102CD30000", "path" : "/usr/lib64/libldap-2.4.so.2", "elfType" : 3 }, { "b" : "7F102CB21000", "path" : "/usr/lib64/liblber-2.4.so.2", "elfType" : 3 }, { "b" : "7F102C904000", "path" : "/usr/lib64/libsasl2.so.3", "elfType" : 3 }, { "b" : "7F102C6B9000", "path" : "/usr/lib64/libgssapi_krb5.so.2", "elfType" : 3 }, { "b" : "7F102C44E000", "path" : "/usr/lib64/libcurl.so.4", "elfType" : 3 }, { "b" : "7F102C1DF000", "path" : "/usr/lib64/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "7F102BFD7000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F102BD1E000", "path" : "/usr/lib64/libc++.so.1", "elfType" : 3 }, { "b" : "7F102BAF3000", "path" : "/usr/lib64/libc++abi.so.1", "elfType" : 3 }, { "b" : "7F102B8D8000", "path" : "/usr/lib64/libunwind.so.8", "elfType" : 3 }, { "b" : "7F102B6BC000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F102B320000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F102E496000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 }, { "b" : "7F102B109000", "path" : "/lib64/libz.so.1", "elfType" : 3 }, { "b" : "7F102AEF2000", "path" : "/lib64/libresolv.so.2", "elfType" : 3 }, { "b" : "7F102AC19000", "path" : "/usr/lib64/libkrb5.so.3", "elfType" : 3 }, { "b" : "7F102A9E5000", "path" : "/usr/lib64/libk5crypto.so.3", "elfType" : 3 }, { "b" : "7F102A7E1000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3 }, { "b" : "7F102A5D5000", "path" : "/usr/lib64/libkrb5support.so.0", "elfType" : 3 }, { "b" : "7F102A3D1000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3 }, { "b" : "7F102A1B6000", "path" : "/usr/lib64/sasl2/libotp.so", "elfType" : 3 }, { "b" : "7F1029FB0000", "path" : "/usr/lib64/sasl2/libcrammd5.so", "elfType" : 3 }, { "b" : "7F1029DAB000", "path" : "/usr/lib64/sasl2/libplain.so", "elfType" : 3 }, { "b" : "7F1029BA6000", "path" : "/usr/lib64/sasl2/liblogin.so", "elfType" : 3 }, { "b" : "7F102999F000", "path" : "/usr/lib64/sasl2/libsasldb.so", "elfType" : 3 }, { "b" : "7F1029791000", "path" : "/usr/lib64/libgdbm.so.4", "elfType" : 3 }, { "b" : "7F1029587000", "path" : "/usr/lib64/sasl2/libscram.so", "elfType" : 3 }, { "b" : "7F1029379000", "path" : "/usr/lib64/sasl2/libdigestmd5.so", "elfType" : 3 }, { "b" : "7F1029174000", "path" : "/usr/lib64/sasl2/libanonymous.so", "elfType" : 3 }, { "b" : "7F1028F6B000", "path" : "/usr/lib64/sasl2/libntlm.so", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x37) [0x56083f78d507]
 mongod(+0x1387045) [0x56083f78d045]
 mongod(+0x16DEDA8) [0x56083fae4da8]
 libpthread.so.0(+0x10F9F) [0x7f102b6ccf9f]
 ??? [0x560845af9780]
 ??? [0x56084590edaf]
-----  END BACKTRACE  -----
Segmentation fault (core dumped)



 Comments   
Comment by Bernie Hackett [ 17/Aug/17 ]

I agree. I can't reproduce this with GCC and have no better ideas.

Comment by Andrew Morrow (Inactive) [ 17/Aug/17 ]

I'm unable to reproduce this with a toolchain we trust. I suspect a Gentoo toolchain issue, but of course cannot prove that. behackett feel free to re-open this if you find additional useful information.

Comment by Andrew Morrow (Inactive) [ 27/Jul/17 ]

behackett - Unfortunately, without more information to go on, I'm not sure what next steps we can take. I strongly suspect a toolchain issue on your platform since we aren't seeing this issue on other platforms. I guess the next thing I'd like to know is whether or not you reproduce this issue with GCC? Also try valgrind and ASAN with GCC even if it doesn't crash, to see if it complains.

Comment by Bernie Hackett [ 17/Jul/17 ]

I can still reproduce this on my other machine. Building with --sanitize=address fails with this error:

Compiling build/scons/opt/sconf_temp/conftest_50.o
Linking build/scons/opt/sconf_temp/conftest_50
/usr/lib64/llvm/4/bin/../../../../lib/clang/4.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_unwind_linux_libcdep.cc.o): In function `__sanitizer::Unwind_GetIP(_Unwind_Context*)':
/var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/sanitizer_common/sanitizer_unwind_linux_libcdep.cc:98: undefined reference to `_Unwind_GetIP'
/usr/lib64/llvm/4/bin/../../../../lib/clang/4.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_unwind_linux_libcdep.cc.o): In function `__sanitizer::BufferedStackTrace::SlowUnwindStack(unsigned long, unsigned int)':
/var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/sanitizer_common/sanitizer_unwind_linux_libcdep.cc:125: undefined reference to `_Unwind_Backtrace'
/var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/sanitizer_common/sanitizer_unwind_linux_libcdep.cc:125: undefined reference to `_Unwind_Backtrace'
/usr/lib64/llvm/4/bin/../../../../lib/clang/4.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_unwind_linux_libcdep.cc.o): In function `__sanitizer::Unwind_GetIP(_Unwind_Context*)':
/var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/sanitizer_common/sanitizer_unwind_linux_libcdep.cc:98: undefined reference to `_Unwind_GetIP'
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)

git hash: a1c67941bf08c69cab04eba20bc9ce9a763e1c7f

Comment by Bernie Hackett [ 11/Jul/17 ]

OK, I have no idea what's changed, but I can't reproduce this anymore regardless of --allocator, --sanitize, --ssl, enterprise modules, or any other combination thereof I've tried.

This is the git hash I'm on right now: 6a649373bde66a24ecb286aceb14bd12bcaa5a1c

I'll try again tonight on my home machine which had the same problem and report back.

Comment by Andrew Morrow (Inactive) [ 06/Jul/17 ]

Interesting that ASAN makes it go away. It feels sort of like stack corruption given that both our backtracer and valgrinds can't give us a real stacktrace. Does it repro if you build just without tcmalloc?

scons CC=$(which clang) CXX=$(which clang++) --libc++ -j24 --disable-warnings-as-errors --nostrip --allocator=system core

Comment by Bernie Hackett [ 02/Jul/17 ]

I wonder if this is some incompatibility between clang 4.x and the version of tcmalloc we vendor? I see we can't use --sanitize and tcmalloc together.

Comment by Bernie Hackett [ 02/Jul/17 ]

Attached full log of stderr and stdout running under valgrind.

Comment by Bernie Hackett [ 02/Jul/17 ]

Running under valgrind:

2017-07-02T10:54:57.522-0700 I COMMAND  [conn5] command pymongo_test.test command: group { group: { finalize: function (obj) { obj.count++; }, $reduce: function (obj, prev) { prev.count++; }, initial: { count: 0 }, cond: {}, key: { a: 1 }, ns: "test" }, $readPreference: { mode: "secondaryPreferred" }, $db: "pymongo_test" } planSummary: COLLSCAN keysExamined:0 docsExamined:4 numYields:1 reslen:173 locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, Collection: { acquireCount: { r: 2 } } } protocol:op_query 108ms
2017-07-02T10:54:57.632-0700 E QUERY    [js] SyntaxError: invalid increment operand @group reduce init:1:10
vex amd64->IR: unhandled instruction bytes: 0x17 0x0 0x0 0x0 0x0 0x50 0x79 0x91 0x17 0x0
vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
==7919== valgrind: Unrecognised instruction at address 0xb123ce3.
==7919==    at 0xB123CE3: ???
==7919==    by 0x17917F3F: ???
==7919==    by 0x1: ???
==7919== Your program just tried to execute an instruction that Valgrind
==7919== did not recognise.  There are two possible reasons for this.
==7919== 1. Your program has a bug and erroneously jumped to a non-code
==7919==    location.  If you are running Memcheck and you just saw a
==7919==    warning about a bad jump, it's probably your program's fault.
==7919== 2. The instruction is legitimate but Valgrind doesn't handle it,
==7919==    i.e. it's Valgrind's fault.  If you think this is the case or
==7919==    you are not sure, please let us know and we'll try to fix it.
==7919== Either way, Valgrind will now raise a SIGILL signal which will
==7919== probably kill your program.
2017-07-02T10:54:57.682-0700 F -        [conn5] Invalid operation at address: 0xb123ce3
2017-07-02T10:54:57.712-0700 F -        [conn5] Got signal: 4 (Illegal instruction).
 
 0x1d635b7 0x1d630f5 0x5c53f3f 0xb123ce3 0x17917f3f
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"108000","o":"1C5B5B7","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"},{"b":"108000","o":"1C5B0F5"},{"b":"5C43000","o":"10F3F"},{"b":"0","o":"B123CE3"},{"b":"0","o":"17917F3F"}],"processInfo":{ "mongodbVersion" : "3.5.9-162-gedc858ca3a", "gitVersion" : "edc858ca3a32974c07f027e6860a52ac6e89039b", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.11.6-gentoo", "version" : "#1 SMP PREEMPT Thu Jun 22 19:31:50 PDT 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "108000", "elfType" : 3, "buildId" : "F6C79C32AE32CB9F905EA683B247B60C4D85AB18" }, { "b" : "4A25000", "path" : "/usr/lib64/valgrind/vgpreload_core-amd64-linux.so", "elfType" : 3 }, { "b" : "4C27000", "path" : "/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so", "elfType" : 3 }, { "b" : "4E37000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "5131000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "5339000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "553D000", "path" : "/usr/lib64/libc++.so.1", "elfType" : 3 }, { "b" : "57F6000", "path" : "/usr/lib64/libc++abi.so.1", "elfType" : 3 }, { "b" : "5A21000", "path" : "/usr/lib64/libunwind.so.8", "elfType" : 3 }, { "b" : "5C43000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "5E5F000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "4000000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x37) [0x1d635b7]
 mongod(+0x1C5B0F5) [0x1d630f5]
 libpthread.so.0(+0x10F3F) [0x5c53f3f]
 ??? [0xb123ce3]
 ??? [0x17917f3f]
-----  END BACKTRACE  -----
==7919== 
==7919== Process terminating with default action of signal 4 (SIGILL): dumping core
==7919==    at 0x5C53DDD: raise (raise.c:53)
==7919==    by 0x1D63163: endProcessWithSignal (signal_handlers_synchronous.cpp:90)
==7919==    by 0x1D63163: mongo::(anonymous namespace)::abruptQuitWithAddrSignal(int, siginfo_t*, void*) (signal_handlers_synchronous.cpp:279)
==7919==    by 0x5C53F3F: ??? (in /lib64/libpthread-2.24.so)
==7919==    by 0xB123CE2: ???
==7919==    by 0x17917F3F: ???
==7919==    by 0x1: ???
==7919== 
==7919== HEAP SUMMARY:
==7919==     in use at exit: 67,008,869 bytes in 12,686 blocks
==7919==   total heap usage: 46,363 allocs, 33,677 frees, 88,667,103 bytes allocated
==7919== 
==7919== LEAK SUMMARY:
==7919==    definitely lost: 4,444 bytes in 42 blocks
==7919==    indirectly lost: 90,767 bytes in 69 blocks
==7919==      possibly lost: 52,240 bytes in 104 blocks
==7919==    still reachable: 66,861,418 bytes in 12,471 blocks
==7919==                       of which reachable via heuristic:
==7919==                         newarray           : 436,288 bytes in 40 blocks
==7919==                         multipleinheritance: 432 bytes in 1 blocks
==7919==         suppressed: 0 bytes in 0 blocks
==7919== Rerun with --leak-check=full to see details of leaked memory
==7919== 
==7919== For counts of detected and suppressed errors, rerun with: -v
==7919== ERROR SUMMARY: 2370 errors from 1000 contexts (suppressed: 0 from 0)
Illegal instruction (core dumped)

Comment by Bernie Hackett [ 02/Jul/17 ]

I've done a bit more investigation. If I build with --sanitize=address and --allocator=system, the segfault does not occur:

$ scons CC=$(which clang) CXX=$(which clang++) --libc++ -j24 --disable-warnings-as-errors --nostrip --sanitize=address --allocator=system core

If I build without those options the segfault does occur:

$ scons CC=$(which clang) CXX=$(which clang++) --libc++ -j24 --disable-warnings-as-errors --nostrip

In both cases I'm building without enterprise modules, and without ssl. I've also upgraded to clang 4.0.1, which made no difference.

Comment by Ian Whalen (Inactive) [ 30/Jun/17 ]

behackett have you had a chance to repro this?

Comment by Andrew Morrow (Inactive) [ 06/Jun/17 ]

Bernie I'm going to assign this back to you while you try some of my suggestions. Since I can't seem to repro it, there isn't a whole lot more that I can do. If you are able to get somewhere with ASAN or valgrind and have some additional actionable data, feel free to re-assign to me and I'll give it a look.

Comment by Andrew Morrow (Inactive) [ 05/Jun/17 ]

behackett - Well, that stack trace is pretty unhelpful, I agree. The fact that the this pointer is nullptr is suggestive, but without additional stack it is hard to know how that happened. I'm also still unable to reproduce this in my zesty chroot with the current tip of master, using clang 4.0. Is this still reproducing for you? If so, my next suggestions would be to build under ASAN (add --sanitize=address --allocator=system to your scons invocation) or run mongod under valgrind --soname-synonyms=somalloc=NONE and see what happens? Perhaps the stack is getting badly enough corrupted that GDB can't see it, but ASAN or valgrind will catch it as it happens?

Comment by Bernie Hackett [ 20/May/17 ]

Sadly the backtrace doesn't provide much information. I have debug symbols installed for glibc and libc++ and built mongod with --nostrip.

$ gdb --args ./mongod --dbpath ~/data/db
GNU gdb (Gentoo 7.12.1 vanilla) 7.12.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./mongod...done.
warning: File "/home/durin/work/mongo/.gdbinit" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
	add-auto-load-safe-path /home/durin/work/mongo/.gdbinit
line to your configuration file "/home/durin/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/home/durin/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
(gdb) run
Starting program: /home/durin/work/mongo/mongod --dbpath /home/durin/data/db
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff28af700 (LWP 23717)]
[New Thread 0x7ffff20ae700 (LWP 23718)]
2017-05-19T20:04:58.417-0700 I CONTROL  [initandlisten] MongoDB starting : pid=23713 port=27017 dbpath=/home/durin/data/db 64-bit host=devbox
2017-05-19T20:04:58.417-0700 I CONTROL  [initandlisten] db version v3.5.7-88-g0acaccdf81
2017-05-19T20:04:58.417-0700 I CONTROL  [initandlisten] git version: 0acaccdf81a00222144253290f9cc3b4fd76e122
2017-05-19T20:04:58.417-0700 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2k  26 Jan 2017
2017-05-19T20:04:58.417-0700 I CONTROL  [initandlisten] allocator: tcmalloc
2017-05-19T20:04:58.417-0700 I CONTROL  [initandlisten] modules: enterprise 
2017-05-19T20:04:58.417-0700 I CONTROL  [initandlisten] build environment:
2017-05-19T20:04:58.417-0700 I CONTROL  [initandlisten]     distarch: x86_64
2017-05-19T20:04:58.417-0700 I CONTROL  [initandlisten]     target_arch: x86_64
2017-05-19T20:04:58.417-0700 I CONTROL  [initandlisten] options: { storage: { dbPath: "/home/durin/data/db" } }
2017-05-19T20:04:58.417-0700 W -        [initandlisten] Detected unclean shutdown - /home/durin/data/db/mongod.lock is not empty.
2017-05-19T20:04:58.433-0700 I -        [initandlisten] Detected data files in /home/durin/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-05-19T20:04:58.433-0700 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-05-19T20:04:58.433-0700 I STORAGE  [initandlisten] 
2017-05-19T20:04:58.433-0700 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-05-19T20:04:58.433-0700 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-05-19T20:04:58.433-0700 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7468M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2017-05-19T20:04:58.567-0700 I STORAGE  [initandlisten] WiredTiger message [1495249498:567091][23713:0x7ffff7fb1d40], txn-recover: Main recovery loop: starting at 4/128
[New Thread 0x7ffff18ad700 (LWP 23719)]
[New Thread 0x7ffff10ac700 (LWP 23720)]
[New Thread 0x7ffff08ab700 (LWP 23721)]
[New Thread 0x7ffff00aa700 (LWP 23722)]
2017-05-19T20:04:58.567-0700 I STORAGE  [initandlisten] WiredTiger message [1495249498:567594][23713:0x7ffff7fb1d40], txn-recover: Recovering log 4 through 5
2017-05-19T20:04:58.642-0700 I STORAGE  [initandlisten] WiredTiger message [1495249498:642845][23713:0x7ffff7fb1d40], file:index-1-526602474019102121.wt, txn-recover: Recovering log 5 through 5
[Thread 0x7ffff00aa700 (LWP 23722) exited]
[New Thread 0x7ffff18ad700 (LWP 23723)]
[Thread 0x7ffff08ab700 (LWP 23721) exited]
[Thread 0x7ffff10ac700 (LWP 23720) exited]
[Thread 0x7ffff18ad700 (LWP 23719) exited]
[New Thread 0x7ffff10ac700 (LWP 23724)]
[New Thread 0x7ffff08ab700 (LWP 23725)]
[New Thread 0x7ffff00aa700 (LWP 23726)]
[New Thread 0x7fffef8a9700 (LWP 23727)]
[New Thread 0x7fffef0a8700 (LWP 23728)]
[New Thread 0x7fffee8a7700 (LWP 23729)]
[New Thread 0x7fffee0a6700 (LWP 23730)]
[New Thread 0x7fffed8a5700 (LWP 23731)]
[New Thread 0x7fffed0a4700 (LWP 23732)]
2017-05-19T20:04:59.637-0700 I STORAGE  [initandlisten] dropping unused ident: collection-0--235991914205857805
2017-05-19T20:04:59.645-0700 I STORAGE  [initandlisten] dropping unused ident: index-1--235991914205857805
2017-05-19T20:04:59.660-0700 I CONTROL  [initandlisten] 
2017-05-19T20:04:59.660-0700 I CONTROL  [initandlisten] ** NOTE: This is a development version (3.5.7-88-g0acaccdf81) of MongoDB.
2017-05-19T20:04:59.660-0700 I CONTROL  [initandlisten] **       Not recommended for production.
2017-05-19T20:04:59.660-0700 I CONTROL  [initandlisten] 
2017-05-19T20:04:59.660-0700 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-05-19T20:04:59.660-0700 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2017-05-19T20:04:59.660-0700 I CONTROL  [initandlisten] 
2017-05-19T20:04:59.660-0700 I CONTROL  [initandlisten] ** WARNING: This server is bound to localhost.
2017-05-19T20:04:59.660-0700 I CONTROL  [initandlisten] **          Remote systems will be unable to connect to this server. 
2017-05-19T20:04:59.660-0700 I CONTROL  [initandlisten] **          Start the server with --bind_ip <address> to specify which IP 
2017-05-19T20:04:59.660-0700 I CONTROL  [initandlisten] **          addressses it should serve responses from, or with --bind_ip_all to
2017-05-19T20:04:59.660-0700 I CONTROL  [initandlisten] **          bind to all interfaces. If this behavior is desired, start the
2017-05-19T20:04:59.660-0700 I CONTROL  [initandlisten] **          server with --bind_ip 127.0.0.1 to disable this warning.
2017-05-19T20:04:59.660-0700 I CONTROL  [initandlisten] 
[New Thread 0x7fffec8a3700 (LWP 23733)]
[New Thread 0x7fffec0a2700 (LWP 23734)]
[Thread 0x7fffec8a3700 (LWP 23733) exited]
[New Thread 0x7fffec8a3700 (LWP 23735)]
2017-05-19T20:04:59.663-0700 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/home/durin/data/db/diagnostic.data'
[New Thread 0x7fffeb8a1700 (LWP 23736)]
[New Thread 0x7fffeb0a0700 (LWP 23737)]
[New Thread 0x7fffea89f700 (LWP 23738)]
[New Thread 0x7fffea09e700 (LWP 23739)]
[New Thread 0x7fffe989d700 (LWP 23740)]
2017-05-19T20:04:59.663-0700 I NETWORK  [thread1] waiting for connections on port 27017
2017-05-19T20:05:01.988-0700 I NETWORK  [thread1] connection accepted from 127.0.0.1:60240 #1 (1 connection now open)
[New Thread 0x7ffff7fb0700 (LWP 23747)]
2017-05-19T20:05:01.988-0700 I NETWORK  [conn1] received client metadata from 127.0.0.1:60240 conn: { driver: { name: "PyMongo", version: "3.5.0.dev0" }, os: { type: "Linux", name: "Gentoo Base System 2.3", architecture: "x86_64", version: "4.11.1-gentoo" }, platform: "CPython 2.7.13.final.0" }
2017-05-19T20:05:01.989-0700 I NETWORK  [thread1] connection accepted from 127.0.0.1:60242 #2 (2 connections now open)
[New Thread 0x7fffe909c700 (LWP 23748)]
2017-05-19T20:05:01.989-0700 I NETWORK  [conn2] received client metadata from 127.0.0.1:60242 conn: { driver: { name: "PyMongo", version: "3.5.0.dev0" }, os: { type: "Linux", name: "Gentoo Base System 2.3", architecture: "x86_64", version: "4.11.1-gentoo" }, platform: "CPython 2.7.13.final.0" }
2017-05-19T20:05:01.990-0700 I NETWORK  [thread1] connection accepted from 127.0.0.1:60244 #3 (3 connections now open)
2017-05-19T20:05:01.990-0700 I -        [conn2] end connection 127.0.0.1:60242 (3 connections now open)
2017-05-19T20:05:01.990-0700 I -        [conn1] end connection 127.0.0.1:60240 (3 connections now open)
[New Thread 0x7fffe8f9b700 (LWP 23751)]
[Thread 0x7fffe909c700 (LWP 23748) exited]
[Thread 0x7ffff7fb0700 (LWP 23747) exited]
2017-05-19T20:05:01.991-0700 I NETWORK  [conn3] received client metadata from 127.0.0.1:60244 conn: { driver: { name: "PyMongo", version: "3.5.0.dev0" }, os: { type: "Linux", name: "Gentoo Base System 2.3", architecture: "x86_64", version: "4.11.1-gentoo" }, platform: "CPython 2.7.13.final.0" }
2017-05-19T20:05:01.992-0700 I NETWORK  [thread1] connection accepted from 127.0.0.1:60246 #4 (2 connections now open)
[New Thread 0x7ffff7fb0700 (LWP 23752)]
2017-05-19T20:05:01.992-0700 I NETWORK  [conn4] received client metadata from 127.0.0.1:60246 conn: { driver: { name: "PyMongo", version: "3.5.0.dev0" }, os: { type: "Linux", name: "Gentoo Base System 2.3", architecture: "x86_64", version: "4.11.1-gentoo" }, platform: "CPython 2.7.13.final.0" }
2017-05-19T20:05:01.995-0700 I COMMAND  [conn4] CMD: drop pymongo_test.test
[New Thread 0x7fffe8e9a700 (LWP 23753)]
2017-05-19T20:05:02.058-0700 E QUERY    [js] SyntaxError: invalid increment operand @group reduce init:1:10
 
Thread 29 "conn4" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7fb0700 (LWP 23752)]
0x000055555c5806c0 in ?? ()
(gdb) bt
#0  0x000055555c5806c0 in ?? ()
#1  0x00007ffff7fad3f0 in ?? ()
#2  0x0000555556438794 in mongo::GroupStage::doWork (this=0x0, out=0x55555c125e60)
    at src/mongo/db/exec/group.cpp:212
#3  0x000055555c0f2f60 in ?? ()
#4  0x0000000000000002 in ?? ()
#5  0x0000000000000000 in ?? ()

Comment by Andrew Morrow (Inactive) [ 04/May/17 ]

behackett - I just tested out 3.4 as well, at e624205252429cc6b629dd3f03b90db957454bd6, and I was similarly unable to reproduce a crash. If you can still reproduce this, can you try running your mongod under GDB while running the test, to see if you can get a more complete backtrace? Can you also test again against the tip of 3.4 (or master) as appropriate, and see if you can still reproduce this issue?

Comment by Andrew Morrow (Inactive) [ 04/May/17 ]

behackett - I just built tip of master (f62ed9c43dbdcd659f622b6f95898a8e71e9db9e) on an Ubuntu Zesty chroot, where clang is 4.0:

$(which clang) --version
clang version 4.0.0-1ubuntu1 (tags/RELEASE_400/rc1)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

I then built mongodb per your build instructions:

buildscripts/scons.py CC=$(which clang) CXX=$(which clang++) -j12 --libc++ --ssl --disable-warnings-as-errors core
$ ./mongod --dbpath ~/var/tmp
2017-05-04T20:01:56.771+0000 I CONTROL  [initandlisten] MongoDB starting : pid=7087 port=27017 dbpath=/home/andrew/var/tmp 64-bit host=acm-workstation
...

I cloned the mongo-python-driver repository at 3359f850197f9129ed4d6f81be8fa96da1a8ff23, and ran the tests:

$ python -m unittest -v test.test_collection.TestCollection.test_group
test_group (test.test_collection.TestCollection) ... ok
 
----------------------------------------------------------------------
Ran 1 test in 0.094s
 
OK

There were no server crashes. Can you try this again with server master? With 3.4.latest?

Comment by Bernie Hackett [ 17/Apr/17 ]

Build command:

scons CC=$(which clang) CXX=$(which clang++) -j24 --libc++ --ssl --disable-warnings-as-errors core

Comment by Bernie Hackett [ 17/Apr/17 ]

$ clang --version
clang version 4.0.0 (tags/RELEASE_400/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm/4/bin

Also libcxx 4.0.0.

Comment by Eric Milkie [ 17/Apr/17 ]

To confirm, you're using vanilla version 4.0.0 of clang, released 25 days ago?
This will be interesting to debug. It might even be a compiler bug.

Comment by Bernie Hackett [ 17/Apr/17 ]

Looking a bit more closely, it looks like the test that causes the segfault intentionally causes an error, testing that we raise the correct exception in PyMongo. For some reason MongoDB master is crashing rather than returning an error. The test is here:

https://github.com/mongodb/mongo-python-driver/blob/3.4.0/test/test_collection.py#L1682-L1683

Generated at Thu Feb 08 04:19:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.