[SERVER-22537] segfault running aggregation query Created: 09/Feb/16  Updated: 19/Nov/16  Resolved: 29/Feb/16

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: 3.2.1
Fix Version/s: 3.2.4, 3.3.3

Type: Bug Priority: Major - P3
Reporter: Adinoyi Omuya Assignee: Max Hirschhorn
Resolution: Done Votes: 0
Labels: code-and-test
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongodb-win32-x86_64-enterprise-windows-64-3.2016-02-18T16-27-59.mdmp     HTML File mongoplay     File wisdom.playback    
Issue Links:
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Steps To Reproduce:

1. Get the data
curl -s https://s3.amazonaws.com/mciuploads/sqlproxy/data/attendees.bson.gz --output attendees.bson.gz
curl -s https://s3.amazonaws.com/mciuploads/sqlproxy/data/flights201406.bson.gz --output flights201406.bson.gz

2. Restore using mongorestore --gzip --archive=attendees.bson.gz and mongorestore --gzip --archive=flights201406.bson.gz

3. Run the aggregation query below on the attendees collection.

Sprint: Query 10 (02/22/16), Query 11 (03/14/16)
Participants:
Linked BF Score: 0

 Description   

Aggregation pipeline (on tableau database):

[
  {
    "$lookup": {
      "from": "flights201406",
      "localField": "airport_code",
      "foreignField": "dest_airport_code",
      "as": "__joined_flights201406"
    }
  },
  {
    "$unwind": {
      "path": "$__joined_flights201406",
      "preserveNullAndEmptyArrays": false
    }
  },
  {
    "$match": {
      "$and": [
        {
          "__joined_flights201406.flight_date": {
            "$gte": ISODate("2014-06-01T00:00:00Z")
          }
        },
        {
          "__joined_flights201406.flight_date": {
            "$lte": ISODate("2014-06-07T11:00:00Z")
          }
        },
        {
          "__joined_flights201406.origin_city_market_id": 31703
        }
      ]
    }
  }
]

Backtrace (sorry, demangler.com is down):

2016-02-09T13:04:18.765-0500 I QUERY    [conn2] getmore test.attendees query: { aggregate: "attendees", pipeline: [ { $lookup: { as: "__joined_flights201406", from: "flights201406", localField: "airport_code", foreignField: "dest_airport_code" } }, { $unwind: { preserveNullAndEmptyArrays: false, path: "$__joined_flights201406" } }, { $match: { $and: [ { __joined_flights201406.flight_date: { $gte: new Date(1401580800000) } }, { __joined_flights201406.flight_date: { $lte: new Date(1402138800000) } }, { __joined_flights201406.origin_city_market_id: 31703.0 } ] } } ], cursor: {} } cursorid:40128142939 ntoreturn:0 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:5215 reslen:4194353 locks:{ Global: { acquireCount: { r: 8090 } }, MMAPV1Journal: { acquireCount: { r: 4045 } }, Database: { acquireCount: { r: 4045 } }, Collection: { acquireCount: { R: 4045 } } } 4726ms
2016-02-09T13:04:19.279-0500 F -        [conn2] Invalid access at address: 0x8
2016-02-09T13:04:19.290-0500 F -        [conn2] Got signal: 11 (Segmentation fault: 11).
 0x1070303b9 0x10702fed8 0x7fff93bd5eaa 0x0 0x10696ea1f 0x1067d4839 0x1067d53bb 0x106ae7a90 0x106ae6b36 0x106ae99bf 0x1069abce1 0x1069abb7f 0x106bbd591 0x106bbd199 0x106b7e6f2 0x106b7d51c 0x106a5bd59 0x106a58d66 0x106760b5f 0x106ff2b6e 0x106ff2311 0x7fff9cb27c13 0x7fff9cb27b90 0x7fff9cb25375
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"106757000","o":"8D93B9"},{"b":"106757000","o":"8D8ED8"},{"b":"7FFF93BD1000","o":"4EAA"},{"b":"0","o":"0"},{"b":"106757000","o":"217A1F"},{"b":"106757000","o":"7D839"},{"b":"106757000","o":"7E3BB"},{"b":"106757000","o":"390A90"},{"b":"106757000","o":"38FB36"},{"b":"106757000","o":"3929BF"},{"b":"106757000","o":"254CE1"},{"b":"106757000","o":"254B7F"},{"b":"106757000","o":"466591"},{"b":"106757000","o":"466199"},{"b":"106757000","o":"4276F2"},{"b":"106757000","o":"42651C"},{"b":"106757000","o":"304D59"},{"b":"106757000","o":"301D66"},{"b":"106757000","o":"9B5F"},{"b":"106757000","o":"89BB6E"},{"b":"106757000","o":"89B311"},{"b":"7FFF9CB24000","o":"3C13"},{"b":"7FFF9CB24000","o":"3B90"},{"b":"7FFF9CB24000","o":"1375"}],"processInfo":{ "mongodbVersion" : "3.2.1", "gitVersion" : "a14d55980c2cdc565d4704a7e3ad37e4e535c1b2", "compiledModules" : [], "uname" : { "sysname" : "Darwin", "release" : "15.2.0", "version" : "Darwin Kernel Version 15.2.0: Fri Nov 13 19:56:56 PST 2015; root:xnu-3248.20.55~2/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/Users/bedrich/mongo/3.2.0-rc0/mongod", "machType" : 2, "b" : "106757000", "buildId" : "393117B97A9A3E4A9A555E3C8169AB7A" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF8DC80000", "buildId" : "54388DF0381333E4BE8D7743A81ACF4D" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF938BE000", "buildId" : "8FC3D139805534989AC56467CB7F4D14" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF9E150000", "buildId" : "6B245C0AF3EA383BA5425B0D0456A41B" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF9CE6D000", "buildId" : "766BC3F541F33315BABC72718A98EA92" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF9527E000", "buildId" : "D3C4AB4023B43BC68C385B8758D14E80" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF9C871000", "buildId" : "F51332690B22388CA57C079667B6291E" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF8D34C000", "buildId" : "C6BD205F4ECE37EEBCABA76F39CDCFFA" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF94600000", "buildId" : "324C91892AF33356847F6F4CE1C6E901" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF8E5AF000", "buildId" : "5F3777A7F07E3D5FBFA3A920FF4170ED" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF8C122000", "buildId" : "09397E0160663179A50C2CE666FDA929" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF8C3E5000", "buildId" : "EDF719D6D2BB38DD8C944272BEFDA2CD" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF9BC77000", "buildId" : "CB745E1F48853F96B38B2093DF488FD5" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF91433000", "buildId" : "163CF63A74553D1FAE578C4475A9204C" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF95046000", "buildId" : "B8D1A5FCCFD53AAB8A1014DDC129710A" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF8CAD3000", "buildId" : "3C2D3ACD0DD1337A824744A910D67A65" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFFA0904000", "buildId" : "49D423297DE9341392C3A473A7E9CF35" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF9E98F000", "buildId" : "EAB38A6C86713B13B50090EC1B912063" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF98A3D000", "buildId" : "5FD7907036CC3D02BEA7BB5D2AE97D5D" }, { "path" : "/usr/lib/system/libsystem_coreservices.dylib", "machType" : 6, "b" : "7FFF9FDE4000", "buildId" : "1B3F5AFCFFCD3ECB8B9A5538366FB20D" }, { "path" : "/usr/lib/system/libsystem_coretls.dylib", "machType" : 6, "b" : "7FFF9572F000", "buildId" : "30AF71346CA73582B9D3507D6ED19A88" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF9C33A000", "buildId" : "945B5FB1DA913D45A961A8FAD53C1E7E" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF93BA7000", "buildId" : "6513635B4ADE3B45BF63ED7AC565B0C9" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF9FDE7000", "buildId" : "0E688457491536DD87985C2EDEE3F1A3" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF9BE15000", "buildId" : "26655445CA97321EB221801CB378D1AA" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFFA044B000", "buildId" : "9EECAB18F02534C48E327EFFA6720EFC" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF9D456000", "buildId" : "1C0410F3F66E3B0DB8AD0D49AB15A529" }, { "path" : "/usr/lib/system/libsystem_networkextension.dylib", "machType" : 6, "b" : "7FFFA0E33000", "buildId" : "DC8A102ABF0231A4891465C34DF6B592" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFFA1108000", "buildId" : "243FADE1255A3B788033F336CD64B817" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF93BD1000", "buildId" : "D3A27E107F083603ACC87A92B2C04BAB" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF9CB24000", "buildId" : "327CECD0B88131538FCC4FD4818B7F16" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF9A344000", "buildId" : "1C8913361B25365DB43E96D5B3BE66B0" }, { "path" : "/usr/lib/system/libsystem_secinit.dylib", "machType" : 6, "b" : "7FFF98A65000", "buildId" : "FD6ECF2C148932CA981B9045B5EB1FAA" }, { "path" : "/usr/lib/system/libsystem_trace.dylib", "machType" : 6, "b" : "7FFFA0E21000", "buildId" : "F0B7622BFB6B31E1870338F57BE84553" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF94640000", "buildId" : "1D0F8265F0263CBD93D3F8DF14FFCE68" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF9D450000", "buildId" : "124E0F0523503774A32C7F5BF38EDE73" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF9E2F4000", "buildId" : "61AB46109304354C9E9BD57198AE9866" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF91AE8000", "buildId" : "9F45830DF1D53CDF94611A5477ED7D1E" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF90487000", "buildId" : "999E610F41FC32A3ADCA5EC049B65DFB" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFFA08B0000", "buildId" : "DCCC81773D0935BC97842A04FEC4C71B" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF97779000", "buildId" : "4243B6B421E9355B9C5A95A216233B96" } ] }}
 mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x1070303b9]
 mongod(_ZN5mongo12_GLOBAL__N_124abruptQuitWithAddrSignalEiP9__siginfoPv+0xF8) [0x10702fed8]
 libsystem_platform.dylib(_sigtramp+0x1A) [0x7fff93bd5eaa]
 ??? [0x0]
 mongod(_ZThn8_N5mongo14DBDirectClient4callERNS_7MessageES2_bPNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEE+0xF) [0x10696ea1f]
 mongod(_ZN5mongo14DBClientCursor11requestMoreEv+0x289) [0x1067d4839]
 mongod(_ZN5mongo14DBClientCursor4moreEv+0x4B) [0x1067d53bb]
 mongod(_ZN5mongo20DocumentSourceLookUp12unwindResultEv+0x5A0) [0x106ae7a90]
 mongod(_ZN5mongo20DocumentSourceLookUp7getNextEv+0x76) [0x106ae6b36]
 mongod(_ZN5mongo19DocumentSourceMatch7getNextEv+0x6F) [0x106ae99bf]
 mongod(_ZN5mongo18PipelineProxyStage11getNextBsonEv+0x61) [0x1069abce1]
 mongod(_ZN5mongo18PipelineProxyStage4workEPm+0x10F) [0x1069abb7f]
 mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x361) [0x106bbd591]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x39) [0x106bbd199]
 mongod(_ZN5mongo12_GLOBAL__N_113generateBatchEiPNS_12ClientCursorEPNS_11_BufBuilderINS_16TrivialAllocatorEEEPiPNS_9TimestampEPNS_12PlanExecutor9ExecStateE+0x72) [0x106b7e6f2]
 mongod(_ZN5mongo7getMoreEPNS_16OperationContextEPKcixPbS4_+0x73C) [0x106b7d51c]
 mongod(_ZN5mongo15receivedGetMoreEPNS_16OperationContextERNS_10DbResponseERNS_7MessageERNS_5CurOpE+0x2A9) [0x106a5bd59]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x1496) [0x106a58d66]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0xFF) [0x106760b5f]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x27E) [0x106ff2b6e]
 mongod(_ZNSt3__114__thread_proxyINS_5tupleIJNS_6__bindIPFPvS3_EJPN5mongo12_GLOBAL__N_124MessagingPortWithHandlerEEEEEEEEES3_S3_+0x61) [0x106ff2311]
 libsystem_pthread.dylib(_pthread_body+0x83) [0x7fff9cb27c13]
 libsystem_pthread.dylib(_pthread_body+0x0) [0x7fff9cb27b90]
 libsystem_pthread.dylib(thread_start+0xD) [0x7fff9cb25375]
-----  END BACKTRACE  -----
Segmentation fault: 11

Ran this on OSX 10.11.2 (15C50) using the following mongod:

db version v3.2.1
git version: a14d55980c2cdc565d4704a7e3ad37e4e535c1b2
allocator: system
modules: none
build environment:
    distarch: x86_64
    target_arch: x86_64

I'm able to reliably reproduce both on WiredTiger and mmapv1.



 Comments   
Comment by Githook User [ 29/Feb/16 ]

Author:

{u'username': u'visemet', u'name': u'Max Hirschhorn', u'email': u'max.hirschhorn@mongodb.com'}

Message: SERVER-22537 Fix DBDirectClient's OperationContext not getting updated.

This fixes an issue where the DBDirectClient used by the $lookup stage
would have the previous OperationContext and trigger an invalid memory
access when sending a getMore request as it unwinds the results.

Calling PipelineProxyStage::doDetachFromOperationContext() now causes
the DBDirectClient's OperationContext to be set to nullptr, and
PipelineProxyStage::doReattachToOperationContext() causes it to be set
to the current OperationContext.

(cherry picked from commit e1928f36d21a4193802d7fbdcb8fcd6df58f7aa7)
Branch: v3.2
https://github.com/mongodb/mongo/commit/139286b38e3d0e1efc1efbe52417310277b3aed8

Comment by Githook User [ 29/Feb/16 ]

Author:

{u'username': u'visemet', u'name': u'Max Hirschhorn', u'email': u'max.hirschhorn@mongodb.com'}

Message: SERVER-22537 Fix DBDirectClient's OperationContext not getting updated.

This fixes an issue where the DBDirectClient used by the $lookup stage
would have the previous OperationContext and trigger an invalid memory
access when sending a getMore request as it unwinds the results.

Calling PipelineProxyStage::doDetachFromOperationContext() now causes
the DBDirectClient's OperationContext to be set to nullptr, and
PipelineProxyStage::doReattachToOperationContext() causes it to be set
to the current OperationContext.
Branch: master
https://github.com/mongodb/mongo/commit/e1928f36d21a4193802d7fbdcb8fcd6df58f7aa7

Comment by Max Hirschhorn [ 29/Feb/16 ]

Requesting backport to the 3.2 branch only because no stage other than $lookup would trigger a getMore on its DBDirectClient, so this bug won't ever manifest on the 3.0 branch or earlier. In particular, DocumentSourceGeoNear runs the "geoNear" command using its DBDirectClient, which always returns a single batch of results.

Comment by Max Hirschhorn [ 23/Feb/16 ]

I ran the aggregation query on the test.attendees collection as described by the steps in the ticket and was able to get more information from ASan. charlie.swanson and I looked through the ASan output and were able to piece together what had happened. The issue is that PipelineProxyStage::doReattachToOperationContext() doesn't necessarily change the operation context of the DBDirectClient underlying the $lookup stage. Currently this is only done by calling MongodImplementation::directClient(). The way that this can manifest is if you need to do a getMore on the DBDirectClient for the same input document to $lookup (i.e. having many documents in the from collection that match).

The proposal is to add a Pipeline::setOpCtx() that will

  1. Update the opCtx member of ExpressionContext. (This is equivalent to the current behavior.)
  2. Call a new DocumentSourceNeedsMongod::setOpCtx() member to update the operation context of the DBDirectClient of any source that has a MongodImplementation injected.

=================================================================
==26400==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000018bb0 at pc 0x000003e82a02 bp 0x7fdd05974bb0 sp 0x7fdd05974ba8
READ of size 8 at 0x60b000018bb0 thread T17
    #0 0x3e82a01 in mongo::OperationContext::getClient() const /home/maxh/debugging/mongo-v3.2/src/mongo/db/operation_context.cpp:46:12
    #1 0x2fe8b61 in mongo::(anonymous namespace)::DirectClientScope::DirectClientScope(mongo::OperationContext*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/dbdirectclient.cpp:59:28
    #2 0x2fe82ea in mongo::DBDirectClient::call(mongo::Message&, mongo::Message&, bool, std::string*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/dbdirectclient.cpp:129:23
    #3 0x2fe92ac in non-virtual thunk to mongo::DBDirectClient::call(mongo::Message&, mongo::Message&, bool, std::string*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/dbdirectclient.cpp:128:22
    #4 0x1f2c5df in mongo::DBClientCursor::requestMore() /home/maxh/debugging/mongo-v3.2/src/mongo/client/dbclientcursor.cpp:210:9
    #5 0x1f2ffd5 in mongo::DBClientCursor::more() /home/maxh/debugging/mongo-v3.2/src/mongo/client/dbclientcursor.cpp:341:5
    #6 0x41b352c in mongo::DocumentSourceLookUp::unwindResult() /home/maxh/debugging/mongo-v3.2/src/mongo/db/pipeline/document_source_lookup.cpp:159:28
    #7 0x41b08b1 in mongo::DocumentSourceLookUp::getNext() /home/maxh/debugging/mongo-v3.2/src/mongo/db/pipeline/document_source_lookup.cpp:69:16
    #8 0x41caae8 in mongo::DocumentSourceMatch::getNext() /home/maxh/debugging/mongo-v3.2/src/mongo/db/pipeline/document_source_match.cpp:67:45
    #9 0x336ddbe in mongo::PipelineProxyStage::getNextBson() /home/maxh/debugging/mongo-v3.2/src/mongo/db/exec/pipeline_proxy.cpp:127:42
    #10 0x336d291 in mongo::PipelineProxyStage::work(unsigned long*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/exec/pipeline_proxy.cpp:72:41
    #11 0x4a8916f in mongo::PlanExecutor::getNextImpl(mongo::Snapshotted<mongo::BSONObj>*, mongo::RecordId*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/query/plan_executor.cpp:393:38
    #12 0x4a87247 in mongo::PlanExecutor::getNext(mongo::BSONObj*, mongo::RecordId*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/query/plan_executor.cpp:322:23
    #13 0x2ac8424 in mongo::GetMoreCmd::generateBatch(mongo::ClientCursor*, mongo::GetMoreRequest const&, mongo::CursorResponseBuilder*, mongo::PlanExecutor::ExecState*, long long*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/commands/getmore_cmd.cpp:421:56
    #14 0x2abc581 in mongo::GetMoreCmd::run(mongo::OperationContext*, std::string const&, mongo::BSONObj&, int, std::string&, mongo::BSONObjBuilder&) /home/maxh/debugging/mongo-v3.2/src/mongo/db/commands/getmore_cmd.cpp:314:30
    #15 0x2f9c6f4 in mongo::Command::run(mongo::OperationContext*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/dbcommands.cpp:1432:19
    #16 0x2f97b1a in mongo::Command::execCommand(mongo::OperationContext*, mongo::Command*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/dbcommands.cpp:1317:18
    #17 0x292d6c7 in mongo::runCommands(mongo::OperationContext*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/commands.cpp:498:9
    #18 0x3be08c6 in mongo::(anonymous namespace)::receivedRpc(mongo::OperationContext*, mongo::Client&, mongo::DbResponse&, mongo::Message&) /home/maxh/debugging/mongo-v3.2/src/mongo/db/instance.cpp:304:9
    #19 0x3bd4e7c in mongo::assembleResponse(mongo::OperationContext*, mongo::Message&, mongo::DbResponse&, mongo::HostAndPort const&) /home/maxh/debugging/mongo-v3.2/src/mongo/db/instance.cpp:533:9
    #20 0x1a1b0fc in mongo::MyMessageHandler::process(mongo::Message&, mongo::AbstractMessagingPort*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/db.cpp:171:17
    #21 0x7984e56 in mongo::PortMessageServer::handleIncomingMsg(void*) /home/maxh/debugging/mongo-v3.2/src/mongo/util/net/message_server_port.cpp:229:17
    #22 0x7fdd150a5181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312
    #23 0x7fdd14bbc47c in clone /build/buildd/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:111
 
0x60b000018bb0 is located 32 bytes inside of 104-byte region [0x60b000018b90,0x60b000018bf8)
freed by thread T17 here:
    #0 0x19845c2 in operator delete(void*) (/home/maxh/debugging/mongo-v3.2/mongod+0x19845c2)
    #1 0x3e84a81 in mongo::OperationContextImpl::~OperationContextImpl() /home/maxh/debugging/mongo-v3.2/src/mongo/db/operation_context_impl.cpp:90:47
    #2 0x5a53eff in mongo::ServiceContext::OperationContextDeleter::operator()(mongo::OperationContext*) const /home/maxh/debugging/mongo-v3.2/src/mongo/db/service_context.cpp:223:5
    #3 0x19cd4ee in std::unique_ptr<mongo::OperationContext, mongo::ServiceContext::OperationContextDeleter>::~unique_ptr() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/unique_ptr.h:184:4
    #4 0x1a1b11b in mongo::MyMessageHandler::process(mongo::Message&, mongo::AbstractMessagingPort*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/db.cpp:175:13
    #5 0x7984e56 in mongo::PortMessageServer::handleIncomingMsg(void*) /home/maxh/debugging/mongo-v3.2/src/mongo/util/net/message_server_port.cpp:229:17
    #6 0x7fdd150a5181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312
 
previously allocated by thread T17 here:
    #0 0x1984002 in operator new(unsigned long) (/home/maxh/debugging/mongo-v3.2/mongod+0x1984002)
    #1 0x5a90819 in boost::detail::up_if_not_array<mongo::OperationContextImpl>::type boost::make_unique<mongo::OperationContextImpl>() /home/maxh/debugging/mongo-v3.2/src/third_party/boost-1.56.0/boost/smart_ptr/make_unique_object.hpp:21:35
    #2 0x5a88df5 in mongo::ServiceContextMongoD::_newOpCtx(mongo::Client*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/service_context_d.cpp:293:12
    #3 0x5a52f65 in mongo::ServiceContext::makeOperationContext(mongo::Client*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/service_context.cpp:179:18
    #4 0x1a1afc2 in mongo::MyMessageHandler::process(mongo::Message&, mongo::AbstractMessagingPort*) /home/maxh/debugging/mongo-v3.2/src/mongo/db/db.cpp:205:5
    #5 0x7984e56 in mongo::PortMessageServer::handleIncomingMsg(void*) /home/maxh/debugging/mongo-v3.2/src/mongo/util/net/message_server_port.cpp:229:17
    #6 0x7fdd150a5181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312
 
Thread T17 created by T0 here:
    #0 0x194ce80 in __interceptor_pthread_create (/home/maxh/debugging/mongo-v3.2/mongod+0x194ce80)
    #1 0x7981eb0 in mongo::PortMessageServer::accepted(std::shared_ptr<mongo::Socket>, long long) /home/maxh/debugging/mongo-v3.2/src/mongo/util/net/message_server_port.cpp:148:26
    #2 0x798327e in non-virtual thunk to mongo::PortMessageServer::accepted(std::shared_ptr<mongo::Socket>, long long) /home/maxh/debugging/mongo-v3.2/src/mongo/util/net/message_server_port.cpp:107:18
    #3 0x7934d4d in mongo::Listener::initAndListen() /home/maxh/debugging/mongo-v3.2/src/mongo/util/net/listen.cpp:352:13
    #4 0x7980d81 in mongo::PortMessageServer::run() /home/maxh/debugging/mongo-v3.2/src/mongo/util/net/message_server_port.cpp:176:9
    #5 0x198c374 in mongo::_initAndListen(int) /home/maxh/debugging/mongo-v3.2/src/mongo/db/db.cpp:652:5
    #6 0x198512f in mongo::initAndListen(int) /home/maxh/debugging/mongo-v3.2/src/mongo/db/db.cpp:657:9
    #7 0x198dd89 in mongoDbMain(int, char**, char**) /home/maxh/debugging/mongo-v3.2/src/mongo/db/db.cpp:891:25
    #8 0x198ce7d in main /home/maxh/debugging/mongo-v3.2/src/mongo/db/db.cpp:702:20
    #9 0x7fdd14ae3ec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287
 
SUMMARY: AddressSanitizer: heap-use-after-free /home/maxh/debugging/mongo-v3.2/src/mongo/db/operation_context.cpp:46 mongo::OperationContext::getClient() const
Shadow bytes around the buggy address:
  0x0c167fffb120: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fffb130: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fffb140: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fffb150: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fffb160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c167fffb170: fa fa fd fd fd fd[fd]fd fd fd fd fd fd fd fd fa
  0x0c167fffb180: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c167fffb190: fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fffb1a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fffb1b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fffb1c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==26400==ABORTING

Generated at Thu Feb 08 04:00:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.