[SERVER-27065] Segfault in building an aggregation Created: 16/Nov/16  Updated: 05/Apr/17  Resolved: 03/Dec/16

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: None
Fix Version/s: 3.5.1

Type: Bug Priority: Major - P3
Reporter: Daniel Gottlieb (Inactive) Assignee: David Storch
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File repro.patch     File sample_fix.patch    
Issue Links:
Duplicate
is duplicated by SERVER-8369 kill cursor of an internal only Clien... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Query 2016-11-21, Query 2016-12-12
Participants:

 Description   

Low severity (extreme circumstances) crash.

[MongoDFixture:job0] 2016-11-14T13:27:33.230-0500 F -        [conn12] Got signal: 11 (Segmentation fault).
[MongoDFixture:job0] 
[MongoDFixture:job0]  0x7f03dbca2092 0x7f03dbca13f9 0x7f03dbca19d7 0x7f03d974e330 0x7f03db21ecb9 0x7f03db21f7ae 0x7f03db2202ee 0x7f03db1c06cb 0x7f03db196e3b 0x7f03db197ed0 0x7f03db7bad13 0x7f03db3bb4ef 0x7f03dafa3c94 0x7f03dafa45fd 0x7f03dbc1b29e 0x7f03d9746184 0x7f03d947337d
[MongoDFixture:job0] ----- BEGIN BACKTRACE -----
[MongoDFixture:job0] {"backtrace":[{"b":"7F03DA7B7000","o":"14EB092","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F03DA7B7000","o":"14EA3F9"},{"b":"7F03DA7B7000","o":"14EA9D7"},{"b":"7F03D973E000","o":"10330"},{"b":"7F03DA7B7000","o":"A67CB9"},{"b":"7F03DA7B7000","o":"A687AE"},{"b":"7F03DA7B7000","o":"A692EE"},{"b":"7F03DA7B7000","o":"A096CB","s":"_ZN5mongo7FindCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderE"},{"b":"7F03DA7B7000","o":"9DFE3B","s":"_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE"},{"b":"7F03DA7B7000","o":"9E0ED0","s":"_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE"},{"b":"7F03DA7B7000","o":"1003D13","s":"_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE"},{"b":"7F03DA7B7000","o":"C044EF","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"7F03DA7B7000","o":"7ECC94","s":"_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE"},{"b":"7F03DA7B7000","o":"7ED5FD"},{"b":"7F03DA7B7000","o":"146429E"},{"b":"7F03D973E000","o":"8184"},{"b":"7F03D9379000","o":"FA37D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.0-rc2-90-ga79c8b6", "gitVersion" : "a79c8b6455ba50d40f42bead5ce24a6093734dc3", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "b" : "7F03DA7B7000", "elfType" : 3, "buildId" : "0606DEFCE5335B31504AAF421CAB12126CE57F65" }, { "b" : "7FFF10321000", "elfType" : 3, "buildId" : "0074678E5FFFF79F46C476077E67057161772F37" }, { "b" : "7F03DA38A000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F03DA186000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F03D9E79000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "33DE07A4AF856622FA878714CCA68CE87C287BEB" }, { "b" : "7F03D9B73000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F03D995C000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "44F5FF332E0FC3442C094539078D17DCF076B6FF" }, { "b" : "7F03D973E000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F03D9379000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F03DA592000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" } ] }}
[MongoDFixture:job0]  mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x7f03dbca2092]
[MongoDFixture:job0]  mongod(+0x14EA3F9) [0x7f03dbca13f9]
[MongoDFixture:job0]  mongod(+0x14EA9D7) [0x7f03dbca19d7]
[MongoDFixture:job0]  libpthread.so.0(+0x10330) [0x7f03d974e330]
[MongoDFixture:job0]  mongod(+0xA67CB9) [0x7f03db21ecb9]
[MongoDFixture:job0]  mongod(+0xA687AE) [0x7f03db21f7ae]
[MongoDFixture:job0]  mongod(+0xA692EE) [0x7f03db2202ee]
[MongoDFixture:job0]  mongod(_ZN5mongo7FindCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderE+0x104B) [0x7f03db1c06cb]
[MongoDFixture:job0]  mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x68B) [0x7f03db196e3b]
[MongoDFixture:job0]  mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x740) [0x7f03db197ed0]
[MongoDFixture:job0]  mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x233) [0x7f03db7bad13]
[MongoDFixture:job0]  mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xCBF) [0x7f03db3bb4ef]
[MongoDFixture:job0]  mongod(_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE+0x194) [0x7f03dafa3c94]
[MongoDFixture:job0]  mongod(+0x7ED5FD) [0x7f03dafa45fd]
[MongoDFixture:job0]  mongod(+0x146429E) [0x7f03dbc1b29e]
[MongoDFixture:job0]  libpthread.so.0(+0x8184) [0x7f03d9746184]
[MongoDFixture:job0]  libc.so.6(clone+0x6D) [0x7f03d947337d]
[MongoDFixture:job0] -----  END BACKTRACE  -----
[MongoDFixture:job0] 2016-11-14T13:27:33.230-0500 F -        [conn7] Invalid access at address: 0x60



 Comments   
Comment by Githook User [ 03/Dec/16 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-27065 cleanup ClientCursor, ClientCursorPin, and CursorManager

  • Makes cursors come into existence pinned. This fixes a
    race condition in which a cursor could time out in between
    being constructed/retrieved and being pinned.
  • Reduces the public interface of ClientCursor. In
    particular, makes ClientCursor's constructor and
    destructor private.
  • Cleans up header file comments in order to more clearly
    indicate expected usage.
    Branch: master
    https://github.com/mongodb/mongo/commit/1e8f34fc476705888f7dec8d06c780de4e556988
Comment by David Storch [ 16/Nov/16 ]

daniel.gottlieb's diagnosis of the problem is this:

The problem is that the pipeline creates a cursor and pins it in separate steps. An aggressive cursor timeout (or unfathomably unlucky scheduling) can cause the cursor to be deregistered by the client cursor monitor before the ClientCursorPin (which does a lookup on the cursorId) sets state that would prevent that.

Specifically, this gives rise to a situation in which ClientCursorPin::_cursor is null. The downstream code does not expect a ClientCursorPin to ever exist without a pinned ClientCursor. This results in a null pointer dereference here:

https://github.com/mongodb/mongo/blob/r3.4.0-rc3/src/mongo/db/commands/pipeline_command.cpp#L563

Generated at Thu Feb 08 04:14:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.