[SERVER-17280] Geo near must pass invalidation messages to index scan used by DensityEstimator Created: 13/Feb/15  Updated: 19/Sep/15  Resolved: 25/Feb/15

Status: Closed
Project: Core Server
Component/s: Concurrency, Geo, Querying
Affects Version/s: 3.1.0
Fix Version/s: 3.0.0-rc11, 3.1.0

Type: Bug Priority: Major - P3
Reporter: Charlie Swanson Assignee: David Storch
Resolution: Done Votes: 0
Labels: 28qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-15539 Invariant failure keyOffset >= 0 duri... Closed
is related to SERVER-16857 Remove PlanStage::invalidate() Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Steps To Reproduce:

This might be hard to reproduce, but enabling fsm_all_sharded.js and running that with --shell-write-mode=compatibility. Note that there are other known issues with fsm_all_sharded, as seen in SERVER-17207, so certain other tests might have to be disabled to get this failure to happen. The easiest way to get this specific test to run is to modify the list of files to run in fsm_all_sharded.js to only include fsm_workloads/yield_geo_near.js. I have not been able to reproduce locally.

Sprint: Quint 3.1.0
Participants:

 Description   

FSM test fails in a sharded environment with this invariant failure. The test interleaves geo_near queries with deletes, inserts, and updates to the (2d) indexed fields. In this environment the collection is sharded on {_id: 'hashed'}.
Here's the backtrace:

m30001| 2015-02-12T19:29:45.476-0500 I -        [conn35] Invariant failure keyOffset < n src/mongo/db/storage/mmap_v1/btree/btree_logic.cpp 2079
 m30001| 2015-02-12T19:29:45.489-0500 I CONTROL  [conn35] 
 m30001|  0x10acf9f59 0x10acae250 0x10ac9b4c3 0x10aab56b7 0x10aaa2171 0x10a84832b 0x10a7bea80 0x10a7be30d 0x10a7b0df5 0x10a7b0fb9 0x10a7c7103 0x10a7ccc6b 0x10a988e30 0x10a988cf9 0x10a70736d 0x10a77e55a 0x10a77f48f 0x10a7803cc 0x10a956c6a 0x10a85ab26 0x10a5a8f81 0x10acbdcdc 0x10ad2d8f1 0x7fff92b36772 0x7fff92b231a1
 m30001| ----- BEGIN BACKTRACE -----
 m30001| {"backtrace":[{"b":"10A59D000","o":"75CF59"},{"b":"10A59D000","o":"711250"},{"b":"10A59D000","o":"6FE4C3"},{"b":"10A59D000","o":"5186B7"},{"b":"10A59D000","o":"505171"},{"b":"10A59D000","o":"2AB32B"},{"b":"10A59D000","o":"221A80"},{"b":"10A59D000","o":"22130D"},{"b":"10A59D000","o":"213DF5"},{"b":"10A59D000","o":"213FB9"},{"b":"10A59D000","o":"22A103"},{"b":"10A59D000","o":"22FC6B"},{"b":"10A59D000","o":"3EBE30"},{"b":"10A59D000","o":"3EBCF9"},{"b":"10A59D000","o":"16A36D"},{"b":"10A59D000","o":"1E155A"},{"b":"10A59D000","o":"1E248F"},{"b":"10A59D000","o":"1E33CC"},{"b":"10A59D000","o":"3B9C6A"},{"b":"10A59D000","o":"2BDB26"},{"b":"10A59D000","o":"BF81"},{"b":"10A59D000","o":"720CDC"},{"b":"10A59D000","o":"7908F1"},{"b":"7FFF92B22000","o":"14772"},{"b":"7FFF92B22000","o":"11A1"}],"processInfo":{ "mongodbVersion" : "3.1.0-pre-", "gitVersion" : "6ca5a81e340f96502bc5f530a8a6fa0d44fea052", "uname" : { "sysname" : "Darwin", "release" : "12.5.0", "version" : "Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/data/mci/shell/src/mongod", "machType" : 2, "b" : "10A59D000", "buildId" : "101EBAE42695358684C0F18E8CA0642B" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF91DB8000", "buildId" : "DF030DDBDF223769A8CD9806DDB84008" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF9785A000", "buildId" : "20E31B9019B93C2AA9EB474E08F9FE05" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF8FAC6000", "buildId" : "65187C6E3FBF3EB8A1AA389445E2984D" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF8ED7E000", "buildId" : "BAAFE0C9BB863CA788C0E3CBA98DA06F" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF979DB000", "buildId" : "08F8731D596139F1AD004590321D24A9" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF98C1D000", "buildId" : "876573D0E9073566A108577EAD1B6182" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF98C8F000", "buildId" : "D26996BFFC5739EB8829F63585561E09" }, { "path" : "/usr/lib/system/libdnsinfo.dylib", "machType" : 6, "b" : "7FFF938D6000", "buildId" : "14202FFBC3CA3FCC94B014611BF8692D" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF95E66000", "buildId" : "F59367C9C110382BA6959035A6DD387E" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF91F92000", "buildId" : "CC9E3394BE16397F926BE579B60EE429" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF970B2000", "buildId" : "2F71CAF86524329EAC56C506658B4C0C" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF90C25000", "buildId" : "BF332AD9E89F387E92A46E1AB74BD4D9" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF8ECE7000", "buildId" : "143B726EDF4737A890AAF059CFD1A2E4" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF8F50D000", "buildId" : "6763BC8E18B83AD98FFAB43713A7264F" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF8EFE2000", "buildId" : "D92DCBC3541C37BDAADEACC75A0C59C8" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF92B22000", "buildId" : "543B05AECFA53EFE8E5877225411BA6B" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF90F75000", "buildId" : "BDCB8566018934C0963435ABD3EFE25B" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF90F8C000", "buildId" : "4FFCA2427F04365F87A6D4EFB89503C1" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF94317000", "buildId" : "4B7993C3F62D3AC1AF92414A0D6EED5E" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF91E7C000", "buildId" : "B434BE5C25AB3EBDBAA75304B34E3441" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF98F26000", "buildId" : "0D99F24E56FE380FB81B4A4C630EE587" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF98D50000", "buildId" : "C49275CC835A3207AFBA8C01374927B6" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF95E6A000", "buildId" : "B739DA63B675387AAD84412A651143C0" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF8EE15000", "buildId" : "92805328CD3634FF9436571AB0485072" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF934A4000", "buildId" : "21703D362DAB3D8B8442EAAB23C060D3" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF981B3000", "buildId" : "70BC645B69523264930CC835010CCEF9" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF9115E000", "buildId" : "CE0C29A3C420339BADAA52F4683233CC" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF91BB6000", "buildId" : "90D31928F48D3E37874F220A51FD9E37" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF8FC25000", "buildId" : "AD5A4CE7CB53313C9FAE673303CC2D35" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF8FBFF000", "buildId" : "D86169F39F31377A9AF3DB17142052E4" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF938D4000", "buildId" : "8548E0DC0D2F30B6B045FE8A038E76D8" } ] }}
 m30001|  mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x10acf9f59]
 m30001|  mongod(_ZN5mongo10logContextEPKc+0x100) [0x10acae250]
 m30001|  mongod(_ZN5mongo15invariantFailedEPKcS1_j+0x183) [0x10ac9b4c3]
 m30001|  mongod(_ZNK5mongo10BtreeLogicINS_13BtreeLayoutV1EE6getKeyEPNS_16OperationContextERKNS_7DiskLocEi+0x1A7) [0x10aab56b7]
 m30001|  mongod(_ZNK5mongo18BtreeInterfaceImplINS_13BtreeLayoutV1EE6Cursor6getKeyEv+0x21) [0x10aaa2171]
 m30001|  mongod(_ZNK5mongo16BtreeIndexCursor6getKeyEv+0x1B) [0x10a84832b]
 m30001|  mongod(_ZN5mongo9IndexScan8checkEndEv+0x160) [0x10a7bea80]
 m30001|  mongod(_ZN5mongo9IndexScan4workEPm+0x5D) [0x10a7be30d]
 m30001|  mongod(_ZN5mongo14GeoNear2DStage16DensityEstimator4workEPNS_16OperationContextEPNS_10WorkingSetEPNS_10CollectionEPd+0x45) [0x10a7b0df5]
 m30001|  mongod(_ZN5mongo14GeoNear2DStage10initializeEPNS_16OperationContextEPNS_10WorkingSetEPNS_10CollectionE+0xE9) [0x10a7b0fb9]
 m30001|  mongod(_ZN5mongo9NearStage4workEPm+0x83) [0x10a7c7103]
 m30001|  mongod(_ZN5mongo15ProjectionStage4workEPm+0x4B) [0x10a7ccc6b]
 m30001|  mongod(_ZN5mongo12PlanExecutor18getNextSnapshottedEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0xA0) [0x10a988e30]
 m30001|  mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x39) [0x10a988cf9]
 m30001|  mongod(_ZN5mongo16Geo2dFindNearCmd3runEPNS_16OperationContextERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEERNS_7BSONObjEiRS9_RNS_14BSONObjBuilderEb+0x117D) [0x10a70736d]
 m30001|  mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKNSt3__112basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEERNS_7BSONObjEiRSA_RNS_14BSONObjBuilderEb+0x3A) [0x10a77e55a]
 m30001|  mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xA1F) [0x10a77f48f]
 m30001|  mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x45C) [0x10a7803cc]
 m30001|  mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5CurOpES3_b+0xE3A) [0x10a956c6a]
 m30001|  mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortEb+0xA36) [0x10a85ab26]
 m30001|  mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xE1) [0x10a5a8f81]
 m30001|  mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x32C) [0x10acbdcdc]
 m30001|  mongod(_ZN5boost12_GLOBAL__N_112thread_proxyEPv+0xB1) [0x10ad2d8f1]
 m30001|  libsystem_c.dylib(_pthread_start+0x147) [0x7fff92b36772]
 m30001|  libsystem_c.dylib(thread_start+0xD) [0x7fff92b231a1]
 m30001| -----  END BACKTRACE  -----
 m30001| 2015-02-12T19:29:45.490-0500 I -        [conn35] 



 Comments   
Comment by Githook User [ 25/Feb/15 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-17280 geo near must propagate saveState()/restoreState() to the DensityEstimator

(cherry picked from commit 77dae5d4457c85e64fa92ebd19daa1ee8027dad6)

Conflicts:
src/mongo/db/exec/geo_near.h
src/mongo/dbtests/query_stage_near.cpp
Branch: v3.0
https://github.com/mongodb/mongo/commit/f4951a12eba1e0cc75adf0422a3654f22eab72a4

Comment by Githook User [ 25/Feb/15 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-17280 geo near must propagate invalidate messages to IndexScan used by DensityEstimator

(cherry picked from commit fe22feda7b5a02918906174d83c999af1a23ba75)

Conflicts:
src/mongo/db/exec/geo_near.h
src/mongo/dbtests/query_stage_near.cpp
Branch: v3.0
https://github.com/mongodb/mongo/commit/73f6151ed1070ad0d45d732c2a7e2e2dcdba2592

Comment by Githook User [ 25/Feb/15 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-17280 geo near must propagate saveState()/restoreState() to the DensityEstimator
Branch: master
https://github.com/mongodb/mongo/commit/77dae5d4457c85e64fa92ebd19daa1ee8027dad6

Comment by David Storch [ 25/Feb/15 ]

Re-opening as saveState() and restoreState() need to be passed through to the density estimator as well.

Comment by Githook User [ 25/Feb/15 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-17280 geo near must propagate invalidate messages to IndexScan used by DensityEstimator
Branch: master
https://github.com/mongodb/mongo/commit/fe22feda7b5a02918906174d83c999af1a23ba75

Comment by Charlie Swanson [ 13/Feb/15 ]

potentially related to SERVER-15539?

Generated at Thu Feb 08 03:43:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.