[SERVER-14723] Crash during query planning for geoNear with multiple 2dsphere indices Created: 29/Jul/14  Updated: 09/Oct/15  Resolved: 03/Mar/15

Status: Closed
Project: Core Server
Component/s: Geo, Querying
Affects Version/s: 2.6.3
Fix Version/s: 2.6.9, 3.0.1, 3.1.0

Type: Bug Priority: Major - P3
Reporter: Mark Percy Assignee: David Storch
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File Mongo 2.6.3 Crash.rtf    
Issue Links:
Related
related to SERVER-13687 Results of $near query on compound mu... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Participants:

 Description   

Hi,

We installed Mongo Server 2.6.3 in our staging environment as a three member replica set in Amazon AWS. Around a day after the initial install, one of the Mongo members crashed with attached crash log. We then rolled back to 2.4.9 which we've been using in production since March 2014 without issue.

Regards,

Mark Percy



 Comments   
Comment by Githook User [ 05/Mar/15 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-14723 fix crash in plan enumerator affecting some geo near queries

(cherry picked from commit 965cbef869f8b864caaf3bf7705f8bd82c2b81ba)
Branch: v2.6
https://github.com/mongodb/mongo/commit/5f889dfec4c625af3413348cdc9b409998d33238

Comment by Githook User [ 04/Mar/15 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-14723 fix crash in plan enumerator affecting some geo near queries

(cherry picked from commit 965cbef869f8b864caaf3bf7705f8bd82c2b81ba)
Branch: v3.0
https://github.com/mongodb/mongo/commit/4c5650f4ce2943693fcc7a939ba69679675116c4

Comment by Githook User [ 03/Mar/15 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-14723 fix crash in plan enumerator affecting some geo near queries
Branch: master
https://github.com/mongodb/mongo/commit/965cbef869f8b864caaf3bf7705f8bd82c2b81ba

Comment by J Rassi [ 15/Aug/14 ]

Hi Mark,

We still need additional information from you to diagnose this problem. Would you be able to provide any of the information requested in my comment above?

Thanks.
~ Jason Rassi

Comment by J Rassi [ 30/Jul/14 ]

Hi Mark,

Sorry to hear that you encountered this issue. I can infer from the stack trace that the crash occurred in the query planner, during planning of a query that included a $geoNear predicate (it could also be a $text predicate, but I would guess your application doesn't use $text if you're running the same application code on top of MongoDB 2.4). So, I would guess that there's a bug that generates a crash in the server when a certain kind of $geoNear predicate is run when certain indexes are present. I'd like to ask for some additional information to further diagnose this issue:

  • Could you upload the full mongod log from this crash as an attachment to this ticket? I'm specifically interested in seeing context in the log for thread "conn6331", to see what it was up to just before the crash happened.
  • Would you be willing to leave mongod running in staging for a day's time with the "-vv" option (verbose log level 2), and upload the log as an attachment to the ticket? This will output a summary of every query to the log before running it, which would help narrow down which query is causing the crash. And, would you be willing to re-upgrade to 2.6.3 in staging for the day, in order to reproduce the issue when the "-vv" option is enabled? A reproduction with "-vv" would significantly speed up the process of figuring out what the failing query is.
  • Where did you obtain the binaries you're running in staging? My initial guess was that you're running the stock "Linux 64-bit" build, but addr2line printed garbage when I tried to run it against your stack trace in combination with our stock Linux 64-bit 2.6.3 server binary. Could you paste the output of running "db.serverBuildInfo()" in the mongo shell against the current 2.4.9 staging mongod on this machine, and also paste the output of running "/<path>/mongod --dbpath /doesnotexist", replacing <path> with the path to your 2.6.3 binaries on this machine?
  • Could you paste the output of running the following against your staging cluster? It prints out the definitions of all indexes created on this cluster. The set of indexes present in the collection associated with the failing query (at time of crash) will be needed to reproduce the issue.

    db.adminCommand("listDatabases").databases.forEach(function(d){ printjson(db.getSiblingDB(d.name).system.indexes.find().toArray()); })

Thanks.
~ Jason Rassi

Comment by David Storch [ 29/Jul/14 ]

Stack trace from the log:

2014-07-21T00:05:47.698+0000 [conn6331] SEVERE: Invalid access at address: 0x1008
2014-07-21T00:05:47.708+0000 [conn6331] SEVERE: Got signal: 11 (Segmentation fault).
Backtrace:0x11da801 0x11d9bde 0x11d9ccf 0x7f2e9dc015b0 0xd6c268 0xd676b6 0xd6a912 0xd6b412 0xd8c04d 0xd21ad6 0xd23b72 0xd25b46 0xd53941 0xb9c972 0xb9ef52 0x76f2bf 0x1190afb 0x7f2e9dbf9f18 0x7f2e9cf0fe9d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11da801]
 /usr/bin/mongod() [0x11d9bde]
 /usr/bin/mongod() [0x11d9ccf]
 /lib64/libpthread.so.0(+0xf5b0) [0x7f2e9dc015b0]
 /usr/bin/mongod(_ZNSt6vectorIPN5mongo15MatchExpressionESaIS2_EEaSERKS4_+0x28) [0xd6c268]
 /usr/bin/mongod(_ZN5mongo14PlanEnumerator23enumerateMandatoryIndexERKNSt3tr113unordered_mapImSt6vectorIPNS_15MatchExpressionESaIS5_EENS1_4hashImEESt8equal_toImESaISt4pairIKmS7_EELb0EEESI_S5_RKSt3setImSt4lessImESaImEEPNS0_13AndAssignmentE+0x466) [0xd676b6]
 /usr/bin/mongod(_ZN5mongo14PlanEnumerator8prepMemoEPNS_15MatchExpressionENS0_15PrepMemoContextE+0x1512) [0xd6a912]
 /usr/bin/mongod(_ZN5mongo14PlanEnumerator4initEv+0x22) [0xd6b412]
 /usr/bin/mongod(_ZN5mongo12QueryPlanner4planERKNS_14CanonicalQueryERKNS_18QueryPlannerParamsEPSt6vectorIPNS_13QuerySolutionESaIS9_EE+0x224d) [0xd8c04d]
 /usr/bin/mongod(_ZN5mongo19getRunnerAlwaysPlanEPNS_10CollectionEPNS_14CanonicalQueryERKNS_18QueryPlannerParamsEPPNS_6RunnerE+0x86) [0xd21ad6]
 /usr/bin/mongod(_ZN5mongo9getRunnerEPNS_10CollectionEPNS_14CanonicalQueryEPPNS_6RunnerEm+0x962) [0xd23b72]
 /usr/bin/mongod(_ZN5mongo9getRunnerEPNS_14CanonicalQueryEPPNS_6RunnerEm+0x96) [0xd25b46]
 /usr/bin/mongod(_ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x1251) [0xd53941]
 /usr/bin/mongod() [0xb9c972]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x442) [0xb9ef52]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9f) [0x76f2bf]
 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x4fb) [0x1190afb]
 /lib64/libpthread.so.0(+0x7f18) [0x7f2e9dbf9f18]
 /lib64/libc.so.6(clone+0x6d) [0x7f2e9cf0fe9d]

Generated at Thu Feb 08 03:35:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.