[SERVER-10285] Seg Fault when parsing Point using 2dsphere Indexes Created: 22/Jul/13 Updated: 11/Jul/16 Resolved: 04/Oct/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Geo |
| Affects Version/s: | 2.4.5 |
| Fix Version/s: | 2.4.7, 2.5.3 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Adrian Grealish | Assignee: | hari.khalsa@10gen.com |
| Resolution: | Done | Votes: | 1 |
| Labels: | crash | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
AWS EC2 machine running latest stable build of mongo 2.4.5 |
||
| Attachments: |
|
| Operating System: | Linux |
| Steps To Reproduce: | Hard to know, we ran the latest in production for 3 whole days before we saw any problem. We are storing Points (lat, lon) and using 2dspherical index and query ing Mongo using geoNear command. We are using the latest mongo java driver 2.9.3 |
| Participants: |
| Description |
|
Issue Status as of October 22nd, 2013 ISSUE SUMMARY USER IMPACT SOLUTION WORKAROUNDS PATCHES Our production system crashed hard yesterday afternoon with what looks like a Segmentation Fault parsing a Point. It took out all 6 Nodes including the master. We unfortunately don't have the query (lat,lon) that cause the issue. Here is the stack trace on master. There seem to be a lot of GEO code changes in this release https://jira.mongodb.org/browse/SERVER-8349 and https://github.com/mongodb/mongo/commit/ba239918c950c254056bf589a943a5e88fd4144c We are rolling back to previous version before all these changes were made.
|
| Comments |
| Comment by auto [ 02/Oct/13 ] |
|
Author: {u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}Message: Backport original commit: ce6607fd28df1cc7c57a6ea03dcb0676e7a009ce |
| Comment by Daniel Pasette (Inactive) [ 16/Sep/13 ] |
|
The fix has not been backported to the 2.4 branch yet. 2.5.3 will be a development release only, not meant for production use. It would be great if you can test 2.5.3 in a development environment though when it comes out. |
| Comment by Eyllo [ 11/Sep/13 ] |
|
Hi there, so does this mean this is solved? and that we should upgrade to 2.5.3? |
| Comment by auto [ 10/Sep/13 ] |
|
Author: {u'username': u'hkhalsa', u'name': u'Hari Khalsa', u'email': u'hkhalsa@10gen.com'}Message: |
| Comment by Eyllo [ 22/Aug/13 ] |
|
Hi Hari, I dunno if this makes any difference or not but we are using a 2d index together with a 2dsphere index. |
| Comment by Eyllo [ 22/Aug/13 ] |
|
Hi Hari, |
| Comment by hari.khalsa@10gen.com [ 21/Aug/13 ] |
|
eyllo Hello. Is there any chance you can log the request and provide me with what is exactly crashing it? Also, what platform are you running on? |
| Comment by Eyllo [ 21/Aug/13 ] |
|
Hi, We are also experiencing a lot of server crashes with something related to this. The crashing problem is really critical as our application also stops just because of some data error? Our server log is as follows: Tue Aug 20 21:34:28.592 Backtrace: |
| Comment by hari.khalsa@10gen.com [ 24/Jul/13 ] |
|
Yes, if you can get the query of death it would make fixing the problem much easier. Looking forward to fixing this. |
| Comment by Adrian Grealish [ 24/Jul/13 ] |
|
we are going to continue to run the load test and have turned up logging on the remaining nodes |
| Comment by Adrian Grealish [ 24/Jul/13 ] |
|
We just saw this same error on a replica node that crashed while we were running load tests against 2.4.3 Jul 23 22:56:57 ip-10-60-69-130 mongod.27017[18174]: Tue Jul 23 22:56:57.680 [conn17514] command location.$cmd command: { geoNear: "presence", near: [ -0.7597462961209844, 52.02757513346383 ], spherical: true, query: { age: { $gte: 88, $lte: 93 } }, limit: 400, maxDistance: 0.1 } ntoreturn:1 keyUpdates:0 locks(micros) r:232891 reslen:29813 232ms |
| Comment by Adrian Grealish [ 23/Jul/13 ] |
|
We are running an off the shelf AMI on a m1.xlarge instance in EC2. We were running 2.4.5 until the crash, then we downgraded. |
| Comment by hari.khalsa@10gen.com [ 23/Jul/13 ] |
|
Hello! The changes you linked were not put into the 2.4 branch. Looking at the logs, I'm not clear on whether or not you're running 2.4.4 or 2.4.5. There are lines in the logs that suggest it's 2.4.4: Jul 22 02:09:34 ip-10-40-123-164 mongod.27017[21395]: Mon Jul 22 02:09:34.330 [initandlisten] db version v2.4.4 Can you tell me about what kind of platform/computer you're running this on? Looks like an off-the-shelf Amazon EC2 instance? |
| Comment by Adrian Grealish [ 22/Jul/13 ] |
|
Not sure if related but we cannot store Point object in the document with this version and latest driver. The point class generated this JSON from here http://geojson.org/geojson-spec.html#id29 { "type": "Point", "coordinates": [100.0, 0.0] }when using Gson. Not sure why BSON can't handle it. Caused by: java.lang.IllegalArgumentException: can't serialize class com.grindr.presence.model.Point |
| Comment by Adrian Grealish [ 22/Jul/13 ] |
|
crashed at Jul 22 00:08:03 |
| Comment by Daniel Pasette (Inactive) [ 22/Jul/13 ] |
|
Can you attach more of the log (or whole log compressed) to this ticket? |