[SERVER-9872] Server deadlock caused by infinite $near & $geoWithin queries on 2dsphere index Created: 06/Jun/13  Updated: 10/Dec/14  Resolved: 08/Jun/13

Status: Closed
Project: Core Server
Component/s: Geo, Querying
Affects Version/s: 2.4.2, 2.4.3
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: James Jory Assignee: hari.khalsa@10gen.com
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 5.6 x86_64, 2.6.18-238.12.1.el5


Issue Links:
Duplicate
duplicates SERVER-9647 Query using compound 2dsphere index w... Closed
Operating System: Linux
Steps To Reproduce:

Unfortunately we have been unable to find specific steps to repeat yet. We have tried different servers and different versions of MongoDB.

Participants:

 Description   

We are staging our application for production deployment but are experiencing periodic server deadlocks during testing that we believe is related to our use of the $near or $geoWithin geo queries. By deadlock I mean that all databases on the server cannot be queried or updated. The CPU also spikes to at/near 100%. We cannot list collections for any of the DBs in the instance. The db.currenOp() function returns all statements in process where all are

{"waitingForLock" : true}

except for one geo query. We've tried both the $near and $geoWithin operators with the same result.

We have experienced this problem on both 2.4.2 and 2.4.3 and on separate physical servers.

Here is the query from db.currentOp().

{
"inprog" : [
{
"opid" : 130836,
"active" : true,
"secs_running" : 3925,
"op" : "query",
"ns" : "vintank.geoFenceActivity",
"query" : {
"$query" : {
"loc.geoPt" : {
"$geoWithin" :

{ "$centerSphere" : [ [ -122.28658676147461, 38.301287089941106 ], 0.0007848061528802386 ] }

},
"pubDt" :

{ "$gte" : ISODate("2013-05-30T17:38:25.423Z") }

},
"$orderby" :

{ "pubDt" : -1 }

},
"client" : "192.168.1.5:44865",
"desc" : "conn3956",
"threadId" : "0x47e27940",
"connectionId" : 3956,
"locks" :

{ "^" : "r", "^vintank" : "R" }

,
"waitingForLock" : false,
"numYields" : 0,
"lockStats" : {
"timeLockedMicros" : {

},
"timeAcquiringMicros" :

{ "r" : NumberLong(2), "w" : NumberLong(0) }

}
}
]
}

Our queries are generated using the Java driver. Below is the relevant Java source for submitting the query (where the query and sort expressions can be determined from the currentOp output above).

ds.getCollection(GeoFenceActivity.class).find(query).limit(Math.max(limit, 1000)).sort(sort).limit(limit);

Since our app is still in testing, the actual lat/lng used in the query is always the same. There are currently only 2,282 documents in this collection.

Here are the indexes on the vintank.geoFenceActivity collection:

{ "v" : 1, "key" :

{ "_id" : 1 }

, "ns" : "vintank.geoFenceActivity", "name" : "id" }
{ "v" : 1, "key" :

{ "site" : 1, "srcId" : 1 }

, "unique" : true, "ns" : "vintank.geoFenceActivity", "name" : "site_1_srcId_1" }
{ "v" : 1, "key" :

{ "pubDt" : 1 }

, "ns" : "vintank.geoFenceActivity", "name" : "pubDt_1", "expireAfterSeconds" : 1209600 }
{ "v" : 1, "key" :

{ "siteType" : 1, "fncPt" : 1, "loc.geoPt" : "2dsphere", "pubDt" : -1 }

, "ns" : "vintank.geoFenceActivity", "name" : "siteType_1_fncPt_1_loc.geoPt_2dsphere_pubDt_-1" }

I checked our mongodb log and didn't see anything out of the ordinary.

This my first bug submission so please let me know if I can get you more information. I've tried to repeat the problem on my local dev system by randomly throwing thousands of queries at the collection but have not been able to reproduce.



 Comments   
Comment by James Jory [ 10/Jun/13 ]

Reporting back after a few days of running with 2.4.4 as promised. Issue has not resurfaced since upgrading to 2.4.4.

Comment by James Jory [ 08/Jun/13 ]

Just upgraded this server to 2.4.4. Everything is working great so far. Will report back in a couple days if still working correctly or if the problem recurs (at which time I'll try changing the index as you suggest).

Thanks!

Comment by hari.khalsa@10gen.com [ 08/Jun/13 ]

Hi! I suspect this is a bug that is fixed in 2.4.4. See here: https://jira.mongodb.org/browse/SERVER-9647

Basically, the index was being created with the fields as descending but the query was treating the index as ascending, so it would loop forever trying to find something descending.

Can you try using 2.4.4? Does that have the same issue? If you can't use 2.4.4, try creating the compound 2dsphere index with all ascending fields and see if that also hangs; I think it won't.

Generated at Thu Feb 08 03:21:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.