[SERVER-9062] mongod crashes when creating a "2dsphere" index on a large (~10 million docs) collection. Created: 21/Mar/13  Updated: 11/Jul/16  Resolved: 29/Mar/13

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 2.4.0
Fix Version/s: 2.4.2, 2.5.0

Type: Bug Priority: Major - P3
Reporter: Greg Brondo Assignee: hari.khalsa@10gen.com
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows 8 64bit. Mongo 2.4 (2008+) 64bit.


Operating System: Windows
Steps To Reproduce:

1) Create new collection
2) insert 10 million docs with {loc:{type: "Point", coordinates: [1,1]}}
3) db.collection.ensureIndex(loc, "2dsphere")

Participants:

 Description   

mongod crashes when creating a "2dsphere" index on a large (~10 million docs) collection. It also crashes during inserts if I create the index before beginning inserts (crashes somewhere north of 800,000 docs).

ERROR output is below:

Thu Mar 21 14:27:55.446 [conn4] build index drn2.events

{ loc: "2dsphere" }

[14:28:00] src\third_party\s2\s2latlng.cc:15: Check failed: is_valid()



 Comments   
Comment by auto [ 28/Mar/13 ]

Author:

{u'date': u'2013-03-25T15:01:40Z', u'name': u'Hari Khalsa', u'email': u'hkhalsa@10gen.com'}

Message: SERVER-9062 don't rely on drem to normalize longitude – explicitly reject OOB values
Branch: v2.4
https://github.com/mongodb/mongo/commit/033f58e7d51c6d3740a22e92c311420ff776e312

Comment by auto [ 27/Mar/13 ]

Author:

{u'date': u'2013-03-25T15:01:40Z', u'name': u'Hari Khalsa', u'email': u'hkhalsa@10gen.com'}

Message: SERVER-9062 don't rely on drem to normalize longitude – explicitly reject OOB values
Branch: master
https://github.com/mongodb/mongo/commit/2ef273f283c7d45b9dfa55f5adf14f2b63069dc7

Comment by hari.khalsa@10gen.com [ 22/Mar/13 ]

We do bounds check/normalize the values...though clearly not perfectly

I suspect the issue lies with how we normalize the longitude values.

Latitude values are clamped to -90,90, and we error if we see a value that is out of range.

Longitude values are converted to radians and then normalized (a floating point modulus, basically) to [-pi, pi]. I think this last normalization step is flawed on Windows for very large longitude values, like the one you provided. This can be avoided by just erroring out like is done with latitude. I'll write a patch and update when it's in. My apologies.

Comment by Greg Brondo [ 22/Mar/13 ]

longitude = -1.95932812755e+23.

Shouldn't correct behavior be to bounds check the lon, lat values? (-180..180, -90..90) ?

Comment by hari.khalsa@10gen.com [ 22/Mar/13 ]

I agree it is a bug. What is the value that caused it to crash? I will fix the behavior.

Comment by Greg Brondo [ 22/Mar/13 ]

I figured it out. One of my rows had an invalid value for latitude (it was in scientific notation). The error message was the clue:

src\third_party\s2\s2latlng.cc:15: Check failed: is_valid()

I believe this to still be a bug as I would think s2latlng.cc would give a Warning ignoring the bad value instead of crashing the mongod process.

Comment by hari.khalsa@10gen.com [ 22/Mar/13 ]

Given the error, I suspect there could be a particular long/lat value that is causing a problem. If there is any way you could find the specific insert from your database that crashes it, that would be very helpful.

Comment by Greg Brondo [ 22/Mar/13 ]

Also, I'm inserting the data with the PyMongo driver. Here's the code:

from pymongo import MongoClient

mongoConn = MongoClient()
mongoDB = mongoConn.drn2
mongoColl = mongoDB.events

c = 0
for line in open("Events_10000000.txt"):
f = line.split(',')
c += 1
if c % 20000 == 0: print c
event = { "loc":

{ "type": "Point", "coordinates": [ float(f[6]), float(f[7]) ] }

}
mongoColl.insert(event)

mongoConn.disconnect()

Comment by Greg Brondo [ 22/Mar/13 ]

I'll try the test now. I actually used real long/lat values obtained from our database so it wasn't just [1,1]. I'll reply shortly.

Comment by hari.khalsa@10gen.com [ 21/Mar/13 ]

OK, so I ran the following tests on both Linux and Win2k8R2:

1. Insert 10M docs of the provided format, build 2dsphere.
2. Ensure 2dsphere, insert 800k documents of the provided format.

...and I didn't have any problems. I haven't run it on a copy of Windows 8 yet, but will do so ASAP.

If you try to reproduce yourself, do you reliably get the error, or did you have it just once?

Comment by hari.khalsa@10gen.com [ 21/Mar/13 ]

Hi! I'm going to try to reproduce this locally. Just to verify, if I do something like this, I should expect to reproduce the crash?:

t = db.crash
t.drop()
t.ensureIndex(

{loc: "2dsphere"}

)
for (var i = 0; i < 800000; ++i) { t.insert({loc: {type: "Point", coordinates: [1,1]}}); }

Are there any points in the collection besides [1,1]?

Thanks!

Generated at Thu Feb 08 03:19:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.