[SERVER-19644] Seg Fault on cloneCollection (specifically gridfs) Created: 29/Jul/15 Updated: 19/Sep/15 Resolved: 04/Aug/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | GridFS, Replication |
| Affects Version/s: | 2.4.9, 3.0.5, 3.1.6 |
| Fix Version/s: | 3.0.6, 3.1.7 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Amit Lichtenberg | Assignee: | Scott Hernandez (Inactive) |
| Resolution: | Done | Votes: | 1 |
| Labels: | RF | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Backwards Compatibility: | Fully Compatible |
| Operating System: | ALL |
| Backport Completed: | |
| Steps To Reproduce: | Run cloneCollection from a remote gridfs chunks collection |
| Sprint: | RPL 7 08/10/15 |
| Participants: |
| Description |
|
The following error started to occur right after we started deploying with the new v 3.0.5:
It replicates every time ever since. |
| Comments |
| Comment by Githook User [ 07/Aug/15 ] | ||||
|
Author: {u'username': u'scotthernandez', u'name': u'Scott Hernandez', u'email': u'scotthernandez@gmail.com'}Message: | ||||
| Comment by Githook User [ 05/Aug/15 ] | ||||
|
Author: {u'username': u'scotthernandez', u'name': u'Scott Hernandez', u'email': u'scotthernandez@gmail.com'}Message: (cherry picked from commit 66abb09a28b2d9ec6955f454b7ff86824ebc56c1) | ||||
| Comment by Amit Lichtenberg [ 04/Aug/15 ] | ||||
|
Thanks! | ||||
| Comment by Githook User [ 04/Aug/15 ] | ||||
|
Author: {u'username': u'scotthernandez', u'name': u'Scott Hernandez', u'email': u'scotthernandez@gmail.com'}Message: | ||||
| Comment by Scott Hernandez (Inactive) [ 31/Jul/15 ] | ||||
|
Looks like there were two interactions when using those versions which caused the problem starting in 3.0.5; this was due to the format of the old data versus the new data coming back from the listCollections command – specifically the new command always returns an "options" field that when missing from pre-2.6 server caused a problem during cloning. This is good news for you as once we fix these two things it should let you keep running without upgrading from 2.4.x, for a while more at least. | ||||
| Comment by Amit Lichtenberg [ 30/Jul/15 ] | ||||
|
I see. Indeed it is probably about time we upgrade the older dbs. So far everything worked so we didn't have a good-enough excuse, but now we do. About your suggestion to copy the documents one at a time - the number of documents at each clone might be large (hundreds or more), so iterating & copying them pythonically takes (too much) time. Thank you anyway for your attention. I would be glad to here farther updates if and when you have them. | ||||
| Comment by Scott Hernandez (Inactive) [ 30/Jul/15 ] | ||||
|
amitlicht, unfortunately using a pre-2.6.x server, like 2.4.9, as a remote source is not supported for 3.0.x. This is covered in various places in the release notes, but not called out specifically for cloneCollection, as it is basically the same for replication and sharding: http://docs.mongodb.org/manual/release-notes/3.0-upgrade/#upgrade-a-replica-set-to-3-0 We will work on finding the root cause of seg fault but it is unlikely that it make it work with 2.4 since that was not a design goal between those versions. I should have more information in the next few days if you are interested in the underlying technical issues. If you upgrade to 2.6.x it will work correctly, of course. Also, it might be just as easy for you to use the python client to copy the documents manually, from one server to the other, and not use cloneCollection at all – that will work on all server versions as longs you use a version of the python driver with support for them (and yes, the drivers have a much larger range of supported server versions). | ||||
| Comment by Amit Lichtenberg [ 30/Jul/15 ] | ||||
|
That's great! Waiting for updates. | ||||
| Comment by Ramon Fernandez Marina [ 30/Jul/15 ] | ||||
|
Hi amitlicht, using 2.4.9 as the source database (and 3.0.5 as the target) I'm able to reproduce the problem. I'm guessing it may have to do with listIndexes, but that's a first hunch – we're investigating. | ||||
| Comment by Amit Lichtenberg [ 30/Jul/15 ] | ||||
|
Got your request about the -vv, I will make sure to use it when trying to replicate it again. About your request for snippets - I'd rather not share our actual source code. I will try to describe in more details what we try to do, tell me if you need more details and I'll try to share what I can: On both DBs we use gridfs to save small-medium sized files (usually no more then 100Ks). To do this, we have 3 collections:
We use pymongo to clone, mongoengine has almost nothing to do with it. The logic is as follows: we collect 3 lists of object ids \ uuids: files, fs_chunks & fs_files. The files list will contain all the (uuids) of file metadatas which should be cloned. The fs_files will contain all the object ids of the gridfs files which are associated with the files metadatas whose ids are in the files list. The fs_chunks will contain all the object ids of the gridfs chunks which are associated with those same files. Next, we use pymongo for the following clone command: At the end of all three clones, we delete the documents in all three lists from the source db using a similar iteration logic. If I had to point out a single thing about our usecase which might sound unusual is the way we clone the gridfs files by accessing the fs.files & fs.chunks collections directly (i.e. our clone operates directly on db.fs.files & db.fs.chunks). This sounds like a hack-ish way to do it, but it has been working so far. Please update me if this helps in any way or if you need more data. As I said, I hope I'll manage to replicate this on Sunday in code & dbs I can share. Thanks, | ||||
| Comment by Ramon Fernandez Marina [ 30/Jul/15 ] | ||||
|
Thanks amitlicht. I tried again to reproduce using pymongo, unsuccessfully. I very much doubt it's an issue in mongoengine, but I'm not familiar with it so that hasn't been part of my testing. If you can share the relevant snippets on how collections are being cloned from your application I'll try those steps on my end. We'll wait for your attempts on Sunday. Dan's request above (using the -vv switch) will yield more verbose logs, which hopefully will give us some useful hints. Regards, | ||||
| Comment by Daniel Pasette (Inactive) [ 30/Jul/15 ] | ||||
|
Hi Amit. If you are able to run the experiment again on Sunday, can you start the mongod server with verbose logging, i.e., mongod -vv | ||||
| Comment by Amit Lichtenberg [ 30/Jul/15 ] | ||||
|
Hi again. Thanks again for your help, | ||||
| Comment by Ramon Fernandez Marina [ 29/Jul/15 ] | ||||
|
amitlicht, thanks for all your help so far. Unfortunately we haven't been able to reproduce this issue on our end, so here are some options going forward:
Would you be able to accommodate either option? Thanks, | ||||
| Comment by Amit Lichtenberg [ 29/Jul/15 ] | ||||
|
Yes, the machine on which the error happened was downgraded to v3.0.4 (by replacing only the mongodb-org-server package). After that point the error did not happen again. The database was not dropped in between - a simple downgrade + starting the server again seem to have solved it. | ||||
| Comment by J Rassi [ 29/Jul/15 ] | ||||
|
Thanks. When you said "had to downgrade it to an older mongo version to keep working", can you confirm that you downgraded the newly-deployed instance to 3.0.4, and that that resolved the issue? To help isolate the problem, I'm particularly interested in whether this issue is reproducible with all 3.0.x versions, or just 3.0.5. | ||||
| Comment by Amit Lichtenberg [ 29/Jul/15 ] | ||||
| ||||
| Comment by J Rassi [ 29/Jul/15 ] | ||||
|
Thanks for the information. Please do let us know when you're able to reproduce this again. In the meantime, I have a few more follow-up questions:
Thanks again. | ||||
| Comment by Amit Lichtenberg [ 29/Jul/15 ] | ||||
|
As per your other question - the clone commands is executed from our application. We use mongoengine over pymongo, if you think it has anything to do with. Specifically the error occurred while trying to replicate data from a remote DB to our main storage, we do it by running cloneCollection selectively on the gridfs content (chunks & files) according to self-implemented availability criteria. | ||||
| Comment by Amit Lichtenberg [ 29/Jul/15 ] | ||||
|
Reproducing will require some time as I no longer have that setup on which the command failed (had to downgrade it to an older mongo version to keep working). I'll see if we can redeploy and reproduce it. The following output was generated on the same remote db, but after running some other tests on it so it's not guaranteed to represent the status at the time of error.
The other commands are no longer relevant on current DB data. | ||||
| Comment by J Rassi [ 29/Jul/15 ] | ||||
|
I'd like to ask for additional information to further diagnose the issue:
Thanks. | ||||
| Comment by Daniel Pasette (Inactive) [ 29/Jul/15 ] | ||||
|
Also, the log output shows the command is passing a query to cloneCollection in both instances (but different query). Is this from your application or from the shell?
| ||||
| Comment by Daniel Pasette (Inactive) [ 29/Jul/15 ] | ||||
|
Can you paste the output from db.fs.chunks.stats() in the mongo shell on the remote collection please | ||||
| Comment by Amit Lichtenberg [ 29/Jul/15 ] | ||||
|
I cannot add the full load due to company privacy issues. I added partial data including boot info & crush data from two different crushes we had on the same instance in a short time. | ||||
| Comment by Ramon Fernandez Marina [ 29/Jul/15 ] | ||||
|
amitlicht, can you please upload full logs for this node since the last restart? I'm looking specifically for startup options and binary information that appear under the [initandlisten] thread at the beginning of the log. Thanks, | ||||
| Comment by Amit Lichtenberg [ 29/Jul/15 ] | ||||
|
Additional info: local DB version (the one running the clone) is 3.0.5. Remote DB version is 2.6. Another clone which operated on a regular (non-gridfs) collection worked fine. |