[SERVER-6877] mongodump raises BSONElement: bad type 117 error Created: 28/Aug/12 Updated: 03/Apr/23 Resolved: 08/Jan/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Admin |
| Affects Version/s: | 2.2.0, 2.2.1 |
| Fix Version/s: | 2.2.2, 2.3.0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Balint Erdi | Assignee: | Spencer Brody (Inactive) |
| Resolution: | Done | Votes: | 10 |
| Labels: | bson, crash | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
mongodump starts to properly create dumps for collections and then crashes, always at the same collection. Here is the dump: Tue Aug 28 07:30:26 Assertion: 10320:BSONElement: bad type 117 The collection contains documents with two ISODate fields and a string field. |
| Comments |
| Comment by Spencer Brody (Inactive) [ 08/Jan/13 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
We believe we have finally reached the bottom of this issue. It looks like it was fixed as part of I'm closing out this ticket, but if anyone experiences this on 2.2.2 or later, please re-open. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 27/Nov/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
We're still trying to get a reproducible test data set to work with. Please let us know if you have one you're able to share or remove/obfuscate sensitive data. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Jan Stanzel [ 26/Nov/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Just a quick note that this issue is still alive and affecting us. Unfortunately, we also have problems reproducing them with a reduced dataset, and we cannot release our whole database. Using mongodump 2.0.7 works so far, but feels rather "dirty".. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Alberto Ornaghi [ 25/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
i know. the data itself is not corrupted. it is dumped correctly. once reimported it works. the fact is that it fails randomly (apparently) to dump it from the running db and the strange thing is that the error is different from time to time. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Spencer Brody (Inactive) [ 25/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi alor, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Alberto Ornaghi [ 25/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ops sorry i've copied and pasted from a previous comment and used: scp -P 722 <zipped archive of data files> aaronboyd@www.10gen.com: i notice only now that it had to be alberto@... | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Spencer Brody (Inactive) [ 25/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
alor, what name did you upload the files under? In the SCP command, this would be whatever you put immediately before "@www.10gen.com:" | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Alberto Ornaghi [ 25/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
to me, it happens randomly on different collections but always when it is creating the metadata.json file for the description of the collection itself. the bson file is correctly dumped. i've uploaded an archive via scp, named audit.zip with the dump data of the offending collection. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Spencer Brody (Inactive) [ 17/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
aaronyo, were you able to figure out which specific index being dropped allowed the dump to work again? If so, what type of index was it? Did it have any special options? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Spencer Brody (Inactive) [ 16/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yes, using mongodump from 2.0.7 is a valid workaround and should be safe, though mongodump from 2.0 does not capture special collection options such as if they are capped. Thanks for the information that this may be related to indexes. Can you (and anyone else who's seeing this) post the results of db.system.indexes.find() on the database having the problems? To the others who have seen this - does removing indexes fix the problem? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Aaron Boyd [ 16/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I discovered that using mongodump v2.0.4 works to dump data from my mongo v2.2.0 db. I then can restore that data using mongorestore v2.2.0. I had missed the short comment about this by Olivier RICARD above, so worth making this clear to others, I think: THERE IS A WORKAROUND: use mongodump 2.0. Could someone from 10gen comment on whether this strategy is safe? The other workaround I discovered is doing a db.copyDatabase(), then deleting all indexes, then dumping. Not great for several reasons, but might workout if you define your indexes somewhere else like in your app. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Aaron Boyd [ 16/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Seems related to indexes. If I restore the db to a known good state, it dumps fine. As soon as I fire up my app (node), which leads to creation of indexes, mongodump starts yielding the bad type errors. When I drop a bunch of indexes, the dump works again... | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Aaron Boyd [ 16/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Spencer – I sent what you requested using the scp command you gave me. The file is called namespaces.out. Hope this helps. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Spencer Brody (Inactive) [ 15/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Can someone who can reproduce this attach the output of db.system.namespaces.find() from the database with the collection that is crashing the dump? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Steven Hirschorn [ 15/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
We're getting this error too. It seems to be intermittent. While investigating why scheduled mongodumps were sometimes creating 0-byte backups I manually backed up the database successfully. I've since tried twice more and got a "assertion: 10320 BSONElement: bad type 26" and before that "bad type 89". It's not clear if it is a particular collection causing the problem, though the mongodump output immediately prior to the error always refers to the same 0-byte (empty) collection. Unfortunately we too can't upload our data, but we'll be grateful for a fix ASAP and happy to help with any other troubleshooting. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Gabor Toth [ 10/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I've tried to reproduce it and i saw some strange things. When i dumped with the old tool (2.0.7) it worked like a charm, the new tool failed, after a mongorestore --drop, the new tool failed again at another collection, after a second try, the new tool failed again at third collection, collection seems to be random. At this point, i deleted documents one by one from the failed collection, and the error occured every time, even with empty collection. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Aaron Boyd [ 10/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Is there anything else I can send you or any other ways we could debug this? I can't just send you my entire db as there is private information in there. When I made a copy, and then deleted the private collections, the dump no longer failed, even the the collection that it was failing on before is still in there. (Same problem László had.) Did anyone else successfully debug this in place? I'm thinking that's what needs to happen. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Spencer Brody (Inactive) [ 09/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
aaronyo, would it be possible for you to send us your data files to help us debug? You can do this by shutting down your server, copying your data files, and sending us a zip of them using the following SCP command:
If it prompts you for a password just leave it empty and hit enter. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Aaron Boyd [ 08/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Actually, no – I was just doing a db.copyDatabase() in the mongo shell. I took another look today and the behavior is pretty erratic. Different collections are failing now. When dumping the live db, it now fails on RequestPO. When I try to dump just that collection (using -c option), it still fails. However, I can dump just that collection from a copy (after using db.copyDatabase). But, I can't dump the entire copy – it fails on ResourceReplacementPO. However, the original live db has no trouble dumping ResourceReplacementPO, and neither does the copy when I dump just that collection. In all cases, when a dump fails, I get a "assertion: 10320 BSONElement: bad type xxx" | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Spencer Brody (Inactive) [ 05/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi ~aaronyo, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Aaron Boyd [ 05/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Same problem on a small collection that definitely has some non ASCII chars in it. When I copy the database, the collection exports just fine. This makes things more difficult to troubleshoot and may explain why you're not able to reproduce the error on your side – something seems to get "fixed" when the copy happens. Let me know if I can be of more help. I can send you whatever data you like on this collection. aaron@ip-10-172-191-216:~/dump$ mongodump -d nomic -c Invite --forceTableScan | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by László Bácsi [ 28/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Unfortunately, this is our main database and I'm not allowed to share it with any third parties. I've tried to drop collections in order to filter them down to a set that still produces the issue but doesn't contain any sensitive information. However, even if I drop only a few collections, the issue goes away. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Spencer Brody (Inactive) [ 27/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
lackac, we would only need the data files for the cartman database. We have an SCP server set up for receiving files larger than can be attached to Jira. You can upload the files by running:
If it prompts you for a password just leave it empty and hit enter. Given how unsuccessful I've been so far at reproducing or identifying the cause, having a copy of datafiles that are exhibiting this behavior would be extremely valuable. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by László Bácsi [ 27/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
We have since migrated our servers to another infrastructure provider. We brought over the original data folders so I suspected that the dump issue woudn't go away, and it haven't. I tried dumping the collections one by one and this way no errors were produced. Here's the output I get when dumping the whole database:
I suspect that the issue is with the categories collection. After the failure the @categories.metadata.json@ file is empty. Only one collection has special options, it's a capped collection but it's not the categories collection. There are no non-ASCII characters in the index names. I can't upload the raw data files as they are quite big. I'm have attached the mongodb logs from during the run of mongodump. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Spencer Brody (Inactive) [ 26/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Another question to anyone who's experienced this - does the collection that the dump break on have any special options set (for example is it capped, a TTL collection, using usePowerOf2Sizes, etc)? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Pierre Baillet [ 26/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Here is the output I get for information:
And then:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Spencer Brody (Inactive) [ 26/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hmm. That's really interesting. To the other people who've seen this issue, does dumping just the one collection that the full mongodump to crash on also cause the dump on just that collection to crash? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Pierre Baillet [ 26/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi, We have the issue here, but I'm not able to clearly identify which collection crashes the tool. The stacktrace happens just after a " [tools] Metadata for core..... to plop.core....json" but dumping this only collection does not crash mongodump. FWIW, it's likely that we have non-ASCII in collection data, but very unlikely in index names | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Spencer Brody (Inactive) [ 26/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
So far we've been unable to reproduce this on our end. lackac or jacek (since you have the smallest collections demonstrating the problem), could one of you upload the following things to help us debug:
If you are uncomfortable uploading that information to this publicly-viewable ticket, please create a new ticket in the Community Private project (which will be viewable only to you and to employees of 10gen) and post the new ticket number here. To everyone who's encountered this: do you have non-ASCII Unicode characters in your index names or the data for the collection that is showing the problem? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Zsolt Márton [ 26/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sorry, I think I made a user error, I skipped the changes in findAndModify interface. Now my tests run correctly. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 26/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
zsolt.marton.hu I am able to dump and restore both the before and after files you have attached. Can you please list the steps you are doing to cause an error? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Zsolt Márton [ 26/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi, I have the same issue, but with findAndModify command. Please find attached the mongodump before the command and after it. Wed Sep 26 10:13:20 [conn1] runQuery called test_teamwork.$cmd { findandmodify: "bc_ut__sha512", query: { _id: BinData }, update: { $set: { a: 320653800 }, $inc: { s: 0, t: 0 } }, upsert: true } , update: { $set: { a: 320653800 }, $inc: { s: 0, t: 0 } }, upsert: true } , update: { $set: { a: 320653800 }, $inc: { s: 0, t: 0 }}, upsert: true } update: { $set: { a: 320653800 }, $inc: { s: 0, t: 0 }} ntoreturn:1 nscanned:0 nupdated:1 fastmodinsert:1 keyUpdates:0 locks(micros) w:267 reslen:91 0ms My bad type is 116. It's with Windows 7 x86_64. I also ran our same test with mongod-2.0.6 and mongodump-2.0.6 (on my win7 host), the bson files in the dumps are the same. If needed, I can run the test on OSX 10.6. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Zsolt Márton [ 26/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dump before and after findAndModify command: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 25/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
lackac or tgabi333, if you can use 2.0.7 to dump the small collections which have the problem and can attach to the ticket, it would help us debug this quickly. Thanks in advance. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Joseph Chen [ 24/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I'm also seeing this issue with 2.2.0 but with a different error: bad type 78
Using the force table scan option that someone recommended did NOT fix the problem. db.coll.validate(true) also did not reveal any problems on the collection. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andrew Bennett [ 07/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I have the same issue with mongodump 2.2.0, but I was able to get mongodump to work with the --forceTableScan option.
The following works fine without any errors.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Olivier RICARD [ 07/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I have the same issue with mongodump 2.2. The workaround is to dump the databases with mongodump 2.0 and it works. The issue does not seem linked with the data but with the tool mongodump. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Gabor Toth [ 07/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Same here when dumping database (version 2.2.0). This collection contains only 4 elements: Fri Sep 7 13:23:28 Metadata for xx.YY to dump/xx/YY.metadata.json db.YY.stats() , db.YY.validate(true) , { "loc" : "0:7cf000", "xnext" : "null", "xprev" : "0:b5000", "nsdiag" : "xx.YY", "size" : 32768, "firstRecord" : "0:7cf0b0", "lastRecord" : "0:7cffb8" } ], , , Collection is readable, seems to be okay. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Jacek Jarmulak [ 04/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Same here, started happening after update to 2.2.0 Tue Sep 4 12:02:51 doing snapshot query | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by László Bácsi [ 04/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I also have the same issue, except it's "bad type 50". It's on 2.2.0 final on a small collection with only 5 items, and db.taxonomies.validate(true) returned with no errors. Here's the mongodump output: Tue Sep 4 13:35:39 cartman.taxonomies to dump/cartman/taxonomies.bson | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by bishwa shrestha [ 03/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I'm also getting similar error from the core server: v2.0.7 Mon Sep 3 13:39:13 [conn50] Assertion: 10320:BSONElement: bad type 117 Please help! | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Balint Erdi [ 28/Aug/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The collection is tiny (135 documents), the validation ran fine in no time and returned no errors. One more thing is that the database (which I'm trying to mongodump now) has been mongodumped and restored from a mongodb 2.0.2 database. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Scott Hernandez (Inactive) [ 28/Aug/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Can you run db.coll.validate(true) from the mongo shell on that collection. This will walk through each document and inspect all data structures for corruption and validity. If the collection is large this may take a while and will cause effectively cause the system to wait to do anything (no writes will be allowed) until it is done. |