[SERVER-6877] mongodump raises BSONElement: bad type 117 error Created: 28/Aug/12  Updated: 03/Apr/23  Resolved: 08/Jan/13

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 2.2.0, 2.2.1
Fix Version/s: 2.2.2, 2.3.0

Type: Bug Priority: Major - P3
Reporter: Balint Erdi Assignee: Spencer Brody (Inactive)
Resolution: Done Votes: 10
Labels: bson, crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File dump-6877.tgz     Text File mongod.log    
Issue Links:
Depends
depends on SERVER-7190 Mongodump metadata.json files should ... Closed
Duplicate
is duplicated by SERVER-7228 Mongo dump fails with BSONElement: ba... Closed
Related
Participants:

 Description   

mongodump starts to properly create dumps for collections and then crashes, always at the same collection.

Here is the dump:

Tue Aug 28 07:30:26 Assertion: 10320:BSONElement: bad type 117
0xaeb951 0x58d0eb 0x55beab 0xb128ef 0x5635d3 0x566ed0 0x568fd1 0xa34f52 0x5545f2 0x7fa722c25c4d 0x554459
mongodump(_ZN5mongo15printStackTraceERSo+0x21) [0xaeb951]
mongodump(_ZN5mongo11msgassertedEiPKc+0x9b) [0x58d0eb]
mongodump(_ZNK5mongo11BSONElement4sizeEv+0x1cb) [0x55beab]
mongodump(_ZNK5mongo7BSONObj10jsonStringENS_16JsonStringFormatEi+0xff) [0xb128ef]
mongodump(_ZN4Dump17writeMetadataFileESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEESt3mapISsN5mongo7BSONObjESt4lessISsESaISt4pairIKSsS7_EEESt8multimapISsS7_S9_SD_E+0x363) [0x5635d3]
mongodump(_ZN4Dump2goESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEE+0x1e50) [0x566ed0]
mongodump(_ZN4Dump3runEv+0x1b61) [0x568fd1]
mongodump(_ZN5mongo4Tool4mainEiPPc+0x1712) [0xa34f52]
mongodump(main+0x32) [0x5545f2]
/lib/libc.so.6(__libc_start_main+0xfd) [0x7fa722c25c4d]
mongodump(__gxx_personality_v0+0x421) [0x554459]
assertion: 10320 BSONElement: bad type 117

The collection contains documents with two ISODate fields and a string field.



 Comments   
Comment by Spencer Brody (Inactive) [ 08/Jan/13 ]

We believe we have finally reached the bottom of this issue. It looks like it was fixed as part of SERVER-7190, which is in 2.2.2.

I'm closing out this ticket, but if anyone experiences this on 2.2.2 or later, please re-open.

Comment by Daniel Pasette (Inactive) [ 27/Nov/12 ]

We're still trying to get a reproducible test data set to work with. Please let us know if you have one you're able to share or remove/obfuscate sensitive data.

Comment by Jan Stanzel [ 26/Nov/12 ]

Just a quick note that this issue is still alive and affecting us. Unfortunately, we also have problems reproducing them with a reduced dataset, and we cannot release our whole database.

Using mongodump 2.0.7 works so far, but feels rather "dirty"..

Comment by Alberto Ornaghi [ 25/Oct/12 ]

i know. the data itself is not corrupted. it is dumped correctly. once reimported it works. the fact is that it fails randomly (apparently) to dump it from the running db and the strange thing is that the error is different from time to time.
unfortunately i cannot send you the complete datafile since it contains lot of data and many (130) others collections.
if it help to understand, my setup is a single machine with a config server, a router and a shard on it.

Comment by Spencer Brody (Inactive) [ 25/Oct/12 ]

Hi alor,
I restored the dump you provided and had no problem dumping it. Rather than a mongodump, would it be possible for you to send a copy of your actual datafiles for this database? You can get the data files by shutting down mongod and then copying all the files that start with "<dbname>." in mongod's data directory, where <dbname> is the name of the database you're having trouble dumping.

Comment by Alberto Ornaghi [ 25/Oct/12 ]

ops sorry i've copied and pasted from a previous comment and used:

scp -P 722 <zipped archive of data files> aaronboyd@www.10gen.com:

i notice only now that it had to be alberto@...
so my fault, uploaded as aaronboyd...

Comment by Spencer Brody (Inactive) [ 25/Oct/12 ]

alor, what name did you upload the files under? In the SCP command, this would be whatever you put immediately before "@www.10gen.com:"

Comment by Alberto Ornaghi [ 25/Oct/12 ]

to me, it happens randomly on different collections but always when it is creating the metadata.json file for the description of the collection itself. the bson file is correctly dumped.
i've not been able to identify a special attributes that causes it. sometimes is a sharded collection, sometimes is a standard collection with just 3 indexes.
the json file is actually created and it seems correct. the mongodump process just seems to crash in the process with bad type error (yesterday 54, today 48).

i've uploaded an archive via scp, named audit.zip with the dump data of the offending collection.

Comment by Spencer Brody (Inactive) [ 17/Oct/12 ]

aaronyo, were you able to figure out which specific index being dropped allowed the dump to work again? If so, what type of index was it? Did it have any special options?

Comment by Spencer Brody (Inactive) [ 16/Oct/12 ]

Yes, using mongodump from 2.0.7 is a valid workaround and should be safe, though mongodump from 2.0 does not capture special collection options such as if they are capped.

Thanks for the information that this may be related to indexes. Can you (and anyone else who's seeing this) post the results of db.system.indexes.find() on the database having the problems? To the others who have seen this - does removing indexes fix the problem?

Comment by Aaron Boyd [ 16/Oct/12 ]

I discovered that using mongodump v2.0.4 works to dump data from my mongo v2.2.0 db. I then can restore that data using mongorestore v2.2.0. I had missed the short comment about this by Olivier RICARD above, so worth making this clear to others, I think:

THERE IS A WORKAROUND: use mongodump 2.0.

Could someone from 10gen comment on whether this strategy is safe?

The other workaround I discovered is doing a db.copyDatabase(), then deleting all indexes, then dumping. Not great for several reasons, but might workout if you define your indexes somewhere else like in your app.

Comment by Aaron Boyd [ 16/Oct/12 ]

Seems related to indexes. If I restore the db to a known good state, it dumps fine. As soon as I fire up my app (node), which leads to creation of indexes, mongodump starts yielding the bad type errors. When I drop a bunch of indexes, the dump works again...

Comment by Aaron Boyd [ 16/Oct/12 ]

Spencer – I sent what you requested using the scp command you gave me. The file is called namespaces.out. Hope this helps.

Comment by Spencer Brody (Inactive) [ 15/Oct/12 ]

Can someone who can reproduce this attach the output of db.system.namespaces.find() from the database with the collection that is crashing the dump?

Comment by Steven Hirschorn [ 15/Oct/12 ]

We're getting this error too. It seems to be intermittent. While investigating why scheduled mongodumps were sometimes creating 0-byte backups I manually backed up the database successfully. I've since tried twice more and got a "assertion: 10320 BSONElement: bad type 26" and before that "bad type 89". It's not clear if it is a particular collection causing the problem, though the mongodump output immediately prior to the error always refers to the same 0-byte (empty) collection.

Unfortunately we too can't upload our data, but we'll be grateful for a fix ASAP and happy to help with any other troubleshooting.

Comment by Gabor Toth [ 10/Oct/12 ]

I've tried to reproduce it and i saw some strange things.

When i dumped with the old tool (2.0.7) it worked like a charm, the new tool failed, after a mongorestore --drop, the new tool failed again at another collection, after a second try, the new tool failed again at third collection, collection seems to be random. At this point, i deleted documents one by one from the failed collection, and the error occured every time, even with empty collection.

Comment by Aaron Boyd [ 10/Oct/12 ]

Is there anything else I can send you or any other ways we could debug this? I can't just send you my entire db as there is private information in there. When I made a copy, and then deleted the private collections, the dump no longer failed, even the the collection that it was failing on before is still in there. (Same problem László had.) Did anyone else successfully debug this in place? I'm thinking that's what needs to happen.

Comment by Spencer Brody (Inactive) [ 09/Oct/12 ]

aaronyo, would it be possible for you to send us your data files to help us debug? You can do this by shutting down your server, copying your data files, and sending us a zip of them using the following SCP command:

scp -P 722 <zipped archive of data files> aaronboyd@www.10gen.com: 

If it prompts you for a password just leave it empty and hit enter.

Comment by Aaron Boyd [ 08/Oct/12 ]

Actually, no – I was just doing a db.copyDatabase() in the mongo shell.

I took another look today and the behavior is pretty erratic. Different collections are failing now. When dumping the live db, it now fails on RequestPO. When I try to dump just that collection (using -c option), it still fails. However, I can dump just that collection from a copy (after using db.copyDatabase). But, I can't dump the entire copy – it fails on ResourceReplacementPO. However, the original live db has no trouble dumping ResourceReplacementPO, and neither does the copy when I dump just that collection.

In all cases, when a dump fails, I get a "assertion: 10320 BSONElement: bad type xxx"

Comment by Spencer Brody (Inactive) [ 05/Oct/12 ]

Hi ~aaronyo,
To confirm, when you say you copy the database, do you mean that you are shutting down the server, copying the data files to a new directory, and then starting up a new mongod pointing at the new data files?

Comment by Aaron Boyd [ 05/Oct/12 ]

Same problem on a small collection that definitely has some non ASCII chars in it.

When I copy the database, the collection exports just fine. This makes things more difficult to troubleshoot and may explain why you're not able to reproduce the error on your side – something seems to get "fixed" when the copy happens.

Let me know if I can be of more help. I can send you whatever data you like on this collection.

aaron@ip-10-172-191-216:~/dump$ mongodump -d nomic -c Invite --forceTableScan
connected to: 127.0.0.1
Fri Oct 5 21:15:06 DATABASE: nomic to dump/nomic
Fri Oct 5 21:15:06 nomic.Invite to dump/nomic/Invite.bson
Fri Oct 5 21:15:06 21 objects
Fri Oct 5 21:15:06 Metadata for nomic.Invite to dump/nomic/Invite.metadata.json
Fri Oct 5 21:15:06 Assertion: 10320:BSONElement: bad type 105
0xad0f71 0x67f28b 0x55beab 0x7036ef 0x5635d3 0x566ed0 0x568fd1 0xb42d62 0x5545f2 0x7f1ec317230d 0x554459
mongodump(_ZN5mongo15printStackTraceERSo+0x21) [0xad0f71]
mongodump(_ZN5mongo11msgassertedEiPKc+0x9b) [0x67f28b]
mongodump(_ZNK5mongo11BSONElement4sizeEv+0x1cb) [0x55beab]
mongodump(_ZNK5mongo7BSONObj10jsonStringENS_16JsonStringFormatEi+0xff) [0x7036ef]
mongodump(_ZN4Dump17writeMetadataFileESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEESt3mapISsN5mongo7BSONObjESt4lessISsESaISt4pairIKSsS7_EEESt8multimapISsS7_S9_SD_E+0x363) [0x5635d3]
mongodump(_ZN4Dump2goESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEE+0x1e50) [0x566ed0]
mongodump(_ZN4Dump3runEv+0x1b61) [0x568fd1]
mongodump(_ZN5mongo4Tool4mainEiPPc+0x1712) [0xb42d62]
mongodump(main+0x32) [0x5545f2]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f1ec317230d]
mongodump(__gxx_personality_v0+0x421) [0x554459]
assertion: 10320 BSONElement: bad type 105
aaron@ip-10-172-191-216:~/dump$ mongodump --version
mongodump version 2.2.0
aaron@ip-10-172-191-216:~/dump$ mongo --version
MongoDB shell version: 2.2.0

Comment by László Bácsi [ 28/Sep/12 ]

Unfortunately, this is our main database and I'm not allowed to share it with any third parties. I've tried to drop collections in order to filter them down to a set that still produces the issue but doesn't contain any sensitive information. However, even if I drop only a few collections, the issue goes away.

Comment by Spencer Brody (Inactive) [ 27/Sep/12 ]

lackac, we would only need the data files for the cartman database. We have an SCP server set up for receiving files larger than can be attached to Jira. You can upload the files by running:

scp -P 722 <zipped archive of data files> laszlo@www.10gen.com: 

If it prompts you for a password just leave it empty and hit enter.

Given how unsuccessful I've been so far at reproducing or identifying the cause, having a copy of datafiles that are exhibiting this behavior would be extremely valuable.

Comment by László Bácsi [ 27/Sep/12 ]

We have since migrated our servers to another infrastructure provider. We brought over the original data folders so I suspected that the dump issue woudn't go away, and it haven't.

I tried dumping the collections one by one and this way no errors were produced.

Here's the output I get when dumping the whole database:

Thu Sep 27 10:02:02 creating new connection to:127.0.0.1:27817
Thu Sep 27 10:02:02 BackgroundJob starting: ConnectBG
Thu Sep 27 10:02:02 connected connection!
connected to: 127.0.0.1:27817
Thu Sep 27 10:02:02 DATABASE: cartman	 to 	dump/cartman
Thu Sep 27 10:02:02 	skipping collection: cartman.system.users.$_id_
Thu Sep 27 10:02:02 	skipping collection: ...
...
Thu Sep 27 10:02:02 	cartman.system.users to dump/cartman/system.users.bson
Thu Sep 27 10:02:02 doing snapshot query
Thu Sep 27 10:02:02 		 1 objects
Thu Sep 27 10:02:02 	Metadata for cartman.system.users to dump/cartman/system.users.metadata.json
Thu Sep 27 10:02:02 	cartman.admin_imports to dump/cartman/admin_imports.bson
Thu Sep 27 10:02:02 doing snapshot query
Thu Sep 27 10:02:02 		 953 objects
Thu Sep 27 10:02:02 	Metadata for cartman.admin_imports to dump/cartman/admin_imports.metadata.json
Thu Sep 27 10:02:02 	cartman.admins to dump/cartman/admins.bson
Thu Sep 27 10:02:02 doing snapshot query
Thu Sep 27 10:02:02 		 1 objects
Thu Sep 27 10:02:02 	Metadata for cartman.admins to dump/cartman/admins.metadata.json
Thu Sep 27 10:02:02 	cartman.domains to dump/cartman/domains.bson
Thu Sep 27 10:02:02 doing snapshot query
Thu Sep 27 10:02:02 		 3 objects
Thu Sep 27 10:02:02 	Metadata for cartman.domains to dump/cartman/domains.metadata.json
Thu Sep 27 10:02:02 	cartman.products to dump/cartman/products.bson
Thu Sep 27 10:02:02 doing snapshot query
Thu Sep 27 10:02:02 		 110411 objects
Thu Sep 27 10:02:02 	Metadata for cartman.products to dump/cartman/products.metadata.json
Thu Sep 27 10:02:02 	cartman.customers to dump/cartman/customers.bson
Thu Sep 27 10:02:02 doing snapshot query
Thu Sep 27 10:02:05 		1214500/1324362	91%	(objects)
Thu Sep 27 10:02:05 		 1324362 objects
Thu Sep 27 10:02:05 	Metadata for cartman.customers to dump/cartman/customers.metadata.json
Thu Sep 27 10:02:05 	cartman.shoppers to dump/cartman/shoppers.bson
Thu Sep 27 10:02:05 doing snapshot query
Thu Sep 27 10:02:08 		2788500/2820074	98%	(objects)
Thu Sep 27 10:02:08 		 2820074 objects
Thu Sep 27 10:02:08 	Metadata for cartman.shoppers to dump/cartman/shoppers.metadata.json
Thu Sep 27 10:02:08 	cartman.categories to dump/cartman/categories.bson
Thu Sep 27 10:02:08 doing snapshot query
Thu Sep 27 10:02:08 		 14 objects
Thu Sep 27 10:02:08 	Metadata for cartman.categories to dump/cartman/categories.metadata.json
Thu Sep 27 10:02:08 Assertion: 10320:BSONElement: bad type 48
0xad0f71 0x67f28b 0x55beab 0x7036ef 0x5635d3 0x566ed0 0x568fd1 0xb42d62 0x5545f2 0x7f080b85076d 0x554459 
 mongodump(_ZN5mongo15printStackTraceERSo+0x21) [0xad0f71]
 mongodump(_ZN5mongo11msgassertedEiPKc+0x9b) [0x67f28b]
 mongodump(_ZNK5mongo11BSONElement4sizeEv+0x1cb) [0x55beab]
 mongodump(_ZNK5mongo7BSONObj10jsonStringENS_16JsonStringFormatEi+0xff) [0x7036ef]
 mongodump(_ZN4Dump17writeMetadataFileESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEESt3mapISsN5mongo7BSONObjESt4lessISsESaISt4pairIKSsS7_EEESt8multimapISsS7_S9_SD_E+0x363) [0x5635d3]
 mongodump(_ZN4Dump2goESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEE+0x1e50) [0x566ed0]
 mongodump(_ZN4Dump3runEv+0x1b61) [0x568fd1]
 mongodump(_ZN5mongo4Tool4mainEiPPc+0x1712) [0xb42d62]
 mongodump(main+0x32) [0x5545f2]
 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f080b85076d]
 mongodump(__gxx_personality_v0+0x421) [0x554459]

I suspect that the issue is with the categories collection. After the failure the @categories.metadata.json@ file is empty.

Only one collection has special options, it's a capped collection but it's not the categories collection. There are no non-ASCII characters in the index names.

I can't upload the raw data files as they are quite big. I'm have attached the mongodb logs from during the run of mongodump.

Comment by Spencer Brody (Inactive) [ 26/Sep/12 ]

Another question to anyone who's experienced this - does the collection that the dump break on have any special options set (for example is it capped, a TTL collection, using usePowerOf2Sizes, etc)?

Comment by Pierre Baillet [ 26/Sep/12 ]

Here is the output I get for information:

Wed Sep 26 22:04:57 [tools]     Metadata for core.computed_best_contributed_picture_sets to paf/core/computed_best_contributed_picture_sets.metadata.json
Wed Sep 26 22:04:57 [tools] Assertion: 10320:BSONElement: bad type 100
0x6ec686 0x9e52d8 0x5620c1 0x5f1076 0x60c1ad 0x569211 0x56b662 0x56d9d4 0x9cb19a 0x55bdc0 0x7fb57d97fc4d 0x55b919
 mongodump(_ZN5mongo15printStackTraceERSo+0x26) [0x6ec686]
 mongodump(_ZN5mongo11msgassertedEiPKc+0x98) [0x9e52d8]
 mongodump(_ZNK5mongo11BSONElement4sizeEv+0x151) [0x5620c1]
 mongodump(_ZN5mongo15BSONObjIterator4nextEv+0x46) [0x5f1076]
 mongodump(_ZNK5mongo7BSONObj10jsonStringENS_16JsonStringFormatEi+0xcd) [0x60c1ad]
 mongodump(_ZN4Dump17writeMetadataFileESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEESt3mapISsN5mongo7BSONObjESt4lessISsESaISt4pairIKSsS7_EEESt8multimapISsS7_S9_SD_E+0x571) [0x569211]
 mongodump(_ZN4Dump2goESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEE+0x1b02) [0x56b662]
 mongodump(_ZN4Dump3runEv+0xea4) [0x56d9d4]
 mongodump(_ZN5mongo4Tool4mainEiPPc+0x11ca) [0x9cb19a]
 mongodump(main+0x30) [0x55bdc0]
 /lib/libc.so.6(__libc_start_main+0xfd) [0x7fb57d97fc4d]
 mongodump() [0x55b919]
assertion: 10320 BSONElement: bad type 100
Wed Sep 26 22:04:57 dbexit:
Wed Sep 26 22:04:57 [tools] shutdown: going to close listening sockets...
Wed Sep 26 22:04:57 [tools] shutdown: going to flush diaglog...
Wed Sep 26 22:04:57 [tools] shutdown: going to close sockets...
Wed Sep 26 22:04:57 [tools] shutdown: waiting for fs preallocator...
Wed Sep 26 22:04:57 [tools] shutdown: closing all files...
Wed Sep 26 22:04:59 [tools]             6/11    54%
Wed Sep 26 22:04:59 [tools] closeAllFiles() finished
Wed Sep 26 22:04:59 [tools] shutdown: removing fs lock...
Wed Sep 26 22:04:59 dbexit: really exiting now

And then:

[22:05][prod] root@a01:/ebs/lvms/vol0# mongodump -d core -dbpath mongodb-shard-4-pink-alpha/ -c computed_best_contributed_picture_sets
Wed Sep 26 22:05:56 [tools] DATABASE: core       to     dump/core
Wed Sep 26 22:05:56 [tools]     core.computed_best_contributed_picture_sets to dump/core/computed_best_contributed_picture_sets.bson
Wed Sep 26 22:05:56 [tools]              3018 objects
Wed Sep 26 22:05:56 [tools]     Metadata for core.computed_best_contributed_picture_sets to dump/core/computed_best_contributed_picture_sets.metadata.json
Wed Sep 26 22:05:56 dbexit:
Wed Sep 26 22:05:56 [tools] shutdown: going to close listening sockets...
Wed Sep 26 22:05:56 [tools] shutdown: going to flush diaglog...
Wed Sep 26 22:05:56 [tools] shutdown: going to close sockets...
Wed Sep 26 22:05:56 [tools] shutdown: waiting for fs preallocator...
Wed Sep 26 22:05:56 [tools] shutdown: closing all files...
Wed Sep 26 22:05:56 [tools] closeAllFiles() finished
Wed Sep 26 22:05:56 [tools] shutdown: removing fs lock...
Wed Sep 26 22:05:56 dbexit: really exiting now
[22:05][prod] root@a01:/ebs/lvms/vol0#

Comment by Spencer Brody (Inactive) [ 26/Sep/12 ]

Hmm. That's really interesting. To the other people who've seen this issue, does dumping just the one collection that the full mongodump to crash on also cause the dump on just that collection to crash?

Comment by Pierre Baillet [ 26/Sep/12 ]

Hi,

We have the issue here, but I'm not able to clearly identify which collection crashes the tool. The stacktrace happens just after a " [tools] Metadata for core..... to plop.core....json" but dumping this only collection does not crash mongodump.

FWIW, it's likely that we have non-ASCII in collection data, but very unlikely in index names

Comment by Spencer Brody (Inactive) [ 26/Sep/12 ]

So far we've been unable to reproduce this on our end.

lackac or jacek (since you have the smallest collections demonstrating the problem), could one of you upload the following things to help us debug:

  • Logs from the mongod for the time period where you run mongodump
  • The output of db.<collection name>.stats() on the collection that is triggering the problem.
  • A file copy (not mongodump) of the actual data files for the database that is triggering the problem? Note, you will need to shut the server down while you copy the data files.

If you are uncomfortable uploading that information to this publicly-viewable ticket, please create a new ticket in the Community Private project (which will be viewable only to you and to employees of 10gen) and post the new ticket number here.

To everyone who's encountered this: do you have non-ASCII Unicode characters in your index names or the data for the collection that is showing the problem?

Comment by Zsolt Márton [ 26/Sep/12 ]

Sorry, I think I made a user error, I skipped the changes in findAndModify interface. Now my tests run correctly.

Comment by Daniel Pasette (Inactive) [ 26/Sep/12 ]

zsolt.marton.hu I am able to dump and restore both the before and after files you have attached. Can you please list the steps you are doing to cause an error?

Comment by Zsolt Márton [ 26/Sep/12 ]

Hi,

I have the same issue, but with findAndModify command. Please find attached the mongodump before the command and after it.

Wed Sep 26 10:13:20 [conn1] runQuery called test_teamwork.$cmd { findandmodify: "bc_ut__sha512", query:

{ _id: BinData }

, update: { $set:

{ a: 320653800 }

, $inc:

{ s: 0, t: 0 }

}, upsert: true }
Wed Sep 26 10:13:20 [conn1] run command test_teamwork.$cmd { findandmodify: "bc_ut__sha512", query:

{ _id: BinData }

, update: { $set:

{ a: 320653800 }

, $inc:

{ s: 0, t: 0 }

}, upsert: true }
Wed Sep 26 10:13:20 [conn1] command test_teamwork.$cmd command: { findandmodify: "bc_ut__sha512", query:

{ _id: BinData }

, update: { $set:

{ a: 320653800 }

, $inc:

{ s: 0, t: 0 }

}, upsert: true } update: { $set:

{ a: 320653800 }

, $inc:

{ s: 0, t: 0 }

} ntoreturn:1 nscanned:0 nupdated:1 fastmodinsert:1 keyUpdates:0 locks(micros) w:267 reslen:91 0ms

My bad type is 116.

It's with Windows 7 x86_64.

I also ran our same test with mongod-2.0.6 and mongodump-2.0.6 (on my win7 host), the bson files in the dumps are the same.

If needed, I can run the test on OSX 10.6.

Comment by Zsolt Márton [ 26/Sep/12 ]

Dump before and after findAndModify command:
dump-6877.tgz

Comment by Daniel Pasette (Inactive) [ 25/Sep/12 ]

lackac or tgabi333, if you can use 2.0.7 to dump the small collections which have the problem and can attach to the ticket, it would help us debug this quickly. Thanks in advance.

Comment by Joseph Chen [ 24/Sep/12 ]

I'm also seeing this issue with 2.2.0 but with a different error: bad type 78

Mon Sep 24 04:33:56 Assertion: 10320:BSONElement: bad type 78
0xad0f71 0x67f28b 0x55beab 0x7036ef 0x5635d3 0x566ed0 0x568fd1 0xb42d62 0x5545f2 0x7feeb9ca2c4d 0x554459 
 mongodump(_ZN5mongo15printStackTraceERSo+0x21) [0xad0f71]
 mongodump(_ZN5mongo11msgassertedEiPKc+0x9b) [0x67f28b]
 mongodump(_ZNK5mongo11BSONElement4sizeEv+0x1cb) [0x55beab]
 mongodump(_ZNK5mongo7BSONObj10jsonStringENS_16JsonStringFormatEi+0xff) [0x7036ef]
 mongodump(_ZN4Dump17writeMetadataFileESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEESt3mapISsN5mongo7BSONObjESt4lessISsESaISt4pairIKSsS7_EEESt8multimapISsS7_S9_SD_E+0x363) [0x5635d3]
 mongodump(_ZN4Dump2goESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEE+0x1e50) [0x566ed0]
 mongodump(_ZN4Dump3runEv+0x1b61) [0x568fd1]
 mongodump(_ZN5mongo4Tool4mainEiPPc+0x1712) [0xb42d62]
 mongodump(main+0x32) [0x5545f2]
 /lib/libc.so.6(__libc_start_main+0xfd) [0x7feeb9ca2c4d]
 mongodump(__gxx_personality_v0+0x421) [0x554459]
assertion: 10320 BSONElement: bad type 78

Using the force table scan option that someone recommended did NOT fix the problem. db.coll.validate(true) also did not reveal any problems on the collection.

Comment by Andrew Bennett [ 07/Sep/12 ]

I have the same issue with mongodump 2.2.0, but I was able to get mongodump to work with the --forceTableScan option.
I also ran db.product_references.validate(true) and everything was clean. Running db.repairDatabase() did not fix the problem.

mongo1:~$ mongodump
 
-- skip several successful collection dumps --
 
Fri Sep  7 11:28:35   production.product_references to dump/production/product_references.bson
Fri Sep  7 11:28:35 doing snapshot query
Fri Sep  7 11:28:35      6583 objects
Fri Sep  7 11:28:35   Metadata for production.product_references to dump/production/product_references.metadata.json
Fri Sep  7 11:28:35 Assertion: 10320:BSONElement: bad type 48
0xad0f71 0x67f28b 0x55beab 0x7036ef 0x5635d3 0x566ed0 0x56a775 0xb42d62 0x5545f2 0x7fca7b21fc4d 0x554459 
 mongodump(_ZN5mongo15printStackTraceERSo+0x21) [0xad0f71]
 mongodump(_ZN5mongo11msgassertedEiPKc+0x9b) [0x67f28b]
 mongodump(_ZNK5mongo11BSONElement4sizeEv+0x1cb) [0x55beab]
 mongodump(_ZNK5mongo7BSONObj10jsonStringENS_16JsonStringFormatEi+0xff) [0x7036ef]
 mongodump(_ZN4Dump17writeMetadataFileESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEESt3mapISsN5mongo7BSONObjESt4lessISsESaISt4pairIKSsS7_EEESt8multimapISsS7_S9_SD_E+0x363) [0x5635d3]
 mongodump(_ZN4Dump2goESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEE+0x1e50) [0x566ed0]
 mongodump(_ZN4Dump3runEv+0x3305) [0x56a775]
 mongodump(_ZN5mongo4Tool4mainEiPPc+0x1712) [0xb42d62]
 mongodump(main+0x32) [0x5545f2]
 /lib/libc.so.6(__libc_start_main+0xfd) [0x7fca7b21fc4d]
 mongodump(__gxx_personality_v0+0x421) [0x554459]
assertion: 10320 BSONElement: bad type 48

The following works fine without any errors.

mongo1:~$ mongodump --forceTableScan

Comment by Olivier RICARD [ 07/Sep/12 ]

I have the same issue with mongodump 2.2. The workaround is to dump the databases with mongodump 2.0 and it works. The issue does not seem linked with the data but with the tool mongodump.

Comment by Gabor Toth [ 07/Sep/12 ]

Same here when dumping database (version 2.2.0). This collection contains only 4 elements:

Fri Sep 7 13:23:28 Metadata for xx.YY to dump/xx/YY.metadata.json
Fri Sep 7 13:23:28 Assertion: 10320:BSONElement: bad type 50
0xad0f71 0x67f28b 0x55beab 0x7036ef 0x5635d3 0x566ed0 0x56a775 0xb42d62 0x5545f2 0x7f615eba1c8d 0x554459
mongodump(_ZN5mongo15printStackTraceERSo+0x21) [0xad0f71]
mongodump(_ZN5mongo11msgassertedEiPKc+0x9b) [0x67f28b]
mongodump(_ZNK5mongo11BSONElement4sizeEv+0x1cb) [0x55beab]
mongodump(_ZNK5mongo7BSONObj10jsonStringENS_16JsonStringFormatEi+0xff) [0x7036ef]
mongodump(_ZN4Dump17writeMetadataFileESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEESt3mapISsN5mongo7BSONObjESt4lessISsESaISt4pairIKSsS7_EEESt8multimapISsS7_S9_SD_E+0x363) [0x5635d3]
mongodump(_ZN4Dump2goESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEE+0x1e50) [0x566ed0]
mongodump(_ZN4Dump3runEv+0x3305) [0x56a775]
mongodump(_ZN5mongo4Tool4mainEiPPc+0x1712) [0xb42d62]
mongodump(main+0x32) [0x5545f2]
/lib/libc.so.6(__libc_start_main+0xfd) [0x7f615eba1c8d]
mongodump(__gxx_personality_v0+0x421) [0x554459]
assertion: 10320 BSONElement: bad type 50

db.YY.stats()
{
"ns" : "xx.YY",
"count" : 4,
"size" : 6088,
"avgObjSize" : 1522,
"storageSize" : 40960,
"numExtents" : 2,
"nindexes" : 3,
"lastExtentSize" : 32768,
"paddingFactor" : 1.0600000000000014,
"systemFlags" : 1,
"userFlags" : 0,
"totalIndexSize" : 24528,
"indexSizes" :

{ "_id_" : 8176, "index_1" : 8176, "index_2" : 8176 }

,
"ok" : 1
}

db.YY.validate(true)
{
"ns" : "xx.YY",
"firstExtent" : "0:b5000 ns:xx.YY",
"lastExtent" : "0:7cf000 ns:xx.YY",
"extentCount" : 2,
"extents" : [

{ "loc" : "0:b5000", "xnext" : "0:7cf000", "xprev" : "null", "nsdiag" : "xx.YY", "size" : 8192, "firstRecord" : "null", "lastRecord" : "null" }

,

{ "loc" : "0:7cf000", "xnext" : "null", "xprev" : "0:b5000", "nsdiag" : "xx.YY", "size" : 32768, "firstRecord" : "0:7cf0b0", "lastRecord" : "0:7cffb8" }

],
"datasize" : 6088,
"nrecords" : 4,
"lastExtentSize" : 32768,
"padding" : 1.0600000000000014,
"firstExtentDetails" :

{ "loc" : "0:b5000", "xnext" : "0:7cf000", "xprev" : "null", "nsdiag" : "xx.YY", "size" : 8192, "firstRecord" : "null", "lastRecord" : "null" }

,
"objectsFound" : 4,
"invalidObjects" : 0,
"bytesWithHeaders" : 6152,
"bytesWithoutHeaders" : 6088,
"deletedCount" : 23,
"deletedSize" : 34456,
"nIndexes" : 3,
"keysPerIndex" :

{ "xx.YY.$_id_" : 4, "xx.YY.$index_1" : 4, "xx.YY.$index_2" : 4 }

,
"valid" : true,
"errors" : [ ],
"ok" : 1
}

Collection is readable, seems to be okay.

Comment by Jacek Jarmulak [ 04/Sep/12 ]

Same here, started happening after update to 2.2.0

Tue Sep 4 12:02:51 doing snapshot query
Tue Sep 4 12:02:51 26 objects
Tue Sep 4 12:02:51 Metadata for VGReporting.Queue to dump/VGReporting/Queue.metadata.json
Tue Sep 4 12:02:51 Assertion: 10320:BSONElement: bad type 55
0x87a46aa 0x84dfdb5 0x85e0177 0x8175d33 0x854e2e2 0x816fc6e 0x817d99d 0x81834b2 0x8472a87 0x816c349 0x131e9c 0x816c131
mongodump(_ZN5mongo15printStackTraceERSo+0x2a) [0x87a46aa]
mongodump(_ZN5mongo10logContextEPKc+0xa5) [0x84dfdb5]
mongodump(_ZN5mongo11msgassertedEiPKc+0xc7) [0x85e0177]
mongodump(_ZNK5mongo11BSONElement4sizeEv+0x1b3) [0x8175d33]
mongodump(_ZNK5mongo7BSONObj10jsonStringENS_16JsonStringFormatEi+0xe2) [0x854e2e2]
mongodump(_ZN4Dump17writeMetadataFileESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEESt3mapISsN5mongo7BSONObjESt4lessISsESaISt4pairIKSsS7_EEESt8multimapISsS7_S9_SD_E+0x39e) [0x816fc6e]
mongodump(_ZN4Dump2goESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEE+0x1d2d) [0x817d99d]
mongodump(_ZN4Dump3runEv+0x3322) [0x81834b2]
mongodump(_ZN5mongo4Tool4mainEiPPc+0x1de7) [0x8472a87]
mongodump(main+0x59) [0x816c349]
/lib/libc.so.6(__libc_start_main+0xdc) [0x131e9c]
mongodump [0x816c131]
assertion: 10320 BSONElement: bad type 55
Done dump at Tue Sep 4 12:02:51 CDT 2012
[root@vbCentos56 xyz]# mongo --version
MongoDB shell version: 2.2.0
[root@vbCentos56 xyz]# mongodump --version
mongodump version 2.2.0

Comment by László Bácsi [ 04/Sep/12 ]

I also have the same issue, except it's "bad type 50". It's on 2.2.0 final on a small collection with only 5 items, and db.taxonomies.validate(true) returned with no errors.

Here's the mongodump output:

Tue Sep 4 13:35:39 cartman.taxonomies to dump/cartman/taxonomies.bson
Tue Sep 4 13:35:39 doing snapshot query
Tue Sep 4 13:35:39 5 objects
Tue Sep 4 13:35:39 Metadata for cartman.taxonomies to dump/cartman/taxonomies.metadata.json
Tue Sep 4 13:35:39 Assertion: 10320:BSONElement: bad type 50
0xad0f71 0x67f28b 0x55beab 0x7036ef 0x5635d3 0x566ed0 0x56a775 0xb42d62 0x5545f2 0x7f237cec176d 0x554459
mongodump(_ZN5mongo15printStackTraceERSo+0x21) [0xad0f71]
mongodump(_ZN5mongo11msgassertedEiPKc+0x9b) [0x67f28b]
mongodump(_ZNK5mongo11BSONElement4sizeEv+0x1cb) [0x55beab]
mongodump(_ZNK5mongo7BSONObj10jsonStringENS_16JsonStringFormatEi+0xff) [0x7036ef]
mongodump(_ZN4Dump17writeMetadataFileESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEESt3mapISsN5mongo7BSONObjESt4lessISsESaISt4pairIKSsS7_EEESt8multimapISsS7_S9_SD_E+0x363) [0x5635d3]
mongodump(_ZN4Dump2goESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEE+0x1e50) [0x566ed0]
mongodump(_ZN4Dump3runEv+0x3305) [0x56a775]
mongodump(_ZN5mongo4Tool4mainEiPPc+0x1712) [0xb42d62]
mongodump(main+0x32) [0x5545f2]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f237cec176d]
mongodump(__gxx_personality_v0+0x421) [0x554459]
assertion: 10320 BSONElement: bad type 50

Comment by bishwa shrestha [ 03/Sep/12 ]

I'm also getting similar error from the core server: v2.0.7

Mon Sep 3 13:39:13 [conn50] Assertion: 10320:BSONElement: bad type 117
0x584612 0x507a3b 0x51a446 0x8e0683 0x8e13c8 0x96026e 0x96501d 0x88ce24 0x88e7cf 0xaa0b38 0x638767 0x7f0a7e074d8c 0x7f0a7d61ec2d
/tmp/mongod(_ZN5mongo11msgassertedEiPKc+0x112) [0x584612]
/tmp/mongod(_ZNK5mongo11BSONElement4sizeEv+0x1cb) [0x507a3b]
/tmp/mongod(_ZN5mongo15BSONObjIterator4nextEv+0x46) [0x51a446]
/tmp/mongod(_ZN5mongo16MultiPlanScannerC1EPKcRKNS_7BSONObjES5_PKNS_11BSONElementEbS5_S5_bb+0x93) [0x8e0683]
/tmp/mongod(_ZN5mongo11MultiCursorC1EPKcRKNS_7BSONObjES5_N5boost10shared_ptrINS0_8CursorOpEEEbb+0x138) [0x8e13c8]
/tmp/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugEPNS_11RemoveSaverEb+0x38e) [0x96026e]
/tmp/mongod(_ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugEb+0x13d) [0x96501d]
/tmp/mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x474) [0x88ce24]
/tmp/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x116f) [0x88e7cf]
/tmp/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78) [0xaa0b38]
/tmp/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638767]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6d8c) [0x7f0a7e074d8c]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f0a7d61ec2d]
Mon Sep 3 13:39:13 [conn50] Assertion: 10331:EOO Before end of object
0x584612 0x50b9ce 0x89a439 0x890cbd 0x5077da 0x88dfb4 0xaa0b38 0x638767 0x7f0a7e074d8c 0x7f0a7d61ec2d
/tmp/mongod(_ZN5mongo11msgassertedEiPKc+0x112) [0x584612]
/tmp/mongod() [0x50b9ce]
/tmp/mongod(_ZNK5mongo7OpDebug8toStringEv+0xce9) [0x89a439]
/tmp/mongod(_ZNK5mongo14LazyStringImplINS_7OpDebugEE3valEv+0xd) [0x890cbd]
/tmp/mongod(_ZN5mongo9LogstreamlsERKNS_10LazyStringE+0x1a) [0x5077da]
/tmp/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x954) [0x88dfb4]
/tmp/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78) [0xaa0b38]
/tmp/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638767]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6d8c) [0x7f0a7e074d8c]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f0a7d61ec2d]
Mon Sep 3 13:39:13 [conn50] AssertionException handling request, closing client connection: 10331 EOO Before end of object

Please help!

Comment by Balint Erdi [ 28/Aug/12 ]

The collection is tiny (135 documents), the validation ran fine in no time and returned no errors. One more thing is that the database (which I'm trying to mongodump now) has been mongodumped and restored from a mongodb 2.0.2 database.

Comment by Scott Hernandez (Inactive) [ 28/Aug/12 ]

Can you run db.coll.validate(true) from the mongo shell on that collection. This will walk through each document and inspect all data structures for corruption and validity. If the collection is large this may take a while and will cause effectively cause the system to wait to do anything (no writes will be allowed) until it is done.

Generated at Thu Feb 08 03:12:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.