[SERVER-7228] Mongo dump fails with BSONElement: bad type xxx Created: 02/Oct/12  Updated: 09/Jan/13  Resolved: 19/Nov/12

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: 2.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Glen Heiberg Assignee: Shaun Verch
Resolution: Duplicate Votes: 1
Labels: dump
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Running a sharded configuration of Mongo 2.2 in Amazon EC2 with 2 replica sets of 2 db nodes each, 1 arbiter for each replSet and 3 config servers.

The config servers and arbiters are EC2 t1.micro instances.
The mongo replica set nodes are EC2 m1.large instances.

All servers are running:
Description: Ubuntu 12.04.1 LTS
Release: 12.04
Codename: precise
Linux 3.2.0-23-virtual #36-Ubuntu SMP Tue Apr 10 22:29:03 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux


Issue Links:
Duplicate
duplicates SERVER-6877 mongodump raises BSONElement: bad typ... Closed
Operating System: Linux
Participants:

 Description   

We experience intermittent failures when doing a mongodump of our production database with errors similar to the one below:

Mon Oct 1 15:30:15 Metadata for Briteblue.activities to /mnt/mongobackup/Briteblue/activities.metadata.json
Mon Oct 1 15:30:15 Briteblue.system.profile to /mnt/mongobackup/Briteblue/system.profile.bson
Mon Oct 1 15:30:15 doing snapshot query
Mon Oct 1 15:30:15 760 objects
Mon Oct 1 15:30:15 Metadata for Briteblue.system.profile to /mnt/mongobackup/Briteblue/system.profile.metadata.json
Mon Oct 1 15:30:15 Assertion: 10320:BSONElement: bad type 95
0xad0f71 0x67f28b 0x55beab 0x7036ef 0x5635d3 0x566ed0 0x568fd1 0xb42d62 0x5545f2 0x7fabf3ca076d 0x554459
mongodump(_ZN5mongo15printStackTraceERSo+0x21) [0xad0f71]
mongodump(_ZN5mongo11msgassertedEiPKc+0x9b) [0x67f28b]
mongodump(_ZNK5mongo11BSONElement4sizeEv+0x1cb) [0x55beab]
mongodump(_ZNK5mongo7BSONObj10jsonStringENS_16JsonStringFormatEi+0xff) [0x7036ef]
mongodump(_ZN4Dump17writeMetadataFileESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEESt3mapISsN5mongo7BSONObjESt4lessISsESaISt4pairIKSsS7_EEESt8multimapISsS7_S9_SD_E+0x363) [0x5635d3]
mongodump(_ZN4Dump2goESsN5boost11filesystem210basic_pathISsNS1_11path_traitsEEE+0x1e50) [0x566ed0]
mongodump(_ZN4Dump3runEv+0x1b61) [0x568fd1]
mongodump(_ZN5mongo4Tool4mainEiPPc+0x1712) [0xb42d62]
mongodump(main+0x32) [0x5545f2]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7fabf3ca076d]
mongodump(__gxx_personality_v0+0x421) [0x554459]
assertion: 10320 BSONElement: bad type 95

Some observations:
The dump always fails when dumping the entire db.
If a dump of the db fails against a specific collection, more often than not, dumping the offending collection on its own is successful
A successful dump of an individual collection may fail 5 minutes later on the same collection
The "BSONElement: bad type" error value changes every time; values seen include 54, 95, 110, 118
The error occurs using mongodump 2.2 on Ubuntu 12 x64, Ubuntu 10 x86, Windows 2008R2, against mongos or directly against a primary node in the replica set.

Oddly, doing a mongodump with the 2.0.7 binary against the very same 2.2 database works perfectly (as does the restore using mongorestore 2.2)

Running db.collection.validate() shows no errors for any of the collections.
We are trying not to having to resort to running a db repair.



 Comments   
Comment by Shaun Verch [ 18/Oct/12 ]

Hi Glen,

It's looking like SERVER-6877 may be exactly the same issue that you are having. Can you still reproduce the problem?

If so, could you do the following?

Next time a mongodump fails, could you log into the mongod that it failed on, increase the log level (http://docs.mongodb.org/manual/reference/command/setParameter/), run mongodump again, and send us the logs from the mongod itself? Even the logs from the last time this failed would be helpful (if you still have the log from around Mon Oct 1 15:30:15).

Thanks!

Comment by Shaun Verch [ 04/Oct/12 ]

Thank you for the report. We are looking into this issue, and will let you know if we need any additional information.

Generated at Thu Feb 08 03:13:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.