[SERVER-17925] mongodump occasional Assertion failure md src/mongo/util/net/message_port.cpp Created: 08/Apr/15  Updated: 13/Apr/15  Resolved: 13/Apr/15

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: 2.4.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Rolf Schreiber Assignee: Sam Kleinman (Inactive)
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

Call

mongodump -o /tmp/outdir

Participants:

 Description   

I get this output when calling mongodump. The assertion failure happens sometimes earlier or later in the process, or sometimes mongodump completes without an error.

connected to: 127.0.0.1
Wed Apr  8 08:17:45.438 all dbs
Wed Apr  8 08:17:45.441 DATABASE: clubs	 to 	/home/obnam/mongodump/clubs
Wed Apr  8 08:17:45.442 	clubs.system.indexes to /home/obnam/mongodump/clubs/system.indexes.bson
Wed Apr  8 08:17:45.443 		 11 objects
Wed Apr  8 08:17:45.443 	clubs.job to /home/obnam/mongodump/clubs/job.bson
Wed Apr  8 08:17:51.481 		Collection File Writing Progress: 200/7378	2%	(objects)
Wed Apr  8 08:17:57.614 		Collection File Writing Progress: 300/7378	4%	(objects)
Wed Apr  8 08:18:04.171 		Collection File Writing Progress: 400/7378	5%	(objects)
Wed Apr  8 08:18:06.322   Assertion failure md src/mongo/util/net/message_port.cpp 195
0x97d206 0x942310 0x9681fc 0x5aa8e2 0x5d3fe0 0x5abf87 0x5b5407 0x583272 0x5836a1 0x58c2eb 0x58eecd 0x57c190 0x56782f 0x7f963ebaeec5 0x57f8bc 
 mongodump(_ZN5mongo15printStackTraceERSo+0x26) [0x97d206]
 mongodump(_ZN5mongo12verifyFailedEPKcS1_j+0xc0) [0x942310]
 mongodump(_ZN5mongo13MessagingPort4recvERNS_7MessageE+0x3ac) [0x9681fc]
 mongodump(_ZN5mongo18DBClientConnection4recvERNS_7MessageE+0x12) [0x5aa8e2]
 mongodump(_ZN5mongo14DBClientCursor18exhaustReceiveMoreEv+0x80) [0x5d3fe0]
 mongodump(_ZN5mongo18DBClientConnection5queryEN5boost8functionIFvRNS_27DBClientCursorBatchIteratorEEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x1c7) [0x5abf87]
 mongodump(_ZN5mongo12DBClientBase5queryEN5boost8functionIFvRKNS_7BSONObjEEEERKSsNS_5QueryEPS4_i+0x337) [0x5b5407]
 mongodump(_ZN4Dump12doCollectionESsP8_IO_FILEPN5mongo13ProgressMeterE+0x3c2) [0x583272]
 mongodump(_ZN4Dump19writeCollectionFileESsN5boost10filesystem4pathE+0x251) [0x5836a1]
 mongodump(_ZN4Dump2goESsN5boost10filesystem4pathE+0x9ab) [0x58c2eb]
 mongodump(_ZN4Dump3runEv+0x136d) [0x58eecd]
 mongodump(_ZN5mongo4Tool4mainEiPPc+0x7d0) [0x57c190]
 mongodump(main+0x2f) [0x56782f]
 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f963ebaeec5]
 mongodump() [0x57f8bc]
assertion: 0 assertion src/mongo/util/net/message_port.cpp:195



 Comments   
Comment by Sam Kleinman (Inactive) [ 13/Apr/15 ]

Thanks for your patience and attention here. I'm glad that your latest mongodump succeeded. I'm going to go ahead and close this ticket for the moment, but we can reopen this issue or open a new ticket if the issue reappears. Again, running validate() and testing against more recent versions of the tools may give us some additional insight if you run into issues like this in the future.

Cheers,
sam

Comment by Rolf Schreiber [ 10/Apr/15 ]

validate() showed no invalid objects (ok=1)
But today mongodump also worked without problem. I'll try again next time when mongodump fails.

Comment by Sam Kleinman (Inactive) [ 10/Apr/15 ]

Given the presence of the {{verifyFailed}] error message, I think it's important that we rule out problems in both the tools doing the validation, and the data itself. To help us trace down the issue, it will help to see can you see if you can read this data with a different client (e.g. the 3.0 mongodump, Pymongo, another driver.) Additionally, can you check the integrity of your data using validate? You can do so by running this command:

db.collection.validate(true)

Note that this is a resource-intensive operation and may have an impact on the performance of your MongoDB instance.

Regards,
sam

Comment by Rolf Schreiber [ 10/Apr/15 ]

I don't think the data is malformed: I executed the mongodump about 5 times within 30 minutes, while items where added to the collection itself at a rate of perhaps one item every five minutes. One of those 5 attempts succeeded, the other 4 times it failed at different (random) progress percentages. If data were malformed, there should be no successful dump.

Comment by Sam Kleinman (Inactive) [ 08/Apr/15 ]

Hello, sorry that you're seeing this error. It looks like some data is failing a validation check. In this situation we want to:

  • See if there is actually malformed data in your database or locate the cause of the error in mongodump if the data is actually valid.
  • See if other clients and more recent versions of mongodump can read the data in the collection.

In pursuit of this, can you help us continue to debug this issue by:

  • Attempting to use both the 3.0 and 2.6 versions of mongodump to dump the contents of clubs.job?
  • Use another client (e.g. Python/node.js/etc.) to query and de-serialize the data in this collection?
  • Provide the name of the client and version of that client that you used to insert and update the data in this collection?

I hope we can understand the underlying issue and find a resolution as soon as possible.

Regards,
sam

Generated at Thu Feb 08 03:46:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.