[SERVER-10004] Problem with streaming data into the database Created: 24/Jun/13 Updated: 17/Jul/13 Resolved: 28/Jun/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Henrik Baastrup | Assignee: | Michael Grundy |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
MongoDb version 2.4.1 on a CentOS 6.0 |
||
| Attachments: |
|
| Operating System: | Linux |
| Steps To Reproduce: | I have attached the part of my program there communicate with the database. I have not attached the data I use as we are talking about 300Mb in BSON format, but if necessary I'm ready to supply them. |
| Participants: |
| Description |
|
I'm streaming data in to the MongoDB on a socket using the MongoDB Wire Protocol. It seems like the DB lose the synchronisme, and I have to close and re-open the socket. This happen about every 2000 record I try to insert. 19:27:28.660 [initandlisten] connection accepted from 10.190.35.153:57153 #1733 (3 connections now open) |
| Comments |
| Comment by Michael Grundy [ 28/Jun/13 ] | ||||||||||||||||||||||||||||||
|
Hi Henrik - You can create a SERVER ticket requesting UDP connection support. I couldn't guarantee that it will get on the schedule anytime soon, but it would register your and anyone else's interest in that feature. If you are adept at c++, I encourage you to try modifying to the code to support a UDP connection. A proof of concept would certainly be interesting to see. Thanks! | ||||||||||||||||||||||||||||||
| Comment by Henrik Baastrup [ 28/Jun/13 ] | ||||||||||||||||||||||||||||||
|
I think I found the problem and I am quite sure that it is NOT on the MongoDB size, I have downloaded the MongoDB source, and yes, it looks very nice!
method. Last I would like to mention that using a TCP connection is a limitation for inserting records fast, a profiling of my program will sow that the most of the time is spend on sending data over the socket. It would be nice with a UDP connection for very fast insert of huge amount of data, that is what I address. | ||||||||||||||||||||||||||||||
| Comment by Michael Grundy [ 27/Jun/13 ] | ||||||||||||||||||||||||||||||
|
The problem doesn't lie in in your event objects, they seem reasonable. Lower down the sendMongoBuffer() method is probably the issue. In addition to the assert, we get this message in the log:
That message length is most likely garbage. Also, the counts your program thinks it has inserted do not match the document count in the collection afterwards. I recommend you compare what you think your app is doing to what is really being sent over the wire (tshark, tcpdump etc) Thanks! | ||||||||||||||||||||||||||||||
| Comment by Henrik Baastrup [ 27/Jun/13 ] | ||||||||||||||||||||||||||||||
|
New changed source. The CdrElementMongoDbStorageHandler class contain the logic to sent data to the MongoDB and store failed records to a file | ||||||||||||||||||||||||||||||
| Comment by Henrik Baastrup [ 27/Jun/13 ] | ||||||||||||||||||||||||||||||
|
BSON file with records there failed to be inserted | ||||||||||||||||||||||||||||||
| Comment by Henrik Baastrup [ 27/Jun/13 ] | ||||||||||||||||||||||||||||||
|
First: I have already looked a lot on the Java driver, I even used it in the beginning. I chose to write my own, because I'm going for speed my objective is get 500000 records inserted at second and would like to rich 1000000 recodes at seconds! (it would be nice if MongoDB had an UDP connection) Second: I do not say that it is not my program there fails. But I opened the bug because it looks like; when I resent the same data after having re-connected I get them in. Now I do not know if the error is on the actual record or the record before, I get an IOException and no bytes sent when I send the actual record, so the database might have closed the connect on the previous record.
and the file load with the following output:
My program told me that I stored 220 records and running the following command I also get 220:
To me this prove that my program do format the data correct. I will attach my data file and my new sources to the bug | ||||||||||||||||||||||||||||||
| Comment by Michael Grundy [ 26/Jun/13 ] | ||||||||||||||||||||||||||||||
|
I ran the test program on the latest version of mongod (2.4.4) and 2.4.1 as you did. The logs have some better information about what is going on. This example is the first error encountered on one run (from 2.4.4):
On a full run of your test program, we get an assortment of Asserts:
It looks like your program corrupts its buffer periodically and somewhat randomly (line of first failure is not the same between runs). This leads to malformed BSON which mongod responds to by hanging up the client connection. I think you should give our Java driver a shot and see if you have similar issues. Edit: I did get the asserts on 2.4.1, not just 2.4.4 | ||||||||||||||||||||||||||||||
| Comment by Henrik Baastrup [ 25/Jun/13 ] | ||||||||||||||||||||||||||||||
|
An other note: If a record fails to insert because the database drop the connection, on re-connect the record is inserted. That was the reason I suspect the database and not my program. Of-cause I might be wrong! | ||||||||||||||||||||||||||||||
| Comment by Henrik Baastrup [ 25/Jun/13 ] | ||||||||||||||||||||||||||||||
|
There exist a bug in the tgz archive I already attached, please overload the Main.java file with this one. | ||||||||||||||||||||||||||||||
| Comment by Henrik Baastrup [ 25/Jun/13 ] | ||||||||||||||||||||||||||||||
|
Test program with data attached. Fell free to contact me if you need help to understand whats going on. | ||||||||||||||||||||||||||||||
| Comment by Henrik Baastrup [ 25/Jun/13 ] | ||||||||||||||||||||||||||||||
|
I have data and a small program in Java there transform them and insert in the Database ready. It is about 68 MB in all, I will try tot attach it to this issue. | ||||||||||||||||||||||||||||||
| Comment by Scott Hernandez (Inactive) [ 24/Jun/13 ] | ||||||||||||||||||||||||||||||
|
Do you have sample data and something we can run to reproduce this? Based on the errors it seems like you are sending bad data to the server. |