[SERVER-4893] Mongo 2.0.2 crashes when a large file is written to GridFS : "Assertion failure a <= 512*1024*1024 util/alignedbuilder.cpp" Created: 07/Feb/12 Updated: 29/Apr/13 Resolved: 15/Apr/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 2.0.2 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Ilya Katsov | Assignee: | Mathias Stearn |
| Resolution: | Incomplete | Votes: | 3 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
CentOS 5.7, single-node deployment, 32GB RAM, 16 cores, 100GB Vertex SSD drive |
||
| Attachments: |
|
||||
| Issue Links: |
|
||||
| Operating System: | Linux | ||||
| Participants: | |||||
| Description |
|
The following code is used to save 1GB file to GridFS:
This constantly crashes server with the following problem:
|
| Comments |
| Comment by Mathias Stearn [ 11/Apr/13 ] | ||||||||||||||||||
|
Has anyone been able to repro with 2.4? I just tested uploading 10,000 files (both large and small) to gridfs and was not able to get this warning to occur. Some of the changes to how journalling is handled in 2.2 for db-level-locking should have helped prevent this from occurring. | ||||||||||||||||||
| Comment by Ben Becker [ 17/Aug/12 ] | ||||||||||||||||||
|
Hi Maxime, That should indeed work, but we still need to address the core issue. The GridFS use case you mentioned may help us reproduce; thanks for the info. Please do let me know if you encounter this issue with SafeMode.Journal enabled. Best Regards, | ||||||||||||||||||
| Comment by Maxime Beaudry [ 15/Aug/12 ] | ||||||||||||||||||
|
Hi Ben, this issues was encountered when inserting multiple files to GridFS. The files are inserted in a single threaded tight loop using the Mongo C# driver. The issue was raised when uploading about 56000 files that range from 0 bytes to 514 Mb. Note that only 410 files out of 56000 were larger than 256 kb (the GridFS chunk size). I tried to use the workaround suggested by Jose but unfortunately the GetLastError method of the C# driver does not support the "j" option. I therefore changed my SafeMode object used when opening the database to have a similar behavior. Here is the code: var safeMode = new SafeMode(true); var connectionString = "mongodb://localhost"; I am currently trying again with this new `safeMode` and I can see that the journal file is always empty. I will let you know the outcome. | ||||||||||||||||||
| Comment by Jose Sebastian Battig [ 14/Aug/12 ] | ||||||||||||||||||
|
Having the same issue, this "solves" the problem from the client side of mongo running out of memory due to writer not catching up:
| ||||||||||||||||||
| Comment by Ben Becker [ 13/Aug/12 ] | ||||||||||||||||||
|
Hi Maxime, Was this encountered while inserting a single gridfs file or multiple files (concurrently)? Also, how large were these files? Thanks, | ||||||||||||||||||
| Comment by Maxime Beaudry [ 11/Aug/12 ] | ||||||||||||||||||
|
Hi Ben and Michel, I am currently evaluating MongoDb and I had a very similar crash. Here is the output of my mongod.exe process on Windows 2008 R2 (x64) when I try to insert lots and lots of data through the C# GridFS API: Fri Aug 10 16:24:12 [conn2339] command DIT.$cmd command: { filemd5: ObjectId('50256deb4921f623207b2ecb'), root: "fs" } ntoreturn:1 reslen:94 280ms ntoreturn:1 reslen:94 1778ms Have you guys made any progress on this bug? Can I help in some way? If this is not fixed, is there any work around that you suggest? | ||||||||||||||||||
| Comment by Eliot Horowitz (Inactive) [ 03/Jul/12 ] | ||||||||||||||||||
|
@jose - that sounds different. can you open a new ticket? also, what version on windows? | ||||||||||||||||||
| Comment by Jose Sebastian Battig [ 03/Jul/12 ] | ||||||||||||||||||
|
I'm having same exact problem. virtualized VM Windows 2008R264 bits. Tried scenario with limited RAM available to MongoD as well as unlimited ability to allocate RAM. Mongo will get up to 5gig of RAM until the DB grows about 8GIG of size it will crash. All attempts so far it crashed more or less at the same place (I can tell exactly, but it was always running the same process after DB grew to the mentioned size). | ||||||||||||||||||
| Comment by Somsak Sriprayoonsakul [ 08/May/12 ] | ||||||||||||||||||
|
We have the same problem. We tried to put a 1GB file into GridFS and, from time to time, mongod crash with similar error message Platform: CentOS 6.2 on X86_64 Here is the error message Tue May 8 22:36:55 [conn4] insert ark.objectcontent.chunks 657ms Tue May 8 22:37:15 Backtrace: Logstream::get called in uninitialized state | ||||||||||||||||||
| Comment by Michel Brazeau [ 10/Apr/12 ] | ||||||||||||||||||
|
Hi Ben, I set my page size setting to "manual settings" with a minimum of 8000MB Tue Apr 10 17:01:09 [conn1] warning assertion failure a <= 256*1024*1024 Then the server shuts down. The log seems identical as before. I'd be very Michel – | ||||||||||||||||||
| Comment by Michel Brazeau [ 10/Apr/12 ] | ||||||||||||||||||
|
Hi Ben, I checked all the system logs in the Windows Event Viewer for the date/time As far as the system is concerned it's a core I7 with 8GB of ram and 337GB Later today I'll try to run a test with something like 4 times the current It is obviously a large stress test. Large records (128Kb) are inserted Michel | ||||||||||||||||||
| Comment by Ben Becker [ 09/Apr/12 ] | ||||||||||||||||||
|
Hi Michel, Could you check your system event logs for any memory or disk related events? Also, how large is your page file? In your case it looks like the journal is growing faster than the OS will allow (due to VM or disk contention). This might be indicative of a lack of available resources, in which case I would recommend adding more shards and/or upgrading the server. That being said, we still need to address how mongod.exe encounters and handles this case. I'm still tracking that down and will update this issue as soon as I know more. | ||||||||||||||||||
| Comment by Michel Brazeau [ 09/Apr/12 ] | ||||||||||||||||||
|
Hi Ben, Thanks for the reply. No I am not using GridFS, just normal inserts but implementing my own I'm using Delphi with a modified version of My test suite has too many internal dependencies to send out as a whole. I changed my test case since then by reducing the size of data. But this The error seems to occur twice. On line 159302 it doesn't seem fatal, but I hope this helps find the problem. We plan on using MongoDb in production http://www.myactiverecords.com to handle the storage and retrieval of electronic records. Best regards, Michel – | ||||||||||||||||||
| Comment by Ben Becker [ 06/Apr/12 ] | ||||||||||||||||||
|
Hi Michel, When you encountered this, where you writing a GridFS file or executing normal inserts? Is this reproducible? Could you attach the logs? Thanks! | ||||||||||||||||||
| Comment by Michel Brazeau [ 04/Apr/12 ] | ||||||||||||||||||
|
Hi Ben, I'm getting the same error and the server crashes on Windows 64 bit in a test case inserting many records about 128Kb in size in a large 18GB database. mongod -version gives: C:\mongodb\bin>mongod -version Thanks, Michel | ||||||||||||||||||
| Comment by Ben Becker [ 09/Feb/12 ] | ||||||||||||||||||
|
Thanks Ilya; will look into this issue soon. | ||||||||||||||||||
| Comment by Ilya Katsov [ 08/Feb/12 ] | ||||||||||||||||||
|
Hi Ben, Driver version is 2.7.2, mongod version is 2.0.2, 64-bit Linux host. | ||||||||||||||||||
| Comment by Ben Becker [ 07/Feb/12 ] | ||||||||||||||||||
|
Hi Ilya, Can you confirm the version of mongod and the java driver are you using? Also, is this a 64-bit Linux host? |