[SERVER-14869] Mongodb crashes without log after creating index Created: 12/Aug/14  Updated: 10/Dec/14  Resolved: 12/Aug/14

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kaz Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-14871 Better error message for FileAllocato... Closed
Operating System: ALL
Steps To Reproduce:

Do continously massive insertions > 3Gb/h and create indexes on one of the newly created collections.

Participants:

 Description   

Hello,
I am experiencing an issue with my server. I created a db on my Win 2008 server, it uses a shared drive (using UNC see configuration file bellow). I am running the server as a service run by a different user (non interactive one).

I do massive inserts (about 3Gb/h). When I create indexes (background=True), SOMETIMES my server crashes without any trace on the log files. Moreover, I am not sure that this only due to indexes....

I got 2 different errors :

  • { "serverUsed" : "localhost:27017" , "ok" : 1 , "n" : 0 , "err" : "Can't take a write lock while out of disk space" , "code" : 14031}

    -> THIS IS WEIRD AS I HAVE >400Gb free on the server

  • OR NOTHING at all, with no trace.

I did not have this error before, but since I have changed the dbpath to another one in a share drive, it is not stable anymore !

Besides, I have also tried to set this option to true: smallFiles, but I got the same problem.

Do you have any idea ?
Many thanks

The configuration file is:
systemLog:
destination: file
path: "//london03/TWXAW0_5/Mongo/logs/mongodb.log"
quiet: false
storage:
journal:
enabled: true
dbPath: "//london03/TWXAW0_5/Mongo/db/"
net:
bindIp: 0.0.0.0
port: 27017

Server :

  • Mongo 2.6.4 64bits version for Windows server 2008
  • Windows server 2008 R2 Enterprise
  • 48 Gb of RAM
  • X5680 Xeon 2 processors @3.33Ghz
  • 400 Gb of available space on the shared drive (UNC: //london03/TWXAW0_5/Mongo/db/)


 Comments   
Comment by Kaz [ 13/Aug/14 ]

I have added a thread on StackOverflow as well if you prefer. Thanks
http://stackoverflow.com/questions/25283575/mongodb-crashes-without-log-after-creating-index

Comment by Kaz [ 13/Aug/14 ]

Thank you very much for your answer. I have spent the whole day yesterday understanding the error and looking at the stack overflow threads. I have not found anything about this error, that's why I think this is a bug on the server.
It seems to be very random as I get new errors.
I have now this error :
2014-08-13T10:03:57.688+0100 [conn2] command MARS.$cmd command: insert

{ $msg: "query not recording (too large)" }

keyUpdates:0 numYields:0 reslen:164 858ms
2014-08-13T10:03:59.498+0100 [conn2] MARS.Trades_20140813 Assertion failure isOk() D:\data\mci\shell\src\src\mongo/db/storage/extent.h 80
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\util\stacktrace.cpp(169) mongo::printStackTrace+0x43
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\util\log.cpp(127) mongo::logContext+0x9c
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\util\assert_util.cpp(115) mongo::verifyFailed+0x14a
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\storage\extent_manager.cpp(229) mongo::ExtentManager::getExtent+0x66
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\storage\extent_manager.cpp(428) mongo::ExtentManager::allocFromFreeList+0x17a
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\storage\extent_manager.cpp(487) mongo::ExtentManager::increaseStorageSize+0x61
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\structure\record_store.cpp(213) mongo::SimpleRecordStoreV1::allocRecord+0x128
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\structure\record_store.cpp(94) mongo::RecordStoreV1Base::insertRecord+0x6c
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\catalog\collection.cpp(232) mongo::Collection::_insertDocument+0xb5
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\catalog\collection.cpp(189) mongo::Collection::insertDocument+0x21d
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\commands\write_commands\batch_executor.cpp(1067) mongo::singleInsert+0x72
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\commands\write_commands\batch_executor.cpp(997) mongo::insertOne+0x17a
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\commands\write_commands\batch_executor.cpp(1031) mongo::WriteBatchExecutor::execOneInsert+0xaf
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\commands\write_commands\batch_executor.cpp(837) mongo::WriteBatchExecutor::execInserts+0x130
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\commands\write_commands\batch_executor.cpp(713) mongo::WriteBatchExecutor::bulkExecute+0x52
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\commands\write_commands\batch_executor.cpp(249) mongo::WriteBatchExecutor::executeBatch+0xb6c
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\commands\write_commands\write_commands.cpp(146) mongo::WriteCmd::run+0x1e6
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\dbcommands.cpp(1357) mongo::_execCommand+0x5e
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\dbcommands.cpp(1564) mongo::Command::execCommand+0xf09
2014-08-13T10:04:00.309+0100 [conn2] mongod.exe ...\src\mongo\db\dbcommands.cpp(1696) mongo::_runCommands+0x4a7

I tried the repair command but it would take days for me (several dozens of Gb ) so I stopped. And I started again the DB, and I had the same issue.

Comment by Ramon Fernandez Marina [ 12/Aug/14 ]

Looks like your database files and your logs are both on a network drive, which means that if there's a problem with the network (or the network drive) mongod may not be able to write anything in the logs, and without logs it's quite difficult to diagnose problems. If you are planning to put your database files on a network drive, I would suggest you consider using a local drive for your logs, so if there's a problem you can gather more information about it.

Note also that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server and tools. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group. If after obtaining logs you find a bug in MongoDB please feel free to open a new SERVER ticket and upload the logs to it.

Finally, the "out of disk space" message may not be completely accurate in this case, so I've opened SERVER-14871 see if it can be improved.

Generated at Thu Feb 08 03:36:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.