[SERVER-13597] MongoDB 2.6 crashes unexpectedly on Windows (2.4 or earlier was stable) Created: 15/Apr/14  Updated: 10/Dec/14  Resolved: 16/Apr/14

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 2.6.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Michael Kennedy Assignee: Mark Benvenuto
Resolution: Duplicate Votes: 0
Labels: Windows, crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive MongoDB Crash.zip    
Issue Links:
Duplicate
duplicates SERVER-13516 Array updates on documents with more ... Closed
Operating System: ALL
Steps To Reproduce:

Not sure, see crash dumps?

Participants:

 Description   

We have been running MongoDB for http://learninglineapp.com and http://develop.com for over a year (maybe almost two). They are hosted in EC2 on Windows server 2008 R2. Until we switched to 2.6 they have been rock solid.

But since we upgraded to 2.6, MongoDB has begun crashing unexpectedly. I don't have much information to put in the description. However, I do have the log file and minidumps. They are attached.

Here is what is in the Windows event log:

The MongoDB (master) service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.

Finally, here is what we get back in .NET as a stack trace (nothing interesting, just the server is missing basically):

MongoDB.Driver.MongoConnectionException: Unable to connect to server localhost:27017: No connection could be made because the target machine actively refused it 127.0.0.1:27017. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:27017
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.Net.Sockets.TcpClient.Connect(IPEndPoint remoteEP)
at MongoDB.Driver.Internal.MongoConnection.Open()
at MongoDB.Driver.Internal.MongoConnection.SendMessage(BsonBuffer buffer, Int32 requestId)
at MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoRequestMessage message)
at MongoDB.Driver.Operations.CommandOperation`1.Execute(MongoConnection connection)
at MongoDB.Driver.MongoServerInstance.Ping(MongoConnection connection)
at MongoDB.Driver.MongoServerInstance.Connect()
at MongoDB.Driver.Internal.DirectMongoServerProxy.Connect(TimeSpan timeout, ReadPreference readPreference)
— End of inner exception stack trace —
at MongoDB.Driver.Internal.DirectMongoServerProxy.Connect(TimeSpan timeout, ReadPreference readPreference)
at MongoDB.Driver.Internal.DirectMongoServerProxy.ChooseServerInstance(ReadPreference readPreference)
at MongoDB.Driver.MongoServer.AcquireConnection(ReadPreference readPreference)
at MongoDB.Driver.MongoCursor`1.MongoCursorConnectionProvider.AcquireConnection()
at MongoDB.Driver.Operations.QueryOperation`1.GetFirstBatch(IConnectionProvider connectionProvider)
at MongoDB.Driver.Operations.QueryOperation`1.Execute(IConnectionProvider connectionProvider)
at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source)
at MongoDB.Driver.Linq.SelectQuery.Execute()
at MongoDB.Driver.Linq.MongoQueryProvider.Execute[TResult](Expression expression)
at System.Linq.Queryable.SingleOrDefault[TSource](IQueryable`1 source, Expression`1 predicate)

I'm happy to work with you (screensharing, whatever) to help track this down.



 Comments   
Comment by Michael Kennedy [ 16/Apr/14 ]

Great, thanks Dan.

Comment by Daniel Pasette (Inactive) [ 16/Apr/14 ]

fix was made for 2.6.1: SERVER-13516
Planning on getting 2.6.1-rc0 out tonight or tomorrow morning.

Comment by Michael Kennedy [ 16/Apr/14 ]

Nice to see you have it reproducible. Any idea when the fix is planned for? If it's not the next point release, I need roll back to 2.4.

Comment by Mark Benvenuto [ 16/Apr/14 ]

It is a duplicate of SERVER-13516. Here is a simple repro:

db.foo.insert(
{ _id : 1, { "TopElement" :  [ 
{ b : 1, suba : [0,1] } ,
(128 more copies)
...
]
}})
 
db.foo.update( {_id : 1},  { $push : { "TopElement.12.suba": "foo" } })

Comment by Michael Kennedy [ 15/Apr/14 ]

That's correct Mark. Thank you for reposting it.

Jason, I restarted my server with verbose logging. It crashes a few times a day. I'll send you the log when it does.

Comment by Mark Benvenuto [ 15/Apr/14 ]

This is stack trace.

mongo::mutablebson::Document::Impl::writeElement<mongo::BSONArrayBuilder>+0x5e [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\bson\mutable\document.cpp @ 2095]
mongo::mutablebson::Document::Impl::writeChildren<mongo::BSONArrayBuilder>+0x90 [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\bson\mutable\document.cpp @ 2147]
mongo::mutablebson::Document::Impl::writeElement<mongo::BSONObjBuilder>+0x15b [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\bson\mutable\document.cpp @ 2112]
mongo::mutablebson::Document::Impl::writeChildren<mongo::BSONObjBuilder>+0x90 [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\bson\mutable\document.cpp @ 2147]
mongo::mutablebson::Document::getObject+0x59 [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\bson\mutable\document-inl.h @ 39]
mongo::update+0xf88 [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\ops\update.cpp @ 749]
mongo::UpdateExecutor::execute+0xd5 [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\ops\update_executor.cpp @ 70]
mongo::multiUpdate+0x748 [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\commands\write_commands\batch_executor.cpp @ 1129]
mongo::WriteBatchExecutor::execUpdate+0x74 [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\commands\write_commands\batch_executor.cpp @ 850]
mongo::WriteBatchExecutor::bulkExecute+0xaf [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\commands\write_commands\batch_executor.cpp @ 719]
mongo::WriteBatchExecutor::executeBatch+0xb6c [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\commands\write_commands\batch_executor.cpp @ 248]
mongo::WriteCmd::run+0x1e6 [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\commands\write_commands\write_commands.cpp @ 146]
mongo::_execCommand+0x5e [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\dbcommands.cpp @ 1357]
mongo::Command::execCommand+0xcf8 [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\dbcommands.cpp @ 1544]
mongo::_runCommands+0x4a7 [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\dbcommands.cpp @ 1650]
mongo::runCommands+0x41 [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\query\new_find.cpp @ 107]
mongo::newRunQuery+0x49d [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\query\new_find.cpp @ 384]
mongo::receivedQuery+0x44f [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\instance.cpp @ 269]
mongo::assembleResponse+0x30b [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\instance.cpp @ 434]
mongo::MyMessageHandler::process+0x111 [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\db\db.cpp @ 202]
mongo::PortMessageServer::handleIncomingMsg+0x677 [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\mongo\util\net\message_server_port.cpp @ 210]
boost::`anonymous namespace'::thread_start_function+0x21 [d:\data\mci\git@github.commongodb\mongo.git\mongodb-mongo-v2.6\src\third_party\boost\libs\thread\src\win32\thread.cpp @ 185]
_callthreadstartex+0x17 [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c @ 314]
_threadstartex+0x7f [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c @ 292]
kernel32!BaseThreadInitThunk+0xd

Comment by J Rassi [ 15/Apr/14 ]

Hi,

Are you able to reproduce this issue? If so, please add the "-vvv" flag to your mongod startup options (or the line "vvv=true" to your mongod config file) and attach the full mongod log file (not just an excerpt, if possible) after reproducing a crash.

~ Jason Rassi

Comment by Michael Kennedy [ 15/Apr/14 ]

Here is the crash in the log (also included in the attached zip file)

2014-04-15T08:17:53.119-0700 [initandlisten] connection accepted from 127.0.0.1:59166 #205 (5 connections now open)
2014-04-15T08:17:58.548-0700 [conn205] *** unhandled exception (access violation) at 0x000000013F5101EE, terminating
2014-04-15T08:17:58.548-0700 [conn205] *** access violation was a read from 0x0000002007377AC2
2014-04-15T08:17:58.548-0700 [conn205] *** stack trace for unhandled exception:
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x178be
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x122c0
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x14eeb
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x17580
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0xc22c9
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x36da78
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x372285
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x1dcc78
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x1ddc94
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x1dfc4f
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x1e0b7c
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x1e2026
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x1f080e
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x1f87c8
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x1f90e7
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x41dee1
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x41ef2d
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x30102f
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0x30312b
2014-04-15T08:17:58.579-0700 [conn205] mongod.exe mongo::parseNumberFromStringWithBase<unsigned char>+0xaeb1e1
2014-04-15T08:17:58.579-0700 [conn205]
2014-04-15T08:17:58.579-0700 [conn205] writing minidump diagnostic file E:\MongoDB\Program\mongod.2014-04-15T15-17-58.mdmp
2014-04-15T08:17:58.657-0700 [conn205] *** immediate exit due to unhandled exception
2014-04-15T08:18:58.734-0700 ***** SERVER RESTARTED *****
2014-04-15T08:18:58.734-0700 Trying to start Windows service 'MongoDB'
2014-04-15T08:18:58.734-0700 Service running

Generated at Thu Feb 08 03:32:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.