[SERVER-9774] Random access exception accessing virtual memory on windows Created: 23/May/13  Updated: 10/Dec/14  Resolved: 20/Jun/14

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 2.4.1, 2.4.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Chad Kreimendahl Assignee: Ramon Fernandez Marina
Resolution: Duplicate Votes: 0
Labels: collector-298ba4e7
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows 8
Windows Server 2012


Attachments: Text File conn3Error.txt     File mongo.dmp    
Issue Links:
Duplicate
duplicates SERVER-13033 Static Initalizer race condition with... Closed
Operating System: Windows
Participants:

 Description   

We've seen this happen in both 2.4.1 and 2.4.3 primarily in Windows 8 environments. Occasionally, following a set of queries, an access violation will cause mongod to exit and crash.

Exception: "The thread tried to read from or write to a virtual address for which it does not have the appropriate access."

(logs & dump attached)



 Comments   
Comment by Chad Kreimendahl [ 05/Feb/14 ]

Part 2: our system.profile of the query in question:

/* 1 */
{
    "op" : "command",
    "ns" : "Dev.$cmd",
    "command" : {
        "mapreduce" : "Coll49",
        "map" : function() {var self = this;
    var dt27946_1 = function(utc) {
    var adjusted = new Date(utc.getTime() + -21600000),
    adjustedYear = adjusted.getUTCFullYear(),
    adjustedMonth = adjusted.getUTCMonth(),
    adjustedDay = adjusted.getUTCDate(); 
    var resultYear = adjustedYear,
    resultMonth = adjustedMonth,
    resultDay = adjustedDay;
    return new Date(Date.UTC(resultYear, resultMonth, resultDay));
   };
   emit({'27946_1': self.F['27946'] && self.F['27946'].V ? dt27946_1(self.F['27946'].V) : null}, {Count0_0: 1});},
        "reduce" : function(key, values) {var result = {Count0_0: 0};values.forEach(function(value) {result.Count0_0 += value.Count0_0;});return result;},
        "query" : {
            "$nor" : [ 
                {
                    "F.27947.Sel" : {
                        "$size" : 1
                    },
                    "F.27947.Sel.Id" : LUUID("27b40bf7-2f2c-a443-a3fe-1f8129a3d3f5")
                }
            ]
        },
        "out" : {
            "inline" : 1
        }
    },
    "ntoreturn" : 1,
    "keyUpdates" : 0,
    "numYield" : 0,
    "lockStats" : {
        "timeLockedMicros" : {
            "r" : NumberLong(7585),
            "w" : NumberLong(0)
        },
        "timeAcquiringMicros" : {
            "r" : NumberLong(17),
            "w" : NumberLong(11)
        }
    },
    "responseLength" : 1097,
    "millis" : 8,
    "ts" : ISODate("2014-02-05T18:28:53.395Z"),
    "client" : "127.0.0.1",
    "allUsers" : [],
    "user" : ""
}

Comment by Chad Kreimendahl [ 05/Feb/14 ]

Part one of updating this... as we are still seeing it: Also, feel free to add windows 8.1 and server 2012 r2 to the environments list.

Wed Feb 05 09:13:59.576 [conn336] *** unhandled exception (access violation) at 0x00007FF704A499EE, terminating
Wed Feb 05 09:13:59.576 [conn336] *** access violation was a read from 0x0000000000000000
Wed Feb 05 09:13:59.576 [conn336] *** stack trace for unhandled exception:
Wed Feb 05 09:13:59.577 [conn339] *** unhandled exception (access violation) at 0x00007FF704A499EE, terminating
Wed Feb 05 09:13:59.577 [conn339] *** access violation was a read from 0x0000000000000000
Wed Feb 05 09:13:59.577 [conn339] *** stack trace for unhandled exception:
Wed Feb 05 09:13:59.577 [conn332] *** unhandled exception (access violation) at 0x00007FF704A499EE, terminating
Wed Feb 05 09:13:59.577 [conn332] *** access violation was a read from 0x0000000000000000
Wed Feb 05 09:13:59.577 [conn332] *** stack trace for unhandled exception:
Wed Feb 05 09:13:59.593 [conn323] command OnspringDev.$cmd command: { mapreduce: "Coll49", map: function() {var self = this;
var dt27946_1 = function(utc) {
var adj..., reduce: function(key, values) {var result = {Count0_0: 0};values.forEach(funct..., query: { $nor: [ { F.27947.Sel: { $size: 1 }, F.27947.Sel.Id: BinData } ] }, out: { inline: 1 } } ntoreturn:1 keyUpdates:0 locks(micros) r:14814 reslen:1097 252ms
Wed Feb 05 09:14:04.154 [conn336] mongod.exe    ...\src\mongo\scripting\engine.cpp(174)                          mongo::Scope::loadStored+0x10e
Wed Feb 05 09:14:04.154 [conn336] mongod.exe    ...\src\mongo\scripting\engine.cpp(423)                          mongo::ScriptEngine::getPooledScope+0x1e5
Wed Feb 05 09:14:04.154 [conn336] mongod.exe    ...\src\mongo\db\commands\mr.cpp(625)                            mongo::mr::State::init+0x6c
Wed Feb 05 09:14:04.154 [conn336] mongod.exe    ...\src\mongo\db\commands\mr.cpp(1166)                           mongo::mr::MapReduceCommand::run+0x82a
Wed Feb 05 09:14:04.154 [conn336] mongod.exe    ...\src\mongo\db\dbcommands.cpp(1927)                            mongo::_execCommand+0x5e
Wed Feb 05 09:14:04.154 [conn336] mongod.exe    ...\src\mongo\db\dbcommands.cpp(2036)                            mongo::Command::execCommand+0x936
Wed Feb 05 09:14:04.155 [conn336] mongod.exe    ...\src\mongo\db\dbcommands.cpp(2128)                            mongo::_runCommands+0x3fd
Wed Feb 05 09:14:04.155 [conn336] mongod.exe    ...\src\mongo\db\ops\query.cpp(46)                               mongo::runCommands+0x46
Wed Feb 05 09:14:04.155 [conn336] mongod.exe    ...\src\mongo\db\ops\query.cpp(952)                              mongo::runQuery+0x43e
Wed Feb 05 09:14:04.155 [conn336] mongod.exe    ...\src\mongo\db\instance.cpp(262)                               mongo::receivedQuery+0x35d
Wed Feb 05 09:14:04.155 [conn336] mongod.exe    ...\src\mongo\db\instance.cpp(401)                               mongo::assembleResponse+0x2f0
Wed Feb 05 09:14:04.155 [conn336] mongod.exe    ...\src\mongo\db\db.cpp(194)                                     mongo::MyMessageHandler::process+0xfa
Wed Feb 05 09:14:04.155 [conn336] mongod.exe    ...\src\mongo\util\net\message_server_port.cpp(207)              mongo::PortMessageServer::handleIncomingMsg+0x578
Wed Feb 05 09:14:04.155 [conn336] mongod.exe    ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(180)  boost::`anonymous namespace'::thread_start_function+0x21
Wed Feb 05 09:14:04.155 [conn336] mongod.exe    f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(314)      _callthreadstartex+0x17
Wed Feb 05 09:14:04.155 [conn336] mongod.exe    f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(292)      _threadstartex+0x7f
Wed Feb 05 09:14:04.155 [conn336] KERNEL32.DLL                                                                   BaseThreadInitThunk+0xd
Wed Feb 05 09:14:04.157 [conn336] writing minidump diagnostic file mongo.dmp
Wed Feb 05 09:14:04.291 [conn336] *** immediate exit due to unhandled exception

Comment by Stennie Steneker (Inactive) [ 13/Aug/13 ]

Hi Chad,

We don't have enough information to be able to reproduce or progress this issue, so I'm going to close as Incomplete.

If you are still seeing this exception with regular frequency, please feel free to reopen the issue with additional details.

Regards,
Stephen

Comment by Tad Marshall [ 23/May/13 ]

Hi Chad,

The stack trace shows that a MapReduce job used a null pointer to read something (read from address 0). It is trying to use a "pooled" JavaScript "scope" which appears to be invalid.

Can you post the MapReduce code that was running at the time of the crash?

Tad

Generated at Thu Feb 08 03:21:24 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.