[SERVER-529] exception while multiple threads exist can cause shell crash Created: 11/Jan/10  Updated: 19/May/14  Resolved: 12/Jan/10

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Aaron Staple Assignee: Aaron Staple
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

I've written the multi thread testing code in the shell to avoid producing this bug, so it's low priority. Here's what happens:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00000010
[Switching to process 67102 thread 0x2103]
0x0021c789 in v8::internal::ThreadManager::EagerlyArchiveThread ()
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) n
Program not restarted.
(gdb) bt
#0 0x0021c789 in v8::internal::ThreadManager::EagerlyArchiveThread ()
#1 0xb0103000 in ?? ()
Cannot access memory at address 0xc0000007

(Here using optimized (non debug) v8, but this bug is easy to reproduce.)

To reproduce:

diff --git a/shell/utils.js b/shell/utils.js
index 9671545..b53f0f5 100644
— a/shell/utils.js
+++ b/shell/utils.js
@@ -387,6 +387,7 @@ if ( typeof _threadInject != "undefined" ){
}

EventGenerator.prototype.addInsert = function( obj )

{ + this._add( "assert( false )" ); this._add( "t.insert( " + tojson( obj ) + " )" ); }

@@ -562,11 +563,12 @@ if ( typeof _threadInject != "undefined" ){

runners.forEach( function( x )

{ x.start(); }

);
var nFailed = 0;

  • runners.forEach( function( x )
    Unknown macro: { if( !x.returnData() ) { ++nFailed; } } );
    -
    - // v8 doesn't like it if we assert before all threads are joined. For now
    - // just avoid that possibility by asserting after threads are joined.
    - assert.eq( 0, nFailed, msg );
    + runners.forEach( function( x ) { assert( x.returnData() ); } );
    +// runners.forEach( function( x ) { if( !x.returnData() ) { ++nFailed; } }

    +//
    +// // v8 doesn't like it if we assert before all threads are joined. For n
    +// // just avoid that possibility by asserting after threads are joined.
    +// assert.eq( 0, nFailed, msg );
    }
    }

and run ./mongo jstests/parallel/insert.js



 Comments   
Comment by auto [ 12/Jan/10 ]

Author:

{'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-529 comment
http://github.com/mongodb/mongo/commit/2652e7faad43dc3c8fc57479dbf88ae16a7b6eab

Comment by Aaron Staple [ 12/Jan/10 ]

Ran another test & it seems to confirm that the problem is as I described.

Comment by Aaron Staple [ 12/Jan/10 ]

Sorry, that part about "threads in v8" should be "threads in boost...boost interface"

Comment by Aaron Staple [ 12/Jan/10 ]

So I'm pretty certain what's happening is as follows: The assertion causes the shell to exit, and on exit v8 starts shutting down (perhaps static objects are destoryed). While this is going on, some of the js threads are still running and they make calls to a partially destroyed v8 engine, resulting in internal v8 assertions and memory errors.

Threads in v8 can't be killed outright (at least not using the v8 interface, maybe it would be possible using a native pthread handle). They can only be interrupted at programmatically defined interruption points. Since a js thread could be in a js only loop at the time an interrupt is requested, so there's no way to interrupt threads in a timely manner in all cases using c++. (There may be a way to implement a js interrupt/timeout, but I haven't investigated.)

A best effort implementation to avoid this bug would be to issue a c++ interrupt on all threads, then wait for them all to finish, and exit if the wait times out. Given that this bug is confined to testing scenarios, I don't think it's worth it to even do that.

Comment by auto [ 12/Jan/10 ]

Author:

{'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-529 when js thread throws an exception, log and check return
http://github.com/mongodb/mongo/commit/386a525c2fc299aeec3943143d5ff04783b4298c

Comment by Aaron Staple [ 11/Jan/10 ]

Also, to clarify, the threads here are JS shell threads created with fork(), new Thread(), etc. These are only used to implement tests currently.

Comment by Aaron Staple [ 11/Jan/10 ]

Note The 'exception' referred to here is a javascript exception.

Generated at Thu Feb 08 02:54:24 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.