[SERVER-8105] SIGSEGV in db/clientcursor.cpp Created: 08/Jan/13  Updated: 08/Mar/13  Resolved: 12/Feb/13

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 2.0.7
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Braam van Heerden Assignee: James Wahlin
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Solaris 10u9 amd64


Attachments: File mongo-sigsegv.tgz    
Operating System: Solaris
Steps To Reproduce:

Cannot reproduce in the lab

Participants:

 Description   

I have an issue with a Mongo 2.0.7 installation. Currently this DB is used only to cache data for communication between unrelated processes. The data is not sharded or replicated, and there is one instance running on each computer. There is a seperate replicated sharded set on the same set of computers, but we cannot replicate the issue in the shardeded set, only in the standalone instance.

From time to time (usually about once every 2 months or so), we are detecting that the Mongo local instance has gone down, interrupting IPC for our application suite, and also losing some client notifications. Unfortunately we have not yet been able to identify a series of environmental or data conditions that can trigger this. Unfortunately we also cannot get a SIGSEGV to generate a stack trace we can look at on Solaris. Instead, we have re-compiled the 2.0.7 source and added our own stack trace routines into the code to try and determine the exact point of the failure. I have attached all the stack traces we have to this ticket.

The stack traces are along these lines:

 
-----------------  lwp# 542 / thread# 542  --------------------
 fffffd7ffe56173a read     (39, 27b153a, 1e)
 0000000000c43bd4 _ZN4redi16basic_pstreambufIcSt11char_traitsIcEE11fill_bufferEb () + 164
 0000000000c449e7 _ZN4redi16basic_pstreambufIcSt11char_traitsIcEE9underflowEv () + 27
 0000000000c3d219 _ZNSs12_S_constructISt19istreambuf_iteratorIcSt11char_traitsIcEEEEPcT_S5_RKSaIcESt18input_iterator_tag () + 559
 0000000000e5f570 _ZN5mongo11sunosPstackERSob () + 480
 00000000008cb880 _ZN5mongo10abruptQuitEi () + 3e0
 00000000008cbdf0 _ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv () + 240
 fffffd7ffe55c2e6 __sighndlr () + 6
 fffffd7ffe550bc2 call_user_handler () + 252
 fffffd7ffe550dee sigacthandler (b, fffffd7ff9a07d78, fffffd7ff9a07a10) + ee
 --- called from signal handler with signal 11 (SIGSEGV) ---
 000000000082dec0 _ZN5mongo6Record5touchEb ()
 000000000079233b _ZN5mongo12ClientCursor5yieldEiPNS_6RecordE () + 6b
 000000000079241c _ZN5mongo12ClientCursor14yieldSometimesENS0_11RecordNeedsEPb () + 9c
 000000000086639f _ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugEPNS_11RemoveSaverEb () + 5df
 00000000008695a6 _ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugEb () + 116
 0000000000810af9 _ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE () + 359
 0000000000811d6b _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE () + ecb
 0000000000e5cbec _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE () + ec
 0000000000707b49 _ZN5mongo3pms9threadRunEPNS_13MessagingPortE () + 269
 fffffd7ffe851506 thread_proxy () + 66
 fffffd7ffe55bfbb _thr_setup () + 5b
 fffffd7ffe55c1e0 _lwp_start ()
 

From what we can determine, the issue is on line 443 in db/clientcursor.cpp, where an invalid pointer (not NULL) is used:

 
db/clientcursor.cpp-433-        }
db/clientcursor.cpp-434-        else {
db/clientcursor.cpp-435-            warning() << "don't understand RecordNeeds: " << (int)need << endl;
db/clientcursor.cpp-436-            return 0;
db/clientcursor.cpp-437-        }
db/clientcursor.cpp-438-
db/clientcursor.cpp-439-        DiskLoc l = currLoc();
db/clientcursor.cpp-440-        if ( l.isNull() )
db/clientcursor.cpp-441-            return 0;
db/clientcursor.cpp-442-        
db/clientcursor.cpp:443:        Record * rec = l.rec();
db/clientcursor.cpp-444-        if ( rec->likelyInPhysicalMemory() ) 
db/clientcursor.cpp-445-            return 0;
db/clientcursor.cpp-446-        
db/clientcursor.cpp-447-        return rec;
db/clientcursor.cpp-448-    }
db/clientcursor.cpp-449-
db/clientcursor.cpp-450-    bool ClientCursor::yieldSometimes( RecordNeeds need, bool *yielded ) {
db/clientcursor.cpp-451-        if ( yielded ) {
db/clientcursor.cpp-452-            *yielded = false;   
db/clientcursor.cpp-453-        }
 

This then subsequently causes a SIGSEGV in db/clientcursor.cpp line 512:

 
db/clientcursor.cpp-502-                CurOp * c = cc().curop();
db/clientcursor.cpp-503-                while ( c->parent() )
db/clientcursor.cpp-504-                    c = c->parent();
db/clientcursor.cpp-505-                warning() << "ClientCursor::yield can't unlock b/c of recursive lock"
db/clientcursor.cpp-506-                          << " ns: " << ns 
db/clientcursor.cpp-507-                          << " top: " << c->info()
db/clientcursor.cpp-508-                          << endl;
db/clientcursor.cpp-509-            }
db/clientcursor.cpp-510-
db/clientcursor.cpp-511-            if ( rec )
db/clientcursor.cpp:512:                rec->touch();
db/clientcursor.cpp-513-
db/clientcursor.cpp-514-            lk.reset(0); // need to release this before dbtempreleasecond
db/clientcursor.cpp-515-        }
db/clientcursor.cpp-516-    }
db/clientcursor.cpp-517-
db/clientcursor.cpp-518-    bool ClientCursor::prepareToYield( YieldData &data ) {
db/clientcursor.cpp-519-        if ( ! _c->supportYields() )
db/clientcursor.cpp-520-            return false;
db/clientcursor.cpp-521-        if ( ! _c->prepareToYield() ) {
db/clientcursor.cpp-522-            return false;   
 

Unfortunately we cannot determine why the pointer becomes invalid, and what conditions are needed for this.



 Comments   
Comment by James Wahlin [ 12/Feb/13 ]

Hi Braam,

At this point it looks like we are stuck without a way to reproduce. I am closing this ticket but please feel free to reopen if you can reproduce or have additional information.

Thanks,
James

Comment by Tad Marshall [ 22/Jan/13 ]

The nice thing that addr2line can do is to show you inlined code that usually isn't visible in the stack trace. I don't know if Solaris is different from Linux in this regard, but on Linux addr2line will show me 20 routines in the call stack from a list of 12 addresses that I pass to addr2line. The 8 "extra" routines are inlined functions that are inferred from the debug information by addr2line.

The most common case we have seen of segfaults is holding pointers to objects that can be freed by other threads. Your segfault seems to be specifically in code that is deliberately touching a record in a memory-mapped database file in order to page-fault it into RAM if it is not already there. The best explanation I can think of is that somehow the database was closed while being accessed, which is supposed to be prevented by locks.

It is possible that there is information in a log file that would show if a file was closed immediately before the segfault, but this is just guessing at a cause. We have not seen reports of crashes like this, so unless we can find a way to reproduce the crash, this may be hard to debug.

Comment by Iwan Aucamp [ 22/Jan/13 ]

If you are unable to do this, given you built mongodb using scons (which will contain debug info), you could try running addr2line which will may give a more complete stack trace including inlined functions.

We do not have debug symbols, but what we normally do is just compare the instructions @ the instruction offset in stack trace to ones in a binary compiled with debugging, here however its not really needed. Stack trace shows that the first instruction in _ZN5mongo6Record5touchEb was being excueted as there is no offset.

The disassembled content of _ZN5mongo6Record5touchEb is :

 
000000000082dec0 <_ZN5mongo6Record5touchEb>:
  82dec0:	8b 07                	mov    (%rdi),%eax
  82dec2:	83 f8 10             	cmp    $0x10,%eax
  82dec5:	7e 25                	jle    82deec <_ZN5mongo6Record5touchEb+0x2c>
  82dec7:	83 e8 10             	sub    $0x10,%eax
  82deca:	48 8d 57 10          	lea    0x10(%rdi),%rdx
  82dece:	48 98                	cltq   
  82ded0:	48 8d 44 07 10       	lea    0x10(%rdi,%rax,1),%rax
  82ded5:	48 39 c2             	cmp    %rax,%rdx
  82ded8:	77 12                	ja     82deec <_ZN5mongo6Record5touchEb+0x2c>
  82deda:	48 8b 05 af 0e 76 00 	mov    0x760eaf(%rip),%rax        # f8ed90 <_GLOBAL_OFFSET_TABLE_+0x1ea8>
  82dee1:	0f be 4f 10          	movsbl 0x10(%rdi),%ecx
  82dee5:	8b 10                	mov    (%rax),%edx
  82dee7:	8d 14 11             	lea    (%rcx,%rdx,1),%edx
  82deea:	89 10                	mov    %edx,(%rax)
  82deec:	f3 c3                	repz retq 
  82deee:	66 90                	xchg   %ax,%ax

Source of Record::touch

    void Record::touch( bool entireRecrd ) {
 
        if ( lengthWithHeaders > HeaderSize ) { // this also makes sure lengthWithHeaders is in memory
            char * addr = data;
            char * end = data + netLength();
            for ( ; addr <= end ; addr += 2048 ) {
                __record_touch_dummy += addr[0];
 
                break; // TODO: remove this, pending SERVER-3711
 
                if ( ! entireRecrd )
                    break;
            }
        }
 
    }

Class definition for Record

    class Record {
    public:
        enum HeaderSizeValue { HeaderSize = 16 };
        int lengthWithHeaders;
        int extentOfs;
        int nextOfs;
        int prevOfs;
 
        /** be careful when referencing this that your write intent was correct */
        char data[4];
 
        int netLength() {
            return lengthWithHeaders - HeaderSize;
        }
        //void setNewLength(int netlen) { lengthWithHeaders = netlen + HeaderSize; }
 
...

Now what its trying to do in 82dec0 is get the value of lengthWithHeaders into eax so it can be compared with HeaderSize. Given this generates a segmentation fault its fair to assume that rdi contained an invalid address - which originated from the caller (ClientCursor::staticYield line 512) as caller sets rdi up with the class base pointer (i.e. this in C++). And since the only way to get from _ZN5mongo12ClientCursor5yieldEiPNS_6RecordE to _ZN5mongo6Record5touchEb is through ClientCursor::staticYield, specifically on line 512 (as Braam indicated), we concluded that the address of rec obtained on line 443 in db/clientcursor.cpp was either invalid when issued or became invalid before _ZN5mongo6Record5touchEb

Also note that this problem occurred at two of our installations, and at each installation more than one time. All occurrences presented the same stack trace.

We are in the process of scheduling and planning an upgrade, quite an onerous task to be honest. If we have not received any feedback regarding this from you by time we have completed planning we will upgrade to 2.2 - and continue to monitor. If we do install 2.2 and face similar issue we will update this issue with the details.

Comment by James Wahlin [ 18/Jan/13 ]

Hi Braam,

We do recommend running with client libraries that are >= to the database version. We can't rule out a bug in an older client version as being the cause for this but there is nothing in the data we have here that would tell us one way or the other.

My suggestion, given feasible, is for you is to upgrade both the client and the database to the latest stable 2.2 version of MongoDB to see whether it addresses.

If you are unable to do this, given you built mongodb using scons (which will contain debug info), you could try running addr2line which will may give a more complete stack trace including inlined functions.

Thanks,
James

Comment by Braam van Heerden [ 14/Jan/13 ]

I am in the process of seeing what the impact of an upgrade on the client's operations will be. However, something else that came to light just now during the investigation is that we are using an older version of the client libraries (1.8.3), and not 2.0.3/2.0.7 as the two server versions. Could this be the cause of the issue, or shouldn't the client version play much of a role in this case?

Comment by Eliot Horowitz (Inactive) [ 10/Jan/13 ]

This doesn't look like anything else we've seen.
Without a repro, nothing jump out at me.

Can you try 2.2.2? There have been various bug fixes, so at least we can rule some things out.

Generated at Thu Feb 08 03:16:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.