[SERVER-4083] MongoDB crashed with corrupted double-linked list error Created: 17/Oct/11  Updated: 29/Feb/12  Resolved: 30/Jan/12

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 2.0.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Evgeny Gridasov Assignee: Mathias Stearn
Resolution: Done Votes: 1
Labels: vmware1
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

RHEL 5.2, running inside VmWare, 2 CPUs, 4Gb of RAM
Linux ***** 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux


Attachments: File mongodb.log    
Operating System: Linux
Participants:

 Description   

An error occurred during an intensive write operation and MongoDB shut down. That was a master server which was configured to replicate to other 3 servers using replication sets.

      • glibc detected *** /apps/mongodb/mongodb-linux-x86_64-2.0.0/bin/mongod: corrupted double-linked list: 0x00002aabd8077450 ***
        ======= Backtrace: =========
        /lib64/libc.so.6[0x31ff8714cc]
        /lib64/libc.so.6(cfree+0x8c)[0x31ff874c5c]
        /apps/mongodb/mongodb-linux-x86_64-2.0.0/bin/mongod(_ZNSt8_Rb_treeISsSt4pairIKSsN5mongo3Top14CollectionDataEESt10_Select1stIS5_ESt4lessISsESaIS5_EE8_M_eraseEPSt13_Rb_tree_nodeIS5_E+0x311)[0x684bd1]
        /apps/mongodb/mongodb-linux-x86_64-2.0.0/bin/mongod(_ZNK5mongo3Top8cloneMapERSt3mapISsNS0_14CollectionDataESt4lessISsESaISt4pairIKSsS2_EEE+0x54)[0x687674]
        /apps/mongodb/mongodb-linux-x86_64-2.0.0/bin/mongod(_ZN5mongo12SnapshotData12takeSnapshotEv+0x92)[0x6817e2]
        /apps/mongodb/mongodb-linux-x86_64-2.0.0/bin/mongod(_ZN5mongo9Snapshots12takeSnapshotEv+0x51)[0x681d01]
        /apps/mongodb/mongodb-linux-x86_64-2.0.0/bin/mongod(_ZN5mongo14SnapshotThread3runEv+0x55)[0x682fd5]
        /apps/mongodb/mongodb-linux-x86_64-2.0.0/bin/mongod(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0xbf)[0x57756f]
        /apps/mongodb/mongodb-linux-x86_64-2.0.0/bin/mongod(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74)[0x579624]
        /apps/mongodb/mongodb-linux-x86_64-2.0.0/bin/mongod(thread_proxy+0x80)[0xa9ea30]
        /lib64/libpthread.so.0[0x32004062f7]
        /lib64/libc.so.6(clone+0x6d)[0x31ff8d1b6d]

Please see attachment with full stack trace.



 Comments   
Comment by Ben Hu [ 22/Jan/12 ]

Yah, my stack trace did not include miniwebserver, I will open a separate ticket https://jira.mongodb.org/browse/SERVER-4741 with my log info.

Comment by Mathias Stearn [ 21/Jan/12 ]

The double-linked list error isn't a real error in most cases. That is, it is fallout from an earlier failure and the inability to recover from a bad state.

@Evgeny: In the log that was posted with this ticket it looks like it was a failure in the built-in web server. Do you know what query caused that? Is there any chance a mongo client connected to the HTTP port rather than the mongo port?

@Ben: If your stack trace didn't include miniwebserver, please open a separate ticket as it is unlikely to be related to this one.

Comment by Ben Hu [ 23/Nov/11 ]

We had same crash with double-linked list error. We are using vmware and our system is Ubuntu 11.04.

It happens in 2.0.1 as well.

Comment by Evgeny Gridasov [ 02/Nov/11 ]

Have you checked the mongodb.log file that I've attached with this issue?
As for your first question - we are using vmware only and are not able to test it on bare metal.
I've also checked with the guys regarding storage subsystem - it is FibreChannel/SAN.

Comment by Mathias Stearn [ 01/Nov/11 ]

Do you still have the logs?

Comment by Mathias Stearn [ 17/Oct/11 ]

Could you attach the full log, not just the last few lines?

Comment by Mathias Stearn [ 17/Oct/11 ]

We are looking into this now. Have you only seen this in vmware or also on bare metal? Also, what does your storage subsystem look like (local disk, FibreChannel, iSCSI, NFS, etc)?

Generated at Thu Feb 08 03:04:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.