Loading...

XML

Word

Printable

JSON

Type: Bug
Resolution: Done
Priority: Major - P3
Fix Version/s: None
Affects Version/s: 2.4.1
Component/s: Stability
Labels:
None
Environment:

Hide
Configuration:
Replica set with 5 members, crashed node was a secondary with delayed replication (1h lag). 3 members (incl. primary) run 2.2.3, this node and another secondary run 2.4.1.

Debian Linux, package from official source:
# dpkg -l | grep mongodb
ii mongodb-10gen 2.4.1 amd64 An object/document-oriented database

# uname -a
Linux tardis 3.2.0-3-amd64 #1 SMP Mon Jul 23 02:45:17 UTC 2012 x86_64 GNU/Linux

~# df
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 314198800 298470360 0 100% /
udev 10240 0 10240 0% /dev
tmpfs 6585076 72 6585004 1% /run
/dev/xvda1 314198800 298470360 0 100% /
tmpfs 5120 0 5120 0% /run/lock
tmpfs 13170140 0 13170140 0% /run/shm

Show
Configuration: Replica set with 5 members, crashed node was a secondary with delayed replication (1h lag). 3 members (incl. primary) run 2.2.3, this node and another secondary run 2.4.1. Debian Linux, package from official source: # dpkg -l | grep mongodb ii mongodb-10gen 2.4.1 amd64 An object/document-oriented database # uname -a Linux tardis 3.2.0-3-amd64 #1 SMP Mon Jul 23 02:45:17 UTC 2012 x86_64 GNU/Linux ~# df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 314198800 298470360 0 100% / udev 10240 0 10240 0% /dev tmpfs 6585076 72 6585004 1% /run /dev/xvda1 314198800 298470360 0 100% / tmpfs 5120 0 5120 0% /run/lock tmpfs 13170140 0 13170140 0% /run/shm

Operating System:
Linux
Steps To Reproduce:

Hide

I have not experienced the disk full situation on any other MongoDB servers, thus I cannot tell if this problem only happens in this very specific situation or in general. Since this is a production deployment I don't want to do this sort of experiment.

Show
I have not experienced the disk full situation on any other MongoDB servers, thus I cannot tell if this problem only happens in this very specific situation or in general. Since this is a production deployment I don't want to do this sort of experiment.
Confidence Status:
None
Work Order:
3

Aha! Reference:
None
Tracking Level:
None
Risk Status:
None
Exec Notes:
None
Goal Name:
None
Goal Link:
None

MongoDB crashed when the disk got full. No suspcious log entries before the one below. dmesg also looks fine. I found several other bug reports relating to full disks, but as far as I can tell they don't apply to the current version.

Sun Apr 14 16:25:33.111 [journal] LogFile::synchronousAppend failed with 24576 bytes unwritten out of 24576 bytes; b=0x7fe7703ee000 errno:28 No space left on device
Sun Apr 14 16:25:33.112 [journal] Fatal Assertion 13515
0xdc7f71 0xd87cf3 0xda410f 0x91f6f5 0x91f922 0x9162eb 0x9166ec 0x916b1b 0xe10879 0x7fecc2687b50 0x7fecc1a2aa7d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdc7f71]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xd87cf3]
/usr/bin/mongod(_ZN5mongo7LogFile17synchronousAppendEPKvm+0x14f) [0xda410f]
/usr/bin/mongod(_ZN5mongo3dur7Journal7journalERKNS0_11JSectHeaderERKNS_14AlignedBuilderE+0x1e5) [0x91f6f5]
/usr/bin/mongod(_ZN5mongo3dur14WRITETOJOURNALENS0_11JSectHeaderERNS_14AlignedBuilderE+0x32) [0x91f922]
/usr/bin/mongod() [0x9162eb]
/usr/bin/mongod() [0x9166ec]
/usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x2eb) [0x916b1b]
/usr/bin/mongod() [0xe10879]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7fecc2687b50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fecc1a2aa7d]

duplicates

SERVER-6924 In RunTime model, when disk full or disk damaged MongoD crashes

Closed

Assignee:: Stennie Steneker (Inactive)

Reporter:: David Gubler

Participants:: David Gubler, Stennie Steneker

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Created:: Apr 15 2013 07:44:23 AM UTC

Updated:: Jan 15 2015 03:28:38 PM UTC

Resolved:: Apr 24 2013 03:42:49 AM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates