[SERVER-14862] mongod segfaults on startup with insufficient space for journal files Created: 12/Aug/14  Updated: 06/Dec/22  Resolved: 16/Nov/16

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Michael O'Brien Assignee: Backlog - Storage Execution Team
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Assigned Teams:
Storage Execution
Operating System: ALL
Participants:
Linked BF Score: 0

 Description   

on master at 2a90d1e2

[root@ip-10-186-161-200 bin]#  ./mongod --port 27999 --dbpath /tmp/data --setParameter enableTestCommands=1 --httpinterface --nopreallocj --sslMode requireSSL --sslPEMKeyFile ../../jstests/libs/server.pem --sslCAFile ../../jstests/libs/ca.pem --sslWeakCertificateValidation
2014-08-11T18:44:01.706-0400 I          [initandlisten] MongoDB starting : pid=1519 port=27999 dbpath=/tmp/data 64-bit host=ip-10-186-161-200
2014-08-11T18:44:01.706-0400 I          [initandlisten]
2014-08-11T18:44:01.706-0400 I          [initandlisten] ** NOTE: This is a development version (2.7.5-pre-) of MongoDB.
2014-08-11T18:44:01.706-0400 I          [initandlisten] **       Not recommended for production.
2014-08-11T18:44:01.707-0400 I          [initandlisten]
2014-08-11T18:44:01.707-0400 I          [initandlisten] db version v2.7.5-pre-
2014-08-11T18:44:01.707-0400 I          [initandlisten] git version: 2a90d1e20230800f8dc5a01ea066873354fcd938 modules: enterprise
2014-08-11T18:44:01.707-0400 I          [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2014-08-11T18:44:01.707-0400 I          [initandlisten] build info: Linux ip-10-180-225-165 2.6.32-220.34.1.el6.x86_64 #1 SMP Thu Mar 28 19:47:49 EDT 2013 x86_64 BOOST_LIB_VERSION=1_49
2014-08-11T18:44:01.707-0400 I          [initandlisten] allocator: tcmalloc
2014-08-11T18:44:01.707-0400 I          [initandlisten] options: { net: { http: { enabled: true }, port: 27999, ssl: { CAFile: "../../jstests/libs/ca.pem", PEMKeyFile: "../../jstests/libs/server.pem", mode: "requireSSL", weakCertificateValidation: true } }, nopreallocj: true, setParameter: { enableTestCommands: "1" }, storage: { dbPath: "/tmp/data" } }
2014-08-11T18:44:01.758-0400 I          [initandlisten] journal dir=/tmp/data/journal
2014-08-11T18:44:01.758-0400 I          [initandlisten] recover : no journal files present, no recovery needed
2014-08-11T18:44:01.758-0400 I          [initandlisten]
2014-08-11T18:44:01.758-0400 E          [initandlisten] Insufficient free space for journal files
2014-08-11T18:44:01.758-0400 I          [initandlisten] Please make at least 3379MB available in /tmp/data/journal or use --smallfiles
2014-08-11T18:44:01.758-0400 I          [initandlisten]
2014-08-11T18:44:01.758-0400 I          [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
2014-08-11T18:44:01.758-0400 F          [initandlisten] Invalid access at address: 0x3ce2e20
2014-08-11T18:44:01.770-0400 F          [initandlisten] Got signal: 11 (Segmentation fault).
 
 0xfc40ae 0xfc3b88 0xfc3cfd 0x7f2d460aa4a0 0x3ce2e20
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"BC40AE"},{"b":"400000","o":"BC3B88"},{"b":"400000","o":"BC3CFD"},{"b":"7F2D4609B000","o":"F4A0"},{"b":"0","o":"3CE2E20"}],"processInfo":{ "mongodbVersion" : "2.7.5-pre-", "gitVersion" : "2a90d1e20230800f8dc5a01ea066873354fcd938 modules: enterprise", "uname" : { "sysname" : "Linux", "release" : "2.6.32-220.34.1.el6.x86_64", "version" : "#1 SMP Thu Mar 28 19:47:49 EDT 2013", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "55C5A8274CB734A03690871DEB58CB710691B77E" }, { "b" : "7FFF8A6FF000", "elfType" : 3, "buildId" : "4339A739E7E40C7BC6FA45243AF6390538DA8EA0" }, { "b" : "7F2D4705D000", "path" : "/usr/lib64/libnetsnmpagent.so.20", "elfType" : 3, "buildId" : "BBC417ED3E99E31E5AA93EA913B9649F3AF46352" }, { "b" : "7F2D46E37000", "path" : "/usr/lib64/libnetsnmphelpers.so.20", "elfType" : 3, "buildId" : "C040838C539ECC7102C3101E1456BCEAB9AB4D28" }, { "b" : "7F2D469EF000", "path" : "/usr/lib64/libnetsnmpmibs.so.20", "elfType" : 3, "buildId" : "0EACCEF16806C09B612E51CFD1845DA86B14B9FA" }, { "b" : "7F2D46715000", "path" : "/usr/lib64/libnetsnmp.so.20", "elfType" : 3, "buildId" : "062251BA67BD95D032162CC73FEB67449E774F08" }, { "b" : "7F2D464FB000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "9E79FB1A0DDCEC497B5D8AFD86CC378ACF8E354A" }, { "b" : "7F2D462B7000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "50487A3480233636C29DBCAD5DE65421808948AB" }, { "b" : "7F2D4609B000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "9CACC898EF34F124A5ECADBD5AE877FF514107CF" }, { "b" : "7F2D45E2F000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "318EAB33420B000D542F09B91B716BACAB1AD546" }, { "b" : "7F2D45A4F000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "70430ED9D4C4559ECB2CCFD7A40736B191FA4A85" }, { "b" : "7F2D45847000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "656BBCC163EB4D2BC5F1404C77B58BF719AF643B" }, { "b" : "7F2D45643000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "7F9E85AB9A7C4416A6A23BF0D525A4BAF062D75D" }, { "b" : "7F2D4533D000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "ED99110E629209C5CA6C0ED704F2C5CE3171513A" }, { "b" : "7F2D450B9000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "72F9090A3145CEF48E0A4202631709ADCE5D1724" }, { "b" : "7F2D44EA3000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "A44499D29B114A5366CD72DD4883958495AC1C1D" }, { "b" : "7F2D44B13000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "E731C64000993D1FD1B443E6D5D6972D149440E8" }, { "b" : "7F2D472A8000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "CEB82E745B0AB8BB7EA28C068327BE1FB068C923" }, { "b" : "7F2D44908000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "8C0C7CAB7F028E4592A8581EB2122FBECAB26B97" }, { "b" : "7F2D4459D000", "path" : "/usr/lib64/perl5/CORE/libperl.so", "elfType" : 3, "buildId" : "0A8E7D74369C1AF1F7C33B8DF8387DE5013898A4" }, { "b" : "7F2D44383000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "EF24A6F82750216C1B473981FCE7CF4CEFB3E789" }, { "b" : "7F2D4416A000", "path" : "/lib64/libnsl.so.1", "elfType" : 3, "buildId" : "B6823FBC4D3E722E38D52690270F8469B2ED3F9A" }, { "b" : "7F2D43F33000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "D7EBBA3CE5425B6EE5B780ACE94D565585C1FDAE" }, { "b" : "7F2D43D30000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "49C9F4BAD47BE86354C56E0749D64A064A699D7B" }, { "b" : "7F2D43AC5000", "path" : "/usr/lib64/librpm.so.1", "elfType" : 3, "buildId" : "61F043213BC6BEEEDA33E5AC5688271D661E2BC4" }, { "b" : "7F2D43896000", "path" : "/usr/lib64/librpmio.so.1", "elfType" : 3, "buildId" : "161B9357C432A95D59C1104FFE611CE175DA0AD4" }, { "b" : "7F2D4368D000", "path" : "/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "E7B49911F1136073DD7DC58E8118CD9A4FBE2A19" }, { "b" : "7F2D43477000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "D053BB4FF0C2FC983842F81598813B9B931AD0D1" }, { "b" : "7F2D43267000", "path" : "/usr/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "6855E5BF5B3634C15F01B1043BD892D727EE3C08" }, { "b" : "7F2D42F81000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "D9A44621797C990C639FF2D5AA452AB559C277DE" }, { "b" : "7F2D42D55000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "D180B6297A9A302693053BD753A85D04A88DE811" }, { "b" : "7F2D42B51000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "6A22EDFF4D4F04A57573E3D1536B6B4963159CD5" }, { "b" : "7F2D42946000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "FF9705F60A59F28CA0FC50720A4F18FA9A889BD6" }, { "b" : "7F2D42743000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8A8734DC37305D8CC2EF8F8C3E5EA03171DB07EC" }, { "b" : "7F2D424E1000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "4D7A7266F3E891524FB3434B559343422B69AF09" }, { "b" : "7F2D421A6000", "path" : "/usr/lib64/libnss3.so", "elfType" : 3, "buildId" : "1CD623312433096C50A2AF396DB1E9CAAB2CB263" }, { "b" : "7F2D41F95000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "1250B1D041DD7552F0C870BB188DC3A34DF2651D" }, { "b" : "7F2D41D80000", "path" : "/usr/lib64/libelf.so.1", "elfType" : 3, "buildId" : "17B1D7503CCD3C5DBC7284A7ADFDFFD84B0B3329" }, { "b" : "7F2D41B5F000", "path" : "/usr/lib64/liblzma.so.0", "elfType" : 3, "buildId" : "6FF9BAEEEE9DDEEF2DFA5CBD36147A75891C0AD4" }, { "b" : "7F2D41932000", "path" : "/usr/lib64/liblua-5.1.so", "elfType" : 3, "buildId" : "6BDB4E1990D6EBA12A5C8D39A7650DB8798BF568" }, { "b" : "7F2D41713000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "BAD5C71361DADF259B6E306A49E6F47F24AEA3DC" }, { "b" : "7F2D4150F000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "A436538388F1F25113FDA834CA2EED524EFA17D6" }, { "b" : "7F2D41307000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "26CC708AC7C0FC1797A2340C024F0ADD0CE054D8" }, { "b" : "7F2D40F93000", "path" : "/lib64/libdb-4.7.so", "elfType" : 3, "buildId" : "437CA0AB593A7383FF1A1700D14AF4998FE93CF3" }, { "b" : "7F2D40D6D000", "path" : "/usr/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "2875A55DBF9490291B2A2759F8FA8054CF8DCE5D" }, { "b" : "7F2D40B68000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "3B5ACC6E00B8031105BA66B0686B2C4DEAFA5B64" }, { "b" : "7F2D40964000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "9214F3DCCA7ADA68137084707DA78FB2501BC1A3" }, { "b" : "7F2D40727000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "DB47BCBF7F973DBD31101FA3AA526A18FABB354F" }, { "b" : "7F2D40522000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "8EF0683858704EF173AB11B1E27076F37F82B7B6" }, { "b" : "7F2D3A484000", "path" : "/usr/lib64/sasl2/libdigestmd5.so", "elfType" : 3, "buildId" : "59CF90146D760E586F0360B194141D1AAE15427F" }, { "b" : "7F2D3A27E000", "path" : "/usr/lib64/sasl2/libcrammd5.so", "elfType" : 3, "buildId" : "81EBAAB9FB79C6FBF16113946E8E51F59D22DCE6" }, { "b" : "7F2D3A079000", "path" : "/usr/lib64/sasl2/libplain.so", "elfType" : 3, "buildId" : "22E4D23A991A44E27DC0D22DB519BC1E22DE3B89" }, { "b" : "7F2D39E73000", "path" : "/usr/lib64/sasl2/libsasldb.so", "elfType" : 3, "buildId" : "20A9C9C04FA83BF4A04AA68833D8774B463BF22E" }, { "b" : "7F2D39C6B000", "path" : "/usr/lib64/sasl2/libgssapiv2.so", "elfType" : 3, "buildId" : "E57DA6F11C14E01FE883007596479A0AA9F6D0AD" }, { "b" : "7F2D39A66000", "path" : "/usr/lib64/sasl2/libanonymous.so", "elfType" : 3, "buildId" : "DEA21FA91ABB7116E372416CBAEC076AECB0B94C" }, { "b" : "7F2D39861000", "path" : "/usr/lib64/sasl2/liblogin.so", "elfType" : 3, "buildId" : "161930A21E55FFADF2FAB366184D2F398D922E92" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x2E) [0xfc40ae]
 mongod(+0xBC3B88) [0xfc3b88]
 mongod(+0xBC3CFD) [0xfc3cfd]
 libpthread.so.0(+0xF4A0) [0x7f2d460aa4a0]
 ??? [0x3ce2e20]
-----  END BACKTRACE  -----



 Comments   
Comment by Eric Milkie [ 12/Aug/14 ]

I wonder if the recent changes to shutdown changed the behavior here? This is a case of early shutdown where things might not be set up yet. In particular, we might be trying to destruct things that aren't yet fully constructed.

Generated at Thu Feb 08 03:36:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.