[SERVER-23865] DB getting shut down if no space for audit log file. Created: 22/Apr/16  Updated: 09/Jul/18  Resolved: 27/Apr/16

Status: Closed
Project: Core Server
Component/s: Logging, Security
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kingsly J Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive MongoDev_Vm_Conf - Copy.zip    
Participants:
Case:

 Description   

Hi Team,
we are using MongoDB 3.2.5 with two shards and each shard is a replica set and also the config server is a replica set. In mongoDB, if there is no space left for the mount where mongo server logs are located, the server itself is getting shut down. we are seeing this in all the mongoDB versions. Please let us know, whether it is designed like that or any internal issue. Because our assumption is that logging will happen asynchronously without affecting the actual worker progress for any servers. Logging failure may throw a warning but will not shut down the server itself.

Regards,
Jebas



 Comments   
Comment by Ramon Fernandez Marina [ 27/Apr/16 ]

Not being able to write to the log is not a fatal condition, and the server continues to operate. I've verified that on my end.

The stack trace you pasted above shows the following:

mongos(_ZN5mongo6logger9LogDomainINS_5audit10AuditEventEE6appendERKS3_+0xDE) [0x8c681e]

which is a call to add an entry in the audit log. This is not related to regular logging.

I looked at the configuration files provided and I don't see the audit log enabled or configured, so I'm not able to provide any more insight without further information. You may want to send the audit and server logs to different filesystems and verify that the reason for the shutdown is the inability to write to the audit log.

Comment by Kingsly J [ 27/Apr/16 ]

Hi Ramon,
Thanks for your replay. Is this the same case for server logs, because as I mentioned earlier even if in case of no space for server logs, DB is getting shut down.

Regards,
Jebas

Comment by Ramon Fernandez Marina [ 26/Apr/16 ]

jebas.kings@gmail.com, what the error log is trying to say is that you've run out of space for your audit log, so the server is shutting itself down. This behavior is expected: when the audit log is enabled, it's not a good security practice to allow operations that should be audited but can't. Please ensure there's enough space to write to your audit log files.

Regards,
Ramón.

Comment by Kingsly J [ 26/Apr/16 ]

Hi Team,
Is there any clue on this issue?

Comment by Kingsly J [ 25/Apr/16 ]

Hi Ramon,
Even today we faced this issue. PFB the error log of a mongos which got shutdowm as because there is no space left in audit log mount.

2016-04-24T06:48:33.673+0530 F -        [conn9] Got signal: 6 (Aborted).
 
 0xca7792 0xca66c9 0xca6ed2 0x3e93a0f710 0x3e93632925 0x3e93634105 0x8c681e 0x8bdcf1 0x8bdf80 0x7d6f74 0xbdcb92 0xbdd94d 0xbec0cd 0xbdc2e6 0x6a10f5 0xc52a3d 0x3e93a079d1 0x3e936e8b6d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"8A7792","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"8A66C9"},{"b":"400000","o":"8A6ED2"},{"b":"3E93A00000","o":"F710"},{"b":"3E93600000","o":"32925","s":"gsignal"},{"b":"3E93600000","o":"34105","s":"abort"},{"b":"400000","o":"4C681E","s":"_ZN5mongo6logger9LogDomainINS_5audit10AuditEventEE6appendERKS3_"},{"b":"400000","o":"4BDCF1"},{"b":"400000","o":"4BDF80","s":"_ZN5mongo5audit20logCommandAuthzCheckEPNS_11ClientBasicERKSsRKNS_7BSONObjEPNS_7CommandENS_10ErrorCodes5ErrorE"},{"b":"400000","o":"3D6F74","s":"_ZN5mongo7Command19_checkAuthorizationEPS0_PNS_11ClientBasicERKSsRKNS_7BSONObjE"},{"b":"400000","o":"7DCB92","s":"_ZN5mongo7Command22execCommandClientBasicEPNS_16OperationContextEPS0_RNS_11ClientBasicEiPKcRNS_7BSONObjERNS_14BSONObjBuilderE"},{"b":"400000","o":"7DD94D","s":"_ZN5mongo7Command20runAgainstRegisteredEPNS_16OperationContextEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi"},{"b":"400000","o":"7EC0CD","s":"_ZN5mongo8Strategy15clientCommandOpEPNS_16OperationContextERNS_7RequestE"},{"b":"400000","o":"7DC2E6","s":"_ZN5mongo7Request7processEPNS_16OperationContextEi"},{"b":"400000","o":"2A10F5","s":"_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE"},{"b":"400000","o":"852A3D","s":"_ZN5mongo17PortMessageServer17handleIncomingMsgEPv"},{"b":"3E93A00000","o":"79D1"},{"b":"3E93600000","o":"E8B6D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.5", "gitVersion" : "34e65e5383f7ea1726332cb175b73077ec4a1b02", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "2.6.32-431.el6.x86_64", "version" : "#1 SMP Sun Nov 10 22:19:54 EST 2013", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "DB88764A814FFB938585371E5DC9F5A0196E8966" }, { "b" : "7FFFB60FF000", "elfType" : 3, "buildId" : "81A81BE2E44C93640ADEDB62ADC93A47F4A09DD1" }, { "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "9E79FB1A0DDCEC497B5D8AFD86CC378ACF8E354A" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "EFF68B7DE77D081BC4A0CB38FE9DCBC60541BF92" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "B933728EE9CE426EBD94C3DF251C78AC838E0E8D" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "3E71CF35EF7C460CC31BE02541DC2F4DB9CF83A0" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "B26528BF6C0636AC1CAE5AC50BDBC07E60851DF4" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "AFC7448F2F2F6ED4E5BC82B1BD8A7320B84A9D48" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "98B028A725D6E93253F25DF00B794DFAA66A3145" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "A44499D29B114A5366CD72DD4883958495AC1C1D" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "1BB4E10307D6B94223749CFDF2AD14C365972C60" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "871059BD552A77064DCFF4CB5DB5222BD39A31C4" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "57BF668F99B7F5917B8D55FBB645173C9A644575" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "56843351EFB2CE304A7E4BD0754991613E9EC8BD" }, { "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "2B2EAE3EB99811D0DBC0FC5979A0A9BFC7B1B1ED" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "95EBB74C2C0A1E1714344036145A0239FFA4892D" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "D02E7D3149950118009A81997434E28B7D9EC9B2" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "6ADE12F76961F73B33D160AC4D342222E7FC7A65" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "5AFCBEA0D62EE0335714CCBAB7BA808E2A16028C" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8A8734DC37305D8CC2EF8F8C3E5EA03171DB07EC" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "D053BB4FF0C2FC983842F81598813B9B931AD0D1" }, { "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "35A038433616143CE209A74D5B7C006E78E09401" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "BAD5C71361DADF259B6E306A49E6F47F24AEA3DC" }, { "b" : "7FDDFC734000", "path" : "/usr/lib64/sasl2/libsasldb.so", "elfType" : 3, "buildId" : "20A9C9C04FA83BF4A04AA68833D8774B463BF22E" }, { "path" : "/lib64/libdb-4.7.so", "elfType" : 3, "buildId" : "4CF43E1CEBBD1469A97108247CE0680B0141D8D7" }, { "b" : "7FDDFC52C000", "path" : "/usr/lib64/sasl2/libgssapiv2.so", "elfType" : 3, "buildId" : "E57DA6F11C14E01FE883007596479A0AA9F6D0AD" }, { "b" : "7FDDFC327000", "path" : "/usr/lib64/sasl2/liblogin.so", "elfType" : 3, "buildId" : "161930A21E55FFADF2FAB366184D2F398D922E92" }, { "b" : "7FDDFC122000", "path" : "/usr/lib64/sasl2/libanonymous.so", "elfType" : 3, "buildId" : "DEA21FA91ABB7116E372416CBAEC076AECB0B94C" }, { "b" : "7FDDFBF1D000", "path" : "/usr/lib64/sasl2/libplain.so", "elfType" : 3, "buildId" : "22E4D23A991A44E27DC0D22DB519BC1E22DE3B89" } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x32) [0xca7792]
 mongos(+0x8A66C9) [0xca66c9]
 mongos(+0x8A6ED2) [0xca6ed2]
 libpthread.so.0(+0xF710) [0x3e93a0f710]
 libc.so.6(gsignal+0x35) [0x3e93632925]
 libc.so.6(abort+0x175) [0x3e93634105]
 mongos(_ZN5mongo6logger9LogDomainINS_5audit10AuditEventEE6appendERKS3_+0xDE) [0x8c681e]
 mongos(+0x4BDCF1) [0x8bdcf1]
 mongos(_ZN5mongo5audit20logCommandAuthzCheckEPNS_11ClientBasicERKSsRKNS_7BSONObjEPNS_7CommandENS_10ErrorCodes5ErrorE+0xF0) [0x8bdf80]
 mongos(_ZN5mongo7Command19_checkAuthorizationEPS0_PNS_11ClientBasicERKSsRKNS_7BSONObjE+0x204) [0x7d6f74]
 mongos(_ZN5mongo7Command22execCommandClientBasicEPNS_16OperationContextEPS0_RNS_11ClientBasicEiPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x5F2) [0xbdcb92]
 mongos(_ZN5mongo7Command20runAgainstRegisteredEPNS_16OperationContextEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x2ED) [0xbdd94d]
 mongos(_ZN5mongo8Strategy15clientCommandOpEPNS_16OperationContextERNS_7RequestE+0x19D) [0xbec0cd]
 mongos(_ZN5mongo7Request7processEPNS_16OperationContextEi+0x876) [0xbdc2e6]
 mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x65) [0x6a10f5]
 mongos(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x26D) [0xc52a3d]
 libpthread.so.0(+0x79D1) [0x3e93a079d1]
 libc.so.6(clone+0x6D) [0x3e936e8b6d]
-----  END BACKTRACE  -----

Comment by Kingsly J [ 25/Apr/16 ]

Hi Ramon,
We have encountered this issue in all the servers. PFA the config files. We are also seeing the same issue while we enable audit logging. Due to space issue, DB is getting shut down.

Regards,
Jebas

Comment by Ramon Fernandez Marina [ 22/Apr/16 ]

I'm not able to reproduce this behavior (3.2.5, 2-shard cluster, 2-node replica set per shard, CSRS). Can you please send us either the configuration files for all nodes in this cluster or the startup logs for them? Also, which nodes are you seeing affected by this behavior (config servers, data nodes, mongos nodes, all of the above)?

Thanks,
Ramón.

Generated at Thu Feb 08 04:04:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.