[SERVER-25701] Invalid query triggers stack trace Created: 19/Aug/16  Updated: 23/Aug/16  Resolved: 23/Aug/16

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.0.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Charles Page [X] Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 14.04
16CPU, 104GB Ram


Attachments: Text File mongod.log    
Issue Links:
Related
related to SERVER-18167 handle BSON validation in OP_COMMAND/... Closed
Operating System: ALL
Steps To Reproduce:

Unknown

Just before the crash MongoDB logged:
Assertion: 10307:Client Error: bad object in message: bson length doesn't match what we found in object with unknown _id

Participants:

 Description   
Original summary

Crash with minor service disruption

Original Description

I'm not quite sure what caused the issue, it occurred in a production environment, recovered by itself. No data loss.

0xf51949 0xef1671 0xed693f 0xed69ec 0x9ee9cd 0x8732a7 0xab7107 0x80e88d 0xf04a6b 0x7ff0bb71a182 0x7ff0ba1e147d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B51949"},{"b":"400000","o":"AF1671"},{"b":"400000","o":"AD693F"},{"b":"400000","o":"AD69EC"},{"b":"400000","o":"5EE9CD"},{"b":"400000","o":"4732A7"},{"b":"400000","o":"6B7107"},{"b":"400000","o":"40E88D"},{"b":"400000","o":"B04A6B"},{"b":"7FF0BB712000","o":"8182"},{"b":"7FF0BA0E7000","o":"FA47D"}],"processInfo":{ "mongodbVersion" : "3.0.3", "gitVersion" : "b40106b36eecd1b4407eb1ad1af6bc60593c6105", "uname" : { "sysname" : "Linux", "release" : "3.16.0-38-generic", "version" : "#52~14.04.1-Ubuntu SMP Fri May 8 09:43:57 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "F56F80CB96B4DBFC070BEB0ADAC7D6B274BFC6B1" }, { "b" : "7FFE7F3F6000", "elfType" : 3, "buildId" : "EFA4172E98FE6C90166BC69E025FF58D9E646CC1" }, { "b" : "7FF0BB712000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7FF0BB4B3000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "A20EFFEC993A8441FA17F2079F923CBD04079E19" }, { "b" : "7FF0BB0D8000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F000D29917E9B6E94A35A8F02E5C62846E5916BC" }, { "b" : "7FF0BAED0000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7FF0BACCC000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7FF0BA9C8000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "76190E922AF7457D078F75C9B15FA184E83EB506" }, { "b" : "7FF0BA6C2000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7FF0BA4AC000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" : "7FF0BA0E7000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7FF0BB930000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf51949]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xef1671]
 mongod(_ZN5mongo11msgassertedEiPKc+0xAF) [0xed693f]
 mongod(+0xAD69EC) [0xed69ec]
 mongod(_ZN5mongo9DbMessage9nextJsObjEv+0x25D) [0x9ee9cd]
 mongod(_ZN5mongo12QueryMessageC2ERNS_9DbMessageE+0x57) [0x8732a7]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x967) [0xab7107]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xDD) [0x80e88d]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x34B) [0xf04a6b]
 libpthread.so.0(+0x8182) [0x7ff0bb71a182]
 libc.so.6(clone+0x6D) [0x7ff0ba1e147d]
-----  END BACKTRACE  -----



 Comments   
Comment by Kelsey Schubert [ 23/Aug/16 ]

Hi Charles_CC,

Thank for uploading the surrounding context. I see that mongod received an invalid query and produced a stack trace. Consequently, it closed the client connection and continued normal operation. I've updated the summary of this ticket to better reflect what transpired.

2016-08-19T05:50:54.386+0000 I NETWORK  [conn106381] AssertionException handling request, closing client connection: 10307 Client Error: bad object in message: bson length doesn't match what we found in object with unknown _id

To clarify, a stack trace resulting from a failed check does not indicate that mongod's availability was compromised. In MongoDB 3.1.5, this check happens earlier and a comprehensive error message is printed instead of the stack trace (SERVER-18167).

For MongoDB-related support discussion please post on the mongodb-users group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group.

Kind regards,
Thomas

Comment by Charles Page [X] [ 19/Aug/16 ]

I've attached the log file, the crash occurs at line 9168.

Comment by Kelsey Schubert [ 19/Aug/16 ]

Hi Charles_CC,

Would you please attach the complete logs including the time before and after this issue?

Thank you,
Thomas

Generated at Thu Feb 08 04:09:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.