[SERVER-20103] F - [conn366] Invalid access at address: 0x177 Created: 25/Aug/15  Updated: 29/Jan/16  Resolved: 26/Sep/15

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 3.0.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Thuong Nguyen Assignee: Ramon Fernandez Marina
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

MongoDB 3.0.4
Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u2 x86_64 GNU/Linux


Attachments: Zip Archive mongod.log.zip     Zip Archive mongodb.log.zip    
Operating System: ALL
Participants:

 Description   

Our Mongodb server (3.0.4) is running on GCE, with 2 replicas and one arbiter. I set it up 3 days ago.
Now it randomly stopped with these errors in /var/log/messages, and I need to start it everytime:

[ 721.309875] mongod[3178] trap int3 ip:1205c31 sp:7f1cb1fe7668 error:0
[ 1875.272667] mongod[5433] trap int3 ip:1205c31 sp:7f973b8f51b8 error:0
[ 2725.869045] mongod[5988] trap int3 ip:1205c31 sp:7ee22fd49148 error:0
[12406.247920] mongod[11255] trap int3 ip:1205c31 sp:7f05acc459c8 error:0
[13922.031178] mongod[13517] trap int3 ip:1205c31 sp:7f1eebef8958 error:0



 Comments   
Comment by Ramon Fernandez Marina [ 29/Jan/16 ]

For the record, SERVER-22334 shows how a missing "var" keyword in the JS code could trigger this issue. MongoDB 3.2 uses SpiderMonkey as the JavaScript engine an it handles this case better than V8.

Comment by Ramon Fernandez Marina [ 27/Aug/15 ]

Thanks for uploading the logs thuongnvbk. It seems that this is triggered by a $where query. Any chance you can provide more details about the data in the collection and the specific $where query that's run against that collection? I'm reading about GCE to try to reproduce on my end.

Comment by Thuong Nguyen [ 27/Aug/15 ]

​​Please take a look at these files.

Thanks,

mongod.log.zip
<https://drive.google.com/a/brodev.com/file/d/0B9YvArvRF6N_RGxpNFNlT1ZheFk/view?usp=drive_web>
​​
mongodb.log.zip
<https://drive.google.com/a/brodev.com/file/d/0B9YvArvRF6N_cURLbW8wbkNqM2c/view?usp=drive_web>

On Wed, Aug 26, 2015 at 2:19 AM, Ramon Fernandez (JIRA) <jira@mongodb.org>

Comment by Thuong Nguyen [ 26/Aug/15 ]

Hi Ramon,
My colleague used click-to-deploy feature of Google Cloud Platform to set up Mongo system. And this feature is just a beta version, I'm not sure it's gonna stable enough for production environment. Maybe they used a custom build (customized by them). This is how they get it done:

https://cloud.google.com/solutions/mongodb/deployment-details

Yesterday, I added new node to replicaSet (same kernel with previous two servers, mongodb 3.0.6 install with package downloaded from mongo website), and it's working fine, nothing weird happened.

Comment by Ramon Fernandez Marina [ 25/Aug/15 ]

thuongnvbk, the stack trace you posted is a crash in V8, from which mongod unfortunately can't recover. We've seen this before with custom builds, when system limits are too low, and on system running software like SELinux or grsecurity.

I'm not familiar with the GCE environment; are there any further details you can gather about the environment that may help understand what kind of system limits does the environment impose on mongod? Also, can you please upload the full logs for one of the affected servers since the last restart until the crash happens?

Thanks,
Ramón.

Comment by Thuong Nguyen [ 25/Aug/15 ]

2015-08-24T23:27:53.461+0000 
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B7A0F9"},{"b":"400000","o":"B799C2"},{"b":"400000","o":"B79D1E"},{"b":"7F60AFF4
A000","o":"F0A0"},{"b":"400000","o":"D08020"},{"b":"400000","o":"D17B6B"},{"b":"400000","o":"D17C20"},{"b":"4000
00","o":"CEB901"},{"b":"400000","o":"CA91DD"},{"b":"400000","o":"D0963F"},{"b":"400000","o":"D096BB"},{"b":"4000
00","o":"D1676F"},{"b":"400000","o":"D16C8E"},{"b":"400000","o":"D7CD5A"},{"b":"400000","o":"E84121"},{"b":"4000
00","o":"AE315A"},{"b":"400000","o":"AE8E31"},{"b":"400000","o":"AD7934"},{"b":"400000","o":"704ED3"},{"b":"4000
00","o":"70527B"},{"b":"400000","o":"6F6B73"},{"b":"400000","o":"7B1EB0"},{"b":"400000","o":"7B21A8"},{"b":"4000
00","o":"7B2239"},{"b":"400000","o":"7B23C7"},{"b":"400000","o":"7C1DF6"},{"b":"400000","o":"5762CD"},{"b":"4000
00","o":"5F40E4"},{"b":"400000","o":"5F506D"},{"b":"400000","o":"5F5D7B"},{"b":"400000","o":"7BABCA"},{"b":"4000
00","o":"6D0F00"},{"b":"400000","o":"42B88D"},{"b":"400000","o":"B2D22B"},{"b":"7F60AFF4A000","o":"6B50"},{"b":"
7F60AE9BB000","o":"DB95D"}],"processInfo":{ "mongodbVersion" : "3.0.4", "gitVersion" : "0481c958daeb2969800511e7
475dc66986fa9ed5", "uname" : { "sysname" : "Linux", "release" : "3.2.0-4-amd64", "version" : "#1 SMP Debian 3.2.
68-1+deb7u2", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "EB7A0F7D9F202DB3
ADD7637D20509D9DE82AC5A1" }, { "b" : "7FFC014E7000", "elfType" : 3, "buildId" : "B302CDC946F4E5B4EB434CD694F0E61
E9277915C" }, { "b" : "7F60AFF4A000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId"
 : "FEF281218797AD6AE726DD5FCEDECADD9E9F51DC" }, { "b" : "7F60AFCE9000", "path" : "/usr/lib/x86_64-linux-gnu/lib
ssl.so.1.0.0", "elfType" : 3, "buildId" : "ED2B7FC51D2E7ADD0D4F1A3667E2B6ED7257663F" }, { "b" : "7F60AF8F1000", 
"path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "7C7F328E0814F339D251A8F8C94
59E50978EC254" }, { "b" : "7F60AF6E9000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" 
: "F58D5DE3E7A2989E915422BA4203FE53DBA449A0" }, { "b" : "7F60AF4E5000", "path" : "/lib/x86_64-linux-gnu/libdl.so
.2", "elfType" : 3, "buildId" : "5D1CA3A3D93ED5B6C6462FFA03E787FDBE4013A3" }, { "b" : "7F60AF1DE000", "path" : "
/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "8711429397A5AF8B6269B867D830EDF6E0225B8D"
 }, { "b" : "7F60AEF5C000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "7F58D6664571
941C86B2D969701A572AD4D7BF1D" }, { "b" : "7F60AED46000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfTyp
e" : 3, "buildId" : "F980B1188708F8D8B5C35D185444AF4CB939AA1E" }, { "b" : "7F60AE9BB000", "path" : "/lib/x86_64-
linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "A745EBA2C16BA80AE1EF1A7A7B70740C2CF1B363" }, { "b" : "7F60B016
6000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9B23F2A44CC8CA6175CBD8D64584B1C7EA5FD
18C" }, { "b" : "7F60AE7A4000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "1EFEB71F
D4999C2307570D673A724EA4E1D85267" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf7a0f9]
 mongod(+0xB799C2) [0xf799c2]
 mongod(+0xB79D1E) [0xf79d1e]
 libpthread.so.0(+0xF0A0) [0x7f60aff590a0]
 mongod(_ZN2v88internal4Heap27ClearJSFunctionResultCachesEv+0x30) [0x1108020]
 mongod(_ZN2v88internal4Heap25GarbageCollectionPrologueEv+0x1B) [0x1117b6b]
 mongod(_ZN2v88internal4Heap14CollectGarbageENS0_15AllocationSpaceENS0_16GarbageCollectorEPKcS5_+0x90) [0x1117c2
0]
 mongod(_ZN2v88internal7Factory7NewCodeERKNS0_8CodeDescENS0_4Code5FlagsENS0_6HandleINS0_6ObjectEEEb+0xD1) [0x10e
b901]
 mongod(_ZN2v88internal8CodeStub7GetCodeEv+0xDD) [0x10a91dd]
 mongod(_ZN2v88internal4Heap17CreateJSEntryStubEv+0x1F) [0x110963f]
 mongod(_ZN2v88internal4Heap16CreateFixedStubsEv+0x3B) [0x11096bb]
 mongod(_ZN2v88internal4Heap20CreateInitialObjectsEv+0x39F) [0x111676f]
 mongod(_ZN2v88internal4Heap5SetUpEb+0x3EE) [0x1116c8e]
 mongod(_ZN2v88internal7Isolate4InitEPNS0_12DeserializerE+0x54A) [0x117cd5a]
 mongod(_ZN2v86LockerC1EPNS_7IsolateE+0x61) [0x1284121]
 mongod(_ZN5mongo7V8ScopeC2EPNS_14V8ScriptEngineE+0x35A) [0xee315a]
 mongod(_ZN5mongo14V8ScriptEngine11createScopeEv+0x21) [0xee8e31]
 mongod(_ZN5mongo12ScriptEngine14getPooledScopeEPNS_16OperationContextERKSsS4_+0xB04) [0xed7934]
 mongod(_ZN5mongo20WhereMatchExpression4initERKNS_10StringDataES3_RKNS_7BSONObjE+0x163) [0xb04ed3]
 mongod(_ZNK5mongo17WhereCallbackReal10parseWhereERKNS_11BSONElementE+0x16B) [0xb0527b]
 mongod(_ZN5mongo21MatchExpressionParser6_parseERKNS_7BSONObjEi+0xD63) [0xaf6b73]
 mongod(_ZN5mongo14CanonicalQuery12canonicalizeERKSsRKNS_7BSONObjES5_S5_xxS5_S5_S5_bbPPS0_RKNS_21MatchExpression
Parser13WhereCallbackE+0xF0) [0xbb1eb0]
 mongod(_ZN5mongo14CanonicalQuery12canonicalizeERKSsRKNS_7BSONObjES5_S5_xxS5_PPS0_RKNS_21MatchExpressionParser13
WhereCallbackE+0x68) [0xbb21a8]
 mongod(_ZN5mongo14CanonicalQuery12canonicalizeERKSsRKNS_7BSONObjES5_S5_xxPPS0_RKNS_21MatchExpressionParser13Whe
reCallbackE+0x49) [0xbb2239]
 mongod(_ZN5mongo14CanonicalQuery12canonicalizeERKSsRKNS_7BSONObjES5_S5_PPS0_RKNS_21MatchExpressionParser13Where
CallbackE+0x27) [0xbb23c7]
 mongod(_ZN5mongo19getExecutorDistinctEPNS_16OperationContextEPNS_10CollectionERKNS_7BSONObjERKSsNS_12PlanExecut
or11YieldPolicyEPPS9_+0x366) [0xbc1df6]
 mongod(_ZN5mongo15DistinctCommand3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x38D)
 [0x9762cd]
 mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+
0x34) [0x9f40e4]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xC1D)
 [0x9f506d]
 mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEE
ERNS_14BSONObjBuilderEbi+0x28B) [0x9f5d7b]
 mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5Cu
rOpES3_+0x77A) [0xbbabca]
mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xB
10) [0xad0f00]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xDD) [0x82
b88d]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x34B) [0xf2d22b]
 libpthread.so.0(+0x6B50) [0x7f60aff50b50]
 libc.so.6(clone+0x6D) [0x7f60aea9695d]
-----  END BACKTRACE  -----

Generated at Thu Feb 08 03:53:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.