[SERVER-63990] Segmentation fault on primary Created: 25/Feb/22  Updated: 10/Jun/22  Resolved: 10/Mar/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Tejas Jadhav Assignee: Edwin Zhou
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-63991 Segmentation fault on primary Closed
is duplicated by WT-4032 parent pages can be evicted while bei... Closed
Operating System: ALL
Participants:

 Description   

We have a MongoDB cluster setup with following configuration

  • 1 primary
  • 2 secondaries (1 connected to clients but not used for reads, other one not connected to clients at all)
  • 1 arbiter

Throughput pattern is write heavy with comparatively lesser number of reads.

A few days back, our primary crashed out of nowhere. It emitted the following traceback,

2022-02-21T02:45:58.828+0000 F -        [thread12203756] Invalid access at address: 0xe8
2022-02-21T02:45:58.863+0000 F -        [thread12203756] Got signal: 11 (Segmentation fault).
 
 0x562f500b44f1 0x562f500b3709 0x562f500b3d76 0x7fca6ebb9390 0x562f4e9abf0f 0x562f4e9aee8b 0x562f4e939294 0x562f4e93a2e2 0x562f4e9332b3 0x562f4e933647 0x562f4e935a22 0x562f4e97ec79 0x7fca6ebaf6ba 0x7fca6e8e551d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"562F4DEC1000","o":"21F34F1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"562F4DEC1000","o":"21F2709"},{"b":"562F4DEC1000","o":"21F2D76"},{"b":"7FCA6EBA8000","o":"11390"},{"b":"562F4DEC1000","o":"AEAF0F"},{"b":"562F4DEC1000","o":"AEDE8B","s":"__wt_split_reverse"},{"b":"562F4DEC1000","o":"A78294"},{"b":"562F4DEC1000","o":"A792E2","s":"__wt_evict"},{"b":"562F4DEC1000","o":"A722B3"},{"b":"562F4DEC1000","o":"A72647"},{"b":"562F4DEC1000","o":"A74A22","s":"__wt_evict_thread_run"},{"b":"562F4DEC1000","o":"ABDC79"},{"b":"7FCA6EBA8000","o":"76BA"},{"b":"7FCA6E7DE000","o":"10751D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.2", "gitVersion" : "489d177dbd0f0420a8ca04d39fd78d0a2c539420", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-1109-aws", "version" : "#120-Ubuntu SMP Fri Jun 5 01:26:57 UTC 2020", "machine" : "x86_64" }, "somap" : [ { "b" : "562F4DEC1000", "elfType" : 3, "buildId" : "90F4CC751C09ABD90756CE2480F0217355B846B5" }, { "b" : "7FFC77559000", "elfType" : 3, "buildId" : "E84E25DBB968253D84247A3C45F8B83709552AC9" }, { "b" : "7FCA6FD9D000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "C8ACB5535F4B7B9880A62B4B38EBE924668F6446" }, { "b" : "7FCA6FB35000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "069C06FA5B5598CB6B27089BD4BEC60469AC85F7" }, { "b" : "7FCA6F6F0000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "3662558D7453C25E2ABE7E6B49D9256F1B8BB1A7" }, { "b" : "7FCA6F4E8000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "BB404D52807964CCC7F0815BC2666688A74B958F" }, { "b" : "7FCA6F2E4000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "BBA6A2E958188C44B9BDA990278EBE8868B85379" }, { "b" : "7FCA6EFDB000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "E178A25E6DB28598588C03D898E44FD79BD16E4D" }, { "b" : "7FCA6EDC5000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FCA6EBA8000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "C557B8146E8079AF46310B549DE6912D1FC4EA86" }, { "b" : "7FCA6E7DE000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30773BE8CF5BFED9D910C8473DD44EAAB2E705AB" }, { "b" : "7FCA6FFB8000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "98D7BC4313D0D8D5E127E06ACF2319829C5CE61D" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x562f500b44f1]
 mongod(+0x21F2709) [0x562f500b3709]
 mongod(+0x21F2D76) [0x562f500b3d76]
 libpthread.so.0(+0x11390) [0x7fca6ebb9390]
 mongod(+0xAEAF0F) [0x562f4e9abf0f]
 mongod(__wt_split_reverse+0x11B) [0x562f4e9aee8b]
 mongod(+0xA78294) [0x562f4e939294]
 mongod(__wt_evict+0xFE2) [0x562f4e93a2e2]
 mongod(+0xA722B3) [0x562f4e9332b3]
 mongod(+0xA72647) [0x562f4e933647]
 mongod(__wt_evict_thread_run+0x72) [0x562f4e935a22]
 mongod(+0xABDC79) [0x562f4e97ec79]
 libpthread.so.0(+0x76BA) [0x7fca6ebaf6ba]
 libc.so.6(clone+0x6D) [0x7fca6e8e551d]
-----  END BACKTRACE  -----



 Comments   
Comment by Edwin Zhou [ 10/Mar/22 ]

Hi tejas.jadhav@gojek.com,

We haven’t heard back from you for some time, so I’m going to close this ticket. If you find this is still an issue for you after upgrading to a supported version and working with our developer community forums, then we'd want to investigate it as a possible bug here in the SERVER project.

Best,
Edwin

Comment by Dmitry Agranat [ 27/Feb/22 ]

Hi tejas.jadhav@gojek.com, As MongoDB 3.6 has reached EOL on April 2021, we'd like to encourage you to start by asking our community for help by posting on the MongoDB Developer Community Forums.

If the discussion there leads you to suspect a bug in the MongoDB supported versions, then we'd want to investigate it as a possible bug here in the SERVER project.

Regarding the mentioned segmentation fault, this might be related to WT-4032, and since you are currently on MongoDB 3.6.2, you are impacted by this issue.

Regards,
Dima

Generated at Thu Feb 08 05:59:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.