<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:53:17 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-83833] shard crash causes WiredTiger to fail replaying the oplog in an infinite loop</title>
                <link>https://jira.mongodb.org/browse/SERVER-83833</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Inserting record with thousands of values in a field that is indexed caused system to hang.&lt;/p&gt;

&lt;p&gt;Database has approximately 60 billion records.&#160; I had inserted a record with ~100,000 values for one key and the mongod process essentially hung.&#160; It was processing the request for 3 days and each shard in the replicaset would not accept new connections.&#160; Reads meanwhile still worked just fine from open connections.&lt;/p&gt;

&lt;p&gt;The application I wrote can handle failures on write/read gracefully, so I restarted the primary shard, and neither secondary (PSS architecture) (26 shards total) shard promoted.&#160; The newly restarted PRIMARY couldn&apos;t connect to any secondary.&lt;/p&gt;

&lt;p&gt;Restarted all the shards in the replicaset and none of them will come back up.&#160; They keep replaying the oplog trying to write this same transaction but keep erroring, then try again, etc and they try to connect outbound, but none of them are listening because of processing the oplog.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Ubuntu 22.04 - 5.15.0-86-generic #96-Ubuntu SMP&lt;/p&gt;

&lt;p&gt;Mongo 6.0.12&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;...&quot;attempts&quot;:13463}}...&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

{&quot;t&quot;:\{&quot;$date&quot;:&quot;2023-12-03T04:27:39.130+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;D3&quot;, &quot;c&quot;:&quot;STORAGE&quot;, &#160;&quot;id&quot;:22413, &#160; &quot;ctx&quot;:&quot;ReplWriterWorker-0&quot;,&quot;msg&quot;:&quot;WT rollback_transaction&quot;,&quot;attr&quot;:{&quot;snapshotId&quot;:1518010}}&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2023-12-03T04:27:39.130+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;D1&quot;, &quot;c&quot;:&quot;WRITE&quot;, &#160; &#160;&quot;id&quot;:4640401, &quot;ctx&quot;:&quot;ReplWriterWorker-0&quot;,&quot;msg&quot;:&quot;Caught WriteConflictException&quot;,&quot;attr&quot;:{&quot;operation&quot;:&quot;applyOplogEntryOrGroupedInserts_CRUD&quot;,&quot;namespace&quot;:&quot;database.collection&quot;,&quot;attempts&quot;:13463}}&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2023-12-03T04:27:39.185+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;D1&quot;, &quot;c&quot;:&quot;WTWRTLOG&quot;, &quot;id&quot;:22430, &#160; &quot;ctx&quot;:&quot;JournalFlusher&quot;,&quot;msg&quot;:&quot;WiredTiger message&quot;,&quot;attr&quot;:{&quot;message&quot;:&lt;/p&gt;
{&quot;ts_sec&quot;:1701577659,&quot;ts_usec&quot;:185089,&quot;thread&quot;:&quot;4152853:0x7fdc39a99640&quot;,&quot;session_name&quot;:&quot;WT_SESSION.log_flush&quot;,&quot;category&quot;:&quot;WT_VERB_LOG&quot;,&quot;category_id&quot;:20,&quot;verbose_level&quot;:&quot;DEBUG&quot;,&quot;verbose_level_id&quot;:1,&quot;msg&quot;:&quot;log_flush: flags 0x4 LSN 28293/7000192&quot;}
&lt;p&gt;}}&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2023-12-03T04:27:39.185+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;D4&quot;, &quot;c&quot;:&quot;STORAGE&quot;, &#160;&quot;id&quot;:22419, &#160; &quot;ctx&quot;:&quot;JournalFlusher&quot;,&quot;msg&quot;:&quot;flushed journal&quot;}&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2023-12-03T04:27:39.236+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;D3&quot;, &quot;c&quot;:&quot;REPL&quot;, &#160; &#160; &quot;id&quot;:21254, &#160; &quot;ctx&quot;:&quot;ReplWriterWorker-0&quot;,&quot;msg&quot;:&quot;Applying op (or grouped inserts)&quot;,&quot;attr&quot;:{&quot;op&quot;:{&quot;lsid&quot;:{&quot;id&quot;:&lt;/p&gt;
{&quot;$uuid&quot;:&quot;64649147-795b-4d60-b933-75f9342cf688&quot;}
&lt;p&gt;,&quot;uid&quot;:{&quot;$binary&quot;:&lt;/p&gt;
{&quot;base64&quot;:&quot;u4nTF1+wmByGgmwndZCCo3FgRx9gUEtGEkFRhsYwq3A=&quot;,&quot;subType&quot;:&quot;0&quot;}
&lt;p&gt;}},&quot;txnNumber&quot;:1151,&quot;op&quot;:&quot;i&quot;,&quot;ns&quot;:&quot;database.collection&quot;,&quot;ui&quot;:{&quot;$uuid&quot;:&quot;4e33280f-7578-49a7-8cf0-4eb149c9d6a2&quot;},&quot;o&quot;:{&quot;_id&quot;:&lt;/p&gt;
{&quot;$oid&quot;:&quot;6566a09432acf3ad46c309d5&quot;}
&lt;p&gt;,&quot;linenum&quot;:35,&quot;sessionids&quot;:&lt;span class=&quot;error&quot;&gt;&amp;#91;&amp;quot;+++KIRT3UUNDXVYN7YCSCNJZ&amp;quot;,&amp;quot;+++UMMUA1LJAMWCME2GDTYD0&amp;quot;,&amp;quot;++-FPRX7E1RKPDKBEUIAK9AR&amp;quot;,&amp;quot;++-IKOOJT1XDCBXJALSRTCDX&amp;quot;,&amp;quot;++1RWUL5LRKT17YZGOPYYYOO&amp;quot;,&amp;quot;++1VB7TRZDLHIQDLEF1KAOO7&amp;quot;,&amp;quot;++37XZVPOOX60IY4LFQQDNZE&amp;quot;,&amp;quot;++3MQNL4LYTBNLBEPSTRQ8SK&amp;quot;,&amp;quot;++3ZDK4AUGOVTGKOKW2EKLS0&amp;quot;,&amp;quot;++4WQCQ3TP4ISAD930LVSO-8&amp;quot;,&amp;quot;++8NKSSBTUZCQWKQPWXMD-GN&amp;quot;,&amp;quot;++9HZQFHPOCKEKNGKEC--ASM&amp;quot;,&amp;quot;++9YB9GIEKTTHPYTBCVI+6SY&amp;quot;,&amp;quot;++AOULKEEIN37QO98A276S+I&amp;quot;,&amp;quot;++AQJA3IN3THJBJJ6IRMEV-P&amp;quot;,&amp;quot;++BEJTRTWPOR-H-6361MTSYG&amp;quot;,&amp;quot;++BEK0DPY0KJ4LXV1A9SBMNG&amp;quot;,&amp;quot;++BXGBXMFTIEGTVSX8M9SDZM&amp;quot;,&amp;quot;++BY-PBDWMJAJ16Y7S75VHEU&amp;quot;,&amp;quot;++C5DWFXPQPR1CTMFVRWPUY9&amp;quot;,&amp;quot;++CNGNVNNKEB+1OF0Z5LMRWM&amp;quot;,&amp;quot;++CS8PDWEGROS4B75E-IBGWG&amp;quot;,&amp;quot;++D0SZNJJ18BI213TPAG5VC3&amp;quot;,&amp;quot;++DN8PF8JENFEORYBVAZQXEV&amp;quot;,&amp;quot;++DPO8TMSJRAXQGWBINV1RGY&amp;quot;,&amp;quot;++E0ZMXRGP3CXMNUOCSJ8ZBV&amp;quot;,&amp;quot;++E2PJRT08CLLINTZL38KTH4&amp;quot;,&amp;quot;++EAOELV8LADXNNMIAAU6ZL2&amp;quot;,&amp;quot;++EBP4XO2ZGCISAPRKG1QKUO&amp;quot;,&amp;quot;++EGPO0VVQJHTCNIZEL8C8G7&amp;quot;,&amp;quot;++EI88+SHXUTJK4MSC18QKTE&amp;quot;,&amp;quot;++EWKA+HFIRLBSHBTXOETUC1&amp;quot;,&amp;quot;++EXZDPX08TGDJA+BTK9OKXP&amp;quot;,&amp;quot;++FXT-QXSYHICYUC+BULLXIV&amp;quot;,&amp;quot;++G64XD4BSY7KHJZIMPHU444&amp;quot;,&amp;quot;++GDS5VBIZCKQNDWALSDXVCT&amp;quot;,&amp;quot;++GTXVO5KSG-W+SXH5IHZTLF&amp;quot;,&amp;quot;++HF91OASMTDLNH2DCELLZBR&amp;quot;,&amp;quot;++HGSLNWUNA+SDSJH9HUX7EN&amp;quot;,&amp;quot;++I6NLNZHZNQL0OTSPIXDCP8&amp;quot;,&amp;quot;++ITOMUQIVWLYMM22ERISKNR&amp;quot;,&amp;quot;++IXC7LZ5E6JM+TGPRZXXYC1&amp;quot;,&amp;quot;++IY3LN4VHEWTLWSMJCT0YRA&amp;quot;,&amp;quot;++IYUNJUVVUXRFUNEJMTY5CZ&amp;quot;,&amp;quot;++J9JYB5ZORHBGYOQUNWWBIR&amp;quot;,&amp;quot;++JGOJ3DXZTXQSH7+TYRA6LE&amp;quot;,&amp;quot;++JH1737JT5GTYPZULVRWQVZ&amp;quot;,&amp;quot;++JIZ6TXRCKMW8GOD6GCIQM+&amp;quot;,&amp;quot;++JRX2QSUYQMT-SWMRGZYJRS&amp;quot;,&amp;quot;++KNX5O5VYPDFVJQB49VUW95&amp;quot;,&amp;quot;++KSYFOKXELJRJL2VMHO9G27&amp;quot;,&amp;quot;++L25IY4BR9FH9W0I0G-4ABV&amp;quot;,&amp;quot;++LI4KECOOTV5JM9HHRVMUTI&amp;quot;,&amp;quot;++LSSYTOXNR-BHBXVL90AXPC&amp;quot;,&amp;quot;++LX9K5V0L5ZWKLXL7J8BK6J&amp;quot;,&amp;quot;++M5DGSW2+AOEXD-H6XZGAZ3&amp;quot;,&amp;quot;++MJ63-8HJCMNDQUKIWOE3IO&amp;quot;,&amp;quot;++O+GA3R6MMIOQDGEL08JERI&amp;quot;,&amp;quot;++ODDSV3IC20TKQIR3WSJASE&amp;quot;,&amp;quot;++OU1Z3OVRRPGE5POKZMJN+H&amp;quot;,&amp;quot;++OZLPHL2MV45ZERWSBXT76Q&amp;quot;,&amp;quot;++PDP3-1DGSCGAPETITMLSEY&amp;quot;,&amp;quot;++POF8VYYFIDCGM9NULFDIIY&amp;quot;,&amp;quot;++Q1MO3L9MBHDCNN40CZDZGE&amp;quot;,&amp;quot;++Q6VYHQNWW3CIQ+B5ELYBGF&amp;quot;,&amp;quot;++QDAOR4RZLMI2GUGIQ8TQDL&amp;quot;,&amp;quot;++QQ8JYRAJP25DALW+HPTYAE&amp;quot;,&amp;quot;++QVTCINSTMZY1KQXJDM1GPJ&amp;quot;,&amp;quot;++R6YUDKAK5WHXD3D7XZKT6J&amp;quot;,&amp;quot;++RI9AMLJHVLKFEVIFYIROMY&amp;quot;,&amp;quot;++RUYJ8OCYVTFGIX8V2ZT4JA&amp;quot;,&amp;quot;++SYHLP2DH8NPCAOOQ10WRCB&amp;quot;,&amp;quot;++TGOTE-UGEOAIGYJ8HQBCSC&amp;quot;,&amp;quot;++THIK9VJYWDDTPQDRSI302B&amp;quot;,&amp;quot;++TSAF6T8YAI2JR-DIDHR7VD&amp;quot;,&amp;quot;++U6HWDH3E5WV7J3LAK0MWLB&amp;quot;,&amp;quot;++UK52G4VQR9IAYA5ILMJQGP&amp;quot;,&amp;quot;++URNZ3TITOEMAGWANHT+RHN&amp;quot;,&amp;quot;++UVNYHMV+HS+9FPQZ4LIQ06&amp;quot;,&amp;quot;++VGOM71BRJTPB1TM0GGQGOE&amp;quot;,&amp;quot;++VJTQBV4NNHDB8CJKUALX1V&amp;quot;,&amp;quot;++W-864G0W0KRSV7GTUZWIQ3&amp;quot;,&amp;quot;++WACSHB3NRHBVYPSVVI3XTX&amp;quot;,&amp;quot;++WAQ852AAH-DHOB5BBR48QX&amp;quot;,&amp;quot;++WIDTRBGRV9ZNHB6G0PFQ5L&amp;quot;,&amp;quot;++WM05POAA0GLAI41IGZXNRJ&amp;quot;,&amp;quot;++WNZERWKUIHR9JQADR0ANIN&amp;quot;,&amp;quot;++WOCC8VGLKAWYQW9WXM2TTN&amp;quot;,&amp;quot;++X-SUWR0M5FOJDRC--JNSUL&amp;quot;,&amp;quot;++XPRKKQW4SVU4CC+EVVPXHP&amp;quot;,&amp;quot;++XSDL6AS1YN+FO+URXGN88G&amp;quot;,&amp;quot;++XURA-PITSNPFWYWK7KAAPI&amp;quot;,&amp;quot;++YG0WP6KAVHEFLFXV34L7-S&amp;quot;,&amp;quot;++YNRYYWV+U8ZA2NBBUP5F1Q&amp;quot;,&amp;quot;++YP+W-OCVM4POQNCO03O3PV&amp;quot;,&amp;quot;++YTLQBI0KJ-PPBEVMXV7BPV&amp;quot;,&amp;quot;++ZDDIUJFLDZCUCSDXG-GRL9&amp;quot;,&amp;quot;++ZNMSBNQ+FAR-NA4ABN7SAL&amp;quot;,&amp;quot;++ZO0MVVQZULPJWT70TOW4N5&amp;quot;,&amp;quot;++ZTWHOCG+SXSGSHG5FWBJ6G&amp;quot;,&amp;quot;++ZZC8CXSDMMGCLUAZI+ACSI&amp;quot;,&amp;quot;+-+AWAFVNG51VRT0LVUTHCDW&amp;quot;,&amp;quot;+-+WODNRZJJSWYJSXQ2WVILZ&amp;quot;,&amp;quot;+---C-OIVPQCDZTZ2HEAQVOP&amp;quot;,&amp;quot;+--ZPFHVPEAMMDAPN7T3YUT4&amp;quot;,&amp;quot;+-2+D2EZRG2ZN5A+2+ERSXAY&amp;quot;,&amp;quot;+-2-O-ZDB0XM3RY-NTDGQQIL&amp;quot;,&amp;quot;+-2TMEMNXQ8UHZEJRFAQRZTY&amp;quot;,&amp;quot;+-4PTRHL9ZOC-ZL2Y3+VMOL7&amp;quot;,&amp;quot;+-50STQP+0DECDNBEKSKHB-E&amp;quot;,&amp;quot;+-5CUYOTSH3AIG-IMJRL-L8G&amp;quot;,&amp;quot;+-5J3DBY3Y-N-63PTFVIGODW&amp;quot;,&amp;quot;+-5VDAN766MQIVHV4E2FBJHR&amp;quot;,&amp;quot;+-7IXR3HTXJNDAQW-E3MVHKQ&amp;quot;,&amp;quot;+-7LMR5TZ8VFYOCFI3FIF5IG&amp;quot;,&amp;quot;+-8KVICJWARD2M5HJFSGY3ML&amp;quot;,&amp;quot;+-9KUIB-JGMXYTVGM38COIHG&amp;quot;,&amp;quot;+-9T1TWRRTNW0W-K14R7Y0J7&amp;quot;,&amp;quot;+-9U0BJZHVLCW7CHH9O-K-QC&amp;quot;,&amp;quot;+-ACUUGGLTEITH3P+A+R19I2&amp;quot;,&amp;quot;+-B-XPRGAVX9TMPWTDTVRZ5V&amp;quot;,&amp;quot;+-BBEXSUOL9JEQUTRH9GZI6P&amp;quot;,&amp;quot;+-BGF3JLNOQS3EZOMA2OV0CX&amp;quot;,&amp;quot;+-BKHNJYXDVOXRJ98I2ER8KN&amp;quot;,&amp;quot;+-BW-OES0BUOIDRHJCAPG0ER&amp;quot;,&amp;quot;+-CMVZJGNA5DOJYGCGH-JZ1+&amp;quot;,&amp;quot;+-CQHTSKRXNL-OSI81J6Y2I8&amp;quot;,&amp;quot;+-CWQNEW7JLQNY-A+6WU-P+I&amp;quot;,&amp;quot;+-DJ-I1-GLTZYOOSEZL-LXGP&amp;quot;,&amp;quot;+-DPNUWWTBV+EMBMXZWYL03S&amp;quot;,&amp;quot;+-DQWANHPA04GO0CYOMHLPGY&amp;quot;,&amp;quot;+-DWMWYZ6UUHYC8ILGYYYFJT&amp;quot;,&amp;quot;+-E5F-7U9QH8-JEWONQDDYQP&amp;quot;,&amp;quot;+-EJC-A73HK4TRK+WFBLP4WS&amp;quot;,&amp;quot;+-EPGZCXFDYERPI6HBSWPMP0&amp;quot;,&amp;quot;+-F2+AAX9LF6GFTTDLDALKMD&amp;quot;,&amp;quot;+-FBZAW0MRRJXOUXGNTCZAPV&amp;quot;,&amp;quot;+-FCP7F6MH2RADZAYVM3DHVG&amp;quot;,&amp;quot;+-FSXRO44UVIKCZ6R50Z7OCM&amp;quot;,&amp;quot;+-HMVAFCLAALREWNBWC5-OKP&amp;quot;,&amp;quot;+-HODPM7AZ3R6OE2ZS8HJSFL&amp;quot;,&amp;quot;+-HQKVU7VTSIVSYWMXJBB6LX&amp;quot;,&amp;quot;+-HRVYLQVJOZELZ4KWV6LR8B&amp;quot;,&amp;quot;+-IIAZ+MYAU+RQSWX5HK22JM&amp;quot;,&amp;quot;+-JDJE37IE72BUD3NSSMTF2G&amp;quot;,&amp;quot;+-JGF3DJR1+AQKAGD04D+MYM&amp;quot;,&amp;quot;+-JL7GHDIML-ZNYX4CISEVSZ&amp;quot;,&amp;quot;+-JPR0KJBB-2QTIVI-TGWH-R&amp;quot;,&amp;quot;+-JXRJ3PJMKQCG8GOW-NQVFL&amp;quot;,&amp;quot;+-K28AS7CF6SX3PQ6QIOQ7ZI&amp;quot;,&amp;quot;+-KJYZVXEVW1SBAI4V1RELGW&amp;quot;,&amp;quot;+-L2P98FMONRPWVEVUF23BZV&amp;quot;,&amp;quot;+-MJM1I0ZJFKT6C-N-F2Z6QU&amp;quot;,&amp;quot;+-MRWJJ4E+DKETFYF9-1NN+2&amp;quot;,&amp;quot;+-MT4DDIWQRGLD2T-VVRLQE+&amp;quot;,&amp;quot;+-MTA+PWK3EF0XK-AQB5A29Z&amp;quot;,&amp;quot;+-N+SO5ST4F5SOWXPEQBAJX6&amp;quot;,&amp;quot;+-NSZMW5V71TOUGP+74P5JKJ&amp;quot;,&amp;quot;+-OH0MSJEJXKWYOMX9RG7UAV&amp;quot;,&amp;quot;+-OUMIL5D6IYEL1S6KHIR+9H&amp;quot;,&amp;quot;+-P11OJZJA3RZ+P2FIRYDWC2&amp;quot;,&amp;quot;+-P7FD-C9PHXEIJCEASDDOHZ&amp;quot;,&amp;quot;+-PH-YJGUFD8PAGY2RBXZWOQ&amp;quot;,&amp;quot;+-PI4LAME4HKV15LI+CZY2DY&amp;quot;,&amp;quot;+-PNRASMXI3QPBAF+FSZUQFH&amp;quot;,&amp;quot;+-Q7GUECU7JUVRIZO67L1VRD&amp;quot;,&amp;quot;+-QCUYF+6M8TTMJOARGGCPBE&amp;quot;,&amp;quot;+-QHOYKH27EQUA+3OJSQSYPV&amp;quot;,&amp;quot;+-RADDP6NXGB3YSOVO80ZKYY&amp;quot;,&amp;quot;+-RAGF5KSEMF9IQE1ZMFYDZ9&amp;quot;,&amp;quot;+-RGTKBZQV1AXEKQSOK9KT3G&amp;quot;,&amp;quot;+-RKTJCHEA1DZF3S81MM6BH+&amp;quot;,&amp;quot;+-RUJ+LC5CE8+YPPXIOR17JH&amp;quot;,&amp;quot;+-RY0L8S3YL8AROA+JKC0COW&amp;quot;,&amp;quot;+-SBMTKIBEYZUPIOX6D89UCC&amp;quot;,&amp;quot;+-SECVDFOSRTE2S5REITZ1WZ&amp;quot;,&amp;quot;+-SLLFJIGFYN52MPS-OASVRI&amp;quot;,&amp;quot;+-SNDMVYUWH2-GAGF86520TG&amp;quot;,&amp;quot;+-SQRHJONZHBZUDQYBCV8BQA&amp;quot;,&amp;quot;+-T2Y4OA1O7Y2ZECGKZTT7OL&amp;quot;,&amp;quot;+-U2TO6VLPTELEE1RTXVN5CR&amp;quot;,&amp;quot;+-UE+WEHVBVHQGGP9WBYWTHL&amp;quot;,&amp;quot;+-UEG6ASOJDR94ZI8RNIWZP7&amp;quot;,&amp;quot;+-UGHTK-8EBKCYJ5C95U7MSD&amp;quot;,&amp;quot;+-UIBM24NX8PZCDMJILFNHGN&amp;quot;,&amp;quot;+-UXUG2F04YJYVGLMJFOE3-A&amp;quot;,&amp;quot;+-UYYYMYWFL6RBWFEFLXVPQ2&amp;quot;,&amp;quot;+-VL2VMTO0ZCNEE-XQMKY+RT&amp;quot;,&amp;quot;+-WA5ZM0QKQGSZKVTWDK6E7A&amp;quot;,&amp;quot;+-WD3NEW5IJAMHRY0VAHY5UW&amp;quot;,&amp;quot;+-WNTAM5XSWWGTJSYO0QGC49&amp;quot;,&amp;quot;+-XC7V9ZCCUIFSVVCQRUX3V9&amp;quot;,&amp;quot;+-YGVJIIQZ+XTVQPGLYBUSMD&amp;quot;,&amp;quot;+-YK-DHG5KAV-Q89DVTNYY78&amp;quot;,&amp;quot;+-YME3PNZ3QTBXQP0JK2ZUPE&amp;quot;,&amp;quot;+-YXIJKZBNJ6GOY4SSBBNDSK&amp;quot;,&amp;quot;+-Z4KWO3OBBVHUCYHEN9XP8Q&amp;quot;,&amp;quot;+-ZPCCUKPFHFDGAYGNIHHBM-&amp;quot;,&amp;quot;+-ZTELTHVROW4LZSKF-KW8NQ&amp;quot;,&amp;quot;+-ZTUC3HS6KBKRMOMY0AN8XQ&amp;quot;,&amp;quot;+-ZZBPMCXWHQBT2LA6V39K-F&amp;quot;,&amp;quot;+0+CK-EEUUXY1DM3XMJUC9Q-&amp;quot;,&amp;quot;+0-DKMCV1LE4XU7KPQZPCJCF&amp;quot;,&amp;quot;+0-LLV1ULPUD0PXNYPJ6T5YH&amp;quot;,&amp;quot;+00JE2MRU3QRWOMAU1VHVLXH&amp;quot;,&amp;quot;+01EEBAETPCIBRKGUPIAM25H&amp;quot;,&amp;quot;+01N8GCPCTZIKTKR649BW1O7&amp;quot;,&amp;quot;+01QV9E2AIJFJRC5TPYMO0-D&amp;quot;,&amp;quot;+02HPZQPP07BN5SKSZONIFQP&amp;quot;,&amp;quot;+02OBZB9SC3DBD1SCJTCVTFG&amp;quot;,&amp;quot;+036-8FHZIVLZ+INC6X6TYNI&amp;quot;,&amp;quot;+03YRMSZYZ2GWYNOD8SUQUEM&amp;quot;,&amp;quot;+05FARD7M3FMZRFUEH6QBY6L&amp;quot;,&amp;quot;+05T8MEPQMLSYRANULM2RERP&amp;quot;,&amp;quot;+06L52FSMIQTEPV9OI5UHQMC&amp;quot;,&amp;quot;+06YM-ALNECUCMG7BM0C4PLR&amp;quot;,&amp;quot;+07HLLKFRUTFD1AAZ6ID8X5+&amp;quot;,&amp;quot;+07TWY0YQZRU1GV2OWZ0BAWU&amp;quot;,&amp;quot;+08MHDMCBE6G+2R3FATH4A+J&amp;quot;,&amp;quot;+08V9YAFQNVDHES6FXCHBO+M&amp;quot;,&amp;quot;+08Y4WNJQ2UNLMXJX4C1GFJB&amp;quot;,&amp;quot;+09-2UGNXYT5CHEUECIZOVBS&amp;quot;,&amp;quot;+0A2RXHECHEC1QVAAQFU2GO9&amp;quot;,&amp;quot;+0ALA4C-OODZXIXG-QKWEKSR&amp;quot;,&amp;quot;+0AR3D4NMAX6A4WX+2SB3O65&amp;quot;,&amp;quot;+0AU+N9U44V6I4EGKJE4PYBL&amp;quot;,&amp;quot;+0BDLICC8DZ0DJKDU6YZSTC8&amp;quot;,&amp;quot;+0BTQNHHUFKSV7POVQ0OGQAR&amp;quot;,&amp;quot;+0C++KWLLTG4R2AYXEBPHZM7&amp;quot;,&amp;quot;+0CADRGL7BJNAHJQN1WA2HWG&amp;quot;,&amp;quot;+0CLDLZQ6DZPEPUDCVNB7QXB&amp;quot;,&amp;quot;+0CLZ6NUO0FMDIHM7BVQUITF&amp;quot;,&amp;quot;+0CNK2OAVMNVD+-E9ATXGHMQ&amp;quot;,&amp;quot;+0D6MNSXYQ2O2KTBDXYZQX4A&amp;quot;,&amp;quot;+0DD06NEKUX1NCAVFMSBLB9I&amp;quot;,&amp;quot;+0DJXT3PQCQCGXHCHWRWEHUO&amp;quot;,&amp;quot;+0DQA822YTUJP5LM8D5LXEK8&amp;quot;,&amp;quot;+0ETFSGVKFFLOHSCDYKQBPVN&amp;quot;,&amp;quot;+0FB1KO8J4S05HA2LLBPBOR9&amp;quot;,&amp;quot;+0FD8U1PCV8RHDQMN8VDPQZK&amp;quot;,&amp;quot;+0FUDWDTX9TDS7WMDDSBARN7&amp;quot;,&amp;quot;+0G1B24WYOJLKJZ-S6AYPVSM&amp;quot;,&amp;quot;+0G4T8G4RUCHQATQPBZX7V0R&amp;quot;,&amp;quot;+0GBXEESZESGRTR5+SPZMINC&amp;quot;,&amp;quot;+0GLPTOTGYECFNQYTZUS9GTM&amp;quot;,&amp;quot;+0GQM-MNT+V6MUBVOCTALTKT&amp;quot;,&amp;quot;+0GYWW6ZWN0JZYEVVJ2N7F2A&amp;quot;,&amp;quot;+0HYDCXBFVPFXBWGIKYCBBAI&amp;quot;,&amp;quot;+0HZRQY2MK8W3CNRCGHC1SNG&amp;quot;,&amp;quot;+0IBNKJ92A4N-1ZUPWBQYSZU&amp;quot;,&amp;quot;+0ICKIPBZUG6NL2FKQWMRRV0&amp;quot;,&amp;quot;+0IXHCKNPWDPAH2GNAPU3V0V&amp;quot;,&amp;quot;+0IXUE8NLIYYCYXOQCXL3ZBC&amp;quot;,&amp;quot;+0J6JC7THCLMO6V49R5LI646&amp;quot;,&amp;quot;+0J96AIEWUJFEBHZWVJHALHI&amp;quot;,&amp;quot;+0JMOAIIYBVGRPUJEMCQTAJL&amp;quot;,&amp;quot;+0JMXYDBSUOALA49GEDC5AIG&amp;quot;,&amp;quot;+0JVNNMB2MGXC0FDSZ5NUEU8&amp;quot;,&amp;quot;+0K-V408F78JT07-D2ZN3HMZ&amp;quot;,&amp;quot;+0KB-2RX6G6R20KWBJRCQPN-&amp;quot;,&amp;quot;+0KFXODTZKXMLBI-NKBMBPEN&amp;quot;,&amp;quot;+0LKKBZF+VX6HQ4SSGEEU26J&amp;quot;,&amp;quot;+0LN0H6NCMUXM9I3AREAOOEJ&amp;quot;,&amp;quot;+0LNGP0W4DZXXJUF2YDL3WTY&amp;quot;,&amp;quot;+0MDHHMGNIFTP-9OB3W3O140&amp;quot;,&amp;quot;+0MMR8X52M5A09+B22QNHHUF&amp;quot;,&amp;quot;+0MWJCASDO8UTDT-QUJN9IA2&amp;quot;,&amp;quot;+0MYKL7LXJWBZLANKGN1DJQV&amp;quot;,&amp;quot;+0MZKK2O714HLW8LVWG9FBVD&amp;quot;,&amp;quot;+0NNP5IPSBABWSHBC342KNBP&amp;quot;,&amp;quot;+0NYNIMSBMCIRWDXS7+O3SOE&amp;quot;,&amp;quot;+0O3M-TCSNX972G9OMPVUIM1&amp;quot;,&amp;quot;+0OJ+RJXW2IYLFB3T0XFFCD5&amp;quot;,&amp;quot;+0OJCL-B5MDUG+FH2ZB3IETF&amp;quot;,&amp;quot;+0OLAQSTKA7ACVKYFW94PDUA&amp;quot;,&amp;quot;+0PTYS-CWAVQXNOWDWXRM+CR&amp;quot;,&amp;quot;+0QGJ8U3297WBHTWESP2FWTM&amp;quot;,&amp;quot;+0R+0DROSOXD0QE5LZMUHUZT&amp;quot;,&amp;quot;+0S6L0E1QBKU4UTRVALAQBNI&amp;quot;,&amp;quot;+0S7NRLHLCQSWQXSCHZMGNA7&amp;quot;,&amp;quot;+0SCP8MJH8SISI237E2VA3EK&amp;quot;,&amp;quot;+0SDFH1EGEAEYEQMQHFDBGAL&amp;quot;,&amp;quot;+0SZOP5DS7XMGUREJEIUHR-V&amp;quot;,&amp;quot;+0T2ZWUOWAMIL0JIUOOM807S&amp;quot;,&amp;quot;+0TFOY9ZUXJNYBVBS2P1IMAT&amp;quot;,&amp;quot;+0TKUTMOAZD8AGCQSVAWF30A&amp;quot;,&amp;quot;+0UIY95RXIKOGSSFV2GBB0KU&amp;quot;,&amp;quot;+0ULQPDUQFFXXJQE9U4PW+KV&amp;quot;,&amp;quot;+0UOUTSX+UGLTM9FVHX2LRTF&amp;quot;,&amp;quot;+0UTW2-WFMN1MPF2717I3V92&amp;quot;,&amp;quot;+0UYTEH-N-DX-HIKMY78QJJ8&amp;quot;,&amp;quot;+0VYT-QGUNWUT854EYXNRM2D&amp;quot;,&amp;quot;+0W81L55OSXXBNJFNLIVDGYH&amp;quot;,&amp;quot;+0WNEZPR4BZWRDG5NNSIRAQX&amp;quot;,&amp;quot;+0XO25UA4P+CLLWYKBBNKAH6&amp;quot;,&amp;quot;+0XPAV-CD7DA-KEGJZYR0EIF&amp;quot;,&amp;quot;+0YD7WQVCXSWQPWF39FZLSNE&amp;quot;,&amp;quot;+0YZDHFUEEP6RAF+MPVSNXBY&amp;quot;,&amp;quot;+0Z-EPTUEFPQB8NWAKRP478B&amp;quot;,&amp;quot;+0ZER+ZQLAQJYIWZ5BDUAHY4&amp;quot;,&amp;quot;+0ZJLQGFI+SOQW3ZYW7RJYGN&amp;quot;,&amp;quot;+0ZPNUYSJFGEXZD9NWG4JSZW&amp;quot;,&amp;quot;+0ZSMRKHGTXLO4SFL8CPA6JG&amp;quot;,&amp;quot;+1++TDWVTNNQXN4DLS9QII+P&amp;quot;,&amp;quot;+1+F5VMU2JOARIUNTR-+FMYW&amp;quot;,&amp;quot;+1+KB+I8IZHEVJHJOXVSVVUW&amp;quot;,&amp;quot;+1-KPPJU7IQALSWUGF7QZZ4+&amp;quot;,&amp;quot;+1-L27J0B1P-YYOKQMFEIRCX&amp;quot;,&amp;quot;+10Q0WMELFLDSJ93EY-4JYUL&amp;quot;,&amp;quot;+12CKX-LJ3EWL1TOAX4EP41F&amp;quot;,&amp;quot;+13S+SD8NX2QTIAYFRKOLRQH&amp;quot;,&amp;quot;+13SPNQGWD0YHKBDCDW8UJCB&amp;quot;,&amp;quot;+14PRSMKDHHO-T3DEZG5BTIF&amp;quot;,&amp;quot;+15QW2PYAX25EJJYKFMEBDHU&amp;quot;,&amp;quot;+179MDCTAVRFHW84NMDBQYWS&amp;quot;,&amp;quot;+17GUGBXIBMDX7HGTWTI7UUA&amp;quot;,&amp;quot;+17ICNPG+OKZBNPXGKF7UOW1&amp;quot;,&amp;quot;+17IEWUYUK7FGENDF+D8ZEZ8&amp;quot;,&amp;quot;+192E6DHEXD7PRUKKLFQXXJA&amp;quot;,&amp;quot;+19WOMK5U8L0WZ-GTKZAOC8Y&amp;quot;,&amp;quot;+1A54IYNNBPZ-R7QOBXAJSFM&amp;quot;,&amp;quot;+1AB2ROEM2T2I+DDTTSJ08HR&amp;quot;,&amp;quot;+1ANP-8+BOEIMCC-S+LYHNTC&amp;quot;,&amp;quot;+1BFVXQ42BSLNK5UTD8JS6WR&amp;quot;,&amp;quot;+1CEWTNV1MTD-LKP7CKQ2UN8&amp;quot;,&amp;quot;+1CF7P3AKIUE3BZN3WMLA8NV&amp;quot;,&amp;quot;+1CGIHBTPKUBPR45PCOR7SC2&amp;quot;,&amp;quot;+1CSBZ0OITD2VE9QF2HDHTK2&amp;quot;,&amp;quot;+1CTBVJ8UIN6G41DJ71OLDLT&amp;quot;,&amp;quot;+1CX+C6ENG4IK9UWMTK1T2X+&amp;quot;,&amp;quot;+1DGKCR7E0QZFIOUQYQ9QZBS&amp;quot;,&amp;quot;+1DZQLTAN7VMNQ1P5LZVRR6N&amp;quot;,&amp;quot;+1E2EADOXCJJS3USXWMWOFJW&amp;quot;,&amp;quot;+1F0MFRNQMOOM5GOWQ6S9BZF&amp;quot;,&amp;quot;+1FKMRKYAHWWJEQWZWXEIPUS&amp;quot;,&amp;quot;+1FTZBMEQIH0TMVAICRHVHUR&amp;quot;,&amp;quot;+1G9SAIZGA4TTU--HXZVPXI-&amp;quot;,&amp;quot;+1GHLQPAFNW8G7BHZ5NAOGTJ&amp;quot;,&amp;quot;+1GLHM-VIUW0G4LPFMLD0C3G&amp;quot;,&amp;quot;+1GN5GMFBRK0ELI9BS83F3-9&amp;quot;,&amp;quot;+1GW8VPHEU9ZAAWHRPYZKKIC&amp;quot;,&amp;quot;+1GYE+DLD3NGJKW-JAIJCLHY&amp;quot;,&amp;quot;+1IP9F5EV1Y4S10ZDSI0OKAR&amp;quot;,&amp;quot;+1IQE+MIFOBNQ6FOCG1P2AX4&amp;quot;,&amp;quot;+1IVUQQMBI7ECWAMNYJMFVL1&amp;quot;,&amp;quot;+1IW8XVTJILT7GONOJRAUPHR&amp;quot;,&amp;quot;+1JSGFN7ATAB49KZG6W-D7BT&amp;quot;,&amp;quot;+1JVEB-MMKHXHBDFIIB528SN&amp;quot;,&amp;quot;+1KYSDIX1AKPYW1NCUQYXP3O&amp;quot;,&amp;quot;+1LE701ZU1KKNV36WPQ6EACF&amp;quot;,&amp;quot;+1LGPSLO+NDDW1KG3JKBUUUD&amp;quot;,&amp;quot;+1LKVG2OQL9ZK1QHCEKB9SZX&amp;quot;,&amp;quot;+1LM-5BJVE3AHH3BURVQEXKB&amp;quot;,&amp;quot;+1LYOWX+NNVKOY1ETOUMY5S-&amp;quot;,&amp;quot;+1MHATVU-SIPQ9FCYVPCCVDP&amp;quot;,&amp;quot;+1MX+YXDQDWZZG07Q32LUOPB&amp;quot;,&amp;quot;+1NFVHVFXTW3BBJACQOEODK1&amp;quot;,&amp;quot;+1NJL99FL+XAZDXUOVDROA37&amp;quot;&amp;#93;&lt;/span&gt;}},&quot;oplogApplicationMode&quot;:&quot;StableRecovering&quot;},&quot;truncated&quot;:{&quot;op&quot;:{&quot;o&quot;:{&quot;sessionids&quot;:{&quot;358&quot;:&lt;/p&gt;
{&quot;type&quot;:&quot;string&quot;,&quot;size&quot;:29}
&lt;p&gt;}}}},&quot;size&quot;:{&quot;op&quot;:14493733}}&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2023-12-03T04:27:39.236+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;D3&quot;, &quot;c&quot;:&quot;STORAGE&quot;, &#160;&quot;id&quot;:22414, &#160; &quot;ctx&quot;:&quot;ReplWriterWorker-0&quot;,&quot;msg&quot;:&quot;WT begin_transaction&quot;,&quot;attr&quot;:{&quot;snapshotId&quot;:1518124,&quot;readSource&quot;:&quot;kNoTimestamp&quot;}}&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</description>
                <environment></environment>
        <key id="2514963">SERVER-83833</key>
            <summary>shard crash causes WiredTiger to fail replaying the oplog in an infinite loop</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="11262" iconUrl="https://jira.mongodb.org/images/icons/statuses/generic.png" description="">Investigating</status>
                    <statusCategory id="4" key="indeterminate" colorName="inprogress"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-server-triage">Backlog - Triage Team</assignee>
                                    <reporter username="mzimmerman@gmail.com">Matthew Zimmerman</reporter>
                        <labels>
                    </labels>
                <created>Sun, 3 Dec 2023 04:33:50 +0000</created>
                <updated>Mon, 1 Jan 2024 08:01:05 +0000</updated>
                                                                                                <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="5949536" author="eric.sedor" created="Wed, 13 Dec 2023 22:35:45 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mzimmerman%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;mzimmerman@gmail.com&quot;&gt;mzimmerman@gmail.com&lt;/a&gt;, I&apos;m glad to hear you found a path forward that didn&apos;t involve forking. My strong suspicion based on your report and reported solution is that this is related to &lt;a href=&quot;https://www.mongodb.com/docs/manual/core/transactions-production-consideration/#wiredtiger-cache&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;current transaction and cache size necessities&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;It may or may not be a bug that we couldn&apos;t get to a better error during a state of high cache thrash.&lt;/p&gt;

&lt;p&gt;I&apos;ve created a secure &lt;a href=&quot;https://amphora.corp.mongodb.com/public/upload/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmb2xkZXJfaWQiOiIyMzk3NTMxNzg5NTMiLCJleHAiOjE3MDUwOTg5MjB9.ax46Em4_3rRWxnGn4gb0dro8XLUHkV9MOmYPooRm6sM&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upload portal&lt;/a&gt; for you. Files uploaded to this portal are hosted on Box, are visible only to MongoDB employees, and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;If you still have this information available, then for each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to that link:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;the mongod logs&lt;/li&gt;
	&lt;li&gt;the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Gratefully,&lt;br/&gt;
Eric&lt;/p&gt;</comment>
                            <comment id="5923067" author="mzimmerman@gmail.com" created="Mon, 4 Dec 2023 14:28:31 +0000"  >&lt;p&gt;Cache full was a hint that I then followed &#8211; the size of our cache was configured to 16GB (our workload is very write-heavy, so having all indexes in RAM wasn&apos;t a thing I was trying to do, though I know that&apos;s best practice).&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Using unmodified code, increasing that cache size then allowed the transaction to be written fully from the oplog and the system didn&apos;t hang.&#160; After this particular transaction was inserted and the shard/replicaset was clean in a PSS architecture, I was then able to stop/restart each shard with the configured 16GB of memory and they&apos;ve been running fine for ~8 hours without my modified code.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Will I hit this limit again somehow in the future?&#160; What is it about inserting a large transaction that would cause this issue?&#160; Any one record can only be 16MB in size... so how could writing just one larger record hang a system with 16GB of memory for the cache configured?&lt;/p&gt;</comment>
                            <comment id="5922231" author="mzimmerman@gmail.com" created="Mon, 4 Dec 2023 03:39:40 +0000"  >&lt;p&gt;After commenting out some of the error handling (attempting to tell mongo that the write succeeded and to just move on with recovering everything else in the oplog (I don&apos;t care about this one transaction that it&apos;s stuck on)), I got another error message further along:&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

{&quot;t&quot;:\{&quot;$date&quot;:&quot;2023-12-04T03:38:49.357+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;D1&quot;, &quot;c&quot;:&quot;WTTXN&quot;, &#160; &#160;&quot;id&quot;:22430, &#160; &quot;ctx&quot;:&quot;ReplWriterWorker-1&quot;,&quot;msg&quot;:&quot;WiredTiger message&quot;,&quot;attr&quot;:{&quot;message&quot;:&lt;/p&gt;
{&quot;ts_sec&quot;:1701661129,&quot;ts_usec&quot;:357612,&quot;thread&quot;:&quot;390727:0x7fb4944ca640&quot;,&quot;session_dhandle_name&quot;:&quot;file:index-76--2567016947708944443.wt&quot;,&quot;session_name&quot;:&quot;WT_CURSOR.insert&quot;,&quot;category&quot;:&quot;WT_VERB_TRANSACTION&quot;,&quot;category_id&quot;:39,&quot;verbose_level&quot;:&quot;DEBUG&quot;,&quot;verbose_level_id&quot;:1,&quot;msg&quot;:&quot;Rollback reason: Cache full&quot;}
&lt;p&gt;}}&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I can see where this is called/triggered in the code, but I&apos;m not sure how to &quot;skip&quot; this to continue along with this troubleshooting idea.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25137"><![CDATA[Server Triage]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 13 Dec 2023 22:35:45 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>dbeng-pm-bot</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-triage</customfieldvalue>
            <customfieldvalue>eric.sedor@mongodb.com</customfieldvalue>
            <customfieldvalue>mzimmerman@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i32ylz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i2kr04:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[eric.sedor@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i32krb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>