<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:52:37 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39627] Audit uses of TransactionParticipant::SideTransactionBlock</title>
                <link>https://jira.mongodb.org/browse/SERVER-39627</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We use SideTransactionBlock to do things outside a transaction, while in a multi-document transaction. E.g., when OpObserverImpl writes the prepare oplog entry or (in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36494&quot; title=&quot;Prevent oplog truncation of oplog entries needed for startup recovery&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36494&quot;&gt;&lt;del&gt;SERVER-36494&lt;/del&gt;&lt;/a&gt;) when TransactionParticipant writes to another collection in the &quot;local&quot; DB.&lt;/p&gt;

&lt;p&gt;It seems error-prone that a side transaction block only partly masks the state of the outer transaction. For example, state methods like TransactionParticipant::inMultiDocumentTransaction() reflect the outer transaction.&lt;/p&gt;

&lt;p&gt;This can cause mistakes when recursing into the TransactionParticipant while doing an operation in a side transaction via&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;TransactionParticipant method -&amp;gt; OpObserverImpl method -&amp;gt; TransactionParticipant::addTransactionOperation&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Audit uses of SideTransactionBlock and consider extending the SideTransactionBlock to more thoroughly mask evidence of the TransactionParticipant&apos;s outer transaction.&lt;/p&gt;</description>
                <environment></environment>
        <key id="698917">SERVER-39627</key>
            <summary>Audit uses of TransactionParticipant::SideTransactionBlock</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="2">Won&apos;t Fix</resolution>
                                        <assignee username="judah.schvimer@mongodb.com">Judah Schvimer</assignee>
                                    <reporter username="jesse@mongodb.com">A. Jesse Jiryu Davis</reporter>
                        <labels>
                            <label>prepare_basic</label>
                    </labels>
                <created>Fri, 15 Feb 2019 21:42:23 +0000</created>
                <updated>Mon, 17 Jun 2019 19:05:33 +0000</updated>
                            <resolved>Mon, 17 Jun 2019 19:05:33 +0000</resolved>
                                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="2285815" author="siyuan.zhou@10gen.com" created="Fri, 14 Jun 2019 20:16:38 +0000"  >&lt;p&gt;Great analysis, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;! I agreed the original concern by Jesse should be resolved using UnreplicatedWritesBlock, since missing UnreplicatedWritesBlock when it&apos;s needed sounds a more fundamental issue anyway. Doing nothing makes sense to me.&lt;/p&gt;</comment>
                            <comment id="2285640" author="schwerin" created="Fri, 14 Jun 2019 18:29:27 +0000"  >&lt;p&gt;SGTM&lt;/p&gt;</comment>
                            <comment id="2285638" author="judah.schvimer" created="Fri, 14 Jun 2019 18:28:24 +0000"  >&lt;p&gt;I&apos;m proposing to do nothing in this ticket. There is definitely still a chance of deadlock using the ACR improperly, but I think for now it makes more sense to fix those one by one. If someone wants to advocate for finishing the POC of having ACRs inherit lockers, I&apos;m open to it, but do not want to invest the time right now to prove this is safe given &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=siyuan.zhou&quot; class=&quot;user-hover&quot; rel=&quot;siyuan.zhou&quot;&gt;siyuan.zhou&lt;/a&gt;&apos;s concern. I will close this ticket &quot;Won&apos;t Fix&quot; in a few days if I hear nothing else, though we can always reopen it.&lt;/p&gt;</comment>
                            <comment id="2282040" author="judah.schvimer" created="Wed, 12 Jun 2019 22:20:09 +0000"  >&lt;ol&gt;
	&lt;li&gt;&lt;tt&gt;OplogSlotReserver&lt;/tt&gt;
	&lt;ol&gt;
		&lt;li&gt;Does not re-enter &lt;tt&gt;TransactionParticipant.&lt;/tt&gt;&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
	&lt;li&gt;profile
	&lt;ol&gt;
		&lt;li&gt;Creating the &lt;tt&gt;system.profile&lt;/tt&gt; collection if needed is in an &lt;tt&gt;UnreplicatedWritesBlock.&lt;/tt&gt;&lt;/li&gt;
		&lt;li&gt;Collection creation isn&apos;t allowed in transactions so there&apos;s nothing else that the &lt;tt&gt;OpObserver&lt;/tt&gt; does on collection creation in a transaction that it does not do normally. This is however a place where we could easily introduce a bug when we add DDL op support in transactions. I don&apos;t have a good forward thinking way of preventing this.&lt;/li&gt;
		&lt;li&gt;We do re-enter the &lt;tt&gt;TransactionParticipant&lt;/tt&gt; for individual &lt;tt&gt;system.profile&lt;/tt&gt; inserts. In &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4b985d5460a2ddb889942f3b4ede04d7051b3921/src/mongo/db/op_observer_impl.cpp#L481-L486&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;OpObserverImpl::onInserts&lt;/tt&gt;&lt;/a&gt; we check if we have a WUOW as a proxy for if we&apos;re doing an insert into &lt;tt&gt;system.profile&lt;/tt&gt; in a transaction. We then choose not to add the operation to the transaction based on that proxy.&lt;/li&gt;
		&lt;li&gt;Sharding also checks for &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4b985d5460a2ddb889942f3b4ede04d7051b3921/src/mongo/db/s/op_observer_sharding_impl.cpp#L109-L112&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;chunks that intersect with inserts&lt;/a&gt; in transactions. This is done unnecessarily for &lt;tt&gt;system.profile&lt;/tt&gt; inserts as a result of mistakenly thinking the insert is happening in a transaction, but it doesn&apos;t seem dangerous, just unnecessary work. These inserts cannot be part of a chunk migration.&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;prepare&lt;/tt&gt; oplog entry
	&lt;ol&gt;
		&lt;li&gt;This actually expects to re-enter the participant in the transaction.&lt;/li&gt;
		&lt;li&gt;Looking at all uses of the participant in &lt;tt&gt;OpObserverImpl&lt;/tt&gt;, &lt;tt&gt;TP::onWriteOpCompletedOnPrimary()&lt;/tt&gt; and &lt;tt&gt;TP::addTransactionOperation()&lt;/tt&gt; are the only functions that change participant state.
		&lt;ol&gt;
			&lt;li&gt;&lt;tt&gt;TP::onWriteOpCompletedOnPrimary()&lt;/tt&gt;: This function needs to be called within a transaction and expects a STB.&lt;/li&gt;
			&lt;li&gt;&lt;tt&gt;TP::addTransactionOperation()&lt;/tt&gt;: This could be modified to prevent accidentally adding operations that occur in STBs to mongodb-transactions, but using UWBs when needed seems less complex, and this change doesn&apos;t feel appropriately motivated.&lt;/li&gt;
		&lt;/ol&gt;
		&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Thus I do not see any work to do now based on this audit.&lt;/p&gt;</comment>
                            <comment id="2281793" author="judah.schvimer" created="Wed, 12 Jun 2019 19:54:37 +0000"  >&lt;p&gt;The original problem this ticket was addressing is hinted at at the bottom of &lt;a href=&quot;https://mongodbcr.appspot.com/220470001/diff/1/src/mongo/db/transaction_participant.cpp&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this cr&lt;/a&gt; on line 2148. An older version of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36494&quot; title=&quot;Prevent oplog truncation of oplog entries needed for startup recovery&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36494&quot;&gt;&lt;del&gt;SERVER-36494&lt;/del&gt;&lt;/a&gt; tried to add a new usage of &lt;tt&gt;SideTransactionBlock&lt;/tt&gt; (STB) that did a write that was intended to be unreplicated. At first the STB didn&apos;t have an &lt;tt&gt;UnreplicatedWritesBlock&lt;/tt&gt; (UWB). When the &lt;tt&gt;OpObserver&lt;/tt&gt; fired for the write in the STB, it attempted to add a new operation to the mongodb transaction that attempted to do the side-transaction write. Since it was not in an UWB, this succeeded and led to a bug where the unreplicated write was put into the transaction oplog entry. While a greater &quot;masking&quot; of transaction state in STB would have fixed this problem, an UWB also would have fixed the problem in a less invasive way. The write was unreplicated and so an UWB was appropriate regardless. &lt;/p&gt;

&lt;p&gt;The goal of this ticket is to replace STB with ACR. To fully do this we would need to give ACR similar treatment to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40498&quot; title=&quot;Writing transaction oplog entries must not take locks while holding an oplog slot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40498&quot;&gt;&lt;del&gt;SERVER-40498&lt;/del&gt;&lt;/a&gt; in terms of the alternative client inheriting the locker of the parent client. This is only necessary for the &lt;tt&gt;OplogSlotReserver&lt;/tt&gt;. That said, as Tess points out, ACRs and STBs not sharing lockers is deadlock prone. Currently there are no known deadlocks in ACR usages, but the &lt;tt&gt;OplogSlotReserver&lt;/tt&gt; would have a deadlock if &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40498&quot; title=&quot;Writing transaction oplog entries must not take locks while holding an oplog slot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40498&quot;&gt;&lt;del&gt;SERVER-40498&lt;/del&gt;&lt;/a&gt; didn&apos;t make STB share a locker with its parent.&lt;/p&gt;

&lt;p&gt;This implies two separate changes:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Make the ACR share its parent locker
	&lt;ol&gt;
		&lt;li&gt;This could be done mostly on its own&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
	&lt;li&gt;Replace STB with ACR
	&lt;ol&gt;
		&lt;li&gt;This requires ACR to share a locker, or at least to create a special mechanism for the &lt;tt&gt;OplogSlotReserver&lt;/tt&gt; to address &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40498&quot; title=&quot;Writing transaction oplog entries must not take locks while holding an oplog slot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40498&quot;&gt;&lt;del&gt;SERVER-40498&lt;/del&gt;&lt;/a&gt;.&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;There are three usages of STB today:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Writing the &lt;tt&gt;prepare&lt;/tt&gt; oplog entry
	&lt;ol&gt;
		&lt;li&gt;The &lt;tt&gt;OpObserver&lt;/tt&gt; code, as can be seen in &lt;a href=&quot;https://mongodbcr.appspot.com/455530001/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this POC&lt;/a&gt; currently makes many assumptions that it can reach back into the &lt;tt&gt;TransactionParticipant&lt;/tt&gt;. One example that was not even addressed in the POC is for deciding to write to the &lt;tt&gt;config.transactions&lt;/tt&gt; table. To complete the POC and replace this STB with an ACR, we would need to remove all places in the &lt;tt&gt;OpObserver&lt;/tt&gt; that call back into the &lt;tt&gt;TransactionParticipant&lt;/tt&gt;, which would be a non-trivial refactor.&lt;/li&gt;
		&lt;li&gt;I don&apos;t currently see any further re-entrance problems here, certainly not of the variety this ticket was originally created for. If we do leave this as is, I will need to double check this.&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
	&lt;li&gt;The &lt;tt&gt;OplogSlotReserver&lt;/tt&gt;
	&lt;ol&gt;
		&lt;li&gt;&lt;b&gt;all&lt;/b&gt; this cares about is having a side-transaction that gets thrown away but needs to share a locker of the parent. STB seems like the right construct for this as it exists today.&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
	&lt;li&gt;Profiling inside of a transaction
	&lt;ol&gt;
		&lt;li&gt;Also not addressed in this POC, this currently relies on maintaining information about the parent client. &lt;tt&gt;profile()&lt;/tt&gt; includes the client address, which an ACR would hide. Sharing a locker here is also important since this does take locks, sometimes database X locks if the &lt;tt&gt;system.profile&lt;/tt&gt; collection needs to be created.&lt;/li&gt;
		&lt;li&gt;Similarly to the &lt;tt&gt;prepare&lt;/tt&gt; oplog entry, this needs slightly further audit for re-entrance problems, though I don&apos;t anticipate any.&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;It seems to me that basically all of the concerns are around making the ACR share its parent locker. Andy is concerned about &lt;tt&gt;currentOp&lt;/tt&gt;, &lt;tt&gt;killOp&lt;/tt&gt;, and &lt;tt&gt;killSession&lt;/tt&gt;. Siyuan is concerned about the fundamental assumptions here. As Tess said though, sharing the locker is important in terms of preventing deadlocks. This seems like the only real motivating purpose of this change right now.&lt;/p&gt;

&lt;p&gt;Siyuan&apos;s concern is valid and I personally do not know what could go wrong by breaking the 1:1 mapping from client:locker.&lt;/p&gt;

&lt;p&gt;To address Andy&apos;s concern we&apos;ll go through the three concerning commands and how the two changes suggested would impact them:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Replace STB with ACR
	&lt;ol&gt;
		&lt;li&gt;currentOp
		&lt;ol&gt;
			&lt;li&gt;For a short period of time during commands that get profiled or during &lt;tt&gt;prepareTransaction&lt;/tt&gt; or &lt;tt&gt;commitTransaction&lt;/tt&gt;, &lt;tt&gt;currentOp&lt;/tt&gt; will start showing a different client similar to other ACRs, this is very short lived in all use cases and should be inconsequential.&lt;/li&gt;
			&lt;li&gt;We could only do this if we&apos;re in a multi-document transaction if that were desired.&lt;/li&gt;
		&lt;/ol&gt;
		&lt;/li&gt;
		&lt;li&gt;killOp
		&lt;ol&gt;
			&lt;li&gt;In the short period of time where currentOp shows different clients, the opId that users see and try to kill will be incorrect. It is unlikely users will try to kill an operation during this time since users generally try to kill long running operations.&lt;/li&gt;
		&lt;/ol&gt;
		&lt;/li&gt;
		&lt;li&gt;killSession
		&lt;ol&gt;
			&lt;li&gt;This will kill the original &lt;tt&gt;OperationContext&lt;/tt&gt; (and not the new one the ACR creates) since that&apos;s what checked out the session. It will abort any unprepared transactions on the session like it does already.&lt;/li&gt;
			&lt;li&gt;The operation will not check for interrupt while in the ACR and so that will not be interrupted, but it may be interrupted once the ACR goes out of scope.&lt;/li&gt;
			&lt;li&gt;This behavior is already the case in any of the ACR usages around the codebase today and will not become significantly more prevalent with this change.&lt;/li&gt;
		&lt;/ol&gt;
		&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
	&lt;li&gt;ACRs inheret lockers
	&lt;ol&gt;
		&lt;li&gt;currentOp
		&lt;ol&gt;
			&lt;li&gt;Lock stats will now be shared by the ACR and its parent. This is the behavior we actually likely want. Right now we&apos;re forgetting some of the locks a given operation acquires when we use an ACR and are effectively giving an inaccurate depiction of what the operation actually did.&lt;/li&gt;
		&lt;/ol&gt;
		&lt;/li&gt;
		&lt;li&gt;killOp
		&lt;ol&gt;
			&lt;li&gt;This is unrelated to the locker.&lt;/li&gt;
		&lt;/ol&gt;
		&lt;/li&gt;
		&lt;li&gt;killSession
		&lt;ol&gt;
			&lt;li&gt;This is unrelated to the locker.&lt;/li&gt;
		&lt;/ol&gt;
		&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;The biggest change appears to be the lock stats becoming more accurate, which would be a win.&lt;/p&gt;

&lt;p&gt;This ends up being a mixed bag. I&apos;d love for some outside input on Siyuan&apos;s more general concern. In the meantime I will actually do the brief audit this ticket requested.&lt;/p&gt;</comment>
                            <comment id="2273340" author="siyuan.zhou@10gen.com" created="Wed, 5 Jun 2019 23:28:26 +0000"  >&lt;p&gt;I think the existing one to one mapping among Client -&amp;gt; OperationContext -&amp;gt; Locker is a fundamental assumption. &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40498&quot; title=&quot;Writing transaction oplog entries must not take locks while holding an oplog slot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40498&quot;&gt;&lt;del&gt;SERVER-40498&lt;/del&gt;&lt;/a&gt; didn&apos;t change this assumption and just let the locker opt out the WUOW as if the locks were acquired outside WUOW. Sharing lockers across clients will break this assumption. I&apos;m wondering about &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=geert.bosch&quot; class=&quot;user-hover&quot; rel=&quot;geert.bosch&quot;&gt;geert.bosch&lt;/a&gt;&apos;s opinion on this.&lt;/p&gt;

&lt;p&gt;Conceptually, it&apos;s not clear to me why side transaction has to mask all states of the running operation context as suggested by this ticket. I feel there are more states we want to keep than stash/mask. Locker is probably not the only. The statistics of the operation, session checkout and whether the operation should conflict with PBWM lock are also bound to operation context that I can think of off the top of my head. Client seems too high-level to isolate the resources here.&lt;/p&gt;

&lt;p&gt;At the code level, when working on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40498&quot; title=&quot;Writing transaction oplog entries must not take locks while holding an oplog slot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40498&quot;&gt;&lt;del&gt;SERVER-40498&lt;/del&gt;&lt;/a&gt;, I felt the only resource we want to stash is the recovery unit / WT transaction since we may not want to or be able to use the existing recovery unit. However, since WUOW manages the lifetime of recovery unit and two-phase locking, we&apos;d have to &lt;a href=&quot;https://github.com/mongodb/mongo/blob/5c2ca8167e61c69b2f88f179f53f829243f33759/src/mongo/db/transaction_participant.cpp#L767-L779&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&quot;release&quot; the resources&lt;/a&gt;. Because of transactions, the lifetimes of recovery unit, locker, two-phase locking are not always the same any longer. If we could have RAII types to manage the lifetime of recovery unit and two-phase locking, separated from the locker itself, side transaction just needs to stash the first two. A WUOW will hold instances of those RAII types.&#160;&lt;/p&gt;</comment>
                            <comment id="2272675" author="milkie" created="Wed, 5 Jun 2019 17:18:35 +0000"  >&lt;p&gt;The code above I wrote for a section that intended to be a handler for an op being killed; the reason for using a new Client was to clear out the current killCode attached to the originalOpCtx.  If one doesn&apos;t do this, no further db locking is possible in an abort handler since the lock would check the kill code and immediately rethrow an Interrupted exception.&lt;br/&gt;
So the answer to Andy&apos;s question is, this was originally a special case where killOp was being handled with special code.  You should be aware of this if you want to use this technique in other places.&lt;/p&gt;</comment>
                            <comment id="2272438" author="judah.schvimer" created="Wed, 5 Jun 2019 15:30:38 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt; may have thought about that. I need to do further investigation and testing to find out.&lt;/p&gt;</comment>
                            <comment id="2272386" author="schwerin" created="Wed, 5 Jun 2019 15:11:13 +0000"  >&lt;p&gt;What does that do for currentOp, killOp and killSessions?&lt;/p&gt;</comment>
                            <comment id="2272373" author="judah.schvimer" created="Wed, 5 Jun 2019 15:05:20 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt; provided the following which works for a given ACR (though doesn&apos;t work in the change from &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40498&quot; title=&quot;Writing transaction oplog entries must not take locks while holding an oplog slot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40498&quot;&gt;&lt;del&gt;SERVER-40498&lt;/del&gt;&lt;/a&gt; to opt out of two phase locking for the locks already held):&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+    // Make a new operation context in order to clear the kill code, if one is present.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+    auto currentLocks = originalOpCtx-&amp;gt;swapLockState(stdx::make_unique&amp;lt;LockerImpl&amp;gt;());&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+    auto cleanupClient = originalOpCtx-&amp;gt;getServiceContext()-&amp;gt;makeClient(&quot;Index build cleanup&quot;);&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+    AlternativeClientRegion acr(cleanupClient);&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+    // After this point, cleanupClient is held inside acr and is not accessible until acr falls out&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+    // of scope.  The new Alternative Client can only be accessed via cc().&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+    auto cleanupOpCtx = cc().makeOperationContext();&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+    auto opCtx = cleanupOpCtx.get();&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+    opCtx-&amp;gt;swapLockState(std::move(currentLocks));&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+    ON_BLOCK_EXIT([&amp;amp;] {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+        auto currentLocks = opCtx-&amp;gt;swapLockState(stdx::make_unique&amp;lt;LockerImpl&amp;gt;());&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+        originalOpCtx-&amp;gt;swapLockState(std::move(currentLocks));&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+    });&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="2266862" author="judah.schvimer" created="Fri, 31 May 2019 15:38:31 +0000"  >&lt;p&gt;I don&apos;t think a PoC would be too hard. I&apos;ll do it next iteration. I knew this ticket might be bigger than originally expected.&lt;/p&gt;</comment>
                            <comment id="2266016" author="schwerin" created="Thu, 30 May 2019 20:45:56 +0000"  >&lt;p&gt;I&apos;m definitely interested in the PoC. We&apos;d need to think about how current-op reporting would look. All the work in the ACR gets assigned to a different client and op context, which might affect metrics.&lt;/p&gt;</comment>
                            <comment id="2266002" author="tess.avitabile" created="Thu, 30 May 2019 20:41:51 +0000"  >&lt;p&gt;On the other hand, having an ACR that does not share the locker is prone to self-deadlock.&#160;&lt;/p&gt;</comment>
                            <comment id="2265996" author="schwerin" created="Thu, 30 May 2019 20:39:34 +0000"  >&lt;p&gt;I&apos;d look at a PoC, but having an ACR where the alternate client shares OpCtx state (the locker) with the principal client&apos;s OpCtx seems error-prone, too. How hard to poc, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;?&lt;/p&gt;</comment>
                            <comment id="2264483" author="judah.schvimer" created="Wed, 29 May 2019 21:46:50 +0000"  >&lt;p&gt;One concern with the above plan is that it somewhat is a layering violation between the &lt;tt&gt;Locker&lt;/tt&gt; and the &lt;tt&gt;Client&lt;/tt&gt;. The &lt;tt&gt;Client&lt;/tt&gt; would have to release and restore the &lt;tt&gt;OperationContext&apos;s&lt;/tt&gt; WUOW similar to in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40498&quot; title=&quot;Writing transaction oplog entries must not take locks while holding an oplog slot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40498&quot;&gt;&lt;del&gt;SERVER-40498&lt;/del&gt;&lt;/a&gt; without unlocking the locks. This should be straightforward and is likely what users of &lt;tt&gt;AlternativeClientRegion&lt;/tt&gt; want. I suspect this would link just fine since &lt;tt&gt;client.cpp&lt;/tt&gt; and &lt;tt&gt;operation_context.cpp&lt;/tt&gt; are both in the &lt;tt&gt;service_context&lt;/tt&gt; library. Given that this prevents complicated to diagnose/fix deadlocks, I think it is worthwhile.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=siyuan.zhou&quot; class=&quot;user-hover&quot; rel=&quot;siyuan.zhou&quot;&gt;siyuan.zhou&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=schwerin&quot; class=&quot;user-hover&quot; rel=&quot;schwerin&quot;&gt;schwerin&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=geert.bosch&quot; class=&quot;user-hover&quot; rel=&quot;geert.bosch&quot;&gt;geert.bosch&lt;/a&gt;, do any of you have any thoughts on this? I can create a POC if that seems helpful.&lt;/p&gt;</comment>
                            <comment id="2264438" author="tess.avitabile" created="Wed, 29 May 2019 21:11:42 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt; and I discussed the following strategy for &lt;tt&gt;SideTransactionBlock&lt;/tt&gt;. We could get rid of &lt;tt&gt;SideTransactionBlock&lt;/tt&gt; entirely and use &lt;tt&gt;AlternativeClientRegion&lt;/tt&gt; everywhere instead. This would have the advantage that the readConcern on the &lt;tt&gt;OperationContext&lt;/tt&gt; and the &lt;tt&gt;TransactionParticipant&lt;/tt&gt; would both be fresh, so code that checks these constructs does not need to consider whether we are in a &lt;tt&gt;SideTransactionBlock&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;Due to&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40498&quot; title=&quot;Writing transaction oplog entries must not take locks while holding an oplog slot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40498&quot;&gt;&lt;del&gt;SERVER-40498&lt;/del&gt;&lt;/a&gt;, this means &lt;tt&gt;AlternativeClientRegion&lt;/tt&gt; would need to share the locker of the parent &lt;tt&gt;OperationContext&lt;/tt&gt;. This would ensure that the thread does not deadlock with itself, and that the max lock timeout for transactions is respected (since the max lock timeout for transactions should apply as long as the thread can block the transaction&apos;s progress). &lt;tt&gt;SideTransactionBlock&lt;/tt&gt; has both of these properties today.&lt;/p&gt;

&lt;p&gt;As part of this work, we could eliminate checks of &lt;tt&gt;OperationContext::getWriteUnitOfWork()&lt;/tt&gt;, since this should now return the same result as &lt;tt&gt;TransactionParticipant::inMultiDocumentTransaction()&lt;/tt&gt;. If this leads to linking issues, we could consider adding an &lt;tt&gt;OperationContext&lt;/tt&gt; flag/decoration that indicates whether the operation is a statement of a multi-document transaction (i.e. it has &lt;tt&gt;autocommit:false&lt;/tt&gt;). This might be preferable anyway, since most users probably expect this to be the semantics of &lt;tt&gt;TransactionParticipant::inMultiDocumentTransaction()&lt;/tt&gt;, when in fact that function returns false once the transaction is aborted/committed.&lt;/p&gt;</comment>
                            <comment id="2159351" author="jesse" created="Fri, 22 Feb 2019 02:29:24 +0000"  >&lt;p&gt;I attached a patch atop git version #2c65bbe9, it&apos;s a sketch of one direction we could go to fix the reentrance problem.&lt;/p&gt;</comment>
                            <comment id="2153955" author="judah.schvimer" created="Fri, 15 Feb 2019 22:03:30 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=blake.oler&quot; class=&quot;user-hover&quot; rel=&quot;blake.oler&quot;&gt;blake.oler&lt;/a&gt;, I could imagine your work looking at the transaction statements at commit time being impacted by this. The prepare oplog entry or the &lt;tt&gt;config.transactions&lt;/tt&gt; write may be added to the operations list in the prepare oplog entry &lt;tt&gt;SideTransactionBlock&lt;/tt&gt;. I have not confirmed this, however.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="729285">SERVER-40466</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="584990">SERVER-36494</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="209751" name="patch" size="3089" author="jesse@mongodb.com" created="Fri, 22 Feb 2019 02:28:41 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>18.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16775"><![CDATA[v4.2]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 15 Feb 2019 22:03:30 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 34 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1032</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 34 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>jesse@mongodb.com</customfieldvalue>
            <customfieldvalue>schwerin@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>siyuan.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hunr4v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr84uf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2999">Repl 2019-06-03</customfieldvalue>
    <customfieldvalue id="3000">Repl 2019-06-17</customfieldvalue>
    <customfieldvalue id="3001">Repl 2019-07-01</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hunde7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>