<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:57:27 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-85325] Classify `OpCtx` decorations based on their performance cost</title>
                <link>https://jira.mongodb.org/browse/SERVER-85325</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The server makes a new &lt;tt&gt;OperationContext&lt;/tt&gt; for each user operation. Instances of &lt;tt&gt;OperationContext&lt;/tt&gt; are decorated, and making and destroying these decorations is expensive. This ticket should go over the list of all decorations on &lt;tt&gt;OperationContext&lt;/tt&gt; and classify them into the following categories:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Trivially constructed and destructed, and already benefit from &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-76788&quot; title=&quot;Elide useless constructor calls for trivially-constructible Decorations&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-76788&quot;&gt;&lt;del&gt;SERVER-76788&lt;/del&gt;&lt;/a&gt;.&lt;/li&gt;
	&lt;li&gt;Can be made cheaper by either making construction/destruction trivial, reusing decorations across multiple operations, simplifying the constructor/destructor, etc.&lt;/li&gt;
	&lt;li&gt;Other (i.e. decorations with complex construction/destruction logic that cannot be easily simplified).&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Once we have this list, we can target the second and third groups by:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Filing a ticket to simplify decorations that belong to the second group.&lt;/li&gt;
	&lt;li&gt;File tickets for decorations that belong to the third group and assign it to the corresponding server team.&lt;/li&gt;
&lt;/ul&gt;
</description>
                <environment></environment>
        <key id="2548103">SERVER-85325</key>
            <summary>Classify `OpCtx` decorations based on their performance cost</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="patrick.freed@mongodb.com">Patrick Freed</assignee>
                                    <reporter username="amirsaman.memaripour@mongodb.com">Amirsaman Memaripour</reporter>
                        <labels>
                    </labels>
                <created>Wed, 17 Jan 2024 19:05:09 +0000</created>
                <updated>Sat, 27 Jan 2024 01:13:51 +0000</updated>
                            <resolved>Sat, 27 Jan 2024 01:13:51 +0000</resolved>
                                                                    <component>Internal Code</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="6046342" author="patrick.freed" created="Sat, 27 Jan 2024 01:13:51 +0000"  >&lt;p&gt;Tickets have been filed, closing this out.&lt;/p&gt;</comment>
                            <comment id="6032691" author="patrick.freed" created="Mon, 22 Jan 2024 23:12:35 +0000"  >&lt;p&gt;I went and looked through the default constructors for all of OperationContexts decorations, and it doesn&apos;t look like any of them contain complex initialization logic. &lt;tt&gt;OperationTimeTrackerHolder&lt;/tt&gt; &apos;s constructor heap allocates an instance of a class that contains a mutex and a long, which could potentially be expensive if done on every operation, but other than that, the constructors are mostly just default and the member fields have simple or trivial constructors (e.g. integers, booleans, vectors, optionals).&lt;/p&gt;

&lt;p&gt;The one notable decoration is CurOpStack, which by size is far larger than any other decoration on OperationContext. In fact, according &lt;a href=&quot;https://jira.mongodb.org/browse/PERF-4731?focusedCommentId=5714086&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-5714086&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;to this comment&lt;/a&gt;, it constitutes over half of the entire decoration buffer. Its constructor also involves an extra small heap allocation. It would seem that the best avenue for improvement here would be to reduce the size of CurOpStack, perhaps by delegating non-required parts of it to separate allocations that we only do on a subset of operations or just by moving some of its state elsewhere. &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-81848&quot; title=&quot;Make decoration construction eager again&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-81848&quot;&gt;&lt;del&gt;SERVER-81848&lt;/del&gt;&lt;/a&gt; has a note about this and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alex.li%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;alex.li@mongodb.com&quot;&gt;alex.li@mongodb.com&lt;/a&gt; mentioned that he&apos;ll be looking into it in a comment on that ticket, so I don&apos;t think anything separate needs to be filed at this time.&lt;/p&gt;

&lt;p&gt;I would be curious to see a flamegraph or some benchmarking results that point to OperationContext decorations as being a performance bottleneck, since my initial look into it suggests that they shouldn&apos;t be overly expensive. The one exception could be CurOpStack, though even that is basically just a 3.5kb allocation, which doesn&apos;t seem egregious.&lt;/p&gt;

&lt;p&gt;Here&apos;s the full list of decorations and some notes about them:&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Trivial constructors/destructors&lt;/b&gt;&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;src/mongo/s/catalog_cache.cpp:106: bool&lt;/li&gt;
	&lt;li&gt;src/mongo/db/repl/always_allow_non_local_writes.cpp:40: bool&lt;/li&gt;
	&lt;li&gt;src/mongo/db/s/scoped_operation_completion_sharding_actions.cpp:69: bool&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&lt;b&gt;Simple / &quot;empty&quot; constructors&lt;/b&gt;&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;src/mongo/db/operation_cpu_timer.cpp:173: std::shared_ptr&amp;lt;OperationCPUTimers&amp;gt;&lt;/li&gt;
	&lt;li&gt;src/mongo/db/pipeline/javascript_execution.cpp:43: std::unique_ptr&amp;lt;JsExecution&amp;gt;&lt;/li&gt;
	&lt;li&gt;src/mongo/db/commands.cpp:634: boost::optional&amp;lt;BSONArray&amp;gt;&lt;/li&gt;
	&lt;li&gt;src/mongo/db/commands.cpp:886: std::shared_ptr&amp;lt;CommandInvocation&amp;gt;&lt;/li&gt;
	&lt;li&gt;src/mongo/db/session/session_catalog.cpp:67: boost::optional&amp;lt;SessionCatalog::ScopedCheckedOutSession&amp;gt;&lt;/li&gt;
	&lt;li&gt;src/mongo/db/serverless/shard_split_donor_op_observer.cpp:69: boost::optional&amp;lt;SplitCleanupDetails&amp;gt;&lt;/li&gt;
	&lt;li&gt;src/mongo/db/repl/tenant_migration_decoration.cpp:39: boost::optional&amp;lt;TenantMigrationInfo&amp;gt;&lt;/li&gt;
	&lt;li&gt;src/mongo/db/s/transaction_coordinator_worker_curop_repository_mongod.cpp:77: boost::optional&amp;lt;TransactionCoordinatorWorkerCurOpInfo&amp;gt;&lt;/li&gt;
	&lt;li&gt;src/mongo/db/query/plan_executor_impl.cpp:91: boost::optional&amp;lt;repl::OpTime&amp;gt;&lt;/li&gt;
	&lt;li&gt;src/mongo/db/query/plan_executor.cpp:52: boost::optional&amp;lt;SharedSemiFuture&amp;lt;void&amp;gt;&amp;gt;&lt;/li&gt;
	&lt;li&gt;src/mongo/db/auth/validated_tenancy_scope.cpp:48: boost::optional&amp;lt;ValidatedTenancyScope&amp;gt;&lt;/li&gt;
	&lt;li&gt;src/mongo/db/transaction_resources.cpp:52: std::unique_ptr&amp;lt;shard_role_details::TransactionResources&amp;gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&lt;b&gt;Cheap but non-trivial/non-empty constructors&lt;/b&gt;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;src/mongo/rpc/metadata/client_metadata.cpp:97: ClientMetadataState&lt;br/&gt;
&#160; - bool, optional&amp;lt;ClientMetadata&amp;gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;src/mongo/rpc/rewrite_state_change_errors.cpp:77: RewriteEnabled&lt;br/&gt;
&#160; - one bool field&lt;/li&gt;
	&lt;li&gt;src/mongo/rpc/metadata/impersonated_user_metadata.cpp:57: synchronized_value&amp;lt;MaybeImpersonatedUserMetadata&amp;gt;&lt;br/&gt;
&#160; - MaybeImpersonatedUserMetadata is a boost::optional&lt;br/&gt;
&#160; - synchronized_value contains a mutex too&lt;/li&gt;
	&lt;li&gt;src/mongo/rpc/metadata/tracking_metadata.cpp:57: TrackingMetadata&lt;br/&gt;
&#160; - a couple of boost::optionals and a bool&lt;/li&gt;
	&lt;li&gt;src/mongo/db/prepare_conflict_tracker.cpp:42: PrepareConflictTracker&lt;br/&gt;
&#160; - two atomics and a tick&lt;/li&gt;
	&lt;li&gt;src/mongo/db/api_parameters.cpp:45: APIParameters&lt;br/&gt;
&#160; - 3 boost::optionals&lt;/li&gt;
	&lt;li&gt;src/mongo/db/storage/storage_engine_change_context.cpp:83: StorageEngineChangeOperationContextDoneNotifier&lt;br/&gt;
&#160; - condvar, mutex, int&lt;/li&gt;
	&lt;li&gt;src/mongo/db/repl/speculative_majority_read_info.cpp:53: SpeculativeMajorityReadInfo&lt;br/&gt;
&#160; - bool, boost::optional&lt;/li&gt;
	&lt;li&gt;src/mongo/db/repl/repl_client_info.cpp:63: LastOpInfo&lt;br/&gt;
&#160; - one bool&lt;/li&gt;
	&lt;li&gt;src/mongo/db/query/query_decorations.cpp:35: QueryKnobConfiguration&lt;br/&gt;
&#160; - couple of bools and size_ts&lt;/li&gt;
	&lt;li&gt;src/mongo/db/query/find_common.cpp:72: AwaitDataState&lt;br/&gt;
&#160; - one date_t and one bool&lt;/li&gt;
	&lt;li&gt;src/mongo/db/write_block_bypass.cpp:43: WriteBlockBypass&lt;br/&gt;
&#160; - one bool&lt;/li&gt;
	&lt;li&gt;src/mongo/client/read_preference.cpp:88: ReadPreferenceSetting&lt;br/&gt;
&#160; - read preference (enum + empty BSONObj)&lt;br/&gt;
&#160; - tags (empty BSONObj)&lt;br/&gt;
&#160; - Seconds&lt;br/&gt;
&#160; - boost::optional&amp;lt;HedgingMode&amp;gt;&lt;br/&gt;
&#160; - bool&lt;/li&gt;
	&lt;li&gt;src/mongo/db/catalog/document_validation.cpp:36: DocumentValidationSettings&lt;br/&gt;
&#160; - one uint8_t&lt;/li&gt;
	&lt;li&gt;src/mongo/db/modules/enterprise/src/audit/audit_deduplicatoin.h:45: AuditDeduplication&lt;br/&gt;
&#160; - one bool&lt;/li&gt;
	&lt;li&gt;src/mongo/db/storage/execution_context.cpp:38: StorageExecutionContext&lt;br/&gt;
&#160; - two sets, a small_vector, and a vector&lt;/li&gt;
	&lt;li&gt;src/mongo/db/op_observer/batched_write_context.cpp:44: BatchedWriteContext&lt;br/&gt;
&#160; - vector, set, two size_ts&lt;br/&gt;
&#160; - does have multiple layers of indirection though, so a couple of function calls&lt;/li&gt;
	&lt;li&gt;src/mongo/db/op_observer/op_observer.cpp:39: OpObserver::Times&lt;br/&gt;
&#160; - vector and an int&lt;/li&gt;
	&lt;li&gt;src/mongo/db/multi_key_path_tracker.cpp:54: MultikeyPathTracker&lt;br/&gt;
&#160; - a vector and a bool&lt;/li&gt;
	&lt;li&gt;src/mongo/db/stats/resource_consumption_metrics.cpp:51: ResourceConsumption::MetricsCollector&lt;br/&gt;
&#160; - medium amount of metrics, but again lots of small stuff like integers&lt;/li&gt;
	&lt;li&gt;src/mongo/db/s/operation_sharding_state.cpp:60: OperationShardingState&lt;br/&gt;
&#160; - couple of maps, some bools&lt;/li&gt;
	&lt;li&gt;src/mongo/db/repl/read_concern_args.cpp:58: ReadConcernArgs&lt;br/&gt;
&#160; - has a bunch of small fields&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&lt;b&gt;Potentially expensive constructors, could be made cheaper&lt;/b&gt;&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;src/mongo/db/operation_time_tracker.cpp:48: OperationTimeTrackerHolder&lt;br/&gt;
&#160; - does heap allocation of mutex and a long&lt;/li&gt;
	&lt;li&gt;src/mongo/db/curop.cpp:210: CurOp::CurOpStack&lt;br/&gt;
&#160; - tons of fields, although most are atomics integers or similar&lt;br/&gt;
&#160; - also involves a heap allocation of a stats member&lt;/li&gt;
&lt;/ul&gt;
</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="2550572">SERVER-85411</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2458965">SERVER-81848</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2330945">SERVER-76788</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="2559099">SERVER-85829</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2559098">SERVER-85828</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2559103">SERVER-85830</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2559107">SERVER-85831</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2559108">SERVER-85832</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2559109">SERVER-85833</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2559111">SERVER-85834</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="506488" name="decoration-dtor.png" size="127143" author="patrick.freed@mongodb.com" created="Thu, 25 Jan 2024 00:32:47 +0000"/>
                            <attachment id="506486" name="decorations-by-binary.txt" size="2281" author="patrick.freed@mongodb.com" created="Thu, 25 Jan 2024 00:31:31 +0000"/>
                            <attachment id="506487" name="dtor-timings.txt" size="9331" author="patrick.freed@mongodb.com" created="Thu, 25 Jan 2024 00:31:31 +0000"/>
                            <attachment id="506199" name="invisible-sharding-ycsb-flamegraphs.zip" size="6346039" author="jack.mulrow@mongodb.com" created="Wed, 24 Jan 2024 15:37:17 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 22 Jan 2024 23:12:35 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 week, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-3598</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>patrick.freed@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 week, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>amirsaman.memaripour@mongodb.com</customfieldvalue>
            <customfieldvalue>patrick.freed@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i38lun:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i2qb4s:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_22250" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Special Downgrade Instructions Required</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="23343"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="8039">Service Arch 2024-01-22</customfieldvalue>
    <customfieldvalue id="8040">Service Arch 2024-02-05</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i387zz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>