<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:23:53 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-30460] Grow the oplog when the replication commit point falls behind the back of where it would normally be truncated to</title>
                <link>https://jira.mongodb.org/browse/SERVER-30460</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;If the replication majority commit point is far enough behind the primary that the oplog entry on the primary that corresponds to the majority point falls to the back of its oplog and gets deleted, this can cause a majority of the secondaries to become too stale to continue replication, and thus require a full resync.  Having a majority of a set&apos;s nodes being in the process of initial sync would mean that there&apos;s no healthy majority to elect a primary, thus the set would then have a prolonged period of no write availability.&lt;/p&gt;

&lt;p&gt;One possible mitigation for this problem is to prevent the primary from deleting ops from its oplog that are ahead or equal to the replication commit point, so that there will always be a common point between the oplogs of the majority of the secondaries and the primary.&lt;/p&gt;</description>
                <environment></environment>
        <key id="411167">SERVER-30460</key>
            <summary>Grow the oplog when the replication commit point falls behind the back of where it would normally be truncated to</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="pavithra.vetriselvan@mongodb.com">Pavithra Vetriselvan</assignee>
                                    <reporter username="spencer@mongodb.com">Spencer Brody</reporter>
                        <labels>
                            <label>prepare_durability</label>
                    </labels>
                <created>Tue, 1 Aug 2017 17:51:17 +0000</created>
                <updated>Mon, 30 Oct 2023 23:14:41 +0000</updated>
                            <resolved>Thu, 3 Jan 2019 15:48:22 +0000</resolved>
                                                    <fixVersion>4.1.4</fixVersion>
                                    <component>Replication</component>
                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>12</watches>
                                                                                                                <comments>
                            <comment id="2104304" author="pavithra.vetriselvan" created="Thu, 3 Jan 2019 15:48:22 +0000"  >&lt;p&gt;Closing this ticket because &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36811&quot; title=&quot;Provide a mechanism for replication to specify the &amp;#39;maximum_truncation_timestamp&amp;#39; for a given &amp;#39;stable_timestamp&amp;#39;&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36811&quot;&gt;&lt;del&gt;SERVER-36811&lt;/del&gt;&lt;/a&gt; should have solved the problem of nodes truncating oplog entries ahead of the commit point since the stableTimestamp will be less than or equal to the replication commit point. &lt;/p&gt;

&lt;p&gt;This issue is only fixed when enableMajorityReadConcern=true.&lt;/p&gt;</comment>
                            <comment id="2103225" author="tess.avitabile" created="Wed, 2 Jan 2019 16:23:59 +0000"  >&lt;p&gt;When resolving this ticket, please clarify that the ticket is only fixed when enableMajorityReadConcern=true.&lt;/p&gt;</comment>
                            <comment id="2103073" author="daniel.gottlieb@10gen.com" created="Wed, 2 Jan 2019 14:34:03 +0000"  >&lt;p&gt;I might be confused, apologies for the clarification request. It sounded like this ticket was being repurposed to ensure the oplog has enough history for recovery when there are in-flight prepared transactions (prepared behind the commit point):&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;As Andy was saying, it seems like we will hit this problem when a prepared but uncommitted/unaborted transaction stays in prepare for a while.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;When the stable timestamp is allowed to move in front of prepared transactions, there would still be a required change for preventing oplog truncation from deleting these in-flight prepared oplog entries. Perhaps that&apos;s being tracked elsewhere (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36494&quot; title=&quot;Prevent oplog truncation of oplog entries needed for startup recovery&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36494&quot;&gt;&lt;del&gt;SERVER-36494&lt;/del&gt;&lt;/a&gt;?), I was really just responding to:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;If we were to not use the functionality provided by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22766&quot; title=&quot;Dynamic oplog sizing for WiredTiger nodes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22766&quot;&gt;&lt;del&gt;SERVER-22766&lt;/del&gt;&lt;/a&gt; and instead specified a Timestamp to an oplog entry that we can truncate, is there work that storage would need to complete first?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;As far as we know, there&apos;s no more storage work required for that.&lt;/p&gt;</comment>
                            <comment id="2102845" author="judah.schvimer" created="Wed, 2 Jan 2019 01:35:49 +0000"  >&lt;p&gt;Since the &lt;tt&gt;stableTimestamp&lt;/tt&gt; is less than or equal to the replication commit point, it seems that this ticket (preventing nodes from truncating newer oplog entries than the commit point) was accomplished in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36811&quot; title=&quot;Provide a mechanism for replication to specify the &amp;#39;maximum_truncation_timestamp&amp;#39; for a given &amp;#39;stable_timestamp&amp;#39;&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36811&quot;&gt;&lt;del&gt;SERVER-36811&lt;/del&gt;&lt;/a&gt; regardless of what the &lt;tt&gt;truncationTimestamp&lt;/tt&gt; is set to. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt;, do you agree?&lt;/p&gt;</comment>
                            <comment id="2097542" author="daniel.gottlieb@10gen.com" created="Thu, 20 Dec 2018 19:13:26 +0000"  >&lt;p&gt;If the only concern is for keeping around prepared, uncommitted oplog entries, support for that was put into storage with &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36811&quot; title=&quot;Provide a mechanism for replication to specify the &amp;#39;maximum_truncation_timestamp&amp;#39; for a given &amp;#39;stable_timestamp&amp;#39;&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36811&quot;&gt;&lt;del&gt;SERVER-36811&lt;/del&gt;&lt;/a&gt;. &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36494&quot; title=&quot;Prevent oplog truncation of oplog entries needed for startup recovery&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36494&quot;&gt;&lt;del&gt;SERVER-36494&lt;/del&gt;&lt;/a&gt; is the repl ticket for tracking the earliest oplog entry that storage must preserve in order to recover from a given stable timestamp.&lt;/p&gt;

&lt;p&gt;A clarification: the Storage API provided with &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36811&quot; title=&quot;Provide a mechanism for replication to specify the &amp;#39;maximum_truncation_timestamp&amp;#39; for a given &amp;#39;stable_timestamp&amp;#39;&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36811&quot;&gt;&lt;del&gt;SERVER-36811&lt;/del&gt;&lt;/a&gt; is really to express the more general contract:&lt;br/&gt;
&lt;tt&gt;setStableTimestamp(stableTimestamp, truncationTimestamp)&lt;/tt&gt;&lt;br/&gt;
To recover forward in the oplog starting at the &lt;tt&gt;stableTimestamp&lt;/tt&gt;, storage is guaranteeing to preserve all oplog dating back to &lt;tt&gt;min(stableTimestamp, truncationTimestamp)&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;Storage doesn&apos;t require that the &lt;tt&gt;truncationTimestamp&lt;/tt&gt; is in any way related to a prepared transaction.&lt;/p&gt;</comment>
                            <comment id="2097468" author="pavithra.vetriselvan" created="Thu, 20 Dec 2018 18:37:20 +0000"  >&lt;p&gt;As Andy was saying, it seems like we will hit this problem when a prepared but uncommitted/unaborted transaction stays in prepare for a while. Did we ever reach a conclusion on allowing the oplog to grow unbounded in size or is that solved by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22766&quot; title=&quot;Dynamic oplog sizing for WiredTiger nodes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22766&quot;&gt;&lt;del&gt;SERVER-22766&lt;/del&gt;&lt;/a&gt; (I guess by continuing to resize the oplog)? &lt;/p&gt;

&lt;p&gt;If we were to not use the functionality provided by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22766&quot; title=&quot;Dynamic oplog sizing for WiredTiger nodes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22766&quot;&gt;&lt;del&gt;SERVER-22766&lt;/del&gt;&lt;/a&gt; and instead specified a Timestamp to an oplog entry that we can truncate, is there work that storage would need to complete first?  &lt;/p&gt;</comment>
                            <comment id="2009690" author="judah.schvimer" created="Thu, 20 Sep 2018 19:26:02 +0000"  >&lt;p&gt;Given the current design of &quot;prepare&quot; I want to re-investigate what is actually required by this ticket.&lt;/p&gt;</comment>
                            <comment id="1645608" author="schwerin" created="Thu, 10 Aug 2017 20:55:59 +0000"  >&lt;p&gt;I don&apos;t believe &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22766&quot; title=&quot;Dynamic oplog sizing for WiredTiger nodes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22766&quot;&gt;&lt;del&gt;SERVER-22766&lt;/del&gt;&lt;/a&gt; provides the requisite functionality. What replication needs is to specify the timestamp of the newest oplog entry that may be truncated, rather than the new maximum size of the oplog. That seems at least partially like storage work. I believe it will be required in 3.8, to support prepared but uncommitted transactions that sit in prepare for a long time. It can stay on the repl backlog, but we will need to collaborate on the design.&lt;/p&gt;</comment>
                            <comment id="1641543" author="michael.cahill" created="Mon, 7 Aug 2017 01:03:28 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=schwerin&quot; class=&quot;user-hover&quot; rel=&quot;schwerin&quot;&gt;schwerin&lt;/a&gt;, the storage-level mechanism was added in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22766&quot; title=&quot;Dynamic oplog sizing for WiredTiger nodes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22766&quot;&gt;&lt;del&gt;SERVER-22766&lt;/del&gt;&lt;/a&gt;.  I think the decision about when to resize (i.e., this ticket) belongs to repl.  Let me know if you disagree or need more from storage to support this work.&lt;/p&gt;</comment>
                            <comment id="1641495" author="daniel.gottlieb@10gen.com" created="Sun, 6 Aug 2017 22:52:16 +0000"  >&lt;p&gt;Do we just want to allow the oplog to grow unbounded in size?&lt;/p&gt;</comment>
                            <comment id="1641485" author="schwerin" created="Sun, 6 Aug 2017 21:52:46 +0000"  >&lt;p&gt;We&apos;re going to have to do this eventually, and then some. Multi-shard transactions will require that the oldest operation of any transaction on a shard in the &quot;prepared&quot; state must remain in the oplog until the transaction is committed or aborted by the coordinator.&lt;/p&gt;</comment>
                            <comment id="1641483" author="daniel.gottlieb@10gen.com" created="Sun, 6 Aug 2017 21:48:30 +0000"  >&lt;p&gt;It wasn&apos;t mandatory, but there was a ticket filed. &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-29215&quot; title=&quot;Coordinate oplog truncate point with checkpoint timestamp&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-29215&quot;&gt;&lt;del&gt;SERVER-29215&lt;/del&gt;&lt;/a&gt; &lt;/p&gt;</comment>
                            <comment id="1641480" author="schwerin" created="Sun, 6 Aug 2017 21:36:02 +0000"  >&lt;p&gt;Somehow, I thought this was part of the mandatory work for the WT engine for one of the storage timestamps projects. Is there not a ticket already filed to this effect, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=michael.cahill&quot; class=&quot;user-hover&quot; rel=&quot;michael.cahill&quot;&gt;michael.cahill&lt;/a&gt;?&lt;/p&gt;</comment>
                            <comment id="1637854" author="daniel.gottlieb@10gen.com" created="Tue, 1 Aug 2017 18:49:21 +0000"  >&lt;p&gt;Straightforward for WiredTiger*&lt;/p&gt;</comment>
                            <comment id="1637826" author="spencer" created="Tue, 1 Aug 2017 18:34:42 +0000"  >&lt;p&gt;I think I&apos;m going to leave this ticket parked with the replication team for now since I think ultimately it&apos;s us who needs to decide what (if anything) we want to do about lagged secondaries.  If we decide this is something we want to do, I will send this over to the storage team for the actual implementation, which according to &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt; is pretty straightforward.&lt;/p&gt;</comment>
                            <comment id="1637775" author="milkie" created="Tue, 1 Aug 2017 17:56:49 +0000"  >&lt;p&gt;This might be easier for storage to do, since we could just amend the capped deleter to not delete anything newer than the oldest timestamp (which will be set when setCommittedSnapshot is called).&lt;/p&gt;</comment>
                            <comment id="1637768" author="spencer" created="Tue, 1 Aug 2017 17:52:41 +0000"  >&lt;p&gt;The alternative to this is to provide backpressure: slowing or failing writes on the primary when the commit point is in danger of falling off its oplog.&lt;/p&gt;</comment>
                            <comment id="1637766" author="spencer" created="Tue, 1 Aug 2017 17:51:42 +0000"  >&lt;p&gt;If we did this we would have to figure out whether we let the oplog grow unbounded (up to the limits of available storage space), or give it some other limit, though any other limit feels arbitrary since we&apos;re already letting the oplog exceed the max size the user has configured it to have.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="383661">SERVER-29215</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="382736">SERVER-29125</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="266840">SERVER-22766</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="584990">SERVER-36494</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>18.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 1 Aug 2017 17:56:49 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 5 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1032</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 5 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>schwerin@mongodb.com</customfieldvalue>
            <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
            <customfieldvalue>pavithra.vetriselvan@mongodb.com</customfieldvalue>
            <customfieldvalue>spencer@mongodb.com</customfieldvalue>
            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htc3mv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hu1uyn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2605">Repl 2018-11-19</customfieldvalue>
    <customfieldvalue id="2606">Repl 2018-12-03</customfieldvalue>
    <customfieldvalue id="2607">Repl 2018-12-17</customfieldvalue>
    <customfieldvalue id="2701">Repl 2019-01-14</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htbppj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>