<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:21:42 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-9879] MutexDebugger detected a reverse mutex acquisition order</title>
                <link>https://jira.mongodb.org/browse/SERVER-9879</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Currently not a real threat because the 2 different orders of lock acquisition doesn&apos;t cross paths. Here&apos;s the detailed report from the user (source: &lt;a href=&quot;https://groups.google.com/forum/?hl=en-US&amp;amp;fromgroups=#!topic/mongodb-dev/g1BULoCq1ck):&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://groups.google.com/forum/?hl=en-US&amp;amp;fromgroups=#!topic/mongodb-dev/g1BULoCq1ck):&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I cherry-picked 2d0f6cc onto v2.2, and added some extra debug printing in&lt;br/&gt;
this branch:&lt;br/&gt;
&lt;a href=&quot;https://github.com/leifwalsh/mongo/compare/mongodb:v2.2...SERVER-6972-backport&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/leifwalsh/mongo/compare/mongodb:v2.2...SERVER-6972-backport&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I also wrote a perl script to understand the locking and show the first&lt;br/&gt;
and second stacktraces for the first observation of each lock ordering:&lt;br/&gt;
&lt;a href=&quot;https://gist.github.com/5732226&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/5732226&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This script parses the output of `./mongo --nodb&lt;br/&gt;
jstests/sharding/remove2.js` and the first time a thread establishes an&lt;br/&gt;
ordering between locks A and B, it shows the most recent stack traces&lt;br/&gt;
where A was locked, and the current stacktrace where we&apos;re locking B.&lt;br/&gt;
Hopefully you can reproduce this if you try, but here is what I got:&lt;br/&gt;
&lt;a href=&quot;https://gist.github.com/5732253&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/5732253&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If I addr2line these, I get these stacks for the establishment of the&lt;br/&gt;
ordering ReplicaSetMonitor before DBConnectionPool (which I understand&lt;br/&gt;
from your past emails to be the invalid one):&lt;/p&gt;

&lt;p&gt;ReplicaSetMonitor is taken at&lt;/p&gt;

&lt;p&gt;mongo::printStackTrace(std::ostream&amp;amp;) at /home/leif/git/mongo-vanilla/src/mongo/util/stacktrace.cpp:39&lt;br/&gt;
mongo::MutexDebugger::entering(char const*) at /home/leif/git/mongo-vanilla/src/mongo/util/concurrency/mutexdebugger.h:77&lt;br/&gt;
mongo::mutex::scoped_lock::scoped_lock(mongo::mutex&amp;amp;) at /home/leif/git/mongo-vanilla/src/mongo/util/concurrency/mutex.h:108&lt;br/&gt;
mongo::ReplicaSetMonitor::remove(std::string const&amp;amp;, bool) at /home/leif/git/mongo-vanilla/src/mongo/client/dbclient_rs.cpp:338&lt;br/&gt;
mongo::dbgrid_cmds::RemoveShardCmd::run(std::string const&amp;amp;, mongo::BSONObj&amp;amp;, int, std::string&amp;amp;, mongo::BSONObjBuilder&amp;amp;, bool) at /home/leif/git/mongo-vanilla/src/mongo/s/commands_admin.cpp:1072&lt;br/&gt;
mongo::Command::runAgainstRegistered(char const*, mongo::BSONObj&amp;amp;, mongo::BSONObjBuilder&amp;amp;, int) at /home/leif/git/mongo-vanilla/src/mongo/s/commands_public.cpp:1665 (discriminator 1)&lt;br/&gt;
mongo::SingleStrategy::queryOp(mongo::Request&amp;amp;) at /home/leif/git/mongo-vanilla/src/mongo/s/strategy_single.cpp:55 (discriminator 1)&lt;br/&gt;
mongo::ShardStrategy::queryOp(mongo::Request&amp;amp;) at /home/leif/git/mongo-vanilla/src/mongo/s/strategy_shard.cpp:49&lt;br/&gt;
mongo::Request::process(int) at /home/leif/git/mongo-vanilla/src/mongo/s/request.cpp:140&lt;br/&gt;
mongo::ShardedMessageHandler::process(mongo::Message&amp;amp;, mongo::AbstractMessagingPort*, mongo::LastError*) at /home/leif/git/mongo-vanilla/src/mongo/s/server.cpp:108&lt;br/&gt;
mongo::pms::threadRun(mongo::MessagingPort*) at /home/leif/git/mongo-vanilla/src/mongo/util/net/message_server_port.cpp:86&lt;/p&gt;

&lt;p&gt;and then DBConnectionPool is taken at&lt;/p&gt;

&lt;p&gt;mongo::printStackTrace(std::ostream&amp;amp;) at /home/leif/git/mongo-vanilla/src/mongo/util/stacktrace.cpp:39&lt;br/&gt;
mongo::MutexDebugger::entering(char const*) at /home/leif/git/mongo-vanilla/src/mongo/util/concurrency/mutexdebugger.h:77&lt;br/&gt;
mongo::mutex::scoped_lock::scoped_lock(mongo::mutex&amp;amp;) at /home/leif/git/mongo-vanilla/src/mongo/util/concurrency/mutex.h:108&lt;br/&gt;
mongo::DBConnectionPool::removeHost(std::string const&amp;amp;) at /home/leif/git/mongo-vanilla/src/mongo/client/connpool.cpp:251&lt;br/&gt;
mongo::ReplicaSetMonitor::~ReplicaSetMonitor() at /home/leif/git/mongo-vanilla/src/mongo/client/dbclient_rs.cpp:258 (discriminator 1)&lt;br/&gt;
void boost::checked_delete&amp;lt;mongo::ReplicaSetMonitor&amp;gt;(mongo::ReplicaSetMonitor*) at /home/leif/git/mongo-vanilla/src/third_party/boost/boost/checked_delete.hpp:39 (discriminator 1)&lt;br/&gt;
boost::detail::sp_counted_impl_p&amp;lt;mongo::ReplicaSetMonitor&amp;gt;::dispose() at /home/leif/git/mongo-vanilla/src/third_party/boost/boost/smart_ptr/detail/sp_counted_impl.hpp:79&lt;br/&gt;
boost::detail::sp_counted_base::release() at /home/leif/git/mongo-vanilla/src/third_party/boost/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146&lt;br/&gt;
boost::detail::shared_count::~shared_count() at /home/leif/git/mongo-vanilla/src/third_party/boost/boost/smart_ptr/detail/shared_count.hpp:309&lt;br/&gt;
boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt;::~shared_ptr() at /home/leif/git/mongo-vanilla/src/third_party/boost/boost/smart_ptr/shared_ptr.hpp:164&lt;br/&gt;
std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt;::~pair() at /usr/include/c++/4.8.1/bits/stl_pair.h:96 (discriminator 1)&lt;br/&gt;
__gnu_cxx::new_allocator&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;::destroy(std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt;*) at /usr/include/c++/4.8.1/ext/new_allocator.h:133&lt;br/&gt;
std::_Rb_tree&amp;lt;std::string, std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt;, std::_Select1st&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;, std::less&amp;lt;std::string&amp;gt;, std::allocator&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt; &amp;gt;::_M_destroy_node(std::_Rb_tree_node&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;*) at /usr/include/c++/4.8.1/bits/stl_tree.h:395 (discriminator 1)&lt;br/&gt;
std::_Rb_tree&amp;lt;std::string, std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt;, std::_Select1st&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;, std::less&amp;lt;std::string&amp;gt;, std::allocator&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt; &amp;gt;::_M_erase_aux(std::_Rb_tree_const_iterator&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;) at /usr/include/c++/4.8.1/bits/stl_tree.h:1744&lt;br/&gt;
std::_Rb_tree&amp;lt;std::string, std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt;, std::_Select1st&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;, std::less&amp;lt;std::string&amp;gt;, std::allocator&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt; &amp;gt;::erase(std::_Rb_tree_const_iterator&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;) at /usr/include/c++/4.8.1/bits/stl_tree.h:828&lt;br/&gt;
std::_Rb_tree&amp;lt;std::string, std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt;, std::_Select1st&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;, std::less&amp;lt;std::string&amp;gt;, std::allocator&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt; &amp;gt;::_M_erase_aux(std::_Rb_tree_const_iterator&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;, std::_Rb_tree_const_iterator&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;) at /usr/include/c++/4.8.1/bits/stl_tree.h:1756 (discriminator 1)&lt;br/&gt;
std::_Rb_tree&amp;lt;std::string, std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt;, std::_Select1st&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;, std::less&amp;lt;std::string&amp;gt;, std::allocator&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt; &amp;gt;::erase(std::_Rb_tree_iterator&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;, std::_Rb_tree_iterator&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;) at /usr/include/c++/4.8.1/bits/stl_tree.h:845&lt;br/&gt;
std::_Rb_tree&amp;lt;std::string, std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt;, std::_Select1st&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;, std::less&amp;lt;std::string&amp;gt;, std::allocator&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt; &amp;gt;::erase(std::string const&amp;amp;) at /usr/include/c++/4.8.1/bits/stl_tree.h:1769&lt;br/&gt;
std::map&amp;lt;std::string, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt;, std::less&amp;lt;std::string&amp;gt;, std::allocator&amp;lt;std::pair&amp;lt;std::string const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt; &amp;gt;::erase(std::string const&amp;amp;) at /usr/include/c++/4.8.1/bits/stl_map.h:726&lt;br/&gt;
mongo::ReplicaSetMonitor::_remove_inlock(std::string const&amp;amp;, bool) at /home/leif/git/mongo-vanilla/src/mongo/client/dbclient_rs.cpp:344&lt;/p&gt;

&lt;p&gt;It seems pretty simple.  ReplicaSetMonitor::remove holds the lock&lt;br/&gt;
ReplicaSetMonitor mutex and erases a ReplicaSetMonitorPtr from the map,&lt;br/&gt;
which calls ~ReplicaSetMonitor.  In the destructor, we call&lt;br/&gt;
DBConnectionPool::removeHost, which promptly grabs the DBConnectionPool&lt;br/&gt;
mutex.&lt;/p&gt;&lt;/blockquote&gt;</description>
                <environment></environment>
        <key id="78308">SERVER-9879</key>
            <summary>MutexDebugger detected a reverse mutex acquisition order</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="kaloian.manassiev@mongodb.com">Kaloian Manassiev</assignee>
                                    <reporter username="randolph@mongodb.com">Randolph Tan</reporter>
                        <labels>
                    </labels>
                <created>Fri, 7 Jun 2013 21:12:38 +0000</created>
                <updated>Fri, 13 Jul 2018 14:47:45 +0000</updated>
                            <resolved>Fri, 13 Jul 2018 14:47:40 +0000</resolved>
                                    <version>2.5.0</version>
                                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="1946707" author="kaloian.manassiev" created="Fri, 13 Jul 2018 14:47:29 +0000"  >&lt;p&gt;As of version 3.4, this reverse mutex acquisition order no longer exists, because ReplicaSetMonitor no longer closes the shard connections, but actually this is happening in removeShard.&lt;/p&gt;</comment>
                            <comment id="570993" author="greg_10gen" created="Fri, 2 May 2014 18:37:19 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=renctan&quot; class=&quot;user-hover&quot; rel=&quot;renctan&quot;&gt;renctan&lt;/a&gt; Is this still an issue given the changes to replicasetmonitor?&lt;/p&gt;</comment>
                            <comment id="356784" author="renctan" created="Mon, 10 Jun 2013 13:51:23 +0000"  >&lt;p&gt;Note: the other mutex ordering (reversed) is from the connPoolSync command path.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 2 May 2014 18:37:19 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 30 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 30 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>greg_10gen</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>randolph@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrmqrz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrenun:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7162</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrvxq7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>