<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:06:35 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-4649] mongod primary locks up and never kills cursors stalling connections for ever</title>
                <link>https://jira.mongodb.org/browse/SERVER-4649</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;At what seems to be random times, the mongod primary never seems to finish some queries.&lt;br/&gt;
Some cursors had been running for 6+ hours. It has only happened on the primary.&lt;/p&gt;

&lt;p&gt;The log always has these entries when this occurs:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;conn123812&amp;#93;&lt;/span&gt; warning: virtual size (129449MB) - mapped size (123342MB) is large. could indicate a memory leak&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 172.16.49.98:40540 #123818&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;conn123816&amp;#93;&lt;/span&gt; end connection 172.16.49.98:40539&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;conn123817&amp;#93;&lt;/span&gt;   Assertion failure cc-&amp;gt;_pinValue &amp;lt; 100 db/clientcursor.h 309&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;conn123817&amp;#93;&lt;/span&gt; killcursors  exception: assertion db/clientcursor.h:309 1ms&lt;/p&gt;


&lt;p&gt;And as we can see in the log, the queries that work seem to take forever:&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;conn123170&amp;#93;&lt;/span&gt; query x.scores nscanned:1015 scanAndOrder:1 nreturned:306 reslen:12260 302422ms&lt;/p&gt;


&lt;p&gt;The server resources doesn&apos;t seem to be exhausted and restarting the server gets everything back to normal again.&lt;/p&gt;</description>
                <environment>Debian Squeeze with official 10gen deb packages.&lt;br/&gt;
Two servers running on a replicaset.&lt;br/&gt;
&lt;br/&gt;
Each server is also running numactl --interleave=all wrapper.&lt;br/&gt;
&lt;br/&gt;
Mixed clients but mainly used by the PHP driver.</environment>
        <key id="27960">SERVER-4649</key>
            <summary>mongod primary locks up and never kills cursors stalling connections for ever</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="kristina">Kristina Chodorow</assignee>
                                    <reporter username="balboah">Johnny Boy</reporter>
                        <labels>
                            <label>pv1</label>
                    </labels>
                <created>Mon, 9 Jan 2012 11:05:38 +0000</created>
                <updated>Wed, 15 Aug 2012 14:24:11 +0000</updated>
                            <resolved>Thu, 22 Mar 2012 22:12:50 +0000</resolved>
                                    <version>2.0.1</version>
                                                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="132251" author="kristina" created="Thu, 14 Jun 2012 01:08:25 +0000"  >&lt;p&gt;@Andreas: for debugging what&apos;s slowing MongoDB down, the mailing list is a better format (&lt;a href=&quot;https://groups.google.com/group/mongodb-user/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://groups.google.com/group/mongodb-user/&lt;/a&gt;).  If this problem continues post-ETA2, please ask about it there.&lt;/p&gt;</comment>
                            <comment id="132057" author="andreas@idztech.com" created="Wed, 13 Jun 2012 19:52:53 +0000"  >&lt;p&gt;This is still very much a big problem, the assertion as you say, are probably harmless and just a symptom of the underlying issue. &lt;/p&gt;

&lt;p&gt;I don&apos;t know why and the only info the oplog is the assertion that get triggered when the DB stops responding (or becomes super slow).&lt;/p&gt;

&lt;p&gt;This happens during normal operation, the DB is performing extremely well and have no issues with the speed. Then all of a sudden this issue appears, the cursors and queues spikes and the site stops loading. This happens about every other week for me atm.&lt;/p&gt;

&lt;p&gt;ETA: And no, no indexes or anything are being build when this happens. It just during normal operation.&lt;/p&gt;

&lt;p&gt;ETA2: I looked in to the index situation and it seems to have been caused by &quot;ensure index&quot; operations, I&apos;ve removed them and the index activity disappeared. I don&apos;t know if that was what caused the other issues, but time will tell.&lt;/p&gt;</comment>
                            <comment id="119207" author="kristina" created="Mon, 14 May 2012 16:59:15 +0000"  >&lt;p&gt;This particular assertion is harmless and is a side-effect of a driver bug. The driver bug is triggered by DB slowness, which is the real issue you should try to debug if you get this assertion.  The &lt;a href=&quot;https://groups.google.com/forum/#!forum/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mailing list&lt;/a&gt; can help you debug general db slowness, Jira isn&apos;t really the place for it.&lt;/p&gt;</comment>
                            <comment id="119044" author="ootada" created="Mon, 14 May 2012 11:57:41 +0000"  >&lt;p&gt;any resolution on this?&lt;/p&gt;</comment>
                            <comment id="96560" author="kristina" created="Wed, 7 Mar 2012 21:31:46 +0000"  >&lt;p&gt;There are a bunch of inserts into OviPets.system.indexes, were you building an index?  What was writing there?  Do you have iostats from the spikes?&lt;/p&gt;</comment>
                            <comment id="90031" author="andreas@idztech.com" created="Mon, 20 Feb 2012 09:56:25 +0000"  >&lt;p&gt;Had a really bad crash day last night, the issues occurred three times in 10 minutes intervals. I&apos;ve uploaded a oplog from the first restart to the next crash 10 minutes later and a MMS screenshot.&lt;/p&gt;</comment>
                            <comment id="89836" author="andreas@idztech.com" created="Sat, 18 Feb 2012 19:02:21 +0000"  >&lt;p&gt;I added a screenshot from MMS when the issue occurs.&lt;/p&gt;</comment>
                            <comment id="88581" author="andreas@idztech.com" created="Wed, 15 Feb 2012 09:53:12 +0000"  >&lt;p&gt;My OpLog from 20:00:00 when the first _pinValue assertion appeared until about 10 minutes later when I restarted the DB.&lt;/p&gt;

&lt;p&gt;I&apos;ve extracted the 32 assertions into it pinvalue.txt&lt;/p&gt;

&lt;p&gt;This MongoDB server is run on a dedicated server with 24GB of RAM and RAID 10 running Ubuntu 11.04. I&apos;m using 2.0.2 and the latest PHP driver (1.2.7).&lt;/p&gt;

&lt;p&gt;This issue happened once a day during the last weekend, we had a bit more load that usual which I&apos;m guessing triggered the issue.&lt;/p&gt;</comment>
                            <comment id="88545" author="eliot" created="Wed, 15 Feb 2012 05:46:49 +0000"  >&lt;p&gt;The pin value error is a bit different, and itself doesn&apos;t cause issues.&lt;/p&gt;

&lt;p&gt;Can add your logs, etc...?&lt;/p&gt;</comment>
                            <comment id="87905" author="andreas@idztech.com" created="Mon, 13 Feb 2012 17:00:50 +0000"  >&lt;p&gt;I commented on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-4617&quot; title=&quot;make _pinValue error message when killCursors called during getMore better&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-4617&quot;&gt;&lt;del&gt;SERVER-4617&lt;/del&gt;&lt;/a&gt; but this seems to be the exact same issue?&lt;/p&gt;

&lt;p&gt;Is there any workaround to avoid this issue until this bug is resolved?&lt;/p&gt;</comment>
                            <comment id="79062" author="balboah" created="Wed, 11 Jan 2012 09:54:01 +0000"  >&lt;p&gt;Sorry to hear that. Thank you for looking into this and I hope we can solve it.&lt;/p&gt;

&lt;p&gt;Last night, the pinValue assert occured again, and number of connection started rising. It never got to the memory warning but it does seem similar to how it behaved. Unfortunately I didn&apos;t notice and wasn&apos;t able to get a currentOp list.&lt;/p&gt;

&lt;p&gt;The error:&lt;/p&gt;


&lt;p&gt;Wed Jan 11 05:14:11 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 172.16.49.121:36532 #82981&lt;br/&gt;
Wed Jan 11 05:14:11 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn82981&amp;#93;&lt;/span&gt;   Assertion failure cc-&amp;gt;_pinValue &amp;lt; 100 db/clientcursor.h 309&lt;br/&gt;
0x57a926 0x5857db 0x7982e9 0x882846 0x889e99 0xaa00c6 0x635bb7 0x7f2e115ea8ba 0x7f2e10ba602d /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x96) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x57a926&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfb) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5857db&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7982e9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19receivedKillCursorsERNS_7MessageE+0xb6) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x882846&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x359) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x889e99&amp;#93;&lt;/span&gt; /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xaa00c6&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x635bb7&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/libpthread.so.0(+0x68ba) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f2e115ea8ba&amp;#93;&lt;/span&gt; /lib/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f2e10ba602d&amp;#93;&lt;/span&gt;&lt;br/&gt;
Wed Jan 11 05:14:11 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn82981&amp;#93;&lt;/span&gt; killcursors  exception: assertion db/clientcursor.h:309 1ms&lt;/p&gt;

&lt;p&gt;Wed Jan 11 05:29:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 172.16.49.121:39143 #84017&lt;br/&gt;
Wed Jan 11 05:29:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn84016&amp;#93;&lt;/span&gt; end connection 172.16.49.121:39142&lt;br/&gt;
Wed Jan 11 05:29:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn84017&amp;#93;&lt;/span&gt;   Assertion failure cc-&amp;gt;_pinValue &amp;lt; 100 db/clientcursor.h 309&lt;br/&gt;
0x57a926 0x5857db 0x7982e9 0x882846 0x889e99 0xaa00c6 0x635bb7 0x7f2e115ea8ba 0x7f2e10ba602d&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x96) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x57a926&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfb) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5857db&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7982e9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19receivedKillCursorsERNS_7MessageE+0xb6) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x882846&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x359) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x889e99&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xaa00c6&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x635bb7&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/libpthread.so.0(+0x68ba) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f2e115ea8ba&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f2e10ba602d&amp;#93;&lt;/span&gt;&lt;br/&gt;
Wed Jan 11 05:29:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn84017&amp;#93;&lt;/span&gt; killcursors  exception: assertion db/clientcursor.h:309 2ms&lt;/p&gt;




&lt;p&gt;Also there seems to be a lot of idle cursors, is this normal?&lt;br/&gt;
PRIMARY&amp;gt; db.currentOp().inprog.length&lt;br/&gt;
6&lt;br/&gt;
PRIMARY&amp;gt; db.currentOp(true).inprog.length&lt;br/&gt;
605&lt;/p&gt;</comment>
                            <comment id="78867" author="scotthernandez" created="Tue, 10 Jan 2012 18:30:43 +0000"  >&lt;p&gt;I&apos;ve looked at the dropbox munin graphs and MMS and unfortunately the logs/graphs don&apos;t all line up to provide a good view of what could be the cause. &lt;/p&gt;

&lt;p&gt;If this happens again, now that MMS is enabled, we will be able to see a better picture of it. Please let us know when this happens again.&lt;/p&gt;</comment>
                            <comment id="78823" author="balboah" created="Tue, 10 Jan 2012 16:41:28 +0000"  >&lt;p&gt;The driver is PHP (5.3.8-1~dotdeb.2) and mongo 1.2.6.&lt;/p&gt;

&lt;p&gt;Yes that is the MMS group.&lt;/p&gt;

&lt;p&gt;The log timezone is UTC/GMT+1 and munin graph is UTC&lt;/p&gt;</comment>
                            <comment id="78798" author="scotthernandez" created="Tue, 10 Jan 2012 16:09:51 +0000"  >&lt;p&gt;Please verify that you are using the PHP driver 1.2.7 and this is your group in MMS: &lt;a href=&quot;https://mms.10gen.com/host/list/4f0bf92f47103e8b01b39d26&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://mms.10gen.com/host/list/4f0bf92f47103e8b01b39d26&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Also, what time zone are those logs in?&lt;/p&gt;</comment>
                            <comment id="78792" author="eliot" created="Tue, 10 Jan 2012 15:57:18 +0000"  >&lt;p&gt;What driver are you using?&lt;/p&gt;</comment>
                            <comment id="78775" author="balboah" created="Tue, 10 Jan 2012 14:45:50 +0000"  >&lt;p&gt;The MMS is also hooked up now.&lt;/p&gt;

&lt;p&gt;I think most of the information is already available in our Munin graphs.&lt;br/&gt;
Here is the gallery of the daily graphs that I think may be interesting: &lt;a href=&quot;https://www.dropbox.com/gallery/2468164/1/Munin?h=08adec&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://www.dropbox.com/gallery/2468164/1/Munin?h=08adec&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;You can easily see the spike in active connections of when the error occurred to see at what time it is.&lt;/p&gt;</comment>
                            <comment id="78760" author="balboah" created="Tue, 10 Jan 2012 12:02:08 +0000"  >&lt;p&gt;I will join MMS and I also have a currentOp log of the time but not with the &quot;true&quot; argument, I hope it still helps a bit.&lt;/p&gt;

&lt;p&gt;I will add a new log when the error occurs again.&lt;/p&gt;

&lt;p&gt;Also upgraded to 2.0.2 which still has the same problem.&lt;/p&gt;

&lt;p&gt;This time I didn&apos;t get the virtual size - mapped size warning but instead:&lt;/p&gt;

&lt;p&gt;Mon Jan  9 19:18:02 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; warning virtual/mapped memory differential is large. journaling:1&lt;/p&gt;

&lt;p&gt;&amp;#8211; snip snip &amp;#8211;&lt;/p&gt;

&lt;p&gt;Mon Jan  9 19:18:03 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn21771&amp;#93;&lt;/span&gt;   Assertion failure cc-&amp;gt;_pinValue &amp;lt; 100 db/clientcursor.h 309&lt;br/&gt;
Mon Jan  9 19:18:03 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn21770&amp;#93;&lt;/span&gt; end connection 172.16.49.121:45331&lt;br/&gt;
0x57a926 0x5857db 0x7982e9 0x882846 0x889e99 0xaa00c6 0x635bb7 0x7f4d6f9968ba 0x7f4d6ef5202d &lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x96) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x57a926&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfb) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5857db&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7982e9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19receivedKillCursorsERNS_7MessageE+0xb6) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x882846&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x359) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x889e99&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xaa00c6&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x635bb7&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/libpthread.so.0(+0x68ba) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f4d6f9968ba&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f4d6ef5202d&amp;#93;&lt;/span&gt;&lt;br/&gt;
Mon Jan  9 19:18:03 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn21771&amp;#93;&lt;/span&gt; killcursors  exception: assertion db/clientcursor.h:309 1ms&lt;br/&gt;
Mon Jan  9 19:18:03 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn21771&amp;#93;&lt;/span&gt; end connection 172.16.49.121:45332&lt;/p&gt;</comment>
                            <comment id="78680" author="scotthernandez" created="Mon, 9 Jan 2012 23:57:04 +0000"  >&lt;p&gt;Can you run db.currentOp(true) when one of these long running queries is running?&lt;/p&gt;

&lt;p&gt;Are you using MMS and if so can you tell use when this happened so we can look at your server stats?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="14359" name="currentop.log" size="1022003" author="balboah" created="Tue, 10 Jan 2012 11:59:29 +0000"/>
                            <attachment id="14953" name="mms.gif" size="66107" author="andreas@idztech.com" created="Sat, 18 Feb 2012 19:02:21 +0000"/>
                            <attachment id="14960" name="mms2.gif" size="133912" author="andreas@idztech.com" created="Mon, 20 Feb 2012 09:56:25 +0000"/>
                            <attachment id="14961" name="oplog - start to crash.log" size="1981927" author="andreas@idztech.com" created="Mon, 20 Feb 2012 09:56:25 +0000"/>
                            <attachment id="14857" name="oplog.log" size="2561021" author="andreas@idztech.com" created="Wed, 15 Feb 2012 09:53:12 +0000"/>
                            <attachment id="14858" name="pinvalue.txt" size="31220" author="andreas@idztech.com" created="Wed, 15 Feb 2012 09:53:12 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>18.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 9 Jan 2012 23:57:04 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 36 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 36 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andreas@idztech.com</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>balboah</customfieldvalue>
            <customfieldvalue>kristina</customfieldvalue>
            <customfieldvalue>scotthernandez</customfieldvalue>
            <customfieldvalue>ootada</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrohan:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hritkf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>23443</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrlh5j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>