<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:20:07 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-9339] rollback files missing after rollback</title>
                <link>https://jira.mongodb.org/browse/SERVER-9339</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I have mongo replica set made up from 3 members: one Primary, one Secondary, and one Arbiter. There are located on 10.10.222, 10.10.0.223, and 10.10.0.155. &lt;/p&gt;

&lt;p&gt;kelly:PRIMARY&amp;gt; rs.status()&lt;br/&gt;
{&lt;br/&gt;
        &quot;set&quot; : &quot;kelly&quot;,&lt;br/&gt;
        &quot;date&quot; : ISODate(&quot;2013-04-12T08:28:41Z&quot;),&lt;br/&gt;
        &quot;myState&quot; : 1,&lt;br/&gt;
        &quot;members&quot; : [&lt;br/&gt;
                {&lt;br/&gt;
                        &quot;_id&quot; : 0,&lt;br/&gt;
                        &quot;name&quot; : &quot;10.10.0.222:27017&quot;,&lt;br/&gt;
                        &quot;health&quot; : 1,&lt;br/&gt;
                        &quot;state&quot; : 1,&lt;br/&gt;
                        &quot;stateStr&quot; : &quot;PRIMARY&quot;,&lt;br/&gt;
                        &quot;uptime&quot; : 34,&lt;br/&gt;
                        &quot;optime&quot; : &lt;/p&gt;
{
                                &quot;t&quot; : 1365755246,
                                &quot;i&quot; : 2767
                        }
&lt;p&gt;,&lt;br/&gt;
                        &quot;optimeDate&quot; : ISODate(&quot;2013-04-12T08:27:26Z&quot;),&lt;br/&gt;
                        &quot;self&quot; : true&lt;br/&gt;
                },&lt;br/&gt;
                {&lt;br/&gt;
                        &quot;_id&quot; : 1,&lt;br/&gt;
                        &quot;name&quot; : &quot;10.10.0.223:27017&quot;,&lt;br/&gt;
                        &quot;health&quot; : 1,&lt;br/&gt;
                        &quot;state&quot; : 2,&lt;br/&gt;
                        &quot;stateStr&quot; : &quot;SECONDARY&quot;,&lt;br/&gt;
                        &quot;uptime&quot; : 24,&lt;br/&gt;
                        &quot;optime&quot; : &lt;/p&gt;
{
                                &quot;t&quot; : 1365755246,
                                &quot;i&quot; : 2767
                        }
&lt;p&gt;,&lt;br/&gt;
                        &quot;optimeDate&quot; : ISODate(&quot;2013-04-12T08:27:26Z&quot;),&lt;br/&gt;
                        &quot;lastHeartbeat&quot; : ISODate(&quot;2013-04-12T08:28:39Z&quot;),&lt;br/&gt;
                        &quot;lastHeartbeatRecv&quot; : ISODate(&quot;2013-04-12T08:28:40Z&quot;),&lt;br/&gt;
                        &quot;pingMs&quot; : 0,&lt;br/&gt;
                        &quot;lastHeartbeatMessage&quot; : &quot;syncing to: 10.10.0.222:27017&quot;&lt;br/&gt;
                },&lt;/p&gt;
                {
                        &quot;_id&quot; : 2,
                        &quot;name&quot; : &quot;10.10.0.155:27017&quot;,
                        &quot;health&quot; : 1,
                        &quot;state&quot; : 7,
                        &quot;stateStr&quot; : &quot;ARBITER&quot;,
                        &quot;uptime&quot; : 34,
                        &quot;lastHeartbeat&quot; : ISODate(&quot;2013-04-12T08:28:39Z&quot;),
                        &quot;lastHeartbeatRecv&quot; : ISODate(&quot;2013-04-12T08:28:40Z&quot;),
                        &quot;pingMs&quot; : 0
                }
&lt;p&gt;        ],&lt;br/&gt;
        &quot;ok&quot; : 1&lt;br/&gt;
}&lt;/p&gt;

&lt;p&gt;They are started in interactive mode and killed by Ctrl-C, so unclear shutdown is not the case. Mongo version is 2.4.0 Linux 64bit with default settings. Database servers time is synchronized using NTP.&lt;/p&gt;

&lt;p&gt;All updatess are upserts with `safe&apos; mode, i.e. &lt;/p&gt;
{getLastError:1}
&lt;p&gt; which as documentation states also means &lt;/p&gt;
{w:1}
&lt;p&gt; and all writes go to the Primary.&lt;/p&gt;

&lt;p&gt;By data loss I mean non-complete documents after applying rollbacks, for example &lt;/p&gt;
{key:Value, req:ReqValue}
&lt;p&gt;. After analyzing many logs (application, mongod, mongostat) I realized that this happens when, for example, the request has been sent to 10.10.0.222 it was Primary initially, and the response has been sent to 10.10.0.223. The whole document is stored in the mongo export file. But after rollback on Y this response is gone.&lt;/p&gt;

&lt;p&gt;I can to reproduce this issue 10 out of 10 times. Client code is Erlang and it&apos;s able to sustain temporary connection failures. If there&apos;s a write failure then the write is repeated later. I can see that missing updates are registered in mongo log files and acknowledged by current Primary (at some poing I traced the erlang driver and saw the replies in Wireshark). The problem seems to be in the rollback mechanics.&lt;/p&gt;
</description>
                <environment>Linux CentOS 2.6.32-358.2.1.el6.x86_64 #1 SMP Wed Mar 13 00:26:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux</environment>
        <key id="71606">SERVER-9339</key>
            <summary>rollback files missing after rollback</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="milkie@mongodb.com">Eric Milkie</assignee>
                                    <reporter username="ten0s">Dmitry Klionsky</reporter>
                        <labels>
                    </labels>
                <created>Fri, 12 Apr 2013 09:21:55 +0000</created>
                <updated>Mon, 11 Jul 2016 17:57:24 +0000</updated>
                            <resolved>Tue, 23 Apr 2013 12:42:29 +0000</resolved>
                                    <version>2.4.0</version>
                                    <fixVersion>2.2.7</fixVersion>
                    <fixVersion>2.4.4</fixVersion>
                    <fixVersion>2.5.0</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="414303" author="auto" created="Thu, 29 Aug 2013 20:17:27 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;milkie&apos;, u&apos;name&apos;: u&apos;Eric Milkie&apos;, u&apos;email&apos;: u&apos;milkie@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9339&quot; title=&quot;rollback files missing after rollback&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9339&quot;&gt;&lt;del&gt;SERVER-9339&lt;/del&gt;&lt;/a&gt; do not query a node that is not in PRIMARY or SECONDARY state&lt;br/&gt;
Branch: v2.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/be66f1b8fc992251f4785adb4c632d3b4677e005&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/be66f1b8fc992251f4785adb4c632d3b4677e005&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="414302" author="auto" created="Thu, 29 Aug 2013 20:17:25 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;milkie&apos;, u&apos;name&apos;: u&apos;Eric Milkie&apos;, u&apos;email&apos;: u&apos;milkie@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9339&quot; title=&quot;rollback files missing after rollback&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9339&quot;&gt;&lt;del&gt;SERVER-9339&lt;/del&gt;&lt;/a&gt; ensure all rollback update/delete paths are recorded&lt;/p&gt;

&lt;p&gt;Conflicts:&lt;br/&gt;
	src/mongo/db/ops/update.cpp&lt;br/&gt;
Branch: v2.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/0364ef4bdd4c17a07801457334aee32b9f601828&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/0364ef4bdd4c17a07801457334aee32b9f601828&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="325683" author="auto" created="Wed, 1 May 2013 14:45:13 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;date&apos;: u&apos;2013-04-30T15:58:46Z&apos;, u&apos;name&apos;: u&apos;Eric Milkie&apos;, u&apos;email&apos;: u&apos;milkie@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9339&quot; title=&quot;rollback files missing after rollback&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9339&quot;&gt;&lt;del&gt;SERVER-9339&lt;/del&gt;&lt;/a&gt; do not query a node that is not in PRIMARY or SECONDARY state&lt;br/&gt;
Branch: v2.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/c6a32dc49c05a78ea322ef13ebb90443bbdb44f1&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/c6a32dc49c05a78ea322ef13ebb90443bbdb44f1&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="324946" author="auto" created="Tue, 30 Apr 2013 16:04:51 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;date&apos;: u&apos;2013-04-30T15:58:46Z&apos;, u&apos;name&apos;: u&apos;Eric Milkie&apos;, u&apos;email&apos;: u&apos;milkie@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9339&quot; title=&quot;rollback files missing after rollback&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9339&quot;&gt;&lt;del&gt;SERVER-9339&lt;/del&gt;&lt;/a&gt; do not query a node that is not in PRIMARY or SECONDARY state&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/8746fa436394d630efec6ee1f4788c932dabb698&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/8746fa436394d630efec6ee1f4788c932dabb698&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="324935" author="auto" created="Tue, 30 Apr 2013 15:42:37 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;date&apos;: u&apos;2013-04-22T18:48:02Z&apos;, u&apos;name&apos;: u&apos;Eric Milkie&apos;, u&apos;email&apos;: u&apos;milkie@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9339&quot; title=&quot;rollback files missing after rollback&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9339&quot;&gt;&lt;del&gt;SERVER-9339&lt;/del&gt;&lt;/a&gt; ensure all rollback update/delete paths are recorded&lt;/p&gt;

&lt;p&gt;Conflicts:&lt;br/&gt;
	src/mongo/db/ops/update.cpp&lt;br/&gt;
Branch: v2.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/6987dd1dd3f4990cb05212c9c77fd924eca4e8ba&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/6987dd1dd3f4990cb05212c9c77fd924eca4e8ba&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="321709" author="milkie" created="Thu, 25 Apr 2013 20:02:33 +0000"  >&lt;p&gt;commit: 30b7ed6470b3c97b04e32b53137ece4314ee8ce8&lt;/p&gt;</comment>
                            <comment id="312016" author="ten0s" created="Fri, 12 Apr 2013 15:34:41 +0000"  >&lt;p&gt;My use case was to make sure that my app survives multiple failovers and continues to function as expected. &lt;br/&gt;
All seemed ok, until I came across this combination of steps.&lt;/p&gt;

&lt;p&gt;&amp;gt;&amp;gt;The rollback directory generally shouldn&apos;t be a part of your normal workflow as you would have to deal with merge conflicts on the data, etc...&lt;br/&gt;
Right. And I made an utility to deal with the rollbacks, and with its help I figured that data is lost and then started to dig deeper.&lt;/p&gt;

&lt;p&gt;&amp;gt;&amp;gt;It sounds like you should be using w=2 write concerns.&lt;br/&gt;
It seems I should.&lt;/p&gt;</comment>
                            <comment id="311998" author="eliot" created="Fri, 12 Apr 2013 15:11:45 +0000"  >&lt;p&gt;While we work on that, what is the use case you are working on?&lt;/p&gt;

&lt;p&gt;The rollback directory generally shouldn&apos;t be a part of your normal workflow as you would have to deal with merge conflicts on the data, etc...&lt;/p&gt;

&lt;p&gt;It sounds like you should be using w=2 write concerns.&lt;/p&gt;</comment>
                            <comment id="311967" author="ten0s" created="Fri, 12 Apr 2013 14:29:37 +0000"  >&lt;p&gt;Right, it should, but it&apos;s not there. Double checked on two machines.&lt;/p&gt;

&lt;p&gt;Linux Mint 32 bit&lt;br/&gt;
db version v2.4.0&lt;br/&gt;
Fri Apr 12 17:27:40.498 git version: ce2d666c04b4a80af58e8bbb3388b0680e8cfeb6&lt;/p&gt;

&lt;p&gt;CentOS 64 bit&lt;br/&gt;
db version v2.4.0&lt;br/&gt;
Fri Apr 12 17:28:59.233 git version: ce2d666c04b4a80af58e8bbb3388b0680e8cfeb6&lt;/p&gt;</comment>
                            <comment id="311956" author="eliot" created="Fri, 12 Apr 2013 14:19:52 +0000"  >&lt;p&gt;There should be a rollback directory in db2.&lt;/p&gt;

&lt;p&gt;Will see if I can reproduce it not being there.&lt;/p&gt;</comment>
                            <comment id="311943" author="ten0s" created="Fri, 12 Apr 2013 13:56:18 +0000"  >&lt;p&gt;Here is  it. &lt;/p&gt;

&lt;p&gt;Open two terminal tabs Tab1 and Tab2&lt;/p&gt;

&lt;p&gt;Tab1&lt;/p&gt;

&lt;p&gt;$ mkdir db1 db2 db3&lt;br/&gt;
$ mongod --replSet test --dbpath ./db1 --logpath ./db1/mongodb.log --port 1111 &amp;amp;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt; 9657&lt;br/&gt;
$ mongod --replSet test --dbpath ./db2 --logpath ./db2/mongodb.log --port 2222 &amp;amp;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;2&amp;#93;&lt;/span&gt; 9773&lt;br/&gt;
$ mongod --replSet test --dbpath ./db3 --logpath ./db3/mongodb.log --port 3333 &amp;amp;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;3&amp;#93;&lt;/span&gt; 9824&lt;/p&gt;

&lt;p&gt;Tab2&lt;/p&gt;

&lt;p&gt;$ mongo localhost:1111/test&lt;br/&gt;
&amp;gt; rs.initiate()&lt;/p&gt;
{
	&quot;info2&quot; : &quot;no configuration explicitly specified -- making one&quot;,
	&quot;me&quot; : &quot;mint-ws:1111&quot;,
	&quot;info&quot; : &quot;Config now saved locally.  Should come online in about a minute.&quot;,
	&quot;ok&quot; : 1
}
test:PRIMARY&amp;gt; rs.add(&quot;mint-ws:2222&quot;)
{ &quot;ok&quot; : 1 }
&lt;p&gt;test:PRIMARY&amp;gt; rs.addArb(&quot;mint-ws:3333&quot;)&lt;/p&gt;
{ &quot;ok&quot; : 1 }
&lt;p&gt;test:PRIMARY&amp;gt; rs.status()&lt;br/&gt;
est:PRIMARY&amp;gt; rs.status()&lt;br/&gt;
{&lt;br/&gt;
	&quot;set&quot; : &quot;test&quot;,&lt;br/&gt;
	&quot;date&quot; : ISODate(&quot;2013-04-12T13:40:10Z&quot;),&lt;br/&gt;
	&quot;myState&quot; : 1,&lt;br/&gt;
	&quot;members&quot; : [&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 0,&lt;br/&gt;
			&quot;name&quot; : &quot;mint-ws:1111&quot;,&lt;br/&gt;
			&quot;health&quot; : 1,&lt;br/&gt;
			&quot;state&quot; : 1,&lt;br/&gt;
			&quot;stateStr&quot; : &quot;PRIMARY&quot;,&lt;br/&gt;
			&quot;uptime&quot; : 142,&lt;br/&gt;
			&quot;optime&quot; : &lt;/p&gt;
{
				&quot;t&quot; : 1365773990,
				&quot;i&quot; : 1
			}
&lt;p&gt;,&lt;br/&gt;
			&quot;optimeDate&quot; : ISODate(&quot;2013-04-12T13:39:50Z&quot;),&lt;br/&gt;
			&quot;self&quot; : true&lt;br/&gt;
		},&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 1,&lt;br/&gt;
			&quot;name&quot; : &quot;mint-ws:2222&quot;,&lt;br/&gt;
			&quot;health&quot; : 1,&lt;br/&gt;
			&quot;state&quot; : 2,&lt;br/&gt;
			&quot;stateStr&quot; : &quot;SECONDARY&quot;,&lt;br/&gt;
			&quot;uptime&quot; : 30,&lt;br/&gt;
			&quot;optime&quot; : &lt;/p&gt;
{
				&quot;t&quot; : 1365773990,
				&quot;i&quot; : 1
			}
&lt;p&gt;,&lt;br/&gt;
			&quot;optimeDate&quot; : ISODate(&quot;2013-04-12T13:39:50Z&quot;),&lt;br/&gt;
			&quot;lastHeartbeat&quot; : ISODate(&quot;2013-04-12T13:40:10Z&quot;),&lt;br/&gt;
			&quot;lastHeartbeatRecv&quot; : ISODate(&quot;2013-04-12T13:40:10Z&quot;),&lt;br/&gt;
			&quot;pingMs&quot; : 1&lt;br/&gt;
		},&lt;/p&gt;
		{
			&quot;_id&quot; : 2,
			&quot;name&quot; : &quot;mint-ws:3333&quot;,
			&quot;health&quot; : 1,
			&quot;state&quot; : 7,
			&quot;stateStr&quot; : &quot;ARBITER&quot;,
			&quot;uptime&quot; : 20,
			&quot;lastHeartbeat&quot; : ISODate(&quot;2013-04-12T13:40:10Z&quot;),
			&quot;lastHeartbeatRecv&quot; : ISODate(&quot;1970-01-01T00:00:00Z&quot;),
			&quot;pingMs&quot; : 0
		}
&lt;p&gt;	],&lt;br/&gt;
	&quot;ok&quot; : 1&lt;br/&gt;
}&lt;br/&gt;
test:PRIMARY&amp;gt; db.coll.update(&lt;/p&gt;
{key:1}
&lt;p&gt;, {$set:{req:&quot;req1&quot;}}, true)&lt;br/&gt;
test:PRIMARY&amp;gt; db.coll.find()&lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;51680a6825870d2c6f155ddf&quot;), &quot;key&quot; : 1, &quot;req&quot; : &quot;req1&quot; }

&lt;p&gt;Tab 1&lt;/p&gt;

&lt;p&gt;$ kill %1&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt;   Done                    mongod --replSet test --dbpath ./db1 --logpath mongodb1.log --port 1111&lt;/p&gt;

&lt;p&gt;Tab 2&lt;/p&gt;

&lt;p&gt;&amp;gt; db = connect(&quot;localhost:2222/test&quot;)&lt;br/&gt;
connecting to: localhost:2222/test&lt;br/&gt;
test&lt;br/&gt;
test:PRIMARY&amp;gt; db.coll.update(&lt;/p&gt;
{key:1}
&lt;p&gt;, {$set:{rsp:&quot;resp1&quot;}}, true)&lt;br/&gt;
test:PRIMARY&amp;gt; db.coll.find()&lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;51680a6825870d2c6f155ddf&quot;), &quot;key&quot; : 1, &quot;req&quot; : &quot;req1&quot;, &quot;rsp&quot; : &quot;resp1&quot; }

&lt;p&gt;Tab 1&lt;/p&gt;

&lt;p&gt;$ kill %2&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;2&amp;#93;&lt;/span&gt;-  Done                    mongod --replSet test --dbpath ./db2 --logpath mongodb2.log --port 2222&lt;br/&gt;
$ mongod --replSet test --dbpath ./db1 --logpath mongodb1.log --port 1111 &amp;amp;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;4&amp;#93;&lt;/span&gt; 10672&lt;/p&gt;

&lt;p&gt;wait 1 min&lt;/p&gt;

&lt;p&gt;$ mongod --replSet test --dbpath ./db2 --logpath mongodb2.log --port 2222 &amp;amp;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;5&amp;#93;&lt;/span&gt; 10941&lt;/p&gt;

&lt;p&gt;Tab 2&lt;/p&gt;

&lt;p&gt;test:SECONDARY&amp;gt; db = connect(&quot;localhost:1111/test&quot;)&lt;br/&gt;
connecting to: localhost:1111/test&lt;br/&gt;
test&lt;br/&gt;
test:PRIMARY&amp;gt; db.coll.find()&lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;51680a6825870d2c6f155ddf&quot;), &quot;key&quot; : 1, &quot;req&quot; : &quot;req1&quot; }

&lt;p&gt;Tab 1&lt;/p&gt;

&lt;p&gt;$ cat ./db2/mongodb.log&lt;br/&gt;
...&lt;br/&gt;
Fri Apr 12 16:42:59.413 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet syncing to: mint-ws:1111&lt;br/&gt;
Fri Apr 12 16:43:00.413 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSyncNotifier&amp;#93;&lt;/span&gt; replset setting oplog notifier to mint-ws:1111&lt;br/&gt;
Fri Apr 12 16:43:04.427 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet we are ahead of the sync source, will try to roll back&lt;br/&gt;
Fri Apr 12 16:43:04.427 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 0&lt;br/&gt;
Fri Apr 12 16:43:04.427 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet ROLLBACK&lt;br/&gt;
Fri Apr 12 16:43:04.427 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 1&lt;br/&gt;
Fri Apr 12 16:43:04.427 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 2 FindCommonPoint&lt;br/&gt;
Fri Apr 12 16:43:04.428 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet info rollback our last optime:   Apr 12 16:41:52:1&lt;br/&gt;
Fri Apr 12 16:43:04.428 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet info rollback their last optime: Apr 12 16:40:53:1&lt;br/&gt;
Fri Apr 12 16:43:04.428 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet info rollback diff in end of log times: 59 seconds&lt;br/&gt;
Fri Apr 12 16:43:04.428 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback found matching events at Apr 12 16:40:53:1&lt;br/&gt;
Fri Apr 12 16:43:04.428 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback findcommonpoint scanned : 2&lt;br/&gt;
Fri Apr 12 16:43:04.428 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet replSet rollback 3 fixup&lt;br/&gt;
Fri Apr 12 16:43:04.454 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 3.5&lt;br/&gt;
Fri Apr 12 16:43:04.454 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4 n:1&lt;br/&gt;
Fri Apr 12 16:43:04.454 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet minvalid=Apr 12 16:40:53 51680ee5:1&lt;br/&gt;
Fri Apr 12 16:43:04.455 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4.6&lt;br/&gt;
Fri Apr 12 16:43:04.455 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 4.7&lt;br/&gt;
Fri Apr 12 16:43:04.456 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 5 d:0 u:1&lt;br/&gt;
Fri Apr 12 16:43:04.590 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 6&lt;br/&gt;
Fri Apr 12 16:43:04.590 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback 7&lt;br/&gt;
Fri Apr 12 16:43:04.655 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet rollback done&lt;br/&gt;
Fri Apr 12 16:43:04.655 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet RECOVERING&lt;br/&gt;
Fri Apr 12 16:43:04.655 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet syncing to: mint-ws:1111&lt;br/&gt;
Fri Apr 12 16:43:05.415 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; replSet SECONDARY&lt;br/&gt;
...&lt;br/&gt;
$ ls ./db1&lt;br/&gt;
local.0  local.1  local.ns  mongodb.log  mongod.lock  test.0  test.1  test.ns&lt;br/&gt;
$ ls ./db2&lt;br/&gt;
local.0  local.ns     mongodb.log.2013-04-12T13-42-52  test.0  test.ns&lt;br/&gt;
local.1  mongodb.log  mongod.lock                      test.1&lt;/p&gt;

&lt;p&gt;There&apos;s no `rollback&apos; directory! The second upsert is gone! Am I missing something?&lt;/p&gt;</comment>
                            <comment id="311922" author="eliot" created="Fri, 12 Apr 2013 13:31:21 +0000"  >&lt;p&gt;Do you have a script to do this or is it manual?  Can you send code?&lt;/p&gt;

&lt;p&gt;I&apos;m a little confused by some of the steps, will look again, but have you considered the following:&lt;/p&gt;

&lt;p&gt;T0: write to primary A&lt;br/&gt;
T1: w=1 response from A to client&lt;br/&gt;
T2: A shuts down before write replicated to B&lt;br/&gt;
T3: B becomes primary without write&lt;br/&gt;
T4: A restarts, rolls back write&lt;br/&gt;
T5: write is now gone, but is in the rollback file on A&lt;/p&gt;

&lt;p&gt;have you checked the rollback file?&lt;/p&gt;

&lt;p&gt;Have you tried this with w=2?&lt;/p&gt;</comment>
                            <comment id="311826" author="ten0s" created="Fri, 12 Apr 2013 09:31:25 +0000"  >&lt;p&gt;Step 5 should be `Kill 10.10.0.223 which is Primary&apos;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="76252">SERVER-9756</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="25890" name="kelly_2013_01_01.mt_messages.2013-04-12T08-31-25.0.bson" size="375981" author="ten0s" created="Fri, 12 Apr 2013 09:25:52 +0000"/>
                            <attachment id="25891" name="mongodb-10.10.0.222.log" size="10087187" author="ten0s" created="Fri, 12 Apr 2013 09:25:52 +0000"/>
                            <attachment id="25892" name="mongodb-10.10.0.223.log" size="2556821" author="ten0s" created="Fri, 12 Apr 2013 09:25:52 +0000"/>
                            <attachment id="25893" name="mongostat-10.10.0.222.log" size="67479" author="ten0s" created="Fri, 12 Apr 2013 09:25:52 +0000"/>
                            <attachment id="25894" name="mongostat-10.10.0.223.log" size="68220" author="ten0s" created="Fri, 12 Apr 2013 09:25:52 +0000"/>
                            <attachment id="25895" name="mt_messages.json" size="928970" author="ten0s" created="Fri, 12 Apr 2013 09:25:52 +0000"/>
                            <attachment id="26106" name="rollback5.js" size="3160" author="spencer@mongodb.com" created="Thu, 18 Apr 2013 22:37:15 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 12 Apr 2013 13:31:21 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        10 years, 24 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/CS-8536'>CS-8536</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            10 years, 24 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>auto</customfieldvalue>
            <customfieldvalue>ten0s</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrmx73:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrn8sf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>49367</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Steps to reproduce (documents are simplified to give you the idea):&lt;/p&gt;

&lt;p&gt;1. 10.10.0.222 is Primary, 10.10.0.223 is Secondary, and 10.10.0.155 is Arbiter.&lt;/p&gt;

&lt;p&gt;2. Start writing 5000 requests and response documents&lt;br/&gt;
Requests are like:&lt;br/&gt;
db.mt_messages.update(&lt;/p&gt;
{key:Value}
&lt;p&gt;, {$set:{req:ReqValue}}, true, false)&lt;br/&gt;
Responses are like:&lt;br/&gt;
db.mt_messages.update(&lt;/p&gt;
{key:Value}
&lt;p&gt;, {$set:{resp:RespValue}}, true, false)&lt;br/&gt;
Both requests and responses are upserts to when they are both received the documents should look like:&lt;/p&gt;
{key:Value, req:ReqValue, resp:RespValue}

&lt;p&gt;3. Kill 10.10.0.222 which is Primary&lt;br/&gt;
4. 10.10.0.223 becomes Primary&lt;br/&gt;
5. 10.10.0.223 Y which is Primary&lt;br/&gt;
6. Start 10.10.0.222 with replSet&lt;br/&gt;
7. 10.10.0.222 becomes Primary&lt;/p&gt;

&lt;p&gt;8. Start 10.10.0.223 without replSet and make export&lt;br/&gt;
$ mongoexport -d kelly_2013_01_01 -c mt_messages -o mt_messages.json&lt;br/&gt;
9. Stop 10.10.0.223&lt;/p&gt;

&lt;p&gt;10. Start 10.10.0.223 with replSet&lt;br/&gt;
11. 10.10.0.223 rollbacks as expected and becomes Secondary&lt;br/&gt;
12. After applying all rollbacks I end up with lost data even though the missing data exist in the export file mt_messages.json.&lt;/p&gt;


&lt;p&gt;Now let&apos;s see the data.&lt;/p&gt;

&lt;p&gt;kelly:PRIMARY&amp;gt; db.mt_messages.count()&lt;br/&gt;
5000&lt;/p&gt;

&lt;p&gt;kelly:PRIMARY&amp;gt; db.mt_messages.find({rqt:{$exists:false}}).count()&lt;br/&gt;
31&lt;/p&gt;

&lt;p&gt;kelly:PRIMARY&amp;gt; db.mt_messages.find({rps:{$exists:false}}).count()&lt;br/&gt;
5000&lt;/p&gt;

&lt;p&gt;kelly:PRIMARY&amp;gt; db.mt_messages.find({rqt:{$exists:false}})&lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;5167c605efb160392cf4fbc2&quot;), &quot;imi&quot; : &quot;11589144&quot;, &quot;omi&quot; : &quot;11582862&quot;, &quot;ri&quot; : &quot;25dfe2d6-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:53.745Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c604efb160392cf4fbb6&quot;), &quot;imi&quot; : &quot;11589157&quot;, &quot;omi&quot; : &quot;11582851&quot;, &quot;ri&quot; : &quot;25e77082-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:53.486Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c604efb160392cf4fbbb&quot;), &quot;imi&quot; : &quot;11589154&quot;, &quot;omi&quot; : &quot;11582857&quot;, &quot;ri&quot; : &quot;25e6a206-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:53.583Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c604efb160392cf4fbbc&quot;), &quot;imi&quot; : &quot;11589151&quot;, &quot;omi&quot; : &quot;11582859&quot;, &quot;ri&quot; : &quot;25e4cc38-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:53.623Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c604efb160392cf4fbba&quot;), &quot;imi&quot; : &quot;11589153&quot;, &quot;omi&quot; : &quot;11582854&quot;, &quot;ri&quot; : &quot;25e6233a-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:53.522Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c604efb160392cf4fbbe&quot;), &quot;imi&quot; : &quot;11589143&quot;, &quot;omi&quot; : &quot;11582858&quot;, &quot;ri&quot; : &quot;25dee606-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:53.647Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c604efb160392cf4fbb8&quot;), &quot;imi&quot; : &quot;11589152&quot;, &quot;omi&quot; : &quot;11582853&quot;, &quot;ri&quot; : &quot;25e57e44-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:53.502Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c604efb160392cf4fbbf&quot;), &quot;imi&quot; : &quot;11589137&quot;, &quot;omi&quot; : &quot;11582865&quot;, &quot;ri&quot; : &quot;25dd3b76-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:53.673Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c605efb160392cf4fbe6&quot;), &quot;imi&quot; : &quot;11589102&quot;, &quot;omi&quot; : &quot;11582905&quot;, &quot;ri&quot; : &quot;25cf1b72-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:54.173Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c605efb160392cf4fbe7&quot;), &quot;imi&quot; : &quot;11589105&quot;, &quot;omi&quot; : &quot;11582906&quot;, &quot;ri&quot; : &quot;25d0161c-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:54.169Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c605efb160392cf4fbe5&quot;), &quot;imi&quot; : &quot;11589113&quot;, &quot;omi&quot; : &quot;11582897&quot;, &quot;ri&quot; : &quot;25d3438c-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:54.167Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c605efb160392cf4fbe2&quot;), &quot;imi&quot; : &quot;11589114&quot;, &quot;omi&quot; : &quot;11582895&quot;, &quot;ri&quot; : &quot;25d371d6-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:54.151Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c605efb160392cf4fbe0&quot;), &quot;imi&quot; : &quot;11589112&quot;, &quot;omi&quot; : &quot;11582896&quot;, &quot;ri&quot; : &quot;25d2f706-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:54.153Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c605efb160392cf4fbe1&quot;), &quot;imi&quot; : &quot;11589118&quot;, &quot;omi&quot; : &quot;11582892&quot;, &quot;ri&quot; : &quot;25d4d4fe-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:54.148Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c605efb160392cf4fbde&quot;), &quot;imi&quot; : &quot;11589115&quot;, &quot;omi&quot; : &quot;11582893&quot;, &quot;ri&quot; : &quot;25d3be5c-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:54.133Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c605efb160392cf4fbdf&quot;), &quot;imi&quot; : &quot;11589119&quot;, &quot;omi&quot; : &quot;11582891&quot;, &quot;ri&quot; : &quot;25d560b8-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:54.127Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c605efb160392cf4fbda&quot;), &quot;imi&quot; : &quot;11589116&quot;, &quot;omi&quot; : &quot;11582889&quot;, &quot;ri&quot; : &quot;25d43594-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:54.114Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c605efb160392cf4fbd9&quot;), &quot;imi&quot; : &quot;11589127&quot;, &quot;omi&quot; : &quot;11582885&quot;, &quot;ri&quot; : &quot;25d8f070-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:54.113Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c605efb160392cf4fbd7&quot;), &quot;imi&quot; : &quot;11589122&quot;, &quot;omi&quot; : &quot;11582882&quot;, &quot;ri&quot; : &quot;25d77d3a-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:54.093Z&quot;) }
{ &quot;_id&quot; : ObjectId(&quot;5167c605efb160392cf4fbd5&quot;), &quot;imi&quot; : &quot;11589131&quot;, &quot;omi&quot; : &quot;11582869&quot;, &quot;ri&quot; : &quot;25db5f2c-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : ISODate(&quot;2013-01-02T18:02:54.090Z&quot;) }
&lt;p&gt;Type &quot;it&quot; for more&lt;br/&gt;
kelly:PRIMARY&amp;gt;&lt;/p&gt;

&lt;p&gt;We miss 31 updates.&lt;br/&gt;
Search the first &quot;imi&quot;:&quot;11589144&quot; which is unique enough for this purpose.&lt;/p&gt;

&lt;p&gt;on 10.10.0.222 the response is registered.&lt;br/&gt;
$ grep 11589144 /var/log/mongodb.log&lt;br/&gt;
Fri Apr 12 11:29:57.026 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn29&amp;#93;&lt;/span&gt; update kelly_2013_01_01.mt_messages query: &lt;/p&gt;
{ ri: &quot;25dfe2d6-a34b-11e2-994b-000c29a75b4d&quot;, imi: &quot;11589144&quot; }
&lt;p&gt; update: { $set: &lt;/p&gt;
{ omi: &quot;11582862&quot;, rpt: new Date(1357149773745), rps: &quot;success&quot; }
&lt;p&gt; } nscanned:0 nupdated:1 fastmodinsert:1 keyUpdates:0 locks(micros) w:148 0ms&lt;/p&gt;

&lt;p&gt;on 10.10.0.223 the request is registered.&lt;br/&gt;
$ grep 11589144 /var/log/mongodb.log&lt;br/&gt;
Fri Apr 12 11:30:12.765 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn18&amp;#93;&lt;/span&gt; update kelly_2013_01_01.mt_messages query: &lt;/p&gt;
{ ri: &quot;25dfe2d6-a34b-11e2-994b-000c29a75b4d&quot;, imi: &quot;11589144&quot; }
&lt;p&gt; update: { $set: { ct: &quot;funnel&quot;, ci: &quot;493b3678-9dc8-11e2-8cce-00269e42f7a5&quot;, ui: &quot;user&quot;, gi: &quot;7dc235d0-c938-4b66-8f8c-c9037c7eace7&quot;, t: &quot;regular&quot;, e: &quot;default&quot;, b: &quot;User&quot;, sa: &lt;/p&gt;
{ a: &quot;375296660002&quot;, t: 1, n: 1 }
&lt;p&gt;, da: &lt;/p&gt;
{ a: &quot;375293615363&quot;, t: 1, n: 1 }
&lt;p&gt;, rd: false, rqt: new Date(1357149779760) } } nscanned:1 nmoved:1 nupdated:1 keyUpdates:0 locks(micros) w:433 0ms&lt;/p&gt;

&lt;p&gt;on 10.10.0.223 whole document is available according to the export file&lt;br/&gt;
$ grep 11589144 mt_messages.json&lt;br/&gt;
{ &quot;_id&quot; : &lt;/p&gt;
{ &quot;$oid&quot; : &quot;5167c605efb160392cf4fbc2&quot; }
&lt;p&gt;, &quot;b&quot; : &quot;User&quot;, &quot;ci&quot; : &quot;493b3678-9dc8-11e2-8cce-00269e42f7a5&quot;, &quot;ct&quot; : &quot;funnel&quot;, &quot;da&quot; : &lt;/p&gt;
{ &quot;a&quot; : &quot;375293615363&quot;, &quot;t&quot; : 1, &quot;n&quot; : 1 }
&lt;p&gt;, &quot;e&quot; : &quot;default&quot;, &quot;gi&quot; : &quot;7dc235d0-c938-4b66-8f8c-c9037c7eace7&quot;, &quot;imi&quot; : &quot;11589144&quot;, &quot;omi&quot; : &quot;11582862&quot;, &quot;rd&quot; : false, &quot;ri&quot; : &quot;25dfe2d6-a34b-11e2-994b-000c29a75b4d&quot;, &quot;rps&quot; : &quot;success&quot;, &quot;rpt&quot; : &lt;/p&gt;
{ &quot;$date&quot; : 1357149773745 }
&lt;p&gt;, &quot;rqt&quot; : &lt;/p&gt;
{ &quot;$date&quot; : 1357149779760 }
&lt;p&gt;, &quot;sa&quot; : &lt;/p&gt;
{ &quot;a&quot; : &quot;375296660002&quot;, &quot;t&quot; : 1, &quot;n&quot; : 1 }
&lt;p&gt;, &quot;t&quot; : &quot;regular&quot;, &quot;ui&quot; : &quot;user&quot; }&lt;/p&gt;

&lt;p&gt;on 10.10.0.223&lt;br/&gt;
$ bsondump kelly_2013_01_01.mt_messages.2013-04-12T08-31-25.0.bson 2&amp;gt;/dev/null | grep 11589144&lt;br/&gt;
IT&apos;S NOT HERE! IT&apos;S GONE!&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10166" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Tests Written</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10154"><![CDATA[Complete]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrob3b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>