<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:38:38 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-35038] Violations of linearizability in MongoDB 3.6.4</title>
                <link>https://jira.mongodb.org/browse/SERVER-35038</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hey folks,&lt;/p&gt;

&lt;p&gt;It looks like I found a violation of linearizability in MongoDB 3.6.4 - it returns stale data after a network partition is healed.&lt;/p&gt;

&lt;p&gt;The docker scripts with repro steps: &lt;a href=&quot;https://github.com/rystsov/consistency-mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/rystsov/consistency-mongodb&lt;/a&gt;&lt;br/&gt;
Logs: &lt;a href=&quot;https://github.com/rystsov/consistency-mongodb/tree/master/experiments/may-17&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/rystsov/consistency-mongodb/tree/master/experiments/may-17&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Structure of the test:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Four nodes in the system&lt;/li&gt;
	&lt;li&gt;Three host MongoDB&apos;s replica set&lt;/li&gt;
	&lt;li&gt;Same nodes host a thin Node.js application with talks with the DB and exposes http interface to read, create and update a key/value pair&lt;/li&gt;
	&lt;li&gt;Fourth node is a client working with http interface of each node and detecting consistency violations&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The client has three async/await coroutines to sequentially update value of its key using an increasing sequence. Each coroutine works with its key and node. Updates to key1 go to node1, key2 to node2 and key3 to node3. Also there are 9 reading coroutines: three coroutines per node to read &quot;key1&quot;, &quot;key2&quot; and &quot;key3&quot; keys.&lt;/p&gt;

&lt;p&gt;All reads were done with &quot;linearizable&quot; read concern and &quot;majority&quot; write concern.&lt;/p&gt;

&lt;p&gt;It looks like the bug may be in the replication layer of the database or the MongoDB driver for Node.js (3.1.0-beta4).&lt;/p&gt;</description>
                <environment></environment>
        <key id="545294">SERVER-35038</key>
            <summary>Violations of linearizability in MongoDB 3.6.4</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="william.schultz@mongodb.com">William Schultz</assignee>
                                    <reporter username="rystsov">Denis Rystsov</reporter>
                        <labels>
                    </labels>
                <created>Thu, 17 May 2018 09:03:36 +0000</created>
                <updated>Wed, 13 Jun 2018 03:36:14 +0000</updated>
                            <resolved>Wed, 13 Jun 2018 03:03:45 +0000</resolved>
                                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>16</watches>
                                                                                                                <comments>
                            <comment id="1919194" author="william.schultz" created="Wed, 13 Jun 2018 03:30:55 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=rystsov&quot; class=&quot;user-hover&quot; rel=&quot;rystsov&quot;&gt;rystsov&lt;/a&gt;, thanks for the bug report. I see now that you&apos;ve already acknowledged a bug in your testing setup, but I figured I might as well follow up just for reference, since I had already done some investigation on this.&lt;/p&gt;

&lt;p&gt;After reading through your code and reproducing your experiments myself, I had a few thoughts. As I understand it, in the histories generated by your test scripts, &lt;tt&gt;write-start&lt;/tt&gt; events are paired with &lt;tt&gt;write-ack&lt;/tt&gt; events, and, similarly,&#160;&lt;tt&gt;read-start&lt;/tt&gt; events paired with &lt;tt&gt;read-ack&lt;/tt&gt; events. The writer threads execute their updates sequentially, and the values being updated increase monotonically. This all seems reasonable. There is something that appears problematic to me, however, with how you detect linearizability violations. At the beginning of this &lt;tt&gt;&lt;a href=&quot;https://github.com/rystsov/consistency-mongodb/blob/cc2e99012979a98399472fe0f71989c1209631ce/client/app/src/main.js#L152-L184&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;read&lt;/a&gt;&lt;/tt&gt; function,&#160;you try to verify linearizability with the following procedure. You first &lt;a href=&quot;https://github.com/rystsov/consistency-mongodb/blob/cc2e99012979a98399472fe0f71989c1209631ce/client/app/src/main.js#L158-L161&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;look for the index of the last acknowledged write attempt to key K&lt;/a&gt; in that key&apos;s write history. Call that index &lt;tt&gt;i_last&lt;/tt&gt;. Then, you do a &lt;a href=&quot;https://github.com/rystsov/consistency-mongodb/blob/cc2e99012979a98399472fe0f71989c1209631ce/client/app/src/main.js#L164&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;linearizable read&lt;/a&gt; of key K, and save that value, call it &lt;tt&gt;v_read&lt;/tt&gt;. Finally, you &lt;a href=&quot;https://github.com/rystsov/consistency-mongodb/blob/cc2e99012979a98399472fe0f71989c1209631ce/client/app/src/main.js#L166-L171&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;step forward in the history of update attempts&lt;/a&gt; for key K, starting at &lt;tt&gt;i_last&lt;/tt&gt;, looking for an update with a value that matches &lt;tt&gt;v_read&lt;/tt&gt;. If you find one, then the &lt;tt&gt;read&lt;/tt&gt; function finishes, without raising an error. However, if you don&apos;t find an update with a value equivalent to &lt;tt&gt;v_read&lt;/tt&gt;, then you &lt;a href=&quot;https://github.com/rystsov/consistency-mongodb/blob/cc2e99012979a98399472fe0f71989c1209631ce/client/app/src/main.js#L172-L177&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;consider it an error&lt;/a&gt;, producing the &apos;never written or stale data&apos; message. This verification method, though, doesn&apos;t appear to take into account previous writes that received no definite acknowledgement or error. Writes that never return a response should be allowed to take effect at &lt;em&gt;any&lt;/em&gt; later time, without violating linearizability.&lt;/p&gt;

&lt;p&gt;As an example of this, consider the &lt;a href=&quot;https://github.com/rystsov/consistency-mongodb/blob/bff87682d77b8d0bc688d527d8ffcd30617d7a7e/experiments/may-17/events&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;events&lt;/a&gt;&#160;history from your may-17 experiment. You claim that the linearizability violation is observed in the following set of events (filtered to only include key3, node1 events, and line numbers included for reference):&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7654: {&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;event&quot;:&quot;read-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7671: {&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;value&quot;:&quot;375&quot;,&quot;event&quot;:&quot;read-ack&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7672: {&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;event&quot;:&quot;read-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7686: {&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;value&quot;:&quot;371&quot;,&quot;event&quot;:&quot;read-ack&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;We see two non-concurrent reads of key3 from node1. The earlier one returns 375 and the later one returns 371. You claim that this cannot be possible because each writer thread updates its key sequentially and monotonically. What is not shown, however, are the write related events for the relevant values i.e. 371, 375, which happen a short time before. This is the part of the event sequence for (key3, node1) read/write events leading up to the violation you described:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7601:{&quot;key&quot;:&quot;key3&quot;,&quot;value&quot;:&quot;371&quot;,&quot;event&quot;:&quot;write-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7606:{&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;event&quot;:&quot;read-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7613:{&quot;key&quot;:&quot;key3&quot;,&quot;value&quot;:&quot;372&quot;,&quot;event&quot;:&quot;write-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7618:{&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;event&quot;:&quot;read-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7625:{&quot;key&quot;:&quot;key3&quot;,&quot;value&quot;:&quot;373&quot;,&quot;event&quot;:&quot;write-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7630:{&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;event&quot;:&quot;read-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7637:{&quot;key&quot;:&quot;key3&quot;,&quot;value&quot;:&quot;374&quot;,&quot;event&quot;:&quot;write-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7642:{&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;event&quot;:&quot;read-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7649:{&quot;key&quot;:&quot;key3&quot;,&quot;value&quot;:&quot;375&quot;,&quot;event&quot;:&quot;write-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7654:{&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;event&quot;:&quot;read-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7667:{&quot;key&quot;:&quot;key3&quot;,&quot;event&quot;:&quot;write-ack&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7668:{&quot;key&quot;:&quot;key3&quot;,&quot;value&quot;:&quot;376&quot;,&quot;event&quot;:&quot;write-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7671:{&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;value&quot;:&quot;375&quot;,&quot;event&quot;:&quot;read-ack&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7672:{&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;event&quot;:&quot;read-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7686:{&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;value&quot;:&quot;371&quot;,&quot;event&quot;:&quot;read-ack&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;It is clear to see that updates to key3 for 371..374 all went unacknowledged. If an invocation of a write never receives a response, it should be valid to occur at any later time. This means it would be valid for the key3=371 update it to occur immediately after the &lt;tt&gt;read-ack&lt;/tt&gt; of key3=375 completed, causing the next read to return 371, which we see in this history. Does this illustrate the same issue that you mentioned as the &quot;bug in your testing setting&quot;?&lt;/p&gt;</comment>
                            <comment id="1919187" author="thomas.schubert" created="Wed, 13 Jun 2018 03:03:46 +0000"  >&lt;p&gt;Thanks for following up, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=rystsov&quot; class=&quot;user-hover&quot; rel=&quot;rystsov&quot;&gt;rystsov&lt;/a&gt;! I&apos;m glad you were able to determine the root cause of the issue.&lt;/p&gt;</comment>
                            <comment id="1919148" author="rystsov" created="Wed, 13 Jun 2018 01:15:24 +0000"  >&lt;p&gt;Everything works as expected, it was bug in my testing setting, sorry for bothering &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="1918850" author="rystsov" created="Tue, 12 Jun 2018 19:57:51 +0000"  >&lt;p&gt;Also I managed to reproduce the issue with Jepsen:&#160;&lt;a href=&quot;https://github.com/rystsov/consistency-mongodb/tree/jepsen&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/rystsov/consistency-mongodb/tree/jepsen&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1900656" author="schwerin" created="Thu, 24 May 2018 14:16:03 +0000"  >&lt;p&gt;Thanks for the report, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=rystsov&quot; class=&quot;user-hover&quot; rel=&quot;rystsov&quot;&gt;rystsov&lt;/a&gt;, and thanks for sharing the code of your reproduction. It will take us a little while to thoroughly analyze the repro, but you&apos;ve given us a lot to get started with.&lt;/p&gt;</comment>
                            <comment id="1900397" author="rystsov" created="Thu, 24 May 2018 07:23:33 +0000"  >&lt;p&gt;By the way, I&apos;ve updated the repo and implemented the HTTP API using Java driver and still was able to run into stale data, logs: &lt;a href=&quot;https://github.com/rystsov/consistency-mongodb/tree/master/experiments/may-24.&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/rystsov/consistency-mongodb/tree/master/experiments/may-24.&lt;/a&gt;&#160;The only difference is that I was able to see violations after I isolated/rejoined a primary twice (see steps for repro steps).&lt;/p&gt;

&lt;p&gt;So it looks like the problem is in replication layer rather than in Node.js driver.&lt;/p&gt;</comment>
                            <comment id="1894919" author="rystsov" created="Fri, 18 May 2018 04:27:25 +0000"  >&lt;p&gt;Unexpected behavior: MongoDB returned stale data when a former isolated primary rejoined the cluster.&lt;/p&gt;

&lt;p&gt;Details:&lt;/p&gt;

&lt;p&gt;I was updating key &lt;ins&gt;key3&lt;/ins&gt; via &lt;ins&gt;node3&lt;/ins&gt; and reading it from all nodes (each node in the replica set was also hosting an app exposing key-value read/write http interface). In the beginning of the experiment &lt;ins&gt;node1&lt;/ins&gt; was primary, I isolated it from the network using iptable rules, waited for 20 seconds and rejoined &lt;ins&gt;node1&lt;/ins&gt; to the cluster. Just after I did, the client read key &lt;ins&gt;key3&lt;/ins&gt; via &lt;ins&gt;node1&lt;/ins&gt; and received data which were older than known last written data on the moment of the request.&lt;/p&gt;

&lt;p&gt;Logs: &lt;a href=&quot;https://github.com/rystsov/consistency-mongodb/tree/master/experiments/may-17&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/rystsov/consistency-mongodb/tree/master/experiments/may-17&lt;/a&gt;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;&quot;node{1-3}.log&quot; are mongod&apos;s logs corresponding to each node&lt;/li&gt;
	&lt;li&gt;&quot;events&quot; contains events of start and end of read/write operations ordered by &quot;wall clock&quot; (Node.js is single threaded so we don&apos;t have ambiguity)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Using &quot;events&quot; we can see that the client first read &quot;key3=375&quot; and then read &quot;key3=371&quot; which is impossible because there is only a single writer whose updates form an increasing sequence so every next read must be greater than the previous:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&quot;key&quot;:(&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;event&quot;:&quot;read-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;value&quot;:&quot;375&quot;,&quot;event&quot;:&quot;read-ack&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;event&quot;:&quot;read-start&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&quot;key&quot;:&quot;key3&quot;,&quot;from&quot;:&quot;node1&quot;,&quot;value&quot;:&quot;371&quot;,&quot;event&quot;:&quot;read-ack&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Versions:&lt;/p&gt;

&lt;p&gt;macOS : 10.13.3 (17D47)&lt;br/&gt;
 Docker: 18.03.0-ce, build 0520e24&lt;br/&gt;
 Ubuntu: 17.10&lt;br/&gt;
 MongoDB: v3.6.4&lt;br/&gt;
 MongoDB Node.js driver: 3.1.0-beta4&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 24 May 2018 14:16:03 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 35 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>william.schultz@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 35 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>schwerin@mongodb.com</customfieldvalue>
            <customfieldvalue>rystsov</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>william.schultz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htyakn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htpb0f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2297">Repl 2018-06-18</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htxwtz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>