<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:36:36 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-55486] Design new integration test facility to use Device-mapper to reproduce disk outage</title>
                <link>https://jira.mongodb.org/browse/SERVER-55486</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;h2&gt;&lt;a name=&quot;Background&quot;&gt;&lt;/a&gt;Background&lt;/h2&gt;

&lt;p&gt;We have a HELP ticket scenario when a faulty disk made all disk I/O to be blocked for indefinite time, which caused the process to enter the uninterruptible sleep state. The main culprit of this state is that when SIGKILL is issued the process is not killed because it&apos;s blocked on a syscall. The user killed the primary mongod server with -9 but it was not killed. After 13 minutes after SIGKILL, the user had to shut down the Amazon EC2 instance to break down hung sessions from multiple mongos proxies to the faulty primary. This happened with v4.0.&lt;/p&gt;

&lt;p&gt;More background on why `kill -9` will never kill the process in the uninterruptible sleep state:&lt;br/&gt;
&lt;a href=&quot;https://askubuntu.com/questions/59811/kill-pid-not-really-killing-the-process-why&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://askubuntu.com/questions/59811/kill-pid-not-really-killing-the-process-why&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Various tricks people use to simulate the uninterruptible sleep state:&lt;br/&gt;
&lt;a href=&quot;https://unix.stackexchange.com/questions/134888/simulate-an-unkillable-process-in-d-state&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://unix.stackexchange.com/questions/134888/simulate-an-unkillable-process-in-d-state&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;More background on why kernel prevents killing process in this kind of state:&lt;br/&gt;
&lt;a href=&quot;https://stackoverflow.com/questions/223644/what-is-an-uninterruptible-process&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://stackoverflow.com/questions/223644/what-is-an-uninterruptible-process&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;and LWN article: &lt;a href=&quot;https://lwn.net/Articles/288056/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://lwn.net/Articles/288056/&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;&lt;a name=&quot;Implementationdetails&quot;&gt;&lt;/a&gt;Implementation details&lt;/h2&gt;

&lt;p&gt;The process of setting the device mapper has multiple steps:&lt;br/&gt;
1. Create a file and map the file to a new loopback device using `losetup`&lt;br/&gt;
2. Use `dmsetup` to map the loopback device to a device mapper device&lt;br/&gt;
3. Use `mkfs.ext4` to format it&lt;br/&gt;
4. Create the directory structure for replica set&lt;br/&gt;
5. Mount one of the replicas (rs1) data directory to the mapper device&lt;br/&gt;
6. Use `dmsetup suspend` to simulate disk outage&lt;/p&gt;

&lt;p&gt;This procedure was already done manually and fully reproduced the production outage.&lt;/p&gt;

&lt;h3&gt;&lt;a name=&quot;Notthesameasnetworkproxy&quot;&gt;&lt;/a&gt;Not the same as network proxy&lt;/h3&gt;

&lt;p&gt;Please note that we already have mongobridge to simulate network errors, however this is not the same. The mongo bridge cannot make the outage in the mongod, it can only make the client to think that mongod has an outage, which is very different from the scenario in HELP ticket.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1657852">SERVER-55486</key>
            <summary>Design new integration test facility to use Device-mapper to reproduce disk outage</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="2">Won&apos;t Fix</resolution>
                                        <assignee username="backlog-server-sharding-nyc">[DO NOT USE] Backlog - Sharding NYC</assignee>
                                    <reporter username="andrew.shuvalov@mongodb.com">Andrew Shuvalov</reporter>
                        <labels>
                            <label>sharding-product-sync</label>
                    </labels>
                <created>Wed, 24 Mar 2021 15:16:57 +0000</created>
                <updated>Tue, 6 Dec 2022 01:29:25 +0000</updated>
                            <resolved>Fri, 23 Apr 2021 14:14:18 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="3681828" author="JIRAUSER1256988" created="Wed, 24 Mar 2021 16:23:11 +0000"  >&lt;p&gt;The trick is that device mapper can be suspended, while loopback device cannot. But device mapper cannot map directly from file. So the topology is:&lt;/p&gt;

&lt;p&gt;file -&amp;gt; loopback device -&amp;gt; device mapper&lt;/p&gt;</comment>
                            <comment id="3681621" author="JIRAUSER1256988" created="Wed, 24 Mar 2021 15:17:43 +0000"  >&lt;p&gt;References: this is related to HELP-22913.&lt;/p&gt;

&lt;p&gt;The document on mongo bridge is:&lt;br/&gt;
&lt;a href=&quot;https://docs.google.com/document/d/1554KDAnJ-i6EQDDhwGuDulP-uFTJESzL3C4dafjqZvk/edit#heading=h.qsuqufq21x78&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.google.com/document/d/1554KDAnJ-i6EQDDhwGuDulP-uFTJESzL3C4dafjqZvk/edit#heading=h.qsuqufq21x78&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="1657867">SERVER-55487</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25134"><![CDATA[Sharding NYC]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 46 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 46 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-sharding-nyc</customfieldvalue>
            <customfieldvalue>andrew.shuvalov@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hz100f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hym6ov:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hz0m9j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>