<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Wed Feb 07 21:36:26 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[CSHARP-306] incorrect index on fs.chunks</title>
                <link>https://jira.mongodb.org/browse/CSHARP-306</link>
                <project id="10041" key="CSHARP">C# Driver</project>
                    <description>&lt;p&gt;In MongoGridFS the index on fs.chunks &lt;/p&gt;
{ files_is : 1, n : 1 }
&lt;p&gt; named files_id_1_n_1 is not created with the unique flag. As a consequence when running in a multi-threaded environment (using multiple connections) every so many file uploads you get a &quot;Command &apos;filemd5&apos; failed: exception: chunks out of order&quot; error. &lt;/p&gt;

&lt;p&gt;Changing: &lt;br/&gt;
   chunks.EnsureIndex(&quot;files_id&quot;, &quot;n&quot;) &lt;/p&gt;

&lt;p&gt;to:&lt;br/&gt;
   chunks.EnsureIndex(&lt;br/&gt;
      new IndexKeysBuilder().Ascending(&quot;files_id&quot;, &quot;n&quot;),&lt;br/&gt;
      new IndexOptionsBuilder().SetName(&quot;files_id_1_n_1&quot;).SetUnique(true)&lt;br/&gt;
   );&lt;/p&gt;

&lt;p&gt;fixes the issue. Or am I missing out on something here and is indeed the git code correct?&lt;/p&gt;

&lt;p&gt;Also note that the check in the else &quot;files_id_1_n_1&quot; index is missing the 1 at the end of the name.&lt;/p&gt;</description>
                <environment>Windows, C#, Visual Studio 2010, multi-threaded, latest Git source</environment>
        <key id="21203">CSHARP-306</key>
            <summary>incorrect index on fs.chunks</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="4" iconUrl="https://jira.mongodb.org/images/icons/priorities/minor.svg">Minor - P4</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="robert@mongodb.com">Robert Stam</assignee>
                                    <reporter username="knippers">Daniel Knippers</reporter>
                        <labels>
                            <label>driver</label>
                    </labels>
                <created>Fri, 19 Aug 2011 08:59:01 +0000</created>
                <updated>Mon, 11 Mar 2019 18:48:28 +0000</updated>
                            <resolved>Mon, 29 Aug 2011 15:35:13 +0000</resolved>
                                                    <fixVersion>1.2</fixVersion>
                                                        <votes>0</votes>
                                    <watches>1</watches>
                                                                                                                <comments>
                            <comment id="51476" author="knippers" created="Tue, 30 Aug 2011 15:14:21 +0000"  >&lt;p&gt;Both the RequestStart and the SafeMode.True option (tested separately of course) resolve the &quot;chunks out of order&quot; error when the index is not set to unique. Note that I inserted about 36 million zipped xml documents already in a test database without the &quot;chunks out of order&quot; error by just creating the files_id/n index with unique option. That makes it even more that I wish I could explain the issue &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/wink.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="51454" author="rstam" created="Tue, 30 Aug 2011 14:25:48 +0000"  >&lt;p&gt;I agree. I also wish I could explain why changing the index to unique seemed to fix the &quot;chunks out of order&quot; error. I suspect it didn&apos;t really fix it, just changed the probability of seeing it to low enough that you weren&apos;t seeing it any more.&lt;/p&gt;

&lt;p&gt;The RequestStart fix (or SafeMode.True) fixes do make sense though. If you have time to test that change to your code &lt;b&gt;without&lt;/b&gt; changing the index to unique that would help verify the theory that the chunks haven&apos;t all arrived at the server in time for the md5 command.&lt;/p&gt;</comment>
                            <comment id="51403" author="knippers" created="Tue, 30 Aug 2011 06:55:37 +0000"  >&lt;p&gt;Thanks for the feedback and support. Adding the database.RequestStart() makes sense. Still I am left with a slightly strange feeling as to why the index fix solved the issue (I like to be able to explain issues for 100%, especially when it comes to databases). &lt;/p&gt;</comment>
                            <comment id="51300" author="rstam" created="Mon, 29 Aug 2011 20:31:35 +0000"  >&lt;p&gt;Good info. So looking at the code I see two things:&lt;/p&gt;

&lt;p&gt;1. You are using the stream-based API (the stream is returned by fileInfo.Open)&lt;br/&gt;
2. You are &lt;b&gt;not&lt;/b&gt; using SafeMode (because the default is off)&lt;/p&gt;

&lt;p&gt;I think you need to surround your code in the StoreFile method with:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;using (database.RequestStart()) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    // the bulk of your method here&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    // RequestStart causes all operations to be done on the same connection&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;In the absence of RequestStart writes to the gridFile stream can be distributed over many connections. And since SafeMode is off the writes can easily back up and be queued in the connections&apos; output buffers. My theory in this scenario is that after the file has been uploaded and the md5 command is sent to the server that not all the chunks have made it up to the server yet.&lt;/p&gt;

&lt;p&gt;Turning SafeMode on would also fix the &quot;chunks out of order&quot; error because it has the side effect of waiting after every write to make sure the write succeeded. With SafeMode on you wouldn&apos;t need the RequestStart, but it wouldn&apos;t hurt either.&lt;/p&gt;

&lt;p&gt;Still not sure how the unique index helps suppress the &quot;chunks out of order&quot; error, but it must have some indirect side effect as you have suggested.&lt;/p&gt;
</comment>
                            <comment id="51296" author="knippers" created="Mon, 29 Aug 2011 20:19:02 +0000"  >&lt;p&gt;Slightly simplified method that is used to store a file:&lt;/p&gt;

&lt;p&gt;private MongoDBRef StoreFile(string FileName)&lt;br/&gt;
{&lt;br/&gt;
   // member in real code&lt;br/&gt;
   // &lt;br/&gt;
   MongoDatabase database = MongoServer.Create(&quot;localhost&quot;).GetDatabase(&quot;TestDB&quot;);&lt;/p&gt;

&lt;p&gt;   MongoGridFSFileInfo fileInfo = database.GridFS.FindOne(FileName);&lt;/p&gt;

&lt;p&gt;   if (fileInfo != null)&lt;br/&gt;
      fileInfo.Delete();&lt;/p&gt;

&lt;p&gt;   MongoGridFSCreateOptions options = new MongoGridFSCreateOptions();&lt;br/&gt;
   options.UploadDate = DateTime.UtcNow;&lt;br/&gt;
   options.Metadata = new BsonDocument().Add(&quot;lastWriteDate&quot;, new FileInfo(FileName).LastWriteTimeUtc)&lt;br/&gt;
      .Add(&quot;createDate&quot;, new FileInfo(FileName).CreationTimeUtc).Add(&quot;compressed&quot;, True);&lt;/p&gt;

&lt;p&gt;   options.ContentType = &quot;application/zip&quot;;&lt;/p&gt;

&lt;p&gt;   fileInfo = new MongoGridFSFileInfo(database.GridFS, FileName, options);&lt;/p&gt;

&lt;p&gt;   using (MongoGridFSStream gridFile = fileInfo.Open(FileMode.Create, FileAccess.ReadWrite))&lt;br/&gt;
   using (FileStream inFile = File.OpenRead(FileName))&lt;br/&gt;
   {&lt;br/&gt;
      // see: &lt;a href=&quot;http://zipstorer.codeplex.com/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://zipstorer.codeplex.com/&lt;/a&gt; + fix from issue #8583&lt;br/&gt;
      // &lt;br/&gt;
      using (ZipStorer zipFile = ZipStorer.Create(gridFile, &quot;&quot;))&lt;/p&gt;
      {
         zipFile.AddStream(ZipStorer.Compression.Deflate, Path.GetFileName(FileName), inFile, DateTime.Now, &quot;&quot;);
         zipFile.Close();
      }
&lt;p&gt;   }&lt;/p&gt;

&lt;p&gt;   return new MongoDBRef(database.Name, database.GridFS.Files.Name, fileInfo.Id);&lt;br/&gt;
}&lt;/p&gt;</comment>
                            <comment id="51292" author="rstam" created="Mon, 29 Aug 2011 20:02:27 +0000"  >&lt;p&gt;Can you share some details about how you upload your GridFS files? Are you using the Upload method of MongoGridFS, or are you using one of the stream based APIs? Are you using SafeMode?&lt;/p&gt;

&lt;p&gt;That might help me figure out how there could be chunks missing.&lt;/p&gt;</comment>
                            <comment id="51287" author="knippers" created="Mon, 29 Aug 2011 19:46:41 +0000"  >&lt;p&gt;Agreed that files_id values are different. The only think I can think of is that enforcing the unique index is using some sort of locking that kind of acts as a synchronized/using lock mechanism for the file insert + md5 hash command.&lt;/p&gt;</comment>
                            <comment id="51259" author="rstam" created="Mon, 29 Aug 2011 18:09:39 +0000"  >&lt;p&gt;If two threads were uploading two GridFS files simultaneously, wouldn&apos;t the files_id values be different so there wouldn&apos;t even be a chance of collisions in the first place?&lt;/p&gt;</comment>
                            <comment id="51255" author="knippers" created="Mon, 29 Aug 2011 17:58:49 +0000"  >&lt;p&gt;Thanks for fixing. Don&apos;t have a validated explanation but what I can think of is that with the unique index two connections wanting to insert a grid fs file (and thus chunks) at exactly the same time get the benefit of the db engine using some sort of locking to make sure the index uniqueness is enforced. &lt;/p&gt;</comment>
                            <comment id="51198" author="rstam" created="Mon, 29 Aug 2011 15:35:13 +0000"  >&lt;p&gt;I have incorporated your suggested changes. You are correct that the driver was not following the GridFS spec exactly.&lt;/p&gt;

&lt;p&gt;However, I cannot explain why changing this index to unique would prevent the &quot;chunks out of order&quot; error message. This error message can be the result of either of two things: a missing chunk or a duplicate chunk. The unique index would prevent duplicate chunks, but not missing chunks. I suspect users seeing this error message are actually missing chunks, in which case the unique index would make no difference. If you have an explanation for how changing this index to unique would resolve the &quot;chunks out of order&quot; I would be very interested.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrh8i7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>14177</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>