<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Wed Feb 07 21:20:26 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[CDRIVER-4262] Memory consumption possibly too high with cursor</title>
                <link>https://jira.mongodb.org/browse/CDRIVER-4262</link>
                <project id="10030" key="CDRIVER">C Driver</project>
                    <description>&lt;h4&gt;&lt;a name=&quot;Summary&quot;&gt;&lt;/a&gt;Summary&lt;/h4&gt;

&lt;p&gt;I was investigating a memory consumption behavior within my own application and I noticed that the Mongo C driver is consuming more memory than I thought was necessary. What I did is that I setup a fake DB with documents that are roughly 2MB in uncompressed size in BSON representation. I then created an application that uses a cursor to iterate over the documents with a batch size of two. I assumed that in any given moment, the memory consumption therefore lies around ~4MB. So I ran the application with Valgrind&apos;s Massif and noticed that the memory consumption was at peak, 8MB in size. So my guessing is that at some point the batch is copied instead of being moved.&lt;/p&gt;

&lt;p&gt;I&apos;ve figured out that the driver copies the server reply in the function&#160;mongoc_cluster_run_opmsg from the statically allocated&#160;reply_local variable to the reply variable.&#160; See this part here:&#160;&lt;a href=&quot;http://example.com&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo-c-driver/blob/ded9ae5e9f2897a283305175aae8e1bbf4021c36/src/libmongoc/src/mongoc/mongoc-cluster.c#L3538&lt;/a&gt;. I&apos;m quite unfamiliar with the C driver so I was not sure if it possible to &quot;move&quot; the memory to the final reply.&#160;&lt;/p&gt;

&lt;p&gt;**I&apos;ve attached the program and the output file of Valgrind.&lt;/p&gt;

&lt;p&gt;&lt;font color=&quot;#1a1a1a&quot;&gt;Environment&lt;/font&gt;&lt;/p&gt;

&lt;p&gt;Linux, Valgrind with driver version 1.20.1&lt;/p&gt;
&lt;h4&gt;&lt;a name=&quot;HowtoReproduce&quot;&gt;&lt;/a&gt;How to Reproduce&lt;/h4&gt;
&lt;ul&gt;
	&lt;li&gt;Populate the database using the script from the ZIP&lt;/li&gt;
	&lt;li&gt;Compile the example from the ZIP&lt;/li&gt;
	&lt;li&gt;Run Valgrind with&#160;valgrind --tool=massif ./build/build/application&lt;/li&gt;
	&lt;li&gt;Inspect the output file, e.g. using MassifVisualizer&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/354773/354773_image-2022-01-13-10-18-34-379.png&quot; height=&quot;489&quot; width=&quot;869&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="1963134">CDRIVER-4262</key>
            <summary>Memory consumption possibly too high with cursor</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="10300" iconUrl="https://jira.mongodb.org/images/icons/priorities/medium.svg">Unknown</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="ezra.chung@mongodb.com">Ezra Chung</assignee>
                                    <reporter username="kai.takac@gmail.com">Kai Takac</reporter>
                        <labels>
                    </labels>
                <created>Thu, 13 Jan 2022 09:25:17 +0000</created>
                <updated>Fri, 27 Oct 2023 13:14:07 +0000</updated>
                            <resolved>Thu, 27 Jan 2022 15:35:00 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="4318412" author="JIRAUSER1261413" created="Thu, 27 Jan 2022 15:34:50 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kai.takac%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;kai.takac@gmail.com&quot;&gt;kai.takac@gmail.com&lt;/a&gt; Thank you once again for the detailed report.&lt;/p&gt;

&lt;p&gt;As you have described, the ~4 MB payload (~2 MB document in a batch of size 2) is being copied during server message response handling, momentarily increasing the total memory usage up to ~8 MB. This momentary copy is fundamentally unavoidable given the C driver&apos;s current implementation of server message response parsing (read entire message from stream into memory buffer, reinterpret memory buffer as read-only RPC data, extract message payload from RPC data view as a BSON document). The extraction of the message payload into a modifiable and owning &lt;tt&gt;bson_t&lt;/tt&gt; object from the read-only RPC data view (whose corresponding memory is owned by the local memory buffer) is the reason behind the temporary duplication of the ~4 MB payload. Eliminating this copy would necessitate a non-trivial refactor of how server message responses are currently being parsed by the C driver.&lt;/p&gt;

&lt;p&gt;As this behavior is working as designed, this bug ticket will be closed.&lt;/p&gt;</comment>
                            <comment id="4290636" author="kevin.albertson" created="Thu, 13 Jan 2022 14:03:14 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kai.takac%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;kai.takac@gmail.com&quot;&gt;kai.takac@gmail.com&lt;/a&gt;, thank you for the detailed report and visualization. We will take a look at this soon.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="354773" name="image-2022-01-13-10-18-34-379.png" size="301748" author="kai.takac@gmail.com" created="Thu, 13 Jan 2022 09:18:36 +0000"/>
                            <attachment id="354772" name="mongoc-memory-example.zip" size="8904" author="kai.takac@gmail.com" created="Thu, 13 Jan 2022 09:24:02 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hzzurz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>