<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Wed Feb 07 21:13:54 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[CDRIVER-2018] Introduce bulk write options to disable BSON validation</title>
                <link>https://jira.mongodb.org/browse/CDRIVER-2018</link>
                <project id="10030" key="CDRIVER">C Driver</project>
                    <description>&lt;p&gt;When &lt;a href=&quot;https://jira.mongodb.org/browse/CDRIVER-1341&quot; title=&quot;Driver should validate BSON documents before insert and update&quot; class=&quot;issue-link&quot; data-issue-key=&quot;CDRIVER-1341&quot;&gt;&lt;del&gt;CDRIVER-1341&lt;/del&gt;&lt;/a&gt; added logic to validate insert, replacement, and update documents (&lt;a href=&quot;https://github.com/mongodb/mongo-c-driver/commit/4b37e655ac066514350d08c882d385c881c7a438&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;4b37e65&lt;/a&gt;), it subtly changed some conventions with bulk write execution. I was alerted to this through the PHP driver&apos;s &lt;a href=&quot;https://github.com/mongodb/mongo-php-driver/blob/1.2.5/tests/manager/manager-executeBulkWrite_error-005.phpt&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;manager-executeBulkWrite_error-005.phpt&lt;/a&gt; test. Validation errors are not reported until the bulk write is executed (&lt;a href=&quot;https://github.com/mongodb/mongo-c-driver/blob/1.6.0/src/mongoc/mongoc-bulk-operation.c#L644&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;), which changes a previous assumption that execution would always ensure the server stream was initialized. The existing checks in &lt;tt&gt;mongoc_bulk_operation_execute()&lt;/tt&gt; for valid client/database/collection objects could theoretically have returned within initializing a stream, but it wasn&apos;t a concern for us.&lt;/p&gt;

&lt;p&gt;As it relates to the PHP driver, this means that we&apos;d only be able to throw a BulkWriteException at execution time (from &lt;tt&gt;Manager::executeBulkWrite()&lt;/tt&gt;) instead of an InvalidArgumentException from the offending method (e.g. &lt;tt&gt;BulkWrite::insert()&lt;/tt&gt;). We actually discovered this because BulkWriteException attempts to refer to the server handling execution, and the selected server was null.&lt;/p&gt;

&lt;p&gt;This may only be an issue for &lt;tt&gt;mongoc_bulk_operation_insert()&lt;/tt&gt;, which may be at a disadvantage because it is void and cannot return a boolean indicating success or failure (unlike the update and replace &quot;with_opts&quot; functions). For this reason, I think we either need a new insert function that can return a boolean or some public API that can expose the &lt;tt&gt;mongoc_bulk_operation_t&lt;/tt&gt;&apos;s error so that we can check it after each attempted insert. The update and replace functions have &quot;with_opts&quot; variants, which return a boolean indicating success or failure, as well as a &lt;tt&gt;bson_error_t *&lt;/tt&gt; output argument to store any error encountered. A similar API for insert would be appreciated.&lt;/p&gt;</description>
                <environment></environment>
        <key id="351247">CDRIVER-2018</key>
            <summary>Introduce bulk write options to disable BSON validation</summary>
                <type id="2" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14711&amp;avatarType=issuetype">New Feature</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="backlog-c-driver">Backlog - C Driver Team</assignee>
                                    <reporter username="jmikola@mongodb.com">Jeremy Mikola</reporter>
                        <labels>
                    </labels>
                <created>Thu, 2 Feb 2017 16:04:31 +0000</created>
                <updated>Wed, 3 May 2017 22:14:11 +0000</updated>
                            <resolved>Thu, 23 Mar 2017 00:39:39 +0000</resolved>
                                    <version>1.6.0</version>
                                    <fixVersion>1.7.0</fixVersion>
                                    <component>libmongoc</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="1530883" author="xgen-internal-githook" created="Thu, 23 Mar 2017 00:39:48 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;ajdavis&apos;, u&apos;name&apos;: u&apos;A. Jesse Jiryu Davis&apos;, u&apos;email&apos;: u&apos;jesse@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/CDRIVER-2018&quot; title=&quot;Introduce bulk write options to disable BSON validation&quot; class=&quot;issue-link&quot; data-issue-key=&quot;CDRIVER-2018&quot;&gt;&lt;del&gt;CDRIVER-2018&lt;/del&gt;&lt;/a&gt; add mongoc_bulk_operation_insert_with_opts&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo-c-driver/commit/8c0e06fe2059bed372d2d13b487e222125069952&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo-c-driver/commit/8c0e06fe2059bed372d2d13b487e222125069952&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1498133" author="jmikola@gmail.com" created="Mon, 13 Feb 2017 21:09:21 +0000"  >&lt;p&gt;Summarizing our discussion earlier today:&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;mongoc_collection_insert()&lt;/tt&gt; and &lt;tt&gt;mongoc_collection_insert_bulk()&lt;/tt&gt; both support an undocumented &lt;tt&gt;MONGOC_INSERT_NO_VALIDATE&lt;/tt&gt; flag, which disables &lt;b&gt;all&lt;/b&gt; validation. &lt;tt&gt;mongoc_collection_insert()&lt;/tt&gt; and &lt;tt&gt;MONGOC_INSERT_NO_VALIDATE&lt;/tt&gt; is used internally by libmongoc for legacy index creation, but there is no evidence of the option being used with &lt;tt&gt;mongoc_collection_insert_bulk()&lt;/tt&gt;. If a new option is implemented per this issue, we&apos;d suggest a name related to legacy index creation rather than a generic name that might encourage users to use it for other situations.&lt;/p&gt;

&lt;p&gt;The PHP driver accumulates all documents to insert in a &lt;tt&gt;mongoc_bulk_operation_t&lt;/tt&gt; via the &lt;a href=&quot;http://php.net/manual/en/class.mongodb-driver-bulkwrite.php&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;MongoDB\Driver\BulkWrite&lt;/tt&gt;&lt;/a&gt; methods prior to the object being passed to &lt;a href=&quot;http://php.net/manual/en/mongodb-driver-manager.executebulkwrite.php&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;MongoDB\Driver\Manager::executeBulkWrite()&lt;/tt&gt;&lt;/a&gt; with a target namespace. There is no public API in libmongoc to access data that has been added to &lt;tt&gt;mongoc_bulk_operation_t&lt;/tt&gt;, otherwise we might be able to check the namespace when &lt;tt&gt;executeBulkWrite()&lt;/tt&gt; is called, pull insert documents out, and defer to &lt;tt&gt;mongoc_collection_insert()&lt;/tt&gt; or &lt;tt&gt;mongoc_collection_insert_bulk()&lt;/tt&gt; with &lt;tt&gt;MONGOC_INSERT_NO_VALIDATE&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;We also noted that the introduction of document validation in libmongoc 1.6.0 (&lt;a href=&quot;https://jira.mongodb.org/browse/CDRIVER-1341&quot; title=&quot;Driver should validate BSON documents before insert and update&quot; class=&quot;issue-link&quot; data-issue-key=&quot;CDRIVER-1341&quot;&gt;&lt;del&gt;CDRIVER-1341&lt;/del&gt;&lt;/a&gt;) may already break behavior for PHP driver and library users today if they are using libbson and libmongoc as shared libraries.&lt;/p&gt;

&lt;p&gt;I pinged &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=derick&quot; class=&quot;user-hover&quot; rel=&quot;derick&quot;&gt;derick&lt;/a&gt; to ask if he had any other ideas for working around this short of the API requested in this ticket.&lt;/p&gt;</comment>
                            <comment id="1495046" author="jmikola@gmail.com" created="Wed, 8 Feb 2017 17:02:04 +0000"  >&lt;p&gt;We discussed creating a &lt;tt&gt;bool mongoc_bulk_operation_insert_with_opts()&lt;/tt&gt; function that also accepts &lt;tt&gt;const bson_t *opts&lt;/tt&gt; and &lt;tt&gt;bson_error_t *error&lt;/tt&gt; arguments. Rather than introduce an option for disabling &quot;dot keys&quot; validation, we discussed having a general &quot;validateBson&quot; option for all of the &quot;with_opts&quot; methods that allows libmongoc to skip its BSON validation introduced in &lt;a href=&quot;https://jira.mongodb.org/browse/CDRIVER-1341&quot; title=&quot;Driver should validate BSON documents before insert and update&quot; class=&quot;issue-link&quot; data-issue-key=&quot;CDRIVER-1341&quot;&gt;&lt;del&gt;CDRIVER-1341&lt;/del&gt;&lt;/a&gt;. That seems more justified if folks are passing pre-validated documents to the bulk write methods, as it allows them to avoid potentially expensive checking (e.g. UTF8).  &lt;/p&gt;</comment>
                            <comment id="1491680" author="jmikola@gmail.com" created="Fri, 3 Feb 2017 15:36:59 +0000"  >&lt;blockquote&gt;&lt;p&gt;&quot;insert&quot; is your only API for insertions?&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Correct. All writes are issued through the &lt;a href=&quot;http://php.net/manual/en/mongodb-driver-manager.executebulkwrite.php&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;MongoDB\Driver\Manager::executeBulkWrite()&lt;/a&gt; method. The base driver has three basic methods for executing commands, queries, and bulk writes.&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;Can you use mongoc_collection_create_index_with_opts?&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;I&apos;m aware of the logic this function and others have to fall back for older server versions, but that is something we handle ourselves in PHPLIB after selecting a server is checking it&apos;s min/max wire versions. This was all done purposefully by design to keep our C layer as slim as possible.&lt;/p&gt;

&lt;p&gt;The only way I can imagine we&apos;d be able to use mongoc_collection_create_index_with_opts is if, at the time a bulk write was executed, we checked if the target collection was &quot;system.indexes&quot; on a pre-2.6 server and then used some API, which does not yet exist, to dissect it and obtain the raw insert statements. We&apos;d then have to throw exceptions if the bulk write consisted of anything more than a series of insert statements. That all seems quite a bit more hacky and fragile than a mechanism to bypass the &quot;dot&quot; validation.&lt;/p&gt;</comment>
                            <comment id="1491462" author="jesse" created="Fri, 3 Feb 2017 04:20:13 +0000"  >&lt;p&gt;&quot;insert&quot; is your only API for insertions? Can you use mongoc_collection_create_index_with_opts? That function tries the modern &quot;createIndexes&quot; command and falls back to inserting into the system.indexes collection with MONGOC_INSERT_NO_VALIDATE set. The driver does all the work for you there.&lt;/p&gt;</comment>
                            <comment id="1491278" author="jmikola@gmail.com" created="Thu, 2 Feb 2017 21:50:13 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jesse&quot; class=&quot;user-hover&quot; rel=&quot;jesse&quot;&gt;jesse&lt;/a&gt;: See my comment in &lt;a href=&quot;https://jira.mongodb.org/browse/PHPC-712?focusedCommentId=1439382&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1439382&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;PHPC-712&lt;/a&gt;:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Note: libmongoc uses a &lt;tt&gt;MONGOC_INSERT_NO_VALIDATE&lt;/tt&gt; constant to disable validation for insert documents. This appears to only be used for legacy index creation.&lt;/p&gt;

&lt;p&gt;If we add validation for &lt;tt&gt;BulkWrite::insert()&lt;/tt&gt; object, we should consider either supporting such an option or inspecting the input argument and skipping validation if it looks like a legacy index specification.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;With libmongoc 1.6.0, there is no way for PHPLIB to create indexes on MongoDB 2.4 and earlier, since &lt;tt&gt;MongoDB\BulkWrite::insert()&lt;/tt&gt; is our only API for insertions. I would propose a &lt;tt&gt;mongoc_bulk_operation_insert_with_opts()&lt;/tt&gt; function that accepts an option to exclude &lt;tt&gt;BSON_VALIDATE_DOT_KEYS&lt;/tt&gt; from the validation flags. In effect, this would force the server to validate that aspect of the document.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="293706">PHPC-712</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="351232">CDRIVER-2017</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="293706">PHPC-712</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hswbq7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1641">C Driver 2017 3 - March 7</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>