<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:21:42 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-50092] [FLE] with encryption on collection and $lookup with two non-encrypted collections fails</title>
                <link>https://jira.mongodb.org/browse/SERVER-50092</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;if you perform an aggregation pipeline &lt;tt&gt;$lookup&lt;/tt&gt;&#160;from a MongoClient with encryption, where both of collections involved are &lt;b&gt;not&lt;/b&gt;&#160;encrypted you&apos;ll get &lt;tt&gt;Pipeline over an encrypted collection cannot reference additional collections&lt;/tt&gt;&#160;&lt;/p&gt;

&lt;p&gt;For example, encrypted collection A, non-encrypted collection B and C and you have pipeline from B : &lt;span class=&quot;error&quot;&gt;&amp;#91;\{$lookup:{from:&amp;quot;C&amp;quot;, localField:&amp;quot;Foo&amp;quot;, foreignField:&amp;quot;Bar&amp;quot;, as:&amp;quot;Baz&amp;quot;}}&amp;#93;&lt;/span&gt; you still get the error above.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1427547">SERVER-50092</key>
            <summary>[FLE] with encryption on collection and $lookup with two non-encrypted collections fails</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="10038" iconUrl="https://jira.mongodb.org/images/icons/subtask.gif" description="">Backlog</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-query-optimization">Backlog - Query Optimization</assignee>
                                    <reporter username="asya.kamsky@mongodb.com">Asya Kamsky</reporter>
                        <labels>
                            <label>FLE</label>
                            <label>qopt-team</label>
                    </labels>
                <created>Tue, 4 Aug 2020 13:18:53 +0000</created>
                <updated>Wed, 31 Jan 2024 10:20:50 +0000</updated>
                                            <version>4.2.8</version>
                                                    <component>Field Level Encryption</component>
                                        <votes>23</votes>
                                    <watches>35</watches>
                                                                                                                <comments>
                            <comment id="6055193" author="JIRAUSER1277054" created="Wed, 31 Jan 2024 10:20:50 +0000"  >&lt;p&gt;Any updates so far or any alternative solutions?&#160; This is really a BIG issue.&#160;&lt;/p&gt;</comment>
                            <comment id="5883398" author="JIRAUSER1275824" created="Wed, 15 Nov 2023 21:46:08 +0000"  >&lt;p&gt;This has been an issue for such a long time. Is everyone that uses CSFLE simply tolerating managing two connections? Have they all moved to some other database provider or encryption mechanism? Or is everyone just encrypting fields explicitly?&lt;/p&gt;

&lt;p&gt;To me this seems like a bug, not an enhancement. Why should querying non-encrypted collections result in an error when CSFLE is enabled? No encryption mechanisms should be engaged after the query is parsed, since no encrypted fields would be found.&#160;&lt;/p&gt;</comment>
                            <comment id="4924292" author="JIRAUSER1271617" created="Tue, 25 Oct 2022 03:18:07 +0000"  >&lt;p&gt;Hello, any update on this?&lt;/p&gt;</comment>
                            <comment id="4400758" author="lamartaa70@gmail.com" created="Wed, 9 Mar 2022 14:47:48 +0000"  >&lt;p&gt;I figured out a work around. Essentially I created a collection that is dedicated to encrypting single string via CSFLE that I want to use as a matching comparison in a Lookup. I insert the string with an auto-encryption client into the collection and then query it with a non-encryption client to get it in its encrypted state (Bson Binary). I then can join a collection in a Lookup that has encrypted strings with another collection that does not, by using a non-encrypted client for both. Then if I want to query the Lookup by an encrypted string I just need to match the Bson Binary data I created in the first collection mentioned above to the encrypted field in the collection that is in the lookup. This way I don&apos;t need the framework to compare the decrypted string with a search string. I just compare an encrypted string field with another temporarily encrypted string.&#160;&#160;&lt;/p&gt;</comment>
                            <comment id="4396587" author="lamartaa70@gmail.com" created="Mon, 7 Mar 2022 22:23:46 +0000"  >&lt;p&gt;Are there any plans to release a fix for this in the near future, or is this backlogged indefinitely? &lt;/p&gt;

&lt;p&gt;I needed to convert over various fields in my database to be encrypted in order to be compliant with Facebook&apos;s Platform data policies. I went the&#160;CSFLE route only to find out in the end that I get this bug on my existing queries that are using lookups/joins because it appears that collections being accessed through an encryption client can not do this. So know I need to perform a refactor on my code to perform any encrypted collections separately from lookups. &lt;/p&gt;

&lt;p&gt;I am pretty disappointed to see that this was brought up two years ago and hasn&apos;t been fixed. Has anyone figured out a better alternative to MongoDB&apos;s&#160;CSFLE that plays nice with server side mongo queries (including joins)?&lt;/p&gt;</comment>
                            <comment id="3533521" author="JIRAUSER1258004" created="Wed, 16 Dec 2020 22:09:06 +0000"  >&lt;p&gt;I can understand that from the perspective of a successful NOSQL development team, &lt;b&gt;joins&lt;/b&gt; being a &quot;convenience&quot; seems a reasonable position. I can understand that smoothing out gaps between joins and encryption is a high-effort, low-reward option.&lt;/p&gt;

&lt;p&gt;The problem from the user perspective is, I think, that MongoDB wrote $lookup, made it available in early 2018, and is trying to establish itself in an environment thoroughly dominated by SQL systems that use joins- and developers that are used to joins. And use joins, as they presume they are able to do so from the MongoDB documentation. If $lookup is a &quot;convenience&quot; that&apos;s not part of the core MongoDB offering, then so- as 3.2 releases- are&#160;&lt;em&gt;basic math operations on aggregations&lt;/em&gt;. My company just ran head-first into this problem while trying to &lt;b&gt;protect PII&lt;/b&gt;&#160;on tables we reference in aggregation.&lt;/p&gt;

&lt;p&gt;We now get to decide whether to rip $lookup out of all of our code and refactor, or to substantially increase the number of connections to a production environment (which, due to MongoDB&apos;s architectural decisions, are more limited than other DBs), or to hope this bug comes out of backlog and into active development soon. Like Tom says, handling connection logic based on field encryption is a huge pain, and like Tal says there&apos;s no &lt;b&gt;good&lt;/b&gt; solution with this bug in place.&#160;I&apos;m going to recommend we refactor, since developer time is more under our control and less firmly limited than the other options, but I hope this description illuminates to the devs why MongoDB users may have a perspective that doesn&apos;t include joins (and joins working with encryption) as a &quot;convenience.&quot;&lt;/p&gt;</comment>
                            <comment id="3503479" author="JIRAUSER1254795" created="Tue, 24 Nov 2020 13:47:34 +0000"  >&lt;p&gt;Hello, any update on this?&lt;/p&gt;

&lt;p&gt;If this issue isn&apos;t going to be resolved in the near future, we are going to have to stop using CSFLE altogether.&#160;&lt;br/&gt;
The workaround of using two separate db connections (one with CSFLE and another without) is causing us trouble, so we have to stop using it, and get back to normal workflow.&#160;&lt;/p&gt;

&lt;p&gt;I have to say I&apos;m surprised this issue isn&apos;t given a much higher priority.. it is a very serious bug IMHO.&#160;&lt;/p&gt;</comment>
                            <comment id="3453376" author="JIRAUSER1257280" created="Tue, 20 Oct 2020 04:52:43 +0000"  >&lt;p&gt;Hi @Asya,&lt;/p&gt;

&lt;p&gt;Yes, I understand it is complex if both collections require encryption. &lt;br/&gt;
We are looking for the solution when both collections do not require encryption. This is needed to have no impact on the existing applications/implementation if we introduce CSFLE on some collections and use the same connection pool (with encryption configuration) to work on both set of encrypted and non-encrypted collections in a database.&lt;br/&gt;
Hope this gives some insight on why this feature is important.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Anu&lt;/p&gt;
</comment>
                            <comment id="3450020" author="gabayto@gmail.com" created="Fri, 16 Oct 2020 19:43:56 +0000"  >&lt;p&gt;Hey @asya,&lt;br/&gt;
The issue is about the case of both* collections are &lt;b&gt;not&lt;/b&gt; encrypted.&lt;/p&gt;
</comment>
                            <comment id="3450007" author="asya" created="Fri, 16 Oct 2020 19:36:30 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=anu.madan%40idemia.com&quot; class=&quot;user-hover&quot; rel=&quot;anu.madan@idemia.com&quot;&gt;anu.madan@idemia.com&lt;/a&gt; this is not a simple problem to address as a second collection requires knowing a different schema for encryption.&lt;/p&gt;

&lt;p&gt;$lookup is not really part of &quot;basic&quot; functionality in normal MongoDB querying - it&apos;s more a convenience added to allow you to avoid a second query in certain cases.   I would say that if this is a blocker you should consider a different approach to querying the two collections (for instance performing two queries rather than doing a single aggregation with $lookup).&lt;/p&gt;</comment>
                            <comment id="3446048" author="JIRAUSER1257280" created="Thu, 15 Oct 2020 13:42:23 +0000"  >&lt;p&gt;Hi, &lt;/p&gt;

&lt;p&gt;Facing the same issue.&lt;br/&gt;
This one is a blocker for our applications as well. Identifying which connection to use just when the query requires aggregation and then switch to a non-encrypted client is a big performance impact. &apos;lookup&apos; is a very basic functionality that needs to be supported and will impact almost all mongodb customers who plan to choose CSFLE.&lt;/p&gt;

&lt;p&gt;Is there an expected date for the resolution of this issue?&lt;/p&gt;

&lt;p&gt;Thanks.&lt;/p&gt;</comment>
                            <comment id="3322788" author="gabayto@gmail.com" created="Wed, 5 Aug 2020 14:57:02 +0000"  >&lt;p&gt;Hey, Also handling this issue,&lt;/p&gt;

&lt;p&gt;Just wanted to say i think its a big issue,&lt;/p&gt;

&lt;p&gt;in every real life scenario doing CSFLE require us to handle some complex logic to decide what connection to use,&#160;&lt;br/&gt;
or use non encrypted connection on error&lt;/p&gt;

&lt;p&gt;very un-natural way to use db connection&lt;/p&gt;</comment>
                            <comment id="3320344" author="asya" created="Tue, 4 Aug 2020 18:34:31 +0000"  >&lt;p&gt;This is a current limitation where using encrypted client assumes that it&apos;s necessary and doesn&apos;t allow $lookup into another collection.&lt;/p&gt;

&lt;p&gt;Current workaround would be to use a non-encrypted client for such aggregations.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="2101078">SERVER-68371</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1515314">JAVA-3859</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25126"><![CDATA[Query Optimization]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5006R00001kmZlfQAE]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 5 Aug 2020 14:57:02 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 week ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-2472</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>sujanmh@noveltytechnology.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 week ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>gus@ryer.io</customfieldvalue>
            <customfieldvalue>lamartaa70@gmail.com</customfieldvalue>
            <customfieldvalue>anu.madan@idemia.com</customfieldvalue>
            <customfieldvalue>asya.kamsky@mongodb.com</customfieldvalue>
            <customfieldvalue>backlog-query-optimization</customfieldvalue>
            <customfieldvalue>ben-wh.luk@aia.com</customfieldvalue>
            <customfieldvalue>sujanmh@noveltytechnology.com</customfieldvalue>
            <customfieldvalue>tal@remotemdr.com</customfieldvalue>
            <customfieldvalue>gabayto@gmail.com</customfieldvalue>
            <customfieldvalue>tyler.elkink@athennian.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxxwi7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr26cv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxxirj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>