<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:13:37 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-26923] OOM Killer Terminates All 3 Nodes in a Shard Using WiredTiger</title>
                <link>https://jira.mongodb.org/browse/SERVER-26923</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hello,&lt;/p&gt;

&lt;p&gt;We&apos;ve recently upgraded our MongoDB deployment to 3.2.10. During this upgrade we intended to migrate the storage engine to the new wiredTiger but ran into stability issues. Seemingly randomly throughout the day all data bearing nodes would crash due to oom-killer termination.&lt;/p&gt;

&lt;p&gt;There are many memory leak issues with wiredTiger in JIRA, most of them fixed. The one we hoped would be beneficial was fixed in 3.2.10 (&lt;a href=&quot;https://jira.mongodb.org/browse/WT-2796&quot; title=&quot;Memory leak in reconciliation uncovered by stress testing &quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-2796&quot;&gt;&lt;del&gt;WT-2796&lt;/del&gt;&lt;/a&gt;), but alas, we still ran into the same problem while running automated tests (not particularly stressful ones) on our cluster.&lt;/p&gt;

&lt;p&gt;We have a multiple environment deployment and the problem presented itself in all lower environments, causing us to reverse the decision to migrate to wiredTiger until we find a way to stabilize it.&lt;/p&gt;

&lt;p&gt;We are using the 1.11 C# driver in our application. The higher environments are both 5 sharded clusters with 3 data bearing nodes in each shard&apos;s replica set. The config servers are not configured as a replica set and will not be migrated to wiredTiger at this point. Our application is cloud hosted in AWS and the number of servers running mongos.exe locally scales up and down automatically according to the load on service queues.&lt;/p&gt;

&lt;p&gt;Please let me know if I can provide any further information. This bug is marked as critical because it involves a severe memory leak, per the table of priorities.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Shy&lt;/p&gt;</description>
                <environment></environment>
        <key id="329920">SERVER-26923</key>
            <summary>OOM Killer Terminates All 3 Nodes in a Shard Using WiredTiger</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="shy@tegrity.com">Shy Tamir</reporter>
                        <labels>
                    </labels>
                <created>Sun, 6 Nov 2016 12:56:40 +0000</created>
                <updated>Sun, 8 Jan 2017 07:54:02 +0000</updated>
                            <resolved>Fri, 6 Jan 2017 22:00:27 +0000</resolved>
                                    <version>3.0.11</version>
                    <version>3.2.10</version>
                                                    <component>Text Search</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="1472058" author="shy@tegrity.com" created="Sun, 8 Jan 2017 07:54:02 +0000"  >&lt;p&gt;Understood, Tom.&lt;/p&gt;

&lt;p&gt;Thanks for the time you&apos;ve put into this. I&apos;ll make sure we have the ticket watched and voted and explain to our developers we must explore other alternatives.&lt;/p&gt;

&lt;p&gt;Thanks again,&lt;br/&gt;
Shy&lt;/p&gt;</comment>
                            <comment id="1458007" author="thomas.schubert" created="Wed, 14 Dec 2016 16:24:53 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=shy%40tegrity.com&quot; class=&quot;user-hover&quot; rel=&quot;shy@tegrity.com&quot;&gt;shy@tegrity.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;m sorry for the difficulties that this issue is causing you. Currently, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26534&quot; title=&quot;Text search uses excessive memory&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26534&quot;&gt;SERVER-26534&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26833&quot; title=&quot;Permit non-blocking $text queries when user doesn&amp;#39;t request score projection&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26833&quot;&gt;&lt;del&gt;SERVER-26833&lt;/del&gt;&lt;/a&gt; both have the fix version &quot;planned but not scheduled.&quot; This fix version indicates that we would like to complete this work, but it is not currently targeted for a specific release.&lt;/p&gt;

&lt;p&gt;We have not heard many reports of users encountering this issue, and, unfortunately, the changes to the underlying logic is significant. If you have not already done so, I would recommend watching and voting on both these tickets, as we take this into account during the planning process.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1457396" author="shy@tegrity.com" created="Wed, 14 Dec 2016 14:53:59 +0000"  >&lt;p&gt;Hi Thomas,&lt;/p&gt;

&lt;p&gt;I wanted to touch base after having had the opportunity to discuss this with our dev leads. We were surprised to learn that such a critical issue is affecting all MongoDB versions and hasn&apos;t been fixed yet. Making changes to our application behavior will require more than a little effort on our part, and does not guarantee a stable database even if we do change our application logic.&lt;/p&gt;

&lt;p&gt;We are currently exploring ways to mitigate this phenomenon per your suggestions, but the only effective solutions so far seem to be in the realm of long-term solutions, postponing our shard reduction efforts for longer than we&apos;d like. Is there any news on what MongoDB version might include a fix for this?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Shy&lt;/p&gt;</comment>
                            <comment id="1450042" author="shy@tegrity.com" created="Tue, 6 Dec 2016 10:23:32 +0000"  >&lt;p&gt;Hi Thomas,&lt;/p&gt;

&lt;p&gt;We&apos;ve experienced OOM-killer on all nodes, but the secondaries go down first, and then the primary gets demoted before going down itself, so the symptoms described in the server issue conform with what we&apos;re seeing in the logs.&lt;/p&gt;

&lt;p&gt;The average object size in the recordings collection is reported by the stats method as 46,555 in the lower affected environment out of 186k objects. And ~26,000 across the higher environments shards out of 3.3m objects.&lt;/p&gt;

&lt;p&gt;Do those numbers sit well with the suspected diagnosis? Barring a change to our application&apos;s logic (which can take time and requires a lot of coordinated effort) would it be reasonable to expect a solution for this server issue in the near future?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Shy&lt;/p&gt;</comment>
                            <comment id="1445887" author="thomas.schubert" created="Wed, 30 Nov 2016 16:30:40 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=shy%40tegrity.com&quot; class=&quot;user-hover&quot; rel=&quot;shy@tegrity.com&quot;&gt;shy@tegrity.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for the detailed information. It looks like you are encountering &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26534&quot; title=&quot;Text search uses excessive memory&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26534&quot;&gt;SERVER-26534&lt;/a&gt;. The increase in memory consumption occurs while the following queries complete.&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-11-21T22:06:33.036-0500 I COMMAND  [conn203] command test.recordings command: find { find: &quot;recordings&quot;, filter: { UniversityId: &quot;b1527c26-46f5-4d33-8e6f-7a5d6cc668b1&quot;, CourseId: { $in: [ &quot;4976b169-e883-4d82-b591-f9dc0936de75&quot; ] }, $text: { $search: &quot;Neurology Clinical Localization Joel Shenker, M.D., Ph.D. University o...&quot; }, RecordingType: { $ne: &quot;Test&quot; }, IsDeleted: false }, projection: { score: { $meta: &quot;textScore&quot; }, $sortKey: { $meta: &quot;sortKey&quot; } }, sort: { score: { $meta: &quot;textScore&quot; } }, shardVersion: [ Timestamp 1000|100, ObjectId(&apos;5553043951c058815a10b78d&apos;) ] } planSummary: IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 } keysExamined:62682 docsExamined:42260 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:713 nreturned:2 reslen:109413 locks:{ Global: { acquireCount: { r: 1430 } }, Database: { acquireCount: { r: 715 } }, Collection: { acquireCount: { r: 715 } } } protocol:op_command 4249ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-11-21T22:06:33.044-0500 I COMMAND  [conn181] command test.recordings command: find { find: &quot;recordings&quot;, filter: { UniversityId: &quot;b1527c26-46f5-4d33-8e6f-7a5d6cc668b1&quot;, CourseId: { $in: [ &quot;770c4a14-12f1-427a-b8fb-eda41148baf5&quot; ] }, $text: { $search: &quot;Neurology Clinical Localization Joel Shenker, M.D., Ph.D. University o...&quot; }, RecordingType: { $ne: &quot;Test&quot; }, IsDeleted: false }, projection: { score: { $meta: &quot;textScore&quot; }, $sortKey: { $meta: &quot;sortKey&quot; } }, sort: { score: { $meta: &quot;textScore&quot; } }, shardVersion: [ Timestamp 1000|100, ObjectId(&apos;5553043951c058815a10b78d&apos;) ] } planSummary: IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 } keysExamined:62682 docsExamined:42260 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:713 nreturned:0 reslen:218 locks:{ Global: { acquireCount: { r: 1430 } }, Database: { acquireCount: { r: 715 } }, Collection: { acquireCount: { r: 715 } } } protocol:op_command 4263ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-11-21T22:06:33.080-0500 I COMMAND  [conn204] command test.recordings command: find { find: &quot;recordings&quot;, filter: { UniversityId: &quot;b1527c26-46f5-4d33-8e6f-7a5d6cc668b1&quot;, CourseId: { $in: [ &quot;a0d9495d-138b-44dd-b221-ab847e23dcf8&quot; ] }, $text: { $search: &quot;Neurology Clinical Localization Joel Shenker, M.D., Ph.D. University o...&quot; }, RecordingType: { $ne: &quot;Test&quot; }, IsDeleted: false }, projection: { score: { $meta: &quot;textScore&quot; }, $sortKey: { $meta: &quot;sortKey&quot; } }, sort: { score: { $meta: &quot;textScore&quot; } }, shardVersion: [ Timestamp 1000|100, ObjectId(&apos;5553043951c058815a10b78d&apos;) ] } planSummary: IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 } keysExamined:62682 docsExamined:42260 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:712 nreturned:0 reslen:218 locks:{ Global: { acquireCount: { r: 1428 } }, Database: { acquireCount: { r: 714 } }, Collection: { acquireCount: { r: 714 } } } protocol:op_command 4282ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-11-21T22:06:33.088-0500 I COMMAND  [conn208] command test.recordings command: find { find: &quot;recordings&quot;, filter: { UniversityId: &quot;b1527c26-46f5-4d33-8e6f-7a5d6cc668b1&quot;, CourseId: { $in: [ &quot;a6ff7224-37cf-499b-9ece-f28ff24bcebf&quot; ] }, $text: { $search: &quot;Neurology Clinical Localization Joel Shenker, M.D., Ph.D. University o...&quot; }, RecordingType: { $ne: &quot;Test&quot; }, IsDeleted: false }, projection: { score: { $meta: &quot;textScore&quot; }, $sortKey: { $meta: &quot;sortKey&quot; } }, sort: { score: { $meta: &quot;textScore&quot; } }, shardVersion: [ Timestamp 1000|100, ObjectId(&apos;5553043951c058815a10b78d&apos;) ] } planSummary: IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 } keysExamined:62682 docsExamined:42260 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:710 nreturned:0 reslen:218 locks:{ Global: { acquireCount: { r: 1424 } }, Database: { acquireCount: { r: 712 } }, Collection: { acquireCount: { r: 712 } } } protocol:op_command 4289ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-11-21T22:06:33.107-0500 I COMMAND  [conn205] command test.recordings command: find { find: &quot;recordings&quot;, filter: { UniversityId: &quot;b1527c26-46f5-4d33-8e6f-7a5d6cc668b1&quot;, CourseId: { $in: [ &quot;81dc93d0-5602-4ba6-b01e-5a76b3b52fdf&quot; ] }, $text: { $search: &quot;Neurology Clinical Localization Joel Shenker, M.D., Ph.D. University o...&quot; }, RecordingType: { $ne: &quot;Test&quot; }, IsDeleted: false }, projection: { score: { $meta: &quot;textScore&quot; }, $sortKey: { $meta: &quot;sortKey&quot; } }, sort: { score: { $meta: &quot;textScore&quot; } }, shardVersion: [ Timestamp 1000|100, ObjectId(&apos;5553043951c058815a10b78d&apos;) ] } planSummary: IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 } keysExamined:62682 docsExamined:42260 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:712 nreturned:0 reslen:218 locks:{ Global: { acquireCount: { r: 1428 } }, Database: { acquireCount: { r: 714 } }, Collection: { acquireCount: { r: 714 } } } protocol:op_command 4326ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-11-21T22:06:33.109-0500 I COMMAND  [conn180] command test.recordings command: find { find: &quot;recordings&quot;, filter: { UniversityId: &quot;b1527c26-46f5-4d33-8e6f-7a5d6cc668b1&quot;, CourseId: { $in: [ &quot;3be4fdb0-879e-4b15-adbf-48b4434a4abe&quot; ] }, $text: { $search: &quot;Neurology Clinical Localization Joel Shenker, M.D., Ph.D. University o...&quot; }, RecordingType: { $ne: &quot;Test&quot; }, IsDeleted: false }, projection: { score: { $meta: &quot;textScore&quot; }, $sortKey: { $meta: &quot;sortKey&quot; } }, sort: { score: { $meta: &quot;textScore&quot; } }, shardVersion: [ Timestamp 1000|100, ObjectId(&apos;5553043951c058815a10b78d&apos;) ] } planSummary: IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 } keysExamined:62682 docsExamined:42260 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:712 nreturned:0 reslen:218 locks:{ Global: { acquireCount: { r: 1428 } }, Database: { acquireCount: { r: 714 } }, Collection: { acquireCount: { r: 714 } } } protocol:op_command 4328ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-11-21T22:06:33.113-0500 I COMMAND  [conn182] command test.recordings command: find { find: &quot;recordings&quot;, filter: { UniversityId: &quot;b1527c26-46f5-4d33-8e6f-7a5d6cc668b1&quot;, CourseId: { $in: [ &quot;fa98c340-e4ce-4462-9273-6c7f22e7b69f&quot; ] }, $text: { $search: &quot;Neurology Clinical Localization Joel Shenker, M.D., Ph.D. University o...&quot; }, RecordingType: { $ne: &quot;Test&quot; }, IsDeleted: false }, projection: { score: { $meta: &quot;textScore&quot; }, $sortKey: { $meta: &quot;sortKey&quot; } }, sort: { score: { $meta: &quot;textScore&quot; } }, shardVersion: [ Timestamp 1000|100, ObjectId(&apos;5553043951c058815a10b78d&apos;) ] } planSummary: IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 } keysExamined:62682 docsExamined:42260 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:713 nreturned:16 reslen:667099 locks:{ Global: { acquireCount: { r: 1430 } }, Database: { acquireCount: { r: 715 } }, Collection: { acquireCount: { r: 715 } } } protocol:op_command 4331ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-11-21T22:06:33.120-0500 I COMMAND  [conn202] command test.recordings command: find { find: &quot;recordings&quot;, filter: { UniversityId: &quot;b1527c26-46f5-4d33-8e6f-7a5d6cc668b1&quot;, CourseId: { $in: [ &quot;c99a2db8-9c81-4fcc-90ee-939b3e15b9f1&quot; ] }, $text: { $search: &quot;Neurology Clinical Localization Joel Shenker, M.D., Ph.D. University o...&quot; }, RecordingType: { $ne: &quot;Test&quot; }, IsDeleted: false }, projection: { score: { $meta: &quot;textScore&quot; }, $sortKey: { $meta: &quot;sortKey&quot; } }, sort: { score: { $meta: &quot;textScore&quot; } }, shardVersion: [ Timestamp 1000|100, ObjectId(&apos;5553043951c058815a10b78d&apos;) ] } planSummary: IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 } keysExamined:62682 docsExamined:42260 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:712 nreturned:0 reslen:218 locks:{ Global: { acquireCount: { r: 1428 } }, Database: { acquireCount: { r: 714 } }, Collection: { acquireCount: { r: 714 } } } protocol:op_command 4328ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-11-21T22:06:33.121-0500 I COMMAND  [conn183] command test.recordings command: find { find: &quot;recordings&quot;, filter: { UniversityId: &quot;b1527c26-46f5-4d33-8e6f-7a5d6cc668b1&quot;, CourseId: { $in: [ &quot;2914fbd2-db56-4905-8bee-f4544b0a81df&quot; ] }, $text: { $search: &quot;Neurology Clinical Localization Joel Shenker, M.D., Ph.D. University o...&quot; }, RecordingType: { $ne: &quot;Test&quot; }, IsDeleted: false }, projection: { score: { $meta: &quot;textScore&quot; }, $sortKey: { $meta: &quot;sortKey&quot; } }, sort: { score: { $meta: &quot;textScore&quot; } }, shardVersion: [ Timestamp 1000|100, ObjectId(&apos;5553043951c058815a10b78d&apos;) ] } planSummary: IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 } keysExamined:62682 docsExamined:42260 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:710 nreturned:0 reslen:218 locks:{ Global: { acquireCount: { r: 1424 } }, Database: { acquireCount: { r: 712 } }, Collection: { acquireCount: { r: 712 } } } protocol:op_command 4340ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-11-21T22:06:33.122-0500 I COMMAND  [conn184] command test.recordings command: find { find: &quot;recordings&quot;, filter: { UniversityId: &quot;b1527c26-46f5-4d33-8e6f-7a5d6cc668b1&quot;, CourseId: { $in: [ &quot;b5e6f12d-e018-402b-b049-278550df2988&quot; ] }, $text: { $search: &quot;Neurology Clinical Localization Joel Shenker, M.D., Ph.D. University o...&quot; }, IsDeleted: false }, projection: { score: { $meta: &quot;textScore&quot; }, $sortKey: { $meta: &quot;sortKey&quot; } }, sort: { score: { $meta: &quot;textScore&quot; } }, shardVersion: [ Timestamp 1000|100, ObjectId(&apos;5553043951c058815a10b78d&apos;) ] } planSummary: IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 } keysExamined:62682 docsExamined:42260 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:711 nreturned:0 reslen:218 locks:{ Global: { acquireCount: { r: 1426 } }, Database: { acquireCount: { r: 713 } }, Collection: { acquireCount: { r: 713 } } } protocol:op_command 4334ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-11-21T22:06:33.122-0500 I COMMAND  [conn206] command test.recordings command: find { find: &quot;recordings&quot;, filter: { UniversityId: &quot;b1527c26-46f5-4d33-8e6f-7a5d6cc668b1&quot;, CourseId: { $in: [ &quot;6baec395-5768-40c9-a9ad-6ac3885fb104&quot; ] }, $text: { $search: &quot;Neurology Clinical Localization Joel Shenker, M.D., Ph.D. University o...&quot; }, RecordingType: { $ne: &quot;Test&quot; }, IsDeleted: false }, projection: { score: { $meta: &quot;textScore&quot; }, $sortKey: { $meta: &quot;sortKey&quot; } }, sort: { score: { $meta: &quot;textScore&quot; } }, shardVersion: [ Timestamp 1000|100, ObjectId(&apos;5553043951c058815a10b78d&apos;) ] } planSummary: IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 } keysExamined:62682 docsExamined:42260 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:713 nreturned:0 reslen:218 locks:{ Global: { acquireCount: { r: 1430 } }, Database: { acquireCount: { r: 715 } }, Collection: { acquireCount: { r: 715 } } } protocol:op_command 4339ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-11-21T22:06:33.125-0500 I COMMAND  [conn186] command test.recordings command: find { find: &quot;recordings&quot;, filter: { UniversityId: &quot;b1527c26-46f5-4d33-8e6f-7a5d6cc668b1&quot;, CourseId: { $in: [ &quot;4ee3c2ba-f12f-4789-a2e8-0a27effca427&quot; ] }, $text: { $search: &quot;Neurology Clinical Localization Joel Shenker, M.D., Ph.D. University o...&quot; }, RecordingType: { $ne: &quot;Test&quot; }, IsDeleted: false }, projection: { score: { $meta: &quot;textScore&quot; }, $sortKey: { $meta: &quot;sortKey&quot; } }, sort: { score: { $meta: &quot;textScore&quot; } }, shardVersion: [ Timestamp 1000|100, ObjectId(&apos;5553043951c058815a10b78d&apos;) ] } planSummary: IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 } keysExamined:62682 docsExamined:42260 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:712 nreturned:0 reslen:218 locks:{ Global: { acquireCount: { r: 1428 } }, Database: { acquireCount: { r: 714 } }, Collection: { acquireCount: { r: 714 } } } protocol:op_command 4338ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-11-21T22:06:33.129-0500 I COMMAND  [conn198] command test.recordings command: find { find: &quot;recordings&quot;, filter: { UniversityId: &quot;b1527c26-46f5-4d33-8e6f-7a5d6cc668b1&quot;, CourseId: { $in: [ &quot;40b1a9d2-aa25-4bd6-92fb-c505b451282d&quot; ] }, $text: { $search: &quot;Neurology Clinical Localization Joel Shenker, M.D., Ph.D. University o...&quot; }, IsDeleted: false }, projection: { score: { $meta: &quot;textScore&quot; }, $sortKey: { $meta: &quot;sortKey&quot; } }, sort: { score: { $meta: &quot;textScore&quot; } }, shardVersion: [ Timestamp 1000|100, ObjectId(&apos;5553043951c058815a10b78d&apos;) ] } planSummary: IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 } keysExamined:62682 docsExamined:42260 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:710 nreturned:6 reslen:203996 locks:{ Global: { acquireCount: { r: 1424 } }, Database: { acquireCount: { r: 712 } }, Collection: { acquireCount: { r: 712 } } } protocol:op_command 4331ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-11-21T22:06:33.131-0500 I COMMAND  [conn185] command test.recordings command: find { find: &quot;recordings&quot;, filter: { UniversityId: &quot;b1527c26-46f5-4d33-8e6f-7a5d6cc668b1&quot;, CourseId: { $in: [ &quot;d0f2c2f6-c45f-4385-add9-a12253eb6b39&quot; ] }, $text: { $search: &quot;Neurology Clinical Localization Joel Shenker, M.D., Ph.D. University o...&quot; }, RecordingType: { $ne: &quot;Test&quot; }, IsDeleted: false }, projection: { score: { $meta: &quot;textScore&quot; }, $sortKey: { $meta: &quot;sortKey&quot; } }, sort: { score: { $meta: &quot;textScore&quot; } }, shardVersion: [ Timestamp 1000|100, ObjectId(&apos;5553043951c058815a10b78d&apos;) ] } planSummary: IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 }, IXSCAN { UniversityId: 1.0, _fts: &quot;text&quot;, _ftsx: 1 } keysExamined:62682 docsExamined:42260 hasSortStage:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:718 nreturned:0 reslen:218 locks:{ Global: { acquireCount: { r: 1440 } }, Database: { acquireCount: { r: 720 } }, Collection: { acquireCount: { r: 720 } } } protocol:op_command 4332ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Therefore, I would recommend modifying your application to reduce the number of concurrent text searches, or modifying these queries to reduce the number of documents examined.&lt;/p&gt;

&lt;p&gt;It looks like these queries are targeted at the secondaries, do you send similar queries to the primary, or have you only experienced OOM killer on secondaries?&lt;/p&gt;

&lt;p&gt;To confirm that &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26534&quot; title=&quot;Text search uses excessive memory&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26534&quot;&gt;SERVER-26534&lt;/a&gt; completely explains the memory consumption, would you please provide the average data size of the documents being examined by these queries?  &lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1442802" author="shy@tegrity.com" created="Sun, 27 Nov 2016 14:43:14 +0000"  >&lt;p&gt;Hi Thomas, and a pleasure to meet you,&lt;/p&gt;

&lt;p&gt;I can share detailed information about the tests and configurations, what specifically is of interest. I&apos;ll gloss over the major details so you can help me out by pointing out what&apos;s left out or needs expanding:&lt;/p&gt;

&lt;p&gt;Infrastructure&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Reproduced in two separate environments of a similar nature.&lt;/li&gt;
	&lt;li&gt;Both environments are sharded clusters, one has only one shard, the other has 5 shards.&lt;/li&gt;
	&lt;li&gt;The servers are all AWS EC2 instances (Memory optimized r3.2xlarge type), running dedicated mongod processes, using assigned ENIs.&lt;/li&gt;
	&lt;li&gt;All communication is secured using self-signed certificates, between the servers with a key file and using SSL for clients.&lt;/li&gt;
	&lt;li&gt;The config servers are all identical but not a replica set.&lt;/li&gt;
	&lt;li&gt;The shard replica sets are all data bearing nodes, and only the 3 data bearing nodes (one in each us-east-1 AZ) exist per replica set.&lt;/li&gt;
	&lt;li&gt;When reproducing the issue only the storage engine for and only for non config data bearing nodes (3 in QA, 15 in PQA).&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Application&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Our application is a video capture and quick-edit software as a service used by higher education institutions to allow their faculty and students to record instructional materials such as lectures - aptly dubbed &quot;Tegrity Lecture Capture&quot;.&lt;/li&gt;
	&lt;li&gt;The nature of the application is such that DB operations are very read and write intensive, especially during peak times but sometimes during routine operations.&lt;/li&gt;
	&lt;li&gt;During manual testing operations for both environments the issue is not reproducible.&lt;/li&gt;
	&lt;li&gt;During a suite of load tests run on PQA the issue is not reproducible.&lt;/li&gt;
	&lt;li&gt;A suite of automated tests performed on both environments, simulating most of the application features&apos; functionalities, reproduces the issue consistently every time we run it, sometimes several times.&lt;/li&gt;
	&lt;li&gt;Nothing but normal activity is recorded in application logs for the times of the crashes.&lt;/li&gt;
	&lt;li&gt;Secondary read preference is set on certain queries (at the query level) that have to do with collecting aggregated data.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;If it will help I can loop in our QA Manager who knows the nature of the tests better than anyone, or one of developer leads for more insight into how the application makes database queries.&lt;/p&gt;

&lt;p&gt;Thanks so much,&lt;br/&gt;
Shy&lt;/p&gt;</comment>
                            <comment id="1442418" author="thomas.schubert" created="Fri, 25 Nov 2016 16:09:57 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=shy%40tegrity.com&quot; class=&quot;user-hover&quot; rel=&quot;shy@tegrity.com&quot;&gt;shy@tegrity.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for uploading the files. I&apos;ve spent some time examining them, but haven&apos;t identified the root cause yet. It appears that the the memory usage is consistent before spiking, causing the OOM. My understanding is that you are able to consistently reproduce this issue in your testing enviroment. Are you able to share the tests and configuration that reproduces this issue?&lt;/p&gt;

&lt;p&gt;Thanks again,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1442333" author="shy@tegrity.com" created="Fri, 25 Nov 2016 11:47:51 +0000"  >&lt;p&gt;Hello,&lt;/p&gt;

&lt;p&gt;Has anyone had a chance to look into this yet.?Just curious...&lt;/p&gt;

&lt;p&gt;Thanks, &lt;br/&gt;
Shy&lt;/p&gt;</comment>
                            <comment id="1440011" author="shy@tegrity.com" created="Tue, 22 Nov 2016 13:51:00 +0000"  >&lt;p&gt;Hello,&lt;/p&gt;

&lt;p&gt;Please find the logging information requested attached to this ticket.&lt;/p&gt;

&lt;p&gt;Please note that we&apos;ve reverted the change so reproducing the issue will take another round of changing it back again and running our tests.&lt;/p&gt;

&lt;p&gt;We&apos;re eager to hear the findings of your analysis.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Shy&lt;/p&gt;</comment>
                            <comment id="1436379" author="shy@tegrity.com" created="Thu, 17 Nov 2016 11:52:52 +0000"  >&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;Please note that we have failed to reproduce this issue in an heretofore untested environment, and so we don&apos;t have the diagnostic data available yet.&lt;/p&gt;

&lt;p&gt;However, we do plan on running the same suite of tests on the environments that were giving us problems during next week, at which time I will upload them here.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Shy&lt;/p&gt;</comment>
                            <comment id="1432788" author="shy@tegrity.com" created="Sun, 13 Nov 2016 15:45:49 +0000"  >&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;Apologies for the delays in getting you those logs.&lt;/p&gt;

&lt;p&gt;We have since reverted the storage engine change and will need to recreate the issue, which will take some time since we need to coordinate it with multiple departments.&lt;/p&gt;

&lt;p&gt;I will post the logs here by the end of the week with any luck and we can progress from there.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Shy&lt;/p&gt;</comment>
                            <comment id="1432780" author="ramon.fernandez" created="Sun, 13 Nov 2016 13:42:52 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=shy%40tegrity.com&quot; class=&quot;user-hover&quot; rel=&quot;shy@tegrity.com&quot;&gt;shy@tegrity.com&lt;/a&gt;, we haven&apos;t heard back from you. If this issue is still critical to you, can you please upload the &lt;tt&gt;diagnostic.data&lt;/tt&gt; directory and the mongod log file for all impacted nodes as Dan requested above?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1427476" author="dan@10gen.com" created="Mon, 7 Nov 2016 00:34:04 +0000"  >&lt;p&gt;Hi Shy,&lt;br/&gt;
Please tar up and the files located in the &lt;tt&gt;diagnostic.data&lt;/tt&gt; directory and the mongod run logs for all impacted nodes.&lt;/p&gt;

&lt;p&gt;Dan&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="322216">SERVER-26534</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="144616" name="Wired Tiger Crash Logs.zip" size="16018722" author="shy@tegrity.com" created="Tue, 22 Nov 2016 13:50:24 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 7 Nov 2016 00:34:04 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 5 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>shy@tegrity.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 5 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dan@mongodb.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>shy@tegrity.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjrrz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsrlsf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;1. Configure shard to use wiredTiger.&lt;br/&gt;
2. Wait an indefinite period of time while automated tests are running (6-12 hours).&lt;br/&gt;
3. Identify the oom-killer and resulting crash in the server&apos;s logs.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrwrfb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>