<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:09:27 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-5628] Insertions in one collection eventually lead to corruption of unrelated collections</title>
                <link>https://jira.mongodb.org/browse/SERVER-5628</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;In an configuration with multiple databases, if records are inserted into collections of one database, records from unrelated collections may be &quot;inserted&quot;* into yet other unrelated collections. There may also be rearrangements in collections in the original database, but I can not determine this as readily as all records in the same database have an id of the same name so I have been using {id_name: {$exists:false}} to find the errors.&lt;/p&gt;

&lt;p&gt;The logs frequently show errors indicating a bad BSON object type and extremely long strings.  Further, validate does not always find the errors and returns &apos;ok&apos; when errors are still present.  &lt;/p&gt;

&lt;p&gt;Other than inserting a couple hundred thousand records into a couple collections at the same time with the Java driver, I have not found a specific way to reproduce this error, but I can provide a valgrind log&lt;/p&gt;

&lt;p&gt;*The bad records do not seem to be inserted with any client insert calls and the journal file does not repair the collections&lt;/p&gt;</description>
                <environment>Centos 6.2 x86_64, 3G RAM</environment>
        <key id="36460">SERVER-5628</key>
            <summary>Insertions in one collection eventually lead to corruption of unrelated collections</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="aaron">Aaron Staple</assignee>
                                    <reporter username="steven.m.cooke@gmail.com">Steven Cooke</reporter>
                        <labels>
                    </labels>
                <created>Tue, 17 Apr 2012 08:18:10 +0000</created>
                <updated>Fri, 8 Mar 2013 15:55:41 +0000</updated>
                            <resolved>Mon, 17 Sep 2012 22:09:45 +0000</resolved>
                                    <version>2.0.4</version>
                                                    <component>Internal Code</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="165263" author="steven.m.cooke@gmail.com" created="Mon, 17 Sep 2012 22:07:29 +0000"  >&lt;p&gt;Aaron,  sorry it&apos;s taken me a while to get back to this.  I have found two&lt;br/&gt;
workarounds in Groovy as described previously and have not seen any issues.&lt;br/&gt;
 The first workaround of checking for every match instead of a single match&lt;br/&gt;
for the key might be slower than needs to be but seems to work as does&lt;br/&gt;
explicitly defining all script variables which is the proper thing.&lt;br/&gt;
 Variables not defined are in a Binding that can be altered outside the&lt;br/&gt;
scope they were originally used, so in a multithreaded script, all bets are&lt;br/&gt;
off.&lt;/p&gt;

&lt;p&gt;My best guess is that my issues were due to the key variable leaking out of&lt;br/&gt;
the scope I thought it was in, but I have not had a chance to test it.&lt;br/&gt;
 Also, I think this is only an issue with Groovy code run in a script&lt;br/&gt;
environment so the issue should vanish if run as compiled classes.&lt;/p&gt;

&lt;p&gt;Please close the issue and thank you for your help and patience.&lt;/p&gt;

&lt;p&gt;Steve&lt;/p&gt;

</comment>
                            <comment id="165233" author="aaron" created="Mon, 17 Sep 2012 21:04:14 +0000"  >&lt;p&gt;Hi Steven,&lt;/p&gt;

&lt;p&gt;Let me know if there&apos;s more to discuss here.  Otherwise I&apos;ll plan to close out the ticket.&lt;/p&gt;</comment>
                            <comment id="155904" author="aaron" created="Wed, 22 Aug 2012 21:41:12 +0000"  >&lt;p&gt;Ok, have you tried to fix the test and is there still an issue?&lt;/p&gt;</comment>
                            <comment id="152922" author="steven.m.cooke@gmail.com" created="Mon, 13 Aug 2012 20:06:43 +0000"  >&lt;p&gt;Aaron,&lt;/p&gt;

&lt;p&gt;  I have not confirmed this, but I think the method variables in the&lt;br/&gt;
closure of the for loop (line 74) are not thread-safe unless declared with&lt;br/&gt;
&quot;def&quot; or the type. Groovy does not have &quot;final&quot; like would be needed for&lt;br/&gt;
anonymous classes om Java, but it is conceivable that without the def&lt;br/&gt;
variables outside the closure do not have the semantics a final variable&lt;br/&gt;
would.  Changing line 72 to include def would be the change.&lt;/p&gt;

&lt;p&gt; This does not explain why there error is not seen when both boolean&lt;br/&gt;
parameters to update are true.  This just may be a timing issue.&lt;/p&gt;

&lt;p&gt;Steve&lt;/p&gt;



</comment>
                            <comment id="152869" author="aaron" created="Mon, 13 Aug 2012 18:14:23 +0000"  >&lt;p&gt;Hi Steven,&lt;/p&gt;

&lt;p&gt;Ok can you run that debugging and send the output?  It&apos;s pretty clear that mongod is getting the wrong &quot;i&quot; values.  (Both the sniffer and mongod logging are reporting incorrect i&apos;s.)  So I&apos;d suggest adding some debugging to your script to figure out how this is happening.&lt;/p&gt;</comment>
                            <comment id="147994" author="steven.m.cooke@gmail.com" created="Fri, 27 Jul 2012 06:27:02 +0000"  >&lt;p&gt;The value of i is not changed after it is set within a single iteration of&lt;br/&gt;
the loop. The error goes away if the delay is increased.  That seems to&lt;br/&gt;
indicate either an issue in the driver or the server or possibly in the&lt;br/&gt;
implementation of Groovy.  Perhaps adding something like:&lt;/p&gt;

&lt;p&gt;def temp = i&lt;br/&gt;
...&lt;br/&gt;
if (i != temp) &lt;/p&gt;
{ exit(1) }

&lt;p&gt;or follow the line:&lt;br/&gt;
   rec = &lt;span class=&quot;error&quot;&gt;&amp;#91;i:i, key: (String)&amp;quot;$chr&amp;quot;&amp;#93;&lt;/span&gt; as BasicDBObject&lt;br/&gt;
with:&lt;br/&gt;
   query = &lt;span class=&quot;error&quot;&gt;&amp;#91;i:i&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;and make the corresponding change to the update statement:&lt;/p&gt;

&lt;p&gt;   result = it.update(query, setrec , true)&lt;/p&gt;

&lt;p&gt;might be convincing.&lt;/p&gt;

&lt;p&gt;There is also a static trace method in DBApiLayer that looks like it can be&lt;br/&gt;
used to output the request sent to the driver but before the driver&lt;br/&gt;
actually sends the request.&lt;/p&gt;


</comment>
                            <comment id="147968" author="aaron" created="Fri, 27 Jul 2012 03:56:14 +0000"  >&lt;p&gt;Hi Steven,&lt;/p&gt;

&lt;p&gt;As I mentioned it seems like a client side issue.  Maybe you can add some debugging to your script to verify what the client is sending, and the values of your &apos;i&apos; variables?&lt;/p&gt;</comment>
                            <comment id="145510" author="steven.m.cooke@gmail.com" created="Fri, 20 Jul 2012 20:17:01 +0000"  >&lt;p&gt;Yes, that is the expected error.  If you increase the sleep time the error&lt;br/&gt;
goes away. Ideas?&lt;/p&gt;

</comment>
                            <comment id="145472" author="aaron" created="Fri, 20 Jul 2012 19:17:21 +0000"  >&lt;p&gt;Hi Steve,&lt;/p&gt;

&lt;p&gt;I ran the test with your modified upsert function.  I am seeing the same behavior, where the client sends an unexpected upsert request:&lt;/p&gt;

&lt;p&gt;Data:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&amp;gt; db.belmont.find( {key:&apos;a&apos; })&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;_id&quot; : ObjectId(&quot;5009acb73a3c2e71463af2a0&quot;), &quot;i&quot; : 113, &quot;key&quot; : &quot;a&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;_id&quot; : ObjectId(&quot;5009acb73a3c2e71463af2c0&quot;), &quot;i&quot; : 130, &quot;key&quot; : &quot;a&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;_id&quot; : ObjectId(&quot;5009acb73a3c2e71463af337&quot;), &quot;i&quot; : 190, &quot;key&quot; : &quot;a&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;_id&quot; : ObjectId(&quot;5009acb73a3c2e71463af3fe&quot;), &quot;i&quot; : 289, &quot;key&quot; : &quot;a&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;_id&quot; : ObjectId(&quot;5009acb73a3c2e71463af426&quot;), &quot;i&quot; : 306, &quot;key&quot; : &quot;a&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;_id&quot; : ObjectId(&quot;5009acb83a3c2e71463af4e0&quot;), &quot;i&quot; : 400, &quot;key&quot; : &quot;a&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;_id&quot; : ObjectId(&quot;5009acb83a3c2e71463af5eb&quot;), &quot;i&quot; : 514, &quot;key&quot; : &quot;a&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;_id&quot; : ObjectId(&quot;5009acb83a3c2e71463af60a&quot;), &quot;i&quot; : 530, &quot;key&quot; : &quot;a&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Mongosniff log:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;127.0.0.1:52419  --&amp;gt;&amp;gt; 127.0.0.1:27017 mass.belmont  83 bytes  id:1d2    466&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        update  flags:1 q:{ i: 119 } o:{ $set: { i: 113, key: &quot;a&quot; } }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Mongod debug log:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Fri Jul 20 12:08:39 [conn1] update mass.belmont query: { i: 119 } update: { $set: { i: 113, key: &quot;a&quot; } } fastmodinsert:1 0ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="144154" author="steven.m.cooke@gmail.com" created="Tue, 17 Jul 2012 18:43:59 +0000"  >&lt;p&gt;Aaron,&lt;/p&gt;

&lt;p&gt; Have you had a chance to look at this again. As you can see from lines 72&lt;br/&gt;
and 74, the value of i does not change inside the loop. Also as pointed&lt;br/&gt;
out, even if wire traffic (using netshark or a similar tool) shows the&lt;br/&gt;
correct number of each update being sent (observing from the client side, I&lt;br/&gt;
believe), an incorrect number of records are inserted.&lt;/p&gt;

&lt;p&gt;  Were you able to run the second version of the script.  I suspect the&lt;br/&gt;
results for mongosniff would be more akin to:   db.arlington.update( &lt;/p&gt;
{
i:1992 }
&lt;p&gt;, &lt;/p&gt;
{ i:1994, key:&apos;b&apos; }
&lt;p&gt;, ... );&lt;br/&gt;
  where 1992 is the expected value of i and &apos;a&apos; is the expected key for the&lt;br/&gt;
record.&lt;/p&gt;

&lt;p&gt;  I&apos;ll try to get more help on the details of the java driver to see if&lt;br/&gt;
there is something there I should be looking for.&lt;/p&gt;

&lt;p&gt;Steve&lt;/p&gt;


</comment>
                            <comment id="142200" author="steven.m.cooke@gmail.com" created="Thu, 12 Jul 2012 04:02:37 +0000"  >&lt;p&gt;Also, I too hastily copied the java code to groovy.  The upsert definition&lt;br/&gt;
should be:&lt;/p&gt;

&lt;p&gt;def  upsert(CountDownLatch latch,  colls, charFn) {&lt;br/&gt;
    def rv = []&lt;br/&gt;
    DOC_COUNT = 2000&lt;/p&gt;

&lt;p&gt;    colls.each { coll -&amp;gt;&lt;/p&gt;


&lt;p&gt;      rv.add({&lt;br/&gt;
          for (int i = 0; i &amp;lt; DOC_COUNT; i++) {&lt;/p&gt;

&lt;p&gt;            coll &lt;/p&gt;
{
               def chr = charFn(it.name);
               rec = [i:i, key: (String)&quot;$chr&quot;] as BasicDBObject
               def setrec = [$set: rec] as BasicDBObject
               it.DB.requestStart()
                        result = it.update([i:i] as BasicDBObject, setrec ,
true, false);
                        //result = it.insert(rec);
                        if (result.error != null) println result
                        it.DB.requestDone();
                        sleep(00)  }
&lt;p&gt;          }&lt;br/&gt;
         // println(&quot;total # of documents after inserting &quot; + DOC_COUNT + &quot;&lt;br/&gt;
docs (should be &quot; + (DOC_COUNT) + &quot;) &quot; + coll.getCount());&lt;/p&gt;

&lt;p&gt;          latch.countDown();&lt;/p&gt;

&lt;p&gt;      });&lt;br/&gt;
    }&lt;br/&gt;
    return rv;&lt;/p&gt;

&lt;p&gt;  }&lt;/p&gt;

&lt;p&gt;requestStart and requestDone made no difference&lt;br/&gt;
The key should now be &apos;a&apos; or &apos;b&apos; for arlington and belmont, respectively&lt;/p&gt;



</comment>
                            <comment id="142198" author="aaron" created="Thu, 12 Jul 2012 03:53:16 +0000"  >&lt;p&gt;I was running mongod, the client, and mongosniff on the same machine and pointed mongosniff at the loopback network interface.&lt;/p&gt;</comment>
                            <comment id="142197" author="steven.m.cooke@gmail.com" created="Thu, 12 Jul 2012 03:50:35 +0000"  >&lt;p&gt;The query and update object should have the same value for i (lines 72 and&lt;br/&gt;
74). Are you running mongosniff on the client, server or both?&lt;/p&gt;

</comment>
                            <comment id="142194" author="aaron" created="Thu, 12 Jul 2012 03:19:21 +0000"  >&lt;p&gt;Hi Steven,&lt;/p&gt;

&lt;p&gt;I ran the script and saw counts != 2000.  Running mongosniff it looks like your test script is sending many update operations like the following:&lt;/p&gt;

&lt;p&gt;	update  flags:1 q:&lt;/p&gt;
{ i: 1992 }
&lt;p&gt; o:&lt;/p&gt;
{ i: 1994, key: &quot;c&quot; }

&lt;p&gt;This is equivalent to&lt;/p&gt;

&lt;p&gt;        db.arlington.update( &lt;/p&gt;
{ i:1992 }
&lt;p&gt;, &lt;/p&gt;
{ i:1994, key:&apos;c&apos; }
&lt;p&gt;, ... );&lt;/p&gt;

&lt;p&gt;Do you expect your test script to generate update operations that look like this, where the &apos;i&apos; value in the query part of the update does not match the &apos;i&apos; value in the update document?&lt;/p&gt;

&lt;p&gt;You can run mongosniff yourself as follows:&lt;/p&gt;

&lt;p&gt;mongosniff --source NET &amp;lt;interface&amp;gt;&lt;/p&gt;

&lt;p&gt;&amp;lt;interface&amp;gt; might be something like eth0, it should be the interface your mongod machine or client machine is sending data on.  You may need to use sudo to run mongosniff.&lt;/p&gt;</comment>
                            <comment id="141602" author="steven.m.cooke@gmail.com" created="Tue, 10 Jul 2012 22:08:38 +0000"  >&lt;p&gt;Aaron,&lt;/p&gt;

&lt;p&gt;  Were you able to run the groovy script?  Let me know if you need any help.&lt;/p&gt;

&lt;p&gt;Steve&lt;/p&gt;</comment>
                            <comment id="138573" author="steven.m.cooke@gmail.com" created="Mon, 2 Jul 2012 13:47:33 +0000"  >&lt;p&gt;Aaron,&lt;/p&gt;

&lt;p&gt;  I think I&apos;ve narrowed down the culprit somewhat by testing the groovy&lt;br/&gt;
code with some different variations.&lt;/p&gt;

&lt;p&gt;1) I checked the wire traffic &amp;#8211; equal number of  arlington/belmont so&lt;br/&gt;
problem may not be on driver side.  The actual commands look correct, but&lt;br/&gt;
 I suppose there are connection keys that could be corrupt&lt;/p&gt;

&lt;p&gt;2) If the update is replaced with insert, the number of records is&lt;br/&gt;
consistently correct, but the documents themselves sometimes end up in the&lt;br/&gt;
wrong place&lt;/p&gt;

&lt;p&gt;3) If inserting, the first record in one of the tables is consistently&lt;br/&gt;
incorrect &lt;span class=&quot;error&quot;&gt;&amp;#91;2.0.5&amp;#93;&lt;/span&gt;. I just updated to 2.0.6 and  this is no longer&lt;br/&gt;
consistent in this way, but there are still errors.&lt;/p&gt;

&lt;p&gt;4) if inserting, the misinserted items have the same $oid as the&lt;br/&gt;
corresponding item in the proper table (same timestamp, machine, inc)&lt;/p&gt;

&lt;p&gt;5) db.requestStart/Done does not seem to make a difference&lt;/p&gt;

&lt;p&gt;6)  Fails frequently &lt;span class=&quot;error&quot;&gt;&amp;#91;1 in 1000&amp;#93;&lt;/span&gt;: result = myCollection.update(&lt;span class=&quot;error&quot;&gt;&amp;#91;i:i&amp;#93;&lt;/span&gt; as&lt;br/&gt;
BasicDBObject, rec , true, false);&lt;br/&gt;
     Fails infrequently &lt;span class=&quot;error&quot;&gt;&amp;#91;1 in 10000&amp;#93;&lt;/span&gt;: result = myCollection.update(&lt;span class=&quot;error&quot;&gt;&amp;#91;i:i&amp;#93;&lt;/span&gt; as&lt;br/&gt;
BasicDBObject, setrec , true, true);&lt;/p&gt;

&lt;p&gt;   i.e. Checking for multiple matches makes a significant difference even&lt;br/&gt;
though there should be only one match. I have not tested whether declaring&lt;br/&gt;
i to be unique makes a difference.&lt;/p&gt;

&lt;p&gt;The table jumping alone does not seem like it would account for all the&lt;br/&gt;
errors I see, namely the data corruption.  I just saw in the docs that&lt;br/&gt;
upserts may lead to field re-ordering.  This is the type of operation I&lt;br/&gt;
would expect might cause what look to be frame-shift errors. Block&lt;br/&gt;
re-ordering is another type of operation I would suspect.&lt;/p&gt;

&lt;p&gt; I have not dug to deeply into the c++ yet, but I think client.cpp is front&lt;br/&gt;
end for networked requests. Yes? Do you know how the requests are queued&lt;br/&gt;
and processed?&lt;/p&gt;

&lt;p&gt;Steve&lt;/p&gt;</comment>
                            <comment id="138215" author="steven.m.cooke@gmail.com" created="Sat, 30 Jun 2012 01:01:44 +0000"  >&lt;p&gt;Aaron,&lt;/p&gt;

&lt;p&gt; I added the line: Logger.getLogger( &quot;com.mongodb.TRACE&quot;&lt;br/&gt;
).setLevel(Level.FINEST);&lt;br/&gt;
 to the start of the script, set the delay to 0,  and counted the instances&lt;br/&gt;
of each table name in the output. There were 2000 messages for each of the&lt;br/&gt;
tables so I don&apos;t think the problem is on the driver side, at least up to&lt;br/&gt;
the &quot;_connector.say&quot; call.  Each of the tables, however was just short of&lt;br/&gt;
2000 records.&lt;/p&gt;</comment>
                            <comment id="137956" author="steven.m.cooke@gmail.com" created="Fri, 29 Jun 2012 15:07:44 +0000"  >&lt;p&gt;Aaron,&lt;/p&gt;

&lt;p&gt;  I haven&apos;t had time to reproduce the logs, but I have a barebones test&lt;br/&gt;
that I think reproduces the error.&lt;/p&gt;

&lt;p&gt;There is a sleep statement on line 74 for 15ms.With that delay I see errors&lt;br/&gt;
in the count of records inserted into the two collections. Increasing over&lt;br/&gt;
50ms and the count is as expected.&lt;/p&gt;

&lt;p&gt;Line 93 has additional collections from a different database.  You can&lt;br/&gt;
uncomment this line and fix the parens and see that records intended for&lt;br/&gt;
one database actually end up in the other (key string case differs between&lt;br/&gt;
dbs).&lt;/p&gt;


&lt;p&gt;Line 73 has an alternative to the simple update with a $set command. Using&lt;br/&gt;
this alternative also exhibits the same behavior.&lt;/p&gt;

&lt;p&gt;The names of collections are the same across databases, but I have changed&lt;br/&gt;
those to be different and still see errors.&lt;/p&gt;


&lt;p&gt;Hope this helps.&lt;/p&gt;



</comment>
                            <comment id="136507" author="aaron" created="Tue, 26 Jun 2012 05:11:51 +0000"  >&lt;p&gt;Hi Steven.  Can you attach the log you are referring to?&lt;/p&gt;</comment>
                            <comment id="136386" author="steven.m.cooke@gmail.com" created="Mon, 25 Jun 2012 19:22:36 +0000"  >&lt;p&gt;Hi Aaron,&lt;/p&gt;

&lt;p&gt;  I just started mongo with objcheck and started simultaneous loads on the&lt;br/&gt;
collections on a fresh store.  I do not see anything reported as errors,&lt;br/&gt;
but I do see lines showing fastmodinserts being made with the wrong data&lt;/p&gt;

&lt;p&gt;Things to note:&lt;br/&gt;
1) The &quot;swaps&quot; seen in logs are from collections with similar table names.&lt;br/&gt;
I.e. Collection1.foo and Collection2.foo where &quot;foo&quot; is the actual name of&lt;br/&gt;
the table.  This may just be coincidence since I only use a handful of&lt;br/&gt;
distinct table names&lt;/p&gt;

&lt;p&gt;2) Not all  &quot;swaps&quot; are logged&lt;/p&gt;

&lt;p&gt;3) The only logged inserts are swaps&lt;/p&gt;



</comment>
                            <comment id="135786" author="aaron" created="Fri, 22 Jun 2012 16:50:54 +0000"  >&lt;p&gt;Hi Steven,&lt;/p&gt;

&lt;p&gt;Another thing to try is to run mongod with the --objcheck option while you are inserting to a fresh database.  This performs validation on insert and will help us determine if you&apos;re running into a client side issue.&lt;/p&gt;</comment>
                            <comment id="135638" author="steven.m.cooke@gmail.com" created="Fri, 22 Jun 2012 10:24:33 +0000"  >&lt;p&gt;From a single web server, yes, all instances share the same&lt;br/&gt;
com.mongodb.Mongo connection.&lt;/p&gt;

&lt;p&gt;One use, however, is to have multiple webservers handle different batches&lt;br/&gt;
of commands. Those web servers will all connect to the same mongodb server&lt;br/&gt;
instance&lt;/p&gt;

</comment>
                            <comment id="135597" author="aaron" created="Fri, 22 Jun 2012 05:25:50 +0000"  >&lt;p&gt;Hi Steven,&lt;/p&gt;

&lt;p&gt;I think Bryan was specifically asking whether all queries are made through a single instance of com.mongodb.Mongo.  Could you let us know?&lt;/p&gt;</comment>
                            <comment id="134573" author="steven.m.cooke@gmail.com" created="Wed, 20 Jun 2012 01:29:38 +0000"  >&lt;p&gt;Bryan,&lt;/p&gt;

&lt;p&gt;  Yes, all requests are through a single connection instance - for the most&lt;br/&gt;
part. On a single machine all db calls share the same instance, but on&lt;br/&gt;
different machines the connections, of course, are not shared.  I would not&lt;br/&gt;
expect the number of client connections to the server to be the issue.&lt;/p&gt;

</comment>
                            <comment id="134549" author="bryan.reinero@10gen.com" created="Tue, 19 Jun 2012 23:16:03 +0000"  >&lt;p&gt;Hi Steven,&lt;/p&gt;

&lt;p&gt;In your use of the Java driver, are you making all queries through a single instance of com.mongodb.Mongo? The Mongo object is thread safe and maintains a connection pool to the database, so you only need to have one instance of the class to handle all your requests. Here a link for a bit  more info: &lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;http://www.mongodb.org/display/DOCS/Java+Driver+Concurrency&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://www.mongodb.org/display/DOCS/Java+Driver+Concurrency&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="134493" author="steven.m.cooke@gmail.com" created="Tue, 19 Jun 2012 20:49:36 +0000"  >&lt;p&gt;Can you give a scenario where interleaving might happen?  I don&apos;t think&lt;br/&gt;
that is possible with my setup as each batch is in its own thread, but I&lt;br/&gt;
can also look closer at how web connections are (re)used.&lt;/p&gt;


</comment>
                            <comment id="134485" author="aaron" created="Tue, 19 Jun 2012 20:09:33 +0000"  >&lt;p&gt;Hi Steven,&lt;/p&gt;

&lt;p&gt;Each of those test scripts runs in a single threaded shell.  The tests are run simultaneously from different shells to generate simultaneous writes to different databases.&lt;/p&gt;

&lt;p&gt;What I mean by interleaving is a situation where two writers are sending messages on a connection and the connection does not get a list of messages in sequence but instead some bytes of one message, some bytes of another message, some more bytes of the first message, etc.  So the messages get mixed up with each other.&lt;/p&gt;</comment>
                            <comment id="134360" author="steven.m.cooke@gmail.com" created="Tue, 19 Jun 2012 18:29:36 +0000"  >&lt;p&gt;Hi Aaron,&lt;/p&gt;

&lt;p&gt; I am little confused by the code.  The while loops are run in separate&lt;br/&gt;
threads? Can you describe what you mean by interleaving?&lt;/p&gt;

&lt;p&gt; I am inserting into multiple collections in a single request but within&lt;br/&gt;
the same database while concurrently doing the same in another&lt;br/&gt;
database/collection set.  With serial insertions into  2 databases with 3&lt;br/&gt;
collections each for 250k docs per collection and each doc contains a 4k&lt;br/&gt;
field. I see no troubles with just java.  I have not had time to try the&lt;br/&gt;
gmongo wrapper, but it looks like it just delegates to the java driver.  I&lt;br/&gt;
have not looked closely and the monkey patching, though.&lt;/p&gt;


&lt;p&gt;So far I have only tried inserts and no upserts which may modify an&lt;br/&gt;
existing doc which does occur in my actual data.  I&apos;ll also try loading one&lt;br/&gt;
collection with actual data.  My real data may be in any language&lt;br/&gt;
(Japanese, Korean, Russian, etc) and my have been converted from MS Word&lt;br/&gt;
documents, so perhaps it is in the documents.&lt;/p&gt;



</comment>
                            <comment id="134347" author="aaron" created="Tue, 19 Jun 2012 17:51:23 +0000"  >&lt;p&gt;Hi Steven,&lt;/p&gt;

&lt;p&gt;I also read and validated the contents of the documents.  Here are the two test scripts I ran (they require the development version of the mongo shell, but I ran them against the 2.0.4 version of mongod).&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;c = db.getSisterDB( &quot;foo&quot; ).c;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;c.drop();&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;docs = [];&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;for( i = 0; i &amp;lt; 1000; ++i ) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    docs.push( { a:0 } );&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;while( 1 ) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    c.insert( docs );&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;c = db.getSisterDB( &quot;bar&quot; ).c;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;c.drop();&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;docs = [];&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;for( i = 0; i &amp;lt; 1000; ++i ) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    docs.push( { a:1 } );&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;while( 1 ) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    c.insert( docs );&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;If you have two threads sharing a connection it&apos;s important to make sure that the messages sent by these threads aren&apos;t being interleaved.  Interleaving could definitely cause invalid documents and the appearance of documents in incorrect namespaces.&lt;/p&gt;</comment>
                            <comment id="133619" author="steven.m.cooke@gmail.com" created="Sun, 17 Jun 2012 04:32:28 +0000"  >&lt;p&gt;With the errors I see, db.repair does not report errors. For example, of&lt;br/&gt;
the handful of databases with errors, only one reports BSONElement type&lt;br/&gt;
errors in the log and only its collections report validation errors.&lt;br/&gt;
 Repair on that collection worked, but removed about 20% of the records in&lt;br/&gt;
one collection. The majority of the incorrect records are still present&lt;/p&gt;

&lt;p&gt;For each collection, validation reports deleted records but delete is never&lt;br/&gt;
called.  Is this normal?&lt;/p&gt;

&lt;p&gt;More details to see if we can find the difference:&lt;br/&gt;
  java driver: 2.7.3&lt;br/&gt;
  gmongo: 0.9.5&lt;br/&gt;
  groovy: 1.8.4&lt;/p&gt;

&lt;p&gt;All threads share the same connection. Can you share your test code or add&lt;br/&gt;
it to the tests on git.  I can not readily make a simple copy of my code or&lt;br/&gt;
data.  I will also start with the QuickTour.java code (&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo-java-driver/blob/master/examples/QuickTour.java&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo-java-driver/blob/master/examples/QuickTour.java&lt;/a&gt;)&lt;br/&gt;
to build tests that emulate my use.  I think that might be easier than&lt;br/&gt;
dissecting the code I already know fails.&lt;/p&gt;

&lt;p&gt;I have looked back at the google group and there are issues with data&lt;br/&gt;
corrruption going back to pre 1.8.6 and as recently as 2.0.x but I could&lt;br/&gt;
not find any resolutions.&lt;/p&gt;




</comment>
                            <comment id="133428" author="aaron" created="Fri, 15 Jun 2012 19:17:55 +0000"  >&lt;p&gt;Hi Steven,&lt;/p&gt;

&lt;p&gt;Last night the jira system we use had an outage, and it looks like the comment you added to this ticket may have been lost.  Could you please re post your message?  Thanks.&lt;/p&gt;</comment>
                            <comment id="132163" author="aaron" created="Wed, 13 Jun 2012 22:07:17 +0000"  >&lt;p&gt;Hi Steven,&lt;/p&gt;

&lt;p&gt;Yep, I was inserting concurrently to the same mongod from two different clients.  I inserted about 10 million documents and then did a repair with no errors.&lt;/p&gt;</comment>
                            <comment id="132075" author="steven.m.cooke@gmail.com" created="Wed, 13 Jun 2012 20:13:04 +0000"  >&lt;p&gt;I&apos;ll see if I can reproduce it with a setup simpler than the one I&lt;br/&gt;
currently have &amp;#8211; groovy/GMongo called through a web interface. I am also&lt;br/&gt;
seeing the same issue in 2.0.6.  Are you inserting concurrently?&lt;br/&gt;
 Previously, I had stated that the changes seemed to happen even if only&lt;br/&gt;
one database was receiving inserts, but with 2.0.6 I have only seen errors&lt;br/&gt;
when multiple databases were in use at roughly the same time. The writes&lt;br/&gt;
are independent so I can not tell if they are truly concurrent.  Roughly,&lt;br/&gt;
with two threads making periodic inserts of about 200 records each, I see&lt;br/&gt;
errors at a rate of about 1 in 1000 after 100k inserts&lt;/p&gt;

&lt;p&gt;Regarding valgrind, I intended to mention earlier that I think chasing down&lt;br/&gt;
the BSON error itself may not be that fruitful. My guess is somewhere there&lt;br/&gt;
is  pointer that is not pointing to the proper buffer and the first visible&lt;br/&gt;
manifestation is the assertion error on the string memory.&lt;/p&gt;

</comment>
                            <comment id="132013" author="aaron" created="Wed, 13 Jun 2012 19:05:01 +0000"  >&lt;p&gt;Hi Steven,&lt;/p&gt;

&lt;p&gt;I did a quick search for the valgrind assertion you encountered and didn&apos;t find any references to it.  So I&apos;m not sure whether the issue has been resolved in the current 3.7.0 valgrind release.&lt;/p&gt;

&lt;p&gt;The validate command does not do a full validation, and the nonzero deletedCount you reported is normal.  (The unused space in your collection is being reported as &quot;deleted&quot;.)&lt;/p&gt;

&lt;p&gt;I also did a test with mongo 2.0.4 where I did batch inserts to two collections in separate databases and did not experience any of the problems you have described.&lt;/p&gt;

&lt;p&gt;I think it would make sense to get some client code that demonstrates this issue, to attempt to rule out a client / driver side issue.&lt;/p&gt;</comment>
                            <comment id="128281" author="steven.m.cooke@gmail.com" created="Thu, 7 Jun 2012 22:40:57 +0000"  >&lt;p&gt;Aaron,&lt;/p&gt;

&lt;p&gt;  What version of valgrind do you recommend.  I am running CentOS 6.2 with&lt;br/&gt;
the latest packaged version, 3.6.0 and trying to avoid installing&lt;br/&gt;
non-packaged apps. Also, the error is not that the BSON string size is&lt;br/&gt;
incorrect.  This is just  a symptom of the larger problem that is occurring&lt;br/&gt;
elsewhere and the type you might see if there were an incorrect block move,&lt;br/&gt;
reuse of an uncleared buffer, incorrect pointer offset, etc&lt;/p&gt;

&lt;p&gt;After the crash I ran repairDatabase.  Attached is a snippet from mongod&lt;br/&gt;
log.  Elided sections before and after are &quot;recursive lock&quot; messages for&lt;br/&gt;
 the collections. Is there any way other than databaseRepair to verify the&lt;br/&gt;
integrity of the mongo data files?&lt;/p&gt;

&lt;p&gt;I have run db.foo.validate(&lt;/p&gt;
{full:true}
&lt;p&gt;) on the databases and no errors are&lt;br/&gt;
reported (&lt;/p&gt;
{..., valid:true, errors:[], ok:1}
&lt;p&gt;).  Perhaps it is part of&lt;br/&gt;
normal operation, but validate does show non-zero deletedCount, but I do&lt;br/&gt;
not delete any documents.  Databases without any inserts have a&lt;br/&gt;
deletedCount = 1 and deletedSize= 8016, for example.&lt;/p&gt;


&lt;p&gt;Currently I am batch inserting docs into collection A and started a&lt;br/&gt;
concurrent batch insert into collection B in a different database.  A&lt;br/&gt;
handful of docs that should be in collection A are now in collection B. In&lt;br/&gt;
both collections the inserts are done periodically, i.e. a couple hundred&lt;br/&gt;
inserts pause, more inserts, etc.&lt;/p&gt;



&lt;p&gt;...&lt;br/&gt;
Mon Jun  4 17:25:26 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn5&amp;#93;&lt;/span&gt; warning: ClientCursor::yield can&apos;t unlock b/c of recursive lock ns: meetup.venues top: { opid: 2061, active: true, lockType: &quot;write&quot;, waitingForLock: false, secs_running: 12, op: &quot;query&quot;, ns: &quot;meetup&quot;, query: &lt;/p&gt;
{ repairDatabase: 1.0 }
&lt;p&gt;, client: &quot;127.0.0.1:38735&quot;, desc: &quot;conn&quot;, threadId: &quot;0x7f4ab09f5700&quot;, connectionId: 5, msg: &quot;index: (3/3) btree-middle&quot;, numYields: 0 }&lt;br/&gt;
Mon Jun  4 17:25:26 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn5&amp;#93;&lt;/span&gt; Assertion: 10334:Invalid BSONObj size: 0 (0x00000000) first element: EOO&lt;br/&gt;
0x5846c2 0x5080e3 0x86ebcf 0x86ed78 0x947c5a 0x8860c4 0x88edca 0x88fcd1 0x5e31db 0x5e36fd 0x850c8c 0x852995 0x8542e9 0x8bd43d 0x9810f2 0x97cdce 0x97e03d 0x941525 0x9433fe 0x889767 &lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x112) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5846c2&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x473) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5080e3&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19CoveredIndexMatcher7matchesERKNS_7BSONObjERKNS_7DiskLocEPNS_12MatchDetailsEb+0x15f) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x86ebcf&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19CoveredIndexMatcher14matchesCurrentEPNS_6CursorEPNS_12MatchDetailsE+0xa8) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x86ed78&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo14processGetMoreEPKcixRNS_5CurOpEiRb+0x3da) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x947c5a&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15receivedGetMoreERNS_10DbResponseERNS_7MessageERNS_5CurOpE+0x1b4) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8860c4&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xf5a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x88edca&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo14DBDirectClient4callERNS_7MessageES2_bPSs+0x81) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x88fcd1&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo14DBClientCursor11requestMoreEv+0x35b) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5e31db&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo14DBClientCursor4moreEv+0x4d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5e36fd&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbbbNS_5QueryE+0x7cc) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x850c8c&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbbbbPi+0x1665) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x852995&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbbbbPi+0x59) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8542e9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x4cd) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8bd43d&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo17CmdRepairDatabase3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x282) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9810f2&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xb5e) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x97cdce&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6fd) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x97e03d&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x941525&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(&lt;em&gt;ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1&lt;/em&gt;+0x172e) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9433fe&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0x889767&amp;#93;&lt;/span&gt;&lt;br/&gt;
Mon Jun  4 17:25:26 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn5&amp;#93;&lt;/span&gt; getmore meetup.venues cursorid:6577276361443070346 exception: Invalid BSONObj size: 0 (0x00000000) first element: EOO code:10334 reslen:20 36ms&lt;br/&gt;
Mon Jun  4 17:25:26 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn5&amp;#93;&lt;/span&gt; build index meetup.venues &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;Mon Jun  4 17:25:27 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn5&amp;#93;&lt;/span&gt; build index done 199999 records 0.749 secs&lt;br/&gt;
Mon Jun  4 17:25:27 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn5&amp;#93;&lt;/span&gt; build index meetup.growth &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;Mon Jun  4 17:25:27 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn5&amp;#93;&lt;/span&gt; build index done 1 records 0 secs&lt;br/&gt;
Mon Jun  4 17:25:27 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn5&amp;#93;&lt;/span&gt; warning: ClientCursor::yield can&apos;t unlock b/c of recursive lock ns: meetup.events top: { opid: 2061, active: true, lockType: &quot;write&quot;, waitingForLock: false, secs_running: 13, op: &quot;query&quot;, ns: &quot;meetup&quot;, query: &lt;/p&gt;
{ repairDatabase: 1.0 }
&lt;p&gt;, client: &quot;127.0.0.1:38735&quot;, desc: &quot;conn&quot;, threadId: &quot;0x7f4ab09f5700&quot;, connectionId: 5, msg: &quot;index: (3/3) btree-middle&quot;, numYields: 0 }&lt;br/&gt;
...&lt;/p&gt;</comment>
                            <comment id="126520" author="aaron" created="Tue, 5 Jun 2012 18:48:11 +0000"  >&lt;p&gt;Hi Steven,&lt;/p&gt;

&lt;p&gt;The most recent stack trace you reported can result from corrupt data files.  We can help diagnose the issue if you provide code to reproduce it or you upgrade valgrind and report the output of the upgraded version.&lt;/p&gt;</comment>
                            <comment id="126255" author="steven.m.cooke@gmail.com" created="Tue, 5 Jun 2012 11:23:47 +0000"  >&lt;p&gt;After applying repairDatabase to each database I was able to resume using&lt;br/&gt;
mongo instance.  I am, however, seeing instances of mis-inserts.&lt;/p&gt;

&lt;p&gt;private def _database(dbname, mongo=null) {&lt;br/&gt;
  def defer = &lt;/p&gt;
{
      def host = mongo ?: &apos;localhost&apos;
      def opts = new MongoOptions()
      opts.connectionsPerHost = 5
      opts.threadsAllowedToBlockForConnectionMultiplier = 10
      opts.autoConnectRetry = true

      new GMongo(new ServerAddress(host), opts)
  }

&lt;p&gt;  // server instance is either the shared instantiated instance in the&lt;br/&gt;
application context&lt;br/&gt;
  // with name &quot;MONGO&quot; or a new instance created with the defer closure&lt;br/&gt;
  // that is saved in the app context with the name &quot;MONGO&quot;&lt;br/&gt;
  server =  getBindingDeferred(&quot;MONGO&quot;, defer) //&lt;br/&gt;
  server.getDB(dbname)&lt;br/&gt;
}&lt;/p&gt;

&lt;p&gt;for each database a map of the collection accessors is created. Something&lt;br/&gt;
like this:&lt;/p&gt;

&lt;p&gt;def database1() {&lt;br/&gt;
  db = _database(&quot;database1&quot;)&lt;br/&gt;
  [ &quot;collection_1&quot;: db.getCollection(&quot;collection_1&quot;),&lt;br/&gt;
    &quot;collection_2&quot;: db.getCollection(&quot;collection_2&quot;),&lt;br/&gt;
    ...&lt;br/&gt;
  ]&lt;br/&gt;
}&lt;/p&gt;

&lt;p&gt;and so on for each database.  The desired collection object is then used to&lt;br/&gt;
get or put documents.&lt;/p&gt;

&lt;p&gt;Note: Instead of instantiating each collection object, the maps could by&lt;br/&gt;
dynamic, e.g.&lt;/p&gt;

&lt;p&gt;   [&quot;collection_1&quot;: &lt;/p&gt;
{ c -&amp;gt; c.call(db.getCollection(&quot;collection_1&quot;))}
&lt;p&gt;, ...]&lt;/p&gt;

&lt;p&gt;and instead of database1().collection_1, database1().collection_1(c) where&lt;br/&gt;
c is a closure taking the actual collection object.  This should preclude&lt;br/&gt;
any chance of a shared collection object being misused, but I do not think&lt;br/&gt;
it is related to documents in the wrong database.&lt;/p&gt;</comment>
                            <comment id="125926" author="steven.m.cooke@gmail.com" created="Mon, 4 Jun 2012 21:40:52 +0000"  >&lt;p&gt;Forgot to answer that &amp;#8211; no geolocation queries&lt;/p&gt;

</comment>
                            <comment id="125919" author="aaron" created="Mon, 4 Jun 2012 21:32:34 +0000"  >&lt;p&gt;Hi Steven,&lt;/p&gt;

&lt;p&gt;I think verbose logging will log the update after it finishes, so if you get a crash before the update finishes it won&apos;t be logged.  What I&apos;m particularly interested in is whether the query/match part of your update uses your geoLocation field, or another geo field.  Do you know if your application is capable of doing this?&lt;/p&gt;

&lt;p&gt;Also, could you send the log from the second crash?  Thanks.&lt;/p&gt;</comment>
                            <comment id="125901" author="steven.m.cooke@gmail.com" created="Mon, 4 Jun 2012 21:04:47 +0000"  >&lt;p&gt;Aaron,&lt;/p&gt;

&lt;p&gt;  The update is to one of four collections, but I do not have application&lt;br/&gt;
logging on the upserts.  Is there a way to increase verbosity in mongod&lt;br/&gt;
logging to show the collection names and key?  Although all incoming&lt;br/&gt;
records should have a key, it is possible the key might be missing so the&lt;br/&gt;
upsert comparison would be on a null.  Might this cause this problem?&lt;/p&gt;

</comment>
                            <comment id="125883" author="aaron" created="Mon, 4 Jun 2012 20:35:11 +0000"  >&lt;p&gt;Hi Steven - is it possible the query part of the update operation that caused this crash was a geo query of some kind?  Is there a way you can tell what the update operation was (perhaps through app server logging)?&lt;/p&gt;</comment>
                            <comment id="125876" author="aaron" created="Mon, 4 Jun 2012 20:24:28 +0000"  >&lt;p&gt;Hi Steven - Could you send the log for the second crash as well?  Thanks&lt;/p&gt;</comment>
                            <comment id="125861" author="steven.m.cooke@gmail.com" created="Mon, 4 Jun 2012 19:51:42 +0000"  >&lt;p&gt;Erik,&lt;/p&gt;

&lt;p&gt;  I gave up on 1.8.x and have now started using 2.0.5.  I have not seen the&lt;br/&gt;
errors I observed before, but I did get  my first server crash while&lt;br/&gt;
inserting.  Attached is the tail of the log.  Connections are from&lt;br/&gt;
JMongoBrowser (10.0.0.73) and from a servlet (10.0.0.107) that is the front&lt;br/&gt;
end to a number of groovy scripts.&lt;/p&gt;

&lt;p&gt;Sidenote:  As I was writing this I noticed mongod crashed again.  A curious&lt;br/&gt;
coincidence is that the connection number for the error here is 2^6 + 1 and&lt;br/&gt;
the connection number for the current crash is 2^7 + 1. Are connections&lt;br/&gt;
allocated in batches of size divisible by some power of 2?&lt;/p&gt;





&lt;p&gt;Sun Jun  3 00:23:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:187 virt:3527 mapped:1424&lt;br/&gt;
Sun Jun  3 00:28:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:187 virt:3527 mapped:1424&lt;br/&gt;
Sun Jun  3 00:33:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:187 virt:3527 mapped:1424&lt;br/&gt;
Sun Jun  3 00:38:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:187 virt:3527 mapped:1424&lt;br/&gt;
Sun Jun  3 00:43:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:187 virt:3527 mapped:1424&lt;br/&gt;
Sun Jun  3 00:48:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:187 virt:3527 mapped:1424&lt;br/&gt;
Sun Jun  3 00:53:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:187 virt:3527 mapped:1424&lt;br/&gt;
Sun Jun  3 00:58:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:187 virt:3527 mapped:1424&lt;br/&gt;
Sun Jun  3 01:03:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:187 virt:3527 mapped:1424&lt;br/&gt;
Sun Jun  3 01:07:43 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:36050 #25&lt;br/&gt;
Sun Jun  3 01:08:11 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.73:3270 #26&lt;br/&gt;
Sun Jun  3 01:08:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn25&amp;#93;&lt;/span&gt; update meetup.venues query: &lt;/p&gt;
{ muid: 2689611 }
&lt;p&gt; update: { $set: { muid: 2689611, name: &quot;Clapham Common&quot;, geoLocation: [ 51.457584, -0.145065 ], address_1: &quot;SW4 9DE&quot;, city: &quot;London&quot;, country: &quot;gb&quot;, email: &quot;&quot;, tags: {} } } fastmodinsert:1 109ms&lt;br/&gt;
Sun Jun  3 01:08:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:190 virt:3529 mapped:1424&lt;br/&gt;
Sun Jun  3 01:10:32 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:36065 #27&lt;br/&gt;
Sun Jun  3 01:13:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:190 virt:3530 mapped:1424&lt;br/&gt;
Sun Jun  3 01:15:16 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:36082 #28&lt;br/&gt;
Sun Jun  3 01:18:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:193 virt:3531 mapped:1424&lt;br/&gt;
Sun Jun  3 01:20:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:36094 #29&lt;br/&gt;
Sun Jun  3 01:23:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:192 virt:3532 mapped:1424&lt;br/&gt;
Sun Jun  3 01:25:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn17&amp;#93;&lt;/span&gt; command admin.$cmd command: &lt;/p&gt;
{ listDatabases: 1 }
&lt;p&gt; ntoreturn:1 reslen:449 142ms&lt;br/&gt;
Sun Jun  3 01:25:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn29&amp;#93;&lt;/span&gt; update meetup.venues query: &lt;/p&gt;
{ muid: 986287 }
&lt;p&gt; update: { $set: { muid: 986287, name: &quot;Dock B, Belmont Harbor North&quot;, zip: &quot;60613&quot;, geoLocation: [ 41.95193, -87.644076 ], state: &quot;IL&quot;, address_1: &quot;3256 North Recreation Drive&quot;, city: &quot;Chicago&quot;, country: &quot;us&quot;, email: &quot;&quot;, tags: {} } } 144ms&lt;br/&gt;
Sun Jun  3 01:28:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:199 virt:3532 mapped:1424&lt;br/&gt;
Sun Jun  3 01:33:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:205 virt:3532 mapped:1424&lt;br/&gt;
Sun Jun  3 01:38:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn29&amp;#93;&lt;/span&gt; update meetup.venues query: &lt;/p&gt;
{ muid: 949106 }
&lt;p&gt; update: { $set: { muid: 949106, name: &quot;Town Tavern&quot;, zip: &quot;20009&quot;, geoLocation: [ 38.920329, -77.041555 ], state: &quot;DC&quot;, address_1: &quot;2323 18th St. NW&quot;, city: &quot;Washington&quot;, country: &quot;us&quot;, email: &quot;&quot;, phone: &quot;202.387.8696&quot;, tags: {} } } 409ms&lt;br/&gt;
Sun Jun  3 01:38:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:213 virt:3532 mapped:1424&lt;br/&gt;
Sun Jun  3 01:43:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn29&amp;#93;&lt;/span&gt; update meetup.venues query: &lt;/p&gt;
{ muid: 686784 }
&lt;p&gt; update: { $set: { muid: 686784, name: &quot;The Magic Castle&quot;, zip: &quot;90028&quot;, geoLocation: [ 34.10456, -118.341816 ], state: &quot;CA&quot;, address_1: &quot;7001 Franklin Ave&quot;, city: &quot;Hollywood&quot;, country: &quot;us&quot;, email: &quot;&quot;, phone: &quot;323-851-3313&quot;, tags: {} } } 170ms&lt;br/&gt;
Sun Jun  3 01:43:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:221 virt:3532 mapped:1424&lt;br/&gt;
Sun Jun  3 01:48:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:228 virt:3532 mapped:1424&lt;br/&gt;
Sun Jun  3 01:53:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:233 virt:3532 mapped:1424&lt;br/&gt;
Sun Jun  3 01:58:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:241 virt:3532 mapped:1424&lt;br/&gt;
Sun Jun  3 02:03:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:248 virt:3532 mapped:1424&lt;br/&gt;
Sun Jun  3 02:08:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:255 virt:3532 mapped:1424&lt;br/&gt;
Sun Jun  3 02:13:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:259 virt:3532 mapped:1424&lt;br/&gt;
Sun Jun  3 02:15:18 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:36253 #30&lt;br/&gt;
Sun Jun  3 02:18:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:268 virt:3533 mapped:1424&lt;br/&gt;
Sun Jun  3 02:23:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:277 virt:3533 mapped:1424&lt;br/&gt;
Sun Jun  3 02:28:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:281 virt:3533 mapped:1424&lt;br/&gt;
Sun Jun  3 02:33:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:289 virt:3533 mapped:1424&lt;br/&gt;
Sun Jun  3 02:38:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:298 virt:3533 mapped:1424&lt;br/&gt;
Sun Jun  3 02:43:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:300 virt:3533 mapped:1424&lt;br/&gt;
Sun Jun  3 02:48:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:301 virt:3533 mapped:1424&lt;br/&gt;
Sun Jun  3 02:53:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:300 virt:3533 mapped:1424&lt;br/&gt;
Sun Jun  3 02:58:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:301 virt:3533 mapped:1424&lt;br/&gt;
Sun Jun  3 03:01:17 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn29&amp;#93;&lt;/span&gt; update meetup.venues query: &lt;/p&gt;
{ muid: 882419 }
&lt;p&gt; update: { $set: { muid: 882419, name: &quot;Winter Park Amtrak Station&quot;, zip: &quot;32789&quot;, geoLocation: [ 28.597743, -81.350183 ], state: &quot;FL&quot;, address_1: &quot;150 E Morse Blvd.&quot;, city: &quot;Winter Park&quot;, country: &quot;us&quot;, email: &quot;&quot;, tags: {} } } 132ms&lt;br/&gt;
Sun Jun  3 03:03:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:309 virt:3533 mapped:1424&lt;br/&gt;
Sun Jun  3 03:08:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:318 virt:3533 mapped:1424&lt;br/&gt;
Sun Jun  3 03:12:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn29&amp;#93;&lt;/span&gt; update meetup.venues query: &lt;/p&gt;
{ muid: 768739 }
&lt;p&gt; update: { $set: { muid: 768739, name: &quot;West Steps, State Capitol&quot;, zip: &quot;95814&quot;, geoLocation: [ 38.574906, -121.486617 ], state: &quot;CA&quot;, address_1: &quot;Capitol Ave&quot;, city: &quot;Sacramento&quot;, country: &quot;us&quot;, email: &quot;&quot;, tags: {} } } 444ms&lt;br/&gt;
Sun Jun  3 03:13:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:323 virt:3533 mapped:1424&lt;br/&gt;
Sun Jun  3 03:15:16 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:36448 #31&lt;br/&gt;
Sun Jun  3 03:18:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn29&amp;#93;&lt;/span&gt; update meetup.venues query: &lt;/p&gt;
{ muid: 7450672 }
&lt;p&gt; update: { $set: { muid: 7450672, name: &quot;DB Bistro Moderne&quot;, zip: &quot;33131&quot;, geoLocation: [ 25.770935, -80.189453 ], state: &quot;FL&quot;, address_1: &quot;255 Biscayne Blvd Way (at the JW Marriott Marquis)&quot;, city: &quot;Miami&quot;, country: &quot;us&quot;, email: &quot;&quot;, tags: {} } } 160ms&lt;br/&gt;
Sun Jun  3 03:18:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn29&amp;#93;&lt;/span&gt; update meetup.groups query: &lt;/p&gt;
{ muid: &quot;288434&quot; }
&lt;p&gt; update: { $set: { muid: &quot;288434&quot;, geoLocation: [ 25.75, -80.2699966431 ], topics: {}, urlname: &quot;Miami-Wine-Tasters&quot;, name: &quot;Miami Wine Tasters&quot; } } moved:1 252ms&lt;br/&gt;
Sun Jun  3 03:18:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:330 virt:3534 mapped:1424&lt;br/&gt;
Sun Jun  3 03:23:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:337 virt:3534 mapped:1424&lt;br/&gt;
Sun Jun  3 03:27:51 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn29&amp;#93;&lt;/span&gt; update meetup.groups query: &lt;/p&gt;
{ muid: &quot;1670903&quot; }
&lt;p&gt; update: { $set: { muid: &quot;1670903&quot;, geoLocation: [ 53.5099983215, -113.629997253 ], topics: {}, urlname: &quot;Calgary-Vavasaur-and-Womanition&quot;, name: &quot;Calgary - Vavasaur and Womanition&quot; } } 116ms&lt;br/&gt;
Sun Jun  3 03:28:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:341 virt:3534 mapped:1424&lt;br/&gt;
Sun Jun  3 03:33:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:345 virt:3534 mapped:1424&lt;br/&gt;
Sun Jun  3 03:38:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:351 virt:3534 mapped:1424&lt;br/&gt;
Sun Jun  3 03:42:15 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn29&amp;#93;&lt;/span&gt; update meetup.venues query: &lt;/p&gt;
{ muid: 7147862 }
&lt;p&gt; update: { $set: { muid: 7147862, name: &quot;Le Commensal&quot;, geoLocation: [ 45.515549, -73.564651 ], state: &quot;QC&quot;, address_1: &quot;1720 St-Denis&quot;, city: &quot;Montr&#239;&#191;&#189;al&quot;, country: &quot;ca&quot;, email: &quot;&quot;, tags: {} } } 165ms&lt;br/&gt;
Sun Jun  3 03:43:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:355 virt:3534 mapped:1424&lt;br/&gt;
Sun Jun  3 03:48:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:362 virt:3534 mapped:1424&lt;br/&gt;
Sun Jun  3 03:53:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:362 virt:3534 mapped:1424&lt;br/&gt;
Sun Jun  3 03:58:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:364 virt:3534 mapped:1424&lt;br/&gt;
Sun Jun  3 04:03:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:364 virt:3534 mapped:1424&lt;br/&gt;
Sun Jun  3 04:08:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:364 virt:3534 mapped:1424&lt;br/&gt;
Sun Jun  3 04:12:20 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:36644 #32&lt;br/&gt;
Sun Jun  3 04:13:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:362 virt:3535 mapped:1424&lt;br/&gt;
Sun Jun  3 04:15:28 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:36657 #33&lt;br/&gt;
Sun Jun  3 04:18:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:364 virt:3536 mapped:1424&lt;br/&gt;
Sun Jun  3 04:23:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:364 virt:3536 mapped:1424&lt;br/&gt;
Sun Jun  3 04:25:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn32&amp;#93;&lt;/span&gt; update meetup.groups query: &lt;/p&gt;
{ muid: &quot;2997162&quot; }
&lt;p&gt; update: { $set: { muid: &quot;2997162&quot;, geoLocation: [ 41.5, -81.66999816889999 ], topics: {}, urlname: &quot;UrbanCode&quot;, name: &quot;UrbanCoders&quot; } } moved:1 371ms&lt;br/&gt;
Sun Jun  3 04:28:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:369 virt:3536 mapped:1424&lt;br/&gt;
Sun Jun  3 04:33:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:373 virt:3536 mapped:1424&lt;br/&gt;
Sun Jun  3 04:38:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:373 virt:3536 mapped:1424&lt;br/&gt;
Sun Jun  3 04:43:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:373 virt:3536 mapped:1424&lt;br/&gt;
Sun Jun  3 04:48:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:373 virt:3536 mapped:1424&lt;br/&gt;
Sun Jun  3 04:53:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:373 virt:3536 mapped:1424&lt;br/&gt;
Sun Jun  3 04:58:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:374 virt:3536 mapped:1424&lt;br/&gt;
Sun Jun  3 05:03:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:375 virt:3536 mapped:1424&lt;br/&gt;
Sun Jun  3 05:08:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:379 virt:3536 mapped:1424&lt;br/&gt;
Sun Jun  3 05:13:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:380 virt:3536 mapped:1424&lt;br/&gt;
Sun Jun  3 05:15:18 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:36834 #34&lt;br/&gt;
Sun Jun  3 05:18:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:378 virt:3537 mapped:1424&lt;br/&gt;
Sun Jun  3 05:23:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:383 virt:3537 mapped:1424&lt;br/&gt;
Sun Jun  3 05:28:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:379 virt:3537 mapped:1424&lt;br/&gt;
Sun Jun  3 05:33:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:381 virt:3537 mapped:1424&lt;br/&gt;
Sun Jun  3 05:38:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:383 virt:3537 mapped:1424&lt;br/&gt;
Sun Jun  3 05:43:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:386 virt:3537 mapped:1424&lt;br/&gt;
Sun Jun  3 05:48:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:388 virt:3537 mapped:1424&lt;br/&gt;
Sun Jun  3 05:53:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:387 virt:3537 mapped:1424&lt;br/&gt;
Sun Jun  3 05:58:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:390 virt:3537 mapped:1424&lt;br/&gt;
Sun Jun  3 06:03:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:390 virt:3537 mapped:1424&lt;br/&gt;
Sun Jun  3 06:08:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:393 virt:3537 mapped:1424&lt;br/&gt;
Sun Jun  3 06:13:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:393 virt:3537 mapped:1424&lt;br/&gt;
Sun Jun  3 06:15:18 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:37039 #35&lt;br/&gt;
Sun Jun  3 06:18:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:394 virt:3538 mapped:1424&lt;br/&gt;
Sun Jun  3 06:23:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:394 virt:3538 mapped:1424&lt;br/&gt;
Sun Jun  3 06:28:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:394 virt:3538 mapped:1424&lt;br/&gt;
Sun Jun  3 06:33:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:394 virt:3538 mapped:1424&lt;br/&gt;
Sun Jun  3 06:38:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:395 virt:3538 mapped:1424&lt;br/&gt;
Sun Jun  3 06:43:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:395 virt:3538 mapped:1424&lt;br/&gt;
Sun Jun  3 06:48:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:395 virt:3538 mapped:1424&lt;br/&gt;
Sun Jun  3 06:53:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:396 virt:3538 mapped:1424&lt;br/&gt;
Sun Jun  3 06:58:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:398 virt:3538 mapped:1424&lt;br/&gt;
Sun Jun  3 07:03:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:397 virt:3538 mapped:1424&lt;br/&gt;
Sun Jun  3 07:08:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:399 virt:3538 mapped:1424&lt;br/&gt;
Sun Jun  3 07:13:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:399 virt:3538 mapped:1424&lt;br/&gt;
Sun Jun  3 07:15:17 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:37234 #36&lt;br/&gt;
Sun Jun  3 07:18:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:405 virt:3539 mapped:1424&lt;br/&gt;
Sun Jun  3 07:23:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:406 virt:3539 mapped:1424&lt;br/&gt;
Sun Jun  3 07:28:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:407 virt:3539 mapped:1424&lt;br/&gt;
Sun Jun  3 07:33:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:405 virt:3539 mapped:1424&lt;br/&gt;
Sun Jun  3 07:38:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:408 virt:3539 mapped:1424&lt;br/&gt;
Sun Jun  3 07:43:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:407 virt:3539 mapped:1424&lt;br/&gt;
Sun Jun  3 07:45:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn32&amp;#93;&lt;/span&gt; update meetup.groups query: &lt;/p&gt;
{ muid: &quot;1658754&quot; }
&lt;p&gt; update: { $set: { muid: &quot;1658754&quot;, geoLocation: [ 41.7700004578, -88.1500015259 ], topics: {}, urlname: &quot;The-Spiritual-Forum&quot;, name: &quot;The Spiritual Forum&quot; } } 277ms&lt;br/&gt;
Sun Jun  3 07:48:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:407 virt:3539 mapped:1424&lt;br/&gt;
Sun Jun  3 07:53:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:408 virt:3539 mapped:1424&lt;br/&gt;
Sun Jun  3 07:58:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:409 virt:3539 mapped:1424&lt;br/&gt;
Sun Jun  3 08:03:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:411 virt:3539 mapped:1424&lt;br/&gt;
Sun Jun  3 08:08:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:408 virt:3539 mapped:1424&lt;br/&gt;
Sun Jun  3 08:13:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:412 virt:3539 mapped:1424&lt;br/&gt;
Sun Jun  3 08:15:22 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:37432 #37&lt;br/&gt;
Sun Jun  3 08:18:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:408 virt:3540 mapped:1424&lt;br/&gt;
Sun Jun  3 08:23:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:409 virt:3540 mapped:1424&lt;br/&gt;
Sun Jun  3 08:28:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:409 virt:3540 mapped:1424&lt;br/&gt;
Sun Jun  3 08:33:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:408 virt:3540 mapped:1424&lt;br/&gt;
Sun Jun  3 08:38:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:409 virt:3540 mapped:1424&lt;br/&gt;
Sun Jun  3 08:43:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:410 virt:3540 mapped:1424&lt;br/&gt;
Sun Jun  3 08:48:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:412 virt:3540 mapped:1424&lt;br/&gt;
Sun Jun  3 08:53:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:409 virt:3540 mapped:1424&lt;br/&gt;
Sun Jun  3 08:58:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:411 virt:3540 mapped:1424&lt;br/&gt;
Sun Jun  3 09:03:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:413 virt:3540 mapped:1424&lt;br/&gt;
Sun Jun  3 09:08:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:411 virt:3540 mapped:1424&lt;br/&gt;
Sun Jun  3 09:13:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:413 virt:3540 mapped:1424&lt;br/&gt;
Sun Jun  3 09:15:28 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:37635 #38&lt;br/&gt;
Sun Jun  3 09:18:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:417 virt:3541 mapped:1424&lt;br/&gt;
Sun Jun  3 09:23:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:416 virt:3541 mapped:1424&lt;br/&gt;
Sun Jun  3 09:28:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:417 virt:3541 mapped:1424&lt;br/&gt;
Sun Jun  3 09:33:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:415 virt:3541 mapped:1424&lt;br/&gt;
Sun Jun  3 09:38:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:415 virt:3541 mapped:1424&lt;br/&gt;
Sun Jun  3 09:43:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:416 virt:3541 mapped:1424&lt;br/&gt;
Sun Jun  3 09:48:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:415 virt:3541 mapped:1424&lt;br/&gt;
Sun Jun  3 09:53:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:417 virt:3541 mapped:1424&lt;br/&gt;
Sun Jun  3 09:58:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:419 virt:3541 mapped:1424&lt;br/&gt;
Sun Jun  3 10:03:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:417 virt:3541 mapped:1424&lt;br/&gt;
Sun Jun  3 10:08:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:420 virt:3541 mapped:1424&lt;br/&gt;
Sun Jun  3 10:13:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:419 virt:3541 mapped:1424&lt;br/&gt;
Sun Jun  3 10:15:21 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:37801 #39&lt;br/&gt;
Sun Jun  3 10:18:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:419 virt:3542 mapped:1424&lt;br/&gt;
Sun Jun  3 10:23:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:418 virt:3542 mapped:1424&lt;br/&gt;
Sun Jun  3 10:28:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:419 virt:3542 mapped:1424&lt;br/&gt;
Sun Jun  3 10:33:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:419 virt:3542 mapped:1424&lt;br/&gt;
Sun Jun  3 10:38:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:423 virt:3542 mapped:1424&lt;br/&gt;
Sun Jun  3 10:43:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:423 virt:3542 mapped:1424&lt;br/&gt;
Sun Jun  3 10:48:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:425 virt:3542 mapped:1424&lt;br/&gt;
Sun Jun  3 10:53:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:424 virt:3542 mapped:1424&lt;br/&gt;
Sun Jun  3 10:58:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:424 virt:3542 mapped:1424&lt;br/&gt;
Sun Jun  3 11:03:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:426 virt:3542 mapped:1424&lt;br/&gt;
Sun Jun  3 11:08:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:424 virt:3542 mapped:1424&lt;br/&gt;
Sun Jun  3 11:13:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:424 virt:3542 mapped:1424&lt;br/&gt;
Sun Jun  3 11:15:19 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:37948 #40&lt;br/&gt;
Sun Jun  3 11:18:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:425 virt:3543 mapped:1424&lt;br/&gt;
Sun Jun  3 11:23:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:427 virt:3543 mapped:1424&lt;br/&gt;
Sun Jun  3 11:28:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:424 virt:3543 mapped:1424&lt;br/&gt;
Sun Jun  3 11:33:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:427 virt:3543 mapped:1424&lt;br/&gt;
Sun Jun  3 11:38:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:425 virt:3543 mapped:1424&lt;br/&gt;
Sun Jun  3 11:43:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:428 virt:3543 mapped:1424&lt;br/&gt;
Sun Jun  3 11:48:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:426 virt:3543 mapped:1424&lt;br/&gt;
Sun Jun  3 11:53:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:429 virt:3543 mapped:1424&lt;br/&gt;
Sun Jun  3 11:58:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:428 virt:3543 mapped:1424&lt;br/&gt;
Sun Jun  3 12:03:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:430 virt:3543 mapped:1424&lt;br/&gt;
Sun Jun  3 12:08:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:428 virt:3543 mapped:1424&lt;br/&gt;
Sun Jun  3 12:13:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:432 virt:3543 mapped:1424&lt;br/&gt;
Sun Jun  3 12:15:24 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:38148 #41&lt;br/&gt;
Sun Jun  3 12:18:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:430 virt:3544 mapped:1424&lt;br/&gt;
Sun Jun  3 12:23:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:431 virt:3544 mapped:1424&lt;br/&gt;
Sun Jun  3 12:28:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:430 virt:3544 mapped:1424&lt;br/&gt;
Sun Jun  3 12:33:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:430 virt:3544 mapped:1424&lt;br/&gt;
Sun Jun  3 12:38:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:433 virt:3544 mapped:1424&lt;br/&gt;
Sun Jun  3 12:43:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:432 virt:3544 mapped:1424&lt;br/&gt;
Sun Jun  3 12:48:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:431 virt:3544 mapped:1424&lt;br/&gt;
Sun Jun  3 12:53:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:431 virt:3544 mapped:1424&lt;br/&gt;
Sun Jun  3 12:58:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:432 virt:3544 mapped:1424&lt;br/&gt;
Sun Jun  3 13:03:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:434 virt:3544 mapped:1424&lt;br/&gt;
Sun Jun  3 13:08:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:434 virt:3544 mapped:1424&lt;br/&gt;
Sun Jun  3 13:13:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:434 virt:3544 mapped:1424&lt;br/&gt;
Sun Jun  3 13:15:24 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:38323 #42&lt;br/&gt;
Sun Jun  3 13:18:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:434 virt:3545 mapped:1424&lt;br/&gt;
Sun Jun  3 13:23:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:434 virt:3545 mapped:1424&lt;br/&gt;
Sun Jun  3 13:28:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:433 virt:3545 mapped:1424&lt;br/&gt;
Sun Jun  3 13:33:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:435 virt:3545 mapped:1424&lt;br/&gt;
Sun Jun  3 13:38:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:434 virt:3545 mapped:1424&lt;br/&gt;
Sun Jun  3 13:43:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:433 virt:3545 mapped:1424&lt;br/&gt;
Sun Jun  3 13:48:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:433 virt:3545 mapped:1424&lt;br/&gt;
Sun Jun  3 13:53:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:433 virt:3545 mapped:1424&lt;br/&gt;
Sun Jun  3 13:58:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:433 virt:3545 mapped:1424&lt;br/&gt;
Sun Jun  3 14:03:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:433 virt:3545 mapped:1424&lt;br/&gt;
Sun Jun  3 14:05:55 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:38451 #43&lt;br/&gt;
Sun Jun  3 14:08:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:433 virt:3546 mapped:1424&lt;br/&gt;
Sun Jun  3 14:13:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:435 virt:3546 mapped:1424&lt;br/&gt;
Sun Jun  3 14:15:19 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:38483 #44&lt;br/&gt;
Sun Jun  3 14:15:20 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:38484 #45&lt;br/&gt;
Sun Jun  3 14:15:20 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn45&amp;#93;&lt;/span&gt; update meetup.groups query: &lt;/p&gt;
{ fbid: &quot;253591661347566&quot;, id: &quot;415615495145181&quot; }
&lt;p&gt; update: { $set: { fbid: &quot;253591661347566&quot;, id: &quot;415615495145181&quot;, graph: { id: &quot;253591661347566_415615495145181&quot;, from: &lt;/p&gt;
{ name: &quot;Pam Bass Rowell&quot;, id: &quot;650285760&quot; }
&lt;p&gt;, to: { data: {} }, message: &quot;When is the feed going to be repaired? I finally tried going back and unfollowing and refollowing all that I follow, but I am afraid I am missing some. &quot;, type: &quot;status&quot;, created_time: &quot;2012-06-03T17:27:14+0000&quot;, updated_time: &quot;2012-06-03T17:27:14+0000&quot;, comments: { data: {} }, properties: {}, actions: {} } } } fastmodinsert:1 338ms&lt;br/&gt;
Sun Jun  3 14:15:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:552 virt:3548 mapped:1424&lt;br/&gt;
Sun Jun  3 14:20:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:437 virt:3548 mapped:1424&lt;br/&gt;
Sun Jun  3 14:25:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:436 virt:3548 mapped:1424&lt;br/&gt;
Sun Jun  3 14:30:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:434 virt:3548 mapped:1424&lt;br/&gt;
Sun Jun  3 14:35:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:436 virt:3548 mapped:1424&lt;br/&gt;
Sun Jun  3 14:40:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:437 virt:3548 mapped:1424&lt;br/&gt;
Sun Jun  3 14:45:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:437 virt:3548 mapped:1424&lt;br/&gt;
Sun Jun  3 14:50:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:436 virt:3548 mapped:1424&lt;br/&gt;
Sun Jun  3 14:55:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:437 virt:3548 mapped:1424&lt;br/&gt;
Sun Jun  3 15:00:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:436 virt:3548 mapped:1424&lt;br/&gt;
Sun Jun  3 15:05:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:436 virt:3548 mapped:1424&lt;br/&gt;
Sun Jun  3 15:10:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:438 virt:3548 mapped:1424&lt;br/&gt;
Sun Jun  3 15:15:29 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:38668 #46&lt;br/&gt;
Sun Jun  3 15:15:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:440 virt:3549 mapped:1424&lt;br/&gt;
Sun Jun  3 15:20:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:439 virt:3549 mapped:1424&lt;br/&gt;
Sun Jun  3 15:25:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:438 virt:3549 mapped:1424&lt;br/&gt;
Sun Jun  3 15:30:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:437 virt:3549 mapped:1424&lt;br/&gt;
Sun Jun  3 15:35:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:439 virt:3549 mapped:1424&lt;br/&gt;
Sun Jun  3 15:40:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:439 virt:3549 mapped:1424&lt;br/&gt;
Sun Jun  3 15:45:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:440 virt:3549 mapped:1424&lt;br/&gt;
Sun Jun  3 15:50:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:438 virt:3549 mapped:1424&lt;br/&gt;
Sun Jun  3 15:55:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:438 virt:3549 mapped:1424&lt;br/&gt;
Sun Jun  3 16:00:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:437 virt:3549 mapped:1424&lt;br/&gt;
Sun Jun  3 16:05:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:439 virt:3549 mapped:1424&lt;br/&gt;
Sun Jun  3 16:10:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:440 virt:3549 mapped:1424&lt;br/&gt;
Sun Jun  3 16:15:21 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:38838 #47&lt;br/&gt;
Sun Jun  3 16:15:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:445 virt:3550 mapped:1424&lt;br/&gt;
Sun Jun  3 16:20:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:438 virt:3550 mapped:1424&lt;br/&gt;
Sun Jun  3 16:25:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:441 virt:3550 mapped:1424&lt;br/&gt;
Sun Jun  3 16:30:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:439 virt:3550 mapped:1424&lt;br/&gt;
Sun Jun  3 16:35:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:441 virt:3550 mapped:1424&lt;br/&gt;
Sun Jun  3 16:40:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:441 virt:3550 mapped:1424&lt;br/&gt;
Sun Jun  3 16:45:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:440 virt:3550 mapped:1424&lt;br/&gt;
Sun Jun  3 16:50:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:438 virt:3550 mapped:1424&lt;br/&gt;
Sun Jun  3 16:55:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:439 virt:3550 mapped:1424&lt;br/&gt;
Sun Jun  3 17:00:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:439 virt:3550 mapped:1424&lt;br/&gt;
Sun Jun  3 17:05:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:439 virt:3550 mapped:1424&lt;br/&gt;
Sun Jun  3 17:10:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:440 virt:3550 mapped:1424&lt;br/&gt;
Sun Jun  3 17:15:25 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:39001 #48&lt;br/&gt;
Sun Jun  3 17:15:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:443 virt:3551 mapped:1424&lt;br/&gt;
Sun Jun  3 17:20:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:442 virt:3551 mapped:1424&lt;br/&gt;
Sun Jun  3 17:25:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:440 virt:3551 mapped:1424&lt;br/&gt;
Sun Jun  3 17:30:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:441 virt:3551 mapped:1424&lt;br/&gt;
Sun Jun  3 17:35:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:441 virt:3551 mapped:1424&lt;br/&gt;
Sun Jun  3 17:40:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:442 virt:3551 mapped:1424&lt;br/&gt;
Sun Jun  3 17:45:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:442 virt:3551 mapped:1424&lt;br/&gt;
Sun Jun  3 17:50:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:440 virt:3551 mapped:1424&lt;br/&gt;
Sun Jun  3 17:55:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:442 virt:3551 mapped:1424&lt;br/&gt;
Sun Jun  3 18:00:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:442 virt:3551 mapped:1424&lt;br/&gt;
Sun Jun  3 18:05:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:440 virt:3551 mapped:1424&lt;br/&gt;
Sun Jun  3 18:10:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:442 virt:3551 mapped:1424&lt;br/&gt;
Sun Jun  3 18:15:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:39223 #49&lt;br/&gt;
Sun Jun  3 18:15:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:446 virt:3553 mapped:1424&lt;br/&gt;
Sun Jun  3 18:20:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:442 virt:3553 mapped:1424&lt;br/&gt;
Sun Jun  3 18:25:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:442 virt:3553 mapped:1424&lt;br/&gt;
Sun Jun  3 18:30:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:442 virt:3553 mapped:1424&lt;br/&gt;
Sun Jun  3 18:35:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:441 virt:3553 mapped:1424&lt;br/&gt;
Sun Jun  3 18:40:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:442 virt:3553 mapped:1424&lt;br/&gt;
Sun Jun  3 18:45:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:443 virt:3553 mapped:1424&lt;br/&gt;
Sun Jun  3 18:50:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:445 virt:3553 mapped:1424&lt;br/&gt;
Sun Jun  3 18:55:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:444 virt:3553 mapped:1424&lt;br/&gt;
Sun Jun  3 19:00:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:443 virt:3553 mapped:1424&lt;br/&gt;
Sun Jun  3 19:05:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:444 virt:3553 mapped:1424&lt;br/&gt;
Sun Jun  3 19:10:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:445 virt:3553 mapped:1424&lt;br/&gt;
Sun Jun  3 19:15:28 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:39387 #50&lt;br/&gt;
Sun Jun  3 19:15:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:446 virt:3554 mapped:1424&lt;br/&gt;
Sun Jun  3 19:20:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:443 virt:3554 mapped:1424&lt;br/&gt;
Sun Jun  3 19:25:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:443 virt:3554 mapped:1424&lt;br/&gt;
Sun Jun  3 19:30:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:446 virt:3554 mapped:1424&lt;br/&gt;
Sun Jun  3 19:35:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:444 virt:3554 mapped:1424&lt;br/&gt;
Sun Jun  3 19:40:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:444 virt:3554 mapped:1424&lt;br/&gt;
Sun Jun  3 19:45:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:446 virt:3554 mapped:1424&lt;br/&gt;
Sun Jun  3 19:50:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:445 virt:3554 mapped:1424&lt;br/&gt;
Sun Jun  3 19:56:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:446 virt:3554 mapped:1424&lt;br/&gt;
Sun Jun  3 20:01:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:445 virt:3554 mapped:1424&lt;br/&gt;
Sun Jun  3 20:06:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:447 virt:3554 mapped:1424&lt;br/&gt;
Sun Jun  3 20:11:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:447 virt:3554 mapped:1424&lt;br/&gt;
Sun Jun  3 20:15:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:39545 #51&lt;br/&gt;
Sun Jun  3 20:16:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:451 virt:3555 mapped:1424&lt;br/&gt;
Sun Jun  3 20:21:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:447 virt:3555 mapped:1424&lt;br/&gt;
Sun Jun  3 20:26:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:446 virt:3555 mapped:1424&lt;br/&gt;
Sun Jun  3 20:31:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:445 virt:3555 mapped:1424&lt;br/&gt;
Sun Jun  3 20:36:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:445 virt:3555 mapped:1424&lt;br/&gt;
Sun Jun  3 20:41:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:446 virt:3555 mapped:1424&lt;br/&gt;
Sun Jun  3 20:46:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:446 virt:3555 mapped:1424&lt;br/&gt;
Sun Jun  3 20:51:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:445 virt:3555 mapped:1424&lt;br/&gt;
Sun Jun  3 20:56:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:445 virt:3555 mapped:1424&lt;br/&gt;
Sun Jun  3 21:01:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:447 virt:3555 mapped:1424&lt;br/&gt;
Sun Jun  3 21:06:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:446 virt:3555 mapped:1424&lt;br/&gt;
Sun Jun  3 21:11:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:446 virt:3555 mapped:1424&lt;br/&gt;
Sun Jun  3 21:15:22 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:39705 #52&lt;br/&gt;
Sun Jun  3 21:16:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:451 virt:3556 mapped:1424&lt;br/&gt;
Sun Jun  3 21:21:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:446 virt:3556 mapped:1424&lt;br/&gt;
Sun Jun  3 21:26:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:449 virt:3556 mapped:1424&lt;br/&gt;
Sun Jun  3 21:31:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:448 virt:3556 mapped:1424&lt;br/&gt;
Sun Jun  3 21:36:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:448 virt:3556 mapped:1424&lt;br/&gt;
Sun Jun  3 21:41:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:449 virt:3556 mapped:1424&lt;br/&gt;
Sun Jun  3 21:43:35 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:39808 #53&lt;br/&gt;
Sun Jun  3 21:43:35 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn53&amp;#93;&lt;/span&gt; build index meetup.growth &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;Sun Jun  3 21:43:35 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn53&amp;#93;&lt;/span&gt; build index done 0 records 0 secs&lt;br/&gt;
Sun Jun  3 21:44:09 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.73:2829 #54&lt;br/&gt;
Sun Jun  3 21:46:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:447 virt:3558 mapped:1424&lt;br/&gt;
Sun Jun  3 21:51:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:449 virt:3558 mapped:1424&lt;br/&gt;
Sun Jun  3 21:51:49 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:39830 #55&lt;br/&gt;
Sun Jun  3 21:55:27 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:39838 #56&lt;br/&gt;
Sun Jun  3 21:56:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:450 virt:3560 mapped:1424&lt;br/&gt;
Sun Jun  3 21:56:36 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:39842 #57&lt;br/&gt;
Sun Jun  3 21:57:05 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:39845 #58&lt;br/&gt;
Sun Jun  3 21:57:48 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.73:3084 #59&lt;br/&gt;
Sun Jun  3 22:01:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:432 virt:3563 mapped:1424&lt;br/&gt;
Sun Jun  3 22:06:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:432 virt:3563 mapped:1424&lt;br/&gt;
Sun Jun  3 22:11:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:433 virt:3563 mapped:1424&lt;br/&gt;
Sun Jun  3 22:15:25 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:40346 #60&lt;br/&gt;
Sun Jun  3 22:16:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:435 virt:3564 mapped:1424&lt;br/&gt;
Sun Jun  3 22:21:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:432 virt:3564 mapped:1424&lt;br/&gt;
Sun Jun  3 22:26:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:432 virt:3564 mapped:1424&lt;br/&gt;
Sun Jun  3 22:30:19 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:40483 #61&lt;br/&gt;
Sun Jun  3 22:30:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn61&amp;#93;&lt;/span&gt; build index songkick.events &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;Sun Jun  3 22:30:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn61&amp;#93;&lt;/span&gt; build index done 0 records 0 secs&lt;br/&gt;
Sun Jun  3 22:30:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn61&amp;#93;&lt;/span&gt; info: creating collection songkick.events on add index&lt;br/&gt;
Sun Jun  3 22:30:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn61&amp;#93;&lt;/span&gt; build index songkick.events &lt;/p&gt;
{ skid: 1 }
&lt;p&gt;Sun Jun  3 22:30:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn61&amp;#93;&lt;/span&gt; build index done 0 records 0 secs&lt;br/&gt;
Sun Jun  3 22:30:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn61&amp;#93;&lt;/span&gt; build index songkick.venues &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;Sun Jun  3 22:30:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn61&amp;#93;&lt;/span&gt; build index done 0 records 0 secs&lt;br/&gt;
Sun Jun  3 22:30:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn61&amp;#93;&lt;/span&gt; info: creating collection songkick.venues on add index&lt;br/&gt;
Sun Jun  3 22:30:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn61&amp;#93;&lt;/span&gt; build index songkick.venues &lt;/p&gt;
{ skid: 1 }
&lt;p&gt;Sun Jun  3 22:30:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn61&amp;#93;&lt;/span&gt; build index done 0 records 0 secs&lt;br/&gt;
Sun Jun  3 22:30:57 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.73:3711 #62&lt;br/&gt;
Sun Jun  3 22:31:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:432 virt:3566 mapped:1424&lt;br/&gt;
Sun Jun  3 22:31:12 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:40508 #63&lt;br/&gt;
Sun Jun  3 22:31:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.73:3719 #64&lt;br/&gt;
Sun Jun  3 22:32:03 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.0.107:40523 #65&lt;br/&gt;
Sun Jun  3 22:32:03 Invalid access at address: 0x7f05c23a746f&lt;/p&gt;

&lt;p&gt;Sun Jun  3 22:32:03 Got signal: 11 (Segmentation fault).&lt;/p&gt;

&lt;p&gt;Sun Jun  3 22:32:03 Backtrace:&lt;br/&gt;
0xa95df9 0xa9651c 0x3d4800f4a0 0x86eb14 0x86ed78 0x9794dc 0x8c9a96 0x8d9930 0x8d9f86 0x8ddc3e 0x8df5f5 0x8dfaf0 0x8e1bcd 0x95f542 0x9642d5 0x88d63c 0x88efdf 0xaa0cc8 0x638997 0x3d480077f1 &lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x3a9) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa95df9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x22c) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa9651c&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib64/libpthread.so.0() &lt;span class=&quot;error&quot;&gt;&amp;#91;0x3d4800f4a0&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19CoveredIndexMatcher7matchesERKNS_7BSONObjERKNS_7DiskLocEPNS_12MatchDetailsEb+0xa4) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x86eb14&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19CoveredIndexMatcher14matchesCurrentEPNS_6CursorEPNS_12MatchDetailsE+0xa8) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x86ed78&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo8UpdateOp4nextEv+0x1ac) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9794dc&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x56) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8c9a96&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner4nextEv+0x110) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8d9930&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv+0x56) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8d9f86&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x11e) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8ddc3e&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x525) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8df5f5&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11MultiCursor10nextClauseEv+0x70) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8dfaf0&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11MultiCursorC1EPKcRKNS_7BSONObjES5_N5boost10shared_ptrINS0_8CursorOpEEEb+0x20d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8e1bcd&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugEPNS_11RemoveSaverE+0x372) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x95f542&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugE+0x125) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9642d5&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x46c) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x88d63c&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x116f) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x88efdf&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xaa0cc8&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x638997&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib64/libpthread.so.0() &lt;span class=&quot;error&quot;&gt;&amp;#91;0x3d480077f1&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Logstream::get called in uninitialized state&lt;br/&gt;
Sun Jun  3 22:32:03 ERROR: Client::~Client _context should be null but is not; client:conn&lt;br/&gt;
Logstream::get called in uninitialized state&lt;br/&gt;
Sun Jun  3 22:32:03 ERROR: Client::shutdown not called: conn&lt;/p&gt;


&lt;ul&gt;
	&lt;li&gt;
	&lt;ul&gt;
		&lt;li&gt;
		&lt;ul&gt;
			&lt;li&gt;
			&lt;ul&gt;
				&lt;li&gt;
				&lt;ul&gt;
					&lt;li&gt;SERVER RESTARTED *****&lt;/li&gt;
				&lt;/ul&gt;
				&lt;/li&gt;
			&lt;/ul&gt;
			&lt;/li&gt;
		&lt;/ul&gt;
		&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="118983" author="milkie" created="Mon, 14 May 2012 01:49:45 +0000"  >&lt;p&gt;Do you have the corresponding server logs from your BSON errors this afternoon?  I would like to see them.&lt;/p&gt;</comment>
                            <comment id="118981" author="steven.m.cooke@gmail.com" created="Mon, 14 May 2012 01:31:28 +0000"  >&lt;p&gt;I am no long inserting anything into mongo so I am not sure I can produce&lt;br/&gt;
any new logs.  I may have some old logs.&lt;/p&gt;

&lt;p&gt;Here is a recent example of a BSON error received after iterating a&lt;br/&gt;
collection of about 580k documents. The error happened after reading 218325&lt;br/&gt;
documents.  I was able to read the remainder after skipping 218300&lt;br/&gt;
documents. The string size translates to ASCII &quot;ts\0\0&quot;. I am using driver&lt;br/&gt;
version 2.7.3&lt;/p&gt;

&lt;p&gt;May 13, 2012 6:54:53 PM com.mongodb.DBPortPool gotError&lt;br/&gt;
WARNING: emptying DBPortPool to mongodb.hierweb.com:27017 b/c of error&lt;br/&gt;
org.bson.BSONException: bad string size: 1953693696&lt;br/&gt;
        at&lt;br/&gt;
org.bson.BasicBSONDecoder$BSONInput.readUTF8String(BasicBSONDecoder.java:456)&lt;br/&gt;
        at&lt;br/&gt;
org.bson.BasicBSONDecoder.decodeElement(BasicBSONDecoder.java:155)&lt;br/&gt;
        at&lt;br/&gt;
org.bson.BasicBSONDecoder.decodeElement(BasicBSONDecoder.java:206)&lt;br/&gt;
        at org.bson.BasicBSONDecoder._decode(BasicBSONDecoder.java:79)&lt;br/&gt;
        at org.bson.BasicBSONDecoder.decode(BasicBSONDecoder.java:57)&lt;br/&gt;
        at com.mongodb.DefaultDBDecoder.decode(DefaultDBDecoder.java:56)&lt;br/&gt;
        at com.mongodb.Response.&amp;lt;init&amp;gt;(Response.java:66)&lt;br/&gt;
        at com.mongodb.DBPort.go(DBPort.java:128)&lt;br/&gt;
        at com.mongodb.DBPort.call(DBPort.java:79)&lt;br/&gt;
        at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:218)&lt;br/&gt;
        at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:189)&lt;br/&gt;
        at com.mongodb.DBApiLayer$Result._advance(DBApiLayer.java:452)&lt;br/&gt;
        at com.mongodb.DBApiLayer$Result.hasNext(DBApiLayer.java:418)&lt;br/&gt;
        at com.mongodb.DBCursor._hasNext(DBCursor.java:509)&lt;br/&gt;
        at com.mongodb.DBCursor.hasNext(DBCursor.java:529)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1375)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1349)&lt;br/&gt;
        at org.codehaus.groovy.runtime.dgm$142.doMethodInvoke(Unknown&lt;br/&gt;
Source)&lt;br/&gt;
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1054)&lt;br/&gt;
        at&lt;br/&gt;
groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)&lt;br/&gt;
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:884)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.callsite.PerInstancePojoMetaClassSite.call(PerInstancePojoMetaClassSite.java:35)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)&lt;br/&gt;
        at common.queryCollection(common.groovy:278)&lt;br/&gt;
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)&lt;br/&gt;
        at&lt;br/&gt;
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)&lt;br/&gt;
        at&lt;br/&gt;
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)&lt;br/&gt;
        at java.lang.reflect.Method.invoke(Method.java:597)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:226)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:52)&lt;br/&gt;
        at common.queryCollection(common.groovy)&lt;br/&gt;
        at common$queryCollection.call(Unknown Source)&lt;br/&gt;
        at Facebook$_eachWall_closure12.doCall(Facebook.groovy:162)&lt;br/&gt;
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)&lt;br/&gt;
        at&lt;br/&gt;
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)&lt;br/&gt;
        at&lt;br/&gt;
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)&lt;br/&gt;
        at java.lang.reflect.Method.invoke(Method.java:597)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)&lt;br/&gt;
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)&lt;br/&gt;
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:884)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)&lt;br/&gt;
        at Facebook.wall(Facebook.groovy:170)&lt;br/&gt;
        at Facebook$wall.callCurrent(Unknown Source)&lt;br/&gt;
        at Facebook.eachWall(Facebook.groovy:161)&lt;br/&gt;
        at Facebook$eachWall.call(Unknown Source)&lt;br/&gt;
        at&lt;br/&gt;
Script6$_recover_wall_closure3_closure10.doCall(Script6.groovy:105)&lt;br/&gt;
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)&lt;br/&gt;
        at&lt;br/&gt;
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)&lt;br/&gt;
        at&lt;br/&gt;
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)&lt;br/&gt;
        at java.lang.reflect.Method.invoke(Method.java:597)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)&lt;br/&gt;
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)&lt;br/&gt;
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:884)&lt;br/&gt;
        at groovy.lang.Closure.call(Closure.java:410)&lt;br/&gt;
        at groovy.lang.Closure.call(Closure.java:423)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.DefaultGroovyMethods.withWriter(DefaultGroovyMethods.java:16122)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.DefaultGroovyMethods.withWriter(DefaultGroovyMethods.java:15977)&lt;br/&gt;
        at org.codehaus.groovy.runtime.dgm$904.invoke(Unknown Source)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)&lt;br/&gt;
        at Script6$_recover_wall_closure3.doCall(Script6.groovy:104)&lt;br/&gt;
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)&lt;br/&gt;
        at&lt;br/&gt;
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)&lt;br/&gt;
        at&lt;br/&gt;
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)&lt;br/&gt;
        at java.lang.reflect.Method.invoke(Method.java:597)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)&lt;br/&gt;
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)&lt;br/&gt;
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:884)&lt;br/&gt;
        at groovy.lang.Closure.call(Closure.java:410)&lt;br/&gt;
        at groovy.lang.Closure.call(Closure.java:423)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.DefaultGroovyMethods.step(DefaultGroovyMethods.java:12626)&lt;br/&gt;
        at org.codehaus.groovy.runtime.dgm$722.invoke(Unknown Source)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)&lt;br/&gt;
        at Script6.recover_wall(Script6.groovy:101)&lt;br/&gt;
        at Script6.run(Script6.groovy:54)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:315)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:111)&lt;br/&gt;
        at&lt;br/&gt;
org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:103)&lt;br/&gt;
        at&lt;br/&gt;
javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:195)&lt;br/&gt;
        at com.hierweb.mdt.ScriptHandler.handle(ScriptHandler.java:206)&lt;br/&gt;
        at&lt;br/&gt;
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)&lt;br/&gt;
        at org.eclipse.jetty.server.Server.handle(Server.java:338)&lt;br/&gt;
        at&lt;br/&gt;
org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:568)&lt;br/&gt;
        at&lt;br/&gt;
org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:995)&lt;br/&gt;
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:560)&lt;br/&gt;
        at&lt;br/&gt;
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:222)&lt;br/&gt;
        at&lt;br/&gt;
org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:403)&lt;br/&gt;
        at&lt;br/&gt;
org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:474)&lt;br/&gt;
        at&lt;br/&gt;
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:437)&lt;br/&gt;
        at java.lang.Thread.run(Thread.java:662)&lt;/p&gt;


&lt;p&gt;The comment in BasicBSONDecoder right before the size check makes me think&lt;br/&gt;
this problem has been around for a while and this is just a workaround&lt;br/&gt;
masking the actual cause of corruption.&lt;/p&gt;


&lt;p&gt;Reading the same collection, this time running from the Netbeans debugger I&lt;br/&gt;
get this error after reading ~57k documents:&lt;/p&gt;

&lt;p&gt;May 13, 2012 8:33:05 PM com.mongodb.DBPortPool gotError&lt;br/&gt;
WARNING: emptying DBPortPool to mongodb.hierweb.com:27017 b/c of error&lt;br/&gt;
java.lang.ArrayIndexOutOfBoundsException&lt;br/&gt;
at java.lang.System.arraycopy(Native Method)&lt;br/&gt;
at org.bson.BasicBSONDecoder$BSONInput._need(BasicBSONDecoder.java:318)&lt;br/&gt;
at org.bson.BasicBSONDecoder$BSONInput.read(BasicBSONDecoder.java:366)&lt;br/&gt;
at org.bson.BasicBSONDecoder$BSONInput.readCStr(BasicBSONDecoder.java:407)&lt;br/&gt;
at org.bson.BasicBSONDecoder.decodeElement(BasicBSONDecoder.java:123)&lt;br/&gt;
at org.bson.BasicBSONDecoder._decode(BasicBSONDecoder.java:79)&lt;br/&gt;
at org.bson.BasicBSONDecoder.decode(BasicBSONDecoder.java:57)&lt;br/&gt;
at com.mongodb.DefaultDBDecoder.decode(DefaultDBDecoder.java:56)&lt;br/&gt;
at com.mongodb.Response.&amp;lt;init&amp;gt;(Response.java:66)&lt;br/&gt;
at com.mongodb.DBPort.go(DBPort.java:128)&lt;br/&gt;
at com.mongodb.DBPort.call(DBPort.java:79)&lt;br/&gt;
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:218)&lt;br/&gt;
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:189)&lt;br/&gt;
at com.mongodb.DBApiLayer$Result._advance(DBApiLayer.java:452)&lt;br/&gt;
at com.mongodb.DBApiLayer$Result.hasNext(DBApiLayer.java:418)&lt;br/&gt;
at com.mongodb.DBCursor._hasNext(DBCursor.java:509)&lt;br/&gt;
at com.mongodb.DBCursor.hasNext(DBCursor.java:529)&lt;br/&gt;
at&lt;br/&gt;
org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1375)&lt;br/&gt;
at&lt;br/&gt;
org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1349)&lt;/p&gt;

&lt;p&gt;No changes  were made to the collection or the code&lt;/p&gt;

</comment>
                            <comment id="118709" author="milkie" created="Fri, 11 May 2012 19:36:22 +0000"  >&lt;p&gt;Thanks for the general overview.  Attach some logs and some sample insert code, with some sample documents, and I&apos;ll try to set something up locally to reproduce.&lt;/p&gt;</comment>
                            <comment id="118641" author="steven.m.cooke@gmail.com" created="Fri, 11 May 2012 16:59:26 +0000"  >&lt;p&gt;Must have hit tab or enter at the wrong time..&lt;/p&gt;


&lt;p&gt;Hi Eric,&lt;br/&gt;
  I have reproduced it multiple time &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;  The general procedure is&lt;br/&gt;
    a) create multiple collections giving each collections an easily&lt;br/&gt;
distinguished id name field, e.g. aid, bid, did&lt;br/&gt;
    b) insert records into all &amp;#8211; probably on the order of 100k. The&lt;br/&gt;
documents I work with have a very wide range in size from hundred of bytes&lt;br/&gt;
to tens of kilobytes.  Think tweets to word docs converted to html. Insert&lt;br/&gt;
the collections&lt;br/&gt;
    c) select documents from collection A having bid or did fields and&lt;br/&gt;
repeat for B and D collections&lt;/p&gt;

&lt;p&gt;   There error rate seems to be around 1 in 10000 for my data and my&lt;br/&gt;
suspicion is that that rate is a function of the mix of my data not a fixed&lt;br/&gt;
periodicity.  Also, because the data seems to be intact data from some&lt;br/&gt;
record, it seems either a read or write frameshift has occurred in such a&lt;br/&gt;
way the location corresponding to BSON size is actually some part of this&lt;br/&gt;
other data.&lt;/p&gt;


&lt;p&gt;  I did not update valgrind, but I am more inclined to believe its output&lt;br/&gt;
than to attribute the error to monitoring tool.&lt;/p&gt;</comment>
                            <comment id="118617" author="milkie" created="Fri, 11 May 2012 15:47:27 +0000"  >&lt;p&gt;Hi Steven,&lt;br/&gt;
I totally believe Valgrind&apos;s output telling us that Valgrind is crashing.  I&apos;m hoping that an upgrade will give us more useful output to help diagnose the bug in mongo code.&lt;/p&gt;

&lt;p&gt;Can you please attach a server log showing the time when you inserted the records and when you ran tests to detect the issue (seeing the BSON errors and extremely long strings would also be good)? Also, provide an example of your code that you are using to insert using the Java driver.  I would like to attempt to reproduce this on my own setup.&lt;/p&gt;</comment>
                            <comment id="118615" author="steven.m.cooke@gmail.com" created="Fri, 11 May 2012 15:41:24 +0000"  >&lt;p&gt;Hi Eric,&lt;/p&gt;

&lt;p&gt;  I have reproduced it multiple time &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;



&lt;p&gt;  I did not update valgrind, but I am more inclined to believe the its&lt;br/&gt;
output than to attribute the error to monitoring tool.&lt;/p&gt;

</comment>
                            <comment id="118578" author="milkie" created="Fri, 11 May 2012 13:54:05 +0000"  >&lt;p&gt;Hi Steven,&lt;br/&gt;
Is this still reproducible?  Did you try upgrading Valgrind?&lt;/p&gt;</comment>
                            <comment id="111572" author="steven.m.cooke@gmail.com" created="Thu, 19 Apr 2012 13:49:29 +0000"  >&lt;p&gt;I restarted with a fresh set collections and can not readily switch to a&lt;br/&gt;
corrupt backup.  Perhaps it is related, but also observe beam.smp from&lt;br/&gt;
erlang/couchbase (erl 5.8.5):&lt;/p&gt;

&lt;p&gt;Apr 17 19:39:48 localhost kernel: beam.smp&lt;span class=&quot;error&quot;&gt;&amp;#91;1929&amp;#93;&lt;/span&gt;: segfault at 0 ip&lt;br/&gt;
00000000004ac8bb sp 00007fb39fcbcc10 error 6 in beam.smp&lt;span class=&quot;error&quot;&gt;&amp;#91;400000+1c6000&amp;#93;&lt;/span&gt;&lt;br/&gt;
Apr 17 19:39:50 localhost abrt&lt;span class=&quot;error&quot;&gt;&amp;#91;2619&amp;#93;&lt;/span&gt;: saved core dump of pid 1899&lt;br/&gt;
(/usr/lib64/erlang/erts-5.8.5/bin/beam.smp) to&lt;br/&gt;
/var/spool/abrt/ccpp-2012-04-17-19:39:48-1899.new/coredump (129990656 bytes)&lt;/p&gt;


&lt;p&gt;The reported reason is &apos;Process /sbin/readahead was killed by signal 11&lt;br/&gt;
(SIGSEGV)&apos;&lt;/p&gt;

</comment>
                            <comment id="111293" author="milkie" created="Wed, 18 Apr 2012 18:38:33 +0000"  >&lt;p&gt;This line: &lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Memcheck: mc_malloc_wrappers.c:113 (add_to_freed_queue): Assertion &apos;VG_(free_queue_volume) &amp;gt;= 0&apos; failed.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;means that valgrind hit an assertion and crashed. It could be due to memory corruption or it could be due to bugs in Valgrind.  Can you upgrade valgrind to the current version and try again?&lt;/p&gt;</comment>
                            <comment id="111271" author="steven.m.cooke@gmail.com" created="Wed, 18 Apr 2012 17:43:25 +0000"  >&lt;p&gt;Tad, I&apos;ve run full validate on problem collections.  If it reports &lt;/p&gt;
{ok:1}
&lt;p&gt;, the collection may still be corrupted. Dropping indexes or running the compact command does not fix the problem.  In the worst case scenario, db.repairDatabase results in total data loss of that collection.&lt;/p&gt;</comment>
                            <comment id="111266" author="steven.m.cooke@gmail.com" created="Wed, 18 Apr 2012 17:38:03 +0000"  >&lt;p&gt;Attached is the valgrind output from restarting mongod after observing corrupted collections&lt;/p&gt;</comment>
                            <comment id="110605" author="tad" created="Tue, 17 Apr 2012 09:13:59 +0000"  >&lt;p&gt;Can you run db.foo.validate(&lt;/p&gt;
{full:true}
&lt;p&gt;) on all the databases that are showing this problem?  Also, if an index is being used in a query that is returning bad results or generating errors in the log, is it possible to rebuild the index and see if that fixes the problem?  What is valgrind telling you?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="17663" name="QuickTour_J_2DB_6T1.groovy" size="2698" author="steven.m.cooke@gmail.com" created="Fri, 29 Jun 2012 15:07:44 +0000"/>
                            <attachment id="17092" name="bson_repair.txt" size="3371" author="steven.m.cooke@gmail.com" created="Thu, 7 Jun 2012 22:40:58 +0000"/>
                            <attachment id="17008" name="merr.txt" size="31242" author="steven.m.cooke@gmail.com" created="Mon, 4 Jun 2012 19:51:43 +0000"/>
                            <attachment id="17662" name="pom.xml" size="2296" author="steven.m.cooke@gmail.com" created="Fri, 29 Jun 2012 15:07:44 +0000"/>
                            <attachment id="16069" name="valgrind.txt" size="31153" author="steven.m.cooke@gmail.com" created="Wed, 18 Apr 2012 17:38:03 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>55.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 17 Apr 2012 09:13:59 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 22 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 22 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>aaron</customfieldvalue>
            <customfieldvalue>bryan.reinero</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>steven.m.cooke@gmail.com</customfieldvalue>
            <customfieldvalue>tad</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hro5k7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrg8rj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8325</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkz2f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>