[SERVER-7106] Invalid Data Inserted into Profiling Collection, Breaking Dex and count function Created: 21/Sep/12 Updated: 15/Feb/13 Resolved: 13/Dec/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Logging |
| Affects Version/s: | 2.2.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Kyle Gato | Assignee: | Tad Marshall |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | collection, profiling | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Linux web-stage-1 3.0.0-15-virtual #25-Ubuntu SMP Mon Jan 2 20:45:15 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux Running Ubuntu On Amazon EC2. Condensed MongoDB Cluster w/ 9 Daemons db version v2.2.0, pdfile version 4.5 kyle@db-stage-1:~$ pip freeze dex FIXME: could not find svn URL in dependency_links for this package: |
||
| Attachments: |
|
| Operating System: | Linux |
| Participants: |
| Description |
|
I am finding data like the data shown below in my system.profile collection for various databases. I've tried disabling profiling, clearing the collection, and re-enabling, at which point the errors occur again after a few minutes. This is affecting our ability to analyze the profile collection,, and can't be a good thing. I created this bug with the creator of dex and he suggested I open a bug with the mongo team: https://github.com/mongolab/dex/issues/4
|
| Comments |
| Comment by Eric Sedor [ 13/Dec/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Ah great! Thank you so much for the pointer. Cheers! | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Eliot Horowitz (Inactive) [ 13/Dec/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sorry for not marking, but we found the root issue: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Eric Sedor [ 13/Dec/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Tad, It's slow-going on this end, too, especially without having reproduced directly. We had a third report of this issue and the user was able to provide the db.stats and validate(true) output. Hopefully this helps! Sincerely, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Tad Marshall [ 09/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Kyle and Eric, Sorry for the delay in following up with you. The validate(true) output says that you have 88 invalid documents and that the details can be found in the log. This is a little clumsy, but validate(true) doesn't have an easy way to capture the assertions that are generated by trying to read the invalid documents, so it just counts the events and refers you to the log to get the details. Can you repeat the validate(true) and post the results along with the fragment of the mongod log file that shows the errors generated? Could you also post the results of "db.system.profile.stats()"? It might also be helpful to get a mongodump of the profiles collection that is giving you difficulty or, if that doesn't work, a copy of the database file itself. We can start with the log of the validate(true) and see if that is enough to identify the problem. How large is the database that you are profiling? If you can post the output of db.stats(), should contain the relevant information. Tad | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Eric Sedor [ 09/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Tad, I developed Dex so I wanted to pop in and offer a stacktrace that was provided to me for this issue. Traceback (most recent call last): The operative code in Dex is very basic, here it is, abbreviated but nearly verbatim: profile_entries = connection[database]['system.profile'].find() Full code is at https://github.com/mongolab/dex/ Thanks! | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kyle Gato [ 21/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dex just analyzes the bson data, so when non bson data is returned it does not know how to understand it. No map reduces or eval's were run against the profile collection. This type of data reoccurs after clearing out the profile collection within minutes. Here is the validate of the profile collection:
Here is a full validate:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Tad Marshall [ 21/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Can you post the output of running validate on this collection (in the mongo shell)? See http://docs.mongodb.org/manual/reference/commands/#validate . Your stack trace shows several SpiderMonkey (JavaScript) functions; were you running map-reduce or eval against the profile collection? Sorry, I'm not familiar with Dex. Do you know what it is doing when it gets these errors? |