[SERVER-46092] Broken index after mongodb version upgrade to 3.6 Created: 12/Feb/20 Updated: 14/Feb/20 Resolved: 14/Feb/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 3.6.12 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Jackie Chu | Assignee: | Eric Sedor |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | index, multikeyIndex | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Heroku with mlab add-on. |
||
| Issue Links: |
|
||||||||
| Participants: | |||||||||
| Description |
|
Problem After upgrading mongodb from v3.4.20 -> v3.6.12 (on mlab add-on for heroku), we have sudden dramatic performance regression on the mongodb. CPU usage went from <10% to 99%, Disk IOPS significantly increased and all burst credits are used up within minutes, which brought down our service (application on heroku). Investigation Our investigation found that the existing index (multi-key) seems to be broken/corrupted after the mongodb upgrade, which caused inefficient query execution plan to be generated & huge # of document examined for all queries. Resolution / Workaround We dropped the index and re-created the same index. The issue seems to be resolved afterward. Question
Collection: "runs"
Index:
Note: "tags" = array of text (huge amount of distant values)
Query:
Execution Plan (problematic) 12 mins - 1M keys examined & 1M docs examined.
Execution Plan (After re-creating same index) 5 ms - 100 keys examined & 100 docs examined. And I noticed that multiKeyPaths is present in this executionStats, but not in the problematic one above.
|
| Comments |
| Comment by Jackie Chu [ 14/Feb/20 ] |
|
Great. Thank you for your promptly response, Eric. |
| Comment by Eric Sedor [ 14/Feb/20 ] |
|
Of course Jackie, Some features of v:2 indexes are indicated here, and you can check index version by index in the output of db.collection.getIndexes() in the shell. The SERVER project is for reporting bugs and feature requests, so for further guidance I must encourage you to reach out to support@mlab.com, or to ask our community by posting on the mongodb-user group or on Stack Overflow with the mongodb tag. Sincerely, |
| Comment by Jackie Chu [ 14/Feb/20 ] |
|
Hi Eric, Thank you for looking into it and it's great for us to learn the cause of that.
Thanks, Jackie |
| Comment by Eric Sedor [ 13/Feb/20 ] |
|
The behavior you are seeing is related to It's worth noting that you can email support@mlab.com for questions about issues with mLab deployments. Sincerely, |