[SERVER-17794] Indexing progress status Created: 30/Mar/15 Updated: 14/Apr/16 Resolved: 01/Apr/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Index Maintenance |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Question | Priority: | Minor - P4 |
| Reporter: | vadyalex | Assignee: | Ramon Fernandez Marina |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Participants: |
| Description |
|
I am facing such problem. I do nightly aggregation jobs that are creating new data and push it to production replica set. To achieve that I push new data to new collection, say stage_my_collection, run indexing in background (to not disturb whole replicaset cluster since it is fully operational) as soon as indexing is done I do rename-drop to make collection available for our services: db.stage_my_collection.renameCollection( "my_collection", true ) The services only read data from target collection but they read it from primary as well as slaves. The problem is I do not have good mechanism to track indexing progress. db.collection.getIndexes() returns indexes as available even though indexing process is in progress. To track progress I made a simple program that connects to each node in replicaset and checks current operation. But the information was misleading. * on admin database. That provides comprehensive information. So far so good. But there is a problem.
But I do have fail signal if any:
Is there a better way for me to track indexing progress for each index? |
| Comments |
| Comment by Ramon Fernandez Marina [ 01/Apr/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
For the progress one can use db.currentOp(), even when there are multiple indexes being built in the background. I created a collection with 20M documents of this shape:
I then started two index builds in the background, and this is what I see:
Index build startup times and failures will be recorded in the logs, so while it may be inconvenient to look in two places I think all the information is there. There's currently no mechanism that centralizes this information for a replica set or a sharded cluster – one must check on each and every node. Please note that system.indexes is an internal system collection, and any changes to it by users may trigger unexpected behavior so I highly recommend you don't use it directly. Since this report is not about a bug in the MongoDB server I'm going to resolve this ticket. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. Regards, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by vadyalex [ 31/Mar/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I have been digging through documentation thinking a bit about my root problem (synchronizing indexing completion across whole replica set) I found an interesting observation: one can create index by do insert of the index description document into collection.system.indexes collection; if insert operation has WriteConsern(3), corresponding to the number of members in the replicaset, program thread will be blocked until all three instances acknowledge indexing process completion; This is perfect. But the indexing process goes in foreground which blocks all other operation on the whole replica set during indexing. Is it by design? Or is it kind of bug? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by vadyalex [ 30/Mar/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Ramon Fernandez, If I understand correctly since MongoDB 2.6 it is possible to build several indexes in the background then how to differentiate which progress corresponds to which progress status? When I meant misleading I probably mean that it is not suitable to track whether indexing process is complete or failed. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 30/Mar/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
vadyalex, the index name is logged at the beginning and end of the operation; for example:
Also db.currentOp() does show progress information:
When you say above that "the information was misleading", what issues were you encountering? |