[SERVER-12844] unable to build background index on secondary Created: 23/Feb/14  Updated: 10/Dec/14  Resolved: 25/Feb/14

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Replication
Affects Version/s: 2.6.0-rc0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Daniel Pasette (Inactive) Assignee: Eric Milkie
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File SERVER-12844.js    
Issue Links:
Related
Operating System: ALL
Steps To Reproduce:
  • start 2 node repl set
  • insert 1000 docs form {x:1}
  • build index on

     ({a:1}, {background:true}) 

Participants:

 Description   

> col.ensureIndex({a:0}, {background: true});
 m31001| 2014-02-22T23:31:26.171-0800 [repl writer worker 1] build index on: server12844.server12844 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "server12844.server12844" }
 m31001| 2014-02-22T23:31:26.171-0800 [repl writer worker 1] 	 added index to empty collection
 m31000| 2014-02-22T23:31:26.176-0800 [conn1] build index on: server12844.server12844 properties: { v: 1, key: { a: 0.0 }, name: "a_0", ns: "server12844.server12844", background: true }
 m31000| 2014-02-22T23:31:26.176-0800 [conn1] 	 building index in background
 m31000| 2014-02-22T23:31:26.180-0800 [conn1] build index done.  scanned 1000 total records. 0.004 secs
> assert.eq(2, col.getIndexes().length);
> 
> sleep(5000);
 m31001| 2014-02-22T23:31:26.218-0800 [repl writer worker 1] build index on: server12844.server12844 properties: { v: 1, key: { a: 0.0 }, name: "a_0", ns: "server12844.server12844", background: true }
 m31001| 2014-02-22T23:31:26.218-0800 [repl writer worker 1] index build failed. spec: { v: 1, key: { a: 0.0 }, name: "a_0", ns: "server12844.server12844", background: true } error: 13130 can't start bg index b/c in recursive lock (db.eval?)



 Comments   
Comment by Eric Milkie [ 21/Apr/14 ]

The MongoDB shell's ensureIndex shell helper will use the system.indexes insertion method if the shell's writeMode is "compatibility". You can enable this write mode by starting the shell with a command line parameter:

./mongo --writeMode=compatibility

Comment by Kent Sibilev [ 21/Apr/14 ]

Thanks for the info. That's great news. Could you please provide a pointer on how to use system.indexes instead? Thanks.

Comment by Eric Milkie [ 21/Apr/14 ]

Hi Kent. I believe your problem is SERVER-13589, which is fixed in 2.6.1. One workaround is to insert index specs directly into system.indexes rather than using the createIndexes command.

Comment by Kent Sibilev [ 21/Apr/14 ]

We've just upgraded to 2.6.0 and run into this issue. A new index is created on master but on the secondaries we get the following in the log files:

2014-04-16T20:56:53.639+0000 [repl writer worker 2] build index on: xxx.messages properties: { v: 1, key:

{ fname: 1.0 }

, name: "fname_1", ns: "xxx.messages", background: true }
2014-04-16T20:56:53.639+0000 [repl writer worker 2] index build failed. spec: { v: 1, key:

{ fname: 1.0 }

, name: "fname_1", ns: "xxx.messages", background: true } error: 13130 can't start bg index b/c in recursive lock (db.eval?)

Any solutions?

Comment by Daniel Pasette (Inactive) [ 25/Feb/14 ]

can no longer repro this issue with new build of mongo server and shell. resolving

Comment by Eric Milkie [ 24/Feb/14 ]

I also don't understand the output in the description. The index build should have been done by a spawned thread [repl index builder 0], but instead it is being done inline using one of the replication threads (so of course it fails). I can't think of why this would happen unless you were doing something like applyOps command.

Comment by Eric Milkie [ 24/Feb/14 ]

(I tried running the test with smoke.py --auth as well, and it still passes for me)

Comment by Eric Milkie [ 24/Feb/14 ]

The attached .js test passes for me. There must be a race so I'll look into that next.

Generated at Thu Feb 08 03:29:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.