[SERVER-36944] applyOps does not permit unknown field names when creating a v:1 index Created: 30/Aug/18  Updated: 29/Oct/23  Resolved: 25/Sep/18

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Replication
Affects Version/s: 3.4.14, 3.6.7, 4.0.2
Fix Version/s: 3.4.18, 3.6.9, 4.0.3, 4.1.3

Type: Bug Priority: Major - P3
Reporter: David Storch Assignee: Benety Goh
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
related to SERVER-37333 [3.4] applyOps does not validate back... Closed
is related to SERVER-28594 non-atomic applyOps should log each i... Closed
is related to SERVER-31161 Index created through applyOps comman... Closed
is related to SERVER-769 Validate top-level & index spec field... Closed
is related to SERVER-36015 Remove references to system.namespace... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0, v3.6, v3.4
Steps To Reproduce:

MongoDB Enterprise > db.adminCommand({applyOps: [{op:"i", ns: "test.system.indexes", o: {ns:"test.test", key: {a:1}, name: "a_1", v: 1, unknown: 1}}]})
{
	"applied" : 1,
	"code" : 197,
	"codeName" : "InvalidIndexSpecificationOption",
	"errmsg" : "The field 'unknown' is not valid for an index specification. Specification: { ns: \"test.test\", key: { a: 1.0 }, name: \"a_1\", v: 1.0, unknown: 1.0 }",
	"results" : [
		false
	],
	"ok" : 0
}

Sprint: Storage NYC 2018-09-24, Storage NYC 2018-10-08
Participants:

 Description   

MongoDB 3.4 introduced validation of index options (SERVER-769). This validation applies to all newly created indexes, regardless of index version. 3.4 also introduced v:2 indexes, so it is illegal for any unknown/unvalidated options to be present in the index catalog for a v:2 index spec. However, old v:1 indexes may exist with unknown options, and oplog entries for creating v:1 indexes with illegal options may exist.

Therefore, the applyOps command should ignore any unknown options when creating a v:1 index. On the other hand, applyOps should error on unknown options for v:2 indexes. This works correctly for op:"c" createIndexes oplog entries:

MongoDB Enterprise > db.adminCommand({applyOps: [{op:"c", ui: UUID("a0aff12b-0a98-429b-9f2a-892b57fcdbd5"), ns: "test.$cmd", o: {createIndexes: "c", key: {a:1}, name: "a_1", v:1, unknown: 1}}]})
{ "applied" : 1, "results" : [ true ], "ok" : 1 }
MongoDB Enterprise > db.adminCommand({applyOps: [{op:"c", ui: UUID("a0aff12b-0a98-429b-9f2a-892b57fcdbd5"), ns: "test.$cmd", o: {createIndexes: "c", key: {b:1}, name: "c_1", v:2, unknown: 1}}]})
{
	"applied" : 1,
	"code" : 197,
	"codeName" : "InvalidIndexSpecificationOption",
	"errmsg" : "The field 'unknown' is not valid for an index specification. Specification: { key: { b: 1.0 }, name: \"c_1\", v: 2.0, unknown: 1.0, ns: \"test.c\" }",
	"results" : [
		false
	],
	"ok" : 0
}

However, when the oplog entry is an insert into system.indexes, unknown options are rejected regardless of index version:

MongoDB Enterprise > db.adminCommand({applyOps: [{op:"i", ns: "test.system.indexes", o: {ns:"test.test", key: {a:1}, name: "a_1", unknown: 1, v: 1}}]})
{
	"applied" : 1,
	"code" : 197,
	"codeName" : "InvalidIndexSpecificationOption",
	"errmsg" : "The field 'unknown' is not valid for an index specification. Specification: { ns: \"test.test\", key: { a: 1.0 }, name: \"a_1\", unknown: 1.0, v: 1.0 }",
	"results" : [
		false
	],
	"ok" : 0
}
MongoDB Enterprise > db.adminCommand({applyOps: [{op:"i", ns: "test.system.indexes", o: {ns:"test.test", key: {a:1}, name: "a_1", unknown: 1, v: 2}}]})
{
	"applied" : 1,
	"code" : 197,
	"codeName" : "InvalidIndexSpecificationOption",
	"errmsg" : "The field 'unknown' is not valid for an index specification. Specification: { collation: { locale: \"simple\" }, ns: \"test.test\", key: { a: 1.0 }, name: \"a_1\", unknown: 1.0, v: 2.0 }",
	"results" : [
		false
	],
	"ok" : 0
}

This is not the expected behavior in the v:1 case. It looks like this is due to applyOps's implementation of insert into system.indexes in terms of the DBDirectClient. Support for anything related to system.indexes was recently removed as part of SERVER-36015, but this issue still affects older versions.



 Comments   
Comment by Githook User [ 25/Sep/18 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-36944 applyOps uses applyOperation_inlock() for system.indexes inserts

(cherry picked from commit bf2cfbd09d6695e2c84b295c3aa539adaf6496a8)
(cherry picked from commit 8fe372afcbad4cd67d46f993291bdc090423caa6)
Branch: v3.4
https://github.com/mongodb/mongo/commit/eb38d663ade9477810458c0b425da66aaeb8b56a

Comment by Githook User [ 25/Sep/18 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-36944 add characterization test for applyOps index creation

(cherry picked from commit 72ff3cb4ee0082e4310505fb10e32eaa133cb0a6)
(cherry picked from commit 0e65a5855d5af2f08d8fcc330adfd7a3bb31a7b4)
Branch: v3.4
https://github.com/mongodb/mongo/commit/1e4f12dc5e4c12e7656270907cf9d49715fa732f

Comment by Githook User [ 25/Sep/18 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-36944 applyOps uses applyCommand_inlock() for system.indexes inserts

(cherry picked from commit bf2cfbd09d6695e2c84b295c3aa539adaf6496a8)
Branch: v3.6
https://github.com/mongodb/mongo/commit/4f1bd3046036a8621c2ad5acde7cdf103a533226

Comment by Githook User [ 25/Sep/18 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-36944 add characterization test for applyOps index creation

(cherry picked from commit 72ff3cb4ee0082e4310505fb10e32eaa133cb0a6)
(cherry picked from commit 0e65a5855d5af2f08d8fcc330adfd7a3bb31a7b4)
Branch: v3.6
https://github.com/mongodb/mongo/commit/c3d0cfa94e423258ad5ebaef6cbe50d5c932dac9

Comment by Githook User [ 25/Sep/18 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-36944 different index name for system.indexes test case in apply_ops_create_indexes.js

(cherry picked from commit 5e46c5444dd56c4a18755cfc7b71ef3c1b44c421)
Branch: v3.6
https://github.com/mongodb/mongo/commit/617b537dbfa9b12233d48993f19af67c35fad7ce

Comment by Githook User [ 25/Sep/18 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-36944 applyOps uses applyCommand_inlock() for system.indexes inserts
Branch: v4.0
https://github.com/mongodb/mongo/commit/bf2cfbd09d6695e2c84b295c3aa539adaf6496a8

Comment by Githook User [ 25/Sep/18 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-36944 add characterization test for applyOps index creation

(cherry picked from commit 72ff3cb4ee0082e4310505fb10e32eaa133cb0a6)
Branch: v4.0
https://github.com/mongodb/mongo/commit/0e65a5855d5af2f08d8fcc330adfd7a3bb31a7b4

Comment by Githook User [ 25/Sep/18 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-36944 different index name for system.indexes test case in apply_ops_create_indexes.js
Branch: v4.0
https://github.com/mongodb/mongo/commit/5e46c5444dd56c4a18755cfc7b71ef3c1b44c421

Comment by Githook User [ 24/Sep/18 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-36944 add characterization test for applyOps index creation
Branch: master
https://github.com/mongodb/mongo/commit/72ff3cb4ee0082e4310505fb10e32eaa133cb0a6

Comment by Tess Avitabile (Inactive) [ 04/Sep/18 ]

Sending to the Storage team, since this issue was introduced by SERVER-28594.

Comment by David Storch [ 30/Aug/18 ]

In that case, my guess is that this was introduced by https://github.com/mongodb/mongo/commit/4b2f7438ec3494cdf5f4f996bc1ac955a0698f40 under SERVER-28594.

Comment by Shane Harvey [ 30/Aug/18 ]

Looks like the server's validation when inserting into system.indexes became more strict in 3.4.14. Here's 3.4.13:

MongoDB server version: 3.4.13
...
MongoDB Enterprise 8333c37e-3473-4f91-8189-c0d4b2ebecf6:PRIMARY> db.adminCommand({applyOps: [{op:"i", ns: "test.system.indexes", o: {v:1, ns:"test.test", key: {a:1}, name: "a_1", "randomOption": 1}}]})
{ "applied" : 1, "results" : [ true ], "ok" : 1 }
MongoDB Enterprise 8333c37e-3473-4f91-8189-c0d4b2ebecf6:PRIMARY> db.test.getIndexes()
[
	{
		"v" : 2,
		"key" : {
			"_id" : 1
		},
		"name" : "_id_",
		"ns" : "test.test"
	},
	{
		"v" : 1,
		"key" : {
			"a" : 1
		},
		"name" : "a_1",
		"ns" : "test.test",
		"randomOption" : 1
	}
]

And 3.4.14:

MongoDB server version: 3.4.14
...
MongoDB Enterprise 9630d9aa-8d77-4f2d-ab6c-47120eb3aa88:PRIMARY> db.adminCommand({applyOps: [{op:"i", ns: "test.system.indexes", o: {v:1, ns:"test.test", key: {a:1}, name: "a_1", "randomOption": 1}}]})
{
	"applied" : 1,
	"results" : [
		false
	],
	"ok" : 0,
	"errmsg" : "applyOps had one or more errors applying ops",
	"code" : 8,
	"codeName" : "UnknownError"
}

Generated at Thu Feb 08 04:44:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.