[SERVER-33089] Unable to start queryable mongod because it failed to regenerate index for admin.system.users Created: 02/Feb/18  Updated: 29/Oct/23  Resolved: 05/Feb/18

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Security
Affects Version/s: 3.4.11
Fix Version/s: 3.4.12, 3.6.3, 3.7.2

Type: Bug Priority: Critical - P2
Reporter: Ankur Srivastava (Inactive) Assignee: Spencer Jackson
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Problem/Incident
Related
is related to SERVER-29259 Authentication/Authorization indexes ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v3.6, v3.4
Sprint: Platforms 2018-02-12
Participants:

 Description   

Unable to start queryable mongod for the backup of mongod version 3.4.10.

indexes on admin.system.user on mongod 3.4.10:

db.system.users.getIndexes()
[
	{
		"v" : 2,
		"key" : {
			"_id" : 1
		},
		"name" : "_id_",
		"ns" : "admin.system.users"
	}
]

indexes on admin.system.user on mongod 3.4.11:

db.system.users.getIndexes()
[
	{
		"v" : 2,
		"key" : {
			"_id" : 1
		},
		"name" : "_id_",
		"ns" : "admin.system.users"
	},
	{
		"v" : 2,
		"unique" : true,
		"key" : {
			"user" : 1,
			"db" : 1
		},
		"name" : "user_1_db_1",
		"ns" : "admin.system.users"
	}
]

Queryable restore uses the latest mongod available to start queryable mongod. So for the backup of mongod 3.4.10 it will use queryable mongod 3.4.11. While starting queryable mongod, it tries to create the extra index introduced in mongod 3.4.11, and it fails because queryable mongod is started in read only mode. Following is the error encountered:

2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] MongoDB starting : pid=20564 port=27812 dbpath=/srv/10gen/bgrid-prod/head01/queryable/5a741effdf9db17ed7c5b844/dbpath/ 64-bit host=brs-qr-2.vpc8.10gen.cc
2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] db version v3.4.11
2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] git version: 34f5bec2c9d827d71828fe858167f89a28b29a2a
2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] modules: enterprise
2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] build environment:
2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten]     distmod: rhel62
2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten]     distarch: x86_64
2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2018-02-02T08:19:12.240+0000 I CONTROL  [initandlisten] options: { config: "/srv/10gen/bgrid-prod/head01/queryable/5a741effdf9db17ed7c5b844/dbpath/conf.yaml", net: { bindIp: "0.0.0.0", port: 27812, ssl: { CAFile: "/srv/10gen/bgrid-prod/head01/queryable/5a741effdf9db17ed7c5b844/dbpath/ca.pem", PEMKeyFile: "/srv/10gen/bgrid-prod/head01/queryable/5a741effdf9db17ed7c5b844/dbpath/serverIdentity.pem", mode: "requireSSL" } }, queryableBackup: { apiUri: "127.0.0.1:8095", memoryQuotaMB: 8192.0, snapshotId: "5a741effdf9db17ed7c5b844" }, security: { authorization: "enabled" }, setParameter: { authenticationMechanisms: "MONGODB-X509" }, storage: { dbPath: "/srv/10gen/bgrid-prod/head01/queryable/5a741effdf9db17ed7c5b844/dbpath/", engine: "queryable_wt", queryableBackupMode: true, wiredTiger: { engineConfig: { cacheSizeGB: 6.0 } } }, systemLog: { destination: "file", path: "/srv/10gen/bgrid-prod/head01/queryable/5a741effdf9db17ed7c5b844/mongod.log" } }
2018-02-02T08:19:12.258+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=6144M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),extensions=[local={entry=queryableWtFsCreate,early_load=true,config={apiUri="127.0.0.1:8095",snapshotId="5a741effdf9db17ed7c5b844",dbpath="/srv/10gen/bgrid-prod/head01/queryable/5a741effdf9db17ed7c5b844/dbpath"}},],readonly=true,
2018-02-02T08:19:18.792+0000 W STORAGE  [initandlisten] Detected configuration for non-active storage engine wiredTiger when current storage engine is queryable_wt
2018-02-02T08:19:18.955+0000 I ACCESS   [initandlisten] No authorization index detected on admin.system.users collection. Attempting to recover by creating an index with spec: { ns: "admin.system.users", v: 2, name: "user_1_db_1", key: { user: 1, db: 1 }, unique: true }
2018-02-02T08:19:18.955+0000 F ACCESS   [initandlisten] Failed to regenerate index for admin.system.users. Exception: Cannot execute a write operation in read-only mode
2018-02-02T08:19:18.955+0000 I STORAGE  [initandlisten] IllegalOperation: Cannot execute a write operation in read-only mode



 Comments   
Comment by Githook User [ 05/Feb/18 ]

Author:

{'email': 'spencer.jackson@mongodb.com', 'name': 'Spencer Jackson', 'username': 'spencerjackson'}

Message: SERVER-33089 Do not regenerate auth indexes in queryable backup mode

(cherry picked from commit ce5691d634b9228b0686e6241e071f6691e46a3f)
Branch: v3.6
https://github.com/mongodb/mongo/commit/855320ba664efb2f80123ef66cc78617bb34dfc1

Comment by Githook User [ 05/Feb/18 ]

Author:

{'email': 'spencer.jackson@mongodb.com', 'name': 'Spencer Jackson', 'username': 'spencerjackson'}

Message: SERVER-33089 Do not regenerate auth indexes in queryable backup mode

(cherry picked from commit ce5691d634b9228b0686e6241e071f6691e46a3f)
Branch: v3.4
https://github.com/mongodb/mongo/commit/bfde702b19c1baad532ed183a871c12630c1bbba

Comment by Githook User [ 05/Feb/18 ]

Author:

{'email': 'spencer.jackson@mongodb.com', 'name': 'Spencer Jackson', 'username': 'spencerjackson'}

Message: SERVER-33089 Do not regenerate auth indexes in queryable backup mode
Branch: master
https://github.com/mongodb/mongo/commit/ce5691d634b9228b0686e6241e071f6691e46a3f

Comment by Spencer Jackson [ 02/Feb/18 ]

I can confirm that the server will start with missing indexes with --queryableBackupMode if index generation is bypassed in readOnly mode.

Comment by Spencer Jackson [ 02/Feb/18 ]

Agreed. There was a bug where old 3.4 servers failed to generate auth indexes. New servers would regenerate them on startup, if they were missing. It looks like the backup came from such an old server. New servers probably shouldn't regenerate the indexes, if the server is in read only mode.

Comment by Bruce Lucas (Inactive) [ 02/Feb/18 ]

Looks like this is probably a side-effect of SERVER-29259.

Generated at Thu Feb 08 04:32:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.