[SERVER-45610] Some reads work while system is RECOVERING Created: 16/Jan/20  Updated: 29/Oct/23  Resolved: 18/Feb/20

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 4.3.4, 4.0.20, 3.6.19, 4.2.9

Type: Bug Priority: Major - P3
Reporter: Matthew Russotto Assignee: Vesselina Ratcheva (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Documented
is documented by DOCS-13417 Investigate changes in SERVER-45610: ... Closed
Duplicate
duplicates SERVER-39596 While a node is not in primary/second... Closed
Related
related to SERVER-46190 Disallow 'dbHash' for nodes in RECOVE... Closed
related to SERVER-48379 ReplSetTest should wait for secondari... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.2, v4.0, v3.6
Steps To Reproduce:

A mongod can be forced to stay in RECOVERING by setting the rsSyncApplyStop failpoint.

--setParameter enableTestCommands=1 --setParameter 'failpoint.rsSyncApplyStop=

{"mode": "alwaysOn"}
Sprint: Repl 2020-02-10, Repl 2020-02-24
Participants:
Case:
Linked BF Score: 32

 Description   

While the replication system is in RECOVERING, we cannot read collection data (error code NotMasterOrSecondary). However, provided we have allowed secondary reads, we CAN do a listDatabases or listCollections or listIndexes. This seems bad, since the results of these reads may represent database states which are inconsistent or at a point in time prior to a previously reported lastDurable time of the node.



 Comments   
Comment by Githook User [ 26/Jun/20 ]

Author:

{'name': 'Vesselina Ratcheva', 'email': 'vesselina.ratcheva@10gen.com', 'username': 'vessy-mongodb'}

Message: SERVER-45610 Reject commands that read data when node is in RECOVERING state

create mode 100644 jstests/libs/all_commands_test.js
create mode 100644 jstests/replsets/db_reads_while_recovering_all_commands.js

(cherry picked from commit 34dc015fcb40b8e4c2c99aadf1a78d7b64de6146)
Branch: v4.2
https://github.com/mongodb/mongo/commit/21184efa95bdb654726b9ad516621965d2581976

Comment by Githook User [ 26/Jun/20 ]

Author:

{'name': 'Vesselina Ratcheva', 'email': 'vesselina.ratcheva@10gen.com', 'username': 'vessy-mongodb'}

Message: SERVER-45610 Reject commands that read data when node is in RECOVERING state

create mode 100644 jstests/libs/all_commands_test.js
create mode 100644 jstests/replsets/db_reads_while_recovering_all_commands.js

(cherry picked from commit 34dc015fcb40b8e4c2c99aadf1a78d7b64de6146)
Branch: v3.6
https://github.com/mongodb/mongo/commit/003b9b85173cf9aee4ecf4bbd45731dcd2d5c6f4

Comment by Githook User [ 26/Jun/20 ]

Author:

{'name': 'Vesselina Ratcheva', 'email': 'vesselina.ratcheva@10gen.com', 'username': 'vessy-mongodb'}

Message: SERVER-45610 Reject commands that read data when node is in RECOVERING state

create mode 100644 jstests/libs/all_commands_test.js
create mode 100644 jstests/replsets/db_reads_while_recovering_all_commands.js

(cherry picked from commit 34dc015fcb40b8e4c2c99aadf1a78d7b64de6146)
Branch: v4.0
https://github.com/mongodb/mongo/commit/8b66a4536d44e025b3d81c49befc2e794282af6b

Comment by Evin Roesle [ 01/Jun/20 ]

No problem! I would like to backport because, while this is not extremely common, when it does happen it can create additional pain during initial sync as described in SERVER-39596 which is already a process that can be very painful for users. This is especially painful for users with a lot of data who usually experience the most pain with initial sync anyways. This can also cause possible data corruption by reading data from a node in RECOVERING which is important to prevent.

Comment by Tess Avitabile (Inactive) [ 01/Jun/20 ]

Ok! Would you mind giving your reasoning, so that we have it for future reference?

Comment by Evin Roesle [ 01/Jun/20 ]

I would like to backport this all the way to version 3.6.

Comment by Tess Avitabile (Inactive) [ 27/May/20 ]

Ping evin.roesle.

Comment by Tess Avitabile (Inactive) [ 13/May/20 ]

This change looks complex to backport. evin.roesle, can you weigh in on the priority of backporting this bug fix?

Comment by Gregory Wlodarek [ 06/May/20 ]

evin.roesle, vesselina.ratcheva: I'd like us to consider backporting this work (at least the dbStats/collStats changes made in dbcommands.cpp) to v3.6, v4.0 and v4.2. This would resolve the issues described in SERVER-39596 that customers were hitting when running the dbStats/collStats command while the node was in a recovering state.

Comment by Githook User [ 18/Feb/20 ]

Author:

{'username': 'vessy-mongodb', 'name': 'Vesselina Ratcheva', 'email': 'vesselina.ratcheva@10gen.com'}

Message: SERVER-45610 Reject commands that read data when node is in RECOVERING state

This reverts commit 542de84ec1e17520bd0d99d54a024ff0e0bc3de2.

create mode 100644 jstests/libs/all_commands_test.js
create mode 100644 jstests/replsets/db_reads_while_recovering_all_commands.js
Branch: master
https://github.com/mongodb/mongo/commit/34dc015fcb40b8e4c2c99aadf1a78d7b64de6146

Comment by Githook User [ 14/Feb/20 ]

Author:

{'username': 'puppyofkosh', 'name': 'Ian Boros', 'email': 'ian.boros@mongodb.com'}

Message: Revert "SERVER-45610 Reject commands that read data when node is in RECOVERING state"

This reverts commit ea7294598dc4621245739da201c0aeaf11aaf957.
Branch: master
https://github.com/mongodb/mongo/commit/542de84ec1e17520bd0d99d54a024ff0e0bc3de2

Comment by Githook User [ 13/Feb/20 ]

Author:

{'name': 'Vesselina Ratcheva', 'username': 'vessy-mongodb', 'email': 'vesselina.ratcheva@10gen.com'}

Message: SERVER-45610 Reject commands that read data when node is in RECOVERING state

create mode 100644 jstests/libs/all_commands_test.js
create mode 100644 jstests/replsets/db_reads_while_recovering_all_commands.js
Branch: master
https://github.com/mongodb/mongo/commit/ea7294598dc4621245739da201c0aeaf11aaf957

Comment by Judah Schvimer [ 22/Jan/20 ]

I wonder if we could do something like view_all_commands.js to ensure we fix all commands.

Comment by Judah Schvimer [ 22/Jan/20 ]

The replSetMaintenance command can also be used to go into RECOVERING.

Generated at Thu Feb 08 05:09:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.