[SERVER-40560] Ensure system-collection protections for local and config system collections Created: 10/Apr/19 Updated: 29/Oct/23 Resolved: 28/Jan/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Security |
| Affects Version/s: | 4.0.5 |
| Fix Version/s: | 4.3.4 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | John Arbash Meinel | Assignee: | Spencer Jackson |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Backwards Compatibility: | Minor Change | ||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||
| Steps To Reproduce: | original reproductionI'll give the exact steps I used to reproduce, including the binary that I am testing against. I'm assuming the issue is not about the binary, though.
It does seem to require that both TestNewProxiedSession and TestResetWhenUnauthorizied are run. I'm guessing it is actually an issue with the test suite tear down followed by another setup. I realize this isn't a minimalist reproduction with only existing mongo tools in play, but given the bit that is failing is an invariant deep in the code, it does seem like a missed expectation. Note also that the test does not fail 100% of the time, but probably close to 90%. So there is a bit of a race involved, just one that typically fails. Note the setup of Mongodb during the test suite is being done here: https://github.com/jameinel/juju-testing/blob/d2cd4adac5bf0330ef9cea5dcc18d485c2cb2b3c/mgo.go#L221 And if there is no --replSet (and also no replSetInitiate call), then the tests pass 100% of the time. I can run other tests/try to find a smaller reproducer if that is relevant. |
||||||||||||||||||||||||||||
| Sprint: | Security 2019-05-06, Security 2019-05-20, Security 2019-06-03, Security 2019-06-17, Security 2019-07-01, Security 2019-08-12, Security 2019-08-26, Security 2019-09-09, Security 2019-09-23, Security 2019-10-07, Security 2019-10-21, Security 2019-11-04, Security 2019-12-16, Security 2019-12-30, Security 2019-01-13, Security 2019-01-27, Security 2020-02-10 | ||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Description |
| Comments |
| Comment by Githook User [ 27/Jan/20 ] | |||||||||||||||||||||||||||||||||
|
Author: {'email': 'spencer.jackson@mongodb.com', 'username': 'spencerjackson', 'name': 'Spencer Jackson'}Message: | |||||||||||||||||||||||||||||||||
| Comment by Eric Sedor [ 16/Apr/19 ] | |||||||||||||||||||||||||||||||||
|
jameinel, we are assigning this to an appropriate team to evaluate against our planned work. Updates will be posted as they happen. | |||||||||||||||||||||||||||||||||
| Comment by Eric Sedor [ 11/Apr/19 ] | |||||||||||||||||||||||||||||||||
|
Hi jameinel, thank you for your report. We especially thank you very much for the additional proactive effort you've taken to find the triggering action for the failure, and related documentation. We'll look into this. | |||||||||||||||||||||||||||||||||
| Comment by John Arbash Meinel [ 11/Apr/19 ] | |||||||||||||||||||||||||||||||||
|
So if there is a bug, it is that local.replset.minvalue isn't local.system.replset.minvalue and/or that Mongod doesn't prevent you from deleting a document that it will later invariant() that it exists. I'm happy that it is a bug in our side that we can fix, but it should be something that users can't actually do.
| |||||||||||||||||||||||||||||||||
| Comment by John Arbash Meinel [ 11/Apr/19 ] | |||||||||||||||||||||||||||||||||
|
oddly, I found it with:
With that, I was able to see that we were indeed removing the collection:
Looking here: https://docs.mongodb.com/manual/reference/local-database/ It doesn't define quite a few of those collections. "local.replset.election" "config.transaction" We already have code that explicitly skips all collections that start with "system.", is there a reason that there are now "local.replset" collections that aren't under "local.system.replset" ?
| |||||||||||||||||||||||||||||||||
| Comment by John Arbash Meinel [ 11/Apr/19 ] | |||||||||||||||||||||||||||||||||
|
more debugging today. If I launch the mongodb server with:
Then it stops crashing. (I was hoping that by setting profiling, I would see what the bad query was that causes it to fail.)
| |||||||||||||||||||||||||||||||||
| Comment by John Arbash Meinel [ 11/Apr/19 ] | |||||||||||||||||||||||||||||||||
|
I'm able to replicate the invariant failure even easier with this change:
Essentially, if I just connect immediately after calling replSetInitiate. Note that I am waiting for it to return, but maybe there is something else I should be waiting for? I haven't been able to replicate this with the mongo shell:
But I also notice that mongo shell always prints out a
so it is doing more than just connecting to mongod.
|