[SERVER-28436] Implement KeysCollectionManager Created: 22/Mar/17  Updated: 06/Dec/17  Resolved: 25/Apr/17

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 3.5.7

Type: Task Priority: Major - P3
Reporter: Misha Tyulenev Assignee: Randolph Tan
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-28435 Implement KeysCollectionCacheReader Closed
Backwards Compatibility: Fully Compatible
Sprint: Sharding 2017-05-08
Participants:

 Description   

KeysCollectionManager is installed on the service context so the LogicalClock can get the key that matches the signature generation.
Its also owns and manages KeysCollection

{Reader Updater}

Pseudo code:

KeysCollectionManager {
public:
    // for reading
    StatusWith<KeysCollectionDocument> getKeyForValidation(OperationContext* opCtx,
                                                           long long keyId,
                                                           const LogicalTime& forThisTime) {
        auto keyStatus = _getKeyWithKeyIdCheck(keyId, forThisTime);
 
        if (keyStatus != ErrorCodes::KeyNotFound) {
            return keyStatus;
        }
 
        auto refreshRequest = _requestRefresh();
        // note: waitFor waits min(maxTimeMS, deadline)
        if (!refreshRequest.waitFor(opCtx, kDeadline)) {
            // return refresh interrupted/timed out
        }
 
        return _getKeyWithKeyIdCheck(keyId, forThisTime);
    }
 
    // for signing
    StatusWith<KeysCollectionDocument> getKeyForSigning(OperationContext* opCtx,
                                                        const LogicalTime& forThisTime) {
        auto keyStatus = _getKey(forThisTime);
 
        if (keyStatus != ErrorCodes::KeyNotFound) {
            return keyStatus;
        }
 
        {
            // refresh retry logic until key.expiresAt > forThisTime or timeout
        }
 
        return _getKey(forThisTime);
    }
 
    void start(ServiceContext*);  // start bg refresher
    void shutDown();              // shutdown bg refresher
    void switchMode(Mode);        // switch between reader or updater
 
private:
    void backgroundThread() { // consider moving thread logic to separate nested class
        // all necessary setup
 
        while (true) {
            auto opCtx = makeOperationContext();
 
            std::shared_ptr<KeysCollectionCache> currentCacheImpl;
            {
                std::lock_guard<std::mutex> lock(_mutex);
                currentCacheImpl = _keysCache;
            }
 
            auto latestKey = currentCacheImpl->refresh(opCtx);
            if (!latestKey.isOK()) {
                // return if nextWakeUp returns inShutdown
                // add back-off?
                continue;
            }
 
            {
                std::lock_guard<std::mutex> lock(_mutex);
                if (_refreshRequest) {
                    _refreshRequest->notification.set();
                    _refreshRequest.reset();
                }
            }
 
            auto nextWakeup = howMuchSleepNeedFor(latestKey.expiresAt);
            // Add backoff to nextWakeup if it has a very small value in a row to avoid spinning.
 
            MONGO_IDLE_THREAD_BLOCK;
            auto status = opCtx.waitFor(nextWakeup);
            // check status for shutdown
        }
    }
 
    std::shared_ptr<Notification> _requestRefresh() {
        std::lock_guard<std::mutex> lock(_mutex);
        if (_refreshRequest) {
            return _refreshRequest;
        }
 
        _refreshRequest = stdx::make_shared<Notification>();
        return _refreshRequest;
    }
 
    StatusWith<KeysCollectionDocument> _getKeyWithKeyIdCheck(long long keyId,
                                                             const LogicalTime& forThisTime) {
        auto keyStatus = _getKey(forThisTime);
 
        if (!keyStatus.isOK()) {
            return keyStatus;
        }
 
        auto key = keyStatus.getValue();
 
        if (keyId == key.keyId) {
            return _key;
        }
 
        if (key.keyId > keyId)
            return ErrorCodes::OldKey;  // caller should decide what to do
 
        // Key not expired but keyId does not match!
        return ErrorCodes::KeyNotFound;
    }
 
    StatusWith<KeysCollectionDocument> _getKey(const LogicalTime& forThisTime) {
        std::lock_guard<std::mutex> lock(_mutex);
 
        auto key = _keysCache.getKey(forThisTime);
 
        if (key.expiresAt < forThisTime) {
            return ErrorCodes::KeyNotFound;
        }
 
        return key;
    }
 
    stdx::mutex _mutex;
    std::shared_ptr<Notification> _refreshRequest;
    std::shared_ptr<KeysCollectionCache> _keysCache;
}



 Comments   
Comment by Githook User [ 25/Apr/17 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-28436 Implement KeysCollectionManager
Branch: master
https://github.com/mongodb/mongo/commit/e7d6650a1fb119dfba667e1f3f50cd11fc09f5bc

Comment by Githook User [ 25/Apr/17 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-28436 Create a basic helper for enabling fail point in cpp test
Branch: master
https://github.com/mongodb/mongo/commit/cac5edbd6fa851beacedbede5140a358388087e9

Comment by Misha Tyulenev [ 03/Apr/17 ]

That could be composed in the updater. i.e.the reader and KeyCollectionManager remain as proposed, while the updater class may contain the reader. This will not affect the Manager as it will just switch the class that is called by the thread

Comment by Randolph Tan [ 30/Mar/17 ]

I just remembered one of the key aspects of the original design: the keys used for signing should have been read from a committed view. That makes me think that the reader should always be active and the updater is active only if it's a config primary.

Comment by Misha Tyulenev [ 30/Mar/17 ]

Thanks for the feedback. The KeyManager does not read or write itself. It owns the thread that schedules work and it implements an API that is called when the node where it runs goes from/to config primary. Going forward it may also serve as a cache.
The transition to primary that is toConfigPrimary(); method may request a read from the reader before it switches strategy to the updater.

Comment by Randolph Tan [ 30/Mar/17 ]

It sounds like the KeyManager would works only in 2 modes - updater or reader. I believe at minimum, the KeyManager of the config should read the latest local key when it transitions to primary. This is because it only refreshes the key periodically and doesn't monitor the new keys in real time.

Comment by Misha Tyulenev [ 30/Mar/17 ]

kaloian.manassiev renctan please review the proposal

Generated at Thu Feb 08 04:18:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.