[CXX-283] MongoDB exception 17234 Created: 14/Jul/14  Updated: 11/Sep/19  Resolved: 11/Sep/14

Status: Closed
Project: C++ Driver
Component/s: None
Affects Version/s: legacy-0.0-26compat-2.6.0
Fix Version/s: legacy-1.0.0-rc1

Type: Task Priority: Critical - P2
Reporter: Rex Conn Assignee: Unassigned
Resolution: Done Votes: 0
Labels: legacy-cxx
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows (2003, 2008, 2012)


Issue Links:
Related
is related to CXX-303 mongo::client::initialize() behaviour... Closed

 Description   

Our installer is installing Mongo 2.6.1, then updating the database with some initial values. On a few systems, we sometimes get a MongoDB exception:

MongoDB Exception: 17234 backgroundJob already running: PeriodicTaskRunner

When those systems get that error, they can never connect to Mongo (waiting & retrying repeatedly just returns the same error).

We can't reproduce this on any of the development systems. Any ideas on what might cause it?



 Comments   
Comment by Githook User [ 11/Sep/14 ]

Author:

{u'username': u'samantharitter', u'name': u'Samantha Ritter', u'email': u'samantha.ritter@10gen.com'}

Message: CXX-283 Clarify documentation for startRunningPeriodicTasks
Branch: legacy
https://github.com/mongodb/mongo-cxx-driver/commit/6a6435990175ff06c77c99fe1ba83614ae94b18d

Comment by Andrew Morrow (Inactive) [ 15/Jul/14 ]

Hi Rex -

That makes sense. I'm happy to hear you were able to find the issue and confirm a fix.

I agree that calling mongo::client::initialize multiple times should have defined behavior, and it clearly shouldn't render the driver unusable in non-obvious ways. We would need to think about what the right behavior ought to be though, and, most likely, such a fix would only be applied on the legacy branch and not backported to 26compat.

Do you mind if I leave this ticket open as a placeholder while we consider options for mongo::client::initialize? If you would prefer to close this since the root issue is sorted out, feel free to file an enhancement ticket in the CXX project requesting stronger behavior guarantees re mongo::client::initialize.

Comment by Rex Conn [ 15/Jul/14 ]

Hi Andrew:

I went back and looked through our code, and we were making a single call to client::initialize in the connection code. However, the developer who did the db initialization code found that on slow systems (overloaded VM's in this case) MongoDB hadn't finished initializing when the first connection was attempted. He added a loop to attempt to reconnect every few seconds, which resulted in client::initialize being called again. Once that happened, we could never connect to Mongo. (Is that WAD?)

I moved the client::initialize call from the connect method to the constructor, and QA hasn't been able to reproduce the problem since.

Thanks for your help.

Rex

Comment by Andrew Morrow (Inactive) [ 15/Jul/14 ]

There should be no direct calls to mongo::client::initialize in the driver code (barring tests/examples). A few comments:

  • You mention that once this issue has arisen, subsequent connection attempts continue to return the same error. This suggests that the code path leading to establishing a connection is somehow leading (perhaps indirectly) to a call to startRunningPeriodTasks.
  • Do you use any sort of RAII pattern with a global static to manage invoking mongo::client::initialize and mongo::client::terminate? Is there any chance that this global static is in a DLL that gets loaded/unloaded?

You stated that this is only happening on production systems, so that limits the diagnostic angles to some degree. Ideally if you had a dev system repro we could just add log messages / breakpoints in mongo::client::initialize, PeriodicTask::startRunningPeriodicTasks, and friends and it would be easier to see if somehow these calls were getting repeated.

Also, if you could provide information on how you built the client driver that would be useful as well.

Thanks,
Andrew

Comment by Rex Conn [ 15/Jul/14 ]

We're not knowingly calling startRunningPeriodicTasks multiple times – we don't have any explicit calls to it. We do call client::initialize, but only once before connecting to Mongo. (Is there another implicit call to client::initialize somewhere in the driver code?)

Thanks.

Rex

Comment by Andrew Morrow (Inactive) [ 15/Jul/14 ]

Hi rconn

Is there any possibility that the function mongo::PeriodicTask::startRunningPeriodicTasks is being called more than once during the lifetime of your application? Keep in mind that mongo::client::initialize invokes PeriodicTask::startRunningPeriodicTasks once itself, so if the application is calling mongo::client::initialize, that already counts as one invocation of startRunningPeriodicTasks.

Thanks,
Andrew

Generated at Wed Feb 07 21:58:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.