[SERVER-35244] Dropping config db in 4.0.0-rc0 fails while it passes in 3.7.9 and earlier Created: 25/May/18  Updated: 29/Oct/23  Resolved: 07/Aug/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.0.0-rc0
Fix Version/s: 4.0.2, 4.1.1

Type: Bug Priority: Major - P3
Reporter: Ankur Srivastava (Inactive) Assignee: Sarah Zhou
Resolution: Fixed Votes: 0
Labels: neweng
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0
Sprint: Sharding 2018-07-16, Sharding 2018-07-30, Sharding 2018-08-13
Participants:

 Description   

trying to drop the config database in mongodb 4.0.0-rc0 fails with following exception:

com.mongodb.MongoCommandException: Command failed with error 40528: 'Direct writes against config.transactions cannot be performed using a transaction or on a session.' on server 127.0.0.1:27500. The full response is { "ok" : 0.0, "errmsg" : "Direct writes against config.transactions cannot be performed using a transaction or on a session.", "code" : 40528, "codeName" : "Location40528" }
 	at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:164)
 	at com.mongodb.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:295)
 	at com.mongodb.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:255)
 	at com.mongodb.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:98)
 	at com.mongodb.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:441)
 	at com.mongodb.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:80)
 	at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:189)
 	at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:264)
 	at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:126)
 	at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:118)
 	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:226)
 	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:217)
 	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:154)
 	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:147)
 	at com.mongodb.operation.DropDatabaseOperation$1.call(DropDatabaseOperation.java:89)
 	at com.mongodb.operation.DropDatabaseOperation$1.call(DropDatabaseOperation.java:86)
 	at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:462)
 	at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:424)
 	at com.mongodb.operation.DropDatabaseOperation.execute(DropDatabaseOperation.java:86)
 	at com.mongodb.operation.DropDatabaseOperation.execute(DropDatabaseOperation.java:45)
 	at com.mongodb.Mongo$3.execute(Mongo.java:837)
 	at com.mongodb.Mongo$3.execute(Mongo.java:818)
 	at com.mongodb.DB.dropDatabase(DB.java:227)



 Comments   
Comment by Githook User [ 08/Aug/18 ]

Author:

{'name': 'Sarah Zhou', 'email': 'sarah.zhou@10gen.com'}

Message: SERVER-35244 Dropping config db in 4.0.0-rc0 fails while it passes in 3.7.9 and earlier

(cherry picked from commit d6e7f0c811acabd3405871d0e09f3d2d84fe9c1c)
Branch: v4.0
https://github.com/mongodb/mongo/commit/e9bbce42db00b374b2fef2b7f2de68ff9c2dbd48

Comment by Githook User [ 07/Aug/18 ]

Author:

{'name': 'Sarah Zhou', 'email': 'sarah.zhou@10gen.com'}

Message: SERVER-35244 Dropping config db in 4.0.0-rc0 fails while it passes in 3.7.9 and earlier
Branch: master
https://github.com/mongodb/mongo/commit/d6e7f0c811acabd3405871d0e09f3d2d84fe9c1c

Comment by Ankur Srivastava (Inactive) [ 31/May/18 ]

kaloian.manassiev We are hitting it now because, mongodb 4.0.0-rc0 creates config db by default on startup. This was not the case in previous versions. We have been using 3.6 driver since march.

Backup tries to delete the config db as part of backup process. Since earlier versions did not create the config db on startup, we did not try to delete it and did not encounter the issue. 

Comment by Kaloian Manassiev [ 31/May/18 ]

Ah yes - this was introduced in 3.6 as part of retryable writes, so that problem would exist in 3.6 as well. ankur.srivastava, are you hitting this now because you started using the 3.6 drivers, which start sessions implicitly?

Comment by Tess Avitabile (Inactive) [ 31/May/18 ]

Okay, great, I'm glad they have a workaround. Sure, that change sounds fine to me. The check is part of the session catalog maintenance, so I think it makes more sense for this work to be done by the Sharding team.

Comment by Kaloian Manassiev [ 31/May/18 ]

I spoke with Ankur yesterday and they could live without dropping the config database because dropping the individual collections still succeeds. So as far as their use case is concerned, they can work around it.

However given that they are running in standalone mode I don't think the drop should be failing. Can you just make the lsid check conditional on whether it is a replica-set or not?

Comment by Tess Avitabile (Inactive) [ 30/May/18 ]

ankur.srivastava, is it okay to close this ticket, since we've diagnosed the issue? Can you follow up separately with kaloian.manassiev about when backup should be dropping the config database?

Comment by Kaloian Manassiev [ 30/May/18 ]

While this particular error while dropping the config/admin databases seems to be something unintended, I actually prefer that it be explicitly disallowed unless the server is running in standalone mode (i.e., for some form of recovery).

There is a lot of critical state stored in the config/admin databases and there should never be a reason for a user to drop them.

Comment by Tess Avitabile (Inactive) [ 29/May/18 ]

Ah, thanks for explaining that the difference is due to the creation of the config database at startup.

I spoke with jeff.yemin, and all 3.6 drivers use "implicit sessions" even when the user does not explicitly start a session: https://github.com/mongodb/specifications/blob/master/source/sessions/driver-sessions.rst#explicit-vs-implicit-sessions. So what you are seeing is expected.

Please let me know if it's important to be able to drop the config database in your tests, and we can discuss possible workarounds.

Comment by Ankur Srivastava (Inactive) [ 29/May/18 ]

I noticed that our tests are failing because config db is not created on fresh start of mongodb 3.7.9, but it is being created on mongodb 4.0.0-rc0. Even though sessions are not being used.

Following is the code which does not uses sessions and throws exception on mongodb 4.0.0-rc0:

import com.mongodb.MongoClient;
import com.mongodb.DB;
import java.util.List;
import java.util.ArrayList;
import java.util.Set;
 
public class Test {
  public static void main(String[] args) {
    MongoClient client = new MongoClient("localhost", Integer.parseInt(args[0]));
    apply(client);
  }
 
  public static void apply(MongoClient mongo) {
    List<String> dbNames = mongo.listDatabaseNames().into(new ArrayList<>());
 
    for (String dbName : dbNames) {
      DB db = mongo.getDB(dbName);
      Set<String> collectionNames = db.getCollectionNames();
 
      for (String collectionName : collectionNames) {
        // Can't drop system collections.
        if (collectionName.indexOf("system.") == 0) {
          continue;
        }
        db.getCollection(collectionName).drop();
      }
 
      if (!"admin".equals(dbName)) {
        System.out.println("Dropping db: " + dbName);
        db.dropDatabase();
        System.out.println("Dropped db: " + dbName);
      }
    }
  }
}

 

output with 3.7.9:

 

May 29, 2018 10:52:27 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Cluster created with settings {hosts=[localhost:32000], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
May 29, 2018 10:52:27 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Cluster description not yet available. Waiting for 30000 ms before timing out
May 29, 2018 10:52:27 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Opened connection [connectionId{localValue:1, serverValue:3}] to localhost:32000
May 29, 2018 10:52:27 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Monitor thread successfully connected to server with description ServerDescription{address=localhost:32000, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 7, 9]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1721083}
May 29, 2018 10:52:27 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Opened connection [connectionId{localValue:2, serverValue:4}] to localhost:32000

output with 4.0.0-rc0

 

 

May 29, 2018 10:48:21 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Cluster created with settings {hosts=[localhost:31000], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
May 29, 2018 10:48:21 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Cluster description not yet available. Waiting for 30000 ms before timing out
May 29, 2018 10:48:21 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Opened connection [connectionId{localValue:1, serverValue:1}] to localhost:31000
May 29, 2018 10:48:21 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Monitor thread successfully connected to server with description ServerDescription{address=localhost:31000, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 0]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1694757}
May 29, 2018 10:48:21 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Opened connection [connectionId{localValue:2, serverValue:2}] to localhost:31000
Dropping db: config
Exception in thread "main" com.mongodb.MongoCommandException: Command failed with error 40528: 'Direct writes against config.transactions cannot be performed using a transaction or on a session.' on server localhost:31000. The full response is { "ok" : 0.0, "errmsg" : "Direct writes against config.transactions cannot be performed using a transaction or on a session.", "code" : 40528, "codeName" : "Location40528" }
	at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:164)
	at com.mongodb.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:295)
	at com.mongodb.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:255)
	at com.mongodb.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:98)
	at com.mongodb.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:441)
	at com.mongodb.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:80)
	at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:189)
	at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:264)
	at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:126)
	at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:118)
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:226)
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:217)
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:154)
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:147)
	at com.mongodb.operation.DropDatabaseOperation$1.call(DropDatabaseOperation.java:89)
	at com.mongodb.operation.DropDatabaseOperation$1.call(DropDatabaseOperation.java:86)
	at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:462)
	at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:424)
	at com.mongodb.operation.DropDatabaseOperation.execute(DropDatabaseOperation.java:86)
	at com.mongodb.operation.DropDatabaseOperation.execute(DropDatabaseOperation.java:45)
	at com.mongodb.Mongo$3.execute(Mongo.java:837)
	at com.mongodb.Mongo$3.execute(Mongo.java:818)
	at com.mongodb.DB.dropDatabase(DB.java:227)
	at Test.apply(Test.java:29)
	at Test.main(Test.java:10)

 

 

Comment by Tess Avitabile (Inactive) [ 29/May/18 ]

That error message would only be generated if the dropDatabase is executed in a session. I tested that it is possible to drop the config database outside of a session in 4.0.0-rc0. Can you please provide a repro script that demonstrates the change in behavior between 3.7.9 and 4.0.0-rc0?

Comment by Ankur Srivastava (Inactive) [ 29/May/18 ]

dropDatabase is not executed in a session. We are using java driver 3.6.3. Same code runs fine for mongodb 3.6, 3.7.9 but fails on 4.0.0-rc0.

Comment by Tess Avitabile (Inactive) [ 25/May/18 ]

Is it possible that the dropDatabase is now executed in a session, when it was not before? Possibly due to a change in the driver? It is illegal in both 3.6 and 4.0 to drop the config database inside of a session. That is, the following example fails on both 3.6 and 4.0:

 (function() {
    const rst = ReplSetTest({nodes: 1});
    rst.startSet();
    rst.initiate();    
    const session = rst.getPrimary().getDB("test").getMongo().startSession({retryWrites: true});
    const testDB = session.getDatabase("test");
    const configDB = session.getDatabase("config");
    assert.writeOK(testDB.c.insert({}));
 
    // Fails with "Direct writes against config.transactions cannot be performed using a  
    // transaction or on a session."
    assert.commandWorked(configDB.dropDatabase());
 
    session.endSession();    
    rst.stopSet();
}());

CC spencer, kaloian.manassiev

Generated at Thu Feb 08 04:39:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.