Index creation on 8.0+ occasionally hangs

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Won't Fix
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Execution
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      During asynchronous tests of the Python driver, we occasionally see createIndexes commands hang indefinitely until they reach our 60 second test timeout. These indexes are being created on collections with up to 100 very small, simple documents. The tests that these failures occur on also use the same database and collection, dropping the collection at the start of each for independence. When this drop succeeds, the driver logs that the earlier test's createIndexes command failed with an IndexBuildAborted error.

      These failures only occur against authenticated, SSL-enabled clusters running MongoDB 8.0+.

      Example logs:

      First test

      [2025/10/24 07:20:15.421] DEBUG    pymongo.serverSelection:logger.py:108 {"message": "Server selection started", "selector": "<function writable_server_selector at 0x7fbdccaddd00>", "operation": "createIndexes", "topologyDescription": "<TopologyDescription id: 68fb85455c4f4a9df6e6059f, topology_type: Sharded, servers: [<ServerDescription ('localhost', 27017) server_type: Mongos, rtt: 0.0010386149606877644>]>", "clientId": {"$oid": "68fb85455c4f4a9df6e6059f"}}
      [2025/10/24 07:20:15.421] DEBUG    pymongo.serverSelection:logger.py:108 {"message": "Server selection succeeded", "selector": "<function writable_server_selector at 0x7fbdccaddd00>", "operation": "createIndexes", "topologyDescription": "<TopologyDescription id: 68fb85455c4f4a9df6e6059f, topology_type: Sharded, servers: [<ServerDescription ('localhost', 27017) server_type: Mongos, rtt: 0.0010386149606877644>]>", "clientId": {"$oid": "68fb85455c4f4a9df6e6059f"}, "serverHost": "localhost", "serverPort": 27017}
      [2025/10/24 07:20:15.421] DEBUG    pymongo.connection:logger.py:108 {"message": "Connection checkout started", "clientId": {"$oid": "68fb85455c4f4a9df6e6059f"}, "serverHost": "localhost", "serverPort": 27017}
      [2025/10/24 07:20:15.421] DEBUG    pymongo.connection:logger.py:108 {"message": "Connection checked out", "clientId": {"$oid": "68fb85455c4f4a9df6e6059f"}, "serverHost": "localhost", "serverPort": 27017, "driverConnectionId": 14, "durationMS": 0.00017049799998858362}
      [2025/10/24 07:20:15.421] DEBUG    pymongo.command:logger.py:108 {"message": "Command started", "clientId": {"$oid": "68fb85455c4f4a9df6e6059f"}, "command": "{\"createIndexes\": \"test\", \"indexes\": [{\"name\": \"num_-1\", \"key\": {\"num\": -1}}], \"lsid\": {\"id\": {\"$binary\": {\"base64\": \"3YaJl0VMRL68j9Kk9D09Eg==\", \"subType\": \"04\"}}}, \"$clusterTime\": {\"clusterTime\": {\"$timestamp\": {\"t\": 1761314827, \"i\": 41}}, \"signature\": {\"hash\": {\"$binary\": {\"base64\": \"NefVupIjjS+iP05oCw6O0Qk+NPs=\", \"subType\": \"00\"}}, \"keyId\": 7564786341519556632}}, \"$db\": \"pymongo_test\"}", "commandName": "createIndexes", "databaseName": "pymongo_test", "requestId": 1280657339, "operationId": 1280657339, "driverConnectionId": 14, "serverConnectionId": 8879, "serverHost": "localhost", "serverPort": 27017}
      # This command hangs until the test is terminated by a 60 second timeout limit.  
      

      Following test

      # Drop command at the start of the test
      [2025/10/24 07:20:15.423] DEBUG    {"message": "Command started", "clientId": {"$oid": "68fb85455c4f4a9df6e6059f"}, "command": "{\"drop\": \"test\", \"lsid\": {\"id\": {\"$binary\": {\"base64\": \"+foRcVjARw2Ztw4oQQHAqw==\", \"subType\": \"04\"}}}, \"$clusterTime\": {\"clusterTime\": {\"$timestamp\": {\"t\": 1761314886, \"i\": 1}}, \"signature\": {\"hash\": {\"$binary\": {\"base64\": \"EJtL1kwUtp4pdtC9I2ZbzCgsuhg=\", \"subType\": \"00\"}}, \"keyId\": 7564786341519556632}}, \"$db\": \"pymongo_test\"}", "commandName": "drop", "databaseName": "pymongo_test", "requestId": 1290974871, "operationId": 1290974871, "driverConnectionId": 15, "serverConnectionId": 9093, "serverHost": "localhost", "serverPort": 27017}
      # Failed createIndex command started by the previous test
      [2025/10/24 07:20:15.423] DEBUG    {"message": "Command failed", "clientId": {"$oid": "68fb85455c4f4a9df6e6059f"}, "durationMS": 60083.457, "failure": "{\"code\": 276, \"codeName\": \"IndexBuildAborted\"}", "commandName": "createIndexes", "databaseName": "pymongo_test", "requestId": 1280657339, "operationId": 1280657339, "driverConnectionId": 14, "serverConnectionId": 8879, "serverHost": "localhost", "serverPort": 27017}
      [2025/10/24 07:20:15.423] DEBUG    {"message": "Connection checked in", "clientId": {"$oid": "68fb85455c4f4a9df6e6059f"}, "serverHost": "localhost", "serverPort": 27017, "driverConnectionId": 14}
      [2025/10/24 07:20:15.423] DEBUG    {"message": "Command succeeded", "clientId": {"$oid": "68fb85455c4f4a9df6e6059f"}, "durationMS": 58.94, "reply": "{\"ok\": 1.0, \"$clusterTime\": {\"clusterTime\": {\"$timestamp\": {\"t\": 1761314887, \"i\": 25}}, \"signature\": {\"hash\": {\"$binary\": {\"base64\": \"p0zbfIZ42ZAGqNitWF74SMTygEg=\", \"subType\": \"00\"}}, \"keyId\": 7564786341519556632}}, \"operationTime\": {\"$timestamp\": {\"t\": 1761314887, \"i\": 25}}}", "commandName": "drop", "databaseName": "pymongo_test", "requestId": 1290974871, "operationId": 1290974871, "driverConnectionId": 15, "serverConnectionId": 9093, "serverHost": "localhost", "serverPort": 27017}
      

      Evergreen task: https://spruce.mongodb.com/task/mongo_python_driver_mongodb_v8.0_test_server_version_python3.11_async_auth_ssl_sharded_cluster_cov_patch_ad1167d01ea03780bae1fdb66ddc88283ce9a983_68fb845a5b57f10007a5ff3e_25_10_24_13_52_09/logs?execution=0&sorts=STATUS%3AASC.

      Another task example: https://spruce.mongodb.com/task/mongo_python_driver_no_c_ext_rhel8_test_standard_v8.0_python3.10_async_auth_ssl_sharded_cluster_patch_ad1167d01ea03780bae1fdb66ddc88283ce9a983_68fb845a5b57f10007a5ff3e_25_10_24_13_52_09/tests?execution=0&sorts=STATUS%3AASC.

      I will attach server logs from these runs once I have them.

            Assignee:
            Unassigned
            Reporter:
            Noah Stapp
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: