[CDRIVER-3318] Issues running some commands via mongoc when using SSL on macOS Created: 14/Aug/19  Updated: 28/Oct/23  Resolved: 25/Oct/19

Status: Closed
Project: C Driver
Component/s: None
Affects Version/s: None
Fix Version/s: 1.16.0, 1.15.3

Type: Bug Priority: Critical - P2
Reporter: Kaitlin Mahar Assignee: Samantha Ritter (Inactive)
Resolution: Fixed Votes: 0
Labels: platform-problems
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Mac OS X 10.14.5
mongoc 1.15.0
MongoDB 4.0.5 standalone


Attachments: HTML File Makefile     File main.c    
Issue Links:
Gantt Dependency

 Description   

I am running a standalone mongod (v4.0.5) started as follows using the certs from mongo-orchestration:

mongod --sslCAFile mongo-orchestration/tests/lib/ca.pem --sslWeakCertificateValidation --sslMode requireSSL --sslPEMKeyFile mongo-orchestration/tests/lib/server.pem

The first command I noticed issues with is buildInfo.

I am able to run buildInfo via the shell as follows:

mongo admin --ssl --sslCAFile code/drivers/mongo-orchestration/tests/lib/ca.pem --eval "db.runCommand({buildInfo: 1})"

However when I try to run it via mongoc (see attached file), the command hangs for 5 minutes and then fails with the output

WARNING:       stream: Failed to buffer 600 bytes
failed to run command: Failed to send "buildInfo" command with database "admin": Failed to read 1620 bytes: socket error or timeout

I have reproduced this both on mongoc 1.14.0 and 1.15.0. 

This is example #1 in the attached file.

You will need to replace the absolute path to the CA file with your own local path.

The second command I ran into issues with is listIndexes. See example 2 in the attached file. This came about running a Swift driver test that creates a couple of indexes and exercises a bunch of options, and then calls mongoc_collection_find_indexes_with_opts to get the results. The function call would hang for a few minutes and eventually print out

WARNING:       stream: Failed to buffer 1 bytes
WARNING:       stream: Failed to buffer 1 bytes

I can't quite figure it out, but the issue has something to do with the size of the createIndexes command run before the call to listIndexes. Possibly the issue just occurs after some number of bytes are read and if your initial command was large enough that one would hang, too.

This only seems to happen in pooled mode. Switching to just creating a new client with the URI solves the problem.

The issues do not occur on Linux.

For now I will use serverStatus in place of buildInfo, and skip the problematic index test when using SSL.

Since we already triggered this issue 2x in our tests inadvertently it seems like it would be fairly easy for a user to encounter as well.



 Comments   
Comment by Githook User [ 13/Dec/19 ]

Author:

{'name': 'samantharitter', 'email': 'samantha.ritter@10gen.com', 'username': 'samantharitter'}

Message: CDRIVER-3318 pass correct message length to SSLRead
Branch: r1.15
https://github.com/mongodb/mongo-c-driver/commit/f387ff0e2c68fe9820628bd8a8f792d5663de087

Comment by Githook User [ 13/Dec/19 ]

Author:

{'name': 'samantharitter', 'email': 'samantha.ritter@10gen.com', 'username': 'samantharitter'}

Message: CDRIVER-3318 add test for ssl hang
Branch: r1.15
https://github.com/mongodb/mongo-c-driver/commit/2493b96885d250f03e0351c1c8a4778465cf2ff6

Comment by Kaitlin Mahar [ 25/Oct/19 ]

Tried it out and our previously failing tests work now. Thanks again!

Comment by Githook User [ 25/Oct/19 ]

Author:

{'username': 'samantharitter', 'email': 'samantha.ritter@10gen.com', 'name': 'samantharitter'}

Message: CDRIVER-3318 pass correct message length to SSLRead
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/ce14000d7728ea96c3cb8207415526b69510fd20

Comment by Githook User [ 25/Oct/19 ]

Author:

{'username': 'samantharitter', 'email': 'samantha.ritter@10gen.com', 'name': 'samantharitter'}

Message: CDRIVER-3318 add test for ssl hang
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/a09fdb13a5429938c510bcc83ef0f8af9ad4b2b6

Comment by Kaitlin Mahar [ 18/Oct/19 ]

Tricky! Good find. Thanks for the update and for fixing this! I will test out the changes once this has been merged. 

Comment by Samantha Ritter (Inactive) [ 18/Oct/19 ]

It turns out that this issue is specific to Secure Transport, not to other ssl libraries on osx.  I discovered that the bug is due to a way that we were integrating with Secure Transport.  Unlike other libraries' SSL read methods, Secure Transport's SSLRead method will block until it reads the number of bytes passed to it or times out.  Rather than passing our actual desired minimum bytes into this method, we pass in the length of the buffer allocated to receive the bytes, which is often greater in size than the message we expect.  We should be doing some math to pass whichever value is smaller, the buffer, or the remaining bytes we'd like Secure Transport to read.

I am not exactly sure why your specific tests caused this bug to surface, and I'm surprised that we have not seen it before!

Comment by Kaitlin Mahar [ 19/Aug/19 ]

I am also encountering this issue trying to run either one of buildInfo or serverStatus against a sharded cluster with SSL on. Started using MongoDB 4.2 and this MO config (just the ssl.json sharded cluster one with SSL file paths changed):

{
    "configsvrs": [
        {
            "dbpath": "$DBPATH/db27117",
            "logpath": "$LOGPATH/configsvr27117.log",
            "port": 27117
        }
    ], 
    "id": "shard_cluster_1", 
    "shards": [
        {
            "id": "sh01", 
            "shardParams": {
                "procParams": {
                    "dbpath": "$DBPATH/db27217",
                    "logpath": "$LOGPATH/sh01.log",
                    "port": 27217
                }
            }
        }, 
        {
            "id": "sh02", 
            "shardParams": {
                "procParams": {
                    "dbpath": "$DBPATH/db27218",
                    "logpath": "$LOGPATH/sh02.log",
                    "port": 27218
                }
            }
        }
    ], 
    "routers": [
        {
            "logpath": "$LOGPATH/router27017.log",
            "port": 27017
        }, 
        {
            "logpath": "$LOGPATH/router27018.log",
            "port": 27018
        }
    ], 
    "sslParams": {
        "sslCAFile": "/Users/kaitlinmahar/code/drivers/mongo-orchestration/tests/lib/ca.pem", 
        "sslOnNormalPorts": true, 
        "sslPEMKeyFile": "/Users/kaitlinmahar/code/drivers/mongo-orchestration/tests/lib/server.pem", 
        "sslWeakCertificateValidation": true
    }
}

And I am running the command in the same fashion as in the buildInfo example in the attached file, with URI:
"mongodb://localhost:27017,localhost:27018/?ssl=true&sslAllowInvalidCertificates=true&sslCertificateAuthorityFile=%2FUsers%2Fkaitlinmahar%2Fcode%2Fdrivers%2Fmongo-orchestration%2Ftests%2Flib%2Fca.pem"

Comment by Kaitlin Mahar [ 15/Aug/19 ]

FYI I also am encountering something similar to example 2 in a test that creates a couple collections using a bunch of options and then tries to call mongoc_database_find_collections_with_opts. (I guess this is not surprising as I assume the methods for listing indexes and collections are implemented very similarly.)

Generated at Wed Feb 07 21:17:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.