[CDRIVER-3522] Stack smashing detected connecting to multiple replica set members with TLS Created: 05/Feb/20  Updated: 28/Oct/23  Resolved: 23/Jul/20

Status: Closed
Project: C Driver
Component/s: None
Affects Version/s: 1.13.0, 1.14.0
Fix Version/s: 1.18.0, 1.17.3, 1.18.0-alpha

Type: Bug Priority: Major - P3
Reporter: Jeremy Mikola Assignee: Roberto Sanchez
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by PHPC-1420 Using replicaSet triggers __fortify_f... Closed
Related
related to CDRIVER-3523 Some return values not considered in ... Backlog
is related to PHPC-1420 Using replicaSet triggers __fortify_f... Closed

 Description   

This was originally reported in PHPC-1420 but I just now got around to investigating the backtrace. The user originally reported this crash with the following connection string:

mongodb://demo:PASSWORD@db2.infra.linker.shop:27017,db1.infra.linker.shop:27017,db3.infra.linker.shop:27017/?replicaSet=rps1&ssl=true&authSource=admin

The ssl and authSource options were originally specified outside of the connection string (via PHP's extra hash argument), but I've merged them in for simplicity.

The crash itself appeared with the following in debug output from the PHP process:

*** stack smashing detected ***: php terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f2f16295b67]
/lib64/libc.so.6(+0x117b22)[0x7f2f16295b22]
/usr/lib64/php/modules/mongodb.so(mongoc_stream_tls_openssl_new+0x3ea)[0x7f2efe76400a]
/usr/lib64/php/modules/mongodb.so(_mongoc_topology_scanner_node_setup_stream_for_tls+0x2d)[0x7f2efe7698ed]
/usr/lib64/php/modules/mongodb.so(_mongoc_async_cmd_phase_initiate+0x7)[0x7f2efe721e27]
/usr/lib64/php/modules/mongodb.so(mongoc_async_cmd_run+0x33)[0x7f2efe722483]
/usr/lib64/php/modules/mongodb.so(mongoc_async_run+0x36d)[0x7f2efe72180d]
/usr/lib64/php/modules/mongodb.so(mongoc_topology_scanner_work+0x10)[0x7f2efe76a2c0]
/usr/lib64/php/modules/mongodb.so(+0x9139b)[0x7f2efe76439b]
/usr/lib64/php/modules/mongodb.so(_mongoc_topology_do_blocking_scan+0x35)[0x7f2efe764cc5]
/usr/lib64/php/modules/mongodb.so(mongoc_topology_select_server_id+0x337)[0x7f2efe7653b7]
/usr/lib64/php/modules/mongodb.so(mongoc_topology_select+0x11)[0x7f2efe765561]
/usr/lib64/php/modules/mongodb.so(mongoc_client_select_server+0x44)[0x7f2efe729d44]
/usr/lib64/php/modules/mongodb.so(+0xb9692)[0x7f2efe78c692]
/usr/lib64/php/modules/mongodb.so(+0xb9793)[0x7f2efe78c793]

The equivalent crash in GDB was a bit more verbose, but also includes some function names missing in the previous log:

Program received signal SIGABRT, Aborted.
0x00007f2f161b42c7 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f2f161b42c7 in raise () from /lib64/libc.so.6
#1  0x00007f2f161b59b8 in abort () from /lib64/libc.so.6
#2  0x00007f2f161f6e17 in __libc_message () from /lib64/libc.so.6
#3  0x00007f2f16295b67 in __fortify_fail () from /lib64/libc.so.6
#4  0x00007f2f16295b22 in __stack_chk_fail () from /lib64/libc.so.6
#5  0x00007f2efe76400a in mongoc_stream_tls_openssl_new (
    base_stream=base_stream@entry=0x55a498722700, 
    host=0x55a498230c98 "2a01:4f8:173:1828::40", opt=0x55a498231968, 
    client=<optimized out>)
    at /usr/src/debug/php-pecl-mongodb-1.6.0~alpha2/NTS/src/libmongoc/src/libmongoc/src/mongoc/mongoc-stream-tls-openssl.c:777
#6  0x00007f2efe762854 in mongoc_stream_tls_new_with_hostname (
    base_stream=base_stream@entry=0x55a498722700, host=<optimized out>, 
    opt=<optimized out>, client=client@entry=1)
    at /usr/src/debug/php-pecl-mongodb-1.6.0~alpha2/NTS/src/libmongoc/src/libmongoc/src/mongoc/mongoc-stream-tls.c:220
#7  0x00007f2efe7698ed in _mongoc_topology_scanner_node_setup_stream_for_tls (
    node=<optimized out>, stream=0x55a498722700)
    at /usr/src/debug/php-pecl-mongodb-1.6.0~alpha2/NTS/src/libmongoc/src/libmongoc/src/mongoc/mongoc-topology-scanner.c:601
#8  0x00007f2efe721e27 in _mongoc_async_cmd_phase_initiate (
    acmd=0x55a498722020)
    at /usr/src/debug/php-pecl-mongodb-1.6.0~alpha2/NTS/src/libmongoc/src/libmongoc/src/mongoc/mongoc-async-cmd.c:245
#9  0x00007f2efe722483 in mongoc_async_cmd_run (acmd=acmd@entry=0x55a498722020)
    at /usr/src/debug/php-pecl-mongodb-1.6.0~alpha2/NTS/src/libmongoc/src/libmongoc/src/mongoc/mongoc-async-cmd.c:114
#10 0x00007f2efe72180d in mongoc_async_run (async=0x55a49822d520)
    at /usr/src/debug/php-pecl-mongodb-1.6.0~alpha2/NTS/src/libmongoc/src/libmongoc/src/mongoc/mongoc-async.c:94
#11 0x00007f2efe76a2c0 in mongoc_topology_scanner_work (ts=0x55a49822eef0)
    at /usr/src/debug/php-pecl-mongodb-1.6.0~alpha2/NTS/src/libmongoc/src/libmongoc/src/mongoc/mongoc-topology-scanner.c:1007
#12 0x00007f2efe76439b in mongoc_topology_scan_once (
    topology=topology@entry=0x55a49822de90, 
    obey_cooldown=obey_cooldown@entry=true)
    at /usr/src/debug/php-pecl-mongodb-1.6.0~alpha2/NTS/src/libmongoc/src/libmongoc/src/mongoc/mongoc-topology.c:471
#13 0x00007f2efe764cc5 in _mongoc_topology_do_blocking_scan (
    topology=topology@entry=0x55a49822de90, error=error@entry=0x7fff34f386a0)
    at /usr/src/debug/php-pecl-mongodb-1.6.0~alpha2/NTS/src/libmongoc/src/libmongoc/src/mongoc/mongoc-topology.c:501
#14 0x00007f2efe7653b7 in mongoc_topology_select_server_id (
    topology=topology@entry=0x55a49822de90, 
    optype=optype@entry=MONGOC_SS_READ, 
    read_prefs=read_prefs@entry=0x55a49822d710, 
    error=error@entry=0x7fff34f38950)
    at /usr/src/debug/php-pecl-mongodb-1.6.0~alpha2/NTS/src/libmongoc/src/libmongoc/src/mongoc/mongoc-topology.c:734
#15 0x00007f2efe765561 in mongoc_topology_select (topology=0x55a49822de90, 
    optype=optype@entry=MONGOC_SS_READ, 
    read_prefs=read_prefs@entry=0x55a49822d710, 
    error=error@entry=0x7fff34f38950)
    at /usr/src/debug/php-pecl-mongodb-1.6.0~alpha2/NTS/src/libmongoc/src/libmongoc/src/mongoc/mongoc-topology.c:616
#16 0x00007f2efe729d44 in mongoc_client_select_server (
    client=client@entry=0x55a4982318e0, for_writes=for_writes@entry=false, 
    prefs=0x55a49822d710, error=error@entry=0x7fff34f38950)
    at /usr/src/debug/php-pecl-mongodb-1.6.0~alpha2/NTS/src/libmongoc/src/libmongoc/src/mongoc/mongoc-client.c:2620
#17 0x00007f2efe78c692 in php_phongo_manager_select_server (
    for_writes=for_writes@entry=false, zreadPreference=<optimized out>, 
    client=0x55a4982318e0, server_id=server_id@entry=0x7fff34f38b9c)
    at /usr/src/debug/php-pecl-mongodb-1.6.0~alpha2/NTS/src/MongoDB/Manager.c:271
#18 0x00007f2efe78c793 in zim_Manager_selectServer (
    execute_data=<optimized out>, return_value=0x7f2f14e15210)
    at /usr/src/debug/php-pecl-mongodb-1.6.0~alpha2/NTS/src/MongoDB/Manager.c:631

Researching __stack_chk_fail suggests that it's merely indicative of when the stack smash was detected, but not where it originated. This Stack Overflow thread suggests using AddressSanitizer to investigate further. Without being able to identify the smash with ASan, I believe anything in the stack trace could be suspect.

The crash itself was reported against PHPC 1.5.5 and 1.6.0alpha2, so I've marked the affected version as libmongoc 1.13.0 and 1.14.0, respectively. In the meantime, I've also asked the user to attempt reproduction using the latest version of the driver (1.7.1), which uses libmongoc 1.16.1.



 Comments   
Comment by Githook User [ 24/Nov/20 ]

Author:

{'name': 'Roberto C. Sánchez', 'email': 'roberto@connexer.com', 'username': 'rcsanchez97'}

Message: CDRIVER-3522 use correct data structure for IPv6
Branch: r1.17
https://github.com/mongodb/mongo-c-driver/commit/579a31d646e0970de4ce29b8961a2af191b90232

Comment by Githook User [ 23/Jul/20 ]

Author:

{'name': 'Roberto C. Sánchez', 'email': 'roberto@connexer.com', 'username': 'rcsanchez97'}

Message: CDRIVER-3522 use correct data structure for IPv6
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/7977659d249f2dd024a7285b267de46aea1e11ef

Comment by Roberto Sanchez [ 17/Jul/20 ]

In the process of working on CDRIVER-3579, I began observing ASAN failures which revealed that the cause of this particular issue was in mongoc-stream-tls-openssl.c in the mongoc_stream_tls_openssl_new function, where a call to inet_pton specifies AF_INET6 but passes a reference to a struct in_addr instead of a struct in6_addr. The difference in size of the two structures was triggering the crash.

Comment by Kevin Albertson [ 12/May/20 ]

Hypothesis: A host listed in the "hosts" list that mismatches TLS validation causes the crash during scan.

Attempted to reproduce by setting up a replica set with a "hosts" list containing a different host name to a secondary (one which fails TLS hostname validation).

I modified /etc/hosts to add these entries:

127.0.0.1 host
127.0.0.1 host2

Created a server certificate with the common name "host", then configured a two node replica set in the mongodb shell:

cfg = rs.confg()
cfg.members[0].host = "host:27017"
cfg.members[1].host = "host2:27017"
rs.reconfig(cfg)

Then connected with example client:

./cmake-build/src/libmongoc/example-client "mongodb://host:27017,host:27018/?tls=true&tlsCAFile=/certs/ca.pem" 

But no crash was observed compiling with ASAN.

Comment by Roberto Sanchez [ 17/Apr/20 ]

I have commented on PHPC-1420 with a request for additional information.

Generated at Wed Feb 07 21:18:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.