[CDRIVER-650] Crash in mongoc_stream_check_closed Created: 14/May/15  Updated: 19/Oct/16  Resolved: 29/May/15

Status: Closed
Project: C Driver
Component/s: libmongoc
Affects Version/s: 1.1.4
Fix Version/s: 1.1.7

Type: Bug Priority: Critical - P2
Reporter: A. Jesse Jiryu Davis Assignee: A. Jesse Jiryu Davis
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File CDRIVER-650-2.py     File example-client.c    
Issue Links:
Depends

 Description   

Crash with backtrace like:

mongoc_stream_check_closed ()
_mongoc_cluster_sendv () 
_mongoc_cursor_next ()
mongoc_cursor_next ()

mongoc_stream_check_closed only checks if "stream" is NULL in debug mode; a release-mode driver will crash if the stream passed to function is NULL.

Likely cause:

  • Start a cursor, complete the initial query on a primary, the cursor has a "hint", the index of its server in driver's array of servers
  • Shut down the primary
  • Do another operation on the cluster, forcing the driver to reconnect and change cluster state (fewer servers, different servers, a server with a cleared stream?)
  • Iterate the cursor to completion - it tries to send a getmore using its "hint", which may be out of bounds now, or point to a different server?, or point to a server with a NULL stream, not sure which of these three is possible
  • Segfault

There seem likely a number of ways the cursor's "hint" can be invalidated. Avoid crashes in all of them for the 1.1.x code.

Good news - this is all completely re-architected and corrected in 1.2, but 1.2. won't be released until EOQ at the earliest.



 Comments   
Comment by Githook User [ 05/Aug/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@emptysquare.net'}

Message: Merge pull request #235 from ajdavis/CDRIVER-650-crash-in-check-closed

CDRIVER-650 crash after replset reconnect
Branch: 1.2.0-dev
https://github.com/mongodb/mongo-c-driver/commit/53682560c307a806bb41c37232e8d4ec182df3ba

Comment by Githook User [ 05/Aug/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@emptysquare.net'}

Message: Merge pull request #235 from ajdavis/CDRIVER-650-crash-in-check-closed

CDRIVER-650 crash after replset reconnect
Branch: 1.2.0-dev
https://github.com/mongodb/mongo-c-driver/commit/53682560c307a806bb41c37232e8d4ec182df3ba

Comment by Githook User [ 05/Aug/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-650 crash after replset reconnect
Branch: 1.2.0-dev
https://github.com/mongodb/mongo-c-driver/commit/aa607cdf4de5052503b867460b40c1aa52756aa2

Comment by Githook User [ 29/May/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@emptysquare.net'}

Message: Merge pull request #235 from ajdavis/CDRIVER-650-crash-in-check-closed

CDRIVER-650 crash after replset reconnect
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/53682560c307a806bb41c37232e8d4ec182df3ba

Comment by Githook User [ 29/May/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@emptysquare.net'}

Message: Merge pull request #235 from ajdavis/CDRIVER-650-crash-in-check-closed

CDRIVER-650 crash after replset reconnect
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/53682560c307a806bb41c37232e8d4ec182df3ba

Comment by Githook User [ 29/May/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-650 crash after replset reconnect
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/aa607cdf4de5052503b867460b40c1aa52756aa2

Comment by A. Jesse Jiryu Davis [ 27/May/15 ]

https://github.com/mongodb/mongo-c-driver/pull/235/

Comment by A. Jesse Jiryu Davis [ 27/May/15 ]

Attaching CDRIVER-650-2.py, which uses MockupDB to confuse the driver, and example-client.c, which can reproduce the segfault. To reproduce, install Python's "pip" and then install MockupDB:

pip install git+git://github.com/ajdavis/mongo-mockup-db.git

Then run "python CDRIVER-650-2.py". The mock servers start up on ports 8000 and 8001.

Now compile and run example-client.c. The sequence of events is:

1. example-client connects with seed list "secondary,primary". Nodes are arranged in the cluster's array as 0 = primary, 1 = secondary.
2. example-client creates cursor & does initial query with read pref "secondary", completes initial read from secondary and stores "1" as its "hint"
3. Secondary shuts down, driver doesn't know it yet
4. example-client starts initial query with a different cursor on primary
5. Primary hangs up, driver gets network error and invalidates its cluster info
6. example-client calls mongoc_cursor_next on the secondary cursor
7. Driver reconnects and recreates its node list, this time without the secondary node
5. Driver then applies the secondary cursor's invalid hint, "1", to read past the end of the nodes list

Generated at Wed Feb 07 21:10:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.