[CDRIVER-586] Bulk Writes sent to wrong server in write command mode Created: 20/Mar/15  Updated: 08/Jan/24  Resolved: 24/Mar/15

Status: Closed
Project: C Driver
Component/s: None
Affects Version/s: 1.1.2
Fix Version/s: 1.2-beta0

Type: Bug Priority: Major - P3
Reporter: Hannes Magnusson Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to PHPC-186 MongoDB\Driver\Server of type Seconda... Closed

 Description   

The bulk->hint is used to do all the precondition checks, including wire version, to decide if we should do legacy op inserts or write commands.

When legacy op inserts are detect we submit the write to the correct bulk->hint server.

However, in the case of write commands this hint gets lost mid-way and a new server selected.

This has two consequences:

  • We could wind up attempting bulk write on a 2.4 server
  • A bulk write to a secondary `local` database will be re-routed to the primary, effectively resulting in dataloss.


 Comments   
Comment by Githook User [ 07/Oct/15 ]

Author:

{u'username': u'hanumantmk', u'name': u'Jason Carey (hanumantmk)', u'email': u'jcarey@argv.me'}

Message: CDRIVER-586 Direct bulk writes to hinted nodes

Bulk writes could have a hint set, which was used for decisions around
legacy op, size of write commands, etc. After all those checks, we
dispatched to whichever node happened to be the primary though.

Added support for a command_simple_with_hint, which allows for a hint.
This lets us dispatch to a specific node, which will error on a non
master, rather than possibly silently succeeding if we route to a
primary.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/b887dab50bec002f67c9e9b0c877bfa36522baa0

Comment by Githook User [ 27/May/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: Merge branch 'pr-233' into 1.2.0-dev

Comment by Githook User [ 27/May/15 ]

Author:

{u'username': u'bjori', u'name': u'Hannes Magnusson', u'email': u'bjori@php.net'}

Message: CDRIVER-586: Send writes to the requested server

This fixes issues (again) where we preselect a node and do all our checks based
on that node, then run the SS machinery again and everything has
changed.

It also fixes intentionaly secondary-writes (e.g. to the local db)
Where the write was incorrectly rerouted to the primary.
Branch: 1.2.0-dev
https://github.com/mongodb/mongo-c-driver/commit/0a314fb388024c8916f8718274aa71e2dabd8b8a

Comment by Githook User [ 24/Mar/15 ]

Author:

{u'username': u'hanumantmk', u'name': u'Jason Carey (hanumantmk)', u'email': u'jcarey@argv.me'}

Message: CDRIVER-586 Direct bulk writes to hinted nodes

Bulk writes could have a hint set, which was used for decisions around
legacy op, size of write commands, etc. After all those checks, we
dispatched to whichever node happened to be the primary though.

Added support for a command_simple_with_hint, which allows for a hint.
This lets us dispatch to a specific node, which will error on a non
master, rather than possibly silently succeeding if we route to a
primary.
Branch: 1.2.0-dev
https://github.com/mongodb/mongo-c-driver/commit/b887dab50bec002f67c9e9b0c877bfa36522baa0

Comment by Hannes Magnusson [ 23/Mar/15 ]

Works! thanks mira.carey@mongodb.com

Comment by Mira Carey [ 23/Mar/15 ]

I pulled together a fix in: https://github.com/mongodb/mongo-c-driver/pull/208

Basically, it adds a private command_simple_with_hint, which we use for write commands to route to a specific node.

Tell me if that fixes your problem

Comment by Hannes Magnusson [ 20/Mar/15 ]

Just realized I was running with uncommitted patch that fixes the batch->hint actually being used - see https://github.com/mongodb/mongo-c-driver/pull/207

The issue of the hint being discarded before executing the cursor is still an issue though, likely reproducable with normal commands.

Comment by Hannes Magnusson [ 20/Mar/15 ]

#0  mongoc_client_command (client=0x11e7d20, db_name=0x7fffffff9960 "phongo.$cmd", flags=MONGOC_QUERY_NONE, skip=0, limit=1, batch_size=0, query=0x7fffffff9c80, fields=0x0, read_prefs=0x1073650)
    at /home/bjori/Sources/phongo/src/libmongoc/src/mongoc/mongoc-client.c:1222
#1  0x00007ffff58d11c1 in mongoc_client_command_simple (client=0x11e7d20, db_name=0x11e5190 "phongo", command=0x7fffffff9c80, read_prefs=0x0, reply=0x7fffffff9d00, error=0x1074068)
    at /home/bjori/Sources/phongo/src/libmongoc/src/mongoc/mongoc-client.c:1267
#2  0x00007ffff58ff0a4 in _mongoc_write_command_insert (command=0x11fb080, client=0x11e7d20, hint=2, database=0x11e5190 "phongo", collection=0x11fb410 "replicaset_writeresult_getserver_002.phpt", write_concern=0x10734c0, 
    result=0x1073e60, error=0x1074068) at /home/bjori/Sources/phongo/src/libmongoc/src/mongoc/mongoc-write-command.c:675
#3  0x00007ffff58ff986 in _mongoc_write_command_execute (command=0x11fb080, client=0x11e7d20, hint=2, database=0x11e5190 "phongo", collection=0x11fb410 "replicaset_writeresult_getserver_002.phpt", write_concern=0x10734c0, 
    result=0x1073e60) at /home/bjori/Sources/phongo/src/libmongoc/src/mongoc/mongoc-write-command.c:825
#4  0x00007ffff58ce121 in mongoc_bulk_operation_execute (bulk=0x1073de0, reply=0x0, error=0x7fffffff9fb0) at /home/bjori/Sources/phongo/src/libmongoc/src/mongoc/mongoc-bulk-operation.c:364
#5  0x00007ffff5906d7d in phongo_execute_write (client=0x11e7d20, namespace=0x11e7ee0 "phongo.replicaset_writeresult_getserver_002.phpt", bulk=0x1073de0, write_concern=0x10734c0, server_id=2, return_value=0x11fb1c0, return_value_used=1)
    at /home/bjori/Sources/phongo/php_phongo.c:512

(gdb) frame 0
#0  _mongoc_cursor_query (cursor=0x11fb4a0) at /home/bjori/Sources/phongo/src/libmongoc/src/mongoc/mongoc-cursor.c:499
499	   if (!(hint = _mongoc_client_sendv (cursor->client, &rpc, 1,
(gdb) l
494	      rpc.query.fields = bson_get_data (&cursor->fields);
495	   } else {
496	      rpc.query.fields = NULL;
497	   }
498	
499	   if (!(hint = _mongoc_client_sendv (cursor->client, &rpc, 1,
500	                                      cursor->hint, NULL,
501	                                      cursor->read_prefs,
502	                                      &cursor->error))) {
503	      GOTO (failure);
(gdb) p cursor->hint
$5 = 0
(gdb) bt
#0  _mongoc_cursor_query (cursor=0x11fb4a0) at /home/bjori/Sources/phongo/src/libmongoc/src/mongoc/mongoc-cursor.c:499
#1  0x00007ffff58df84e in _mongoc_cursor_next (cursor=0x11fb4a0, bson=0x7fffffff9a58) at /home/bjori/Sources/phongo/src/libmongoc/src/mongoc/mongoc-cursor.c:818
#2  0x00007ffff58df561 in mongoc_cursor_next (cursor=0x11fb4a0, bson=0x7fffffff9a58) at /home/bjori/Sources/phongo/src/libmongoc/src/mongoc/mongoc-cursor.c:746
#3  0x00007ffff58d11dc in mongoc_client_command_simple (client=0x11e7d20, db_name=0x11e5190 "phongo", command=0x7fffffff9c80, read_prefs=0x0, reply=0x7fffffff9d00, error=0x1074068)
    at /home/bjori/Sources/phongo/src/libmongoc/src/mongoc/mongoc-client.c:1270
#4  0x00007ffff58ff0a4 in _mongoc_write_command_insert (command=0x11fb080, client=0x11e7d20, hint=2, database=0x11e5190 "phongo", collection=0x11fb410 "replicaset_writeresult_getserver_002.phpt", write_concern=0x10734c0, 
    result=0x1073e60, error=0x1074068) at /home/bjori/Sources/phongo/src/libmongoc/src/mongoc/mongoc-write-command.c:675
#5  0x00007ffff58ff986 in _mongoc_write_command_execute (command=0x11fb080, client=0x11e7d20, hint=2, database=0x11e5190 "phongo", collection=0x11fb410 "replicaset_writeresult_getserver_002.phpt", write_concern=0x10734c0, 
    result=0x1073e60) at /home/bjori/Sources/phongo/src/libmongoc/src/mongoc/mongoc-write-command.c:825
#6  0x00007ffff58ce121 in mongoc_bulk_operation_execute (bulk=0x1073de0, reply=0x0, error=0x7fffffff9fb0) at /home/bjori/Sources/phongo/src/libmongoc/src/mongoc/mongoc-bulk-operation.c:364
#7  0x00007ffff5906d7d in phongo_execute_write (client=0x11e7d20, namespace=0x11e7ee0 "phongo.replicaset_writeresult_getserver_002.phpt", bulk=0x1073de0, write_concern=0x10734c0, server_id=2, return_value=0x11fb1c0, return_value_used=1)
    at /home/bjori/Sources/phongo/php_phongo.c:512

As you can see in the backtrace, the provided hint is dropped by the mongoc_client_command_simple() call.

Generated at Wed Feb 07 21:09:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.