[SERVER-29006] Query failed with error code 6 with YCSB workload Created: 27/Apr/17  Updated: 27/Oct/23  Resolved: 23/Jun/17

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: sandip Assignee: Mark Agarunov
Resolution: Gone away Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongodb.py.txt    
Operating System: ALL
Steps To Reproduce:

Mentioned in the description.

Participants:

 Description   

Hi ,

I have the following setup 4 nodes, 1 client
Client -> node1 - node2 - node3 - node4
node1 and node2 config server is running
node[1-4] are also acting as shard server , Sharding is enabled for YCSB database and for collection usertable.

Mongos is running on client.

YCSB: https://github.com/brianfrankcooper/YCSB/tree/master/mongodb
Workload: I insert 10M doc , it works fine, When i run workloade issue started to happen

With Version 3.4 Enterprise it crashed. Issue looks similar SERVER-27701

With version 3.5.6 Lot of queries throwing following Error
com.mongodb.MongoQueryException: Query failed with error code 6 and error message



 Comments   
Comment by Mark Agarunov [ 23/Jun/17 ]

Hello sannandi@indiana.edu,

Thank you for the information, I am glad this is now working for you. As this is no longer an issue, I've closed this ticket.

Thanks,
Mark

Comment by sandip [ 22/Jun/17 ]

Hi ,

I figure it out and make it work. It was a problem related to multiple issues.

1. The YCSB repo uses old mongo JDK, the implementation does some leak. if I use latest one 3.4.2 it works good.
2. I need to set up ulimit -n 32000 and sysctl net.core.somaxconn = 4096 , the failures are due to system limit.

Thanks,

Comment by Mark Agarunov [ 22/Jun/17 ]

Hello sannandi@indiana.edu

We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide the complete log files from both mongod and mongos when this issue is present?

Thanks,
Mark

Comment by Mark Agarunov [ 09/Jun/17 ]

Hello sannandi@indiana.edu,

Thank you for providing the script. Looking into this further, the error code you are seeing, 6, is "HostUnreachable". As this can be caused by a few different paths in the code, I would like to request the complete log files from both mongod and mongos when this issue is present so that we can get a better idea of what the issue may be. I've generated a secure upload portal so that you can send us these files privately.

Thanks,
Mark

Comment by Mahbubur Rub Talha [ 02/Jun/17 ]

Hello

i'm facing same issue on my mongodb cluster. I have a collection in which sharding is not enabled. When i run single find query it seems ok. But during concurrent query it started showing exception "com.mongodb.MongoQueryException: Query failed with error code 6 and error message 'Operation aborted.' on server" . And on "Exit Codes and Statuses" of mongodb documentation there are nothing about error code 6

Comment by sandip [ 02/May/17 ]

The configuration is here:
Client -> node1 - node2 - node3 - node4
node1 and node2 config server is running
node[1-4] are also acting as shard server , Sharding is enabled for YCSB database and for collection usertable.
Mongos is running on client.

Only a Single instance of mongos running on client

Commands used to run:
Attached the script mongodb.py.txt

What is the configuration you are using ?

Comment by Mark Agarunov [ 02/May/17 ]

Hello sannandi@indiana.edu,

I have not yet been able to reproduce the error that you've described, and was hoping to get some more information. If possible, please provide the following:

  • The complete output of YCSB
  • The complete logs from mongos
  • The complete logs from all mongod nodes
  • The configuration you are using for mongod and mongos
  • The exact command you are using to execute YCSB

The should help shed some light on what may be causing this behavior.

Thanks,
Mark

Comment by Jeffrey Yemin [ 27/Apr/17 ]

I don't think this is a networking issue between client and mongos. Running against MongoDB >= 3.2, the driver will only throw a MongoQueryException in response to a non-ok response from the server to either the find or getMore command (in this case probably a find since all YCSB queries are for single documents). The error code of 6 and the error message of "'End of file' on server x.x.x.x:27017." is what's being returned from mongos in the command response, e.g.:

{
  ok: 0, 
  code : 6, 
  errmsg : "'End of file' on server x.x.x.x:27017."
}

Comment by Daniel Pasette (Inactive) [ 27/Apr/17 ]

that's the error message given by the java driver back to the client. It looks like a networking issue between the client and the mongos.

Comment by sandip [ 27/Apr/17 ]

Only error message i can see is
com.mongodb.MongoQueryException: Query failed with error code 6 and error message 'End of file' on server x.x.x.x:27017.

All the clusters are up while the test is on. Is there anything wrong with the setup ? As I have mentioned if i use only write query it works fine.

Comment by Daniel Pasette (Inactive) [ 27/Apr/17 ]

What is the error on the mongos?
Are all the nodes in the cluster up and running at the time of the error message?

Comment by sandip [ 27/Apr/17 ]

The YCSB workload e is 95% scan and 5% insert. If I change the workload to 100% insert it works fine, When I do scan operation it is throwing error. The earlier version was crashing. With version 3.5.6 crash issue on mongos has fixed but exception starts to happen.

I am using 4 instances of YCSB and each one doing 30 threads.

Generated at Thu Feb 08 04:19:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.