[CDRIVER-27] non-blocking version (with support for libev) Created: 24/Nov/10 Updated: 16/Nov/21 Resolved: 06/Aug/16 |
|
| Status: | Closed |
| Project: | C Driver |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | TBD |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Eliot Horowitz (Inactive) | Assignee: | Backlog - C Driver Team |
| Resolution: | Won't Fix | Votes: | 9 |
| Labels: | internal-woes | ||
| Σ Remaining Estimate: | Not Specified | Remaining Estimate: | Not Specified |
| Σ Time Spent: | Not Specified | Time Spent: | Not Specified |
| Σ Original Estimate: | Not Specified | Original Estimate: | Not Specified |
| Attachments: |
|
||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||
| Sub-Tasks: |
|
||||||||||||||||||||||||
| Comments |
| Comment by A. Jesse Jiryu Davis [ 06/Aug/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I investigated for a customer whether an async C Driver is needed to support high-throughput applications talking to MongoDB over a high-latency high-throughput network. I ran some scaling tests and found that the C driver scales well to 4000 threads on a commonly deployed server VM. My test rig is a pair of EC2 m4.xlarge Ubuntu 14.04 machines in the US East zone. Using the latest C driver code on master, I wrote two programs, thread-loadtest-client.c and thread-loadtest-server.c. On both the client and server machines I configured a smattering of scalability options:
I compiled the programs and started the server program on one machine:
It chooses a port to listen on and accepts connections, spawning a thread for each. On each connection it receives "ismaster" requests from the test client program. For each request it waits 100ms, and responds. On the client machine I ran:
... where "XXX" is the internal IP of the server machine, YYY is the listening port, and ZZZ is the number of threads for the test run. The client program starts the threads, waits until they've all connected to the server program, then runs them concurrently for about 10 seconds to measure throughput in commands-per-second. The results:
Throughput scales linearly through 3500 threads. Around 4000 threads it hits a throughput of 371k commands per 10 seconds. Adding any more than 4000 threads barely increases throughput. CPU wasn't saturated on the client or server, so I think 4000 threads overcame latency and saturated the network, which was my goal. The program scales decently because each thread spends most of its time waiting for the high-latency response, so there's little unnecessary context-switching. (Starting and joining thousands of threads is time-consuming, but I don't include that overhead in my measurement.) I conclude that an async rewrite of the C Driver is not necessary to handle a high-latency connection to MongoDB; it's probably not even helpful. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Alex Reviyou [ 12/Apr/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hello, it look slike this issue existed for a while... Can you please let us know if this issue will even be resolved? We would like to see if we can use gridfg+nginx for our production system within several months... THanks, |