[SERVER-13663] Perf regression for workload that updates 1 row with 1 or many clients Created: 18/Apr/14  Updated: 13/Jun/23  Resolved: 13/Jun/23

Status: Closed
Project: Core Server
Component/s: Performance, Write Ops
Affects Version/s: 2.6.0
Fix Version/s: 4.1 Desired

Type: Bug Priority: Major - P3
Reporter: Mark Callaghan Assignee: Backlog - Performance Team
Resolution: Won't Do Votes: 0
Labels: perf-stop-regressions, pperf
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PDF File 249.pdf     PDF File 260.pdf     File run.sh    
Issue Links:
Depends
depends on SERVER-13685 refactor fast-path for _id queries to... Closed
depends on SERVER-13337 Re-enable fast path for _id equality ... Closed
Related
Assigned Teams:
Product Performance
Operating System: ALL
Steps To Reproduce:

I used my fork of jmongosysbench - https://github.com/mdcallag/sysbench-mongodb as my changes have yet to be pushed upstream.

This is the only option I set for mongod:
journalCommitInterval=300

Then I load 400M rows into 1 collection (in the blog post I used 8 collections, but the repro case here only needs 1):
bash run.sh 1 400000000 1 5 yes no $( date +%s )

Then I run the test first for one client thread:
t=10; dop=1; bash run.sh 1 1 $dop $t no yes $( date +%s )

And then for 32 client threads:
t=10; dop=32; bash run.sh 1 1 $dop $t no yes $( date +%s )

Participants:

 Description   

My results are at http://smalldatum.blogspot.com/2014/04/biebermarks.html. Using the same hardware for the test and 2.4.9 versus 2.6 (rc2 or release) I get more updates/second with 2.4.9 versus 2.6 – 1.22X more at 32 concurrent clients and 1.11X more updates/second with 1 client.



 Comments   
Comment by Miguel Angel Nieto [ 13/Jun/23 ]

ger.hartnett@mongodb.com this ticket refers to 2.x version of MongoDB, I think it won't be needed anymore.

Comment by Ramon Fernandez Marina [ 15/May/15 ]

rui.zhang, can we re-test with a recent release (3.0.3 is the latest stable) and see if this is still an issue?

Thanks,
Ramón.

Comment by hari.khalsa@10gen.com [ 22/Apr/14 ]

Thanks for the ticket, mdcallag.

There is a fast path for find-by-_id that is being used in 2.4.x that is not being used in 2.6.x. I believe this is responsible for some of the regression you're seeing. I've created SERVER-13685 to track this.

Comment by Mark Callaghan [ 22/Apr/14 ]

Figured out how to make google perftools work, so here is a CPU profile from 1 client thread doing random queries over 1M row table. All data is in the OS filesystem cache – no disk reads. The client ran for 5 minutes and the average QPS is 15662 for 2.4.9 and 10286 for 2.6.0.

Looks like the new overhead for 2.6 is from the optimizer and maybe the parser.

Comment by Eliot Horowitz (Inactive) [ 20/Apr/14 ]

I'm seeing an issue with both java driver versions, but much worse with the newest.

Comment by Mark Callaghan [ 19/Apr/14 ]

I used mongo-java-driver-2.11.4.jar for all of my tests. If it helps I will repeat with another version.

Comment by Eliot Horowitz (Inactive) [ 19/Apr/14 ]

Seems a big part of the issue is new write commands.
If you use the old version of the java driver (r2.11.4) with 2.6 I see much better performance.
Need to dig into why.

Comment by Mark Callaghan [ 19/Apr/14 ]

Will provide more details on Monday, but using same database with 8 collections and concurrent point queries I see a larger regression in 2.6. Will repeat test with 1 thread (client). At 8 threads I get 94k QPS with 2.4.9 and 62k QPS with 2.6.0 release. Per "top" the Java client uses 1.5X more CPU with 2.4.9 than with 2.6.0 as expected. The context switch rate is about 1.4X more with 2.4.9 which is as expected and suggests this isn't a mutex contention issue. But mongod for 2.4.9 does not use 1.5X more CPU, instead 2.6.0 is using 1.2X more CPU even though 2.4.9 is doing 1.5X more QPS.

For this test, regression at 8 & 16 threads was worse than at 32 and 40 threads. Test host has 40 cores with hyperthread enabled and peak QPS was at 32 threads.

Top-n sources of CPU per "perf" on Linux...

for 2.4.9

     6.19%   mongod  mongod               [.] (anonymous namespace)::cpp_alloc(unsigned long, bool)                                                                                                                                                                                                                                                                                                                                                                                                                            
     2.99%   mongod  libpthread-2.13.so   [.] pthread_mutex_lock                                                                                                                                                                                                                                                                                                                                                                                                                                                               
     2.58%   mongod  mongod               [.] _ZN12_GLOBAL__N_121do_free_with_callbackEPvPFvS0_E.constprop.41                                                                                                                                                                                                                                                                                                                                                                                                                  
     2.32%   mongod  mongod               [.] mongo::KeyV1::woCompare(mongo::KeyV1 const&, mongo::Ordering const&) const                                                                                                                                                                                                                                                                                                                                                                                                       
     2.04%   mongod  mongod               [.] mongo::ps::Rolling::access(unsigned long, short, bool)                                                                                                                                                                                                                                                                                                                                                                                                                           
     1.58%   mongod  libc-2.13.so         [.] __strlen_sse42                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
     1.52%   mongod  libstdc++.so.6.0.16  [.] std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()                                                                                                                                                                                                                                                                                                                                                                                          
     1.49%   mongod  mongod               [.] boost::shared_ptr<mongo::Projection> boost::make_shared<mongo::Projection>()                                                                                                                                                                                                                                                                                                                                                                                                     
     1.48%   mongod  mongod               [.] mongo::BtreeBucket<mongo::BtreeData_V1>::find(mongo::IndexDetails const&, mongo::KeyV1 const&, mongo::DiskLoc const&, mongo::Ordering const&, int&, bool) const                                                                                                                                                                                                                                                                                                                  
     1.48%   mongod  mongod               [.] boost::detail::shared_count::~shared_count()                                                                                                                                                                                                                                                                                                                                                                                                                                     
     1.44%   mongod  mongod               [.] mongo::BucketBasics<mongo::BtreeData_V1>::KeyNode::KeyNode(mongo::BucketBasics<mongo::BtreeData_V1> const&, mongo::__KeyNode<mongo::DiskLoc56Bit> const&)                                                                                                                                                                                                                                                                                                                        
     1.30%   mongod  libc-2.13.so         [.] __memcpy_ssse3                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
     1.27%   mongod  mongod               [.] tc_malloc                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
     1.22%   mongod  [ip_tables]          [k] ipt_do_table                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
     1.20%   mongod  libstdc++.so.6.0.16  [.] _ZNSs4_Rep10_M_disposeERKSaIcE.part.2                                                                                                                                                                                                                                                                                                                                                                                                                                            
     1.03%   mongod  mongod               [.] mongo::Projection::init(mongo::BSONObj const&)     

And for 2.6.0 release

6.01%   mongod  mongod               [.] operator new(unsigned long) 
2.95%   mongod  libpthread-2.13.so   [.] pthread_mutex_lock
2.67%   mongod  mongod               [.] mongo::KeyV1::toBson() const
2.43%   mongod  mongod               [.] operator delete(void*)
1.68%   mongod  libc-2.13.so         [.] __strlen_sse42 
1.51%   mongod  libstdc++.so.6.0.16  [.] _ZNSs4_Rep10_M_disposeERKSaIcE.part.2
1.33%   mongod  mongod               [.] boost::intrusive_ptr<mongo::BSONObj::Holder>::~intrusive_ptr()
1.32%   mongod  libc-2.13.so         [.] __memcpy_ssse3
1.24%   mongod  mongod               [.] mongo::BucketBasics<mongo::BtreeData_V1>::KeyNode::KeyNode(mongo::BucketBasics<mongo::BtreeData_V1> const&, mongo::__KeyNode<mongo::DiskLoc56Bit> const&)
1.16%   mongod  mongod               [.] tc_malloc
1.09%   mongod  libc-2.13.so         [.] vfprintf
1.03%   mongod  mongod               [.] mongo::BSONElement::size() const
0.99%   mongod  libstdc++.so.6.0.16  [.] std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()
0.98%   mongod  libc-2.13.so         [.] __memcmp_sse4_1
0.94%   mongod  libstdc++.so.6.0.16  [.] std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&)
0.85%   mongod  mongod               [.] mongo::QueryPlanner::plan(mongo::CanonicalQuery const&, mongo::QueryPlannerParams const&, std::vector<mongo::QuerySolution*, std::allocator<mongo::QuerySolution*> >*) 
0.84%   mongod  mongod               [.] mongo::newRunQuery(mongo::Message&, mongo::QueryMessage&, mongo::CurOp&, mongo::Message&)
0.81%   mongod  mongod               [.] mongo::ParsedProjection::make(mongo::BSONObj const&, mongo::MatchExpression const*, mongo::ParsedProjection**)
0.76%   mongod  mongod               [.] tc_free                               

Comment by Mark Callaghan [ 18/Apr/14 ]

Comparing 2.4.9 with 2.6.0 release using "top" with 1 client thread

  • 2.4.9 does 1.15X more updates/second
  • 2.4.9 java client uses 1.13X more CPU
  • 2.4.9 mongod uses 0.92X (less) the CPU compared to 2.6.0

So I see a similar pattern at 1 thread as at 32 threads, but maybe the regression is worse with 32 threads. So I will focus on profiling at 1 thread first to see what uses more CPU.

Generated at Thu Feb 08 03:32:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.