[SERVER-5918] invalid memory writes in sharding test suite Created: 24/May/12  Updated: 11/Jul/16  Resolved: 11/Jun/12

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.1.1
Fix Version/s: 2.1.2

Type: Bug Priority: Major - P3
Reporter: Eric Milkie Assignee: Greg Studer
Resolution: Done Votes: 0
Labels: buildbot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-4262 when dropping collections need to inv... Closed
Operating System: ALL
Participants:

 Description   

To reproduce, run "test sharding" with Valgrind
Might be part of the cause for http://buildbot.mongodb.org/builders/Linux%2032-bit%20debug/builds/1763/steps/test_1/logs/stdio

==32699== Invalid write of size 8
==32699==    at 0x14D3821: mongo::DBClientConnection::setSoTimeout(double) (dbclientinterface.h:1025)
==32699==    by 0x14D337A: mongo::ScopedDbConnection::_setSocketTimeout() (connpool.cpp:391)
==32699==    by 0x12300E4: mongo::ScopedDbConnection::ScopedDbConnection(std::string const&, double) (connpool.h:206)
==32699==    by 0x15AA081: mongo::ChunkManager::calcInitSplitsAndShards(mongo::Shard const&, std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> > const*, std::vector<mongo::Shard, std::allocator<mongo::Shard> > const*, std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> >*, std::vector<mongo::Shard, std::allocator<mongo::Shard> >*) const (chunk.cpp:913)
==32699==    by 0x15AA5CD: mongo::ChunkManager::createFirstChunks(std::string const&, mongo::Shard const&, std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> > const*, std::vector<mongo::Shard, std::allocator<mongo::Shard> > const*) (chunk.cpp:960)
==32699==    by 0x1142872: ShardingTests::ChunkManagerCreateBasicTest::run() (sharding.cpp:101)
==32699==    by 0x114AE3E: mongo::unittest::TestHolderBase<ShardingTests::ChunkManagerCreateBasicTest>::run() (unittest.h:86)
==32699==    by 0x1220DC0: mongo::unittest::Suite::run(std::string const&) (unittest.cpp:121)
==32699==    by 0x1221511: mongo::unittest::Suite::run(std::vector<std::string, std::allocator<std::string> > const&, std::string const&) (unittest.cpp:183)
==32699==    by 0x15EBF8D: mongo::dbtests::runDbTests(int, char**, std::string) (framework.cpp:254)
==32699==    by 0xF03134: main (dbtests.cpp:29)
==32699==  Address 0x6cf8be0 is 14 bytes after a block of size 34 free'd
==32699==    at 0x4A06336: operator delete(void*) (vg_replace_malloc.c:457)
==32699==    by 0x3AB6AA2FD2: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (basic_string.h:235)
==32699==    by 0x1651995: mongo::ConnectionString::connect(std::string&, double) const (dbclient.cpp:87)
==32699==    by 0x14D1E77: mongo::DBConnectionPool::get(std::string const&, double) (connpool.cpp:197)
==32699==    by 0x12300C1: mongo::ScopedDbConnection::ScopedDbConnection(std::string const&, double) (connpool.h:205)
==32699==    by 0x15AA081: mongo::ChunkManager::calcInitSplitsAndShards(mongo::Shard const&, std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> > const*, std::vector<mongo::Shard, std::allocator<mongo::Shard> > const*, std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> >*, std::vector<mongo::Shard, std::allocator<mongo::Shard> >*) const (chunk.cpp:913)
==32699==    by 0x15AA5CD: mongo::ChunkManager::createFirstChunks(std::string const&, mongo::Shard const&, std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> > const*, std::vector<mongo::Shard, std::allocator<mongo::Shard> > const*) (chunk.cpp:960)
==32699==    by 0x1142872: ShardingTests::ChunkManagerCreateBasicTest::run() (sharding.cpp:101)
==32699==    by 0x114AE3E: mongo::unittest::TestHolderBase<ShardingTests::ChunkManagerCreateBasicTest>::run() (unittest.h:86)
==32699==    by 0x1220DC0: mongo::unittest::Suite::run(std::string const&) (unittest.cpp:121)
==32699==    by 0x1221511: mongo::unittest::Suite::run(std::vector<std::string, std::allocator<std::string> > const&, std::string const&) (unittest.cpp:183)
==32699==    by 0x15EBF8D: mongo::dbtests::runDbTests(int, char**, std::string) (framework.cpp:254)
==32699== 



 Comments   
Comment by auto [ 08/Jun/12 ]

Author:

{u'login': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-5918 better CUSTOM ConnectionString type for testing, fixes casting problems
Branch: master
https://github.com/mongodb/mongo/commit/eb6e62a8eb28ec87f30dbaad0ffecee8f89d4e27

Comment by Greg Studer [ 24/May/12 ]

Probably fastest actually to keep the current override and use ConnectionString::OTHER to indicate. Otherwise we'll need pluggable connection types, which is a pretty big change.

Comment by Greg Studer [ 24/May/12 ]

Thanks for tracking this down - the issue is that, for testing, it's useful to replace the connections to shards with connections to the local DB. May need to create a separate kind of connection string.

Comment by auto [ 24/May/12 ]

Author:

{u'login': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-5918 disable memory-corrupting test until it can be fixed properly
Branch: master
https://github.com/mongodb/mongo/commit/7c608e90c32c33e2a39e29ca88e0ce3fccddf480

Comment by Eric Milkie [ 24/May/12 ]

Looks like the commit d1fd245ea8ec1a75974169fc0f07d21dc3217a96 for SERVER-4262 either exposed a bug or created a bug.

Comment by Eric Milkie [ 24/May/12 ]

Findings:

    void ScopedDbConnection::_setSocketTimeout(){
        if( ! _conn ) return;
        if( _conn->type() == ConnectionString::MASTER )
            (( DBClientConnection* ) _conn)->setSoTimeout( _socketTimeout );

The C style cast is causing memory corruption due to a bug where somehow a DBDirectClient connection is unexpectedly getting added to the connection pool.
If a dynamic_cast had been used, the program could have aborted instead when it detected that _conn was not a DBClientConnection*.

It remains to be discovered why the DBDirectClient* was added to the pool.

Generated at Thu Feb 08 03:10:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.