[SERVER-10427] segfault when calling mongo::ScopedDbConnection::getScopedDbConnection(connection_string) with replicaset Created: 04/Aug/13 Updated: 06/Aug/13 Resolved: 06/Aug/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Internal Client |
| Affects Version/s: | 2.4.1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Ingo Schramm | Assignee: | Tad Marshall |
| Resolution: | Done | Votes: | 0 |
| Labels: | connection, crash, driver, replicaset | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Linux Ubuntu, Redhat |
||
| Operating System: | Linux | ||||||||||||
| Steps To Reproduce: | With a little of bad luck - or Redhat (?) -, this code is sufficient.
|
||||||||||||
| Participants: |
| Description |
|
Hi, as a surprise I suffer from a segfault when I connect to a replicaset of MongoDB 2.4.3 with C++ driver 2.4.1. RS consists of 1 master and 2 slaves. The surprise comes from the fact that exactly the same code works well in most environments (Ubuntu, Redhat) and even in the above environment (Redhat) in another application against the same replicaset. Code (executed in a lambda):
Stacktrace:
You may note the libc-2.15 which I LD_PRELOAD because the test machine has another libc version than the build machine. Both application and driver are built against libc-2.15 of course and the preload does work well with other app which uses exactly the same code. Any hints appreciated. Cheers |
| Comments |
| Comment by Ingo Schramm [ 06/Aug/13 ] |
|
Obviously not. Just close it - and thanks again. But you could give the SConstruct patch mentioned above a try. Cheers, |
| Comment by Tad Marshall [ 06/Aug/13 ] |
|
Hi Ingo, Is this not a problem with MongoDB then? Should we close this ticket? Tad |
| Comment by Ingo Schramm [ 06/Aug/13 ] |
|
Sorry, my fault so far. I tracked the problem down to some shared library loaded not from where I expected it to be loaded. Thanks for your help anyways! Cheers |
| Comment by Ingo Schramm [ 05/Aug/13 ] |
|
Upgraded driver to 2.4.5, result is exactly the same. Running locally against RS on mongod 2.4.5 - no crash. Running remotely against RS on mongod 2.4.3 - crash. #0 0x00007ffff7b20d3d in inet_pton () from /home/ingo.schramm/mdbtest-pkg/lib/libc-2.15.so BTW I had to patch the SConstruct file line 85: from: boostLibs = ["thread", "filesystem", "system"] Cheers |
| Comment by Ingo Schramm [ 05/Aug/13 ] |
|
No, I do not touch ReplicaSetMonitor at all. Also, the crash happens before I'd get the chance to do so. I will fire local tests against newer versions of everything, but it may take some time to set it all up. Unfortunately, I cannot reproduce the crash locally (yet), only in the data center. Cheers |
| Comment by Tad Marshall [ 05/Aug/13 ] |
|
Hi Ingo, Thanks for all the answers! Do you call ReplicaSetMonitor::remove(), or use ReplicaSetMonitor directly in any way? It will be used "on your behalf" internally, but direct calls to it are capable of causing problems. You are right that several of the issues we've identified are related to exiting while active replica sets exist. We may have those issues fixed in the latest master branch code, but we also may still have remaining issues there. There are definitely issues of that type in the 2.4.1 and 2.4.3 versions. We have separate issues in one of our unit tests, but that unit test uses a "mock" to simulate a replica set and we haven't yet narrowed down the problem to a specific location. It could be in code that all replica sets use or it could be in the mock only. Tad |
| Comment by Ingo Schramm [ 05/Aug/13 ] |
|
Hi Tad! Yes, exactly 2.4.1 of the C++ driver with version 2.4.3 of MongoDB on the machine where it crashes. The installation is quite complex and quite big. I will give 2.4.5 of the driver a try today. This is quite a fun since we build against our own SDK and integrating scons into our CMake/make landscape is a task on its own. Build OS: Linux 3.2.0-51-generic #77-Ubuntu SMP Wed Jul 24 20:18:19 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux GCC g++ (GCC) 4.7.2 Build command like (test build): g++ -D_DEBUG -D_GNU_SOURCE -D_STDC_LIMIT_MACROS -D_STDC_CONSTANT_MACROS -finput-charset=UTF-8 -std=c++0x -Wall -Wextra -Werror -pedantic -O2 -g -gstrict-dwarf -fno-inline -fno-inline-functions -fPIC -fopenmp -isystem(SDK stuff) -I(Src stuff) C++ Driver is also build like that against SDK. Test OS: Linux 2.6.32-358.6.1.el6.x86_64 #1 SMP Fri Mar 29 16:51:51 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux All other libraries are SDK (namely boost 1.48.0). Yes, we use C++11 all over the place, makes life a lot easier. The issues you mentioned are mostly realated to thread destruction, right? As far as I see, here it is more realated to connection, when starting. The application in question is aimed to run forever. Indeed, it uses a lot of threading, and the driver is called from within a boost thread. The last (and only) log from the driver is like: Fri Aug 2 17:09:38.901 starting new replica set monitor for replica set rs-name with seed of host1:1001,host2:1001,host3:1001 Hope that may help. Cheers, |
| Comment by Tad Marshall [ 04/Aug/13 ] |
|
Hi Ingo, Thanks for the report. Are you using version 2.4.1 of the C++ driver with version 2.4.3 of the MongoDB server as you stated? What is the reason for using two different versions? This is probably unrelated to the issue you're hitting, but to reproduce your problem we should probably use the exact versions that you are using. Why not use the latest 2.4.5 version? You are using "auto" in your sample code, which suggests C++11. We have ongoing work to bring the codebase to perfect compatibility with C++11, but we currently do not use features of C++11 and we do not build in C++11 mode for our released versions. What compiler version are you using?
What is the command line that you are using to build your program (including all options you are specifying)? Can you give us the precise OS version you are using to test this code?
We have had some issues related to using replica sets from programs using the C++ driver: Thanks! Tad |