[SERVER-6989] mongodump segfaults when host is passed a replSet string Created: 10/Sep/12  Updated: 11/Jul/16  Resolved: 10/Dec/12

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: 2.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: siddharth.singh@10gen.com Assignee: Tad Marshall
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-7801 Potential access to freed memory by T... Closed
Duplicate
duplicates SERVER-7006 mongo tools seg fault in 2.2.0 with r... Closed
Related
is related to SERVER-7612 explicit primary read pref does not ... Closed
Operating System: ALL
Participants:

 Description   

When mongodump is run against a replica set string :

./mongodump --host foo/host1,host2 it throws a segmentation fault. See below :

sid@ferrari:~/10gen/mongo$ ./mongodump --host foo/ferrari:31000,ferrari:31001
Mon Sep 10 13:11:12 starting new replica set monitor for replica set foo with seed of ferrari:31000,ferrari:31001
Mon Sep 10 13:11:12 successfully connected to seed ferrari:31000 for replica set foo
Mon Sep 10 13:11:12 changing hosts to { 0: "ferrari:31000", 1: "ferrari:31001" } from foo/
Mon Sep 10 13:11:12 trying to add new host ferrari:31000 to replica set foo
Mon Sep 10 13:11:12 successfully connected to new host ferrari:31000 in replica set foo
Mon Sep 10 13:11:12 trying to add new host ferrari:31001 to replica set foo
Mon Sep 10 13:11:12 successfully connected to new host ferrari:31001 in replica set foo
Mon Sep 10 13:11:12 Primary for replica set foo changed to ferrari:31000
Mon Sep 10 13:11:12 replica set monitor for replica set foo started, address is foo/ferrari:31000,ferrari:31001
Mon Sep 10 13:11:12 [ReplicaSetMonitorWatcher] starting
connected to: foo/ferrari:31000,ferrari:31001
Mon Sep 10 13:11:12 all dbs
Logstream::get called in uninitialized state
Mon Sep 10 13:11:12 deleting replica set monitor for: foo/ferrari:31000,ferrari:31001
Segmentation fault



 Comments   
Comment by Tad Marshall [ 10/Dec/12 ]

The repro cases in this ticket have been resolved in other tickets. The remaining known issues are described in SERVER-7801.

Comment by Tad Marshall [ 30/Nov/12 ]

SERVER-7801 is the root cause of the crash in git commit 70361a4125edeeae8427bae122ac361a1c6c2697.

Comment by Tad Marshall [ 29/Nov/12 ]

The commit that fixed this crasher was:

Author:

{u'date': u'2012-11-12T19:07:44Z', u'email': u'randolph@10gen.com', u'name': u'Randolph Tan'}

Message: SERVER-7612 explicit primary read pref does not work well with shard versioning
Ensure that selectNodeUsingTags will use the same connection to the primary with checkMaster
Branch: master
https://github.com/mongodb/mongo/commit/53ffd6d8e442a8d09569845e02e9dee528950481

Comment by Tad Marshall [ 28/Nov/12 ]

Testing the specific commit that Gianfranco mentioned above, I can reproduce a crash ... but I can't with today's code. I'll try to find the specific commit that fixed it.

So, this appears to be fixed in master, but we don't know what fixed it so we can't backport the fix until we find which commit we need.

Comment by Tad Marshall [ 28/Nov/12 ]

Unable to duplicate in Windows 2.3.1.

C:\Users\Tad>"Documents\Visual Studio 2010\Projects\mongodev\mongodump.exe" --db test -c test --host sh1/Tad-W7-VBox:30011,Tad-W7-VBox:30012 -o dump\test-sh1 -vvvvv
Wed Nov 28 14:23:30.533 starting new replica set monitor for replica set sh1 with seed of Tad-W7-VBox:30011,Tad-W7-VBox:30012
Wed Nov 28 14:23:30.549 BackgroundJob starting: ConnectBG
Wed Nov 28 14:23:30.565 successfully connected to seed Tad-W7-VBox:30011 for replica set sh1
Wed Nov 28 14:23:30.580 ReplicaSetMonitor::_checkConnection: Tad-W7-VBox:30011 { setName: "sh1", ismaster: true, secondary: false, hosts: [ "Tad-W7-VBox:30011", "Tad-W7-VBox:30012" ], primary: "Tad-W7-VBox:30011", me: "Tad-W7-VBox:30011", maxBsonObjectSize: 16777216, localTime: new Date(1354130610580), ok: 1.0 }
Wed Nov 28 14:23:30.580 changing hosts to { 0: "Tad-W7-VBox:30011", 1: "Tad-W7-VBox:30012" } from sh1/
Wed Nov 28 14:23:30.580 trying to add new host Tad-W7-VBox:30011 to replica set sh1
Wed Nov 28 14:23:30.580 BackgroundJob starting: ConnectBG
Wed Nov 28 14:23:30.596 successfully connected to new host Tad-W7-VBox:30011 in replica set sh1
Wed Nov 28 14:23:30.596 trying to add new host Tad-W7-VBox:30012 to replica set sh1
Wed Nov 28 14:23:30.612 BackgroundJob starting: ConnectBG
Wed Nov 28 14:23:30.612 successfully connected to new host Tad-W7-VBox:30012 in replica set sh1
Wed Nov 28 14:23:30.627 creating new connection to:Tad-W7-VBox:30011
Wed Nov 28 14:23:30.627 BackgroundJob starting: ConnectBG
Wed Nov 28 14:23:30.627 connected connection!
Wed Nov 28 14:23:30.643 dbclient_rs nodes[0].ok = true Tad-W7-VBox:30011
Wed Nov 28 14:23:30.643 dbclient_rs nodes[1].ok = true Tad-W7-VBox:30012
Wed Nov 28 14:23:30.643 _check : sh1/Tad-W7-VBox:30011,Tad-W7-VBox:30012
Wed Nov 28 14:23:30.658 ReplicaSetMonitor::_checkConnection: Tad-W7-VBox:30011 { setName: "sh1", ismaster: true, secondary: false, hosts: [ "Tad-W7-VBox:30011", "Tad-W7-VBox:30012" ], primary: "Tad-W7-VBox:30011", me: "Tad-W7-VBox:30011", maxBsonObjectSize: 16777216, localTime: new Date(1354130610658), ok: 1.0 }
Wed Nov 28 14:23:30.674 dbclient_rs nodes[0].ok = true Tad-W7-VBox:30011
Wed Nov 28 14:23:30.674 dbclient_rs nodes[1].ok = true Tad-W7-VBox:30012
Wed Nov 28 14:23:30.674 Primary for replica set sh1 changed to Tad-W7-VBox:30011
Wed Nov 28 14:23:30.674 ReplicaSetMonitor::_checkConnection: Tad-W7-VBox:30011 { setName: "sh1", ismaster: true, secondary: false, hosts: [ "Tad-W7-VBox:30011", "Tad-W7-VBox:30012" ], primary: "Tad-W7-VBox:30011", me: "Tad-W7-VBox:30011", maxBsonObjectSize: 16777216, localTime: new Date(1354130610674), ok: 1.0 }
Wed Nov 28 14:23:30.674 dbclient_rs nodes[0].ok = true Tad-W7-VBox:30011
Wed Nov 28 14:23:30.690 dbclient_rs nodes[1].ok = true Tad-W7-VBox:30012
Wed Nov 28 14:23:30.690 ReplicaSetMonitor::_checkConnection: Tad-W7-VBox:30012 { setName: "sh1", ismaster: false, secondary: true, hosts: [ "Tad-W7-VBox:30012", "Tad-W7-VBox:30011" ], primary: "Tad-W7-VBox:30011", me: "Tad-W7-VBox:30012", maxBsonObjectSize: 16777216, localTime: new Date(1354130610690), ok: 1.0 }
Wed Nov 28 14:23:30.690 creating new connection to:Tad-W7-VBox:30012
Wed Nov 28 14:23:30.690 BackgroundJob starting: ConnectBG
Wed Nov 28 14:23:30.705 connected connection!
Wed Nov 28 14:23:30.721 dbclient_rs nodes[0].ok = true Tad-W7-VBox:30011
Wed Nov 28 14:23:30.721 dbclient_rs nodes[1].ok = true Tad-W7-VBox:30012
Wed Nov 28 14:23:30.721 replica set monitor for replica set sh1 started, address is sh1/Tad-W7-VBox:30011,Tad-W7-VBox:30012
connected to: sh1/Tad-W7-VBox:30011,Tad-W7-VBox:30012
Wed Nov 28 14:23:30.721 BackgroundJob starting: ReplicaSetMonitorWatcher
Wed Nov 28 14:23:30.768 BackgroundJob starting: ConnectBG
Wed Nov 28 14:23:30.768 [ReplicaSetMonitorWatcher] starting
Wed Nov 28 14:23:30.799 dbclient_rs _selectNode found local secondary for queries: 1, ping time: 0
Wed Nov 28 14:23:30.799 BackgroundJob starting: ConnectBG
Wed Nov 28 14:23:30.830 DATABASE: test   to     dump\test-sh1\test
Wed Nov 28 14:23:30.830         test.test to dump\test-sh1\test\test.bson
Wed Nov 28 14:23:30.846                  42 objects
Wed Nov 28 14:23:30.846         Metadata for test.test to dump\test-sh1\test\test.metadata.json

Duplicated in Windows 2.2.2:

C:\Users\Tad>"Documents\Visual Studio 2010\Projects\mongo22\mongodump.exe" --host sh2/Tad-W7-VBox:30021,Tad-W7-VBox:30022-vvvvv
Wed Nov 28 14:47:22 starting new replica set monitor for replica set sh2 with seed of Tad-W7-VBox:30021,Tad-W7-VBox:30022
Wed Nov 28 14:47:22 successfully connected to seed Tad-W7-VBox:30021 for replica set sh2
Wed Nov 28 14:47:22 changing hosts to { 0: "Tad-W7-VBox:30021", 1: "Tad-W7-VBox:30022" } from sh2/
Wed Nov 28 14:47:22 trying to add new host Tad-W7-VBox:30021 to replica set sh2
Wed Nov 28 14:47:22 successfully connected to new host Tad-W7-VBox:30021 in replica set sh2
Wed Nov 28 14:47:22 trying to add new host Tad-W7-VBox:30022 to replica set sh2
Wed Nov 28 14:47:22 successfully connected to new host Tad-W7-VBox:30022 in replica set sh2
Wed Nov 28 14:47:22 Primary for replica set sh2 changed to Tad-W7-VBox:30021
Wed Nov 28 14:47:22 replica set monitor for replica set sh2 started, address is sh2/Tad-W7-VBox:30021,Tad-W7-VBox:30022
Wed Nov 28 14:47:22 [ReplicaSetMonitorWatcher] starting
connected to: sh2/Tad-W7-VBox:30021,Tad-W7-VBox:30022-vvvvv
Wed Nov 28 14:47:22 all dbs
Wed Nov 28 14:47:22 DATABASE: test       to     dump/test
Wed Nov 28 14:47:22     test.test to dump/test/test.bson
Wed Nov 28 14:47:22 doing snapshot query
Wed Nov 28 14:47:22              58 objects
Wed Nov 28 14:47:22     Metadata for test.test to dump/test/test.metadata.json
Wed Nov 28 14:47:22 DATABASE: admin      to     dump/admin
_DEBUG warning trying to lock a mongo::mutex during static shutdown
_DEBUG warning trying to lock a mongo::mutex during static shutdown
_DEBUG warning trying to lock a mongo::mutex during static shutdown
Logstream::get called in uninitialized state
_DEBUG warning trying to lock a mongo::mutex during static shutdown
Wed Nov 28 14:47:22 deleting replica set monitor for: sh2/Tad-W7-VBox:30021,Tad-W7-VBox:30022

Unhandled exception at 0x000000013fe35a2d in mongodump.exe: 0xC0000005: Access violation reading location 0xffffffffffffffff.

>Debug.ListCallStack
 Index  Function
--------------------------------------------------------------------------------
*1      mongodump.exe!std::_Tree<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::vector<mongo::HostAndPort,std::allocator<mongo::HostAndPort> >,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,std::vector<mongo::HostAndPort,std::allocator<mongo::HostAndPort> > > >,0> >::_Lbound(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & _Keyval="sh2") 
 2      mongodump.exe!std::_Tree<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::vector<mongo::HostAndPort,std::allocator<mongo::HostAndPort> >,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,std::vector<mongo::HostAndPort,std::allocator<mongo::HostAndPort> > > >,0> >::lower_bound(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & _Keyval="sh2") 
 3      mongodump.exe!std::map<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::vector<mongo::HostAndPort,std::allocator<mongo::HostAndPort> >,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,std::vector<mongo::HostAndPort,std::allocator<mongo::HostAndPort> > > > >::operator[](const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & _Keyval="sh2") 
 4      mongodump.exe!mongo::ReplicaSetMonitor::_cacheServerAddresses_inlock() 
 5      mongodump.exe!mongo::ReplicaSetMonitor::~ReplicaSetMonitor() 
 6      mongodump.exe!mongo::ReplicaSetMonitor::`scalar deleting destructor'() 
 7      mongodump.exe!boost::checked_delete<mongo::ReplicaSetMonitor>(mongo::ReplicaSetMonitor * x=0x00000000004f8f80) 
 8      mongodump.exe!boost::detail::sp_counted_impl_p<mongo::ReplicaSetMonitor>::dispose() 
 9      mongodump.exe!boost::detail::sp_counted_base::release() 
 10     mongodump.exe!boost::detail::shared_count::~shared_count() 
 11     mongodump.exe!boost::shared_ptr<mongo::ReplicaSetMonitor>::~shared_ptr<mongo::ReplicaSetMonitor>() 
 12     mongodump.exe!std::_Pair_base<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> >::~_Pair_base<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> >() 
 13     mongodump.exe!std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> >::~pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> >() 
 14     mongodump.exe!std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> >::`scalar deleting destructor'() 
 15     mongodump.exe!std::_Destroy<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > >(std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > * _Ptr=0x00000000004dca38 ("sh2", {px=0x00000000004f8f80 pn={...} })) 
 16     mongodump.exe!std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > >::destroy(std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > * _Ptr=0x00000000004dca38 ("sh2", {px=0x00000000004f8f80 pn={...} })) 
 17     mongodump.exe!std::_Dest_val<std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > >,std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > >(std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > > & _Alval={...}, std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > * _Pdest=0x00000000004dca38 ("sh2", {px=0x00000000004f8f80 pn={...} })) 
 18     mongodump.exe!std::_Tree<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,boost::shared_ptr<mongo::ReplicaSetMonitor>,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > >,0> >::_Erase(std::_Tree_nod<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,boost::shared_ptr<mongo::ReplicaSetMonitor>,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > >,0> >::_Node * _Rootnode=0x00000000004dca20) 
 19     mongodump.exe!std::_Tree<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,boost::shared_ptr<mongo::ReplicaSetMonitor>,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > >,0> >::clear() 
 20     mongodump.exe!std::_Tree<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,boost::shared_ptr<mongo::ReplicaSetMonitor>,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > >,0> >::erase(std::_Tree_const_iterator<std::_Tree_val<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,boost::shared_ptr<mongo::ReplicaSetMonitor>,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > >,0> > > * _First=0x00000000002bf7b8 ("sh2", {px=0x00000000004f8f80 pn={...} }), std::_Tree_const_iterator<std::_Tree_val<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,boost::sha
red_ptr<mongo::ReplicaSetMonitor>,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > >,0> > > * _Last=0x00000000002bf780 (<Bad Ptr>, {px=0xcdcdcdcdcdcdcdcd pn={...} })) 
 21     mongodump.exe!std::_Tree<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,boost::shared_ptr<mongo::ReplicaSetMonitor>,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > >,0> >::_Tidy() 
 22     mongodump.exe!std::_Tree<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,boost::shared_ptr<mongo::ReplicaSetMonitor>,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > >,0> >::~_Tree<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,boost::shared_ptr<mongo::ReplicaSetMonitor>,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > >,0> >() 
 23     mongodump.exe!std::map<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,boost::shared_ptr<mongo::ReplicaSetMonitor>,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > > >::~map<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,boost::shared_ptr<mongo::ReplicaSetMonitor>,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<mongo::ReplicaSetMonitor> > > >() 
 24     mongodump.exe!`dynamic atexit destructor for 'mongo::ReplicaSetMonitor::_sets''() 
 25     mongodump.exe!doexit(int code=0x00000000, int quick=0x00000000, int retcaller=0x00000000) 
 26     mongodump.exe!exit(int code=0x00000000) 
 27     mongodump.exe!__tmainCRTStartup() 
 28     mongodump.exe!mainCRTStartup() 
 29     kernel32.dll!BaseThreadInitThunk() 
 30     ntdll.dll!RtlUserThreadStart() 

This is the bug that was fixed in SERVER-6915 for 2.3.0, but it somehow was not marked for backport.

If we backport commit ca91de625f1b565f38621635f595326b70d30f37 this will be fixed (but we probably want the whole set of commits for SERVER-6915 and anything that was marked as a dup of it).

Generated at Thu Feb 08 03:13:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.