[SERVER-13624] mongorestore causes mongod crashing Created: 17/Apr/14  Updated: 10/Dec/14  Resolved: 02/May/14

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: 2.4.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kang Jiang Assignee: Matt Kangas
Resolution: Duplicate Votes: 0
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Snow Leopard 10.6.8


Attachments: File fs.chunks.bson     File fs.chunks.metadata.json     File fs.files.bson     File fs.files.metadata.json    
Issue Links:
Duplicate
duplicates SERVER-11650 Unittests freeing unallocated thread ... Closed
Operating System: OS X
Participants:

 Description   

Mongod crashed when mongorestoring fs.chunk collection, all the other collections restored successfully.

mongod log:

mongod(93427,0x1046c2000) malloc: *** error for object 0x102800000: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Thu Apr 17 11:35:11.570 Got signal: 6 (Abort trap).
 
Thu Apr 17 11:35:11.575 Backtrace:
0x10006d99b 0x1000b5fdb 0x7fff849491ba 0 0x1000b9242 0x1000bbd91 0x1000bc4ad 0x1000c3a29 0x1000c5a29 0x1000b5304 0x3 
 0   mongod                              0x000000010006d99b _ZN5mongo15printStackTraceERSo + 43
 1   mongod                              0x00000001000b5fdb _ZN5mongo10abruptQuitEi + 987
 2   libSystem.B.dylib                   0x00007fff849491ba _sigtramp + 26
 3   ???                                 0x0000000000000000 0x0 + 0
 4   mongod                              0x00000001000b9242 _ZN5mongo6listenEi + 482
 5   mongod                              0x00000001000bbd91 _ZN5mongo14_initAndListenEi + 1457
 6   mongod                              0x00000001000bc4ad _ZN5mongo13initAndListenEi + 29
 7   mongod                              0x00000001000c3a29 _ZL11mongoDbMainiPPcS0_ + 29561
 8   mongod                              0x00000001000c5a29 main + 9
 9   mongod                              0x00000001000b5304 start + 52
 10  ???                                 0x0000000000000003 0x0 + 3

There's a similar report at mongo_user group



 Comments   
Comment by Matt Kangas [ 02/May/14 ]

Jiang, sorry for the confusion. When I asked for SHA1 I meant "git object name", not SHA1 of the tarball.

I just realized that, since you're using v2.4.8, the problem is definitely SERVER-11650. The underlying issue is https://code.google.com/p/gperftools/issues/detail?id=581 . We disabled debugallocation on darwin in commit 2b66528 which first shipped in r2.5.4. It has not been backported to v2.4.

The correct workaround on v2.4 is as we discussed above.

  • Build without --dd
  • Or build with --allocator=system
Comment by Kang Jiang [ 30/Apr/14 ]

Hi Matt

I think this sha1 is the checksum of the source package archive. Because homebrew doesn't download the source from your git repo. It downloads from your official site.

The mongodb formula:

class Mongodb < Formula
  homepage 'http://www.mongodb.org/'
  url 'http://downloads.mongodb.org/src/mongodb-src-r2.4.8.tar.gz'
  sha1 '59fa237e102c9760271df9433ee7357dd0ec831f'
......

There is no dependency on gperftools in the mongodb formula and I checked that I haven't installed gperftools using homebrew on my machine.

Comment by Matt Kangas [ 29/Apr/14 ]

59fa237e102c9760271df9433ee7357dd0ec831f is not a valid commit id in the mongodb/mongo repository.

Please check if the Homebrew formula is using Homebrew's version of gperftools, rather than our patched version in src/third_party. If so, this would explain the issue you're seeing.

Comment by Kang Jiang [ 29/Apr/14 ]

Hi Matt

OK. This issue happens when I (a) build from source and (b) happens in both debug and release version.

I first built it using homebrew. I check the sha1 in formula, it's '59fa237e102c9760271df9433ee7357dd0ec831f'.

Now I try to build with the command "scons all --allocator=system", and it's OK. The problem is gone.

Comment by Matt Kangas [ 28/Apr/14 ]

Jiang, you mentioned this before but I'd like to confirm: you only see this issue (a) when you build v2.4.8 yourself and (b) it is a debug build?

Which SHA1 are you building against? (Tag r2.4.8 is a350fc38922fbda2cec8d5dd842237b904eafc14)

To help isolate this further, can you try the following and see if you can still reproduce the failure?

  • Compile an optimized build (remove "--dd")
  • Compile with the system allocator, not tcmalloc (add "--allocator=system")

Currently the oldest OS X system I have available to test against is the following, which is Mac OS X 10.6.8 but a newer Xcode. Notably the compiler is Clang, not GCC.

$ system_profiler SPSoftwareDataType | grep 'System' && g++ --version
    System Software Overview:
      System Version: Mac OS X 10.6.8 (10K549)
i686-apple-darwin10-llvm-g++-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)
Xcode 4.2
Build version 4C199

Comment by Kang Jiang [ 26/Apr/14 ]

Hi, Matt

Yes. I'm using OS X Snow Leopard 10.6.8.
I used "scons all --dd" to build the debug version.

"g++ --version" yields

i686-apple-darwin10-g++-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3)

"xcodebuild -version" yields

Xcode 3.2.6
Component versions: DevToolsCore-1809.0; DevToolsSupport-1806.0
BuildVersion: 10M2518

Comment by Matt Kangas [ 26/Apr/14 ]

Ah, I see you specified the OS X version: Snow Leopard 10.6.8

Comment by Matt Kangas [ 26/Apr/14 ]

Hi Jiang,

Can you provide the full and exact "scons" invocation that you used to build mongod?

Also:

  • What OS X version? ("uname -a")
  • Which compiler? (output of "g++ --version" and "xcodebuild -version")

SERVER-11650 (version 2.5.4) resulted in a failure message similar to yours on OS X debug builds. Recently we patched our vendored gperftools to resolve it in SERVER-13528 (version 2.7.0).

Comment by Kang Jiang [ 21/Apr/14 ]

I built a debug version of mongod 2.4.8.

And when I tried to run mongorestore, the mongod crashed with the following calling stack:

mongod(917,0x105518000) malloc: *** error for object 0x103800000: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
 
Program received signal SIGABRT, Aborted.
0x00007fff8492c932 in select$DARWIN_EXTSN ()
(gdb) bt full
#0  0x00007fff8492c932 in select$DARWIN_EXTSN ()
No symbol table info available.
#1  0x00000001005544c0 in mongo::Listener::initAndListen (this=0x105128148) at src/mongo/util/net/listen.cpp:228
	fds = {{
    fds_bits = {1536, 0 <repeats 31 times>}
  }}
	ret = 0
	socks = {
  <std::_Vector_base<int,std::allocator<int> >> = {
    _M_impl = {
      <std::allocator<int>> = {
        <__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, 
      members of std::_Vector_base<int,std::allocator<int> >::_Vector_impl: 
      _M_start = 0x10510e0b8, 
      _M_finish = 0x10510e0c0, 
      _M_end_of_storage = 0x10510e0c0
    }
  }, <No data fields>}
	maxfd = 10
	maxSelectTime = {
  tv_sec = 0, 
  tv_usec = 10000
}
#2  0x000000010055a2bd in mongo::PortMessageServer::run (this=0x105128140) at src/mongo/util/net/message_server_port.cpp:129
No locals.
#3  0x00000001000d7f44 in mongo::listen (port=27017) at src/mongo/db/db.cpp:286
	options = {
  port = 27017, 
  ipList = {
    static npos = 18446744073709551615, 
    _M_dataplus = {
      <std::allocator<char>> = {
        <__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, 
      members of std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Alloc_hider: 
      _M_p = 0x7fff70ed8518 ""
    }
  }
}
	server = ('mongo::MessageServer' *) 0x105128140
#4  0x00000001000d8d63 in mongo::_initAndListen (listenPort=27017) at src/mongo/db/db.cpp:674
	is32bit = false
	missingRepl = 0
#5  0x00000001000d8d8f in mongo::initAndListen (listenPort=27017) at src/mongo/db/db.cpp:684
No locals.
#6  0x00000001000de8f2 in mongoDbMain (argc=1, argv=0x7fff5fbff070, envp=0x7fff5fbff080) at src/mongo/db/db.cpp:1295
	staticObserver = {
  <boost::noncopyable_::noncopyable> = {<No data fields>}, 
  members of mongo::StaticObserver: 
  static _destroyingStatics = false
}
#7  0x00000001000de937 in main (argc=1, argv=0x7fff5fbff070, envp=0x7fff5fbff080) at src/mongo/db/db.cpp:727
	exitCode = 32767

Comment by Kang Jiang [ 19/Apr/14 ]

I downloaded the pre-built binary version(2.6.0) from official site and mongorestore works pretty fine.

Is there anything wrong in my building environment? This bug happens in both 2.4.8 and 2.6.0 if built from source.

Generated at Thu Feb 08 03:32:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.