[SERVER-3845] Process aborted on initial sync when trying to access it via HTTP console Created: 14/Sep/11  Updated: 04/Feb/15  Resolved: 23/Sep/11

Status: Closed
Project: Core Server
Component/s: HTTP Console
Affects Version/s: 2.0.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Aristarkh Zagorodnikov Assignee: Tony Hannan
Resolution: Duplicate Votes: 2
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux shaula 2.6.32-32-server #62-Ubuntu SMP Wed Apr 20 22:07:43 UTC 2011 x86_64 GNU/Linux


Issue Links:
Duplicate
duplicates SERVER-3907 Broken pipe of HTTP client lets mongo... Closed
Operating System: Linux
Participants:

 Description   

Stopped secondary, removed any data from it (to resync from primary), restarted, tried to monitor its progress using HTTP console, got the following.
Restarting after crash allowed it to continue with resyncing and eventually becoming secondary properly.

Wed Sep 14 11:35:06 [initandlisten] MongoDB starting : pid=32663 port=27017 dbpath=/mnt/disk1/mongodb 64-bit host=shaula
Wed Sep 14 11:35:06 [initandlisten] db version v2.0.0, pdfile version 4.5
Wed Sep 14 11:35:06 [initandlisten] git version: 695c67dff0ffc361b8568a13366f027caa406222
Wed Sep 14 11:35:06 [initandlisten] build info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST
Wed Sep 14 11:35:06 [initandlisten] options: { bind_ip: "l1-1.shaula.bru,c1.shaula.bru,127.0.0.1", config: "/etc/mongodb.conf", dbpa
Wed Sep 14 11:35:06 [initandlisten] journal dir=/mnt/disk1/mongodb/journal
Wed Sep 14 11:35:06 [initandlisten] recover : no journal files present, no recovery needed
Wed Sep 14 11:35:06 [rsStart] couldn't connect to localhost:27017: couldn't connect to server localhost:27017
Wed Sep 14 11:35:06 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Wed Sep 14 11:35:06 [rsStart] replSet info you may need to run replSetInitiate – rs.initiate() in the shell – if that is not alrea
Wed Sep 14 11:35:06 [websvr] admin web console waiting for connections on port 28017
Wed Sep 14 11:35:06 [initandlisten] waiting for connections on port 27017
Wed Sep 14 11:35:07 [initandlisten] connection accepted from 10.1.1.24:55354 #1
Wed Sep 14 11:35:07 [initandlisten] connection accepted from 10.1.1.23:45047 #2
Wed Sep 14 11:35:07 [initandlisten] connection accepted from 10.1.1.22:56325 #3
Wed Sep 14 11:35:07 [initandlisten] connection accepted from 10.1.1.25:41100 #4
Wed Sep 14 11:35:16 [initandlisten] connection accepted from 127.0.0.1:43383 #5
Wed Sep 14 11:35:16 [rsStart] trying to contact l1-1.perseus.bru
Wed Sep 14 11:35:16 [rsStart] trying to contact l1-1.perseus.bru:27017
Wed Sep 14 11:35:16 [rsStart] trying to contact l1-1.sheratan.bru:27017
Wed Sep 14 11:35:16 [rsStart] replSet got config version 6 from a remote, saving locally
Wed Sep 14 11:35:16 [rsStart] replSet info saving a newer config version to local.system.replset
Wed Sep 14 11:35:16 [FileAllocator] allocating new datafile /mnt/disk1/mongodb/local/local.ns, filling with zeroes...
Wed Sep 14 11:35:16 [FileAllocator] creating directory /mnt/disk1/mongodb/local/_tmp
Wed Sep 14 11:35:16 [FileAllocator] done allocating datafile /mnt/disk1/mongodb/local/local.ns, size: 16MB, took 0.025 secs
Wed Sep 14 11:35:16 [FileAllocator] allocating new datafile /mnt/disk1/mongodb/local/local.0, filling with zeroes...
Wed Sep 14 11:35:16 [FileAllocator] done allocating datafile /mnt/disk1/mongodb/local/local.0, size: 64MB, took 0.083 secs
Wed Sep 14 11:35:16 [FileAllocator] allocating new datafile /mnt/disk1/mongodb/local/local.1, filling with zeroes...
Wed Sep 14 11:35:16 [rsStart] replSet saveConfigLocally done
Wed Sep 14 11:35:16 [rsStart] replSet STARTUP2
Wed Sep 14 11:35:16 [rsSync] ******
Wed Sep 14 11:35:16 [rsSync] creating replication oplog of size: 3380MB...
Wed Sep 14 11:35:16 [FileAllocator] done allocating datafile /mnt/disk1/mongodb/local/local.1, size: 128MB, took 0.182 secs
Wed Sep 14 11:35:16 [FileAllocator] allocating new datafile /mnt/disk1/mongodb/local/local.2, filling with zeroes...
Wed Sep 14 11:35:18 [rsHealthPoll] replSet info member l1-1.sheratan.bru:27017 is up
Wed Sep 14 11:35:18 [rsHealthPoll] replSet member l1-1.sheratan.bru:27017 is now in state PRIMARY
Wed Sep 14 11:35:18 [rsHealthPoll] replSet info member l1-1.perseus.bru:27017 is up
Wed Sep 14 11:35:18 [rsHealthPoll] replSet member l1-1.perseus.bru:27017 is now in state ARBITER
Wed Sep 14 11:35:19 [FileAllocator] done allocating datafile /mnt/disk1/mongodb/local/local.2, size: 2047MB, took 2.934 secs
Wed Sep 14 11:35:19 [FileAllocator] allocating new datafile /mnt/disk1/mongodb/local/local.3, filling with zeroes...
Wed Sep 14 11:35:51 [FileAllocator] done allocating datafile /mnt/disk1/mongodb/local/local.3, size: 2047MB, took 32.025 secs
Wed Sep 14 11:36:20 [websvr] Socket http response send() errno:104 Connection reset by peer 10.4.8.101:58023
Wed Sep 14 11:36:20 terminate() called, printing stack:
0xa834ed 0x7fd2145f8d16 0x7fd2145f8d43 0x7fd2145f8e3e 0x5a55cf 0x65029c 0x5ae424 0x652e5e 0xa9ea30 0x7fd2148489ca 0x7fd213df770d
/usr/bin/mongod(_ZN5mongo11myterminateEv+0x4d) [0xa834ed]
/usr/lib/libstdc++.so.6(+0xcad16) [0x7fd2145f8d16]
/usr/lib/libstdc++.so.6(+0xcad43) [0x7fd2145f8d43]
/usr/lib/libstdc++.so.6(+0xcae3e) [0x7fd2145f8e3e]
/usr/bin/mongod(ZN5mongo6Socket4sendEPKciS2+0x2ef) [0x5a55cf]
/usr/bin/mongod(_ZN5mongo13MiniWebServer8acceptedENS_6SocketE+0x4cc) [0x65029c]
/usr/bin/mongod(_ZN5mongo8Listener13initAndListenEv+0x494) [0x5ae424]
/usr/bin/mongod(_ZN5mongo15webServerThreadEPKNS_11AdminAccessE+0x9e) [0x652e5e]
/usr/bin/mongod(thread_proxy+0x80) [0xa9ea30]
/lib/libpthread.so.0(+0x69ca) [0x7fd2148489ca]
/lib/libc.so.6(clone+0x6d) [0x7fd213df770d]
Wed Sep 14 11:36:20 Got signal: 6 (Aborted).

Wed Sep 14 11:36:20 Backtrace:
0xa83fc9 0x7fd213d44af0 0x7fd213d44a75 0x7fd213d485c0 0xa835bb 0x7fd2145f8d16 0x7fd2145f8d43 0x7fd2145f8e3e 0x5a55cf 0x65029c 0x5ae4
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
/lib/libc.so.6(+0x33af0) [0x7fd213d44af0]
/lib/libc.so.6(gsignal+0x35) [0x7fd213d44a75]
/lib/libc.so.6(abort+0x180) [0x7fd213d485c0]
/usr/bin/mongod(_ZN5mongo11myterminateEv+0x11b) [0xa835bb]
/usr/lib/libstdc++.so.6(+0xcad16) [0x7fd2145f8d16]
/usr/lib/libstdc++.so.6(+0xcad43) [0x7fd2145f8d43]
/usr/lib/libstdc++.so.6(+0xcae3e) [0x7fd2145f8e3e]
/usr/bin/mongod(ZN5mongo6Socket4sendEPKciS2+0x2ef) [0x5a55cf]
/usr/bin/mongod(_ZN5mongo13MiniWebServer8acceptedENS_6SocketE+0x4cc) [0x65029c]
/usr/bin/mongod(_ZN5mongo8Listener13initAndListenEv+0x494) [0x5ae424]
/usr/bin/mongod(_ZN5mongo15webServerThreadEPKNS_11AdminAccessE+0x9e) [0x652e5e]
/usr/bin/mongod(thread_proxy+0x80) [0xa9ea30]
/lib/libpthread.so.0(+0x69ca) [0x7fd2148489ca]
/lib/libc.so.6(clone+0x6d) [0x7fd213df770d]

Logstream::get called in uninitialized state
Wed Sep 14 11:36:20 ERROR: Client::shutdown not called: websvr



 Comments   
Comment by Chris Soyars [ 21/Sep/11 ]

I've got this problem as well...

Tue Sep 20 22:06:18 [websvr] Socket http response send() errno:32 Broken pipe 172.16.202.208:56272
Tue Sep 20 22:06:18 terminate() called, printing stack:
0xa834ed 0x3bb64bce16 0x3bb64bce43 0x3bb64bcf2a 0x5a55cf 0x65029c 0x5ae424 0x652e5e 0xa9ea30 0x3bb580673d 0x3bb50d44bd
/usr/bin/mongod(_ZN5mongo11myterminateEv+0x4d) [0xa834ed]
/usr/lib64/libstdc++.so.6 [0x3bb64bce16]
/usr/lib64/libstdc++.so.6 [0x3bb64bce43]
/usr/lib64/libstdc++.so.6 [0x3bb64bcf2a]
/usr/bin/mongod(ZN5mongo6Socket4sendEPKciS2+0x2ef) [0x5a55cf]
/usr/bin/mongod(_ZN5mongo13MiniWebServer8acceptedENS_6SocketE+0x4cc) [0x65029c]
/usr/bin/mongod(_ZN5mongo8Listener13initAndListenEv+0x494) [0x5ae424]
/usr/bin/mongod(_ZN5mongo15webServerThreadEPKNS_11AdminAccessE+0x9e) [0x652e5e]
/usr/bin/mongod(thread_proxy+0x80) [0xa9ea30]
/lib64/libpthread.so.0 [0x3bb580673d]
/lib64/libc.so.6(clone+0x6d) [0x3bb50d44bd]
Tue Sep 20 22:06:18 Got signal: 6 (Aborted).

Comment by Tony Hannan [ 19/Sep/11 ]

I see the bug in mongo/util/net/miniwebserver.cpp line 169:

> sock.send( response.c_str(), response.size() , "http response" );

This line is not protected by an exception handler. Socket.send throws a SocketException if the send failed (in this case because connection was reset by peer).

Generated at Thu Feb 08 03:04:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.