[SERVER-6770] Unexpected shutdown Created: 15/Aug/12  Updated: 08/Mar/13  Resolved: 27/Nov/12

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 2.0.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Rosco Assignee: Mathias Stearn
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux mongo04 3.3.8-gentoo #4 SMP Wed Jul 18 13:04:16 MSK 2012 x86_64 Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz GenuineIntel GNU/Linux


Operating System: Linux
Participants:

 Description   

Wed Aug 15 09:21:10 [initandlisten] connection accepted from 10.20.30.12:50992 #8309
Wed Aug 15 09:27:34 [initandlisten] connection accepted from 10.20.30.22:46205 #8425
Wed Aug 15 09:28:17 [conn8438] authenticate:

{ authenticate: 1, user: "root", nonce: "e97451fc6d5afae7", key: "963507d7a5755183d9b8db6b164d2589" }

Wed Aug 15 09:28:17 [conn8438] auth: couldn't find user root, catalog_ru_front.system.users
Wed Aug 15 09:28:18 [conn8342] authenticate:

{ authenticate: 1, user: "root", nonce: "f972f5128f581f33", key: "2d41a1cd667473100afc7aa1e59bb74e" }

Wed Aug 15 09:28:18 [conn8342] auth: couldn't find user root, catalog_ru_front.system.users
Wed Aug 15 09:28:18 [conn8410] authenticate:

{ authenticate: 1, user: "root", nonce: "6a11bcc7ae506e7e", key: "2bc5033ccf036a313f3c2efaf86abb14" }

Wed Aug 15 09:28:18 [conn8410] auth: couldn't find user root, catalog_ru_front.system.users
Wed Aug 15 09:28:19 [conn8330] authenticate:

{ authenticate: 1, user: "root", nonce: "de4efc4e81d4b177", key: "027a55fb363f8215f10a592bc2e6dde5" }

Wed Aug 15 09:28:19 [conn8330] auth: couldn't find user root, catalog_ru_front.system.users
Wed Aug 15 09:28:19 [conn8249] authenticate:

{ authenticate: 1, user: "root", nonce: "26cdb8dab282c5f1", key: "1b46f5c138e3136fbdf6b848fff22fd1" }

Wed Aug 15 09:28:19 [conn8249] auth: couldn't find user root, catalog_ru_front.system.users
Wed Aug 15 09:28:19 [conn8330] authenticate:

{ authenticate: 1, user: "root", nonce: "7f739c3d81fd7cdf", key: "55e52b1a1498d185bcdf9174e595d141" }

Wed Aug 15 09:28:19 [conn8330] auth: couldn't find user root, catalog_ru_front.system.users
Wed Aug 15 09:28:19 [conn8430] authenticate:

{ authenticate: 1, user: "root", nonce: "f65f3fe1841b924e", key: "c248ef18d1e6913841ab992002f512c5" }

Wed Aug 15 09:28:19 [conn8430] auth: couldn't find user root, catalog_ru_front.system.users
Wed Aug 15 09:28:20 [conn8425] authenticate:

{ authenticate: 1, user: "root", nonce: "834599d720569d56", key: "face44461efecb52b8a92188bbef7c75" }

Wed Aug 15 09:28:20 [conn8425] auth: couldn't find user root, catalog_ru_front.system.users
Wed Aug 15 09:28:21 [conn8309] authenticate:

{ authenticate: 1, user: "root", nonce: "3100892acaeb2bf", key: "645efadcfc740aad24b7940f4c2d713d" }

Wed Aug 15 09:28:21 [conn8309] auth: couldn't find user root, catalog_ru_front.system.users
Wed Aug 15 09:28:21 [conn8425] authenticate:

{ authenticate: 1, user: "root", nonce: "f38a3165032e4190", key: "efebf5ea3a4bec02585fbcb826187280" }

Wed Aug 15 09:28:21 [conn8425] auth: couldn't find user root, catalog_ru_front.system.users
Wed Aug 15 09:28:21 Got signal: 6 (Aborted).
Wed Aug 15 09:28:21 Backtrace:
Wed Aug 15 09:28:21 [conn8309] ERROR: Client::~Client _context should be null but is not; client:conn
Wed Aug 15 09:28:21 [conn8309] ERROR: Client:hutdown not called: conn
Wed Aug 15 09:28:21 [conn8425] ERROR: Uncaught std::exception: std::bad_alloc, terminating
Wed Aug 15 09:28:21 dbexit:
Wed Aug 15 09:28:21 [conn8425] shutdown: going to close listening sockets...
Wed Aug 15 09:28:21 [conn8425] closing listening socket: 6
Wed Aug 15 09:28:21 [conn8425] closing listening socket: 7
Wed Aug 15 09:28:21 [conn8425] closing listening socket: 8
Wed Aug 15 09:28:21 [conn8425] removing socket file: /var/run/mongodb/mongodb-27017.sock
Wed Aug 15 09:28:21 [conn8425] shutdown: going to flush diaglog...
Wed Aug 15 09:28:21 [conn8425] shutdown: going to close sockets...
Wed Aug 15 09:28:21 [conn8425] shutdown: waiting for fs preallocator...
Wed Aug 15 09:28:21 [conn8425] shutdown: lock for final commit...
Wed Aug 15 09:28:21 [conn8425] shutdown: final commit...
Wed Aug 15 09:28:21 [conn8425] shutdown: closing all files...
Wed Aug 15 09:28:21 [conn8425] closeAllFiles() finished
Wed Aug 15 09:28:21 [conn8425] journalCleanup...
Wed Aug 15 09:28:21 [conn8425] removeJournalFiles



 Comments   
Comment by Mathias Stearn [ 28/Aug/12 ]

Are you using a gentoo build or our binaries? If it is the gentoo build, I'd suggest trying out binaries from http://www.mongodb.org/downloads. Also please try to include about ~100 lines before the stacktrace next time.

Also, it seems that both failures occur after attempts to log in as a non-existant user "root". Is there any chance this server is under attack or otherwise compromised? Is this server properly firewalled off from the public Internet?

Comment by Denis Bozhok [ 28/Aug/12 ]

It happens again. Now we have got a backtrace:
http://pastebin.com/CJjZiyzx

Comment by Denis Bozhok [ 23/Aug/12 ]

Sorry for delay.

"Has this happened before?"
No, it`s not.

"Could someone/something on the host issues a sigabort to the process?"
No, we have no such processes. We only have nrpe (nagios) daemon that checks connection to mongodb.

"What init script are you using and what user are you running mongod as?"
We use Gentoo based destro and using it init script.

"Are there any events at the same time in the system logs?"
Only nrpe checks, nothing more.

Comment by Scott Hernandez (Inactive) [ 15/Aug/12 ]

Has this happened before? Could someone/something on the host issues a sigabort to the process?

What init script are you using and what user are you running mongod as? Are there any events at the same time in the system logs?

Generated at Thu Feb 08 03:12:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.