[SERVER-9607] One of our secondaries crashed Created: 07/May/13  Updated: 10/Dec/14  Resolved: 15/May/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.2.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Jerry Hoffmeister Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: Linux
Participants:

 Description   

One of our secondaries crashed on Saturday and here's the log from the event:

Sat May 4 23:35:04 [initandlisten] connection accepted from 10.50.233.182:53504 #1002184 (39 connections now open)
Sat May 4 23:35:05 [conn13712] command admin.$cmd command:

{ writebacklisten: ObjectId('50d36dbc7e80d0454290d50e') }

ntoreturn:1 keyUpdates:0 reslen:44 300000ms
Sat May 4 23:35:16 [conn1002183] end connection 10.94.38.225:37345 (38 connections now open)
Sat May 4 23:35:16 [initandlisten] connection accepted from 10.94.38.225:37350 #1002185 (39 connections now open)
Sat May 4 23:35:22 [conn198198] command admin.$cmd command:

{ writebacklisten: ObjectId('50f4ae4c820d4bbfd39165d0') }

ntoreturn:1 keyUpdates:0 reslen:44 300000ms
Sat May 4 23:35:30 [initandlisten] connection accepted from 10.73.46.111:39610 #1002186 (40 connections now open)
Sat May 4 23:35:34 [conn1002184] end connection 10.50.233.182:53504 (39 connections now open)
Sat May 4 23:35:34 [initandlisten] connection accepted from 10.50.233.182:53509 #1002187 (40 connections now open)
Sat May 4 23:35:42 Invalid access at address: 0x7ee3a15e5000 from thread: conn643637

Sat May 4 23:35:42 Got signal: 7 (Bus error).

Sat May 4 23:35:43 Backtrace:
0xaffd31 0x558bb9 0x559142 0x7f1760cecc60 0x7f176002a4a6 0x670aa3 0x666efa 0x8318b3 0x7b2fc4 0x7b3f68 0x5703f2 0xaedfc1 0x7f1760ce3d8c 0x7f1760085c2d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x558bb9]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x559142]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) [0x7f1760cecc60]
/lib/x86_64-linux-gnu/libc.so.6(memcpy+0x46) [0x7f176002a4a6]
/usr/bin/mongod(_ZN5mongo22fillQueryResultFromObjERNS_11_BufBuilderINS_16TrivialAllocatorEEEPKNS_10ProjectionERKNS_7BSONObjEPKNS_12MatchDetailsEPKNS_7DiskLocE+0xd73) [0x670aa3]
/usr/bin/mongod(_ZNK5mongo12ClientCursor22fillQueryResultFromObjERNS_11_BufBuilderINS_16TrivialAllocatorEEEPKNS_12MatchDetailsE+0x18a) [0x666efa]
/usr/bin/mongod(_ZN5mongo14processGetMoreEPKcixRNS_5CurOpEiRb+0x573) [0x8318b3]
/usr/bin/mongod(_ZN5mongo15receivedGetMoreERNS_10DbResponseERNS_7MessageERNS_5CurOpE+0xd04) [0x7b2fc4]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x8b8) [0x7b3f68]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x5703f2]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xaedfc1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6d8c) [0x7f1760ce3d8c]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f1760085c2d]



 Comments   
Comment by Eliot Horowitz (Inactive) [ 15/May/13 ]

That indeed looks like a hardware issue.

If anything else comes up, please let us know.

Comment by Jerry Hoffmeister [ 13/May/13 ]

looks like this is a hardware issue... happened again and from the syslog:

200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)
May 12 00:12:31 RPTDB-RS05-Zuse1d-S01 kernel: [21251499.795988] end_request: I/O error, dev xvdb, sector 1450274624
May 12 00:12:36 RPTDB-RS05-Zuse1d-S01 init: mongodb main process (29674) terminated with status 14
May 12 00:15:01 RPTDB-RS05-Zuse1d-S01 CRON[5376]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Comment by Jerry Hoffmeister [ 07/May/13 ]

I should also add that I've restarted mongod and it appears to be re-syncing just fine.

Generated at Thu Feb 08 03:20:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.