[SERVER-10590] can't run rs.status(), invalid utf-8 string Created: 21/Aug/13  Updated: 10/Dec/14  Resolved: 24/Aug/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.2.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: charity majors Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
Related
Operating System: ALL
Participants:

 Description   

Around 10:40 pm last night, all my mongo alerts went off for one of our shards because rs.status() is now returning junk.

parse:PRIMARY> rs.status()
Wed Aug 21 16:20:05 decode failed. probably invalid utf-8 string [syncThread: 10334 Invalid BSONObj size: 247791731 (0x7300C50E) first element: �@@timezone_offset_epoch_ms: ?type=-16]
Wed Aug 21 16:20:05 why: TypeError: malformed UTF-8 character sequence at offset 78
Wed Aug 21 16:20:05 TypeError: malformed UTF-8 character sequence at offset 78 src/mongo/shell/utils.js:1018

It's returning the same junk on the primary and all secondaries.

Any idea how to fix? It's basically impossible for us to run a repairDatabase on this shard due to the 1) size of the data and 2) resetting all the padding factors would kill us.

Thanks.



 Comments   
Comment by Daniel Pasette (Inactive) [ 24/Aug/13 ]

moved to community private ticket.

Comment by charity majors [ 21/Aug/13 ]

Any ideas for narrowing down the corrupted collection? I'm able to run db.stats() on all databases. Not finding anything in the log around that time. Happy to upload the logfile if you move this ticket to private.

Comment by charity majors [ 21/Aug/13 ]

Replication is also busted.

Wed Aug 21 16:24:03 [rsSync] Assertion: 10334:Invalid BSONObj size: 247791731 (0x7300C50E) first element: ¹^HÈ@@^Ptimezone_offset_epoch_ms: ?type=-16
0xb07561 0xacd449 0xacd5cc 0x5721e8 0x796fed 0x874ac4 0x6583e4 0x814697 0x814988 0x861ace 0x822b5b 0x9a42db 0x9a9a38 0x9ad318 0x9ad36a 0x9ad77c 0xb4d3d9 0x7f206b2cbe9a 0x7f206a5decbd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x99) [0xacd449]
/usr/bin/mongod() [0xacd5cc]
/usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x468) [0x5721e8]
/usr/bin/mongod(_ZN5mongo13unindexRecordEPNS_16NamespaceDetailsEPNS_6RecordERKNS_7DiskLocEb+0x11d) [0x796fed]
/usr/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPKcPNS_6RecordERKNS_7DiskLocEbbb+0x204) [0x874ac4]
/usr/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4b4) [0x6583e4]
/usr/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x814697]
/usr/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x814988]
/usr/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6e) [0x861ace]
/usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x822b5b]
/usr/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x9a42db]
/usr/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x388) [0x9a9a38]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x9ad318]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9ad36a]
/usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0xac) [0x9ad77c]
/usr/bin/mongod() [0xb4d3d9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f206b2cbe9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f206a5decbd]
Wed Aug 21 16:24:03 [rsSync] replSet syncThread: 10334 Invalid BSONObj size: 247791731 (0x7300C50E) first element: ¹^HÈ@@^Ptimezone_offset_epoch_ms: ?type=-16

Generated at Thu Feb 08 03:23:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.