|
Sorry, that's what I meant.
12-19 was large and correct.
01-07 was blank.
Blank wouldn't be caused by a crash, so not sure where that came from.
|
|
Really? Here the 12-19 hexdump was huge (and the corresponding DB works), only the 01-07 was small.
Are you sure they were both small?
|
|
Those files are all 0 except for the file header.
That's what you would get if you started with a clean database.
The .ns file looks correct, but the .0 and .1 are from an empty db.
If those were from a kill -9, we might expect some garbage data, but nothing would 0 the files out.
|
|
Done - files are hexdump-2010-12-19 and hexdump-2011-01-07 on your SCP server.
|
|
By blank I mean the files have the file header, but the rest of the data is 0s.
Any sort of crash can't zero out the entire file.
Can you do
hexdump imeveryone.0
and send the output
|
|
Both files were created after stopping the DB process. They're not blank here - a tar listing is below - do the files at your end match?
(imeveryone)mike imeveryone $ tar -tzvf corruptdb-2011-01-07.tgz
drwxr-xr-x 0 mike staff 0 7 Jan 14:43 db/
rw------ 0 mike staff 128335872 7 Jan 14:43 db/imeveryone.0
rw------ 0 mike staff 134217728 7 Jan 14:43 db/imeveryone.1
rw------ 0 mike staff 16777216 7 Jan 14:43 db/imeveryone.ns
-rwxr-xr-x 0 mike staff 6 7 Jan 14:43 db/mongod.lock
(imeveryone)mike imeveryone $ tar -tzvf db-2010-12-19.tgz
drwxr-xr-x 0 mike staff 0 7 Jan 16:14 db/
rw------ 0 mike staff 67108864 7 Jan 16:15 db/imeveryone.0
rw------ 0 mike staff 134217728 7 Jan 16:15 db/imeveryone.1
rw------ 0 mike staff 16777216 7 Jan 16:15 db/imeveryone.ns
|
|
Those files are all 0 except for the .ns files.
Are you sure those are the correct files?
|
|
Can you confirm when the corruptdb files where created?
Were those taken right after the crash?
|
|
Thanks Eliot,
I've reuploaded the files:
Corrupt DB is : corruptdb-2011-01-07.tgz
Working backup is : db-2010-12-19.tgz
mongod on the server where the problem happened was 1.4.4 on Linux, per
mongo-users I've tried repair tools all the way up to 1.7.4 on OS X.
Mike
|
|
Thanks Eliot,
I've reuploaded the files:
Corrupt DB is : corruptdb-2011-01-07.tgz
Working backup is : db-2010-12-19.tgz
mongod on the server where the problem happened was 1.4.4 on Linux, per mongo-users I've tried repair tools all the way up to 1.7.4 on OS X.
Mike
|
|
@mike, also, what version were you on?
|
|
@mike, if you could upload an old backup and absolutely any logs would help a lot.
also, can you tell us exactly what the files upload were.
something seems off of about them that I wouldn't expect.
|
|
@mike, an older backup would help trying to understand the issue.
|
|
FYI I have the same issue. In my case, I've tried repairing the DB in Mongo on both Ubuntu and OS X, with the same result.
I have SCPd a tarball to www.10gen.com:corruptdb.tgz . Any help would be sorely appreciated!
|
|
How big is the .tgz?
How big is the raw data?
Can you try --repair on a 64-bit machine?
To send us privately you can:
scp -P 722 <file> <any username>@www.10gen.com:
<any password>
then comment with the username you used
|
|
Other databases are failing, too:
$ mongod --repair
Wed Oct 27 13:24:09 MongoDB starting : pid=68923 port=27017 dbpath=/data/db/ 32-bit
Wed Oct 27 13:24:09 db version v1.6.4, pdfile version 4.5
Wed Oct 27 13:24:09 git version: nogitversion
Wed Oct 27 13:24:09 sys info: Darwin whitestar.local 9.8.0 Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_44
Wed Oct 27 13:24:09 [initandlisten] ****
Wed Oct 27 13:24:09 [initandlisten] ****
Wed Oct 27 13:24:09 [initandlisten] need to upgrade database devise-test-suite-mongoid with pdfile version 4.5, new version: 4.5
Wed Oct 27 13:24:09 [initandlisten] starting upgrade
Wed Oct 27 13:24:09 [initandlisten] devise-test-suite-mongoid repairDatabase devise-test-suite-mongoid
Wed Oct 27 13:24:09 [initandlisten] devise-test-suite-mongoid.system.namespaces Assertion failure magic == 0x41424344 db/pdfile.h 241
0x774d1 0x8689c 0x2becf9 0x2ae50d 0x2b1112 0x2c8c0c 0x1610c7 0x2cdf97 0x2d2514 0x2d63cb 0x2d6f73 0x2d7e08 0x14b9d2 0x286414 0x291e2a 0x292b9a 0xbbbf4 0x9e6b9 0x2553ed 0x25753e
0 mongod 0x000774d1 _ZN5mongo12sayDbContextEPKc + 193
1 mongod 0x0008689c _ZN5mongo8assertedEPKcS1_j + 300
2 mongod 0x002becf9 _ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE + 137
3 mongod 0x002ae50d _ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE + 61
4 mongod 0x002b1112 _ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE + 146
5 mongod 0x002c8c0c _ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi + 1228
6 mongod 0x001610c7 _ZN5mongo11UserQueryOp5_initEv + 407
7 mongod 0x002cdf97 _ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE + 311
8 mongod 0x002d2514 _ZN5mongo12QueryPlanSet6Runner3runEv + 788
9 mongod 0x002d63cb _ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE + 667
10 mongod 0x002d6f73 _ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE + 99
11 mongod 0x002d7e08 _ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE + 40
12 mongod 0x0014b9d2 ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1 + 3714
13 mongod 0x00286414 _ZN5mongo16getDatabaseNamesERSt6vectorISsSaISsEERKSs + 1636
14 mongod 0x00291e2a _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE + 5594
15 mongod 0x00292b9a _ZN5mongo14DBDirectClient4callERNS_7MessageES2_b + 122
16 mongod 0x000bbbf4 _ZN5mongo14DBClientCursor4initEv + 244
17 mongod 0x0009e6b9 _ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii + 601
18 mongod 0x002553ed _ZN5mongo6Cloner2goEPKcRSsRKSsbbbb + 797
19 mongod 0x0025753e _ZN5mongo9cloneFromEPKcRSsRKSsbbbb + 78
Wed Oct 27 13:24:09 [initandlisten] assertion 0 assertion db/pdfile.h:241 ns:devise-test-suite-mongoid.system.namespaces query:{}
Wed Oct 27 13:24:09 [initandlisten] Assertion: 10290:bad system.namespaces object
{ $err: "assertion db/pdfile.h:241" }
0x85b9f 0x2557eb 0x25753e 0x2a9bda 0x3d052f 0x3d43b4 0x3d6e7f 0x3d77ba 0x3d932f 0x1c16 0x2
0 mongod 0x00085b9f _ZN5mongo11msgassertedEiPKc + 367
1 mongod 0x002557eb _ZN5mongo6Cloner2goEPKcRSsRKSsbbbb + 1819
2 mongod 0x0025753e _ZN5mongo9cloneFromEPKcRSsRKSsbbbb + 78
3 mongod 0x002a9bda _ZN5mongo14repairDatabaseESsRSsbb + 1562
4 mongod 0x003d052f _ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE + 111
5 mongod 0x003d43b4 _ZN5mongo15repairDatabasesEv + 1156
6 mongod 0x003d6e7f _ZN5mongo14_initAndListenEiPKc + 1071
7 mongod 0x003d77ba _ZN5mongo13initAndListenEiPKc + 42
8 mongod 0x003d932f main + 4735
9 mongod 0x00001c16 start + 54
10 ??? 0x00000002 0x0 + 2
Wed Oct 27 13:24:09 [initandlisten] exception in initAndListen std::exception: bad system.namespaces object
{ $err: "assertion db/pdfile.h:241" }
, terminating
Wed Oct 27 13:24:09 dbexit:
Wed Oct 27 13:24:09 [initandlisten] shutdown: going to close listening sockets...
Wed Oct 27 13:24:09 [initandlisten] shutdown: going to flush oplog...
Wed Oct 27 13:24:09 [initandlisten] shutdown: going to close sockets...
Wed Oct 27 13:24:09 [initandlisten] shutdown: waiting for fs preallocator...
Wed Oct 27 13:24:09 [initandlisten] shutdown: closing all files...
Wed Oct 27 13:24:09 closeAllFiles() finished
Wed Oct 27 13:24:09 [initandlisten] shutdown: removing fs lock...
Wed Oct 27 13:24:09 dbexit: really exiting now
|
|
It fails with 1.6.4, too. What's the best way to get a tar.gz of the databases to you privately?
$ mongod --repair
Wed Oct 27 13:20:52 MongoDB starting : pid=68706 port=27017 dbpath=/data/db/ 32-bit
Wed Oct 27 13:20:52 db version v1.6.4, pdfile version 4.5
Wed Oct 27 13:20:52 git version: nogitversion
Wed Oct 27 13:20:52 sys info: Darwin whitestar.local 9.8.0 Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_44
Wed Oct 27 13:20:52 [initandlisten] ****
Wed Oct 27 13:20:52 [initandlisten] ****
Wed Oct 27 13:20:52 [initandlisten] need to upgrade database benchmark-gridfs with pdfile version 4.5, new version: 4.5
Wed Oct 27 13:20:52 [initandlisten] starting upgrade
Wed Oct 27 13:20:52 [initandlisten] benchmark-gridfs repairDatabase benchmark-gridfs
Wed Oct 27 13:20:52 [initandlisten] benchmark-gridfs.system.namespaces Assertion failure magic == 0x41424344 db/pdfile.h 241
0x774d1 0x8689c 0x2becf9 0x2ae50d 0x2b1112 0x2c8c0c 0x1610c7 0x2cdf97 0x2d2514 0x2d63cb 0x2d6f73 0x2d7e08 0x14b9d2 0x286414 0x291e2a 0x292b9a 0xbbbf4 0x9e6b9 0x2553ed 0x25753e
0 mongod 0x000774d1 _ZN5mongo12sayDbContextEPKc + 193
1 mongod 0x0008689c _ZN5mongo8assertedEPKcS1_j + 300
2 mongod 0x002becf9 _ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE + 137
3 mongod 0x002ae50d _ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE + 61
4 mongod 0x002b1112 _ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE + 146
5 mongod 0x002c8c0c _ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi + 1228
6 mongod 0x001610c7 _ZN5mongo11UserQueryOp5_initEv + 407
7 mongod 0x002cdf97 _ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE + 311
8 mongod 0x002d2514 _ZN5mongo12QueryPlanSet6Runner3runEv + 788
9 mongod 0x002d63cb _ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE + 667
10 mongod 0x002d6f73 _ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE + 99
11 mongod 0x002d7e08 _ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE + 40
12 mongod 0x0014b9d2 ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1 + 3714
13 mongod 0x00286414 _ZN5mongo16getDatabaseNamesERSt6vectorISsSaISsEERKSs + 1636
14 mongod 0x00291e2a _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE + 5594
15 mongod 0x00292b9a _ZN5mongo14DBDirectClient4callERNS_7MessageES2_b + 122
16 mongod 0x000bbbf4 _ZN5mongo14DBClientCursor4initEv + 244
17 mongod 0x0009e6b9 _ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii + 601
18 mongod 0x002553ed _ZN5mongo6Cloner2goEPKcRSsRKSsbbbb + 797
19 mongod 0x0025753e _ZN5mongo9cloneFromEPKcRSsRKSsbbbb + 78
Wed Oct 27 13:20:52 [initandlisten] assertion 0 assertion db/pdfile.h:241 ns:benchmark-gridfs.system.namespaces query:{}
Wed Oct 27 13:20:52 [initandlisten] Assertion: 10290:bad system.namespaces object
{ $err: "assertion db/pdfile.h:241" }
0x85b9f 0x2557eb 0x25753e 0x2a9bda 0x3d052f 0x3d43b4 0x3d6e7f 0x3d77ba 0x3d932f 0x1c16 0x2
0 mongod 0x00085b9f _ZN5mongo11msgassertedEiPKc + 367
1 mongod 0x002557eb _ZN5mongo6Cloner2goEPKcRSsRKSsbbbb + 1819
2 mongod 0x0025753e _ZN5mongo9cloneFromEPKcRSsRKSsbbbb + 78
3 mongod 0x002a9bda _ZN5mongo14repairDatabaseESsRSsbb + 1562
4 mongod 0x003d052f _ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE + 111
5 mongod 0x003d43b4 _ZN5mongo15repairDatabasesEv + 1156
6 mongod 0x003d6e7f _ZN5mongo14_initAndListenEiPKc + 1071
7 mongod 0x003d77ba _ZN5mongo13initAndListenEiPKc + 42
8 mongod 0x003d932f main + 4735
9 mongod 0x00001c16 start + 54
10 ??? 0x00000002 0x0 + 2
Wed Oct 27 13:20:52 [initandlisten] exception in initAndListen std::exception: bad system.namespaces object
{ $err: "assertion db/pdfile.h:241" }
, terminating
Wed Oct 27 13:20:52 dbexit:
Wed Oct 27 13:20:52 [initandlisten] shutdown: going to close listening sockets...
Wed Oct 27 13:20:52 [initandlisten] shutdown: going to flush oplog...
Wed Oct 27 13:20:52 [initandlisten] shutdown: going to close sockets...
Wed Oct 27 13:20:52 [initandlisten] shutdown: waiting for fs preallocator...
Wed Oct 27 13:20:52 [initandlisten] shutdown: closing all files...
Wed Oct 27 13:20:52 closeAllFiles() finished
Wed Oct 27 13:20:52 [initandlisten] shutdown: removing fs lock...
Wed Oct 27 13:20:52 dbexit: really exiting now
|
|
Can you try with 1.6.4
If that fails - it would be great if we could get the raw data.
|
Generated at Thu Feb 08 02:58:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.