[SERVER-14048] mongodump against mongos can't send dump to standard output Created: 26/May/14  Updated: 11/Jul/16  Resolved: 13/Jun/14

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: 2.6.1
Fix Version/s: 2.6.4, 2.7.2

Type: Bug Priority: Major - P3
Reporter: Kay Agahd Assignee: J Rassi
Resolution: Done Votes: 0
Labels: community-team
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

64 bit, wheezy


Issue Links:
Depends
Duplicate
Related
related to TOOLS-5 mongodump --out - removes many inform... Closed
Operating System: Linux
Backport Completed:
Sprint: Server 2.7.2
Participants:

 Description   
Issue Status as of Jun 13, 2014

ISSUE SUMMARY
When used against mongos, mongodump can't sent the dump data to standard output.

USER IMPACT
mongodump will appear to hang waiting for data and the dump will never finish.

WORKAROUNDS
Send the dump data to a directory instead:

mongodump ... --out dump.dir

AFFECTED VERSIONS
The mongodump tool from MongoDB 2.6 production releases up to version 2.6.3 is affected by this issue.

FIX VERSION
The fix is included in the 2.6.4 production release.

RESOLUTION DETAILS
The logic mongodump uses to determine whether to query the database with "exhaust mode" was performing a read from an improperly-initialized variable. The initialization was corrected.

Original description

mongodump got stuck without any message. As the dump file didn't grow anymore, I had to kill the mongodump process.

mongodump --host sx547:27017 -u admin -p XXXX --authenticationDatabase admin --db offerStore --collection offer --query '{_id:{$lte:540867465}}' -vvvvv --out - > chunk01.bson

I had a look into the dumped chunk01.bson file using bsondump, which showed an error at the end. Is it because I killed mongodump or is it the reason that mongodump never terminated?

2014-05-25T23:37:36.051+0200 Assertion failure amt == (size_t)( size - 4 ) src/mongo/tools/tool.cpp 330
2014-05-25T23:37:36.056+0200 0x11a8571 0x114bff9 0x112fe3e 0x112234d 0x75c3cb 0x112135d 0x1124497 0x1124f32 0x7f5d905aaead 0x75bad9
 bsondump(_ZN5mongo15printStackTraceERSo+0x21) [0x11a8571]
 bsondump(_ZN5mongo10logContextEPKc+0x159) [0x114bff9]
 bsondump(_ZN5mongo12verifyFailedEPKcS1_j+0x17e) [0x112fe3e]
 bsondump(_ZN5mongo8BSONTool11processFileERKN5boost11filesystem34pathE+0x99d) [0x112234d]
 bsondump(_ZN8BSONDump5doRunEv+0x9b) [0x75c3cb]
 bsondump(_ZN5mongo8BSONTool3runEv+0x10d) [0x112135d]
 bsondump(_ZN5mongo4Tool4mainEiPPcS2_+0xb7) [0x1124497]
 bsondump(main+0x42) [0x1124f32]
 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7f5d905aaead]
 bsondump() [0x75bad9]
assertion: 0 assertion src/mongo/tools/tool.cpp:330


The same procedure using mongoexport worked well. The json file was finished within seconds:

mongoexport --host sx547:27017 -u admin -p XXXX --authenticationDatabase admin --db offerStore --collection offer --query '{_id:{$lte:540867465}}' --out - > chunk01.json



 Comments   
Comment by Githook User [ 16/Jun/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-14048 mongodump can't use exhaust mode with mongos

(cherry picked from commit a6793da36b0026c9b0ef50228f40472b31b5c76d)
Branch: v2.6
https://github.com/mongodb/mongo/commit/75da8d92f3d4ba4d72b3c05c06fd933721fb1230

Comment by Githook User [ 16/Jun/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-14048 Remove unnecessary member variables from Dump

(cherry picked from commit 435ff367fac3e02427aba5129e1f103ec603eb1e)
Branch: v2.6
https://github.com/mongodb/mongo/commit/9e74eeb7c3598dc50f42d605163cd2366c2b3dab

Comment by Githook User [ 16/Jun/14 ]

Author:

{u'username': u'ehershey', u'name': u'Ernie Hershey', u'email': u'ernie.hershey@10gen.com'}

Message: Revert "SERVER-14048 Remove unnecessary member variables from Dump"

This reverts commit eed2eeacd17559b5d708920483b6c00dcc1ec21d.
Branch: v2.6
https://github.com/mongodb/mongo/commit/91025943ace37749d176bb3c66ae256a229236b7

Comment by Githook User [ 16/Jun/14 ]

Author:

{u'username': u'ehershey', u'name': u'Ernie Hershey', u'email': u'ernie.hershey@10gen.com'}

Message: Revert "SERVER-14048 mongodump can't use exhaust mode with mongos"

This reverts commit 6a193b142b32d954d97f16a63308815e551ed885.
Branch: v2.6
https://github.com/mongodb/mongo/commit/854863bfcf9bed1b3ccbbf430f02e266ff6ff7cc

Comment by Githook User [ 16/Jun/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-14048 mongodump can't use exhaust mode with mongos

(cherry picked from commit a6793da36b0026c9b0ef50228f40472b31b5c76d)
Branch: v2.6
https://github.com/mongodb/mongo/commit/6a193b142b32d954d97f16a63308815e551ed885

Comment by Githook User [ 16/Jun/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-14048 Remove unnecessary member variables from Dump

(cherry picked from commit 435ff367fac3e02427aba5129e1f103ec603eb1e)
Branch: v2.6
https://github.com/mongodb/mongo/commit/eed2eeacd17559b5d708920483b6c00dcc1ec21d

Comment by Githook User [ 13/Jun/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-14048 mongodump can't use exhaust mode with mongos
Branch: master
https://github.com/mongodb/mongo/commit/a6793da36b0026c9b0ef50228f40472b31b5c76d

Comment by Githook User [ 13/Jun/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-14048 Remove unnecessary member variables from Dump
Branch: master
https://github.com/mongodb/mongo/commit/435ff367fac3e02427aba5129e1f103ec603eb1e

Comment by Kay Agahd [ 28/May/14 ]

Ok, no problem, I've sent you the logs via scp -P 722 mongodump_<server>.log SERVER-14048@www.mongodb.com:
The shards are as follows:

shards:
        {  "_id" : "offerStoreIT",  "host" : "offerStoreIT/s163:27017,s167:27017,s171:27017" }
        {  "_id" : "offerStoreIT2",  "host" : "offerStoreIT2/s551:27017,s552:27017,s553:27017" }


Primary of shard1 is s167 and primary of shard2 is s551. The router is sx547.

Here is the mongdump command, so you see the timestamp:

root@sx547:/home/admin# mongodump --host sx547:27017 -u admin -p XXXX --authenticationDatabase admin --db offerStore --collection offer --query '{_id:{$lte:540867465}}' -vvvvv --out - > chunk01.bson
2014-05-29T00:51:16.527+0200 creating new connection to:sx547:27017
2014-05-29T00:51:16.528+0200 [ConnectBG] BackgroundJob starting: ConnectBG
2014-05-29T00:51:16.529+0200 connected to server sx547:27017 (172.16.66.17)
2014-05-29T00:51:16.529+0200 connected connection!
connected to: sx547:27017
^C
root@sx547:/home/admin#

Comment by Ramon Fernandez Marina [ 28/May/14 ]

kay.agahd@idealo.de, I don't think we'll need the logs to track this down, but I could be mistaken – we'll let you know if that's the case (feel free to upload them anyway if you have them at hand).

Comment by Kay Agahd [ 28/May/14 ]

Dan and Ramon, I confirm that --out dumpdir works.
Do you still need the logs from mongos and mongod or is this obsolete since you've found the reason (stdout)?
Here come the collection stats (btw. the collection is called offer, the database is called offerStore, hence db.offer.stats():

mongos> use offerStore
switched to db offerStore
mongos> db.offer.stats()
{
        "sharded" : true,
        "systemFlags" : 1,
        "userFlags" : 1,
        "ns" : "offerStore.offer",
        "count" : 88372057,
        "numExtents" : 156,
        "size" : 230068466352,
        "storageSize" : 246133119296,
        "totalIndexSize" : 34935410272,
        "indexSizes" : {
                "_id_" : 2821938224,
                "asin_1" : 694592080,
                "bokey" : 5533688496,
                "categoryBokey" : 6052472048,
                "clickCount" : 2078396432,
                "importGroup_1_importId_1_missingSince_1" : 866206320,
                "mappedCatalogCategory" : 3422269200,
                "missingSince" : 2533554352,
                "onlineProductIds" : 2423423632,
                "shopId_1__id_1" : 3856651904,
                "shopId_1_merchantId_1" : 4652217584
        },
        "avgObjSize" : 2603.407391003697,
        "nindexes" : 11,
        "nchunks" : 4811,
        "shards" : {
                "offerStoreIT" : {
                        "ns" : "offerStore.offer",
                        "count" : 51561396,
                        "size" : 135425641600,
                        "avgObjSize" : 2626,
                        "storageSize" : 148823682016,
                        "numExtents" : 90,
                        "nindexes" : 11,
                        "lastExtentSize" : 2146426864,
                        "paddingFactor" : 1,
                        "systemFlags" : 1,
                        "userFlags" : 1,
                        "totalIndexSize" : 17182248272,
                        "indexSizes" : {
                                "_id_" : 1398987184,
                                "bokey" : 2907328368,
                                "categoryBokey" : 2816329488,
                                "mappedCatalogCategory" : 1668991408,
                                "clickCount" : 948800272,
                                "missingSince" : 1300899712,
                                "onlineProductIds" : 1154998992,
                                "shopId_1_merchantId_1" : 2137541616,
                                "asin_1" : 463726368,
                                "importGroup_1_importId_1_missingSince_1" : 473913664,
                                "shopId_1__id_1" : 1910731200
                        },
                        "ok" : 1
                },
                "offerStoreIT2" : {
                        "ns" : "offerStore.offer",
                        "count" : 36810661,
                        "size" : 94642824752,
                        "avgObjSize" : 2571,
                        "storageSize" : 97309437280,
                        "numExtents" : 66,
                        "nindexes" : 11,
                        "lastExtentSize" : 2146426864,
                        "paddingFactor" : 1,
                        "systemFlags" : 1,
                        "userFlags" : 1,
                        "totalIndexSize" : 17753162000,
                        "indexSizes" : {
                                "_id_" : 1422951040,
                                "bokey" : 2626360128,
                                "categoryBokey" : 3236142560,
                                "mappedCatalogCategory" : 1753277792,
                                "clickCount" : 1129596160,
                                "missingSince" : 1232654640,
                                "onlineProductIds" : 1268424640,
                                "shopId_1_merchantId_1" : 2514675968,
                                "asin_1" : 230865712,
                                "importGroup_1_importId_1_missingSince_1" : 392292656,
                                "shopId_1__id_1" : 1945920704
                        },
                        "ok" : 1
                }
        },
        "ok" : 1
}
mongos>

Comment by Ramon Fernandez Marina [ 27/May/14 ]

kay.agahd@idealo.de, we're observing the behavior you describe when using mongodump against mongos, but only when sending the dump to the standard output. We're investigating this issue; in the meantime please continue to use mongoexport, or specify a directory name for the output (instead of using the standard output):

mongodump ... --out chunk01.dump

Comment by Daniel Pasette (Inactive) [ 27/May/14 ]
  • Instead of sending the dump output to stdout, specify a dump location using --out dumpdir and the mongodump informational output will go to stdout/stderr.
  • For the log output, you will have to capture it from mongos and the mongod processes.
  • Also, can you post the collection stats, i.e., db.offerStore.stats()
Comment by Kay Agahd [ 27/May/14 ]

1) Yes, I can reproduce it.
2) Where mongodump writes its logs? Even with -vvvv I don't see any output. Which server log do you need? I'm running mongodump on the router (mongos). Do you need the logs from the mongos or from the primaries or secondaries of the shards?

Comment by Daniel Pasette (Inactive) [ 27/May/14 ]

That error msg indicates that the size of the document in the bson header didn't match up with the size of the document on disk. It's not possible to say for sure if the problem was caused by killing the process, but seems likely.

You say the job finished with mongoexport in seconds.

  • Are you able to repeat the hang when running mongodump again?
  • Can you post the server and mongodump log?
Generated at Thu Feb 08 03:33:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.