[SERVER-2011] mongodump, bsondump and mongoexport should only write to stderr in case of a serious error Created: 26/Oct/10  Updated: 26/Oct/15  Resolved: 08/Apr/13

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: None
Fix Version/s: 2.5.0

Type: Improvement Priority: Minor - P4
Reporter: Micah Wedemeyer Assignee: Shaun Verch
Resolution: Done Votes: 8
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-9282 Audit tools for correct handling of l... Closed
Duplicate
is duplicated by SERVER-8106 bsondump writes progress to stdout Closed
Related
related to TOOLS-1200 Mongodump messages could not be separ... Closed
related to SERVER-9268 Use existing logging framework to sta... Closed
is related to SERVER-3112 mongo* tools logs should not log to s... Closed
is related to SERVER-8713 bsondump output interferes with progr... Closed
is related to TOOLS-152 Tools should provide way to redirect ... Accepted
is related to DOCS-1982 Document: --quiet option Closed
Participants:

 Description   

I'm the "author" of the AutoMongoDump tool on github (github.com/
micahwedemeyer/automongobackup). I basically took the very nice
AutoMySQLDump bash script and modified it for use with MongoDB.
Thanks, by the way, for making the mongodump CLI so similar to
mysqldump. I assume that wasn't an accident.

Anyways, it seems that every time mongodump runs, it writes some debug
output (like "connected to: 127.0.0.1:27017") to stderr. The
automysqldump script is set up to count anything written to stderr as
a serious error and alert the user that the dump has probably failed.
I've hacked around this for my mongo port, assuming that everything in
stderr is just warning messages, not fatal errors. I tried playing
with analyzing the output to stderr a little, but it seemed like a bad
road to head down.

I'd like to request that if anyone chooses to refactor or work on
mongodump, please modify it only write to stderr if a serious error
occurs. That way I can update the script to alert the user in case of
serious errors.



 Comments   
Comment by Shaun Verch [ 08/Apr/13 ]

The --quiet option now will silence all informational messages, which should account for this use case.

Comment by auto [ 08/Apr/13 ]

Author:

{u'date': u'2013-04-03T22:43:32Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}

Message: SERVER-2011 Added --quiet option to tools
Branch: master
https://github.com/mongodb/mongo/commit/ac39ed282dc9610a115d09cbf88cd1bdf0fa3ba6

Comment by auto [ 14/Dec/12 ]

Author:

{u'date': u'2012-10-30T21:52:08Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}

Message: SERVER-2011 Prevent mongoexport from writing to a file called '-'
Branch: master
https://github.com/mongodb/mongo/commit/fa652ca77fa7eac38282a03412c133f7464fb8db

Comment by auto [ 14/Dec/12 ]

Author:

{u'date': u'2012-10-30T20:59:09Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}

Message: SERVER-2011 Fix handling of stdout vs stderr in mongoexport
Branch: master
https://github.com/mongodb/mongo/commit/e842f8ca919017ef455b1099a6cd369ae83b895b

Comment by Eliot Horowitz (Inactive) [ 03/Jun/12 ]

This is mostly done - but want to keep open until we audit everything and make sure everything is standardized across all tools.

Comment by Nils Toedtmann [ 27/Apr/12 ]

Well as a sysadmin what you often need for automation is dump tools which

  • spit out the result on stdout for further processing (e.g. filter, compress, mail, ...) without having to handle temporary files
  • do report real errors on stderr so i get notified that a dump failed and i have to do something about it
  • do not report non-errors which clutter my root mail with false-positives

E.g. there should be a way of saying this:

mongodump  --host example.net  --show-only-real-errors-on-stderr-but-no-supplementary-information  --out - | gzip -c | uuencode -m - | mail -s "mongo backup `hostname` `date +%Y%m%d`" backup-archive@example.com

Comment by Krzysztof Wilczynski [ 26/Apr/12 ]

Hi,

So...

When having "quiet" and/or "silent" flags, then we have to rely on exit code, which following what is already known to work well would be 0 for success and 1 (or any non-zero) for failure.

Now, when "out" flag points to a file name, then supplementary info should go on standard output and errors on standard error, including correct exit code being set. Of course, "quiet" will still silence any supplementary information as well as errors form being shown.

When "out" equals to "-", then both supplementary information and errors should go on standard error (since standard out is now taken). Appropriate exit code still has to be set, plus "quiet" flag when set would still suppress printing any extraneous information (on standard error this time).

All of this would make something like that possible:

mongodump --quiet --host example.net --out a.dump || echo 'Velociraptor ate my database!!!11eleven'

My 2p. Hope that helps

KW

Comment by Nils Toedtmann [ 26/Apr/12 ]

Thank you for taking care of this!

Flags like -q|--quite or -s|--silent are common. I'd prefer it the other 
way round though: don't print out any non-errors on stderr unless explicitly 
told so with something like -v|--verbose or --debug.

(sorry, need to use 'noformat' escaping for markup to not strike out my comment)

Comment by Brandon Diamond [ 26/Apr/12 ]

I'm working on fixing this behavior. Many of these messages were emitted to stdout originally; we moved them to stderr to ensure that messages do not interfere with output when dumping objects to stdout.

Any thoughts on adding a flag to disable non-error output to stderr? Would this be a reasonable compromise?

Comment by Eric Herot [ 26/Apr/12 ]

Can I third this? I'm a sysop trying to use meskyanichi's very handy ruby backup script "backup" and for the twenty or so DB backups we do, this is the ONLY thing I get emailed about, and I get an email for all 15 mongo databases!!!

This behavior is incorrect, extremely annoying (even dangerous), and exceptionally simple to fix. Why has it not been taken care of?

Comment by Nils Toedtmann [ 22/Feb/12 ]

Tools which write non-critical info to stderr are a pain for every sysadmin. Cron and backup tools (like e.g. rsnapshot) expect that only errors which need to be fixed are written to StdErr, and they will rootmail all stderr to alert the sysadmins. Hence we have to silence these noisy tools (2>/dev/null), introducing the risk that when there are actual errors, we won't notice

If mongodump is meant to be used for automated/scriped jobs, this should be fixed.


BTW this seems to be a re-occurence of SERVER-564

Comment by Krzysztof Wilczynski [ 17/Sep/11 ]

Hello,

I also would like to see this problem solved not only for the sake of "automongobackup" (which is a very nice addition to the tool set surrounding Mongo), but also to maintain consistency concerning that sort of tools – clear separation what goes onto stdout and stderr is always a plus. Said that... having "connected to:" during normal healthy run of "mongodump" is not quite that consistent. Perhaps there is a greater idea hidden here which I am missing

Having a look at the source code ...

                (_usesstdout ? cout : cerr ) << "connected to: " << _host << endl;

Also ...

        if ( _matcher.get() )
            (_usesstdout ? cout : cerr ) << processed << " objects processed" << endl;

I am not sure why these two lies are being written onto standard error

Thank you in advance for looking into this

KW

Generated at Thu Feb 08 02:58:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.