[SERVER-4312] Add "key" missing readline/bash (EMACS-style) command line features to the shell Created: 17/Nov/11  Updated: 11/Jul/16  Resolved: 08/Mar/12

Status: Closed
Project: Core Server
Component/s: Shell
Affects Version/s: None
Fix Version/s: 2.1.1

Type: Bug Priority: Major - P3
Reporter: Tad Marshall Assignee: Tad Marshall
Resolution: Done Votes: 7
Labels: rn
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

All


Attachments: Microsoft Word EMACS-style command line editing in the Mongo shell.xls    
Issue Links:
Depends
Duplicate
is duplicated by SERVER-2852 Linenoise doesn't work well with some... Closed
is duplicated by SERVER-3914 Ctrl commands do not work in shell Closed
is duplicated by SERVER-4286 Meta commands don't work in the shell Closed
is duplicated by SERVER-4480 Autocompletion in shell is broken Closed
is duplicated by SERVER-3809 Mongo shell tab completion Closed
is duplicated by SERVER-3484 Linenoise support for bash-like funct... Closed
is duplicated by SERVER-5692 The shell command line editor is quit... Closed
is duplicated by SERVER-5711 DB shell is missing emacs yank command Closed
Related
is related to DOCS-121 Document the mongo shell's line-editi... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

I'm folding all of the existing and future requests for readline-style (bash-style) command line editing features into a single ticket so that we can do more effective estimation and planning. So, requests for ctrl-Z job control, ctrl-R history search, ctrl-K kill and ctrl-Y yank and everything like it is all here. I'll call this a "bug" since it worked in version 1.8 (when linked with readline) though it's basically a "feature" for linenoise.



 Comments   
Comment by auto [ 12/Mar/12 ]

Author:

{u'login': u'tadmarshall', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 Simplify previous "kill buffer" fix

Just skip the join code when no previous kill buffer exists,
simpler code with less duplication.
Branch: master
https://github.com/mongodb/mongo/commit/cd54317107d9acf63dc5de6c466990ca9d234462

Comment by auto [ 12/Mar/12 ]

Author:

{u'login': u'tadmarshall', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 Don't read kill buffer that isn't there

If your first "kill" operation is for zero characters, the initial
kill ring buffer isn't saved. If this is immediately followed by
a kill operation that does kill text, the code was trying to join
the two kills, but the first one never got a buffer. Only join
text when there is something to join ... if no buffer to join with,
just store the new text in a slot by itself. Most visible in debug
builds, but a bug in any build.
Steps to reproduce:
1) Start shell
2) Type "a"
3) Press Ctrl-K
4) Press Ctrl-U
Branch: master
https://github.com/mongodb/mongo/commit/3a59385162569f7d2dbf0aec0bfb03c52ace706c

Comment by Sam Kleinman (Inactive) [ 08/Mar/12 ]

Linking to existing documentation issue (DOCS-121) and removing documentation needed flag.

Comment by Tad Marshall [ 08/Mar/12 ]

This was essentially finished for 2.1.0 as shipped, but I held the ticket open in case additional features were written. A couple of 2.0 features were found to have been clobbered by my changes; the ones we know about are fixed in this version. I'm resolving this ticket as having achieved the goal of "key" missing readline features restored; we can reopen it or open new tickets for bugs or remaining missing features.

Comment by auto [ 08/Mar/12 ]

Author:

{u'login': u'tadmarshall', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 Fix two display bugs in linenoise

I broke the fetching of the screen width with some code rearrangement
weeks ago and finally dug into it. Can't use console_out to fetch
screen width because it is set in enableRawMode() which is (now)
called later. This caused bizarre behavior where the cursor would
walk up the screen as you recalled previous history ... I thought this
was UTF-8 related, it isn't. Second bug is trying to refresh the line
after displaying "Display all %d possibilities?" in tab completion:
don't do it in that case, we already fixed the line and moved to the
next one.
Branch: master
https://github.com/mongodb/mongo/commit/f84147cf3fa29aea65529ac5cce3c56678530a48

Comment by Tad Marshall [ 04/Mar/12 ]

This is the list for version 2.1.0 of the mongo shell. If this doesn't include something that is wanted, it may be best to submit a new ticket asking for it ... I think that the "key" features are now covered.

EMACS-style shell editing keystrokes/commands

Keystroke readline function name (or invented name) Works in Mongo 2.0? Works in Mongo 2.1.0?
Ctrl-A beginning-of-line Yes Yes
Ctrl-B backward-char Yes Yes
Ctrl-D delete-char (or exit shell) Yes (needs work) Yes
Ctrl-E end-of-line Yes Yes
Ctrl-F forward-char Yes Yes
Ctrl-G abort No Yes
Ctrl-H [Backspace] backward-delete-char Yes Yes
Ctrl-I [Tab] complete Yes (needs work) Yes
Ctrl-J accept-line Yes Yes
Ctrl-K kill-line No (deletes instead) Yes
Ctrl-L clear-screen Yes Yes
Ctrl-M accept-line Yes Yes
Ctrl-N next-history Yes Yes
Ctrl-O "Return-then-next-history" No No
Ctrl-P previous-history Yes Yes
Ctrl-Q quoted-insert No No
Ctrl-R reverse-search-history No Yes
Ctrl-S forward-search-history No Yes
Ctrl-T transpose-chars Yes (needs work) Yes
Ctrl-U unix-line-discard No (deletes instead) Yes
Ctrl-V quoted-insert No No
Ctrl-W unix-word-rubout No Yes
Ctrl-X ( start-kbd-macro No (should we?) No
Ctrl-X ) end-kbd-macro No (should we?) No
Ctrl-X e call-last-kbd-macro No (should we?) No
Ctrl-X Ctrl-U undo No No
Ctrl-Y yank No Yes
Ctrl-Z Suspend (job control) No Yes (Linux)
Ctrl-_ (underscore) undo No No
Del delete-char Yes Yes
Meta-B backward-word No Yes
Meta-C capitalize-word No Yes
Meta-D kill-word No Yes
Meta-F forward-word No Yes
Meta-Backspace backward-kill-word No Yes
Meta-L downcase-word No Yes
Meta-N non-incremental-forward-search-history No No
Meta-P non-incremental-reverse-search-history No No
Meta-R revert-line No No
Meta-T transpose-words No No
Meta-U upcase-word No Yes
Meta-Y yank-pop No Yes
Meta-Ctrl-Y yank-nth-arg No No
Meta-. (period) yank-last-arg No No
Meta-_ (underscore) yank-last-arg No No
Meta-< beginning-of-history No Yes
Meta-> end-of-history No Yes
Meta-? possible-completions No No
Meta-backslash delete-horizontal-space No No
Meta-= possible-completions No No
Meta-* insert-completions No (should we?) No
Meta-<digits> digit-argument No No
Left-arrow backward-char Yes Yes
Right-arrow forward-char Yes Yes
Up-arrow previous-history Yes Yes
Down-arrow next-history Yes Yes
Home beginning-of-line Yes Yes
End end-of-line Yes Yes
Ctrl-left-arrow backward-word No Yes
Ctrl-right-arrow forward-word No Yes
Meta-left-arrow (Emacs) backward-word No Yes
Meta-right-arrow (Emacs) forward-word No Yes
Comment by auto [ 05/Jan/12 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 add Meta-< and Meta->, improve history logic

Add support for the Meta-< (beginning-of-history) and Meta->
(end-of-history) keystrokes. Improve the handling of unsaved
input on the "current" (new) command line when beginning to
move through history with ctrl-R, up-arrow, Meta-<, etc.
Branch: master
https://github.com/mongodb/mongo/commit/717c7a2d79272f03f73f3214fbb61b59e22836ee

Comment by auto [ 05/Jan/12 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 use PromptBase instead of PromptInfo in calls

The called code only references the base members, so it should
reference the base class, not the derived class; mostly for
consistency and possibly future flexibility.
Branch: master
https://github.com/mongodb/mongo/commit/f720eb38b1372564d9fc01bde3b348d26feed04e

Comment by auto [ 05/Jan/12 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 add paging for long command completion lists

If the list of displayed command completion possibilities is more
than one screen, display "-More-" and wait for a keystroke to
display more. Display one additional line on Return or Newline,
display another screenful on space, y or Y, stop the list on n, N,
q or Q, and stop the list after showing ^C on ctrl-C.
Branch: master
https://github.com/mongodb/mongo/commit/813420ded1be8ec4168e242522507f889614db31

Comment by auto [ 03/Jan/12 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 rewrite tab completion to work like readline

Replace the old logic for rotating through completion possibilities
with logic that mimics readline's behavior. We now only complete
the portion that is unambiguous (the same in all possibilities) and
show a list of matches if tab is hit again after a failure to
complete a match. If the number of matches is over 100, we ask
before displaying the list. Still to come – paging code to pause
after displaying a screen's worth of matches.
Branch: master
https://github.com/mongodb/mongo/commit/e7fe1c16621ae483629c5663cb42da1e02d32934

Comment by auto [ 22/Dec/11 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 fix logic for meta-C (capitalize)

My logic for capitalizing didn't match readline, now it does.
Branch: master
https://github.com/mongodb/mongo/commit/504b8c51ca05c707dd60aa89d4f78358abb2a066

Comment by auto [ 22/Dec/11 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 add meta-C, meta-L and meta-U to mongo shell

Added meta-C (capitalize word), meta-L (lowercase word) and
meta-U (uppercase word) readline features to the shell. Changed
the InputBuffer struct to a class, most members are now private.
Merged some common code for ctrl-R and ctrl-S keystrokes when
already in search mode.
Branch: master
https://github.com/mongodb/mongo/commit/6dbda206551284b242c768a122243e3dcf897cc1

Comment by auto [ 21/Dec/11 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 consolidate some code into a class/struct

Added new InputBuffer struct to hold the buf, len and pos
variables that were being passed to everything, moved some
code into methods in that struct. Folded linenoiseRaw into
linenoise.
Branch: master
https://github.com/mongodb/mongo/commit/e65a90d143c8155a5a8dfb4f34c2152e638e39c5

Comment by auto [ 20/Dec/11 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 fix ctrl-L on Linux with 80 character prompt

Linux doesn't advance the cursor to the next line after
writing in column 80, so we need to do it ourselves. Probably
time for common code so the same fix isn't in multiple places.
Branch: master
https://github.com/mongodb/mongo/commit/2baf1bcf679df8e72b6d54beff19f9b40972ee4f

Comment by auto [ 18/Dec/11 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 add incremental history search to the shell

Added ctrl-R and ctrl-S to search history. It's a pretty close
clone of readline but not quite exact and there may be bugs
and/or rough spots. Probably some fine tuning to come.
Branch: master
https://github.com/mongodb/mongo/commit/706b9e89a8a466111929330be449e250253a7dc8

Comment by auto [ 12/Dec/11 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 implement kill ring for mongo shell

Add code to enable Meta-Y to rotate through killed text. Ring size
is set at 10 right now, but it's a compile-time constant and could
be changed if desired. 10 is what readline defaults to.
Branch: master
https://github.com/mongodb/mongo/commit/6ebb2c50b3ce9a82fec7be921dd7996c7bd28545

Comment by auto [ 12/Dec/11 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 fix Alt+Backspace on xterm

Have to special-case Alt+Backspace on xterm. The "normal"
processing was giving us Meta-_ which is a different (and
not yet implemented) function.
Branch: master
https://github.com/mongodb/mongo/commit/81c688139fc335d57b5b2f64e5deb68e0cffcae9

Comment by auto [ 12/Dec/11 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 implement kill/yank in mongo shell

A single kill buffer is now working in the mongo shell. All of
the commands that are supposed to add text to the kill buffer now
do so, and accumulate text properly with repeated kills in either
direction (to the left or to the right) or both. No Meta-Y (rotate
kill buffer, "yank-pop") feature in this checkin, so it's not a
kill ring yet.
Branch: master
https://github.com/mongodb/mongo/commit/59b8d78a690bfa72b9faa0cd76857ea2ed7f094b

Comment by Tad Marshall [ 11/Dec/11 ]

From doing more testing and online reading, I can't demonstrate some of the features on my list (which came largely from "Learning the bash Shell" by Cameron Newham & Bill Rosenblatt) but there are other features that aren't on the list above that work. readline (and bash and the mongo shell version 1.8) accept Ctrl-_ to undo a single change, and this can be repeated to undo multiple changes. So, you can press the Backspace key 20 times (which deletes text, does not kill it, so that it could be yanked), and then press Ctrl-_ repeatedly to bring back all of the deleted text. This works under Gnome Terminal from Ubuntu 11.10 with both bash and mongo 1.8.2. readline also supports numeric arguments for "commands", including the command of inserting a character. So, you can hit Esc then 5 then 'g' and five 'g's will be inserted. You can also hold down Alt and type digits to specify the numeric argument. Negative arguments are also legal and work with some commands (but not all). Typing Esc - 1 0 <right arrow> will move the cursor to the left by 10 characters, but Esc - 3 Ctrl-W kills to the left and does it a single time.

I was wrong in my comment from Dec 08 2011 01:15:53 AM UTC that readline doesn't support Meta-Y to rotate the kill ring. I must have been testing in the wrong terminal. It won't work for me in Ubuntu's xterm with default settings, but it works fine in Gnome Terminal using mongo version 1.8.2.

It will be a judgement call how many of the more advanced features make the cut as "key" features. Kill and Yank are must-haves as is Ctrl-R search; if you have an opinion about what is vital and what you would never use, add a comment to this ticket, thanks!

Comment by auto [ 11/Dec/11 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 several fixes, features and cleanups

Fix parsing of Alt+<alphabetic char> for xterm so Alt+b and Alt+f
work. Beep on failure to parse a keyboard key the way bash and
version 1.8.2 do. Allow Meta-<left arrow> and Meta-<right arrow>
as synonyms for Ctrl-<left arrow> and Ctrl-<right arrow>; bash
and version 1.8.2 don't allow this, but Emacs does, and it's
harmless and helps us not fight muscle memory for Emacs users.
Fix incorrect deletion on Meta-Backspace; this key works differently
from Ctrl-W and shouldn't share code with it. Add ESC prefix for
Meta in Windows (Alt key already worked). Implement Meta-D to
delete the word to the right of the cursor. Remove the 'fd' file
handle and all references to it. It was always zero and served no
purpose. Always read from stdin and write to stdout; they can be
redirected but we still use the same handles (0 and 1).
Branch: master
https://github.com/mongodb/mongo/commit/0fdbec10d663e94e3a69ae169a35501f826822c1

Comment by auto [ 11/Dec/11 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 rewrite keyboard parsing for Linux

Replace the nested if statement with a ton of new code to parse
the escape sequences that the various Linux terminal programs
give us when arrow keys (for example) are pressed while Ctrl or
Alt are held down. Make ESC followed by <key> generate the same
internal code as Alt+<key> even when the terminal gives us different
stuff. Meta-B and Meta-F now work on all platforms. Rip out the
unsupported terminal "feature", it does nothing useful. Turn on
the Linux keyboard debugging feature in all builds, it is useful
and doesn't get in the way. Striving to work correctly on Gnome
Terminal, xterm, aterm, konsole, rxvt and yakuake, code was written
with this set targeted but testing is limited to implemented
functions (e.g. Meta-B and Meta-F). More testing once more features
are written.
Branch: master
https://github.com/mongodb/mongo/commit/deefd0741bacdeeff77c9ae1b51af2f9e25c2095

Comment by auto [ 09/Dec/11 ]

Author:

{u'login': u'', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-4312 add some framework for readline work, ctrl-W is in

Beef up the keyboard debugging support on Linux to make it easier
to figure out what keys are doing, add a framework for encoding
key combinations (e.g. Ctrl-Meta-left-arrow), add code to encode
the Ctrl and Alt keys on Windows, add code for Meta-B/Ctrl-left-arrow
and Meta-F/Ctrl-right-arrow (only working on Windows right now), add
code for Ctrl-W forall platforms.
Branch: master
https://github.com/mongodb/mongo/commit/e221038722d54c7e5623bacbce6a8a9fa5d846b7

Comment by Tad Marshall [ 08/Dec/11 ]

I forgot Ctrl-left-arrow (same as Meta-B) and Ctrl-right-arrow (same as Meta-F). Also, I had expected to need to implement a kill-ring the way Emacs does it, so Ctrl-Y yanks back the most recent (possibly accumulated) killed text, and then Meta-Y cycles through previously killed text, but readline doesn't seem to do this. And for those unfamiliar with Emacs' terminology, "Meta" means either the Alt key or hitting Esc first, so Meta-B can be either Alt+B or Esc then b. Both key patterns should work and do the same thing.

Comment by Tad Marshall [ 07/Dec/11 ]

Here is the state of things on 7 December 2011. Every feature with a blank in the "Works in Mongo 2.1?" column is a candidate to get done for 2.1, but the ones where the "Works in Mongo 2.0?" column says "No (should we?)" seem questionable, inapplicable or low value to me.

EMACS-style shell editing keystrokes/commands

Keystroke Description Works in Mongo 2.0? Works in Mongo 2.1?
Ctrl-A Beginning-of-line Yes Yes
Ctrl-B Character-left Yes Yes
Ctrl-D Delete-right or exit shell Yes (needs work) Yes
Ctrl-E End-of-line Yes Yes
Ctrl-F Character-right Yes Yes
Ctrl-G Cancel-editing-mode No  
Ctrl-H [Backspace] Delete-left Yes Yes
Ctrl-I [Tab] Command-completion Yes (needs work)  
Ctrl-J Return Yes Yes
Ctrl-K Kill-to-end-of-line No (deletes instead)  
Ctrl-L Refresh-screen Yes Yes
Ctrl-M Return Yes Yes
Ctrl-N Next-line (history) Yes Yes
Ctrl-O Return-then-next-history No  
Ctrl-P Previous-line (history) Yes Yes
Ctrl-Q Insert-verbatim (readline) No (should we?)  
Ctrl-R Reverse-search (history) No  
Ctrl-S Forward-search (history) No  
Ctrl-T Transpose-characters Yes (needs work) Yes
Ctrl-U Kill-to-beginning-of-line No (deletes instead)  
Ctrl-V Insert-verbatim (bash) No (should we?)  
Ctrl-V Backward-kill-word (readline) No (should we?)  
Ctrl-W Backward-kill-word No  
Ctrl-X ( Begin-keyboard-macro No (should we?)  
Ctrl-X ) Finish-keyboard-macro No (should we?)  
Ctrl-X e Play-keyboard-macro No (should we?)  
Ctrl-X / Show-filename-completions No (should we?)  
Ctrl-X ~ Show-username-completions No (should we?)  
Ctrl-X $ Show-variable-completions No (should we?)  
Ctrl-X @ Show-hostname-completions No (should we?)  
Ctrl-X ! Show-command-completions No (should we?)  
Ctrl-X Ctrl-R Load-readline-init-file No (should we?)  
Ctrl-X Ctrl-V Display-version-info (bash) No (should we?)  
Ctrl-Y Yank No  
Ctrl-Z Suspend (job control) No Yes (Linux)
Del Delete-right Yes Yes
Meta-B Word-left No  
Meta-C Capitalize-next-word-and-move-past-it No  
Meta-D Kill-word-right No  
Meta-Ctrl-E Expand-alias No (should we?)  
Meta-F Word-right No  
Meta-Backspace Backward-kill-word No  
Meta-Tab Attempt-complete-from-history (???) No  
Meta-L Lowercase-next-word-and-move-past-it No  
Meta-N Non-incremental-forward-search No  
Meta-P Non-incremental-backward-search No  
Meta-R Revert-line No  
Meta-T Transpose-words No  
Meta-U Uppercase-next-word-and-move-past-it No  
Meta-Ctrl-Y Insert-first-word-in-previous-command No  
Meta-Del Kill-word-left No  
Meta-. (period) Insert-last-word-in-previous-command No  
Meta-_ (underscore) Insert-last-word-in-previous-command No  
Meta-^ Expand-history (???) No  
Meta-< Beginning-of-history No  
Meta-> End-of-history No  
Meta-? Show-completions No  
Meta-/ Attempt-filename-completion No (should we?)  
Meta-~ Attempt-username-completion No (should we?)  
Meta-$ Attempt-variable-completion No (should we?)  
Meta-@ Attempt-hostname-completion No (should we?)  
Meta-! Attempt-command-completion No (should we?)  
Meta-\ Delete-whitespace No  
Meta-= Show-completions No  
Meta-* Insert-all-completions No (should we?)  
Meta-{ Insert-all-completions-inside-braces No (should we?)  
Left-arrow Character-left Yes Yes
Right-arrow Charcter-right Yes Yes
Up-arrow Previous-line (history) Yes Yes
Down-arrow Next-line (history) Yes Yes
Home Beginning-of-line Yes Yes
End End-of-line Yes Yes
Comment by Tad Marshall [ 07/Dec/11 ]

Excel spreadsheet with bash/readline feature list and Mongo shell support

Comment by Kristina Chodorow (Inactive) [ 06/Dec/11 ]

Just want to note that support for Alt/Meta combinations is important (e.g., Alt-D, Alt-backspace, Alt-<arrow keys>).

Comment by Tad Marshall [ 06/Dec/11 ]

I added ^Z job control (commit fa62a374a8818ec2c8981176c60c9cd0a01eb8c1) based on code from https://github.com/leejo/mongo (https://github.com/mongodb/mongo/pull/111), thanks leejo!

Comment by Tad Marshall [ 02/Dec/11 ]

I got most of the way into creating a spreadsheet listing all of the stuff that we might, could or should do in addressing this bug/feature, but I need to finish it and add it to this Jira ticket.

Generated at Thu Feb 08 03:05:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.