Right, it's theoretically dangerous, or I'd say practically: "please think when using it"... though I personally couldn't find any other serious problematic example other than typing something clearly dangerous in the first place. And I personally found it immensely useful in the other, non-dangerous cases. So I just took the chance and went with the idea, and then wanted to share it with you, so you might also find it useful, maybe! :) Also... umm... you know, a tool isn't really truly Unixy if you can't hurt yourself with it, right? ;P
That said, I'm starting to think — should I include a note about the potential dangers in the readme?... they seemed glaringly obvious to me (at least in case of rm-like commands), but can there be readers/users, to whom they'd not be obvious?...
edit: Ok, I tried to add a warning in the Readme, that's something I can do easily, and if it helps someone, worth it...
I'm not so versant about the state of permission / capability systems on Linux.
But if there is a way for you to drop the capability to modify the filesystem before running the first pipeline, you should definitely do it by default (and provide a switch to override it if somebody knows what they are doing.)
Sounds interesting; but is there any way I could change to a different user if not running as root?... also, even if yes, how do I dynamically create a non-privileged user, or find a pre-existing one?
I would try creating a non-privileged user "up" during installation and switch to that user under the hood before running the command via `su`.
(Though it would complicate installation which is effectively non-existent since there is a single binary; and tightly couple the application with the environment. It's a trade-off anyway.)
(Edit: Somebody mentioned the user "nobody" which seems to be a better alternative.)
In any case I think that the whitelist approach would work better as it would be nice if the tool knew in advance which commands work properly and which ones don't. That way it could inform its users about the allowed & unallowed commands.
Commands with side effects (or "impure" in the FP terminology) don't make much sense anyway to be used within this. The main value is fast iteration, in order to verify the expected output of the pipeline. For me it doesn't make much sense to use it with commands whose main purpose is to modify the filesystem instead of generating / transforming data and writing it to stdout.
So the criteria for the commands that would make it into the whitelist may be "not having side effects", "writing to stdout", and optionally "reading from stdin".
Fastly iterating side effects is unsafe as well, as has already been pointed out.
This can be a fine data analysis tool actually, unfortunately only for command line geeks. The experience is actually not so far from analyzing data with VIM interactively by piping it to an external UNIX command and getting the result dataset back into the editor. VIM hackers will know :)
One thing to watch out for is that a process running as "nobody" has permission to manipulate (e.g. kill) other processes running as that user. This sometimes includes system processes. Allowing any user to run arbitrary commands as nobody is technically a privilege escalation, and therefore should be avoided.
A single-purpose user with its own group would have this problem only to a lesser degree (you'd be able to mess with other users' "up" invocations, but not any system processes).
I actually like the user idea better. The problem with a whitelist is that even useful commands can have subtle or little known dangerous modes, like "find . -exec rm {}"
Ah. Good point. Whitelisting commands would have already been a bit painful, and now your comment shows that the parameters need to be whitelisted/blacklisted as well, which would be crazy.
In a world in which shell commands respected the UNIX philosophy, "find" wouldn't have a silly option like "exec", and other commands wouldn't mix read / write / pure data transform operations in a single command.
But it is what it is. So yeah, protection probably needs to be implemented in the user level, for maximum safety.
Maybe an alternative and/or complementary solution would be to profile each inputted command to detect if they are attempting write operations (maybe with "strace" or something like that), and cancel the evaluation of the command in the next iterations and/or show a warning.
> The experience is actually not so far from analyzing data with VIM interactively by piping it to an external UNIX command and getting the result dataset back into the editor. VIM hackers will know :)
- Send the whole content of the buffer as standard input to the given command and rewrite the buffer with the data read from the standard output of it:
:%! grep network -A2
- Examine the returned dataset. Iterate:
:%! grep :
- Examine & iterate:
:%! cut -d: -f2-
- Examine & iterate:
:%! paste - -
This way, you can examine the output of each step of the pipeline individually, so you can construct your command incrementally. And it is up to you to decide at which point a command will be run instead of it being run automatically following every keypress.
Since the dataset returned by the last command will be visible in the current buffer, you will be able to examine & play with it full screen. You will be able to clean or transform some parts manually (this is frequently needed in data science).
You can always return to the previous/next dataset by pressing u/Ctrl+R (for undo and redo), and examine your command history by pressing ":" and then Ctrl+P/Ctrl+N. (Or you can open the command history quickfix menu by pressing "q:" to view/copy the last several commands at once.)
And since you are in a full blown text editor, you can take advantage of other helpful features such as folding, saving the current dataset to a temporary file, etc.
If you are comfortable with more than a few UNIX filters, VIM can be a very convenient and fun tool to play with data.
Nice tool! Your main use case for destroying files is “xargs rm”, which would be used when receiving a piped argument list.
Could you have this command run as user “nobody” by default, and have a flag to run as a real user? Or as others have mentioned, use capabilities on the spawned processes to prevent anything other than standard input/output?
This is one of the main reasons that I ultimately decided for my Strukt.app to not be built on top of built-in shell commands. For an interactive tool, it's crucial for each operation to be able to report whether it's idempotent, does writeback, etc.
Strukt doesn't do 'rm' yet (and nobody has ever requested it), but it does have other operations that do writeback. Once you add one, automatic updates are disabled, and you have to explicitly run the pipeline each time you want it run.
We really need a new cross-platform low-level framework for lego-like operations, like shell commands but updated for the 21st century.
PowerShell has a design answer for this - each cmdlet takes parameters, but the environment provides common parameters relating to loging, error handling, storing output in variables, and it provides support for "WhatIf" and transactions.
WhatIf instructs the command not to make any changes, but merely to log what it would do. You can "Remove-Item C:\Windows -Force -WhatIf" and it will output "What if: Performing the operation "Remove Directory" on target "C:\windows"" but not delete anything.
(implementing support for "whatif" is optional, so you can't casually use it on any given cmdlet without first checking that cmdlet supports it and makes sensible use of the parameter instead of ignoring it, but the idea is there).
Yep, that's basically the solution that I came up with, too. I have a "dry-run?" flag planned, if I ever end up with enough writeback operations where it'd be useful.
(That's the common Unix shell name for it, though it's not entirely consistent -- which is another problem with using Unix built-ins!)
One thing I've noticed is that it's really frustrating (and not at all helpful) to require perfect validity from a shell-like tool. There's just too many times where you want to edit something into something else, without needing to find a perfect path of valid and meaningful syntax at every intermediate step. So if you have a flag on an operation that doesn't understand that flag, the flag will have a (?) badge to indicate that it won't be understood or used, but the pipeline will still run.
I feel this app should have a limited free mode, like "10" results or something. I'd love to try it, but just trying for $24 seems like a gamble, as I am not sure I'd like it or not.
If you’re on Mac OS X , try installing the “trash” app (from brew iirc) and retrain yourself to never use bare rm anymore. There’s probably similar tools on Linuxes.
Theory is: rm is dangerous and should not be used day to day.
Using btrfs on Linux I have a job that takes a CoW snapshot of the whole filesystem every 10 minutes just so that I can undo whoopsies. I don't retain all of these snapshots, only the most recent one.
Journaling works well here too, but in my experience the interface sucks on every platform. Eg, it should be trivial to view recent unlinks. There’s simply more reason to invest in backup software, which has an excellent interface even if it is a “heavy” operation.
That is not what journaling is for. Journaling is for maintaining filesystem consistency. If you want access to recently deleted files that is a different task.
On my system I have snapshots of my home directory created every 15 minutes. Not quite the same thing, but it also helps me when I have incorrectly modified a file.
And do what when tmp gets full and services & applications start crashing?
When someone implements that undo, someone's going to need to write a tool that really does remove files, really. Will the next guy wonder why really-rm doesn't have undo built in?
You could do what is more or less an industry standard when it comes to this and warn the user that there's not enough room to allow an undo when deleting said file(s) and prompt if they want to delete it permanently instead.
I am similarly not worried on my Linux system, which also has frequent snapshots turned on. Instead I worry about bugs in btrfs, but that is another story.
That's the first thing I thought and it's mentioned in the readme file itself.
I don't think I'd ever want to use this tool. Every keystroke could be an arbitrary command, and I wonder about the performance of commands with a lot of I/O or processing.
Conceptually it's really neat and I get it, but practically I find things like parameter completion in Fish much more useful.
This looks like a fun and helpful tool! (Perhaps best to ignore the notice from lshw and not run it as root, ha ha. Maybe even only in some kind of sandbox. I bet they could add a --safe flag so it would make an on-the-fly Docker image with a copy of your cwd, and run the commands in there. Then you have different problems I suppose, but still....)
I've never seen `paste - -` before, but that is amazing too! I can't count how many times I've wanted something like that. Surely that will help my own shell pipelines in the future.
Also I've never seen `foo |& bar` before. What a neat idea! The surprisingly-valid creativity of it reminds me of Duff's Device. Has anyone else ever found uses for that idiom?
After so many years working in the shell, it's such a rare and delightful treat to learn new tricks. Thank you for sharing this with us!
Making a docker container to run the command in is a good idea, but not helpful if the container doesn't have the same filesystem. I love the idea of something like this, but I'm not sure if I would ever use it out of fear of unintended actions.
If you didn't know about paste(1), maybe you don't know about comm(1) either. There are a couple other like that that I typically fail to remember because I use them only every now and then but when I do they usually are a lifesaver.
Looks great. Personally, I have been using a "cache" function to cache output of expensive commands. This has helped me iterate faster on pipelines that call into apis.
function cache {
cmd="$*"
name=$(echo "$cmd" | base64)
if [[ -f ./cache/$name.exit ]] && [[ $(cat ./cache/$name.exit) -eq 0 ]]; then
echo "cached"
else
mkdir -p ./cache
eval "$cmd" > ./cache/$name.out
echo $? > ./cache/$name.exit
fi
cat ./cache/$name.out
}
cache ls /
I like this. I currently do the "plumbing" the OP refers to using files. This has generally worked for me, but want to give this tool a test run and see if it's faster than my current approach and this cache command.
I do progressive filtering a lot. It's convenient to have all the intermediate results somewhere temporarily.
What I do is typing the command in one acme window with the working directory, and middle mouse swipe it. Acme shows the result of that command in a +Errors window with the same working directory. I change the name of this new popped up window (to something like `+Errors-cmd`, type a command that operates on the file name `/mnt/acme/$winid/body`, and middle mouse swipe it again. Acme shows the result of the new command under `+Errors` again. Rinse and repeat until I'm satisfied.
I guess this way it gives a convenient temporary place for all the intermediate texts. I can of course edit them in place if I want.
> But you'd be careful writing "rm" anywhere in Linux anyway, no? Also, why would you want to pipe something into "rm"?
But the thing is that you don't need to intend to pipe to "rm". Maybe you were typing something else, like having a command "rmore" or something. This danger is also not strictly limited to `rm` and `dd`, and you have to be careful that no substrings from the start of the command you intend to write cannot be interpreted as something dangerous.
The author's theory there about user behavior seems dangerously wrong.
I'm quite careful writing command lines because the whole experience of the shell is that of working with sharp knives: you get a lot of power but if you screw up, you'll feel the pain. The point of this tool is to take away a lot of the pain that teaches people caution. Its whole theory is "just try a lot stuff as you explore what you want".
In their shoes I'd look at using some of the container/security magic as a way of nerfing commands. If the on-a-keypress runs work in a way where they can't make changes to a filesystem, that seems way better to me. Even better if the tool then reports, "Would have deleted 532 files in a warning color at the top of the output."
A safety measure I picked up from a sysadmin while watching over their shoulder: start writing nifty command lines by prefixing them with # first, to prevent havoc when fat fingering the [enter].
It also executes command and process substitutions, opens files etc. But granted, it's a useful trick in some cases, after passing the fat-fingering stage.
What I like about SQL is that it's got the double safety of ";". To accidentally run a SQL statement before it's ready in a command line, you'd have to both add the ";" AND hit Enter, or have the bad practice of adding ";" before the statement is ready, or use a bad SQL command line that sees the ";" as optional.
Yes. Like Enter? People might still like the fact that the cursor doesn't need to move from wherever it's editing the command to run it, and can still continue to edit there.
That's the main direction I currently seem to be coalescing on based on feedback I'm getting. That said, one person on lobste.rs suggested trying to play with Linux capabilities, to make the filesystem immutable... now that would be a seriously fancy trick if it works!
Can I say that I hope you don't make this tool too Linux-specific. It seems to run pretty reasonably on Mac and would probably also work on BSD variants. Since I use all three (Mac, BSD, Linux) on a semi-regular basis, I'm pretty unlikely to integrate a tool into my Unix workflows if it doesn't work reliably on most unixes, so if you can look into cross-platform ways to accomplish things, that would widen the audience of people who could get value from the tool.
For example, one way to address the rm problem would be to give it a "safe mode" that drops user permissions and runs as nobody. You might also create a blacklist of common, potentially-dangerous unix commands that you don't execute. Both of those are examples of how to address the need without resorting to Linux-only tricks.
How can I do the "drop user permissions" thing? I haven't seen such a keyword mentioned by anyone yet, sounds interesting. Does this not require some kind of syscall, however, thus not really being cross-platform anyway?
‘nobody’ is an only UNIX thing. It’s just a user and group (like root, $USER, etc) except ‘nobody’ is meant to have no permissions to any folders nor files.
In practice it’s only as good as your users ability to avoid the temptation of ‘chmod 777’ (for example) but it seems a good place to start.
As an aside, I had the same idea to do a tool like this as well. Except mine would have been built into the $SHELL I’m writing (as that has a heavy focus on being more IDE-like than traditional shells). I scraped the plan for pipe previews precisely because of the dangers we’re discussing. However your tool makes a lot more sense because at least that is manually invoked whereas my original plan was to have that feature automatic and baked into the shell - which is an order of magnitude more dangerous. So I’m glad someone else has ran with this idea
I've seen somebody mention the idea of applying this to a shell, but only after pressing some special key (e.g. Ctrl-Enter). Would probably make it un-dangerous enough? This seems to be what people want of up too, anyway.
As to the "nobody", from what I'm reading, it seems you'd first have to be root, to be able to switch to "nobody"... so this doesn't really seem to be useful to me in this case... :/
I've tested it, and it seems to prevent deleting files with rm. What doesn't work, however, is that it also prevents writing the results to up1.sh. Perhaps if writing to the file fails (or you detect the process is running as nobody), you could send the finished pipe sequence to stdout instead of a shell script. Then, people could run it like:
The solution there is not to set any writable bits for the up executable. Then only root will be able to write to it (which is ideally what you want for any tools within /usr/bin (whatever) anyway
That’s an interesting argument having it hot-key triggered. The shell in its current form already supports hit-keys so I could plug right into your tool verbatim that way. The only issue is you call fork shell so any $SHELL specific behaviours of my shell would be lost.
I appreciate this is a personal project and sometimes there is nothing more annoying than having feature requests; but if you did ever decide to add a flag for choosing alternative shells then drop me a message ( raise it as an issue on github.com/lmorg/murex ) and I’ll add ‘up’ as an optional 3rd party plug in.
i fear a blacklist won't be enough. better a user configurable whitelist. every time a command is not on the whitelist, don't run it, but give the user the option to add it to the list instead (with a multi key action, so that it can't be added to easily and by accident)
Consider that the filesystem isn't necessarily the only thing you can mess up by accident. It's much more unlikely, but you could accidentally trigger a bunch of unwanted network requests (e.g. piping into `xargs curl` or something), among other possible things with external effects.
Maybe cutting off network access could be done too? That would reduce some potentially useful features, but it could be a parameter to enable it (hey, we're talking Linux, never enough parameters! ;). That is, assuming that a syscalls/capabilities trick like that is possible at all. (Anyone can say more here? Some kernel hackers? This is HN, ain't it?)
Yeah, I think almost any ability to produce an effect other than printing to stdout and stderr should probably be restricted by default, with a flag to enable it. If people adopt this and frequently use specific capabilities, they can add a shell alias to add enable their preferred set of capabilities.
Or, one possible alternative would be to always prevent any side-effects other than printing output, and then display a warning saying something like "This command tried to access the network/modify the filesystem/be naughty in some other way. To allow this and re-run the command, press Control+Shift+Enter"
capabilities should certainly work. for portability, setting a non-existant proxy should also help. most commandline tools should honor shell proxy variables and fail the request.
I think from a design perspective, it's better to build your sanity checks and safeguards into the UI instead of some fancy abstraction that's running in the background. Throw in some options to make it user configurable and you have a pretty slick solution that behaves in a way that'll feel familiar to gurus.
I think requiring a keypress by default to run any commands and maybe even throwing up a warning for a list of known dangerous ones (perhaps user configurable) would work well. A really fancy solution would be to do this stuff via some sort of linting.
I don't see how linting could work, I'm afraid. There are obscure options in zip, git, grep, and what else, not to mention full-blown interpreters like awk and perl, that can edit files via a myriad of syntaxes, and then there's the halting problem. Similarly with whitelist/blacklist, I'm not currently really convinced it could help much; take `xargs rm`, vs. some hyphotetical `foobar -rm` command + params (where maybe -r = recursive, -m = monitor?) Keypress, on the other hand, sounds to me like it could be a reasonable default compromise... still with an option to switch to "fully accelerated, fast mowing" mode at any point, if one so desires...
Too fancy in my opinion. Personally, I prefer KISS solutions. Having an immutable filesystem in up would be unexpected, and I'm sure there's other destructive things that can be done that don't involve the filesystem. As odd as it may seem, I may also want to be able to do filesystem destructive things in up, why shouldn't I be able too?
EDIT: As an example, I may exploratively be playing with find options to match different sets of files in a directory hierarchy. Once I see I've matched the files I want, I may want to add a `-delete` to the end of the find conditions I specified to delete them all. That seems like a useful use of up, but making the filesystem immutable would disallow it.
Hm. I think in my ideal vision, you'd then quit up, and somehow magically have the pipeline ready at your fingertips on the shell prompt, so that you can just add the `-delete` there in a "regular" fashion, and press Enter. Not quite sure how this could be done with bash easily. I think the upN.sh file is not a bad approximation however. (Someone on reddit (I think) seemed to mention that zsh seems to have some functionality which could allow building something like that over up.) Interestingly, this seems to me to support the idea that limiting syscalls/capabilities could be a good approach (if at all possible, that is). Potentially maybe also cutting off network access? But even if I add some safety options, even if I make them the defaults, I'll try really hard to keep an option available for people who want to play with it raw, with no safety belts.
edit: I totally get you as far as KISS goes; that's why I built and shared the tool as is in the first place ;)
> Interestingly, this seems to me to support the idea that limiting syscalls/capabilities could be a good approach (if at all possible, that is). Potentially maybe also cutting off network access?
Yeah, I still think it's not a good approach. rcthompson gave an example with network access, but that doesn't mean that bad things can only happen with filesystems or network access. You can also kill processes, shutdown the computer, and many more things. I don't think you can guess what mistakes the user might do, or which were really mistakes or were intentional.
Also, if you can avoid special permissions or capabilities for your core functionality, then that's better. People should be conservative in giving special permissions to programs, and I can't think it makes sense to give "up" the ability to make a filesystem immutable even if it's in an isolated namespace.
Hmmm; so what would you think of the other popular suggestion, to add a (probably default) mode, where only after pressing, say, Ctrl-Enter, the command would be executed? With another shortcut allowing to switch back and forth to the "no seatbelts, confident, fast moving, fast mowing hacker" mode?
I would just do the Enter thing (I can't think of a reason to prefer Ctrl-Enter), and forget about the other mode. If you really want to do that other mode with a super-restricted environment, just have the code prepared to be denied permission (to setup that environment) and not assume that it has been granted. There will be users that won't feel comfortable adding those permissions/capabilities to "up".
you are building up a pipeline, you don't want it to be destructive until you are done.
so at the very end you could apply a special action that now runs the resulting command with safety disabled.
or toggle between safe and unsafe mode at will, with safe mode being default.
it's not necessary to have the whole up utility run with filesystem disabled. but it could apply the restriction only to the pipeline its executing. that would allow it to selectively apply the restriction as requested by the user.
Another solution is to only allow commands from a whitelist to be run. If the user types something not on the list you say "rm is not on the list of allowed commands, press enter to run and add it".
I think the whitelist makes more sense, since you can never say for sure that you've added every potentially harmful program to the blacklist. If it's easy to add things to the whitelist, as the parent comment proposed, I think the idea would work very well, especially considering most people have a relatively small number of commands they pipe other commands into.
Btw, if you want to see all the commands you've ever piped data into, sorted by frequency, you can use this command (which I put together using `up`!)
Another thing you might want to investigate (for Linux, not sure about cross-platform alternatives) is OverlayFS[1], which allows you to create union filesystems where some layers are read-only and the top layer is read-write. This can allow destructive operations like rm without actually removing anything from the lower (read-only) layers, using whiteouts.
I'd have replied to this on lobste.rs but don't have an account. I'm not a heavy poster but I'd love an invite if anyone has one going :)
I need your email for an invitation; if you can send me your address via my own email (see my profile), or via keybase, I'll send you an invite. I don't see your email advertised in your profile here on HN, so my hands are tied for now :)
Can you elaborate a bit more? Which functionality are you referring to? And why did you disable it? Really curious! (Author of up here) I get it that by eshell you mean a shell in Emacs? (That's at least what I'm getting as a first result from google)
I ultimately just didn't find I used it much. If i am iterating, org-mode with shell sources fits a bit more natural for me.
Edit: and to be clear, I meant specifically to have the output below the command with the command still in edit mode. Was not trying to say it is the same as up.
Well, that's only completely horrific. Cool idea, but should definitely be limited to a fixed list of commands or something. Even then, some stuff that might be handy like xargs could be quite dangerous...
I was thinking about similar tool to this one and this was the biggest obstacle I could think of. Also write touch and create a file for each keystroke.
To become something more than a hack you have a few choices:
- a blacklist - will never be enough, still arguments may be problematic
- a whitelist - will always be not enough, arguments may be problematic, but a bit less than with a blacklist
- limiting permissions somehow - tricky
The last and best option IMHO would be to wrap it in a sandbox, where all filesystem access is behind an overlay (i.e. mount namespaces and overlayfs). This way if you are satisfied you can apply changes (if there is anything to change). The overlay would be removed and created on each keypress. It may be also possible to wrap process access, so one could safely play with kill, but I'm not sure. Even network settings to some extent. But there is nothing you can do with curl -X POST or something similar.
Hmm, is there a straightforward way to be a bit more careful without detracting from the usefulness? What about a default blacklist of commands with the ability to override it through a config file?
I don't think a blacklist could possibly be comprehensive enough. I think you'd have to use some OS permission-limiting system to prevent it and any subprocesses it spawns from have any write access to the filesystem.
Author here: I hope something like that (syscall/capabilities limits) could work. If it is possible, it would just mostly solve the problem, I believe. I'm kinda starting to realize, that probably any command modifying some external state is potentially somewhat risky already, by potentially spinning some exponential feedback loop. (One person on lobste.rs mentioned that foo.bak.bak.bak.bak files could easily get created.) Regardless, I'm generally considering adding a shortcut/option to pause/unpause, and only execute on Ctrl-Enter when paused.
Another possibly reasonable option would be to create a (configurable) whitelist of commands that are considered safe, and keep running the pipeline automatically as long as it only contains whitelisted commands. Any time a non-whitelisted command is introduced, stop auto-running and require Ctrl+Enter or something, until the command once again consists of only whitelisted commands.
This would save you, for example, if you had a custom command called "gr" which was short for "get rid of current directory" (obviously chosen as a pathological example since it's a prefix of grep). As you type the word "grep", auto-running is paused because "g", "gr", and "gre" are not whitelisted, and then once "grep" is fully typed, it recognizes that "grep" is on the whitelist and resumes auto-running. And it never ran the dangerous "gr" command.
Yah, a whitelist of commands which includes bash would probably be best. You'd be fine using it and can simply switch to chainsaw mode by adding bash to the command
I hate to jump straight to Docker, but that seems to be a quick way to restrict access to the local file system. This of course limits utility, but would be much safer. Plus I think the usefulness of a tool like up is primarily in munging the input text anyway.
Docker is an obviously bad solution to this. If you can run a docker file you defacto have root on that computer ^1.
Firejail could do exactly the same thing, but without requiring the user running it download an entire second operating system, or requiring them to have root. Also, the sandboxing mechanisms that docker uses are just generally available and aren't hard to use, so if they went that way they may as well just use the actual syscalls that do what they want instead of importing and entire other operating system to run your commands.
This is where my rant about docker, and the habits it encourages, would go. If I could figure out a way to phrase it politely.
Wow, firejail seems super interesting, thanks a lot for the idea and mention! I'm not sure if I'll manage to use it, but certainly a good direction for some further research!
The problem I was suggesting could be solved Docker wasn't with the privileges of up itself, but the problem of commands you write within up being potentially destructive. I didn't say I thought Docker was a good solution.
Thanks a lot for the link! I'll totally try to look into this. If it really proves to be as easy as a single syscall... Oh, wow, now that'd be a killer...
It does occur to me that if this did system-call redirection and banned the unlink (and maybe rename?) syscalls from working in it's executed commands, you could get a fair degree of safety.
I think most of the useful commands ought to be able to be run in a namespace where they can’t do much. Eg if they can’t see any files then they can’t delete them. Unfortunately they wouldn’t then be able to read config files. Eg I would expect grep looks for a config file to decide what colour to highlight matches and so on.
Perhaps one could run them as a fresh user with few permissions, except it could still write to files if they are writable by “other”
A solution is to have the tool run first in a container or sandbox where any changes can be rolled back ("test mode").
Once the command looks promising on the test container, run the command again in a fresh container to confirm, and maybe see a list of affected files. Finally run on the main filesystem outside of any test container.
Edit: similar ideas with family snapshots already suggested elsewhere in the thread
One possible way to deal with some of those cases would be to intercept those commands and instead display something about what would happen e.g. `deletes 42 files`
Yes this is an amazingly dangerous and stupid idea. Sure it looks cool but so does a Lamborghini until it catches fire or you crash it into a tree at 150mph.
Giving a user an opportunity to correct mistakes before they kill themselves is a major feature of Unix. Sure when you’ve decided you know what you’re talking about but don’t it will quite obediently shoot your face off. We don’t want to make that bit easier though.
Edit: please don’t take this as derision but more factual. It certainly looks cool and opens the idea for further discussion.
Actually it’s more RTFM before you drive the car. A powerful skill many have forgotten which is supplanted by autonomous tools to make tripping over claymores easier.
Stupid is too harsh, and I don't want to discourage anyone from contributing to open source, but I'm not sure this is all that useful either (at least not to me). Is the utility in not having to press Enter because this'll do it for me between every character? I can only think that the reason people like it so much is because it looks cool or most people don't know how to use command line editing keybindings effectively and like that they won't have to hit up arrow and move slowly across the command to edit. The fact that it doesn't have readline support, timestamped history support, or output the output of the last command to stdout at the end already seem like feature losses when compared to using a shell normally.
At least one neat benefit I can imagine is not flooding your history with 20 minor edits to a regex in a long pipeline, and not flooding your stdout with garbage from a bad regex
Inversion of the output (viewing the top, instead of the bottom, without scrolling on longer outputs) is also of note
Imagine if this were simply the shell, with all its normal editing facilities, but laid out in this fashion. It seems obvious to me theres a usefulness anytime you’re trying to create a longer pipeline.. and doing the same thing the shell normally, but leaving a mess behind
Your list of issues are all resolvable (and probably trivially so), and thus not really relevant to the question of whether this thing is worth having in the first place (which requires more fundamental questioning). IE the issue with rm executing when you’re typing in ‘rmore’ is absolutely fatal in the current design and makes it unusable. That can’t be fixed without a bunch of hacks, or losing one of its main features (swapping to enter-to-run is more than fine imo). With such a fatal flaw, it could be fair to call it stupid and useless.
But for lacking readline support on first public listing..?
I couldn't find a good readline library in pure Go, that would also interact well with the tcell package I am using for the TUI. This annoyed me really hard, but eventually I said, screw that, and went with the minimum I had to write to make an input box usable at all :)
I disagree. The example in the GIF is something I've done many times, usually with a bunch of temporary files. Where the commands are side-effect free and relatively cheap, but the specifics are fiddly, this looks to be really useful. But not universally applicable, no!
I think at the point something like this becomes useful though, I've usually just dropped into writing a Python script - which more often then not is the right choice because there's a high probability I'm going to need to do more things soon anyway if it's getting that complicated.
After watching the GIF on the README.md I was like "WHERE HAVE YOU BEEN ALL MY LIFE" (actually just the past 10'ish years)!!!
Then I was like, I bet quite a few ancient wizards out there have some pretty amazing techniques they have picked up over the year. And that it would be great to be a "fly on the wall" watching some of these wizards do simple tasks. And that a YouTube channel watching them do simple tasks could be quite revealing and therefore very educational.
Paul Irish did something like this a few years ago and I still use a bunch of the tools he introduced into my purview. Really, really improved productivity!
Not a series, but a great one-off video by Gary Bernhardt, A Raw View into my Unix Hackery, linked below. I haven't bought his courses, but based on what I've seen of his talks and free material, they may get you part of the way (but I don't think all of the way) to what you're describing.
If anyone has taken his courses, I'm sure they could give better feedback on those. Either way, watch the video.
> And that a YouTube channel watching them do simple tasks could be quite revealing and therefore very educational.
Does such a channel exist? All I've ever found has been tutorials. I want to watch unix veterans use the tools. I've found that to be the best way to learn. Hell, that's how I learned vim.
I do a pretty similar version of this with command-| in textmate, but I've wished for/thought about building a command like this to speed up the process!
The textmate approach does have the advantage (or perhaps disadvantage, for large enough inputs!) of emitting immutable "views" of the data at each step, rather than (presumably?) re-evaluating the pipeline each time, which is nice if the steps of the pipeline are the slow part, or to go back 2 steps and start a new pipeline. Maybe a potential future flag? :-)
Automatically emitting an `up<N>.sh` script is super clever too!
I (partially) recreated your `seq 1 15` example here [1] -- no sleep, but my lshw wasn't as interesting as yours :-) I'm hitting command-shift-| to bring up the "filter through command" dialog.
i have not used textmate, but the description let me come up with this idea. maybe it's useful:
consider the command
A | B | C | D
when you process it in up, split the command at the | and process each one separately. that is first run
A;
save the output into a buffer (BA)
then pipe BA into B, save that into BB, and so on, until you have 4 buffers containing the outputs of A; A|B; A|B|C; and A|B|C|D;
now when the user edits, you can see which section is changed. say the user changes the command C into C1, so the pipeline becomes: A|B|C1|D
at this point you can pipe buffer BB (which has the output from A|B) into C1; write that into a new buffer BC1; and pipe that into D;
the old buffers C and D can be removed at this point, or you can keep them around in case the user undoes their change and goes back to an old version.
Really cool. The key principle that this tool highlights is the advantage of rapid iteration. Whenever you have a problem, spending the time up front to shorten your test iterations will pay off massively in the long run of the problem's life.
This is nice for linear pipes. But occasionally I find myself building more of a graph.
Bash lets you provide file arguments from pipes via the <() syntax (it's turned into a fd reference via proc behind the scenes). And you can also wire up additional pipes with fd renumbering, though this gets ugly in bash, some other shells are more flexible there, e.g dgsh.
A tool to iteratively build a graph of commands and pipes would be the next step.
That's totally where my long-term vision goes! :D But I'm very unsure if I'll have enough time and resources to be able to reach it :/ My strong hope is however that the Luna language[1] & team may get us there faster, and to an even nicer place actually :)
Hmmm, you made me think and try to recollect. And to my surprise, I can't really remember the fact of The Presentation being an influence. But I am quite sure it must have been, I can see no other option. Hm, memory is a tricky thing. And Bret Victor left his imprint on our whole industry.
I use it all the time to select a command from my shell history. It is a very convenient addition to zsh-autosuggestions (one of those things that you use hundreds times per day without even thinking about it).
Late to the party. This screams like it needs to be part of the shell instead of an addition.
E. G. I type a command in the shell, end it with pipe, and press enter, it doesn't delete the command but runs a preview of the output top 10 lines? This way I can continue to the edit the command without losing the context, the pipe, or I can decide to redirect it to a file!
Hmm, interesting thought! The tool needs to do some ugly buffering however; would there be place for it in a shell? Also, I suppose some kind of "Ctrl-Enter" or something would be a better idea instead? So that it could also be hit at any point in the line... But how about scrolling through the preview?
The most frequent use case for this for me would be interactive grep. The "fzf" tools has this along with fuzzy search. And the text doesn't disappear while you type "grep" like it does with "up".
I will definitely try this out too though when I make a pipeline with sed, cut, etc.
Just an awesome tool, I love those moments of "ah that's trivial, why didn't I think of this already" and it definitively applies for this tool. So special kudos for the idea, but also how little go code the implementation took (ignoring dependencies).
There is already some "pre-release" discussion going on over at lobsters btw.
Thanks! :) See also my "prior art" note in the readme ;) I also had the "why didn't somebody think of this already" moment when I thought of it ;) And somebody in fact more or less did, just I wasn't able find it then :)
Reminds me of fzz [0], which is a similar tool but focused more on interactive modification of prescribed components of a known command/pipeline than appending a pipeline.
Novel and interesting idea but honestly i can't see any other use for this than grep and that point you might as well just pipe the output straight into vi, less or your other favourite editor where incremental search is one keystroke away.
Sure, the shell has some powerful tools like jq that editors usually lack but otherwise several levels combo piping through cut, wc, awk, etc are quite rare and are usually only done in crazy shell scripts that actually should be rewritten in more appropriate language.
And the dangers of automatically running any commands shouldn't be downplayed. If you only whitelist a few programs this would be much safer, most users would only use this for grep. Or run the whole thing through a docker container.
rm is not the only issue. Another problem may arise if your command is not idempotent with respect to the state of the system. Especially if it is not idempotent with respect to system state and requires multiple parameters, since it will execute many times as the parameters are typed. Maybe having a configurable debounce on key press with a reasonable default would be sufficient to get around this issue.
What would be really cool is if this tool ran in a pretend mode and showed you what the results would be without actually mutating the system. Then, once the user is happy with the results, the command can be executed for real.
This whole concept reminds me of helm for emacs :-). Thanks for sharing!
Thanks for the ideas, and for the good words! :) Do you know of any nice screencast/video where some similar functionality of helm would be shown? Does it also work for shell commands in some way? I'm not an emacs user...
> This is achieved by boosting any typical Linux text-processing utils such as grep, sort, cut, paste, awk, wc, perl, etc., etc., by providing a quick, interactive, scrollable preview of their results.
lshw | grep network -A2 | grep : | cut -d: -f2-
A better idea would be to work on tools that have structured output, so people can select the keys they want and don't have to scrape.
$ Get-NetAdapter | where status -eq 'up' | select InterfaceDescription
InterfaceDescription
--------------------
Qualcomm Atheros QCA61x4A Wireless Network Adapter
Sounds like fun, but I think there are other ways to achieve a similar workflow without an extra tool. For example, I like to use a tmux session with two panes. On the one side there is a `vim myscript.sh` and on the other side there is a `watch myscript.sh`.
Very similar result and using rm is not that dangerous as you can finish writing your code and it gets executed as soon as you save the file (if you don't like the watch delay you can use one of the many inotify based watch tools or set -n 0.1 or something that suits your needs).
Well damn! This is a really elegant solution for a problem that has been bothering me for quite a while, and for which my own planned solution was much more complex :O
One thing that probably would be useful is when commands are not completely typed or parametrized at the end of the pipeline to still show the last result along with the error message instead of replacing everything and thus removing any context the user had.
Eheh; sorry, MVP! ;) and thanks a lot for taking it lightly and commenting in such a kind and playful way :D It's totally on the long list in some way... please see https://github.com/akavel/up/issues/2 ! :)
I don't think LD_PRELOAD would be enough; there are programs which don't use libc and reach straight for syscalls (e.g. the whole Go ecosystem/runtime)
What about making the tool behave sanely so you don't have to play tricks with the OS to protect yourself from its behavior?
If I type "grep", I don't want it to try to execute "g", "gr", and "gre" as I type it -- not if those commands don't exist, and especially not if they do.
Just don't execute a command unless the user requests it (say, by typing Enter). Problem solved.
It would remove the live execution bits which are kinda neet. I think the cli has a lot of life left in it and ideas like this are worth exploring. Perhaps compromises are necessary but I would love a tool that pretends to execute something shows the results and gives
the option of committing those changes. Think of a find | xargs grep | rm
That then shows the output but prompts the user for commit.
