Hi all, TopShell is a reactive scripting environment that I've been working on for a while. It's purely functional with type inference that runs as you type, as well as a bit of autocompletion.
You can work with SSH, files and HTTP requests, and there's a pure DOM library for scraping. You can draw with SVG or HTML.
The primary use case I'm focusing on is ad-hoc scripting: When you have a task that involves some files, some servers, some HTTP requests, some command line tools, some data manipulation -- or a combination of all that -- and you just want to get a correct result quickly without fuzz.
TopShell looks very interesting, thank you for sharing!
This is a problem domain which I've been thinking a lot about recently. I write tons of shell (bash) scripts. I hate bash and its peculiarities, but it remains one of the quickest ways for me to get work done; I've tried to use many other languages to fill the gap of writing scripts that start as quick one-offs but have the potential to evolve into a more refined piece of software over time, but so far haven't had much luck.
Most languages force me to go to great lengths for everything I want to do in this domain:
- implementing command-line arguments/flags/subcommands: either overly-simplistic (getopts) or overly complex (cobra)
- force me set up error handling for everything that could possibly go wrong
- aren't flexible enough to transform 'soupy' data between various representations without basically writing a full-blown custom parser
- can sometimes be difficult to distribute a minimal-dependency release to remote machines or other users
Functional languages are awesome for many of these things, but can be awkward when dealing with the side-effects that we normally want when dealing with scripting tasks.
So far, what I gather from your readme/wiki is that TopShell is:
a) A functional language seemingly inspired by ML, Haskell, et al.
b) A reactive development environment for the language, à la Jupyter
I love the code examples in the readme; I think this is something most people want to see but a lot of projects overlook. However, I think the readme is a bit barren on the philosophical front. You have a great wiki page [0] for that - I think it could be condensed and integrated into your readme so new visitors have a good idea of your vision.
As someone who would consider using TopShell, my concerns are:
- I can't find documentation on how I can use my own editor/environment if I want, rather than the web interface (which is nice for its notebook-like experience, but I don't see it fitting into my normal workflow)
- I want to continue to integrate with programs which adhere to the unix philosophy; how does TopShell fit into this?
- How dedicated are you to the development/maintenance of the project?
- Do you forsee growing a community of contributors (and if so, how?), or do you want to keep this as a personal pet project?
- Do you think the project will be alive in 1, 3, 5 years?
> I've tried to use many other languages to fill the gap of writing scripts that start as quick one-offs but have the potential to evolve into a more refined piece of software
Groovy is pretty great for this. I have tonnes of things that start off as one liners and end up as full blow JVM (Java/Groovy) applications.
Thank you for the long and thoughtful comment. I think you bring up a lot of valid points, and I don't have complete answers to all of them yet, but I'll do my best to address them here.
> Functional languages are awesome for many of these things, but can be awkward when dealing with the side-effects that we normally want when dealing with scripting tasks.
I think that's a common sentiment, but I have a different take on it - purely functional programming is actually great at expressing intended effects - as opposed to side-effects!
With the right standard library, I think it might be possible to make I/O at least as easy as in imperative scripting languages like Python. As a small example, compare reading a file as a string in Python and TopShell:
Python:
file = open(“testfile.text”, “r”)
text = file.read()
TopShell:
text <- File.readText "testfile.text"
Does that translate to bigger examples? I think it does, but a more serious efford to compare the two would be required for a convincing argument.
> I can't find documentation on how I can use my own editor/environment if I want, rather than the web interface
This is not yet possible. What I have in mind for this is moving the code execution to the backend and having some way for an editor to communicate with the TopShell server - perhaps through the Language Server Protocol.
> I want to continue to integrate with programs which adhere to the unix philosophy; how does TopShell fit into this?
TopShell can use such programs, but TopShell scripts can't currently become such programs. I'm still looking for the right way to do this - one way would be to have a `main : System -> Stream Bytes -> Stream Bytes` where `System` contains command line arguments and so on, and `Stream Bytes -> Stream Bytes` is stdin to stdout.
> How dedicated are you to the development/maintenance of the project?
I'm not sure what an answer to that looks like - though what I can say is that I use TopShell for all my own ad-hoc scripting needs.
> Do you forsee growing a community of contributors (and if so, how?), or do you want to keep this as a personal pet project?
I hope for a community of contributors, but I have no experience in community building per se. There are many low-barrier ways to contribute though - for example, by extending the standard library. Any tips?
> Do you think the project will be alive in 1, 3, 5 years?
I do think so, but as it is, there's a "bus factor" of 1. Please don't bet your business on it right now unless you're prepared to take over maintenece on a moments notice.
> Can sometimes be difficult to distribute a minimal-dependency release to remote machines or other users
To me, a script is a single file I can send to a friend (or put on a machine), and it will just run. Most scripting languages don't really do this, because they assume that package management is handled somewhere else, and implicitly depend on an environment. When TopShell eventually gets a `main` function, the goal will be that all you need to run a script is the script file and TopShell.
> TopShell can use such programs, but TopShell scripts can't currently become such programs. I'm still looking for the right way to do this - one way would be to have a `main : System -> Stream Bytes -> Stream Bytes` where `System` contains command line arguments and so on, and `Stream Bytes -> Stream Bytes` is stdin to stdout.
That actually sounds perfect. If you were able to implement this, I could see this tool coming in very handy for tons of things.
I’ll be keeping a close eye on the project, thank you for your thorough response!
I was initially put off by going straight to the repo's declared homepage (the Playground) and just seeing two empty text areas. The repo README isn't quite enough on its own - as you say, the interactive scripting experience is a huge part of the value here.
I strongly recommend doing what you can to get visitors to that experience as directly as possible. For example: pre-filling the Playground with a short, good example (e.g. the JSON-to-table one). Many people won't get as far as pasting that into the empty Playground. Once I tried it, everything made considerably more sense to me. (I'm not a regular FP coder; I've done some Elm, some Miranda in college, that's it.)
There are plenty of other things you can do (embed a couple of GIFs in the README; offer a choice of examples in the Playground) but I'd focus on (a) getting people to the Playground (b) prefilling the Playground with an example (c) add a few explanatory comments to the example.
Again, this looks great, and I'm eager to use it. Congrats!
That's a great idea - I will look into that. Perhaps even a little menu with a few different examples. One challenge is that the Playground is sandboxed, so the more interesting examples involving eg. SSH or scraping are mostly off limits.
Thanks for the feedback, and be sure to leave an experience report if you do try it! TopShell is very young, and its future evolution will be guided by concrete use cases.
It is indeed in the Haskell family of languages, just like Elm and PureScript.
TopShell is all about the out-of-box experience - not just a language, but an environment that emphasizes an interactive scripting experience, where you can see data as you fetch and manipulate it.
The reason is that I do Scala at my day job, and thus I'm simply more experienced in that than Haskell.
While I've written a compiler in Haskell in the past, and prefer the language in many ways, I don't know how to do front end programming in Haskell, compile Haskell to efficient JS or debug performance problems. I also don't know if there's a good Haskell IDE these days. PureScript would have been risky for me, since I haven't done anything in it before.
The first example is a HTTP request, aren't network requests inherently unpure? as they rely on state outside of the program and could cause side effects...
Purely functional programming languages can't have side effects. But they can have intended effects.
Like Haskell, TopShell uses monads to represent effects, and eg. Http.fetch returns a Task, which is similar to the IO monad in Haskell.
The returned Task is a pure description of how to perform the HTTP request. Every time you apply Http.fetch to the same arguments, you get exactly the same description back, thus maintaining referential transparency.
Only the top level knows how to interpret such a description and perform the effects (eg. open a HTTP connection etc). Thus you compose Tasks into bigger tasks and then let the top level run them.
I've been working on it for a little over a year. The biggest challenge so far has been designing and implementing a type system with the flexibility required for ad-hoc scripting. Building a standard library that can tackle dirty real world problems as nicely as possible is an ongoing challenge too.
In TopShell, Streams are analogous to pipes. The primary way you use TopShell Streams is by combining and transforming them into new streams. The interface for streams is very similar to what you might find in a Functional Reactive Programming library, and indeed it can be used to eg. animate SVG based on data that's streamed from somewhere.
Why would ad-hoc scripting require more flexibility with a type system? What kind of flexibility? Is it that you need more types? Or more union types to handle all the different variations that are the same, but all slightly different?
Anonymous records and variants are convenient in a scripting language because you don't have to declare them. This in turn leads to anonymous record types and anonymous sum types, which require something beyond HM for type inference, and I wanted to avoid introducing row variables.
TopShell uses type constraints for this, and I've never implemented a type constraint solver before TopShell. There are also some extra things to do around checking explicit type annotations, when the user provides them.
You can work with SSH, files and HTTP requests, and there's a pure DOM library for scraping. You can draw with SVG or HTML.
I'm looking forward to your feedback!