Nothing necessarily surprising about this. Yes Go is made by Google, but not by all of Google, just like this project isn't necessarily a first-class Microsoft project (although it might be). And Microsoft has contributed significant amounts of open source in the past few years. Lua is also a great language to support first-class, as its popularity is increasing more each year due to its ease of embedding.
> The VM is designed differently (I'm encouraging Brian to write a blog post about this), but in a nutshell it is designed for better error handling and for adding a debugger
Looking at the VM, they implement function calls by recursively re-entering the VM in Go. The VM implements neither tail calls nor coroutines, and I don't see how they could implement the latter without some serious refactoring.
Perhaps by adding a debugger they mean they implemented the VM such that they could repurpose existing Go debuggers, which is why there's a 1:1 mapping between the Lua call chain and the Go runtime call chain.
This was open sourced very early. It is being built with the intent of supporting Helm v3 and by open sourcing it early it enables other Helm maintainers to get involved and to import it into Helm v3 early.
I expect more docs, details, and improvements will be coming
I'd think one of the big advantages of rewriting Lua in a GC'd language would be that you could ditch the stack API, which is necessary to keep track of pointers but not so intuitive to use. Seems like they didn't take that route though.
What's the alternative to a stack API? If you look at Python and Perl they use a combination of (1) an explicit stack similar to Lua, (2) instantiation of heavy-weight list or array objects for passing arguments, (3) explicit reference counting, and (4) code generation.
Lua's stack protocol can sometimes be verbose, but I find it infinitely more simple and elegant than the existing alternatives.
I think you tend to see long, unwieldy code blocks that invoke Lua functions multiple times because you can, despite how tricky it can be to track the top of your stack. You rarely see this in Perl or Python because it becomes impossible to manage long before that point. So the solution for Lua is the same as for other interfaces--use shorter, simpler functions and compose them. This is usually easier and more performant to do in Lua, anyhow, because there's less fixed boiler plate and indirection necessary; not to mention Lua gives you many more facilities for stashing and managing your C state across calls (like closure upvalues, userdata values, continuation context cookies, etc) without resorting to global variables either inside or outside the VM.
Yeah, implementing a precise GC is a pain, but in Go the hard work of that is already done for you. If I were to write my own Lua-in-Go, my inclination would be to try to make an API something like this
The above is shorter and also significantly more efficient because you're not creating the intermediate table object. Even in Go, given Lua's coroutine and tailcall semantics (assuming they were implemented) the Go compiler would likely be forced to always heap allocate the argument list table. There's a reason PUC Lua is blazingly fast, and its partly the same reason why LuaJIT is so fast--the Lua stack protocol is a thin abstraction that easily admits a simple, efficient implementation yet still managing to provide an effective interface boundary that doesn't leak (beyond the fact of the protocol itself).
I don't think there's any good middle ground here. The best options are (1) Lua's explicit stack protocol or (2) some sort of type inference that permitted a direct call, like lua_tcall(L, "five", 5), into the VM. You can actually implement the latter in C somewhat easily (at least if you stick to simple data types) by using _Generic and __VA_ARG__ macros to generate the stack pushing code.
But I rarely see this done because it's a lot of magic for little gain. And where I have seen it done it's always been annoying because it's too leaky--invariably you have to fallback to the stack pushing code because there's a limit to the types of automagic type coercions you can accomplish between two drastically different languages. So you have to learn their magical wrapper routines in addition to the regular Lua API.
Many years ago I abused __VA_ARG__, __builtin_types_compatible_p, and libffi so I could invoke ad hoc functions as libevent callbacks without having to proxy the call through a function that unwrapped a pointer-to-void cookie. See http://25thandclement.com/~william/projects/delegate.c.html I still think it's kinda clever but I can't remember the last time I actually used it. Even in code bases already using delegate.c I ended up returning to the slightly more repetitive but transparent and idiomatic patterns for installing libevent events.
rather than having to roll your own. Part of the function of the stack protocol is to minimize the number of unique functions needed to load and store values across language boundaries. So there's a single [low-level] function for communicating a number value in a type-safe manner, rather than a multitude of functions--one routine for setting a number value for a table string key, another for a number value for a table number key, another for a string value for a table number key, etc. The permutations explode, and unless the host language has some sort of generics capability then it adds significant complexity both to the interface and to implementation maintenance. (See Perl XS.) Alternatively you can reduce the number of permuted interfaces by using printf-style formatting strings and va_lists, but that's not type safe. (See Py_BuildValue.)
Notably, Lua 5.3 removed lua_pushunsigned, lua_tounsigned, luaL_checkint, luaL_checklong, and many similar routines as there was never an end to the number of precomposed type coercing, constraint checking auxiliary routines people demanded. They decided that the only way to win that game was to not play at all. But with C11 _Generic and a dash of compiler extension magic you can actually implement a richer set of type agnostic (but type safe) numeric load/store interfaces in just a handful of auxiliary routines. For better or worse, though, Lua doesn't target C11 nor wish to depend on compiler extensions. "Batteries not included" is a whole 'nother debate in the Lua universe.
> Lua's stack protocol can sometimes be verbose, but I find it infinitely more simple and elegant than the existing alternatives.
This. TL;DNR, used a C/Lua binding to explain the Lua stack to a fellow engineer. Took a few post-it notes to keep track of 4 levels of stack, 1 pcall back to Lua to sort an array, and forward the results on. It just worked, was elegant, and blew our minds.
Today on Friday at 5pm I was talking about a ‘problem’ we had with a Lua daemons compared to their C brothers. Due to the use of tables, each time you queried them, the output order was always random. Computers don’t care, but a big user is also engineers and finding fields wandering all over the place is a pita. So what if we ordered the keys alphabetically instead? The code is Lua -> C -> IPC -> Output. The order is set in the Lua->C binding, so what if we sorted it in the binding.
But the tables are arbitrary size and you’d have to allocate memory, put them in a structure of some kind, etc, etc. Then we realized we could just put all the keys in a Lua array, and sort them with table.sort(). We could then use the array to get key/values in order, all from within the C binding.
Being late Friday, and having talked it through let’s just do it. Plus it was a handy teaching moment to explain to a fellow engineer how Lua binding worked, including a call back into Lua. By 6pm we had the binding recompiled, tested and it’s just worked.
Lessons learned:
1. The stack can seem confusing, but once you learn how to work with it, it’s pretty amazing.
2. The Lua reference docs are amazing. They clearly list what they pop and what they push onto the stack.
3. Calling back into Lua is.
4. It looks like write only code. Reading the code needs a post it note to keep track of the stack, along with the Lua reference manual. -- comments...
Based on my experience porting Wren's interpreter to Go, I would expect this to be about 3x slower than a C interpreter (without JIT). But someday I hope to be pleasantly surprised.
I don't believe Go has been optimized for compiling an interpreter, so perhaps there is low-hanging fruit, should the Go team decide that it's a priority.
I haven't open sourced it (yet). It's incomplete, has a slightly different syntax than Wren, and I basically ran out of steam and switched to another hobby.
But if someone else wants to work on it, it might be fun.
Go was originally intended as a systems level language, suitable for writing extremely efficient servers, dealing directly with raw bytes where needed, therefore enabling writing code at both high and low levels. VMs typically take advantage of the lowest level of code that their language supports to be as efficient as possible. There's no reason Go code can't be as fast as C++ code with enough time for optimizations, and there's no reason a VM written in Go has to be slower than any other low-level program written in Go.
I believe there are compiler improvements that can be made without changing the Go language, like computed goto's for large switch statements. That will help.
However, to really do as well as C, Go would need a more space-efficient union type. A Go interface is always two pointers due to how the garbage collector works, and that's not going to be as efficient as using something like NaN-encoding for a union between a pointer and a float64.
For go code to be as fast as C it would necessarily not look like go code as you’d have to avoid anything that would involve non-deterministic allocation or garbage collection. Sure, it’s possible. But it doesn’t make sense.
I wonder what sort of VM monitoring will be available. Something that was interesting with HekaD's lua integration was the ability to monitor CPU usage and evict bad actors. Can it see and perhaps even limit?
This was something I was wondering about with Cloudflares isolate implementation as well. I read that in node/v8 you can inspect an isolates CPU usage, but no way to limit it apparently? If you start out with 1/2 of a CPU but your program gets squeezed down to 1/1000 as more isolates are added or existing isolates use more time you could be in for a big surprise as your program no longer performs adequately, perhaps to the point of not functioning properly at all.
Isolates are limited to a single thread. There's no built in way to throttle an isolate's CPU usage, but you can manage the thread it belongs to. Which means isolates aren't inherently worse for CPU contention than processes would be, you just need to roll your own throttling instead of using something like cgroups.
People always over look this, but it's critical to getting to a truly fair-use shared platform. You can do it with PUC lua using debug hooks. Not other runtime I know of has this feature.
You should read the source you are linking to instead of being needlessly snarky.
The function you call emits IR, and it's doc block says:
// EmitIR compiles the Lua script similarly to Compile but instead
// uses the long listing options to capture the compiled IR dump.
// EmitIR returns the Lua bytecode encoded as a string or any error.
luac is the Lua compiler, not a reflection of what language the engine is written in. I'm not saying it doesn't call a compiler written in C, but executing a binary named `luac` doesn't mean that it does https://www.lua.org/manual/5.3/luac.html
Rust has good bindings for PUC Lua, but this looks like it could have several advantages over Rio's runtime. I did a quick search for any Rust bindings to Go libraries but haven't found any yet. I opened an issue in rlua to discuss bindings to golua, even though that would definitely be a separate project, but we have discussed there before other Lua runtimes. If anyone else is interested in using this library in Rust, here are those issues:
shiplift uses an API. Looks like calling Go from Rust is never done. I would guess because of the huge performance hit at 3 orders of magnitude worse than C (according to the users.rust-lang link above). Will still be interesting to see how Golua impacts the wider Lua ecosystem.
Would this allow Lua code to directly use Goroutines, and take advantage of Go's features for concurrency and parallelism?
I'd love to see a high-level language like Lua (or Python, Ruby etc.) with good threading support. Building a language runtime in Go might be a good way to get that support more easily?
While I'm wishing for things, could I have good single-threaded performance too? This would likely need a JIT, which unfortunately doesn't seem possible in Go at the moment [0].
Maybe I should give up on my desire for a high-level language and just use Go itself, where concurrency, parallelism and good performance are all available today...
You have various lisps at your disposal, with my favourite Guile offering a full working Concurrent ML implementation instead of the almost-CML offered by Go.
I'll admit I don't use or know Go, but I have been in the Tarantool docs lately. It is a JIT compiled "drop in replacement for Lua 5.1 or JIT" and has fibers and channels inspired from Go. https://medium.com/tarantool-database/dbms-as-an-application...
You don't have to call it giving up. Go is pretty easy to pick up if you're used to high level languages. I would recommend reading a pretty short book called "concurrency in go" (Katherine Cox-Buday) for some really concise education on it.
This is great, since I've been meaning to do some stuff in Lua instead of JavaScript (mostly Functions-like stuff that require simple scripting). Having more options is always nice.
Android's OpenGL/Vulkan debugger, Fucshia TCP/IP stack, some driver research, a GCloud hypervisor, a basic POSIX kernel implementation presented at this year's USENIX are some of the examples of Go as C replacement.
It was quietly open sourced earlier this week at an early state so we could start looking at importing it into something else (https://helm.sh v3). You likely will not find any examples, yet. It's not release quality quite yet so there will likely be changes as it becomes production quality.
I'm amazed people found this already. The team over there who owns this is excellent. I expect it to work nicely once they have had a little time to get everything together.
I see that contribution is guarded by CLA agreement. I often wonder: if I don't want to sign a CLA, couldn't I just fork the whole repository and keep going on my own?
I don't want to be "that guy" but I think its a valid question to ask- why didn't they write this in ... ... .... Rust?
I think go and rust serve two somewhat different use cases- go being for distributed computing in a nutshell, and rust for fast running programs. And I think Rust would have been a better choice
Edit: looks like everybody and their dog has a Lua engine now- maybe I'll create one in rust
Idiomatic Go code tries to use cgo sparingly, if at all. And the Lua embedding API makes calls into C with a very high frequency. I haven't tried, but I suspect that the combination of these two factors would make embedding a Lua engine written in C (or, say, in Rust with a C ABI) into a Go program disastrous for the performance of Go's green threading runtime.
Alternatively, while it would be nice to have a Lua engine entirely in Rust, Rust goes to great lengths to make calling into C have no runtime cost, so you generally just see people writing Rust wrappers over the C API, which tend to impose varying degrees of additional safety on top (Lua internally does a ton of setjmp/longjmp, among other things, so it's not easy). A good example of "wrap and add safety" is Chucklefish's wrapper, being used for their game Witchbrook: https://github.com/kyren/rlua
The why not Rust is a valid question. It's because this was put together with an intent of being embedded in an application written in Go (https://helm.sh).
I'm a Helm maintainer so I'm familiar with the history of how this came to be.
What is the advantage of using a full blown language for Helm configuration instead just using something, which is a bit more flexible, but not Turing complete like: https://github.com/dhall-lang/dhall-lang (Or something similar)
As Helm so far could operate with a simple template language , and now suddenly jumps to a Turing-complete configuration language - this seems quite large step... What is the justification?
Good question. For 90+% of things a simple template languages works. But, there are cases where you need something more. For example, the features that were baked into Helm v2 to support installing OpenStack in Kubernetes were complex. IIRC, no one on GitHub has used these but they are complicated and in use privately. That complexity should be pushed to the chart rather than Helm. Helm shouldn't need to hold that complexity, which is in the direct line for everyone, for those complicated apps. This is where Lua comes in.
Lua provides the ability to have the language embedded easily (it was designed for that) for use in cross platform situations (Windows included).
One of the motivations (linked in another comment) may reveal why:
> We also wanted clean and idiomatic Go APIs
If you want to use this as a scripting/plugin language for Go programs (say, for those distributed computing programs you mentioned), it's pretty defensible to make it integrate with Go better by... building it in Go.
I am by no means an expert on that particular subject, but from what little I do know, both Go and Lua (the regular C version) have garbage collectors optimized for low latency; I may be wrong, but if I am not, it looks like a good match, doesn't it?
Likely they wanted an idiomatic Go API, and maybe had more in-house Go knowledge than Rust knowledge.
I can imagine how exposing a Rust API to a non-Rust program could be as problematic as a C++ API: your common denominator is a C API, which adds cumbersomeness and removes (some of) the safety.
The more interesting question is "why Lua?" The issue others have linked says "We need a Lua 5.3 engine", I'd be curious why. Presumably that means they want to use some existing Lua plugins? Is this something that's built into Azure?
I think you got it backwards. Go is good for toy-interpreters, compilers, programming languages in general. Pretty bad for distributed computing though, Rust is definitely better for that use case if your only choices are Rust and Go.
Rust's type system is derived from the ML variants, and as such is fantastic for writing compilers and interpreters. ML stands for "meta language", ie. it's named after how good it is at writing compilers in.
Kind of, I prefer the productivity I get from the GC in ML language variants.
Having done some compiler related work back when we had Caml Light, OCaml was still called Objective Caml and Mirada was being turned into Haskell, I don't see what I would gain from having to deal with the borrow checker.
The discussion here is between Rust & Go. My point is swaping the GC for the borrow checker in order to get ADTs is a net win to me.
If we're veering off into hypotheticals that don't have anything to do with the question at hand, I personally write compilers and interpreters in Rust rarher than another ML because my use cases tend to be worst case latency sensitive enough that a GC is a non starter.
