There is no real substance in this blog post. At the end:
I chose Go because it is being developed to solve the same problems I have to solve. All by a group of scientists an programmers that are each twice as smart as I have ever been (on my best day).
So, what are the problems you are trying to solve? And why can Go solve them better than, Java, Erlang, C++, Scala, or any other language?
If you could answer that question, you'd have an interesting blog post. And if the answer is 'concurrency via Goroutines and channels' (it always is), explain why they are better than Erlang or Akka's actors, STM in Clojure or Haskell, or other proposals, and you'd have a stellar blog post.
"explain why they are better than Erlang or Akka's actors, STM in Clojure or Haskell,"
They aren't. I analyzed Go for my personal use, and I have a high degree of familiarity with Erlang and Haskell. The conclusion I came to was that it is a step back from Erlang and Haskell's concurrency support, and that I didn't care to step back into those problems. It's easier to do concurrency correctly in Go than it is in Python or C, but it's still pretty easy to make it blow up in your face and take down your whole program. You to both learn Go, and then learn all the implicit (and to some extent still forming) rules about exactly how you can safely handle channels, which the compiler can't help with.
(I tried to keep an open mind, but I'm still of the opinion Erlang's processes with asynchronous mailboxes is better than Go's synchronous channels. Note that's two differences, not one; sync vs. async, and mailboxes vs. channels. And they are effectively synchronous, even when you set a buffer size, because there's still a hard cap on when the channel is full; semantically the difference between a 1-element sync channel and a 10-element sync channel is pretty minimal, you get the same crashes and failures either way if you misuse them, it just takes a bit longer in the 10-element case. Oh, and since "synchronous" seems to have had its meaning mangled here in the past few years, in this case it means that the writing and reading of a channel must coordinate.)
But if you're coming from a traditional Algol language, it's a big step forward. I'm sure coming to Go from Python or Lisp feels like a blast of fresh air, because it really is an improvement. And it gives you this without having to abandon your Algol roots, which most of the rest of your alternatives do (except Akka, though I don't how hard those are to use correctly since they're in a mutable language), and that is saying something.
But if you're coming from a traditional Algol language, it's a big step forward. I'm sure coming to Go from Python or Lisp feels like a blast of fresh air, because it really is an improvement. And it gives you this without having to abandon your Algol roots, which most of the rest of your alternatives do (except Akka, though I don't how hard those are to use correctly since they're in a mutable language), and that is saying something.
You hit the nail on the head. It does feel like a blast of fresh air. I've been eyeing Erlang and Haskell, but have not had to chance to do something worthwhile with them.
Yes. I guess it's a bit late, but let me be clear: I'm critical of Go, but it is a good language (and from me that's saying something, I have a high bar), it has great potential, and I wouldn't be surprised I still find myself working in it in a few years. It's either at or very quickly approaching "bet your startup on it" status. The tooling is fantastic and there are a lot of other great ideas in it.
My personal experience base is freaking weird, and if you don't have Erlang or Haskell already under your belt, then I recommend considering Go quite strongly. You won't be bothered by some of the things I'm bothered by.
And I will find some fresher story to say this on at some point, to be fair.
I'm using Emacs.
Mainly because I'm proficient in Emacs and found it good fitting into my needs.
If you want to argue that Emacs is not the IDE because it doesn't has feature X, then please not that it has great integrated internal environment. For Go "IDE" features check this:
http://dominik.honnef.co/posts/2013/03/writing_go_in_emacs/
That's not an IDE. What's the value of static typing if you have you don't have any tools which make use of it to provide code analysis, completion and refactoring options?
I didn't say anything about performance, and that's because I don't think performance has much to do with it. If Go used asynchronous mailboxes, its performance at most would be ever so slightly worse, it would still be much better than Erlang, and Go's performance loss would probably entirely disappear in microbenchmarks anyhow.
Indeed, that's one of my frustrations with Go, one of the big reasons I chose to leave it alone. The ways in which it is much less safe to use concurrently than Erlang or Haskell aren't performance related, they're related to the choice of primitives. (In particular, I can't see how to build an OTP equivalent for Go. This is a big problem for me, since I was looking at it to replace Erlang, but I'm not ripping out something that has demonstrated how bullet-proof it is to replace it with something more fragile like Go, when performance is not my problem.)
To be clear, I'm just talking about the concurrency primitives, and I do not believe that Go's concurrency primitives are better than Haskell or Erlang's, full stop. That does not mean Go is not a better language for some tasks, and I further believe Go's concurrency primitives are a step up over most current "conventional" languages, equally full stop. For one thing, to program in Erlang or Haskell requires programming in Erlang or Haskell, which is why I mentioned the fact that Go is still an Algol-esque language. That's a legitimate advantage. I've paid the price of learning the alternate paradigms, but I would generally recommend Go to a larger corporate deployment before I'd recommend trying to convince a couple dozen people to learn Erlang. So much... whining.
>>I didn't say anything about performance, and that's because I don't think performance has much to do with it.
Sure but that is why I brought it up. Performance and safety are very often two sides to the same tradeoff. You were saying you thought the concurrency of Erlang was safer, and I was proposing that depending on the problem domain, a little risk can be worth the performance gain. That being said, I think Go makes concurrency easy to get right (much less risky than say C/C++), it just doesn't ensure correctness.
>>...Go is still an Algol-esque language. That's a legitimate advantage. I've paid the price of learning the alternate paradigms, but I would generally recommend Go to a larger corporate deployment before I'd recommend trying to convince a couple dozen people to learn Erlang.
Sure but that is why I brought it up. Performance and safety are very often two sides to the same tradeoff.
Haskell is pretty much the counter example. But Haskell will always to be weird to most imperative programmers. This is why I am far more exited by Rust than by Go, it promises to deliver performance and safety in an imperative language.
Also, there are actor implementations on top of languages that are more performant than Erlang. E.g. see Akka for Java and Scala.
I said "very often" for this reason, there is always an exception to the rule. I do love Haskell too.
>>This is why I am far more exited by Rust than by Go, it promises to deliver performance and safety in an imperative language.
To be clear we were talking about concurrency safety, not type and memory safety; Go is type and memory safe. If an organization using Go chose to tell programmers they could not send pointers between goroutines or share global state, Go would be just as communication safe as Rust. Just in Go, this communication safety is optional.
> To be clear we were talking about concurrency safety, not type and memory safety; Go is type and memory safe.
Not if you're using parallelism (GOMAXPROCS > 1).
> If an organization using Go chose to tell programmers they could not send pointers between goroutines or share global state, Go would be just as communication safe as Rust. Just in Go, this communication safety is optional.
It is in Rust as well. You just have to declare that you're opting into data races; safety is the default and enforced by the compiler if you don't opt in.
> I do not believe that Go's concurrency primitives are better than Haskell or Erlang's, full stop.
Unless I'm mistaken (and please correct me if I am!), synchronous primitives can be used to implement asynchronous primitives, but the converse isn't true. If that's the case, why would Erlang's asynchronous primitives not be inferior to Go's synchronous primitives?
Go vs Haskell/Erlang has a similar showing in TechEmpower's Web Framework Benchmarks
Many servers solve the c10k problem (not that it is an issue for many users anyway), but it always seemed as much interesting to me how your code is structured with a highly concurrent server. E.g. Yesod and Snap (via WAI) allow you to stream the request and response with conduit (and enumerator) and build nice and composable pipelines. Play 2 is also moving in that direction with their iteratee implementation:
Iteratee/pipe-based processing would never be possible in Go, since it does not offer parametric polymorphism and typeclasses. Sure, you could write transformations to take and produce interface{}s, but it will give you as much type safety as a dynamic languages and type switches in every function.
This is not the answer you are looking for. But I'm still figuring out that part. One of my problems have been performance/memory usage.Like many of you out there, I don't have endless funds to cover the infrastructure costs. So I have to make with run of the mill VPSs and web hosts. Go's speed and lower memory consumption (compared to my implementation in Python/Django), allows for my financials to stay in check. This project is privately funded, and I'm not SV rich. I will continue talking about the problems as they keep appearing. That is the purpose of the blog. I appreciate the bluntness.
Please do not take my comment as a personal attack. Blogging is useful and practice makes perfection :). I was more surprised that it is at the top of Hacker News and as such it's more a critique on voting behaviour ;).
I am looking forward to see your search innovations...
Oh, I know its not personal. I know that the HN crowd seeks more technical content. Which I will be providing. Its just that, as of right now, this is pretty much what I got. But keep posted. I am building and releasing often.
>>So, what are the problems you are trying to solve? And why can Go solve them better than, Java, Erlang, C++, Scala, or any other language?
Compared to X, Go has Y:
Java: Less verbose, faster builds, native code (no JVM needed), cleaner OO, structural typing, better memory usage, structural typing, more convenient concurrency
Erlang: Better performance, better memory use, cleaner syntax, easier to train people on/hire for, statically typed
C++: Type and memory safety, much simpler syntax, better build times, large standard library, structural typing, garbage collection, easy and safe concurrency
Odd choice. A little more research could've yielded a migration to something like Clojure with the benefits of Lisp and an industry-backed stack (JVM).
Of course using an offbeat stack is going to hurt - doesn't mean you have to give up Lisp too.
A decision made odder still since being an independent project, one would think maximizing individual productivity would be the only hope of yielding anything novel and useful in a field like search that isn't a fitting of an off-the-shelf to a business problem. (Something I've done before, it was quite neat.)
Go is not a language focused around individual productivity in any capacity or scope. It's the language of pyramid-building Egyptian Pharaohs. (Throw 1,000 lives at it, it'll get 'er done...)
I say this as someone rather envious of any language that has a native compiler that builds static binaries. (Oh, OCaml...)
As we ponder on what orangetide is up to, lets keep in mind he's not explained or revealed anything at all about Nuuton and for all we know - could be having us on for the sake of self-promotion/promoting Go.
The incorrect speculation about the origins and purpose of Go make this even stranger.
Go exists to make people like Rob Pike happy. The Plan9 and Suckless community of programmers. Those types. That it happened to solve the problems of some Google programmers who were making it was almost incidental.
As we ponder on what orangethirty is up to, lets keep in mind he's not explained or revealed anything at all about Nuuton and for all we know - could be having us on for the sake of self-promotion/promoting Go.
Well, you are right. I am promoting Nuuton. After a almost a year of working on it and not speaking much about it, I decided to start opening up. I don't want for it to fail due to being too secretive. Time will tell if this is the right decision. But I'm having a great time building in Go. If all fails, I will still have the knowledge of doing it.
Edit
I also considered Clojure. Even spoke about it to some people on this board. But it seemed to take me along the same path Lisp did. Still a fine language, though. I have nothing against it. It did not make much sense in my mind.
> Go is not a language focused around individual productivity in any capacity or scope.
Seems to have been my impression as well. Lisp with its macros can make an individual programmer who doesn't need to worry about coding conventions incredibly productive. I see Rust has a semi-powerful macro system too although not to the extent of Lisp, as it doesn't look like you can evaluate arbitrary code at compile time.
> Go is not a language focused around individual productivity in any capacity or scope.
I don't know the motivation behind the people developing Go, but I found it very easy and convenient to work with it, almost as easy as with Python sometimes, which is mind-blowing IMO given the static, compiled, strict, etc. nature of Go. So again, don't know whether the guys behind the language were motivated by personal productivity, but they have definitely achieved it.
Though I didn't really try Lisp other than quick tutorials, so I can't make a comparison with that.
>Google started developing Go for one simple reason: to simplify the act of writing search systems.
Not really.
For one, Google didn't develop Go (it wasn't a company decision to get a new language, as Java was for SUN, .NET was for MS and Dart was for Google). Go was merely started by some a team working at Google as a side project and it grew a little from there. They do pay several people to work on it nowadays, but it's not some "bet the company" language. Also, Google hasn't mandated replacing any C++/Java stuff with Go, and nobody is touching their search engine code. If it happens, it happens organically and for lesser stuff.
Second, Go was developed for "systems" software (albeit in the sense of network servers and such), not search specifically.
> Go was merely started by some a team working at Google as a side project
History shows that a lot of extremely successful things started out that way.
> and it grew a little from there
That's a bit of an understatement. Go has hundreds of contributors, and is already used by thousands of companies.
> They do pay several people to work on it nowadays, but it's not some "bet the company" language.
Definitely not. For that matter, neither is Dart.
> Also, Google hasn't mandated replacing any C++/Java stuff with Go, and nobody is touching their search engine code. If it happens, it happens organically and for lesser stuff.
You make it sound like Go isn't being used at Google, which is disingenuous. No, it probably hasn't replaced the crawler and pagerank logic--it'd be silly to rewrite all of that--but it is continuously being picked for new projects internally. Some of the few such projects that have become public are Vitess, the database frontend that powers YouTube, and the code that powers dl.google.com.
(That there is no known mandate to replace existing code isn't indicative of much. It's a bad move to rewrite that much code for any reason.)
> Second, Go was developed for "systems" software (albeit in the sense of network servers and such), not search specifically.
It started out as a "systems language" (which roughly translates to "for writing backend software", but was commonly misunderstood), however that description has been removed since Go has proved useful in other areas. It is now simply a general-purpose programming language.
>>Go was merely started by some a team working at Google as a side project
>History shows that a lot of extremely successful things started out that way.
Also a lot of failures. But both cases are beside the point. I didn't correct him on whether Go is good or not. Just in that it was not started officially by Google.
>That's a bit of an understatement. Go has hundreds of contributors, and is already used by thousands of companies.
And yet it's still a totally marginal language in the grand scheme of things. Not even 1/100 of what something like Java or .NET is, for example.
>Definitely not. For that matter, neither is Dart.
Agreed on Dart (and I don't think I said it was). But .NET and Java were.
>You make it sound like Go isn't being used at Google, which is disingenuous. No, it probably hasn't replaced the crawler and pagerank logic--it'd be silly to rewrite all of that--but it is continuously being picked for new projects internally. Some of the few such projects that have become public are Vitess, the database frontend that powers YouTube, and the code that powers dl.google.com.
Yes, I've read about both, I follow Go since the beginning. Those two quite humble projects are the only things that have become publicly known for 2 years now. A db load balancer (which had some Go issues they had to code around even), and an abandoned part of Google infrastructure in C++ that a Go team member volunteered to rewrite in Go.
>however that description has been removed since Go has proved useful in other areas. It is now simply a general-purpose programming language.
I believe the reason it that description has been removed was because people complained that "systems software" means specifically OSs, drivers and such, not just backend software.
> It started out as a "systems language" (which roughly translates to "for writing backend software",
Not really. While Go was presented as suitable for writing classically backend software like databases, it was also presented as suitable for writing compilers, programming tools, debuggers, IDEs, and web browsers, none of which are "backend software."
Yeah, that's what I meant by "roughly." I was trying to point out that a lot of people assumed "systems" meant "kernel/real-time development", which Go, while not decidedly bad, is not that well-suited for.
To the point that it's developing at a rapid pace: Yes and no. There is a lot of development happening on Go, but the APIs that you can access in the stable releases are not changing behavior.
The Go 1 compatibility contract guarantees that no breaking API changes will happen for many years, barring some serious security issue: http://golang.org/doc/go1compat.html If it compiles now, it should continue to compile and work as expected in the future.
I wanted it to be pure Python because I love Python. That is until I had to work with threads. After spending some time working with gevent, I decided to simply use another language to deal with that.
Yes, I know. Had to work on a commercial project that used gevent. Boy was that a mess. The project was a server that had to handle 30K+ hits/second. The client's engineering dept could only get it to do 3K. When I saw they were using gevent (along with a custom framework that lacked any documentation), I knew their goal had a long way to go. Last thing I heard, they were having trouble hiring people to work with it. I'm glad I dodged that bullet.
If you are seriously hoping for this to be a full time business then I would strongly recommend you read "The Lean Startup" or similar. This blind technical pursuit of perfection is a pretty classic approach for us techies and something i'm guilty of myself. It was actually that process that made me realise what my strengths actually were and what I would need to do to start a world class company (work with great people).
If this isn't really a commercial project then just ignore the above and happy hacking, perhaps you should consider open sourcing it?
I'm always telling people here about how they should sell first, and code later. This project has been challenging in that regard, but I have been spending a lot of time figuring out the business aspect of search. It is challenging because one of my main goals is to respect privacy. These days, violating the privacy of the users is a sure way to cash in. But I don't want to do that. As a result, I've had to write a couple of versions to see how it would work out. My current plan incorporates a product line right into the stack. That way, if/when the time comes, I can focus on writing the necessary parts. No need to scramble and try to make it fit.
I do follow the lean principle. But one cannot blindly adhere to them. Nuuton is not the sort of project where you just try and get something to work. There had to be a foundation first. But from now on, development will follow the build-measure-learn cycle. I am open sourcing parts of it. Mostly Go libraries that might result from developing it.
Things got complicated quickly. I was writing too much code to achieve simple things. Plus I was writing a lot of one-off code. I prefer to write as much reusable code a I can. Mostly because I'm lazy and I like to keep things simple.
I'd be interested to hear more details on this. Writing non-reusable code and lisp seems antithetical to me. Do you mean the lack of libraries and interfaces to other services?
Yeah, it sounds very dubious to me that the amount of code in Go ends up being much less than what's required to achieve the same in Lisp. From what I've seen Go looks verbose in comparison with the lack of macros and higher level programming.
It has proven to be less verbose for this project. Probably due to my lack of Lisp skills. I have been in love with Lisp for years, and study it actively. But I'm not a PG with it. Sadly, my natural programming skills are not that great.
Its a combination of that factor, and my lack of understanding of the problem. Personally, I did not feel Lisp was very fitting to my work flow and goals. It still is one of my favourite languages, but not the right choice for this project.
So I guess if I posted about how I chose C++ and Perl to write Enginuity Search Engine I'd get no love on HN because its boring, stable, scales and just works.
Maybe not. Or maybe you would, if your reasoning were well-presented and reflected a logical approach and conclusion, and if it were insightful in a way that people found interesting and useful.
Well I might have a go at it soon in our engineering section of our company blog. Have some interesting network performance analysis info to share as well.
I chose Go because it is being developed to solve the same problems I have to solve. All by a group of scientists an programmers that are each twice as smart as I have ever been (on my best day).
So, what are the problems you are trying to solve? And why can Go solve them better than, Java, Erlang, C++, Scala, or any other language?
If you could answer that question, you'd have an interesting blog post. And if the answer is 'concurrency via Goroutines and channels' (it always is), explain why they are better than Erlang or Akka's actors, STM in Clojure or Haskell, or other proposals, and you'd have a stellar blog post.
But really, this at the top of Hacker News?