I built a fault-tolerant system for binding lots of disparate and network-unaware services via a monitored event-driven communication protocol to a network-fronted cloud architecture. It's called Katamari, and it's the version of Fuzed (this same concept just for Rails instances) that I modified to run as the majordomo of Powerset's realtime query analysis layer. It let us take un-threaded and network-unaware software that talks over local file descriptors and turn them into a cloud service.

Katamari also has some fairly good fault tolerance characteristics. It aggressively tries to maintain a cloud and it has no real single points of failure. It recognizes hung or crashed instances of individual workers and restarts them. It also supports arbitrary versioned resources existing in parallel, which is great when you want to do A/B testing.

It was also designed so that it was usable in both a development and production environment. While currently little slow for most production uses (~2ms overhead for most queries, with a special case of ~12ms), it's plenty fast and runs in less than 10mb of memory on most linux systems, and can transparently go from a 1-box to a N-box solution.

Microsoft open-sourced it for me (after 2 years of bureaucratic nonsense), so I am just pulling out the useless and crufty parts before I put it up on GitHub. I've given 2 talks about the technology, one for Fuzed at Railsconf and one on Katamari at the last Erlang Factory. I hope to have a modernized version maintained this year, and start moving the tech forward.

Katamari is about 10k lines of code; with about 2/3 of that being Erlang and the remaining 1/3 being Ruby code for the glue (Ruby was Powerset's primary glue and binding language, so it is the only supported binding). Katamari aggressively leverages the characteristics of Erlang to accomplish this feature set; its safe to say all the fault tolerance wouldn't have been doable without Erlang's implementation providing the basics for it.

ErlyDTL, a Django Template Language compiler: http://github.com/evanmiller/erlydtl

Chicago Boss, an MVC web framework in progress: http://github.com/evanmiller/ChicagoBoss

This doesn't get mentioned enough, but Erlang has excellent compiler tools, so if you're willing to roll up your sleeves, you can usually hack the language to fit your problem. For instance, Chicago Boss ships with a nice but completely non-standard database querying syntax, described here:

http://www.chicagoboss.org/api-db.html

You will weep for joy when you rewrite a gnarly SQL query with a few well-chosen Unicode set operators.

The compiler tools also make it easy to generate Erlang AST and compile it to BEAM (Erlang bytecode). That's how ErlyDTL works. I haven't done performance benchmarks myself, but the folks at Zotonic report that their Django templates and controller code come out 10X faster than comparable PHP code.

I think a couple of things prevent Erlang from being a do-all language:

1. Lack of libraries. The situation has been steadily improving, but Erlang's VM and processing model makes it difficult to draw upon the "C tradition" in the same way that Perl, Ruby, and the others can take a popular C library and write a thin wrapper around it.

2. Inferior OS integration, at least with UNIX. For instance, custom signal-handling facilities are non-existent, and you have to jump through strange hoops to do simple stuff like close only the write end of a read-write pipe, or get the exit code of an exec'd process. Weak sauce for anyone who has spent a lot of time with Perl, Ruby, or C.

ErlyWeb, an open source web framework: http://github.com/yariv/erlyweb/.

Twoorl, an open source Twitter clone: http://github.com/yariv/twoorl/.

Vimagi, a graffiti-like facebook app: http://apps.facebook.com/vimagi-paint.

Also, parts of Facebook chat (I work at Facebook).

Also, thanks for your work on mysql driver, which I happen to use for PavoMe.

HN is such a nice community :)

I built a very specialized mail server (MTA).

It was a tedious, frustrating experience, even though you'd think it would be the perfect language.

It ended up being too slow (probably because of all the string processing)

In my entire programming career, I've never before had to join a mailing list and ask questions to get things done, that should tell you something.

I'd be interested in hearing more about the frustrations.

yes indeed I'm increasingly using couchdb, planning on picking up erlang and it would be useful to avoid the mailing list if you can pass info along...

This was a few years ago, so my memory is foggy and things might have changed, but I remember how uncomfortable erlang felt in a unix environment. I might have felt the same way about java, since its also sandboxy.

I think the first time I hit the mailing list was because I couldn't figure out how erlang can boot up and change its effective user id. (setuid). The reason being it needs to grab a privleged port as root. Well forget that, it has no idea what that means. The recommended way is to use iptables port forwarding. I think thats how couchdb does it (or one of those nosql things)

Logging also didn't seem to fit well with linux. You can manage to write some stuff to syslog, but erlang really prefers its confusing way of logging. See things like this http://www.erlang.org/doc/man/sasl_app.html.

Oh and if you decide to just write data to the error log in the simpliest way possible, like this error_logger:error_msg("An error occurred in ~p~n", [a_module]). Guess what, it doesn't open that log file in append mode, so it wipes it out every time you restart the erlang process. You have to write and install your own error log handler to open the file in append mode.

Want to do some dns lookups? Sounds simple. It starts up a gen_server thread... and starts caching the results! Talk about principle of most surprise. And it doesn't handle mx records. But if you get on the mailing list the guys will tell you there's an undocumented library deep in the distro that will handle this, and probably won't change. So you give up in frustration and just write a port to run dig.

It just went on and on.

You can really tell that erlang evolved to run self sufficient on special hardware and not in a supportive unix environment.

And of course there are the basic language frustrations. What bothered me the most was simply the lack of a 'return'. This meant you couldn't 'short circuit' a function, like "if something then return", and it often resulted in another level of nesting, even for the simpliest stuff. Yes I know that sometimes you could use function pattern matching to factor this out, but not as much as you hope. Having a 'return' statement in erlang is part of a religious war, which I'm not going to fight, but I'm strongly on the having side.

I found some erlang things worked great for my project, like the server and finite state machine 'behaviors', but when I realized the string processing was so slow I just switched to stackless python, wrote it in a fraction of the time, never used it, and ended up keeping the existing perl version :)

You can write a MTA in any language, so its probably not the best fit for erlang. The best fit for erlang would be something that only erlang could do, and just suffer through it ;-)

Where did you get all that from? Erlang was never intended for research but as a tool for writing real switch applications. Yes, the original Prolog interpreter was slow, but that has not been used for almost 20 yrs. We never had "big iron" computers to play with, and never really needed them either as the system has been efficient enough to run on computers that were cost effective for the actual applications. The implementations have always been optimised for applications that are highly concurrent and extremely fault-tolerant, like telephone switches. Yes, it was a bit slow in being able to use more cores but that was because Ericsson, who still maintains the BEAM implementation, did not have need of it. Now, it will take as many cores as you can throw at it, and be able to use them if you have written a truly concurrent application (Amdahl's law). Transparently too.

Ericsson could have had a need for multicore already back in 1997, when the first SMP Erlang prototype was written, but the space and power budgets for the processor boards made it impossible to get any matching hardware for it. SMP in those days was not multicore, but several CPUs. It was only usable in servers - not in embedded systems.

I remember reading "Erlang - A survey of the language and its industrial applications" (Joe Armstrong - in INAP'96)

Now I've checked again, so let me summarize:

It started to spread "outside the lab" in 1987. In 1995 there were three commercial applications: network simulator, mobility server and distributed resource controller. First is a simulator, not running on the specialized hardware, the third was running on "a network of Windows NT machines." The second: "acts as a call control system when linked to the PBX"

OK, the second was running on the additional SPARC-based machine, not big iron, but AFAIK certainly more powerful than the PC's of that time.

So I don't see anything that was targeting really "specialized hardware" particularly not what we'd think as embedded (less power consumption, slower processors than the PC CPU's). It was a research project, and it had traditionally performance problems.

Even today, after so many years, when you compare Erlang HiPe (Hi Perf, you can imagine others then!) with something higher than C, let's then say Lua JIT it's easy to see which one is faster:

http://shootout.alioth.debian.org/u32/benchmark.php?test=all...

Edit: Or for multi core 64 bits with Mono:

http://shootout.alioth.debian.org/u64q/benchmark.php?test=al...

Do you realize that rvirding is Robert Virding, one of the creators of Erlang?

I worked on a spreadsheet compiler at hypernumbers.com and some backend stuff for vidiowiki.com (video transcoding, file uploading/streaming and some other smaller stuff).

Erlang excels at distribution. People tend to forget its raison d'être is fault-tolerance (not concurrency), so when you set off working on an Erlang application you presume that it will run on 2+ machines. That's why it's great for things like Facebook Chat and Riak. In addition to those, it's also good for writing control and supervision layers, where workers are written in Python, Java, C etc.

While I'm at it, I'd like to plug http://veldstra.org/whyerlang/ It's in a real need of an update, and I'd love to hear any feedback.

Banking systems, both retail and commercial. Logistics. Embedded. Clusterware. Telecoms. Robotics, AI. When it comes to messaging, don't forget RabbitMQ and all the SMS systems.

Francesco

For those of you who don't now, Francesco Cesarini is the founder and CTO of Erlang Training and Consulting, and has been a major proponent of Erlang for years, also having co-written the book "Erlang Programming" (http://oreilly.com/catalog/9780596518189 - it's a great book, I highly recommend it myself)

Simple stuff, but:

- An IRC bot -- great fit for the language

- A URL shortener service (three times) -- first on Yaws, then on Nitrogen, then on ChicagoBoss (Hi EvanMiller!)

Erlang is the most fun I've ever had writing code. While a lot of that is the functional language aspect, I find it an extremely intuitive language at least at a low level (once you get past the 'there are 4 ways to end a line?!?!' stage). I'm still getting used to thinking through bigger problems in an Erlang way, but thats more a function of my brain than the language.

I think Erlang gets a lot of unfair knocks given it's pigeonholed reputation for 'concurrency and fault tolerance'. Certainly it has it's limitations (libraries), but generally, I think its a pretty well-rounded general purpose language.

A Facebook application that had nearly 2 million users.

A game data aggregation platform and feed system for EA.

A lot of open source Erlang stuff (http://github.com/ngerakines)

I have built the backend (game logic and comet) for http://www.iluminatus.ro, a trivia-like multiplayer game for the Romanian market. We're going to release an english version soon.

It was a great fit. It lacks Unicode processing capabilities by default, but there are a few open source libraries for that. The only thing I dislike about Erlang is the poor string manipulation and lack of clean structures (Erlang's records are an ugly hack IMO).

http://hypernumbers.com - an online spreadsheet which is a native web app - lurching towards a proper launch after a lot of development.

Erlang is a great fit - works well, allows you to build a native stack without any cross-language impedance. Your application is cluster ready from the get-go and allows you to focus on the functionality rather than any of the utilities required to make the damn thing stay up.

I built a Twitter client, with j2me on handsets and Erlang server at the back to handle all communications with twitter and few other services for image hosting, etc. It was my second project in Erlang and was very rewarding experience. Part I especially liked, is that I could deploy updates to the server while it was running, with no desruption to clients at all. http://pavo.me/

Phonebooth, a hosted telephone system http://www.phonebooth.com

Great fit. OTP process restart characteristics. Per-tenant process separation. Process abstraction, in general, rather than threads. Finite state machine behaviors. Live code upgrades. Mnesia database.

Only real downside was it took the rest of the team a bit of time to become effective in Erlang... about 4 to 6 months.

Not me but deserves attention: http://www.wings3d.com/

"Wings 3D is a subdivision modeller with an user interface that is easy to use for both beginners and advanced users"

I wrote a distributed sat-solver in Erlang (good old fashioned search problem). It was my intro project to get to know the language. Has been great fun and now Erlang is in charge of computers talking to each other while Java takes care of the heavy lifting. Also a similar problem solved with Erlang (nothing to do with me), and featured here on hacker news...

http://blog.ezyang.com/2010/08/tour-of-preach-distributed-er...

Great stuff. Erlang may make you smile (again).

We built a system to connect teams via texting and conferencing. http://www.connectteams.com. Erlang provides the web front end, the backend telecommuncations, and the reporting interfaces. All perfect fits.

Super late (a misjudged apache-bench got me banned for a day or something), but: http://darcsden.com/alex/aliter

It's a reverse engineering of a popular MMORPG called Ragnarok Online (or RO). Last time I was working on it, you could log in with the official client, walk around, teleport, chat, etc. (The walking had to be done in C, though; one pathfind per walk request is not something I'd want to leave to Erlang.)

Erlang is perfect for any messaging systems. And on other hand Erlang works well for GIS:

Vector Map Server - a BSc thesis project. Vector geospatial data streaming server, very similar thing to server's streaming vector data to Google Earth or Map24. Also works as WFS server. Has to handle a lot of data, but since data is mostly immutable it works well.

Backend service for not yet released liveblogging site. Done in collaboration with few other developers. Massive pubsub system. Done tests with 1M concurrent users on 200 small-node EC2 cluster with end-to-end latency mostly < 1.5s.

Texspace - spatial search server based on Microsoft Research paper. Fuzzy spatial search capable of working with completely unstructured queries. Had pretty big issues with processing, indexing and analyzing big amounts of data due to slowness of all DB solutions available at that time. Writing geometry algorithms in Erlang is not trivial due to immutability, since all of them are described in imperative languages.

IM subsystem for microblogging site (which is pretty big in Asia) - implemented 5 proprietary protocols. Erlang is perfect for such tasks! The only issue was lack of documentation for proprietary protocols ;)

Map routing server - implemented optimized A* search with route graph generalization. Writing fast A* is tricky in immutable world. Books like Okasaki's "Pure Functional Data Structures" FTW!

WMS Tiling / TMS service with caching. Simple and easy task for Erlang. Mnesia + file-level cache.

Overseer - Nitrogen + CouchDB app for tracking employees time by doing screenshots via VNC and marking them as "work" / "not work". A breeze :)

Erlang FTW!

SMS/MMS gateway (aggregator) with SMPP, CIMD and MM7 support and AMQP interface (RabbitMQ as a broker). Also building a simple SMSC right now.

It's a perfect fit.

I'm currently building a web-based IRC client with bouncer-like functionality - ie, stays online when you close the browser, remembers backlog etc.

Built github.com/RJ/playdar-core in erlang, after initially writing it in C++/boost

Not mine, but there is a CMS built in Erlang: http://zotonic.com

- we wrote the backend of http://bidit.eu in Erlang (front end in Rails) - we're writing the backend for a TV co-viewing chat application for the iPad in Erlang right now - we're rolling out a new Nitrogen-powered website for Inaka Networks this weekend

I built one of the fist comprehensive call detail record collection systems for a SIP voip network. I also built a least cost routing system for same.

A options feed aggregation and charting backend.

A package management system.

many many other small apps.

Cheers, Martin Logan erlangcamp.com

Not mine, and though I can't find a page on their site that states it, Vendetta Online[1] uses Erlang heavily for their server code. I went to a conference they spoke at a while back, they've said it works wonderfully for them and lets them handle large amounts of players easily.

They are also working to release a game engine (NAOS[2]) built from their server code, which also uses Erlang pretty heavily

[1]: http://www.vendetta-online.com/ [2]: http://www.guildsoftware.com/products.html

I did several small Erlang projects and now working on two relatively large.

Erlang is a good fit for:

* Irregular concurrency: Task-level, Fine-grained parallelism

* Network servers

* Distributed systems

* Middleware: Parallel databases, Message Queue servers

* Soft Realtime / Embedded applications

* Monitoring, control and testing tools

Not so good fit for:

* Concurrency more appropriate to synchronized parallel execution: Data Parallelism

* Floating-point intensive code (HPC)

* Text Processing / Unicode (Unicode support now much better)

* Traditional GUI (some improvement with wx now)

* Hard Realtime applications

* Extensive interop with other languages/VMs (some improvement here with NIFs and Erjang - Erlang on JVM)

* No Windows 64-bit VM

I using Erlang for the things listed under "not so fit", because it's advantages out-weight some incomvinience.

I built a real-time news widget that streams (hyper)local news and public records (marriage licenses, liquor permits, etc). Media companies partner with us http://nozzlmedia.com we've doubled the time visitors spend on their Web pages.

Erlang was a good fit for us, mostly because of the tools available around it, such as Nitrogen http://nitrogenproject.com a powerful JavaScript/Erlang framework. Erlang apps seem robust and lean.

Started etap: http://github.com/ngerakines/etap which ngerakines took over.

Iterate: http://github.com/zaphar/iterate an agile user story tracking app. Which is mostly useable but still in development. Used nitrogen which I found to be a really good web framework for dynamic applications.

Mostly toy projects, many of them still in development.

A little homemade web framework. A blog compiler to generate a static site. A one-on-one chat client to be integrated with other already existing websites, a browser-based version of the Apples to Apples game, a brainfuck interpreter, maze solver, etc.

We have an erlang cluster that runs all our services @ http://thecadmus.com

High number of processes distributed across multiple nodes with supervisor trees taking care of them all has worked well for us.

- a key-value stored modeled caching system that can run functions during set/get, have expiry, counters, counters that wait for N as a buffer before running transactions, and machine learning - all as primitives. the tail-recursive processes acting as mini-servers holding state and the best thing is that you have a mini-automatic scaling if you combine this well with expiring of processes. increase load when there's a need. they die when there's no more need/lesser traffic.

- a distributer crawler that works with python. from a typical rdbms approach that brought down creating inverted index's from more than an hour to few seconds. the euroka moment was when we stopped porting/writing for another language, sequentially, for-loop style approach to a more - processes and message passing. big 'why didnt i do this moment' a few years ago!

- a website , dashboard for the ad/content network using yaws on the LYME stack

- and btw - all that rumour about unicode is just bs. actually it was a blessing for us that content aka strings are just integers which when converted to binaries not only takes less space, pattern matches like a dream but all of a sudden your sytem can handle any language because to it - it's all just integers, english or swahili. that was a huge bonus even when getting investors, new untapped market opp's.

- an ad network, ( content network i guess without saying)

- a fire-and-froget thumbnail/screenshot system with queues & talking to imagemagick . i love writing and fire-and-forget tools, and with erlang, its so darn easy to write proceses's that have a life of their own. sure at some point you need to compromise on erlang's unique ability to handle RAM vs giving that up to research on how to handle large mnesia tables ( mnesia = inbuild db that has a limit on size of tables) . but it's worth the tradeoff.

case in point - our idea of scaling was...'reducing nodes' ie increasing efficiency

-2008 a million hovers in the year, 4 nodes

-2009 a million hovers every month, 3 nodes

-2010 a million hovers every week, 2 nodes i could bring it down to 1 node for doing the tasks as well, on just one 1 8gb RAM node.

the idea of reducing your costs, means that much lesser to profitability. ( we're now profitable, pretty much on auto-pilot on the backend, the 3-4 member team concentrating on increasing toplines,sales now )

in addition... - a wrapper to iui for nitrogen framework for building iphone webapps easily - wrappers to tokyocabinet ive contributed to called medici - remember writing wrappers to other things like rrd, aws utils, etc

all of the above have been used at hover.in, more at http://slideshare.net/bosky101

there's nothing i can think of that i wont/cant do in erlang anymore.

~B @hoverin

Let me guess, all this over engineering nonsense set your startup back 3 years.

OpenPoker