It's not just WebDAV that's an abomination (returning HTTP status codes and messages in XML, WTF were they thinking), but vCard and iCal as well.
The vCard "group construct" (see rfc6350) is one of the dumbest things ever added to a spec. It seems a trivial thing to add, but completely screws up your internal storage and manipulation formats. It's horrible, on top of all the other horribleness that is vCard.
Of course, I'm biased, given we're trying to push alternatives to IMAP, CardDAV and CalDAV - http://jmap.io/ - but please do check it out. Having actually worked on IMAP servers and clients, CalDAV servers and clients and CardDAV servers and clients, we've learned a lot about creating a saner alternative.
>returning HTTP status codes and messages in XML, WTF were they thinking
You're giving me so many ideas right now for a follow-up post.
>but vCard and iCal as well
The group construct is bad, the syntax is bad, but other than that, I never had to implement or parse it properly. Vdirsyncer is mostly passing strings through.
I suspect that this part is much harder to replace than CardDAV and CalDAV, since it's not just a file structure with a ton of bullcrap on top of it. For that reason I currently have those file formats in the remoteStorage folder structure i'm syncing to, but it's still a step forward. I might switch to jCard and jCal, but I don't think it'd be worth the breakage.
So one of your guys just pinged me in private about this, and I responded lengthily. Here's the basic summary:
* I didn't know JMAP was also about calendar and contacts.
* Those parts seem nice.
* However, I'm the kinda guy who self-hosts his data servers. JMAP seems like a powerful protocol. I'm not sure those two things mix well, since JMAP server implementations probably will require a good database, not just a dumb FS like in remoteStorage.
* However, I would like to collaborate wherever possible.
> JMAP server implementations probably will require a good database, not just a dumb FS like in remoteStorage.
To do JMAP well you need to be able to calculate change sets, which does mean a database, though a fairly light one. I think you could do a server without delta updates by always returning a cannotCalculateChanges error in response to getUpdates calls, but it would be very inefficient on both the client and the wire.
> However, I would like to collaborate wherever possible.
Sure! Best place to start is probably the jmap-discuss list:
> For anybody who wants to be "careful" when syncing and thinks they want this option, consider putting your collections into git repos and commit after sync.
There's a demo/beta DAV<->JMAP proxy out there by fastmail. You might want to look into that (and keep the current DAV backend in the meantime).
I didn't notice that JMAP includes calendars and contacts. Looks good!
Personally, I'd prefer a dumb storage with a generic API over yet another custom protocol/API just for these data formats alone. Worked great on people's local drives before the Web, and I think it should work similar on the Web.
Dumb storage is fine if you only have a single actor that can maintain a copy of the state at all times. As soon as you've got multiple actors on the data (another client on your desktop or phone, or even something doing mail delivery), then you need a way to determine what changed on the server so you can update your state.
If the server has no ability to tell you what changed, then you're left with having to download and check everything. On a large data set, that's pretty much impossible to do quickly and without a lot of network traffic.
Of course, no server is actually that dumb - even a file listing with file sizes can get you part of the way there. But if you've got 10000 files in a file store, that list can still get pretty heavy. If you're willing to make the server smarter, eventually you can get to the point where the server can give you only what changed since the last time you checked. JMAP isn't unique in this; IMAP has MODSEQs, *DAV has collection synchronisation, etc.
JMAP specifically doesn't really care much about the actual format of the data it works with. The only thing it really needs is an immutable ID, so you could use the same model to store all sorts of things (and at FastMail we do, with things like client settings).
> Of course, no server is actually that dumb - even a file listing with file sizes can get you part of the way there. But if you've got 10000 files in a file store, that list can still get pretty heavy.
Exactly. remoteStorage has ETAGs in folder listings for that. The point is that you can implement a folder structure that enables you to update say just the last week of events plus upcoming ones, which is usually nowhere near 10000. Except with CalDAV you can't (according to the article, I haven't actually looked into it myself).
I guess you can dump a [Maildir](http://cr.yp.to/proto/maildir.html) (or at least a modified version) into a remoteStorage, but this is not really an acceptable protocol for mobile clients with limited space. As mentioned in the blogpost I have the same problems with storing calendars in remoteStorage.
Hey robmueller, I think it bears moving this paragraph on proxy.jmap.io
The service is not running with all of the security measures employed on a production site such as FastMail, so DO NOT USE THIS PROXY FOR ACCOUNTS WITH SENSITIVE DATA.
above the "Oauth to a gmail account or log in to an IMAP server below".
Of course it stands to reason that one shouldn't grant access to third parties to one's gmail, but you may save some heartache from people who are enthusiastic-past-the-point-of-reason if you move that bit up.
I would have just sent you a PR but proxy.jmap.io isn't hosted on github pages.
My pet peeve with vCard is the lack of an encoding and the resulting mess with various implementations. JMAP fixes this by enforcing utf8. You have my vote. :)
To be fair the newer versions of vcard also specify this, and I haven't had a single encoding issue with even the older versions. Every server just seems to send UTF-8, no questions asked.
So far all my phones had a different way to encode Unicode strings. When I wanted to merge the data, I tried a bunch of Python libraries and all failed differently. In the end I wrote my own one-off script.
There's actually not that much to write about. It's a regular company with share holders, the share holders just happen to be some of the staff.
Fortunately the owners all get on very well, have similar beliefs in which technical, financial and marketing directions we want to head in, and love what we do. The result is that everything just runs pretty smoothly, so far at least :)
I think I mentioned in the interview though, the difference in vibe between being owned by Opera and being owned by the staff isn't as big as you'd think. Much more important is just actually loving what you do, the things you create, and the people you work with. All of those are absolutely great, both as they were under Opera and now back under staff ownership IMHO. I wonder if other FastMail staff will have their own thoughts :)
> I wonder if other FastMail staff will have their own thoughts :)
Ok, thanks for the nice soapbox, boss :)
I've been at FastMail for ~2.5 years. The first 1.5 was as Opera. I haven't noticed a substantial difference between the two owners. Under Opera there was a very small amount of bureaucratic tension in a couple of places but that's to be expected given the amount of freedom we had. There's some decisions we've been able to make more quickly now we're on our own, but I don't think there's anything in particular we would have done differently had we not left Opera. I do miss the occasional trips to Norway though...
What I love the most about FastMail is the trust. Competent technical people can recognise other competent technical people, and though some days I feel like bit of a fraud because everyone at the company is crazy smart, I know I'm good enough to pass most of the time. They obviously agreed, because on my first day I was given complete, unrestricted access to everything - keys to the kingdom. Along with that I got told "do cool stuff and don't fk it up".
It probably took me 18 months to really understand that they actually meant that. If I have an idea I can run with it and don't need to ask for permission. If its complicated and I don't understand something, I'm expected to get help. If I screw it up, then I'm expected to get up at 3am and fix it, and deal with the support calls and the PR fallout, and the general embarrassment around the office.
The flip side is that I have to take complete responsibility for my actions. If we agree to as team to do something, we're all responsible for driving it, and we're all expected to hold each other to account. If I promise to deliver something, then it has to be delivered. No one will get fired for a mistake - fear doesn't exist in our dojo - but if you screw up someone is going to pull you aside and ask what happened and how we can avoid it next time. Though that's kind of cool too because you learn pretty quick that if you screw up its easier to just write an email to the team telling them so and explaining what needs to change to stop it happening again.
All of this only works if everyone is really passionate about the service and is happy to get their hands into whatever thing is important that day and take responsibility for everything. We've had staff in the past that were good at what they did, but didn't deeply care about the service. So they had a tendency to not completely polish everything, or overengineer, or not step outside of their area of interest, and it didn't work as well.
A possible downside of all this is that none of us seem to have a strong distinction between "work" and "home". We work wherever there's network, from all over the world, at all hours. Its pretty normal for me to get home (assuming I didn't arbitrarily decide to work from home or a coffee shop or whatever that day), spend a couple of hours with my family, and then get straight back into it. But that's only because most days I sit back down at my desk and can't think of anything else I'd rather be doing!
I'm not actually one of these owner-staff people, but "just" an employee, but really that only seems to be make any difference when we're talking about financial or legal stuff. On technical matters, we're all equal. I've pulled up one or more of the owners before on stupid technical decisions. I've walked in and said "you've got too much on your plate, I'm taking this and this". I will defer to them on stuff related to financials and legals because ultimately its their money and necks on the line, but that's fine because my opinion is always heard and valued.
Yeah, this sounds like an impossible magical fairy land. I still don't quite believe it some days, but it really is this great. That's the culture RobM and others have built over the life of the company. I don't know how deliberate it was, but I'm going to work hard to make sure it stays that way because every day I'm excited to get up and go to work, and a little disappointed to have to leave and go home and/or go to bed.
Have you tried FastMail.fm? It's not free, but has a fast web UI, excellent IMAP support, customer service, etc. Search other HN threads to see what others think of it. (disclaimer: I co-own it with some of the other staff)
3. Each SSE connection is regarded as a regular site connection. So if people open up 6 tabs to your site, you might suddenly find your site is unresponsive to users because every available connection is used by an SSE connection. You might need to implement some inter-tab communication system instead.
Yep. But that's why we've unified the family plan levels to be the same as the personal ones.
So the only difference now is it's $5/y extra to have the family "container", which lets you create accounts in your own domain rather than having an @fastmail.fm account and aliases. Having that container then also lets you setup multiple accounts, manage them, share folders between users, etc. If you only have 1 account, then yes, it's a bit annoying, but I don't think $5/y is too onerous.
I really wish companies would get SSL vs TLS vs STARTTLS right. I wrote about this on our blog a while back and ended up turning it into a help page because people still seem to get confused.
> The Spamhaus people are bad guys. I gradually realized that during the time I worked on spam filters. They presumably started out with good intentions, but the position they're in has corrupted them.
That's a very attacking statement to call people "bad guys". Not just "they're not doing a good job", but actually "bad people". Ouch.
Worse, I think it's totally wrong.
While I do agree with the problem of "power corrupts", I believe that Spamhaus have been highly successful at avoiding that.
I've dealt with and spoken with people at Spamhaus regularly, and they're smart people fighting the good fight. They know what they're doing, and take their responsibilities seriously. They don't blindly attack people or use threats or accuse them of misdeads. They gather evidence about bad behaviour and act on it.
The proof is in the pudding. 100's of RBLs have come and gone over the years, either run as temporary projects that the owner gave up on one day, or gone the "power corrupts" option and just ended up listing so many IPs they've generated too many false positives (e.g. SPEWS).
Spamhaus RBLs are still being run today, and being done so very successfully. Virtually every small/medium email server I know uses them in one form or another (blocking or scoring). They generally have very high block rates, and very low false positive rates.
From a time when Al Iverson was still keeping stats on the various RBLs out there, you can see the Spamhaus zen RBL on it's own generally caught 75% of spam with 0% false positives.
Certainly some listings are controversial (eg google docs), but it's always been for a good reason, and forced the provider of the service to come to terms with the fact their spam policies were lax or their service was being seriously abused by spammers. They were thus forced to take action, something they should have been doing anyway.
Without Spamhaus, the internet would be a way worse place, with way more spam/junk emails and websites.
Spamhaus is not as bad as some other blacklists have been, certainly. Nor do I think they started out as bad people, though you have to admit that running a blacklist might tend to attract a certain type of person. Nor do I think they're stupid, or that their lists are ineffective (at the times when they're not deliberately blacklisting innocent people). But like others who've gone into the blacklist business, they do seem to have lost their way morally. I wrote about an example here: http://paulgraham.com/spamhausblacklist.html
So you're still saying that maybe they didn't start that way, but right now they are bad people, who have lost their way morally.
I think that's an incredibly harsh accusation for people that are doing an awful lot of work collecting evidence and fighting real spammers on the internet (http://www.spamhaus.org/rokso/index.lasso), and again, I totally disagree with you.
> though you have to admit that running a blacklist might tend to attract a certain type of person
I think you could tar so many people in so many industries with broad brush stroke sterotypes like that, it seems an unhelpful generalisation to make.
From the article you link:
> As of this writing, any filter relying on the SBL is now marking email with the url "paulgraham.com" as spam.
The SBL is an IP based RBL, nothing to do with domains, so the above statement is patently false. And if anyone was doing IP lookups of URI's in emails and using the SBL for that (which I've never even heard of), that's clearly a misuse of the SBL anyway, because that's not what the SBL is supposed to be used for.
The Spamhaus Block List ("SBL") Advisory is a database of IP addresses which do not meet Spamhaus's policy for acceptance of inbound email and therefore from which Spamhaus does not recommend the acceptance of electronic mail.
---
So it should only be used to block machines sending email, nothing about the content thereof.
There's RHSBLs (like SURBL and URIBL) that are related to dealing with URI's in emails, that's nothing to do with IP RBLs like SBL.
> Why? Because the guys at the SBL want to pressure Yahoo, where paulgraham.com is hosted, to delete the site of a company they believe is spamming
What's that got to do with the SBL again? The SBL is purely about what IP addresses "from which Spamhaus does not recommend the acceptance of electronic mail", nothing about websites. So that whole accusation feels wrong. Mixing up email sending servers and websites, domains and IPs, and absolutely no evidence for it at all.
It was the SBL. I don't remember the details of how my emails were getting blocked. But they were obviously getting blocked, because if they hadn't been, I'd never have found out about the problem. At the time the Spamhaus guys themselves didn't deny that they'd blacklisted large numbers of innocent Yahoo Store users. Their defense was that the end justified the means.
You seem naive about the nature of evil if you think that it somehow precludes doing constructive work. Bad people don't wake up every morning thinking "what evil shall I do today?" What distinguishes them is that they cross lines other people won't. But the situations that test them may come up fairly infrequently.
"At the time the Spamhaus guys themselves didn't deny that they'd blacklisted large numbers of innocent Yahoo Store users. Their defense was that the end justified the means."
Putting this into context, this wouldn't have been the first step. This type of measure was typically implemented after it becomes increasingly clear that Yahoo would not, or could not, adopt measures to reduce the amount of spam coming from their mail servers.
One of Yahoo's general weaknesses is that it takes over 24 hours from sending a complaint until appropriate action is taken (that's why comments on their main sites - e.g. News - contains oodles of spam, and other types of abusive comments). On the typical life-cycle of email spam this is far too long - if a site is ever closed at that point, and so these abusive sites tend to still be up when the email recipient is clicking on those links. Closing a site after the damage has been done is just a never-ending game of whack-a-mole.
Blackholing bigger and bigger chunks of Yahoo Stores is then an escalating step until either Yahoo addresses the spam situation appropriately, or their customers see that Yahoo cannot sustainably provide the service customers are paying for and they either leave or seek legal remedies. At that point innocent customers are paying the price for living in a bad neighbourhood. The question is, why didn't Yahoo do a better job in controlling the level of abuse through Yahoo stores? That Spamhaus felt it necessary to escalate through to blocking chunks of ip addresses indicates Yahoo Stores fell significantly short of what was needed to reduce the spam coming from their servers. The indicative belief from the anti-spam community at that point is: it's mostly clear that the revenue generated from hosting spammers is more important to Yahoo Stores than being able to provide their innocent customers with the level of service they paid for.
From my perspective, SpamHaus were one of the cleaner, more diplomatic black lists around the time of the Yahoo Stores problem. It's been a few years since I last poked around in the anti-spam community. Last I've seen of Spamhaus they didn't defend a legal challenge in California raised by a confirmed spammer, because California doesn't have jurisdiction over UK-located organisations, and so the spammer got a default ruling in his favour ( http://www.theregister.co.uk/2007/03/23/e360insight_lawsuit/ , http://www.spamhaus.org/organization/statement.lasso?ref=3 ).
I'm one of the main developers and was one of the (now previous) owners of Fastmail.
Fastmail has always been a small company, there are just 3-4 main developers (myself, Bron, Richard and recently in the last year, Kurian), and a couple of support staff scattered around world. For that small size, I think we've managed to build a pretty great product with lots of niche and power features, loyal users, and apart from a small disaster in 2006 (2-3 day outage for a big chunk of users), we've also been incredibly reliable, especially in recent years.
Fastmail was nicely profitable, but not spectacularly so. We're basically all geeks, and we don't have a marketing or sales department that can grow our customer base significantly (we tried, but it didn't work out, and we probably should have put more effort in, but didn't... because we probably preferred to spend time just building neat stuff, or fixing that edge case bug, or doing that fun thing... like I said, geeks).
I think we had to face facts a bit, we were a small fish with limited resources in a market that has become severely more competitive in recent years. We needed to invest a bunch of time and money in updating our interface, and adding new features (especially better mobile syncing).
And coincidentally, it's around that time that Opera came along and started talking to us. Despite being half a world apart, there's a lot of fit between the companies. They use a lot of perl, we do to. They're a company run by technology people, creating a product that's loved by geeks, is highly customisable, has a loyal fan base, and despite it's small size, punches above it's weight. I think that describes us pretty much as well.
So the timing was right, and Opera have an interest in picking up email as a core competency, and a bunch of ideas on what they want to improve, what they want to build. The other Fastmail guys were also interested in new opportunities, and we're all becoming Opera staff and are committed to working there for a few years at least. There's already plans for some staff to move to Norway to work, a change of life after 5 years of just 3 of us in a single office (apparently the Norwegian lessons are paying off... Jeg vil gjerne et øl til)
So it'll be an interesting change, and something new I'm looking forward to. I've been working for Fastmail for 10 years now. It's been a great time. I've loved building the product and the company. Like anything, there's been ups (it's fun developing a site that customers really love and tell you about) and downs (some people are addicted to being able to access their email, and running a 24/7 email service means that if people can't get to their email for even just 1 minute, you'll start hearing about it). After 10 years, it'll be strange having a boss again. I've met a bunch of the Opera people, and it'll be really great working with them. I know the other staff are looking forward to it as well.
It'll also be great to have Neil on board as well. He worked for us over a couple of summers, and basically designed the entire "new" web interface, all the HTML, CSS and JS. We've already got 80% of a whole new AJAX interface done (remember in programming though, the first 80% takes 80% of the time, the remaining 20% takes the other 80% of the time), and I'm looking forward to completely finishing that off, and working on a bunch of new stuff.
Hmmm, this story went on longer than I expected. Hope it's interesting to someone...
I too hope you've done well out of the deal. I've been a paying Fastmail customer for ~7 years and am very happy with it. The 2006 downtime and a few other hiccups around the same time were annoying, but since then things have felt rock solid.
I find the spam filter better than the one in Gmail, so thanks to whoever's done the hard work with that.
And thank you for keeping the old web interface around, I prefer it over the newer one. There's nothing particularly wrong with the newer one, I'm just very used to the way the old one works and find it fast enough without AJAX enhancements. That you kept the old interface going is one of the reasons I stayed with Fastmail, if you'd forced a switch to the new one I'd have probably gone to Gmail.
Congratulations is in order. I've been using Fastmail since at least 2001 and have been a paying user for the last eight years. I hope it continues to be the quality service it has been going forward.
The vCard "group construct" (see rfc6350) is one of the dumbest things ever added to a spec. It seems a trivial thing to add, but completely screws up your internal storage and manipulation formats. It's horrible, on top of all the other horribleness that is vCard.
Of course, I'm biased, given we're trying to push alternatives to IMAP, CardDAV and CalDAV - http://jmap.io/ - but please do check it out. Having actually worked on IMAP servers and clients, CalDAV servers and clients and CardDAV servers and clients, we've learned a lot about creating a saner alternative.