49 comments in 5 hours and almost all of them talk about the technical aspects or about working at Google.
This is about more than that, it's about decentralization as a way to empower people so that in the end we don't need centralized companies or governments to control our data.
Apple (and other companies) control what you're allowed to download. Google (and other companies) control what emails get through to you and your email history. Microsoft (and other companies) control your hardware.
He forgot to mention the larger and more disturbing point; many of these companies are American and so they're under the jurisdiction of the NSA and FBI (and CIA if you're not from around there). With centralization, law enforcement has easy and direct access to things. The only barrier is a warrant and even that isn't a barrier as we saw in the AT&T NSA wiretapping case.
He wouldn't want to work at Google or many other companies because they're pushing for centralization which brings certain political/social effects that he dislikes.
So can we please have a discussion about the political and social implications of decentralization vs centralizaton rather than the technical aspects??
I imagine a different future, where if Alice wants to talk to Bob and
Bob wants to talk to Alice, there’s no unaccountable intermediary that
can interfere with their communication, whether they’re speaking text,
or video, or 3-D models, or simulation.
I never thought I'd live in a society where the value of privacy or anonymity was up for debate. These are real world concepts that I thought everyone took for granted in their routine communications, but are being tossed aside on the Internet merely because finding a technical solution seems too hard (and it is very, very hard).
Maybe we should reframe the debate around the concept of intimacy instead. There are a lot of activities that are perfectly innocent, but I still wouldn't want to do them in a crowded room or in my boss's office. The Internet is becoming the primary medium of electronic communication. I should be able to use it to have an intimate discussion without it being data mined by anyone. There's no reason any third party should be able to detect the conversation is even taking place, and anonymity should be at my discretion, subject to the acceptance of the party I'm communicating with.
I can appreciate that thought, but I think you are overlooking that we are moving from the primary medium of the telephone for these types of communications. Telephones certainly weren't anonymous, or undetectable. In fact when they first started they weren't even private but quasi public (to the operator and others on the party line).
I'm not arguing with the ideal, just pointing out that we didn't transition straight from face to face, anonymous and undetectable to the internet. We passed through the telephone phase, which last many decades, and arguably had less tools available to do what you wanted. Internet communication is an outgrowth of the telephone and its network, but arguably with some work is able to provide more of what you want.
Your home phone was the equivalent of a gmail address. A pay phone was closer to a home email server, or an anonymous internet cafe web browsing session. If its shocking that this debate isn't larger about web decentralization, it should be doubly shocking that it never seemed to occur at all, so far as I can tell, during the phone age.
But regardless, your ISP will always be present, and your packets will always traverse many dozens of pieces of networking equipment, owned by several dozen big faceless institutions, along the way. Services can be eliminated, the network cannot. True point to point communication in the manner the OP and yourself seem to want truly requires point to point communication. Maybe we need some innovation in the ham radio space.
I'm old enough to remember party lines, so your point is well taken. But even with the advantages of the telephone, it soon became obvious that people resented the constant eavesdropping of nosy neighbors. It's frustrating to have to fight that battle all over again, especially when someone like Eric Schmidt says, "If you have something that you don't want anyone to know, maybe you shouldn't be doing it in the first place." [1] I get it, the Internet is turning the world into a small town, but sometimes living in a small town sucks.
I agree, the discussion here has immediately taken a technical dimension for the same reasons that engineers are really poor at evaluating the real world value and far-reaching social implications of the technology they create.
This is not an article about the technical difficulties of running today's Internet or email. It is about the transition from revolution to institution. Centralisation limits the disruptive potential of the Internet and in the long term will transform it from an enabling technology into another obstacle that needs to be overcome.
This is just another step in a long series of advancements that have transformed human existence: organised Agriculture, Centralised Government, The Written Word, The Legal System, Mathematics, Science, Democracy, Free Markets, Finance, The Industrial Revolution, Publishing, Education, Modern Medicine, Personal Automobiles, and the Mass Media. In the idealistic minds of the pioneers, the potential was evident and intentions often subversive. Today these have invariably matured into terribly inefficient monstrous empires that are fiercely resistant to innovation or new ideas. With the resources to move the world, why should they move for anyone?
Personal Computing and the Internet are new and exciting, the gatekeepers remain engineers because they are the only ones that seem to be able to get it to do anything. Despite the convenience argument, building a centralised infrastructure will lead to the inevitable takeover by the same maniacs that run everything else. Of course, it may take a few generations before Google's superbly engineered infrastructure is operated by a Manufacturing/Pharmaceutical/Petroleum/Media/Retail/Finance/Defense conglomerate whose alumni occupy every significant position in Government. Why should we bother?
"After this, let's build the biggest, most bad-ass super robot weapon ever because it's cool and fun, we'll use a master password because it's more convenient and surely that bully that eats all our lunches might be watching intently, but he's too stupid to understand advanced control theory."
That being said, I would like to work at Google, the smart kids building the biggest, most bad-ass super robot weapon ever.
Here's the issue. No one cares. Well, I care, and presumably a number of other readers care, but compared to the total sum of internet users we're just a rounding error.
You know why people are using services like Gmail? Because it just works. Have you ever tried setting up your own mail server? I like to think that I'm pretty damn skilled with "the computer" but after a day of tweaking I'm still not sure mine is operating properly.
The sad fact is that being idealistic is not enough. You have to be idealistic and better than the bad guys. If you offer people a system that is hard to use, wastes their time, and/or is simply inferior to other options, no one will ever use it no matter how idealistically pure it is. Then you just sound like an asshole when you say "you're all morally inferior for refusing to degrade your experience."
So. Fix your system. Make it better than what we currently have. Then come back and convince me to care (hint: if your solution involves end-users installing and maintaining multiple servers, you're doing it wrong).
Me care too. But centralization, in the right hands, seems able to produce some genuine advances. I hate that the iPad is closed. But I really think its ease of use, and the ethic of usability it requires for "app" success, and its functionalities, make it a compelling step forward. I really wish, for many reasons, I could better articulate why.
It's very hard to see how a decentralized community could implement such a vision -- it _might_ arrive at a similar or better place through evolution. But look how long Linux has had to hit the desktop target.
I don't think centralization is the inevitable winner. The decentralized can and do follow these innovations, and the centralized seem to struggle to keep up with the pace of innovation. But lately we've seen some advantages for centralization that decentralization hasn't yet met.
You have nicely restated the point of the last four paragraphs of my post‡. However, your tone makes it sound like you think you're disagreeing with me. This leads me to believe my post was not clear to you, but I am unfortunately at a loss as to how I could have made it any clearer. Do you have any suggestions?
‡ actually, you added some points of your own as well. To be specific, I never said anything about moral inferiority, and your "hint" is idiotic, analogous to the guy who thinks most Americans will never own their own vehicle because it's too much work to take care of the horses properly.
Ironically, I think that is exactly what the OP was saying. He would rather be working on solving the distributed problems than solving the centralized problems. He recognizes central is working better today, but that doesn't mean it will be tomorrow.
To be fair, the author (appears to) work for Canonical who has had success working to make idealistic software more approachable. So it appears to me that he isn't some Ivory Tower FOSS advocate, and is merely stating a personal preference.
"kragen has worked for a variety of companies doing mostly cool things. In early 2006, he pulled up stakes and headed to South America, to begin exploring how he can empower people through technology. And to work on his Spanish."
It's heartening to see so much interest in my little post; it was quite a surprise to come home from a weekend traveling around to dance contact improv and discover hundreds of comments waiting for me.
It's disappointing that so many of the comments focus on one or another point about why things are the way they are: spam filtering is hard and benefits from secrecy from spammers, centralized software is currently more usable, etc. My post was about values, about what kind of a world we can be building, not about which tactics are expedient in the world we currently live in. People with the same values can get together to discuss what tactics to use to advance their goals, but it's no use in suggesting to me that I should use a tactic that advances goals I oppose because that tactic is more expedient!
I do understand that the mail post is about why he doesn't want to work at Google and not about demonizing Google. Yet, what still strikes me odd is that many people feel like Google and other big companies should act the way they want.
"Their “real names” policy on Google+ is one example; it makes it likely that only people who feel they have no repercussions to fear from anyone, ever, will write there."
And that is fine in my opinion. Not everybody needs to be on the Google+. It's their playground, let them run it the way they want.
It is not that simple. Big companies have an impact on society and culture, that cannot and should not be ignored.
Much like there are systems in place to protect the market from companies (or more accurately itself), it needs to be considered if that impact is negative, if such a negative impact is somehow encouraged (like monopolies) and if so what can be done about it.
Democracy works because we assume that communication can be done freely and anonymously. If these assumptions don't hold true for major communication platforms such as Facebook, Twitter or Google+ we do have a significant problem.
The fact that you don't have to join these platforms becomes irrelevant if the societal impact extends beyond them or not-joining prevents you from communicating with others in society.
> Democracy works because we assume that communication can be done freely and anonymously.
The real name policy on Google+ doesn't change anything about that. People can still communicate anonymously, just not via Google+. Democracy does not require anonymity on every communication platform imaginable. There also isn't anonymous internet access, and there aren't anonymous domain names (or only few).
Democracy depends on the State allowing anonymity, not on every communication provider implementing it. Google thinks that it's system profits from accountability, and they have the right to enforce it on their platform.
If it's a single provider doing it, without anything close to a dominant market share (the current situation), I agree. It would become more problematic if it became de facto the standard communication platform, though, or even a large share of it. For example, if Facebook messages replace email for 90% of the population, you will find it very difficult to avoid using Facebook messages.
I don't see it as much different than the State, really. Companies with smallish market share are like local governments: relatively harmless, because you can just avoid the ones you don't like. Bigger governments and companies get increasingly hard to avoid; it's easier to move out of Pittsburgh than to move out of the U.S.A., and it's easier to avoid Google Plus than to avoid the big-4 telecoms.
If a G+ or Facebook achieve 90% of communication volume, it means that anonymity isn't an important factor for most people and you could in fact argue that the high market share is due to the fact that there are less problems with spam and abuse because there is accountability. If a significant amount of people desire anonymous communication, there will be a communication platform for them.
Also you can avoid G+ and the big-4 telecoms much easier than you can avoid living in a state. The need to live is much greater that need to have internet access, no matter how you slice it.
You can avoid living in a particular state more easily than you can avoid internet access, I'd argue. In 2011, the hardship of going without internet is much higher than the hardship of moving from California to Nevada, for example.
I do agree that national-level laws are more coercive, in part because many countries are just very large, and in part because immigration laws make it hard to move.
"Democracy works because we assume that communication can be done freely and anonymously. If these assumptions don't hold true for major communication platforms such as Facebook, Twitter or Google+ we do have a significant problem."
Face to face communication has always been available freely and anonymously. I'm not quite certain that any other form of communication has ever been either freely available or remotely anonymous for anyone remotely like an average person.
What communications platform has it ever held true for?
1.) Telegraph -- cost money, not universally accessible, open, message assured to be read by at least 2 3rd parties.
2.) Mail -- costs money, need a registered address, handled by a quasi govt agency, can be read by 3rd party.
3.) Land line Telephone -- costs money, not universally accessible, requires registered address, every action is logged including recipient and duration. Calls can be listened to by 3rd party.
4.) Cell Phones -- costs money, not universally accessible, burner can be had with no address or name. Calls can be listened to by 3rd party. Location tracking available.
5.) Home Internet -- costs money, not universally accessible, requires registered address, everything routed through ISP who keeps user logs, network that can intercept.
It's unclear to me why the service layer attracts so much attention, when the underlying network layer currently makes it largely impossible to have what you want for the average person. Having anonymity at the service layer is almost confusing when the underlying framework guarantees it is largely not real. There are all sorts of options already available at the service layer that can give all sorts of false appearances, many fewer at the network layer to make those false appearances turn real. Twitter and Facebook and G+ are trivial issues in comparison to getting around the fact that backbone and access points are a network owned by 3rd party intermediaries (just as with all the previous communications platforms above).
While all the above points are true, there is a big difference between picking up the phone to call someone in 1950 and posting an update on a social networking site in 2011.
I would submit that there are two factors that make the difference: 1) laws regulating the service in question and 2) ease of data aggregation / mining.
In both cases there are barriers to interception & analysis of communication that effectively provide or used to provide a certain level of expected privacy and anonymity. Of course no communication is 100% secure and of course anything can be intercepted, but how hard is that to do?
In 1950 if you wanted to get pictures of someone's house, learn the names of their known associates, find out what books they own, what magazines they read, what opinions they have, their work history, their goals, their aspirations, which of their relatives they are closest to, etc. etc. -- your work would have been cut out for you. It would involve physical trips to places like courthouses, time and effort in taking pictures, developing film, driving around, possibly snooping where you shouldn't, asking lots of people lots of questions, and so on.
Now all that is out there with a few mouse clicks and not only is that the case, it is rapidly becoming the social norm that it should be out there and if it isn't, as Mr. Schmidt says, then you must have something to hide.
The genie of cheap, accessible, centralized data storage and data mining is out of the bottle. At least with email there is some sort of legal precedent to treat communications as privileged. With social networks and the like, users give up their rights to the commercial entity that provides the service. And as others have pointed out, rolling your own services is not an attractive alternative.
Agreed, the network layer can be intercepted, but the network layer either has to be hacked into or somebody in law enforcement has to leave a data trail regarding the interception. At the service layer, it is becoming assumed that you give up all rights to your data, and this is something whose repercussions have yet to be fully felt.
I think this misses the wider point. Google are basically saying: "we can't guarantee your anonymity so we won't pretend we can".
By enforcing a real name policy they are sending a message to those that need real anonymity: use a system designed from the ground up to give you privacy.
Anonymity and pseudonymity are two orthogonal concepts. Google could easily offer pseudonymity without making any sort of offer of anonymity. Arguing against pseudonyms with "We can't guarantee your anonymity" is nothing but a straw man.
I don't think the everyman views them as separate concepts.
People honestly believe that if you're using a pen name, then you're anonymous to those who you haven't told your real name to. Note even the terminology used here "Real Name", as in anything that isn't your government ID name is not actually a name. Pseudonyms in formal language are 'names', but not so in informal language.
"You are not required to use your Real Name to use the Service provided by Google. However Google makes no assurance that your true identity cannot or will not be disclosed even if you make use of a Pseudonym to use the Service."
... or words to that effect in the TOU would address the matter just fine for most people. For those who truly require military/intel grade anonymity, there are alternatives. For those who just want to engage with the general public without revealing their real name to all and sundry (absent hacking or subpoenas), the above should be sufficient.
If Google wish to go the extra mile and provide assurable anonymity, that's another matter.
Otherwise, this is a case of the perfect being the enemy of the good.
I guess an issue for many hackers is that whether or not to work for Google is a real, nagging question. Many of us will have the skill-sets that cause Google to seek us out when they are on hiring sprees. Some will certainly get offers.
For hackers, Google is clearly an interesting company to work for. They have good infrastructure, good technology, a culture that is likeable to hackers, competitive pay, and an ability to make world-changing technology.
So, all in all, it's an enticing offer.
However, for many hackers, there's the question if Google's overall corporate direction fits our values. Combined with the fact that Google is so powerful, one wonders if Google is the ally or the threat.
I believe that the article make the point that if you have the abilities to land a job at Google, you might as well have the abilities to change things for the better without Google.
Exactly. Right now for good hackers the job market is such that they can pretty much choose where to work. And the market is even so good that most are able to do these decisions based on matching corporate values and not just salary.
Yeah, I expected this to be yet another post about how evil google really is, but instead I found a very well written and very insightful post with very interesting references. It might not apply to everyone, but it's a great read.
Let me explain. I think we have a very fundamental disagreement.
I believe that some actions are good and some actions are evil, according to, among other things, whether they harm others, whether they are fair, and whether they are honest. I believe that people should do things that are good and not do things that are evil, whether in their capacity as employees or otherwise; big companies are not moral agents, but their employees are. I can assure you that most of the people who work on Google+ believe that, too, which is one of the reasons they work at Google instead of somewhere else.
And I believe that the “real names” policy is evil, because it strengthens existing imbalances of power in the world, which is unfair and also harms people.
If I'm wrong, and it is not evil, then I do not want Google (by which I mean a large group of people, especially those with a lot of influence) to abandon it. But if I'm right, and it is evil, then I do want Google to abandon it. That is, I don't want Google to conform to my personal preferences; I want Google to do good and not evil, and I have an opinion about what that consists of, an opinion that might be wrong.
Does that clarify? What in the above "strikes you odd"?
I somewhat see where you are going and I do like it. Employees should work in a company where they believe in the way of working and feel comfortable doing the job. Agreed on that.
But I see the problem arising on how to define what's evil and what is not.
There are lot of things which can be thought as strengthening existing imbalances of power in the world. Even the Google search could be considered to be such.
I can't figure out why "real names" policy is evil. I don't see it harming people. From what I have been using Facebook and Google+, I actually enjoy and prefer it so that people use real names.
Not all products are suited for everybody and alternatives for different use cases exist. I'm sure that Google has lot to offer for people who don't want to use their real name publicly.
> Employees should work in a company where they believe in the way of working and feel comfortable doing the job. Agreed on that.
That's not what I said, although it's probably a good HR policy.
I didn't say people should do things that they think are good and not evil (at work or otherwise).
I said people should do things that are good and not evil.
If I thought what you think I thought, it would be counterproductive to bring up these issues publicly, because it might make people uncomfortable by making them question their values.
> But I see the problem arising on how to define what's evil and what is not.
Yes, of course it's difficult to know what's good and what's evil (it's not a question of "definition" unless moral relativism is correct), and if past experience is any guide, we probably won't come to any kind of permanent consensus. That doesn't mean that discussing it is pointless. We can probably come to better conclusions about what's good and what's evil with the discussion than without it.
> There are lot of things which can be thought as strengthening existing imbalances of power in the world. Even the Google search could be considered to be such.
Someone might think Google Search strengthens existing imbalances of power, but I don't think so. And of course it does the opposite of harming people.
> I can't figure out why "real names" policy is evil. I don't see it harming people.
I don't really want to get into the whole #nymwars discussion here, but suffice it to say that the broad consensus among Google+ users is that you're wrong about that — which is what I meant by saying that Google's policy was undemocratically chosen.
the problem in the real name policy, to me, is more related to the fact that they want your ID name while they claim to need the name you go by usually.
Would google accept _why as a name for a user? Probably not unless this _why is a well known guy. But what if you are not well known, but your friends call you that way? To me they should accept it based on their terms...
What concerns me about RN policies is the immoderacy with which they're often pursued, not the idea itself. It's actually (gasp!) a good idea at root. I have no problem with allowing pseudonyms and role accounts-- I think G+ and Facebook should-- but Google is right not to want profiles with names like Fuckhead89 and MiKeY ~:~ j0n3S on the service.
Giving less leniency (in terms of the decision whether to can spammers) to non-real-name accounts makes sense. Stopping impersonation is obviously the right thing to do. Preemptively treating accounts as negative-value accounts simply because they look* pseudonymous is the wrong thing to do.
What I think needed to be said in the boardroom conversations is that the damage done by deleting a good (even pseudonymous) profile is about 100 times that done by allowing a bad profile. Bad profiles (spammers, trolls) will ruin a site, but they're only dangerous if numerous enough to establish a critical mass. The cost of ripping out a good, potentially pseudonymous profile, is a nasty blog post that makes the rounds on the internet.
There's a lot of interesting grey area between surrendering everything to Google and rolling your own mail server. I unplugged from Google this month but I'd still rather pay a little money to fastmail.fm, for instance, than fight spam myself.
The net isn't the wild west anymore but it doesn't have to be a sterile walled garden either.
This guy really should go work for Google and figure out the problems they need to deal with running a service like Gmail. Even for just a little while.
At work we had a researcher from Yahoo Mail come in and give a presentation on the machine learning techniques they use to try and stop spammers abusing their mail servers. It was eye-opening to learn just what kind of hourly battle they face to keep spam out of their systems and the ways they are trying to combat it. It was even more enlightening when the presenter told stories about the problems that machine learning can't solve - like people within the company being bribed to whitelist spam companies based in Vegas.
On the surface it's such a simple problem, and I'm sure anyone who's tried to prevent their web application's outgoing mail being marked as spam by the evil corporations of Yahoo and Google will have had the desire to go write a blog post saying what a crock of shit the whole thing is and how they would never take part in that. But here's the thing - those systems are in place because if they weren't, email would be a completely useless form of communication at this point.
The people sending spam make _millions_ of dollars abusing a system which is popular because its open and based on trust. That kind of money combined with greed gives people all different levels of drive and incentive to get their emails about bigger penises and viagra through to your inbox. Every time they prevent one form of attack, these guys will create a new one.
To do this they do things like install mail servers on unsuspecting user's machines, specifically targeting Yahoo/Hotmail/Google users because their IP will obviously need to be trusted by those companies. They will also hack into other people's private mail servers. They will spoof email headers and pretend they're someone else. They will hire people, experts, who will find new ways of breaking in to servers they detect as having mail servers running on them. All this just to get past the spam filters and prevention that make email a useful form of communication to begin with.
And let's forget the people who couldn't set up their own mail server for just a second. I like to think I know what I'm doing. After installing Postfix and jumping through all the hoops to get my emails whitelisted by Gmail and making sure I didn't have an open relay on my mail server, you know what happened? Someone managed to hack in by brute force anyway. I only noticed because of the _millions_ of automated replies that were coming in every day from dead email accounts or people that were out of office.
Now, I could have worked hard to fight this. I could have did something other than changing my passwords and hoping they didn't get crack them again. But the point is - I only ran a mailserver to get email delivered to me on my personal domain. I didn't want to have to fight and battle and dedicate myself to solving this problem. I wanted to take this thing for granted. I just wanted to send and receive email. Instead bad people could not only sit there and read all my incoming mail - but they could use my server to spam people and get me blacklisted and blocked from so many other services I worked so hard to be trusted by. And they did all this without even specifically targeting me. I was a statistic to them, someone who simply didn't know what they know. In the end, I moved my personal mail account to Google Apps, free of charge. Problem solved.
By using Gmail or Yahoo Mail or Hotmail - you are almost definitely more secure than setting up your own mailserver. You have people paid hundreds of thousands of dollars a year working full time to make sure your data is secure. I mean if privacy is your reason not to use Gmail, then I hope for your sake your mail server is secure. Maybe you think it is. I know I did too.
And all these people complaining about advertisements based on the content of their emails. Yahoo Mail had a team of like 30 people just doing _research_ on how to stop spammers. Then all these other people working on support. How does that service get provided to us _free of charge_ without advertisements or some sort of monetisation? I know in some people's heads they think it's literally just a Bayesian classifier and some hand-coded rules, but it's so beyond that.
And of course, let's not forget the fact that a lot of people would not be able to set up their own mail server anyway. Maybe you don't need them, but Hotmail, Gmail and Yahoo Mail enable hundreds of millions of people to communicate _for free_ with other people around the world that otherwise wouldn't be technically competent enough to buy a domain name and set up a local mail server. It lets you communicate with them too, because they don't get frustrated wading through hundreds of spam emails just to read the good stuff.
And that system only works because we have good guys that are fighting the bad guys who want to ruin it for the rest of us. And this is just the one example of email. Which has all this decentralised and open properties that you desire. I am reminded of Diaspora when they released a first beta of their code and it got absolutely torn to shreds for security reasons, and we haven't heard much since.
The real world sucks.
That's why I think it might be a good idea for you to go work for Google.
Thank you, this post was good and informative. Nevertheless I think you missed his main point and concentrated on something that was merely incidental to it.
Yes, spam fighting is hard. Yes, it's probably easier with huge centralized installations (he actually observed that at this point the centralization offers advantages over the decentralized model.) But his main point was not about spam nor even about e-mail in general. His point was that it is worth putting the additional effort into making decentralized systems work. This is definitely not what Google are doing.
Google charges for Postini's standalone service. Their integration of Postini's function into Gmail and it being "free" in exchange for some of your freedoms is the entire point of this discussion and the mailing list post.
The OP said they should license it (the spam filtering portion specifically). They do. If you don't want to pay you can also use it for free. Simply put, I don't see his point.
That's neat. But it's also the first I've heard of it. Do they promote it outside of the business world? I get the impression that they don't think of it as something a CS department or non-profit would use.
Have you looked for it? It's pretty tied in with their Google Apps offering which is where I've seen it, despite not being in the market for it.
It's no surprise that they concentrate on marketing their business software to businesses though. That's who's going to pay for it. A CS department is going to use whatever the school uses and a non-profit is better served by Google Apps (which is discounted heavily for non-profits).
Your solution to admin incompetence is for a centralized service to eliminate the admin. Why can't the service just provide competence? If dozens are people are working round the clock to eliminate spam for your free mail service, why can't they package that and let you control your own data?
The centralized solution you've proposed carried to it's fullest extent is basically eliminating email altogether, where a small cabal of whitelisted services are only able to pass messages to each other. If spam detection software must remain secretive and proprietary at these big companies, this is basically a capitulation to the spammers.
The anti-spam systems work because they are based on content of emails and properties across the providers entire user-base. Every time you click "Mark as spam" you are contributing data for all users in the service. In a decentralised service, even if people agreed to submit all their emails and information for the greater good (which they probably wouldn't), the data still needs to be centralised somewhere and secured by experts. The blacklist/whitelist of notorious spammers and servers needs to be maintained somewhere. You end up having a committee to do that, an elected/trusted group of people and they need to deal with appeals, etc.
Two:
If the logic for blocking spam were public, don't you think that would make it much easier for spammers to circumvent?
Edit - I can't reply to the user below. Must be some HN feature. But the logic for accepting an email is essentially a decision tree, it is based on data and evolves over time. It is a very different problem from something like encryption.
Fyi - On HN, after a message is posted, there's a delay before anyone can reply. The farther "down" the message is, the longer the delay. The logic is that this delay will prevent uninteresting back-and-forth flamewars. I'm guessing that's the HN feature you were talking about.
By analogy: "the logic for encrypted two-way communication (e.g. RSA) is public, don't you think that makes it much easier for hackers to intercept your credit card details?".
Enough has been said about security - or spam filtering in this case - by obscurity.
You are drawing an invalid analogy between cryptography and filtering. The only reason cryptography works with open algorithms is that the keys can be kept secret. To a very large extent in filtering the specific algorithms are as analogous to cryptographic keys as they are to the other parts of crypto-systems. That is, filtering algorithms are like very primitive cryptography where there was no separation between the system and the keys.
If you can propose a spam-filtering algorithm which would not be circumvented if its exact implementation were known, I'd seriously love to hear it. That would basically be a magic bullet for all spam.
Spam filtering is a wicked problem. The solutions are contextual, and there's no one single tool that will slay the dragon.
That said, a great many anti-spam solutions work by well-known and publicly available methods. DKIM (header signing) actually utilizes PKI. DNSBLs are publicly queryable (in some cases the zonefiles may be downloaded), Bayesian and rules-based filters are also generally available.
The real challenges are:
1. Spam is cheap. Spam mail outnumbers ham (non-spam mail) by 100:1 or better. There's a lot of it.
2. Distinguishing spam from ham is contextual, and people's contexts differ.
3. False positives are expensive. Wrongly classifying ham as spam carries far worse consequences than wrongly classifying spam as ham (false negatives). Filters must skew to permissive.
4. There's little central agreement on methods, there are many old systems in existence. We've seen a few small advances (DKIM, SPF) in the past decade, but brute-force content filtering is still required.
5. Even well-established strong verification tools are too technically advanced for the vast majority of the userbase, and/or are unappealing to others. PGP MIME-encoded email signatures (strong cryptographic identity verification) dates to 1991, fer crissakes! Getting even corporate-supported users to employ this properly is at best difficult (though it's becoming ever so slightly more common largely due to compliance requirements). For others, repudiability is important.
6. It's an arms race. Spammers change methods (many based on automated tools assuring rapid widespread adoption of new methods) based on new anti-spam methods.
7. Client and server (MUA/MTA) support for tools which would facilitate whitelisting of users and mail peers is difficult. Centralizing mail gateways can complicate the issue if those core gateways emit proportionately high levels of spam (I see or have seen middlin' amounts of spam from Hotmail, Yahoo, GMail, AOL, and other large email service providers, though generally they're pretty good).
That said: whitelisting, reputation systems (sender, server, DKIM, SPF), authentication (DKIM, PGP), contextual (Bayesian), and rules-based (e.g.: SpamAssassin) properly used do make the situation tenable. But this requires extensive support largely for the administrator of an email gateway. End-users may be forgiven for thinking spam is a "solved problem", though at their level it largely is.
What ultimately will solve the email spam problem will be for email to be superseded by another communications channel (SMS, weblogs, social sites, etc.) to the extent that spammers focus their energies there. It's an economic problem, and if the economics fail to support spamming, the (smart) spammers will move elsewhere.
They can't provide competence in a box because there's no free lunch. What would motivated a free e-mail provider would hand you the keys to the castle? If you want this product, get ready to pay for it.
I also think you're placing a mistaken emphasis on data. It's address books, not your data, that provide lock-in on these services. As far as I know, any of them will let you wrest your e-mail from their claws via IMAP or POP. The hard part is telling your contacts to mail you at <address>@gmail.com instead of <address>@hotmail.com
Full disclosure: I recently accepted a job from Google. My opinions on this matter are mine alone and are not based on any confidential information. I forward e-mail from my own domain to gmail. I also run a mixmaster anonymous remailer.
"Someone managed to hack in by brute force anyway. I only
noticed because of the _millions_ of automated replies that were coming in every day from dead email accounts or people
that were out of office."
This is not a description of your email server being cracked. It's a description of someone Joe-jobbing pretending to send mail from your domain. Duckgo for mitigation techniques..
In fairness, there's not enough information provided to determine which this is, though my suspicion is that OP wouldn't know the difference regardless.
It is truly disheartening to see you'd been downvoted when I came into this thread.
The truth is, the OP's domain was probably considered to be in a bad "neighborhood" because his mail server had been compromised for spamming purposes at one point or another. It's dreadfully easy to either misconfigure a mail server or to end up with your mail server compromised.
Regardless, it's easy to hate on Google, especially in a primarily entrepreneurial forum where those posting are often trying to solve tough problems with far fewer resources. But Google is solving tough problems, even when you feel you've been wronged by an algorithm. Gmail has had an unbelievably successful spam filter for years, forcing the competition to rise to the occasion and match it, to the point where people forget how serious a problem spam is. It's not trivial, and it doesn't mean there's a democratic crisis when your e-mails end up in a spam bin. Especially when it's quite likely because your mail server was compromised.
> Regardless, it's easy to hate on Google, especially in a primarily entrepreneurial forum where those posting are often trying to solve tough problems with far fewer resources. But Google is solving tough problems,
I didn't feel the 'hate'. I read that he didn't particularly care for Google's approach. He certainly says nothing about Google not solving tough problems.
I thought it was a pretty fair piece actually, giving Google credit where it's due, and without trying to demonize them; just stating that he doesn't agree with where they're going.
No, our mail server has never been compromised for spamming purposes. I'm well aware of how easy it is to misconfigure a mail server, and it's not that I think we are too smart or paranoid to have done so; it's just that in the years that we've been struggling with that problem, we've never discovered that misconfiguration, or discovered outgoing spam (other than bounces from e.g. kragen-tol-request.)
I hope I didn't come across as "hating on Google."
All it takes to be considered a "bad neighborhood" is to have a dynamic, ISP-owned IP, as I found out when I tried to send mail from my personal server. And yes, I'm too cheap to pay Comcast even more money for a static IP.
I run my own mailserver too and saw some similar problems from early on, though not AFAIK with gmail in particular. If it ever has been compromised, I doubt it was right away.
So... "it's too hard so I'll just let google handle all my email". Works fine until people starting blocking google mail because they don't trust them. This isn't "might happen one day" - it happens to me today already. You're just punting on the real issue, kicking the can a few months down the road.
Who's blocking Google? Do you mean everything from Google servers or only @gmail.com addresses? If so it's trivial to get a domain and still be using Google for your email.
A family member's office is one I know (along with a few others) that are firmly in "MS Exchange" mode, and they've blocked mail from gmail and other google mail servers, because "google's not secure". Of course, they let hotmail mail through just fine :)
I never got spam before I used gmail. Now maybe this has more to do with timing, but it seems like putting everyone's emails on the same domain just makes things easier for spammers. Seems to me like spam is a problem caused by centralization, not solved by it.
It also seems like putting everyone's information in one place makes it easier for hackers to harvest, as well. Gmail probably has a security hole somewhere, too. If gmail's hole is discovered, everyone's emails are compromised (or a large number of people). If a private server gets compromised, there isn't as much there. There's not as much motivation to hack 1000 servers to get 1000 people's information as there is to hack 1 server to get 1000 people's information (although I recognize that one server is going to be a lot harder to crack on average).
I'm open to an education on this topic, as I don't know the methods of modern spammers/crackers.
It also seems like putting everyone's information in one place makes it easier for hackers to harvest, as well.
Google dreams of being able to handle all that information on one server.
Besides that, it's not incredibly common (albeit not impossible) for people to steal information by actually hacking directly into their servers, especially with someone like Google. More likely ways to get at someone's email is through XSS or phishing attacks.
>If Alice’s email gets marked as spam, Bob ought to be able to find out why — and fix it!
While Gmail doesn't exactly let you figure out why, you can nevertheless fix it. That's what the Not Spam action is for. I've had mail land in the spam folder that shouldn't have, it only took a few 'not spam' actions to retrain it to let it through again. You're also free to backup your Gmail through both imap and pop. I never got the Gmail paranoia--the worst they do from my perspective is possibly deep-analyzing my emails in an effort to better serve me ads. They possibly sell the data to others (though I've seen no evidence of this) for them to serve me better ads. All these ads I don't ever see anyway because I use AdBlock Plus making their efforts pointless for my account.
I'm not a fan of the rhetorical conflation of decentralized computing with democracy. His other material I don't really want to comment on.
Do you know if Not Spam works locally or globally? Assume I'm emailing Alice and Bob and get flagged as spam. Alice checks her spam folder every day and keeps flagging my mails as not spam. Bob just assumes Google's spam filter works and has never looked in his spam folder. Will my emails eventually get through to both Alice and Bob or just to Alice?
I'd bet with fairly high odds (and hope) Gmail whitelists locally. If Alice thinks your mails aren't spam but Bob does, I would want Alice to get your mails normally but Bob to be able to keep ignoring you, implying a need for local blacklists and whitelists with a reasonable set of defaults. If Alice + some large number of others think you're alright, maybe you'll be promoted to not being flagged as spam by default. Bob might then just have to take the action of marking your things as spam to invoke a (hopefully) local spam/blacklist rule.
There are a few functions which whitelist locally like adding someone to your contacts. But I think "Not Spam" moves it to the inbox and provides feedback into the voting algorithm without whitelisting at any scope.
So far the "real name" policy is the only major, undemocratic, incident that I've seen from a centralized online service. Does anyone know of any other incidents that make case against centralized servers?
Nonetheless, I agree with him that we need to make decentralized computing practical. The best example I've seen of this is Opera Unite: http://www.youtube.com/watch?v=ivshJ-qyg5w
There is also Freenet, but so far (from reading their mailing lists) they are discussing about changing their load management. I've tried Freenet and it slows down my machine (not very practical), but it is the only software project I've seen that distributes the hosting of digital content among peers instead of a centralized server: http://freenetproject.org/
So far the "real name" policy is the only major, undemocratic, incident that I've seen from a centralized online service. Does anyone know of any other incidents that make case against centralized servers?
Easy access for law enforcement, easy access for the NSA, CIA, FBI, everyone else. The attack on Gmail by Chinese hackers used the interface that Google provides to law enforcement to use. There's also the commercial access part. Some companies sell their centralized databases to 3rd parties.
About freenet; it encrypts all network traffic so that no one else knows what is being transferred. That causes quite a bit of a slowdown. Also, it's Java and on some machines it can use up a lot of RAM, especially on older machines.
The alternative is encrypting your emails and letting GMail store that. The problem with that is that they still know when you sent an email, and who you sent it to. Just the fact that your email is encrypted can be taken as sign of guilt by law enforcement. But it's still an option.
> The alternative is encrypting your emails and letting GMail store that. The problem with that is that they still know when you sent an email, and who you sent it to. Just the fact that your email is encrypted can be taken as sign of guilt by law enforcement.
Not sure but thinking back to those Chinese hackers they might be interested in knowing who's talking to who.
It's also useful for anyone who wants to smear someone else. "Oh politician so-and-so is talking to such a person in secret, I wonder what they're saying".
Diaspora was/is a hint of the future for distributed social networks, but too bad it was/is a bit poorly executed.
To be honest, I had hopes that Google+ would be part of an open platform with federation, considering Google Talk already uses XMMP and Google Buzz used OStatus partially (just for broadcasting, not for receiving, though). I even commented about that, right here on HN.
The real-names policy changes all that, I lost my hope on Google leading a social network decentralization, so it's all a bit sour for me.
http://norayounis.com/2008/11/04/308
The impressive flickr account of blogger and friend Hossam el-Hamalawy
is facing censorship by flickr themselves! First they disabled
pro-palestine photos he had taken in Ireland, then they started
searching their old drawers for reasons to un-list his account from
public search.
decentralization-stories
http://www.worldchanging.com/archives/009100.html
The most credible voice in the Ossetian war, he tells us, was a
Georgian blogger who’d fled Abkhazia for Russia. His LiveJournal
account was highly critical both of Moscow and of Sakashvili, and was
widely read in the Russian blogosphere. But a flurry of denial of
service attacks, launched by a set of zombie computers likely
controlled by Russian hackers, disabled LiveJournal for an hour, and
forced the owners of LiveJournal to ask the blogger to leave the
service so that future attacks wouldn’t take down the platform. He
moved to Wordpress, but had the same experience. If governments are
able to unleash attacks that can disable whole platforms, it’s likely
that they’ll successfully silence many online voices.
decentralization-stories
http://elphabawest.blogspot.com/2008/12/microsoft-live-has-n...
Microsoft Live blogging used to support anonymous commenting, but they
were tracking the commenters’ identity --- and when they turned off
anonymous commenting, all the previously-anonymous comments became
non-anonymous. decentralization-stories
http://ascii.textfiles.com/archives/1617
Eviction, or the Coming Datapocalypse
(AOL deleted a lot of people's web pages with scant 4 weeks' warning)
decentralization-stories
https://www.eff.org/deeplinks/2011/04/googles-lack-transpera...
Earlier this month, Google removed Grooveshark’s popular app from the Android Market for violation of the Android terms of service, later informing Grooveshark that the removal was related to a “complaint from the RIAA" but nevertheless refusing to provide an actual legal or policy basis for the takedown.
Yeah - I'm not a fan of centralisation either, but what this article misses is that the most important thing enabled by the internet is not blocked by any of Google's practices.
What is the most important thing? In my view - it enables the formation of "Super Groups" - which I think will represent the most significant cultural change since the dawn of language.
All you need for the formation of super groups are sufficiently cheap and efficient signalling processes. Google has perhaps contributed to this drop in signalling costs as much any company on the internet.
Anyhoo - for those who want to know what a super group is - I wrote about it here:
The breathless gibberish of your 'Super Group' claims are at least rendered entertaining by their unfortunate nomenclatural confusion with Journey, Asia and Bad English. Any technology that encourages more "supergroups" to form should be killed with fire, lest it inflict another Crosby, Stills, Nash and Young-style 50-year musical reign of terror on us.
"What is the most important thing? In my view - it enables the formation of "Super Groups" - which I think will represent the most significant cultural change since the dawn of language."
Errr ?!
That's why religion always wins. Makes people believe such things.
@grovulent: I am willing to engage the idea. But the amount of suppression(of contradictions) your referred post demands is too high(that i did not finish it). I believe it was the same for the other poster. May be you can express it differently and post again.
I'm interested in your idea and I read your G+ post. What is a super group? You gave some examples but I can't figure out what thing your examples have in common.
Also touches on the discoverability problems with using mailing lists like this. I wish I'd been reading this list thirteen years ago when it was started.
> I want my list mail to serve as prior art to stop
obvious patents from being granted, or to revoke them or the obvious claims in them in court
Do discussions of ideas really count as prior art, or must there also be at least an effort of implementation ("reduction to practice" according to wikipedia[1])?
In theory, yes; if the invention is "known or used by others in this country [the US]" prior to the date of the invention by the patenting inventor, it counts as prior art.
It's tricky, though, because that's the invention date, not the filing date, and the "inventor" could lie about the date of the invention. The safest approach is to publish my ideas in print in a foreign country, ideally in a small print run in a language spoken by nobody at Intellectual Ventures, and then wait a year before posting to the list.
Prior art does not have to include a reduction to practice; conception of the invention is sufficient. Lore has it that Charles Hall's waterbed patent was narrowed substantially by a reference to Stranger in a Strange Land, which described waterbeds.
but if you used a centralized server for your blog, wouldn't the date/time stamp of that server provide the same information as a mail server would?
In other words: wouldn't that file/database entry timestamp (ie: wordpress' time -- just an example/name of a blog not necessarily available at that time) serve the same purpose?
Yes — but if an evil person were running a blog like that, they could write posts about other people's inventions and put fake old dates on them. They could also fake the timestamps added by their mail server, but not the timestamps added by the mail servers of their subscribers. So the timestamp wouldn't have as much legal weight.
I think there's always a balance between something being open and transparent (and possibly free and opensource, as those two things usually, though not always, go hand in hand) and something being closed and proprietary. I personally think having an opensource search engine - where all the rules for SEO are well known, constantly refactored and updated by the community - would be awesome. Would it be economically viable? Can't tell you that.
Doing many things, including filtering spam, is more difficult in a decentralized environment. (It is curious that email, itself decentralized, has come to be dominated by several large service providers; I wonder how much of this is due to economies of scale for fighting spam and other attacks relative to other economies of scale relative to things not characterizable as an economy of scale? Search of documents published in a decentralized manner on the web is another example.) Many things are even easier in a completely centralized manner, thus G+, Facebook, Twitter, and their morbid predecessors. For all their issues, architecturally decentralized email, web, and internet are much more valuable than the 2011 versions of AOL, CompuServe, and Prodigy silos. So thank you to all working on making the next bits of decentralized architecture work. I imagine it is possible to do a bit of this work at Google et al, but it is clearly way, way down the priority list of any such companies.
> Google wants you to keep your mail in
Gmail instead of on your home computer
Offline Gmail is cool.
"Mail Settings" (gear in top right corner) -> "Offline"
Doesn't work in all browsers since Google Gears was deprecated in the newest versions of Chrome and Firefox, but it's only a minor hassle to run an earlier version of Firefox for the offline mode and syncing.
Very helpful to me and quite easy/convenient to set up, even with the recent Gears deprecation.
'Apple wants to relegate websites to second-class status on their popular computers, and exercises viewpoint censorship on what “apps” they allow in their “app store”.'
It was originally intended that you would make web apps to access on the IPhone and they did not want developers making apps for it. Consumers and developers demanded that feature so you got what you wished for. (not you per se)
Every time I see one of these "Why I don't want to work at Google" articles I think to myself, "Don't worry, they don't want you to work there either."
My frustration with the freedom box is that the way they pitch it, it will only appeal to the nerdiest nerds. I think it's an awesome idea, but for it to make any impact it has to have better packaging.
I think there is a fair bit of difference between 'closed' centralized control in the form of apple / facebook, which is generally proprietary and secretive, compared to the 'open' centralization that google espouses ... they are one of the great proponents of open source ... their android OS for example is open source. Given that, I probably still wouldn't want to work for them though, from what I've heard theres a lot of c++ coding at 1am in the morning. ;-)
"Apple wants to relegate websites to second-class status on their popular computers,"
Does he realize that Apple computer has been around a lot longer than the internet? And that they've been making an OS with applications on it a lot longer than there was a web browser? So maybe that's actually what Apple DOES and does well?
However, at the same time I like the whole "Apps" (AppStore, Steam, Market) concept because it helped create whole businesses. Lots of small people making real money in the age of piracy...
The internet dates from, depending on how you date it, 1969, sometime in the mid-1970s, or January 1, 1983. Apple dates from April 1, 1976. Maybe 34 (edit: 35) years is "a lot longer" than 28 years to you?
Anyway, Apple doesn't do that on their less popular computers — Macintoshes.
Kragen is obviously not well informed of how the Internet works regardless of how long he's been online. Or to use one of his piss poor analogies, just because a cat is 20 years old does not mean he knows every cat in the neighborhood. His descriptions of DSL, wireless networking, p2p, network topology, and his guesses about the infrastructure behind major "evil" web sites are about as narrow sighted as a twenty year old feline.
This is about more than that, it's about decentralization as a way to empower people so that in the end we don't need centralized companies or governments to control our data.
Apple (and other companies) control what you're allowed to download. Google (and other companies) control what emails get through to you and your email history. Microsoft (and other companies) control your hardware.
He forgot to mention the larger and more disturbing point; many of these companies are American and so they're under the jurisdiction of the NSA and FBI (and CIA if you're not from around there). With centralization, law enforcement has easy and direct access to things. The only barrier is a warrant and even that isn't a barrier as we saw in the AT&T NSA wiretapping case.
He wouldn't want to work at Google or many other companies because they're pushing for centralization which brings certain political/social effects that he dislikes.
So can we please have a discussion about the political and social implications of decentralization vs centralizaton rather than the technical aspects??