I can understand that a company wants to do away with the distraction of having to worry about running a datacentre but is using one of the major cloud services cheaper than running your own? Amazon and Microsoft seem to be making healthy profits off their cloud services. That's part of your cost.
Now it may be that even if more expensive, it is a better use of your capital (renting instead of owning so that you can invest your capital in higher return investments like hiring better people). But is it cheaper, at an equivalent level of service / redundancy?
> but is using one of the major cloud services cheaper than running your own?
At list prices? Pretty much never. Nobody sane with installations like Evernote are paying remotely close to the published prices, though.
The cloud providers rarely make sense in terms of cost for smaller businesses unless you heavily rely on batch jobs (spinning servers up/down) or other of their more advanced services. There are many valid reasons for smaller companies to consider cloud deployments, but I've yet to see one where it was the cheaper alternative.
> The cloud providers rarely make sense in terms of cost for smaller businesses unless you heavily rely on batch jobs (spinning servers up/down) or other of their more advanced services. There are many valid reasons for smaller companies to consider cloud deployments, but I've yet to see one where it was the cheaper alternative.
The cost of a server isn't just the hourly/monthly fee for running it. It's also the cost of waking up at 3am to turn it back on. If you're managing your own bare metal then you have to deal with that one yourself and I can assure you it's never cheaper than the cloud unless you value your time at $0.
Sure you can save some money by running a server yourself but if your business it as the point where that's the deciding factor in being solvent than you might as well close shop.
This is why you should run your entire application on bare metal (colo or dedicated) with failover to the cloud.
The benefit of the cloud is its elasticity. You can handle a 2000% traffic increase by spinning up VMs instantly; no need to wait a day or week or month for bare metal provisioning.
And you should do this! But once you've structured your application this way, you are already horizontally scalable. There is no reason the core of your capacity cannot be on bare metal, especially now that container orchestration platforms are bringing feature parity to cloud and bare metal. Only provision VMs for handling temporary traffic spikes.
You can generally predict traffic costs for next 1, 2, or 3 months. You can't predict traffic spikes as easily.
So why not invest in dedicated servers or collocated hardware, just enough to cover your expected traffic? Send any excess capacity, whether due to traffic spike or hardware failure, to dynamically provisioned cloud services. This way you capture the economic advantage of elastic cloud pricing while achieving full utilization of your capex investment.
I've heard this burst-to-cloud story for many years now - Paul Maritz pitched it heavily as the pillar of VMware's value/strategy when he took over there.
I've yet to ever see or hear of it properly executed in practice. I think it's a really hard thing to try and bolt on to a legacy system, and if you build a new ground-up app this way going full cloud to start is easy and cheap for a while until you're successful and then the economics are.. complex.
I think there's also the weird angle to it that the folks who have the skill to architect apps like this are very hard to come by.
Would love to be proven wrong and hear about folks who've seen burst-to-cloud work out in practice though!
> I've yet to ever see or hear of it properly executed in practice.
Isn't Netflix the poster child for this, and haven't they been for a very long time now? Not the content, which resides in their and others CDNs, but the back-end, webservers, databases and management systems are all supposed to be in the cloud now (after a long time migrating).
> I think it's a really hard thing to try and bolt on to a legacy system
There are different levels. Architecting a sharded database solution before you need to might be a bit excessive, but designing for multiple web, storage, or processing servers, etc from the start even if using bare metal is smart, and allows (relatively) easily adding capacity even if you are just colocating somewhere. At the point where you are already able to add capacity by scaling hardware, the cloud is a fairly attractive target, especially since automating the steps you take to add/remove hardware is likely trivial if you've already been doing it manually for a while, as the rough edges have likely been smoothed away somewhat already..
Netflix is not a good example for one big reason: their cost of running servers is dwarfed by their other costs, possibly by two orders of magnitude. Netflix can and does waste money on servers.
At the dollar spend for the infrastructure for most apps, the labor costs of dealing with physical servers is not worth it. Saving $1K/month on infrastructure is meaningless if it costs me $10K in labor costs.
Your numbers only work for the little guy when labor costs are discounted to zero.
Renting a dedicated server requires no labor cost of your own (if a drive fails, your provider replaces it), and is almost always cheaper than the equivalent cloud resource, especially when it comes to bandwidth.
That still leaves sysadmins that you no longer need when using the cloud. The cloud allows your developers to run free without the need for infrastructure and sys admin teams to worry about the servers.
Very little sysadmin needed if you go with a managed dedicated server provider. They pretty much take care of most of that for you. A little more expensive than a non-managed dedicated server but oftentimes still much cheaper than the cloud. Plus, you can't really completely get away from sysadmin when working with the cloud either. You actually get a lot less support than you do with a managed dedicated server provider.
Note: I heavily use AWS and managed dedicated servers.
> That still leaves sysadmins that you no longer need when using the cloud
Looking at our timesheets, managing cloud setups takes more time on average than managing bare metal server setups, and the amount of time spent handling hardware issues make up ~1%-2% of ops time spent on the bare metal setups for our systems. We "save" that on the cloud setups, but end up having spend more to account for dealing with crappy APIs to do all kinds of things we have much more direct control over instead.
You'd have to price in the cognitive load for engineering for that setup, and the recruiting costs / turnover costs for anyone that had to be exposed to the colo center. Also, the colo center! Speed & reliability are features, so you'd have to make sure to pick a colo center that had a competitive setup in terms of peering & backbone speeds, power redundancy, etc. Ideally you'd also be in more than 1 colo unless the cloud is acting as a backup datacenter, so you'd need staff in more than 1 region, lest you fall victim to something like the flooding in NYC and the colocated services that fell down during hurricane sandy.
You'd need to price in continued investment into that colocated hardware too, parts break down, cyclical improvements to hardware happen, etc. It's not simplistic at all, maybe there's money to be saved, but unless your hosting costs are really dragging down profit, I'd pinch pennies elsewhere personally.
> You'd have to price in the cognitive load for engineering for that setup, and the recruiting costs / turnover costs for anyone that had to be exposed to the colo center.
All of which tends to be more expensive if you replace the "colo center" at the end with "cloud provider".
Operating a cloud setup is by no means free of operational issues.
> so you'd need staff in more than 1 region
If you're too small to need that, you use managed hosting, pay a tiny premium over bare metal colo providers, but still far less than a cloud setup.
> It's not simplistic at all
Yes, it is, it is trivial to model, and for companies that don't have the experience, there are consultancies everywhere that will do it for you (that's part of what I do for a living - it's always the easiest jobs to get paid for, because there are hard numbers that shows them how much more they'll be saving that year than what they'll be paying me).
> but unless your hosting costs are really dragging down profit, I'd pinch pennies elsewhere personally.
I regularly deal with clients where hosting costs are measured in tens of percent of total operating costs. For a lot of internet related business, hosting is a massive drag.
Data has weight. You'd need to duplicate some portion of your data between on premise and cloud. I feel the better approach is doing 100% on cloud compute with bare metal CDN / reverse proxy
That can certainly be cost effective if bandwidth is your main cost. Bandwidth cost is the most atrociously overpriced part of the big public cloud providers, so it's certainly worth pricing out, in particular for people who already are on a cloud provider and where moving would be complex.
Exactly. The beauty of this is that building to scale to cloud as needed makes bare metal even more cost effective compared to cloud, because suddenly you can plan for far higher utilization rate for your fixed hardware than you could before.
One note: you don't even necessarily need your own colocated equipment. If you're in a location with expensive real estate and/or power it is often hard to beat dedicated rented servers for smaller installations, because you often have more flexibility in renting servers further from wherever you have staff than what would work if you colo and need to be able to reach your setup in person.
Rented managed servers still tends to beat cloud with good margins.
> The cost of a server isn't just the hourly/monthly fee for running it. It's also the cost of waking up at 3am to turn it back on. If you're managing your own bare metal then you have to deal with that one yourself and I can assure you it's never cheaper than the cloud unless you value your time at $0.
You have to deal with outages when using cloud providers too. So you design for availability in either case, and pay someone to wake up at 3am when there's an actual emergency.
We're the ones taking the calls if our customers systems goes down at 3am, and it eats into my profits if it happens because they pay us monthly fees that are structured so they're far more profitable for us if nothing goes wrong.
Guess what's most profitable for me? Bare metal setups. Despite the fact that we end up charging my customers less to manage them, because it's harder to justify high fees when the overall costs are so much lower.
> Sure you can save some money by running a server yourself but if your business it as the point where that's the deciding factor in being solvent than you might as well close shop.
I had a client last year where we cut the hosting costs including the hourly rates for operational tasks by 90% by moving them off AWS.
While they were on AWS their hosting fees made up ~30% of their burn rate. It's an extreme example, but there are a lot of companies out there for whom hosting makes up a sufficient portion of their cost that it is a matter of survival for them to cut them.
"> but is using one of the major cloud services cheaper than running your own?
At list prices? Pretty much never. "
???
It's almost always 'cheaper' for small companies to use the cloud.
It costs a lot of money to pay people to figure what hardware to buy, install servers, networking gear, maintain it, pay for bandwidth, backup etc.
But the cost becomes 'non starter' when you start to access all the nice things that cloud offers: Lambdas, S3, easy backups, etc. etc. and the fact that it 'scales on demand'.
Then of course there are the financing aspects: you don't pay for expensive servers up front, you 'lease' them effectively - which has serious working capital benefits.
All those things have considerable business value.
So sure - if you want to compare only 'server instance costs' - maybe 'buying your own instance' - specifically on a server by server basis ... maybe it's cheaper to buy a server.
But for most small businesses - the advantage of the cloud far outweighs the cost.
I'd argue that 'rolling your own infrastructure' would be risky and costly initiative that boils down to a 'cost optimization' only, and can't really be considered until a business reaches an operational degree of maturity and visibility wherein someone could really sit down and plan quite far into the future.
And even then I'm not so sure - this is why so many big and successful companies still use the cloud.
> It costs a lot of money to pay people to figure what hardware to buy, install servers, networking gear, maintain it, pay for bandwidth, backup etc.
> Then of course there are the financing aspects: you don't pay for expensive servers up front, you 'lease' them effectively - which has serious working capital benefits.
There are solutions other than cloud and going to the extreme of owning servers, switches, maintenance, bandwidth and space in a datacenter.
OVH (an example due to my familiarity) will sell you a dedicated server with 2 x E5 2650v3, 256GB of RAM, 4TB of disk and 500Mbit guaranteed bandwidth (1Gbps burst) for $269/mo. This includes most of the annoying stuff, like replacing failed components, dealing with network issues and such. The bandwidth, conveniently, is also not limited aside from the speed you're given. You can use 500Mbit/s for every second of a month if you want. That's 164TB.
The cost of a n1-highmem-32, which is the closest alternative I could find on GCE is $1030/mo, plus disk ($348/mo for 2TB), plus bandwidth (164TB is $13k from Google).
Also, the cost of owning and running your own hardware isn't necessarily expensive, particularly if you're small. When you're running a fleet of thousands, yes, you definitely want dedicated people for that. When you can fit all your infrastructure in a single rack, it's not much trouble to call up Dell and ask for a quote, install it in a local DC then forget about it. Most DCs give you "remote" or "smart" hands that can do basic maintenance like swapping a failed HDD for low or no cost.
Dedicated/colocation aren't for everyone but they're certainly not as scary as some people around here seem to think. And in terms of cost, I daresay that if you're doing anything bandwidth heavy, the cloud is near-certainly costing you insane amounts of money. You can get decent gigabit transit (~300TB/mo) these days for $400/mo while Google, Amazon etc. charge in the thousands for that kind of traffic.
There is some variance in how people interpret "cloud". Does colocating your own machine at a remote ISP constitute being in the "cloud"? What about leasing a server? What about a virtual server, but a traditional one where it's essentially no different than a single leased server? What if your multiple real and/or virtual leased servers are in the same private network at the remote location? What if there's an API available to you (even if it deploys hardware servers)? What if extra services, such as storage, can be attached?
Different people will likely consider "cloud computing" to apply at different thresholds with different features. I also consider "in the cloud" (handled remotely in the internet) to be different than "cloud computing" (handled by a cloud service), and "cloud services" to me could mean either one depending on context. I find most your statements apply truthfully to what I consider "in the cloud", that is, remote hosting. I don't think they necessarily apply when used with "cloud services" platforms. All of a sudden, all the money you save not having to deal with hardware and paying someone to configure and maintain it is either back (you need someone to deal with the craziness of each cloud services platform, and they might cost more), or offset by higher costs, if you don't have a business case that requires you be able to scale later, or add network storage, etc. Better yet, if all you need is a website, there's plenty of services that do away with all that to just provide hosting of various sizing and shapes. Is that a "cloud service"? Is that a cloud service? Probably, but in a conversation about GCE and AWS, maybe it's not actually what's being talked about.
> It costs a lot of money to pay people to figure what hardware to buy, install servers, networking gear, maintain it, pay for bandwidth, backup etc.
That's part of what I do for a living, yet my clients tends to have recovered my fees in 2-3 months from the savings they make on reduced hosting costs. Those savings include factoring in the ongoing operational costs.
50%+ savings is not unusual. We've achieved 90% in rare cases (high bandwidth requirements - bandwidth costs are extreme at the cloud providers).
> But the cost becomes 'non starter' when you start to access all the nice things that cloud offers: Lambdas, S3, easy backups, etc. etc. and the fact that it 'scales on demand'.
Nothing in a bare metal setup precludes you from using cloud functionality when/where it makes economic sense to do that rather than implement an alternative. I have plenty of hybrid setups running.
> Then of course there are the financing aspects: you don't pay for expensive servers up front, you 'lease' them effectively - which has serious working capital benefits.
That's what we do when we use colocated servers too. There is no difference there.
> So sure - if you want to compare only 'server instance costs' - maybe 'buying your own instance' - specifically on a server by server basis ... maybe it's cheaper to buy a server.
I'm not comparing only server instance costs. I'm comparing total cost of operating fully managed environments, including total staff costs including staff overheads.
> I'd argue that 'rolling your own infrastructure'
You don't need to roll your own infrastructure - you can go to managed hosting providers, or the multitude of consultancies like mine that will do it for you. Heck, I'll happily sign deals with people where I'm paid in a given percent of the saving I generate, because the savings are large and predictable.
> And even then I'm not so sure - this is why so many big and successful companies still use the cloud.
Big, successful companies don't pay the published rates. I have direct knowledge of companies with 75%+ discounts from major cloud providers. At those rates: yes, it is cost effective.
First of all it is important to understand that building/running your own cloud isn't only huge investment and high risk but also comes high cost of opportunity because every cent you spend in infrastructure you hope will be cheaper is not spend to increase the real value of your product. Not making this investment until your organisation and IT are "cloud-ready" decreases this risk. I have seen companies invest more than double the cost of a cloud provider to build their own cloud without having access to the features that a public cloud provider provides.
For me the questions come down to the proportion of resources to revenue and how "cloud-ready" you are.
Hm, but how much of those features they don't have access to would actually be used? After a few decades in the trenches, I've come to (mostly, kind of) accept that on average, most companies do not push the leading edge, and settle for relatively baseline feature sets of all of the software they use. As long as a deployment hits those feature points, they don't aggressively push for learning, testing, deploying and using new features.
This is of interest to me because no matter how I slice it, when analyzed from this "baseline feature" perspective, cloud solutions of conventional software that require deploying upon conventional platforms are expensive, and I figure I must be missing the picture. I'm much more excited about cloud solutions like AWS Lambda that de novo solutions are built around, as those do seem to have more promise delivering the much-talked-about 2-10X cost improvements compared to conventional software, but still at a high cost of vendor lock-in due to lack of cloud API standards adoption by vendors.
There were a lot of basic features missing. After 2 years of initial Openstack deployment there was no DNSaaS available and the loadbalancer solution was very unstable. This was partly caused by the organisation because nobody knew how to build a cloud system and first first few iterations of this private cloud were build like traditional bare metal infrastructure. If the investment for the private cloud had been used to adapt a public cloud provider for all non critical applications the benefit of the organisation would be greater.
There are tools like Kubernetes / Mesos that can help abstract the infrastructure layer so you can move between different providers. I have seen kubernetes federations setups between kubernetes clusters on AWS and kubernetes clusters that were hosted inhouse.
Building/running your "own" cloud doesn't require owning bare metal servers. You can rent and lease for a fraction of the price of renting cloud instances. Once you start paying up front of reserved instances, the capital invested tends to start stacking against you.
You certainly can burn through ridiculous amount of money if you insist on purchasing servers outright and pick expensiv brands.
Conversely, the "private cloud" deployment I run are mostly rented servers with month-to-month contracts. No upfront infrastructure spend - we grow them a server at a time. We also run some bare metal setups and they're generally cost-competitive with rented servers, but there it certainly depends on the size of the installation.
I recommend OVH.com. They provide bare metal servers in a semi-automated way and have locations in US, Canada and France.
For the type of application that I run, where our demand is not very elastic, I find it very cost-effective and more flexible (as long as you're comfortable with administrating KVM and ZFS). I also like that with ZFS, I can easily sync snapshots between providers at the filesystem-level.
Hetzner and OVH are my go-to providers if I don't have specific needs that precludes it (in Hetzners case, geography - I don't think they have data centres outside of Germany).
I don't tend to care that much what control I have over the network as most of the systems I deploy are set up to allow us to tie in compute or storage resources "anywhere", which means we put encrypted overlay networks in place for all inter-server traffic no matter what to get a consistent fabric to build on top of. It costs a bit of CPU resource, but I've yet to run into anywhere where's it's been an issue.
It is cheaper for my company. We have very high seasonal demand on one of our websites. Our busy months need 10 times as many servers as the low months. With amazon we pay for the servers only when we use them. When we did it ourselves the servers spent most of the year idle, but using electricity and maintenance but earning us nothing just so we could handle out peak demand. Now we pay nothing for them when we are not using them.
December is one of our low months: those servers we were hitting hard in September are now helping some online store keep up with the Christmas rush. I believe this is factored into the price we pay.
We do a lot of our own work in house. In general we have enough IT staff that it is more cost effective to keep our own servers in house, but that is for loads where the variation is relatively small.
Reminds me of the "buying locally grown food" argument that's premised on the assumption that buying locally grown food requires less carbon emissions because the food doesn't have to be transported from far away. HOWEVER that argument ignores that even after factoring in transportation, those larger industrial farms are farm more efficient than your local farm. So even with transportation the industrially sourced food might be more carbon efficient.
Only if they are so efficient that after they get their cut, it's still more efficient than rolling your own.
Because these cloud services have inherent lock-in, they don't have enough incentive to keep margins low. Indeed, Amazon's margins are extremely high after factoring in bandwidth costs.
Agree, but that's my question. Does the savings achieved by these cloud providers more than offset their margin, or is the motivation to move to the cloud solely a return on capital / removing a distraction?
It seems that their motivation was enabling the use the "machine learning" as they say or the "Big Data Analytics" as the user threeseed recognized in his post here.
25:30 - Rationale, background
53:00 - Discussion of financial models, move from capex to opex
Keep in mind that the volume of data in terms of video assets for broadcast quality linear playout as well as numerous continuous high bitrate outgoing video streams is very large.
As they note in the video, their financial modelling showed that it made financial sense to move to AWS if it was "all-in", allowing them to fully close down a number of large data centres.
If all you are using Amazon/Microsoft for is EC2 style hosting then yes it's expensive.
But if you are interested in Big Data Analytics as well (as Evernote would be) then it is a bargain. It is very expensive to run your own managed Hadoop cluster, build the data ingestion and scheduling components, event store, NoSQL store, machine/deep learning pipelines not to mention AI technologies and have them all integrated and cohesively packaged.
As one author once wrote "Whenever a company says 'In order to serve you better', you can be pretty sure they're about to do something that will make your life more miserable." I like the point of xkcd 1150 too. But Evernote actually does take money from their note taking customers.
Except, I get to share one person across several systems.
Simplified example:
$100k/year for a Linux engineer
That same person can upgrade 10 Linux servers (on-premise company) or 1000 linux servers (cloud provider). It may take the 1000 linux servers slightly longer, but the cost to service is still significantly lower.
There's also a lot less variability in skills of the Linux engineer.
I don't think people quite understand that cloud computing is a consolidation and elimination of backend computing jobs, not the increase. The only reason there is an increase in the market, is because the overall pie is growing.
You know Facebook, Apple, Microsoft, Google, etc and other major companies use CDN's like Akamai to deliver their content to users right?
Building a few data centers may not be not be hard and may be cheaper, what's hard is serving content fast and as close to your users as possible in the world. For that how many data centers could you really build?!
> is using one of the major cloud services cheaper
> than running your own?
Depends on the size and to some extent the distribution. Above a certain size (and this varies based on the mix of memory, storage, and CPU cycles) it is cheaper to run your own stuff in a space leased from a data center. However that size generally has to be achieved at every data center you want to run in. For example, if your function was that it saves money above size 10, And you need 7 units in the US and 7 units in Europe, it is probably not cost effective to run your own even though your up to 14 units of capacity, but if you're running 15(US) and 12(EU).
That said, you still have to have the the operational expertise to run it and with that expertise will come an understanding of the actual costs. For example, at Blekko we were running 1500 machines in a data center and had just over 12,000 hard drives. Those drives are going to fail over time so you need build process around continually renewing the drives. You assume that you'll be able to get some replaced under warranty but you should also simply depreciate the cost of drives over 3 years. So you spend $110 on a drive, you depreciate it at cents per day. That is an "expense" you incur as you buy replacement drives. So your storage is "costing" you $36,000/month ($0.10 * 12,000 drives * 30 days). At the time it was the 2TB generation and our system used simple triple replication for data protection so it represented 8PB of data protected storage, of which, at any given time we were utilizing roughly 5PB. S3 from Amazon [1] is .021/GB per month so 5 petabytes (just what we were using) would be $105,000 per month (so roughly triple the cost). And there is the network 'in/out' costs which raise the prices further and you don't pay for your own storage. Most of our storage management costs (detecting bad drives, reformatting, scheduling for RMA) was done automatically but it did take an data center engineer one day each weak to work through the list of drives that had been determined to be bad, swap in replacements from the 'incoming' drive box, and put the outgoing into the 'rma' box (40 drives in the box) and once the outgoing box was full, print out the list of serial numbers and generate a standardized RMA to Western Digital and to then put the shipping label on the box. So in addition to the depreciation cost you add 20% of the cost of a date center engineer who is doing the mechanical work of moving dead drives out.
So you can cost it out this way for your own infrastructure and figure out which is cheaper, and for a lot of people it will be cheaper to run it in the cloud because you really don't need multiple petabytes of data or thousands of cores on line or terabytes of RAM. And some people build their systems in efficiently so that they are spending way more on people keeping them running than they might if they had designed them differently. glindahl had honed a set of scripts and tweaked Nagios bits that made running lots of machines fairly efficient but not everyone has someone like that on staff.
But the place where it is a real win, and I think this was the economic story that got Evernote to move, is that when you downsize your infrastructure needs it is trivial when you are in the cloud, and it is non-trivial when they are your own assets. Data center leases are generally multi-year and they are loath to let you sublet space or reduce space.
Since Evernote has been shrinking this makes a lot of sense for them, when their user base shrinks their costs shrink with it. And that is the "killer app" for cloud based infrastructure.
Perhaps phrase the question differently, to challenge the premise it is 'cheaper':
"Could they set up a datacentre as an independent subsidiary that they were client to, but that was run on a commercial basis for other customers too?" i.e. commercially viable.
If the answer is "Yes" as it was for Amazon years ago, then it seems like a no-brainer - do it yourself. Otherwise, such decisions calls for greater scrutiny. For example, should HR services be run internally, or outsourced. Outsourcing some functions don't work even they're more expensive on first analysis, like the network effects internal HR can have, or the answer may be "No" for some services, Payroll a common example.
Arguably their core business is synchronized notes and well-done applications. The storage, hosting, networking, electricity etc are all implementation details.
I think economy of scale will kick in at some point allowing the cloud providers to profit while being a cheaper service than what the clients could have done themselves.
Evernote’s engagement with Google engineers was a pleasant surprise to
McCormack. The team was available 24/7 to handle Evernote’s concerns
remotely, and Google also sent a team of its engineers over to Evernote’s
facilities to help with the migration.
Those Google employees were around to help troubleshoot any technical
challenges Evernote was having with the move. That sort of
engineer-to-engineer engagement is something Google says is a big part of
its approach to service.
It's interesting to read that, given that one of the perennial complaints about GCE on Hacker News is the relative lack of support compared to AWS or Azure. Is it just that when you're a customer that big, even Google is willing to give you personalized service? Or is this a sign of a change in the GCE support model?
Option 3. This is PR and not a postmortem containing facts you can use to inform your decisions.
I fear there is no journalism here. No attempt to uncover real motivations, nothing remotely negative, no dissenting comment from current/past engineers, nothing about who paid what to whom, the incestuous relationships of ex-Googlers (Evernote CEO is ex-Google), how much Google is losing to take on high profile customers, who is placing these stories or anything other than pushing the narrative that "everybody must move to Google Cloud".
I run a service on Firebase (now part of Google Cloud Platform) and ran into problems recently. To my surprise, the support was excellent. And I've only a few thousand users.
As for whether that level of support was acquired with the rest of Firebase or is native to GCE, I don't know.
> Or is this a sign of a change in the GCE support model?
It's standard practice for all professional services (not limited to tech or cloud). When you are a major client you always get privileges and custom support.
It's a mix of you pay so much for the service that they can give you some help + it's good for PR and case studies + it helps them to gather feedback on their services.
That's just the way business do business (outside of the 20 years old children at SV companies who are scared to talk to sales guys and support).
Google does not have any engineers in the countries where are company operates or has contractors. Many people on HN are not from the US or not from locations having this option, so most of our judgment comes from using the online help or having a call with Amazon/Google/MS. Based on my experience, I still think Amazon wins. I bet it was really helpful for Evernote to get such a support though.
The Stanford HPC team reported last year having sent
more than 2PB to Google drive using Lustre/HSM. This was using the public API and the fact that gdrive is free for *.edu sites.
So a large part of choosing GCE is possibly because CEO was a Googler and he was very familiar with how Google works and his contact at Google helped him scored a better negotiation and possibly even better support. Although I won't deny GCE team would snatch Evernote regardless because it's an important client to show off in GCE's client portfolio.
> The decision to go with Google over another provider like AWS or Azure was driven by the technology team at Evernote, according to Greg Chiemingo, the company’s senior director of communications. He said in an email that CEO Chris O’Neill, who was at Google for roughly a decade before joining Evernote, came in to help with negotiations after the decision was made.
Evernote didn't get everything for free. They still pay Google with some NDA stuff in the contract (better support, better pricing model, some free stuff here and there). I wouldn't have a second doubt the CEO's past employment at Google influenced the decision. After all, he had to sign it.
You're right; they didn't get anything for free. They gave up access to their logo and the story for discounts. That's how the game is played. And there's no problem with that! Both sides win.
I just thought it was funny that they mentioned the donuts at all. Sales/solution teams bring stuff like that all the time. It's totally irrelevant, until you realize that the original article was a puff piece.
> Meaning private data? Is this covered in the TOS?
Your question is right on the mark. The last attempted privacy policy change, less than two months ago, did cause some reactions:
"The latest update to the Privacy Policy allows some Evernote employees to exercise oversight of machine learning technologies applied to account content"
"The policy changes have to do with machine learning, which Evernote says it is using to “help you get the most out of your Evernote experience.” Evernote wants to let its machine learning algorithms crunch your data, but it doesn’t want to stop there — the company also wants to let some of its employees read your notes so it can ensure that the machine learning is functioning properly."
Evernote wanted to take advantage of the cloud to help with features based on machine learning that it has been developing
I noticed there are other companies citing machine learning as a reason to move to one of the big three (Xero.com gave the same reason when moving to AWS). Could someone give more Background Information to this and what are the exact Advantages?
According to this article, they intended to analyze the notes taken by their users and also have their "employees read your notes so" they "can ensure that the machine learning is functioning properly" and that to "help you get the most out of your Evernote experience."
I believe that they simply wanted to enable the targeted ads or sell the newly generated data, and that they still want to do this. But maybe somebody here has more insight.
What I get from this is that they want to use Google's available horsepower in machine learning and maybe, trying to build their own TensorFlow equivalent would have cost them extra and have added risk of becoming rapidly out of date, since Google is investing heavily in it.
I'm sure they've known that for a while but the company is about as old as the big public cloud providers and at that time I was certainly nervous about using them.
If they started today I imagine they wouldn't have dreamed of having their own datacenters.
I would pay good money to Evernote if they had a "backup all my data multiple times, removing all possibilities of losing anything". I would gladly pay for Premium if that was an option.
You should look at CloudHQ. Their PDF backups to e.g. Google Drive preserve everything out of Evernote. Used it once to get away from Evernote while still preserving search of old stuff.
Nah. It's not a good simple nice solution. Google also hates Desktop Apps (except their browser). And IMHO Google Keep makes a much better job for short notes without wasting whitespace.
That was a surprisingly frustrating article to read. It kept repeating the exact same things just worded slightly differently over and over again, like the four or five times it mentions machine learning being a major part of Evernote's decision to go with Google, without ever explaining what, exactly, that machine learning would be.
The whole thing reads like a press release targeted at an audience that doesn't know anything about the actual mechanics of cloud computing.
Edit: Fortunately Evernote have published a great five-part series on their blog[1] which goes into much greater detail about the transition to GCS, although there's still no explanation of how machine learning fits into their future plans.
That's more or less because it is - it's aimed at enterprise IT 'decision makers' with predominantly on prem stacks, and is a glorified marketing case study.
Sadly the article never describes HOW they transfer the data. Simply sending 3PB on the wire will cost a fortune.
Maybe Google does have some non-public service like AWS Snowball?
Nothing against the author, but it sounds like another story that Google PR pitched and influenced. It is normal these days. I am sure AWS does the same. That is why the technical details may be light. The submitter is likely a Social Media Manager for Google (nothing wrong with that - kudos for not obscuring it!)
I, too, have wondered about the technical details when I see stories like this. If the data was originally stored in S3, they probably used the Cloud Storage Transfer Service (https://cloud.google.com/storage/transfer/). The secret sauce appears to be parallelization of the transfers.
When I copied hundreds of GB of data from one bucket of to another AWS bucket, it took me days to do! Then I realized it was because I was running a single threaded process. The slowest part was getting the list of objects to copy. If I had a separate, faster index of the objects, then I could have done a copy in parallel much more quickly and max out the bandwidth available to me. (Maybe that is part of the rationale of Netflix's S3mper?).
I'm curious to hear other thoughts on accomplishing fast transfers of S3 data.
Edit: theoretically speaking, if you had 1 TB of data to transfer, and a paltry 100 Mbps (12.5 MB/s) connection, you could do this in under 24 hours if the entire network was fully utilized. Not a network engineer so go easy on me if I am wrong!
As for costs, sending 1 TB out will cost $90,000 in S3 transfer costs, before any other discounts.
Hi! Author here. (Nice to meet you.) I can tell you that Evernote, not Google, pitched this story to me. While the Google Cloud PR team was aware of its existence, they did not contact or attempt to influence me when it came to its content. (Literally, all that they did was confirm the on-site presence of their engineers, and that an engagement of that sort wasn't a one-off favor to Evernote.)
Unfortunately, a technical deep dive on the exact parameters of the transition wasn't really in scope for this piece. I tried to include detail where I could, and Evernote has a really wonderful 5-part blog series that goes into many of the details of the transition. (It's now linked in this post.)
To provide some additional details on the transfer: Evernote was going from data centers in California that it owned and operated to Google, not from S3. Technically speaking, the migration isn't 100% complete, as Evernote is still finishing the transfer of its attachment files to Google.
Evernote did not disclose the exact details of how that transfer was executed on the record, and thus it did not end up in my piece. Thanks for reading!
Happy to add clarity where I can -- it is my job, after all. Suspicion and skepticism are healthy, and I totally hear the criticism that the piece feels marketing copy-esque.
For what it's worth, the Evernote team seems genuinely happy with the way the transition went, along with their interactions with Google.
The most straightforward is running aws s3 command but as you noted it's single threaded, slow as hell. Imagine you have 100TB of data to migrate from one bucket to another. If the command encounter an error (which is very likely to happen), you have to start again.
I think the only other option left is use Spot Fleet EMR. I believe the S3 API takes a marker for pagination so you can probably keep paginate to list the next N keys (of course assume your current bucket is no longer accepting new objects... otherwise create a Lambda function and let new file to be copied to the new bucket on create event).
I don't remember if aws s3 cp --recursive bucket1 bucket2 requires local copies first, or it is a remote copy-only operation.
I have always wanted AWS to allow S3 bucket to be transferred from one account to another with simply transferring ownership.
I did try EMR + s3distcp a while ago. AWS Support recommended it and provided instructions in their docs. It was not very fast for me. The data I was trying to copy was about 400 GB of small 5-20 kb files - not ideal for fast copying. I gave up and used a different method.
I suspect Evernote, if using S3, would have a similar data profile of many small objects.
the timing of these blog post is actually just perfect.. probably they knew that the "geeeks", after reading the PR article, wanted to have more insights.
Yeah, this is actually AWS Snowball. But the Google Cloud does no have a competing offering. It has none publicly, so it would be of real interest if they only have it for big customers handing over their first born son or for everyone on request but not advertising it like AWS.
3PB over a fast, peered connection isn't unreasonable (and again, this would all be ingress to GCS which we don't charge for, but your provider would). The limiting factor would be how much you want to attack your storage layer at home.
Disclosure: I work on Google Cloud (but wasn't involved in this migration, nor even particularly aware).
First, getting data into the cloud is cheap, it's outbound bandwidth that's expensive.
Second, that's peanuts for a big company. Most wouldn't notice a 100k bills for a one of transfer. Not to mention it's negligible compared to the size of storage and processing.
Why would it not make any sense? They probably did an analysis on this and decided that GCP overall is a better solution to their problem.
I regularly move companies to the cloud because they cannot afford to setup their on services and just want to focus on the business side of tech and write software only that delivers business value. These companies tend to use queue services, CDNs, scalable file stores, clustered database servers, disposable environments for research. If they had to operate these services, it would be a horrendous cost and huge amount of effort to set this all up and keep it running. Cloud vendors to these things for a reasonable price and it is ready to be used today. I find it very hard to think of any scenario where cloud does not win the game, this is why Google and Microsoft is betting on it.
I regularly move people off cloud providers once they realise how expensive they are.
We usually cut peoples fully loaded costs in two or more by moving them to dedicated hosting. Far more if they're bandwidth intensive (Google and AWS charge in the region of 5x to 50x as much for bandwidth as most other hosting alternatives)
There is certainly segments that use a lot of disposable environments and complex services where it makes sense to go to cloud providers, but I've yet to see a client where it came out as the cheapr option.
If they're big enough to negotiate steep discounts, then that certainly changes - I know one company that has 75% discounts over the public prices from one of the major cloud providers, and they're not that large (yearly spend in the high six digits).
How would you move an S3 user off cloud?
Do you have networking, load-balancing and performance experts on call if something goes sideways?
I guess we are aiming for different companies, my clients want scalability and do not want to deal with BGP issues, power outages, vendor RMAs and the other things people forget to add to the basket when comparing cloud vendors to dedicated hosting. Amazon buys networking equipment and servers on a discount you would not able to buy, they also get a great discount for the networking bandwidth and energy as well. It would be very hard to beat them in the datacenter game. I worked for companies where we were in charge of datacenters (I have built few) and companies using cloud services. Based on my experience you cannot meet with the reliability, security, availability and efficiency of AWS/GCP when using a small dedicated hosting provider.
> How would you move an S3 user off cloud? Do you have networking, load-balancing and performance experts on call if something goes sideways?
The vast majority of S3 users have loads that can be handled with a couple of low end dedicated servers.
If you can afford to pay the S3 bandwidth charges, then by the time you can't serve the load with a handful of servers, you're paying so much that you can afford hefty retainers with consulting/devops companies larger than mine and still make massive savings.
S3 can be cost effective, but only if you rarely retrieve the data.
> I guess we are aiming for different companies, my clients want scalability and do not want to deal with BGP issues, power outages, vendor RMAs and the other things people forget to add to the basket when comparing cloud vendors to dedicated hosting.
Same with my customers, which is why they pay me to set up and operate systems for them. Still ends up far cheaper than cloud setups.
> Amazon buys networking equipment and servers on a discount you would not able to buy, they also get a great discount for the networking bandwidth and energy as well.
I'm sure they do, the problem is they're not passing those savings on. Especially on bandwidth.
As an example, S3 bandwidth out for up to 10TB/month is $90 per TB. I pay ~$2 per TB for most of my own projects. Depending client needs we use providers that charge from ~$2/TB to $20/TB.
It sounds to me as if you've not actually looked at prices at alternative providers.
^ This. I recently moved a bandwidth intensive app off of AWS to a LiquidWeb dedicated server and have been able to save over 50%. The cloud is not the cheapest option and not always needed for small/medium sized businesses/apps.
You also have to take into account the number of people it takes to run the data center operations in looking at the total costs. If they had 20 people required to operate running their own servers, that's probably around $2m/year in personnel costs, whereas they might be able to reduce that to, say, 5 people and save $1.5m just on that.
Please excuse the loose relation to this specific aspect of their relationship, but there's something bothering me about Google & Evernote.
I recently set up a consumer grade LG Android smartphone. A few days later I got a push notification inviting me to install Evernote[1]. I wonder who gets paid for that: the telco, LG, or Google? How does one unsubscribe from such app adverts? By disabling Google Play Store notifications? I really didn't know push ads for apps are a thing. Besides the annoyance, it seems like an unfair advantage over all the other top apps.
Thanks!! I was hoping someone might drop a tip like that, so I kept the notification around. Indeed.. After a long press, it reveals that it's owned by the Evernote[1] app. Apparently it comes preinstalled. The gear button results in a crash, but I was able to block notifications under the info icon. Probably best to take the time and purge all the bundled apps, before another one pops up.
I have the Kindle app on my phone and it sends a notification just about every day. I didn't mind because often their recommendations were interesting to me and I would occasionally buy the book they were pushing.
Then they started pushing magazines. I wish there was some way to tell Amazon that I'm not going to subscribe to a magazine. I ended up turning off notifications.
I just realized I can make another ~700 polite off-topic questions, before my karma runs out. That's actually very acceptable :-D. First time I see it as currency.
Now it may be that even if more expensive, it is a better use of your capital (renting instead of owning so that you can invest your capital in higher return investments like hiring better people). But is it cheaper, at an equivalent level of service / redundancy?