We at Instacart use them heavily. It is a system my team built called Bento Remote ("Bento" is a local orchestration tool we previously built). We built Bento Remote ourselves because Codespaces was in its infancy at the time and really couldn't meet our needs for things like pre-built images, preserved disks, warm pools, live patching, and being behind our firewall. Bento Remote was and is hugely successful and likely one of the largest boosts to developer productivity in our history.
With Bento Remote get a fresh new fully dedicated EC2 machine in about two minutes from when you run `bento remote create`. It is continuously tested on every merge and verified to function. Everything you need is preinstalled and ready to go. Just connect VSCode to it and start coding. No futzing around getting your local environment to work. Break something on your Bento Remote? Throw it away and get a new machine. Switching projects? Grab a new machine from the pool that is preconfigured for that project. Your settings travel with you.
We had a potentially unique set of circumstances though. Our full stack development requirements were VERY high and needed 64GB to even run our largest and most active apps. This made other 3rd party tools fantastically expensive. Building it ourselves let us fit instacart specific needs and workflows. As well as do clever things like hibernating the EC2 instance after hours when the users laptop is idle. This plus a host of other measures were essential to making this not only cost effective but a large net gain.
So I highly recommend everyone at least take a look and evaluate if you have the need. Start with the SaaS versions and see.
The cheapest instance I could find approached $1k USD per year without egress costs. If you can build all of that, why use EC2 over your own hardware? If you’re spinning up containers, why not use the dev’s workstation for that? You already paid for it, right?
How is buying a your dev a single top of the line expensive 64GB RAM MacBook equivalent to on-demand EC2 instances?
Despite containers working locally, the ease of having a remote image with everything preinstalled is huge. Lots of large companies offer internal dev cloud environments - I’ve always found it to be a frictionless experience versus local setup at other companies.
Suppose you’re building a feature that requires $newlib as a dependency. How is that handled? Do you request the addition of $newlib and wait for a new remote image to be deployed for everyone before you can work on your feature branch?
Right. The devs have full sudo access. They can install and configure whatever they want. Our Bento tool has pre/post setup hooks for all services. So teams can automate this. When something is commonly used we just preinstall it for them. Or some just contribute to the repo. It is all fairly straight forward Ansible and shell scripts. We push nightly to everyone's existing machines.
We actually moved away from containers for dev purposes for almost all of our apps, though many teams still use them. Mainly because they are a bad developer experience. We invested in making apps run consistently in a native way.
Bento was actually our first attempt at doing all of this on our local MacBooks. Many users found that their machine would slow to a crawl and heat up. M1 Macs and now OrbStack instead of Docker Desktop help reduce that. Moving off of docker helped as well. Ec2 gives us the consistent image that is easily testable and repeatable. Egress costs here are negligible.
We dont' keep the machine running all of the time. It stays up running during the users work hours (~8 hours) and then hibernates (the machine is off but memory persisted) which reduces EC2 costs (minus EBS) to zero. We detect activity on their laptop and then wake it when needed.
It sounds like the number of instances needed fluctuates heavily, maybe max number of concurrent instances needed in a month is ~500 64GB instances while average could be ~100 instances?
We generally try to do things that give a return on investment and don't burn money for no reason. This one most certainly had the ROI. It took 4-ish people to build it in under a year and it removed significant pain and increased developer velocity for hundreds of other developers. Many of whom are working on things that allowed us to be profitable at our IPO.
We're not just a shopping cart product. I thought the same before I first joined but it turns out getting groceries to your door is incredibly hard to do at nation-wide scale.
Sounds roughly right. Though not all of us were full time on it at all times. We also paid that just once about two years ago. It has been part time feature and maintenance work since then.
Now run that same estimate assuming we saved 600 engineers only 1 hour per week (a very conservative number by our research and feedback). $120/hr * 1 hour saved per week * 4 weeks * 600 engineers. We paid it back in less than four months, conservatively.
With Bento Remote get a fresh new fully dedicated EC2 machine in about two minutes from when you run `bento remote create`. It is continuously tested on every merge and verified to function. Everything you need is preinstalled and ready to go. Just connect VSCode to it and start coding. No futzing around getting your local environment to work. Break something on your Bento Remote? Throw it away and get a new machine. Switching projects? Grab a new machine from the pool that is preconfigured for that project. Your settings travel with you.
We had a potentially unique set of circumstances though. Our full stack development requirements were VERY high and needed 64GB to even run our largest and most active apps. This made other 3rd party tools fantastically expensive. Building it ourselves let us fit instacart specific needs and workflows. As well as do clever things like hibernating the EC2 instance after hours when the users laptop is idle. This plus a host of other measures were essential to making this not only cost effective but a large net gain.
So I highly recommend everyone at least take a look and evaluate if you have the need. Start with the SaaS versions and see.