Hi all. We're really flattered to be on the home page of hacker news. Thanks!!
You caught us by surprise. As you can see we've built in the ability to gracefully handle a lot of traffic but it's not ideal because you need to wait in a queue until we have enough capacity. So I wanted to jump on here and give you an update of a release we're putting out next week.
FastOrSlow does actual browser simulations from around 12 locations around the world. When we first architected it, we wanted to provide as many locations as possible including places like South Africa. That meant in some cases we needed actual bare-metal servers. So we built it on bare metal in data centers around the world.
We experienced some moderate popularity about a month ago and realized we needed to do a better job of scaling on demand. So we re-evaluated moving on to AWS. Around the same time AWS launched in Cape Town and a few other locations, and so we were able to get the kind of coverage we wanted globally.
So we kicked off an AWS migration. We're days away from moving into AWS and using spot instances to ramp up and down fast based on demand. But as I said, you caught us by surprise.
So please give FastOrSlow a try today, but know that by around late next week we'll be doing a much better job of handling your requests FAST without having to wait in line.
Thanks for your interest and patience. Myself and the lead architect, Ryan, will be here to answer any questions and of course listen to your suggestions.
Thanks!! I run Wordfence which does quite well. Profitable and the team is around 35 really great folks with a focus on solid engineering. So we fund it ourselves. For a going concern like us the costs are very modest and we can afford to run it indefinitely without charging.
In fact just the bit of brand awareness we get for our main product, Wordfence probably more than justifies the costs.
We actually created this as an internal tool because there wasn't anything that would give you a clear indication of site performance from DNS lookup to connect to TTFB to page render from around the world. Our own market is global and even a 1% drop in conversion really costs. After building it we realized that it's very little incremental effort to make it available for all.
As I mentioned we're about to migrate into AWS where we run much of our other infrastructure. The financial model and eng design we came up with keeps our costs surprisingly low. We can handle huge traffic spikes and it roughly doubled the cost, even with the spikes. So I'm confident we can run this for the long haul as a free product.
A few things I can commit to: Firstly, it's extremely unlikely that you'll see ads on FastOrSlow other than our own company name. That's never been our model and I can't stand when a great site or service gets smothered in ads.
In addition, our company has been around since 2006 and we've been made acquisition offers many many times by large companies that you know well. We've consistently turned them down and will continue to do so because we are profitable, love our independence and I think we all know what happens to many great products after acquisition. I'm telling you this because I want you to know that this won't be changing hands to a company that will try to monetize it in obnoxious ways.
No plans to offer a paid service right now. Our next release is for internal use and will allow us to do performance surveys across large numbers of sites and present statistically significant insights as research to the online community.
Happy to answer any other questions. Thanks for your interest!
Wordfence is a solid product. I 'inherited' my wife's business's website a few years ago from a small marketing firm. It was heavily infected with malware. I tried several tools to clean it up and Wordfence is the one tool that consistently helped me remove all malware and keep it off. I was using the free version for quite a while and just upgraded to a paid license yesterday as a matter of fact.
In the world of Wordpress plugins it is a gem in the rough.
Should have mentioned, but it's probably obvious, that we also don't need funding. So won't be raising VC with the pressures that come with that. Nothing against folks who do, but that's our situation.
> We actually created this as an internal tool because there wasn't anything that would give you a clear indication of site performance from DNS lookup to connect to TTFB to page render from around the world.
Out of curiosity, did you look into any paid services like SpeedCurve or Calibre? There are also OSS projects like SiteSpeed or WebPageTest.
Full disclosure: I work at SpeedCurve. We've had customers cancel their accounts in the past because they wanted to build their own performance monitoring tool. To the best of my knowledge, most of those customers have come back because building and maintaining a tool like that is no easy task.
...Which I guess is a strange way of saying congratulations on building a performance monitoring tool!
A WAF specifically marketed at WordPress is a clever idea.
You mentioned that Wordfence has a team of 35 behind it. This seems like a large team, and I'm curious to know what the distribution of roles is like? Also interested to know how many users and paying customers Wordfence has?
We focus fairly heavily on engineering, particularly on QA. My co-founder Kerry Boyte ran QA at Symantec, eToys, BBC, and a few other well known companies. She actually built BBC's digital TV testing lab herself. Ran testing for the MS Exchange version of Norton back in the 90s. And ran the web QA team at eToys before the suits messed things up there.
I'm a dev (ceo these days) and I wrote the first version of Wordfence. Also worked at many big and small companies in South Africa, London and USA. We're US based now. And so we're both engineering focused, and our process is to only ship rock solid code via excellent QA.
Funny thing. Finding vulnerabilities is actually just QA. And a vuln is just a celebrity bug. So this culture really helps foster great security research too.
Another big area we invest in is customer service. Our CS team does a great job and we also focus substantial energy on community support in free forums, which really helps foster goodwill from the community and build a relationship with folks who use our free product, many of whom become paying customers.
I started my career in ops in the 90s before moving to dev and am passionate about high performance (as you can tell from fastorslow) - and so we spend a lot of energy and budget on solid operations to support our products.
Kerry is kind of a math guru and is our COO and CFO these days. So we run a tight ship in terms of financials, which is pretty much why we're still around. Cashflow plan updated daily. Awesome projections. Awesome granularity. Sorry if I'm over-sharing, but I love entrepreneurship and perhaps this will give other folks starting out some insight.
We have around 3 to 5 million free customers. Can't share how many paying, but it's a lot.
I haven't coded for some time now. In 2015 I completely handed off coding on Wordfence to a brilliant developer named Matt who is still with us. I've moved my focus 100% to leadership, strategy and somewhat of a CTO role, but I share that with the strong tech leads in our engineering team.
Hope that helps. We're a private company, so I couldn't just dump our internal data, but I hope that gives you some flavor of who we are, our focus, team distribution and culture.
Thank you for sharing the history of the company Defiant, about the people who develop WordFence, and the guiding principles and approach in management.
The overview was insightful, to see what makes up a well-run operation: how the engineering, tight financial management, quality assurance, performance, security, customer support and fostering the user community - all tie together to make a solid, growing business.
It definitely earns trust and respect for the company's culture and decisions. As a dev learning the ropes in a CTO role, this kind of open communication about the internal workings of a successful company and product is great food for thought.
To decisively not seek acquisitions and venture capital, and instead value independence, profitability, good finances, engineering, QA.. This will stay in my mind as a role model.
If there's an opportunity where I work, I'll likely recommend WordFence or other services by the company. (In fact, performance profiling of large/heavy sites has come up a number of times this year.)
Elsewhere a possible blog post on migrating to AWS was mentioned. I love that style of articles, describing the thought process that goes into technical decisions, how it was planned and implemented. Hope I can catch it here if/when it's published.
I feel that half my job as WordPress developer is knowing or finding quality plugins. It can be a real jungle sometimes. Reading your posts leave no doubt in my mind that Wordfence should be my go-to security plugin in the future.
You're saying all the right things :) thanks for the awesome level of detail, too! You guys seem like a really cool company. I'll definitely be making good use of this product in the future; thanks for making it and sharing it!
Thank you! We're hiring!! (Sorry, but I'd be remiss if I didn't mention that). We're currently hiring PHP developers and for QA. visit our company website at https://www.defiant.com for details.
One thing that might help a bit is to make the profiles you've already done public (well a curated list of popular sites maybe so you don't leak information). Have a list on the home page, so I can see what a report looks like, even if it's not the page I want.
If you get a 502 or if it takes a few seconds to load, we're busy adding another DB reader, so just reload and it will appear. Should stabilize as the new DB reader comes online. HN is slamming us right now.
I wish we'd thought of that! Once we're in AWS we'll be posting research because we've created an internal tool to do large scale performance surveys e.g. comparing hosting providers or performance of sites using certain libs etc. So I'll put that on the blog and HN might pick it up.
FastOrSlow is built using Lighthouse. :) We do actual browser simulations using headless Chrome and lighthouse to get the performance data. We do it programmatically from all the locations, bring the data back and build the report.
Thank you for the tool and the details you've shared! Is there any particular reason you opted to run your own instances of Lighthouse rather than use the PageSpeed Insights API provided by Google? API quota limitations, perhaps?
With the positive comments here, perhaps the HN mods wouldn't mind if you posted an update yourself once the AWS migration is completed? I know there's rules around that, but I think they are okay with the occasional self-promotion, especially for great, free tools.
We would be more than happy to chat about how we migrated to AWS, the differences, benefits, down-sides, surprises. Will be a fun post. I'll let rbritton know and we'll put it together, and of course you can try the product too if you like. FastOrSlow has a blog, so we'll post it there.
Lighthouse returned error: ERRORED_DOCUMENT_REQUEST. Lighthouse was unable to reliably load the page you requested. Make sure you are testing the correct URL and that the server is properly responding to all requests. (Status code: 429)
Does the tool detect when the site responds with a "we don't want you here" page from specific regions? E.g. the Cloudfront blacklist, Cloudflare captcha/proof-of-work challenge, etc.
These pages are usually quick to render and may skew results.
Would you consider writing a post on your experiences moving to AWS and spot instances for dynamic scaling? I'd love to hear more about the architectural changes you made for that.
It will depend a bit on how fast or slow the sites are in front, but a general estimate is about 10-15 sites per minute within the limits of the current architecture. Position 2889 will be a while, but it will be today (depending on time zone).
Hi, great tool. I am in line (1,201) can you talk a little about how the dynamic spot instances work? Do you kick off spot instance based on CPU load or network usage of all other spot instances in the cluster? Are all regions created equal, or do some regions get more requests than others?
I have to imagine your bandwidth costs from AWS aren't gonna be cheap.
I think you understand this, but just want to be clear that this is only launching next week. We're currently still on baremetal.
It'll be based on sustained queue length averages rather than any kind of load number. We can also bump it up manually.
Our estimates for some fairly severe traffic based on historic data are quite reasonable for bandwidth costs. But we're a medium sized business, so we have revenue to support that. I think individuals and very small teams can get really surprised by this.
If you're just starting out and don't need dynamic scaling and the tooling that AWS provides, I'd seriously look into colo so that you can buy bandwidth using 95th percentile billing. We use a combination of servers that we own in data centers using 95th billing and AWS. Depends on use case and we have gotten very good at financially modeling future costs for a given config.
We have a lot of jobs in the queue right now. Just passed 1000 folks waiting. Just wanted to reassure you that your job will finish. Just leave the browser window open, take a break and come back. Even if you close your browser, as long as you come back to the same URL, your report will be there once the job is done.
If someone closes the browser, you could remove the user from the queue to save time for now. Lot of users would be closing the browser and might not come back.
This looks like a very cool tool, thank you! I totally understand the "getting surprised". This of all this interest (and the big queue!) as confirmation that this tool is of general interest.
Hi! I work on Lighthouse and Page Speed Insights (similar service to what you got here). Nice to see another player in this space :) congrats on the HN bump!
Wondering if you've had any pain points integrating Lighthouse that you wouldn't mind sharing. Anything more difficult than it should be, or were the docs lacking in some way, etc.
504 ERROR
The request could not be satisfied.
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
Generated by cloudfront (CloudFront)
Request ID: Ba3iKlMjJUrRy2BYUVqVLmm4hcOF70KSJgcfFnU2SUdQFLTIriKq8A==
Thanks. Seeing the same thing. Our team is on it. Just woke up our head of ops at 5am his time. Surprisingly Scott sounded awake and as if he was expecting my call. :-)
It seems cool but there are like 1k sites before mine, i wonder if you guys can implement a feature to email the report after the report is generated in the mean time (seems like a small feature).
I don't expect the site to be fully crowded all of the time, but have you thought about adding caching for previously processed sites? Perhaps a choice to see either a previously generated report or create an entirely new one.
The reason I ask is that I did a report last night, and checked it on my phone in the early morning. I put my site's address in again from my desktop now, and I see that I am 1,000+ in line. Even though a recent report was made for this site already, I can't seem to access it.
Tip: if your site is going to redirect from the bare domain to something else (e.g. `https://example.com` -> `https://example.com/home/`) make sure you enter the final destination, letting it redirect means that the cost of the redirects will be included in the report, which is unlikely to be what you want.
Unless that redirect is something your users are likely to experience, in which case you shouldn't cheat and provide a URL that delivers a different loading delay than visitors will encounter.
Well...in many cases all it really tells you is the roundtrip network latency, which may or may not be something you are interested in.
In my case it actually was sort of interesting - it definitely suggested that the redirects don't have the correct caching headers, because they clearly weren't being served off the CDN. However, this redirect is a pretty unusual case for us, because most of our traffic is either from SEO or links from notifications we send users.
I agree that you should be looking at the experience most users are going to have, but if the first experience for most users is a redirect you should probably do something about that.
Yep. We try to indicate when you're being affected by redirects since `http` -> `https` and `example.com` -> `www.example.com` are common ones. Those can often mean hitting an origin server instead of an edge server first.
Yup! As I noted in a comment above, this is just not a common case for us - usually our inbound traffic is coming from SEO which sends you to the right place. But when I am typing it into a search box, it's real easy to forget.
I am going to look into getting those redirects cached though, there's no reason we shouldn't be able to serve them off the edge.
I've had some issues like "There is a more recent report available for this URL. Would you like to view it?" but then when I try to, it says "No reports found".
These pages use an awful lot of CPU when they’re active—in Firefox Nightly on Windows, a completed results page was consuming an entire core, which is not usual.
Yeah, let me know when it actually works. Started at about 1200 in the queue and 4 hours later I'm still number 81, and have been sitting at 81 for the last hour.
Sounds interesting, but I think you launched too soon.
Looks like a queue worker issue. From eng:
"We're moving about 5000 events per second now. Definitely shrinking now, and memory usage is dropping too."
We're using Chrome's Lighthouse running in containers with a focus on normalized performance between locations, security and responsiveness. This is built from the ground up by us.
Quote from the eng team: "Never actually heard of it. We're build around the raw data provided from the built-in Lighthouse audits plus some of our own custom ones"
Besides the underlying insinuation (what's wrong with tools copying others if the ideas hit all the right notes?), what recommendations did Lighthouse copy?
WPT is mainly a supertool—here's a ton of data, go dig into the waterfall and network requests with some visualizations. I think there's a page for recommendations but to be honest I've never used it nor seen anyone talk about it.
With Lighthouse's performance report, we aim to provide specific actionable advice with estimates for how impactful it will be.
You caught us by surprise. As you can see we've built in the ability to gracefully handle a lot of traffic but it's not ideal because you need to wait in a queue until we have enough capacity. So I wanted to jump on here and give you an update of a release we're putting out next week.
FastOrSlow does actual browser simulations from around 12 locations around the world. When we first architected it, we wanted to provide as many locations as possible including places like South Africa. That meant in some cases we needed actual bare-metal servers. So we built it on bare metal in data centers around the world.
We experienced some moderate popularity about a month ago and realized we needed to do a better job of scaling on demand. So we re-evaluated moving on to AWS. Around the same time AWS launched in Cape Town and a few other locations, and so we were able to get the kind of coverage we wanted globally.
So we kicked off an AWS migration. We're days away from moving into AWS and using spot instances to ramp up and down fast based on demand. But as I said, you caught us by surprise.
So please give FastOrSlow a try today, but know that by around late next week we'll be doing a much better job of handling your requests FAST without having to wait in line.
Thanks for your interest and patience. Myself and the lead architect, Ryan, will be here to answer any questions and of course listen to your suggestions.
Regards,
Mark.