Yeah, which is why it is only 10K -- and not the 100K like was stated earlier by someone - if it were the 100K then it would be very similar to a typical YC round of funding.
We've used HackerOne at a startup I work at (10-20 employees).
We had to turn it off because we were getting bombarded every couple days with the same issues, that were just run by crackers/hackers running basic pen test scripts. They all seemed to have the same toolkit, and would just run the same tests and report the same bugs. Most of which were either invalid, or just not a priority and, so, a waste of our time to read. The write-up of the bug was also poor, with poor English, and this causes wasted time..
Before signing up for another bug bounty program I'd want to know that:
1) The testers were not mostly just amateur crackers running the same toolkit on 100 sites per day, and the same toolkit that 10 testers ran yesterday.
2) The amount of dupe reports was basically 0.. If we get a bug reported and we ignore it, and make zero response, we still do not want to get the same report 10 times over the next 2 months.
3) The write-ups should have proper English, good grammar, and be very clear.
4) If a user reports 10 bugs, and we only want to pay for 1, that should be totally fine. The other 9 are either dupes that we have ignored before, or new reports that are just not a priority or worth looking at.
5) We basically never want to get into a negotiation with the hackers over if a payout should be $2000 because 10 bugs were reported when we know of all the bugs and, basically, don't value them.
Your experience is exactly why we're building Federacy.
Bug bounties can be an incredibly efficient way to work with outside security researchers to find vulnerabilities, test for best practices, etc., but done poorly, can cause more damage then they help. We want to make them work for startups as well as they do for companies like Dropbox, Shopify, and Google. We have our work cut out for us -- but if we're successful, we think it could materially improve how startups secure themselves.
All the dev teams we've been part of share the same challenges. We're always overburdened with work on revenue-producing features, so being flooded with more work that ultimately doesn't add much value in securing our software is the last thing we want.
Right now our solution for spam, dupes, and low-quality reports is to be extremely selective with the security researchers we allow on the platform.
We're launching in private beta so James and I can hand-pair researchers, help companies write their VRP, and review every vulnerability report.
Other ideas we’re working on:
- Very clear “Known Issues” / “Not Issue/Out of Scope” sections
- De-duping based on comparing report attributes
- Utilizing machine learning to improve de-duping based on description of vulnerability
- Collaboration. Encouraging companies to look at their approved outside researchers as a part of their team and building tools to facilitate this
Do you think any of these would help? Are there other ideas we should be focusing on that might solve these problems more efficiently?
My 2 cents: I used to work on the appsec team at Twitter and can attest that we could not get Mopub to ever resolve any of your security vulnerabilities.
Noise is certainly a problem on bug bounty platforms but our team handled all of that - by the time vulnerabilities reached you they were already valid, triaged, important issues to resolve.
> We're always overburdened with work on revenue-producing features
This is the bigger problem - if your leadership doesn't care about security then it doesn't matter whether you use Hackerone or Federacy or something else, it's still not going to be a priority. This was the case with RB, in my personal opinion.
Of course many companies do care or want to care but still need some handholding - I think Federacy can provide them a lot of value and wish you a lot of success in that.
Hah, yeah, this stuff is hard and acquisitions make it even harder.
I think you started a month after I left. We built a lot at MoPub in a short period of time and when we were acquired I had a mile-long backlog. The Twitter security team was great though and built a war-room during integration. We worked some intense hours leading up to the IPO and over the Holidays, and I’m proud of the work we all did. We migrated a sprawling stack that supported what was then the largest mobile ad exchange and billions of sub-second auctions over just a few weeks. Most of the MoPub team transitioned to other projects and teams quickly though and I left not that long after.
Totally agree that it starts at the top. If the C-level doesn’t care, there just won’t be the resources it takes to build good, secure software. We intend to focus on supporting companies who do care, and we think this focus will also impact how companies using Federacy interact with researchers. We want outside researchers to be viewed as allies, not as a burden.
Have any thoughts on how we can best accomplish this?
Every bug bounty platform has tried to be "selective" in the researchers they allow in when they start. You'll soon discover that selective doesn't scale.
The only way you are going to disrupt the current market is by hiring on your own salaried pentesting talent to participate.
What do you think caused being selective not to scale at other platforms? What do you think we can do to keep the quality of our researchers extremely high?
What we’ve heard in talking about this to a bunch of talented researchers is that they’ve been frustrated with payout rates (too low for amount of work), tone of the interactions between researcher and company, number of opportunities/companies where they can add value (given their skillset - many have said they do the work in large part to learn).
I think there is probably a lot we can do to create/keep balance in the marketplace to address a lot of these if we take things slow.
Would love to hear more of your thoughts on the strategy of building out our team with salaried pentesting talent. Why do you think that is critical to adding a lot of value for startups?
We've tossed around ideas like this -- including something similar to how Numerai uses staking for their data science competitions. The security researcher would stake a small amount based on their confidence that the report is an impactful vulnerability.
I think it's an interesting idea, but could be complicated to get right. We’re also wary of creating barriers that are too prohibitive for some of the really great and hard-working researchers in the world.
I think an easy solution may be to build good vetting tools and a thorough process: a short application, technical interview, and/or trial periods for new researchers. Right now though, we’re personally reviewing every researcher. :)
A big part of this, too, is providing the environment where researchers can learn and emphasize their existing contributions. I think there’s a lot we can do there, while still allowing researchers to provide a lot of value.
That'd be interesting--a small, maybe even just $1-10, deposit that gets refunded if the bug is legitimate.
I don't think punishing dupes is a good idea though, because a researcher has no idea (and should have no idea) whether their bug has been found before, so dupes should probably still result in a refund.
However, as a kid who has no credit card, but has found some pretty spicy bugs (and gotten rewarded for them), it would make it impossible for me to report them.
We definitely don't want to discourage you from contributing. It also doesn't necessarily have to be money, you could stake reputation you've previously earned.
The dupes problem is super important, in my opinion, because it's currently an unpleasant experience for both sides. Not getting paid out for valid work that has simply been reported before (but not disclosed) can make doing this kind of research as a freelancer unfeasible, while triaging duplicate reports burns time for dev teams.
We've tried to build out in-scope/out-of-scope functionality that makes it super simple to keep your scopes current (could even update automatically via API). We definitely want to build out additional functionality that makes publicly acknowledging known, 'won't fix', and non-impactful issues super easy, perhaps by pulling most of the information from a duplicate report. Do you think that’d be useful?
The other thing we want to really focus on is the disclosure process, and encouraging companies to do it as often and soon as possible.
