It looks like they are just querying yahoo for results and ads as a proxy and it sounds like, through some contractual/technical limitations, they have to make that request from a yahoo controlled domain name (duckduckgo-owned-server.yahoo.net). However, they say they fully control the server and yahoo does not get to touch it, as yahoo delegated DNS to them.
The change does not seem malicious and if what they say is true, then the implementation might not be too bad. The volume of traffic they get probably means you cannot profile an individual from Yahoo's side. However, this potentially means that your search history is leaked to Yahoo, albeit in an aggregate manner with other DDG users, which may have attacks that I'm not aware of.
What I can see, however, is a lot of noscript users get startled by this change as they will see DDG use a script from yahoo.net (actually duckduckgo-owned-server.yahoo.net) as by default noscript does not show the full domain.
The cynic in me says that it all starts like this. With multiple, gradual ToS changes, eventually, it will become like any other company out there. And what if Yahoo acquires DDG?
* Offer something for free, with VC money.
* Gain market share since you don't need to worry about profit.
* Realise you need to make money, so start turning into the companies you stole market share from.
Just want to point out that although that may be a model for many startups, DuckDuckGo is already profitable through (non-tracking) advertising and affiliate revenue. More info: https://duck.co/help/company/advertising-and-affiliates
I think it's very, very reasonable to be skeptical of any claims about DDGs actual concern about privacy. Actions speak far louder than words.
Gabe made his first few million by making a classmates.com clone that was the opposite of respectful about privacy. Those actions made clear that he has no ethical or moral stance regarding privacy, and that he will happily violate privacy for profit.
Add to that the fact that he took venture funding. Venture funding aims for a maximized exit price, not a moral stance. The moral stance will disappear the moment your site gains meaningful traction. It's what Ayn Rand (one of your founder's favorte authors) would have wanted.
DDG users who care about privacy are, to be blunt, laughably gullible. Your current business is pro-privacy because:
1) the search context creates sufficient value even without the personal target; and
2) the tech is a dumb layer on top of other tech, and it can't provide personalizaton.
As such, yegg's pretending he cares about privacy... but he's just selling the limitations of his lousy tech as benefits. It's a PT Barnum move. As soon as the tech is better, the privacy will disappear.
I don't want to be rude, but I don't think DuckDuckGo has actually taken much market share to worry about yet, so maybe it's not following the usual startup route.
Maybe. It's not difficult to be profitable though. Servers are cheap. I don't think it's going to trouble Google though, which is a shame. We could do with some serious search competition.
Might have used the wrong term, but the above comment was dead earlier, and several of his other comments are dead despite not seeming to violate any rules.
That link is pretty good at explaining the DNS delegation, TLS and NoScript. However, as DDG runs on AWS, they don't have sole control of their servers anyway.
Edit: output of a DNS/Ping tool I wrote for a book:
Enter a hostname or IP address:
duckduckgo.com
Performing DNS lookup of duckduckgo.com
Complete, duckduckgo.com = 54.229.105.92
Performing reverse DNS lookup of 54.229.105.92
Complete, 54.229.105.92 = ec2-54-229-105-92.eu-west-1.compute.amazonaws.com
Complete, duckduckgo.com = 54.229.105.203
Performing reverse DNS lookup of 54.229.105.203
Complete, 54.229.105.203 = ec2-54-229-105-203.eu-west-1.compute.amazonaws.com
Complete, duckduckgo.com = 46.51.197.89
Performing reverse DNS lookup of 46.51.197.89
Complete, 46.51.197.89 = ec2-46-51-197-89.eu-west-1.compute.amazonaws.com
Complete, duckduckgo.com = 176.34.135.167
Performing reverse DNS lookup of 176.34.135.167
Complete, 176.34.135.167 = ec2-176-34-135-167.eu-west-1.compute.amazonaws.com
Complete, duckduckgo.com = 176.34.155.20
Performing reverse DNS lookup of 176.34.155.20
Complete, 176.34.155.20 = ec2-176-34-155-20.eu-west-1.compute.amazonaws.com
Complete, duckduckgo.com = 176.34.131.233
Performing reverse DNS lookup of 176.34.131.233
Complete, 176.34.131.233 = ec2-176-34-131-233.eu-west-1.compute.amazonaws.com
Pinging duckduckgo.com 4 times
Ping attempt #1 of 4
Success
30 ms
Ping attempt #2 of 4
Success
30 ms
Ping attempt #3 of 4
Success
30 ms
Ping attempt #4 of 4
Success
29 ms
Press any key to exit...
Same source as before. Yes, I assume that's their hosting company.
Enter a hostname or IP address:
startpage.com
Performing DNS lookup of startpage.com
Complete, startpage.com = 37.0.87.7
Performing reverse DNS lookup of 37.0.87.7
Complete, 37.0.87.7 = rt87bb0-37-7.routit.net
Complete, startpage.com = 89.146.4.146
Performing reverse DNS lookup of 89.146.4.146
Complete, 89.146.4.146 = rt4bb146-89-146.routit.net
Complete, startpage.com = 145.131.132.81
Performing reverse DNS lookup of 145.131.132.81
Complete, 145.131.132.81 = rt132bb131-145-81.routit.net
Pinging startpage.com 4 times
Ping attempt #1 of 4
Success
26 ms
Ping attempt #2 of 4
Success
27 ms
Ping attempt #3 of 4
Success
26 ms
Ping attempt #4 of 4
Success
28 ms
Press any key to exit...
Protip for the supertip: I just use !s for startpage. One keystroke lesser. :) It's the one I use most of the time.
Although my default search engine is DDG, for my use (technical and otherwise, with date based searches) DDG is still more like a toy that doesn't provide me the relevant results more than half the time. So I stick to startpage most of the time and sometimes use Google directly (with tracking related precautions/protections taken).
So they data mine this traffic at the nameserver level, just to reasonably say that DDG doesn't share info? Making it technically your computer sharing the info, by way of their forced implementation? Seems like the ad blockers are making the right move here.
This comment is puzzling, since authoritative nameservers only see actual end-user traffic on an occasional basis given the cornucopia of resolver caches in near-unanimous use (as well as the fat TTL on the yahoo.net side of the delegation). This denies Yahoo! the reliable data that you seem to think they've successfully pulled over on everyone, at least if I understand your comment correctly which I'll admit I'm having a hard time doing.
It's also an odd concern given that most everybody these days uses Google or their ISP's resolvers which would be far more interesting "nameserver level data mining." (I have a hard time believing Google successfully logs that traffic in detail, however.)
I can't grok what you're asking me but I'm almost certain the answer is no, as you've already inferred. Extrapolating DNS analytics to actionable intel is incredibly misleading in nearly every case on the service side, particularly since there are a number of caches that consider TTL a guideline and since large shared caches are commonplace.
Understanding what your company is doing if you run the resolver is where it makes more sense. But on the service side? You might as well hire a cat to report metrics by pawing a ouija board, and then you at least get cuddles with your random numbers.
Think of it this way: DNS is basically a different form of ARP, in a certain manner of speaking. I wouldn't try to quantify performance data from either except to troubleshoot operations.
So they data mine this traffic at the nameserver level, just to reasonably say that DDG doesn't share info?
What does this mean? If you mean nameservers might get to see your IP, that isn't true. At most they'll see your primary DNS resolver's IP, but if you use Google DNS or Open DNS they will only see that (I've implemented CDNs, and this is a very big problem for them if they operate at the DNS level - which most do).
I've been astounded at how rare it is for a search API to offer date information. I talked to Gabriel about the problem many moons ago, and even with their significantly greater clout (than a random dude working on a tiny project that uses search engine APIs), and despite asking for it, they couldn't get that info from their partners. So, it's good to see they finally made it happen with Yahoo.
But, I thought the Yahoo BOSS search API was gone, and the only way to use Yahoo search was basically just a black box sort of deal (i.e., you pass them the query and they hand you results, which you display as yahoo sent them). I'm not sure I understand what DDG is doing, and when you'd hit those Yahoo results. It's especially convoluted since Yahoo gets its results from Bing, right?
Always loved DuckDuckgo - kinda came across as a search engine that didn't seek to takeover the world, and only sought to fill a (necessary) niche. That seems more genuine than "don't be evil".
The basic problem with Google is the age old software engineer problem, not knowing when enough is enough.
They kept tweaking results and trying to be helpful, and over time building a "bubble" around the individual user rather than trying to produce the objectively best results based on the query.
There's some truth to the pattern, though; Once you're no longer the underdog, those seeking power will seek you out, and it's hard to prevent infiltration by the power-hungry.
Good things dont scale. Sergey Brin & Larry Page could have been the old Japanese man in the subway station sushi joint making the the best bites of a lifetime for a few hundred humans a day. They decided to go a different route.
I don't think I follow. A good web search engine cannot be a niche product with a few users. The bigger the resources, the better the results. That's an impossible problem to overcome by merely being really good at something. DDG overcomes it, reasonably effectively, by getting results from a huge list of partners.
If there weren't huge companies offering search APIs, DDG simply couldn't compete with Google (without massive scale of their own). As it is, DDG still provides somewhat weaker results in some cases; it is still my primary search engine on all of my devices, and I like it, but when I'm frustrated by the results, I make use of !g and usually find what I'm looking for.
> The underdog is always everyone's favorite until they somehow wind up on top.
Google was everyone's favorite for a while even after becoming the market leader. And it seems the reasons to dislike google are pretty diverse (although around here I mostly hear "because they forced g+ login on youtube" or "because they killed reader". FTR my biggest annoyances with them was 1: when they killed off Google Desktop Search, 2: the years when they insisted on fuzzing any search until it returned millions of results )
>> although around here I mostly hear "because they forced g+ login on youtube" or "because they killed reader"
You forgot {{insert all possible privacy concerns}}, and on that wave, total disrespect for user choices. As a very tech savvy human I still can't dodge all their malicious attempts to collect my data, imagine the non savvy human fellow.
Please help me understand why DuckDuckGo decided for the query to go in the URL, as opposed to being encrypted on the client before being sent to the server as a get or a post.
Most users probably have their browsing history switched on, and synced with their friendly browsing software provider anyway, totally reversing any privacy advantage a user might potentialy gain by using DuckDuckGo.
> Another way to prevent search leakage is by using something called a POST request, which has the effect of not showing your search in your browser, and, as a consequence, does not send it to other sites. You can turn on POST requests on our settings page, but it has its own issues. POST requests usually break browser back buttons, and they make it impossible for you to easily share your search by copying and pasting it out of your Web browser's address bar.
I want to use DDG. I've made it my default search engine. But the result page is just so slow. Both in slow to load (v Google), and actual responsiveness. It's just slightly laggy, or at least it feels like that. Google has neither problem. I'm willing to put up with not-excellent relevancy, but the slowness is killing me.
I'm using FF 49 64-bit on Win8. Am I anomalous or does anyone else feel this at all?
Which of course depending on their ranking implementation seems kinda odd, as you could expect a big part of the users of SO also to be the ones using DDG. But again if it categorically undervalues results from there, then those users will of course find alternatives.
Might be the query then, I usually get the proper SO result as one of the first 3 matches. Anyway if you're getting results that are really off, hit the 'Send feedback' button. A DDG employee here on HN claimed they use this actively to make results better - hard to teel if that is so, of course.
This is what I've noticed as well. Searching for Rust programming questions is very hard with DDG, whereas Google gets relevant results at the top. Otherwise, DDG and Google are pretty much on par.
I generally have no problems with it. I've tried FF/Chrome on Win8/10 and Arch. The load time might be marginally slower than google but the UI is fine.
I haven't noticed but I wonder if the complexity of making the infinite results pop into the page as you scroll is causing the difference. For me this feature alone is worth a mild slowdown in responsiveness. On Google you gave to click to get to results >10.
You're not anomalous. I'm not usually using DDG, but checked it out on chrome 51 64-bit on Win10, compared with google and there is definitely noticeable lag on DDG in many areas, though NOT to the extent of making page unusable.
I've noticed that DDG is maybe twice as slow as Google (still damn fast), and much faster than the links returned in the results. That said, I have recently noticed that occasionally, no results load at all, and I F5-it.
I do sure hope that any agreement DDG has made with Yahoo doesn't get cancelled by their eventual new owner.
But if Dan Gilbert ends up being the buyer my guess is that he'd actually like Gabriel Weinberg enough to continue. Not so much with Verizon or the private equity firms bidding.
DDG is a meta search engine that uses mainly Yahoo and/or Yandex (depending on the region).
Yahoo had turned their Yahoo BOSSS API (acccess to Bing search engine) from free to pay-for-each-1000-queries and completely discontinued the API earlier this year. So now we learn DDG gets access, but no one else of the smaller fishes (other meta search engines, etc).
For the layman like me: if I do a search for "cute cats", does the results comes from Bing after it has been through Yahoo before DDG shows it? Who does the actual searching?
For those who'd been missing date-restricted searches, DDG now offers these, thanks to the Yahoo partnership.
It's not as flexible as Google's date-ranged searches, in which specific start and end dates can be specified, but you now can restrict searches to the past day, week, month, or year. I'd noticed this (and submitted to HN) about a month back.
I switched to using StartPage[1] a while back, because I got sick of waiting for DDG to add date-filtering. I've kind of got used to StartPage now, so not sure if I'll switch back... although some of DDG's 'bangs' were handy.
That's why I'd prefer a search engine that was not using AWS, or Yahoo. Too many needless dependencies.
Out of curiousity I'm starting a counter today. Will post something when this IP addr fails. Feel free to take a guess how long it will be used by DDG.
And when it does change, if ever, I have a one line shell script that uses ed to remove entries from files, so all it takes to add a new IP addr to HOSTS is typing
"We've worked closely with Yahoo to implement these new features in accordance with contractual terms and our privacy policy. To be as transparent as possible, we've written up details on the technical implementation."
As a search user, I couldn't care less about this stuff. I don't think it's even possible to build a good search engine "in accordance with contractual terms".
I think the point the previous poster was making is that even though it's regulated, there are so many black boxes that it's rather easy to sneak in a violation and no one notices (whether the company is doing it or it's accidental). Just look at how many organic product recalls happen in any given year due to various causes.
Why do they use www.duck.co as their development url and www.duckduckgo.com as their search engine name?
The latter is just way too long to type. Especially for casual users who has google set at their main search engine. Compare URL lengths:
duckduckgo.com 14 (10+4)
google.com 10 (6+4)
yahoo.com 9 (5+4)
yandex.ru 9 (6+3)
bing.com 8 (4+4)
baidu.cn 8 (5+3)
I really want to like duckduckgo, but the name is really not well chosen. Even google (with a relatively long name) at least have two repeating characters.
I can't remember the last time I typed in a search engine URL. I type my queries into my browser URL field and let it do the right thing. When I want Google, I use "!g query" (because DDG is my default, and that's how you tell it to redirect the query to Google).
Under what circumstances do you type in the name of your preferred search engine?
When I want to use the non-default search engine (non-default is usually ddg). When trying a search engine for the first time. When using a public computer.
Biggest problem is on mobile. On Chrome (for iPhone), I have the following options:
1. Google
2. Yahoo!
3. Bing
I don't see any way to add a search engine not on the list.
This doesn't change the fact, but you really shouldn't expect anything else from a browser made by a search engine provider. They probably wouldn't even allow the other two options, if they could get away without monopolization lawsuit.
DuckDuckGo is to long and ddg.gg url is kinda makes it disservice. It instantly gets labeled as "nerd-stuff" if you recommend it to some non-techy friend, you just know they never going to even try it.
Wow, what are our options now?! Also, Yahoo now has information about general searches made by DDG users even if they can't get info on a particular individual.
Yahoo, the company that saves your unsent and deleted email drafts forever and will turn them over to law enforcement even though they say that should be impossible?
Maybe Yahoo is sincere in their privacy pledge, maybe they're not, but one thing has already been proven: they don't have a good enough grasp of how their own servers work to make that promise.
Very disappointed in DuckDuckGo. I expect they won't be around for too much longer, much like Yahoo.
It looks like they are just querying yahoo for results and ads as a proxy and it sounds like, through some contractual/technical limitations, they have to make that request from a yahoo controlled domain name (duckduckgo-owned-server.yahoo.net). However, they say they fully control the server and yahoo does not get to touch it, as yahoo delegated DNS to them.
The change does not seem malicious and if what they say is true, then the implementation might not be too bad. The volume of traffic they get probably means you cannot profile an individual from Yahoo's side. However, this potentially means that your search history is leaked to Yahoo, albeit in an aggregate manner with other DDG users, which may have attacks that I'm not aware of.
What I can see, however, is a lot of noscript users get startled by this change as they will see DDG use a script from yahoo.net (actually duckduckgo-owned-server.yahoo.net) as by default noscript does not show the full domain.