Can you email me - silverlock at cloudflare - with your ticket ID and domain name so I can understand what broke?

Ultimately it looks like the existing DS records for your domain weren't removed (and you can see that in your DNSViz output). Still have some questions for "how" it was working beforehand (see the email for those).

For others: I'll let the OP share what details they would like to, as this is their domain.

I'm nervous about migrating a domain to CF which is used for glue records, and want to have immediate support access if something goes wrong.

Roughly one hour after I e-mailed @elithrar who kindly reached out and offered to expediate the issue, the broken DNSSEC records were partly fixed. The domain once again resolved through all major DNSes, and public access was restored. At that point dnsviz.net told me that A, MX, etc. records were "insecure", though name resolution worked fine. A few minutes ago I took another look with dnsviz and it's now telling me that all records are secure. Everything looks normal again.

Thanks a bunch for helping out, @elithrar. I really appreciate that you were proactive.

If the problem had somehow fixed itself or if the support ticket had gotten any attention or feedback at all within a day or two instead of just being "snoozed" by support staff, I wouldn't have made any noise about it. After four days of complete silence a bit of "cry-baby consumer activism" seemed like the only resort.

If CF reconnects to me with an update on why the domain dead-locked and why it took 4 days to untilt everything I'll add that info as well.

I've been OP and this has been an update about my domain woes.

All plans come with support. Even the free plans (community, or email, the bot will deflect the request but if you email you're still stuck, you will get a reply _eventually_ (due to heavy support load, it can take a while though).

The correct procedure would be:

* turn off DNSsec on old registrar (and wait a day or two)

* update NS and/or migrate domain

* wait a while and make sure it works

* turn on DNSsec in CF dash and update DNSsec settings in the domain

It's not that DNSsec doesn't work -- it's doing exactly what it's supposed to be doing.

Transport security is like HTTPS. DNSSEC was the equivalent of PGP signing every webpage. The former brings value to the end user, the latter not so much.

Even the government has issued memo M-18-23 ("Shifting From Low-Value to High-Value Work") that rescinds the requirements for the government to implement DNSSEC.

WebPKI is a joke because it lacks name constraints and so isn't and can't be hierarchical. DNSSEC is a true PKI -- you can still have multiple roots if you like and don't trust ., but it's got name constraints, so whatever domain you graft an alternate PKI at, from there on down you get bound to that PKI. This is really, truly fantastic.

Add DANE and you have a complete replacement for WebPKI.

DNSCrypt is needed to increase confidentiality, it's cheaper than DNSQuic and such things. Unfortunately .'s and com's and major TLDs' NSes are unlikely to want to waste CPU cycles on any DNS confidentiality solution, and even if some TLDs did, unless clients use QName minimization, users gain no confidentiality -- . and the TLDs all have to adopt it.

The two, together, would be truly fantastic.

If you aren't a fan of DV certificates (as you point out verified by resolution), you can always restrict your trust store to only CA certificates that sign EV certificates (verified by business records).

If you do, then you get MITM protection.

If you don't, but choose to use QName minimization, you still get a modicum of MITM protection: because the attacker would have to choose to get in the middle without having enough knowledge of whether a particular upcoming TLS connection (or whatever) will be of particular interest.

Really, DNSSEC is infinitely better than the WebPKI, even WebPKI+CT, especially when DNSSEC clients use QName minimization, and even more so when clients pin copies of . from time to time.

Source: me, having lost access to my domain for 4 days for reasons that are not yet fully clear to me.

[1] - https://ianix.com/pub/dnssec-outages.html

My understanding is that people break things.

> Just culture is a concept related to systems thinking which emphasizes that mistakes are generally a product of faulty organizational cultures, rather than solely brought about by the person or persons directly involved. In a just culture, after an incident, the question asked is, "What went wrong?" rather than "Who caused the problem?".

Prominent (and very effective) example: Aviation safety.

DNSSEC is both easy to break and hard to fix. https://sockpuppet.org/blog/2015/01/15/against-dnssec/

Upgrading to a non-free plan?

You don't have to upgrade to enterprise, but even their $20/mo plan comes with support.

(Also, I hate to victim-blame here but using DNSSEC was a bad idea in the first place)

(from that article, that is from 2015 and woefully outdated)

> With TLS properly configured, DNSSEC adds nothing.

This is false. DNSSEC adds address lookup security through response integrity, whereas TLS (only!) adds transport layer security to the endpoint you're connected to (hence the name). If you find a record in DNS with DNSSEC enabled, you know that the response is exactly as the sender intended it to be (and when connecting to the address returned for A- or AAAA-records, you'll be connecting to the intended IP address). Without DNSSEC, this is impossible to guarantee and record interception / MitM would be an attack vector.

Additionally, "With TLS correctly configured" also implies CAA being set up, which only can be done securely using DNSSEC. As to why CAA must be configured: Not all CAs are made the same; and re-routing network traffic is fairly doable if you only need to target one of the many public CAs. Targeting only that CA allowed in the CAA record is presumably much harder.

> Securing DNS lookups isn’t a high-priority task

> DNSSEC’s real job is thus to replace the TLS CA system. This plan is called DANE.

No, DNSSEC's job _is_ to secure DNS lookups. DANE is only one scheme that is made possible by DNSSEC; Secure CAA checking being another.

> Real-world DNSSEC therefore relies on RSA with PKCS1v15 padding.

Correct, but also relies on Ed25519 and P-256. A lot of authorative servers are still using the legacy RSA keys, but another lot is using P-256 and Ed25519 too.

> [sections] DNSSEC is Expensive To Adopt / Deploy

This is partly true, but any security is expensive to adopt/deploy. DNSSEC is fairly easy nowadays, though, with many hosted DNS services providing some form of DNSSEC.

> DNSSEC doesn’t secure browser DNS lookups.

It would, if you allowed your browser to recurse.

> DNSSEC is Unsafe

> Authenticated denial. Offline signers. Secret hostnames. Pick two.

That's fine. Secrecy doesn't add security; Authenticated denial and Offline signers do.

> DNSSEC is Architecturally Unsound

I disagree with the conclusion here. Sure, it might be useful for US gov to be the writer of the spec, but what public scrutiny DNSSEC has had implies that the security part is sound.

I really wish my bank would use it and stop calling javascript from domains unknown to me :)

Slack had a 24 hour outage doing exactly that; so I don't think "revert to non-DNSSEC without issues" is trivial at all.

(More broadly though the operational overhead of software is really high these days in a lot of ways. I think that's true of anything, not just HTTPS, but there are a lot of other historical factors leading to that.)

I think it's a bit of a leap to suggest that just doing things like banning mass surveillance would magically make systems more stable or make 15yo operating systems suddenly relevant on the net again. We'd probably still need a lot of the stuff we have in place already. However, I suggest we try it anyway because there's only one way to find out and oh well we won't lose anything valuable anyway.

Mass surveillance is not the only reason to have HTTPS everywhere. It protects not just from snoopers, but from MITM attacks.

* Inserting malicious JavaScript

* Changing content on trusted websites in order to mislead people

* Replacing downloadable application binaries with versions that contain malicious code

Points 2 and 3 are the same, they're about integrity which could be had cheaper with content-addressing (hashes uniquely identifying the content) rather than pulling in the full TLS+CA machinery.

Like Airtel used to (still does?) in India.

But, you make a mistake in firmware version XYZ and there is an RCE in it. So you pull it off your site and now XZZ is the latest version.

Only problem is, anyone that can MITM you can serve version XYZ that the client will accept and make the machine exploitable by an RCE.

I'm sorry but you're not thinking this through very carefully. People still care about authentication of read-only stuff, in the same way much (and it should be all) open source software, particularly from repositories, is signed these days. A great deal of mischief can be done by modifying info in flight, even ignoring privacy concerns entirely. Plenty of not just governmental but corporate bodies could benefit by being able to trivially rewrite whatever populations (or targeted segments of populations) read. Even ignoring what a vector it could be for other attacks. In principle, we could have some universal standard for signing and authenticating as unaltered websites without bothering to encrypt them. But frankly that seems pointless vs just having encryption as well.

Further, like all practical public crypto use in the face of adversaries, there is a lot of benefit from using it universally and thus "hiding in the herd". Otherwise, the mere usage of crypto itself is a signal, and also easier to target and block (not just technically but via laws). Whereas when it's just baked into literally everything that's much harder to outright infeasible, and also destroys that extra bit of signal.

This was all debated and considered extensively while the moves to universal HTTPS were happening. People moved read-only sites as well for a reason.

And the entire discussion was primarily motivated by pervasive surveillance. Which is the point I was making that we're living in a bad equilibrium (low trust society) where there is an attacker and there is a costly defense against the attacker that demonstrates that this particular kind of attack can be rendered useless but we cannot stop paying for the defense because as soon as we do the attacks would resume. If we could instead solve the regulatory problem and forbid surveillance then we would not have to pay that price.

> Further, like all practical public crypto use in the face of adversaries, there is a lot of benefit from using it universally and thus "hiding in the herd". Otherwise, the mere usage of crypto itself is a signal, and also easier to target and block (not just technically but via laws). Whereas when it's just baked into literally everything that's much harder to outright infeasible, and also destroys that extra bit of signal.

That's just a different angle on the surveillance, no? If we had no surveillance then nobody would be there to observe those bits of information you would leak by using or not using encryption.

> In principle, we could have some universal standard for signing and authenticating as unaltered websites without bothering to encrypt them. But frankly that seems pointless vs just having encryption as well.

There are plenty of benefits such as lower latency, much simpler zero-copy IO on the server side (sendfile), improved caching, less energy and silicon area wasted on encryption, less technological obsolescence, a smaller your-ciphersuite/OpenSSL-has-flaws maintenance treadmill. To some extent we could even do without CAs (via content-addressable data).

These may all be papercuts, but we're still getting cut because we can't collectively just tell the NSA to get off our lawn even though we have the means to keep them out anyway.

Your banks support numbers, election poll information, binary downloads/checksums are some really critical ones but the list is really endless given the wide range of possible adversaries and their motives.

One big advantage to pushing HTTPS everywhere is that we don't have to trust people to be able to correctly predict which read only content is sensitive.

I do wish browsers handled expired certs for longstanding sites in a way that was clearer to nontechnical people. We should have the ability to look at a project like the Internet Archive to know the history of a site in terms of both content and the certs it was served under.

HTTPS is wonderful because it offers a guarantee that the data isn’t tampered with (except with corporate root CAs, but that is fuckery).

But LE doesn't remove the compatability challenges. If you needed to ship a device today that would sit in a box for 10 years and then get online and get an update via https, that's really hard to do. TLS protocols sometimes get discouraged, and CA changes happen, etc.

I'm a web programmer and I have no idea how the law enforcement could in any way help. Nor do I want them to. The idea that I have to cooperate with law enforcement to put a site online is absurd.

See "Tech support scams" on YouTube to see what's being done today. We're talking about billion-dollar crime organizations.

You're updating the firmware on a server. The firmware is signed, so the attacker cannot outright put their own firmware on your system. The version you're using currently is secure, and the version you want to go to is secure, but there are versions in between that are insecure. All an attacker needs to do is modify the DNS and http stream to feed the firmware with an RCE to you, and then they can directly take over your server.

Can you provide a source, or even just reasoning, to why this would be true? In my experience, MiTM is a common enough attack vector used by criminals.

Sorry, but this argument doesn't hold water

And this coming from someone who supports systems still running NT 4

I have fallback rules enabled on all of my domains - TLS 1.3 is preferred, but older editions will be supported if the need arises (1.2, 1.1, and 1.0 (on a single domain))

We could return back to IPsec, or tunnel everything under https as a more modern version of IPsec, but those solutions are all disliked depending on who you ask.

There are other solutions such as MTA-STS and DNS-over-HTTP but the end-to-end validation of DNSSEC is pretty powerful.

It does create a lot of operational risk, as you’ve discovered. It also checks a box if you’re building a system for the US Federal .gov.

tptacek has written about this at length on this site and other places.

Just comment on HN and they'll crawl out of the woodwork.

I registered a domain at Google Domains.

Then I configured the domain at CloudFlare.

At first it worked OK then I started getting SERVFAIL.

I found the problem was there was still DNSSEC configuration set up at Google Domains. I deleted that and everything worked OK.

Cloudflare was not at fault in my case.

By paying for the cheapest plan, or any plan at all for that matter.

Cloudflare's kind policy of zero markup on domain registrations on a free plan is remarkably generous. OK, sure, the traffic data has an obvious value to them, but maybe the support environment could improve with, I dunno, a tiny 5% markup.

If this was that important then you should not have used the free plan.

There is by users' own hands no way out of domain registration issues like these, sooner than 30-45 days when the domain can be transferred once again. Those who decide to offer registrar services, even for free, must hold some liability towards the users and the ecosystem and offer some support to make sure their product actually works.

Its frankly, disgusting.

Recently I had to reach out to @CloudflareSupport on Twitter to get my several day old report of bad Warp+ routing on the forums looked at. It was eventually fixed but it was done in silence and really wasn't something that should've happened in the first place. Nor has there been a followup report on what went wrong.

It’s weird; If I was affected by a bug on Cloudflare, my first instinct would not be to start giving them money in order to be allowed to inform them about it.

We're on a pro plan and have had an outstanding support ticket since March 22nd. With the last cloudflare response being 19 days ago.

I can't seem to get cloudflare to talk directly to backblaze (it's a domain mapping issue) and playing the middle-man in a back and forth between cloudflare and backblaze support seems to be recipe for not getting things resolved promptly.

I know it's covid times and organizations may be short staffed but compare this to cloud66 support who implemented a whole code update to support a special edge case for a non-paying customer within 48 hours. That makes an almost 2 month old unresolved ticket seem a bit tired.

@jgrahamc I'll email you ticket details in case you'd like to take a look.

I've had nothing but problems with them personally

I know some people swear by them ... I'm in the "swear at them" camp

DNSSEC signed is basically just that the TLD servers has a DS record listed for the domain. In order to remove dnssec you remove the DS record. This can be easy or hard depending on the interface that the TLD, but in theory very simple.

The reason why its recommended to remove dnssec before transfer is to allow caches to timeout with the old DS record to expire. Some TLD also automatically remove DS when you do a transfer and a name server change, as it is a rather clear signal that the old key won't be useful. There is however some exciting new technology called multi-signer which is intended to resolve this problem in the future.

It's still pretty early days for DNSSEC, if you're going to use it it's worthwhile to know a lot about it. Just look at the several Slack outages caused by their attempts to implement it. Eventually the tooling will catch up, and registrars will all give you warnings about moving DNS and registration and the importance of syncing up your keys but we just aren't there yet.