I don't know if others have experienced it, but I run a monitoring server from Hetzner and have daily issues with IPv6 latency and packet loss (edit: Finland DC).
I monitor 3 other IPv6 locations, the monitoring server will very randomly throw alerts, and only from Hetzner. Yet, when I opened a ticket, I was told it was the fault of the other providers, despite the mtr traces showing otherwise, and not having issues outside Hetzner.
Hopefully more IPv6 users means that I won't be the only one impacted by those networking issues. I find IPv6 useful for servers that are not public-facing. They are firewalled of course, but it also means I can access them directly from home without hops or VPN (my home having a static IPv6 address).
I wouldn't use their Finland DC for anything serious. Peering isn't good and they seem to route a lot of the traffic via their Germany network. Both DCs in Germany are way better.
> Stuff like that always makes me wonder how much of it is down to the NSA being hooked straight into DE-CIX
It has been long established in the datacenter industry that if you build a building in $wierdplace, it is far more beneficial to your bottom line to haul your own traffic to a major exchange point than to pay to have each carrier come to you.
Further, everyone needs to stop worrying about the NSA. Yes they tap cables, no they don't care about what you ate for lunch. Dozens of countries monitor all traffic that crosses their borders, China does the same shit on all China Telecom owned fiber worldwide, other countries probably have fiber tap subs now, corporations will do it too if it makes them money. Encrypt your traffic, deploy TLS on sites you control, and stop trying to frame run of the mill networking decisions or any little issue you have on the internet on the hidden hand of espionage trying to steal your My Little Pony NFTs.
> It has been long established in the datacenter industry that if you build a building in $wierdplace, it is far more beneficial to your bottom line to haul your own traffic to a major exchange point than to pay to have each carrier come to you.
Unless you have a large enough customer base and can pressure carriers into peering with you in weird places. Deutsche Telekom AG in Germany does not peer at DE-CIX because they just don't want to. It's also the reason why YouTube regularly stutters in the evening for DTAG customers.
> Unless you have a large enough customer base and can pressure carriers into peering with you in weird places
You are confusing peering (settlement free interconnect) with interconnection points. If you wanted to build a datacenter in Finland and buy transit to reach the greater internet, it will always be cheaper to buy your own fiber to {London,Amsterdam,Frankfurt,etc} than to pay each carrier to extend their network into Finland.
> Deutsche Telekom AG in Germany does not peer at DE-CIX because they just don't want to.
This is false. DTGC is present at DE-CIX and is pushing between 10-20G of traffic. They may not want to peer with _you_, but they are peering across the exchange.
> It's also the reason why YouTube regularly stutters in the evening for DTAG customers.
You don't know that. Most YouTube traffic is served by Google Global Cache servers and doesn't transit the internet or public exchanges. DTAG may not have enough caches or capacity provisioned, but that is a decision they have made about their customers experience. Google will happily give them more cache nodes if they ask.
If you don't like DTAG's peering policies, go work there and change them. But it is completely irrelevant to the topic at hand, which is physical interconnections between datacenters and carriers.
Can somebody explain the government's defense in that case?
"The court said DE-CIX could not cite article 10 of Germany’s Basic Law, which guarantees the privacy of communications, because the company was not directly affected by the BND operations."
I honestly don't understand how an argument like that holds water.
I assume this is common knowledge, but I personally find Hetzner to be exceptionally good.
It's always been good to me when I was using their European data centers, but that was always a bit of a bummer because of latency. Now that they have a DC in the US, I just can't think of a single good reason to use other cloud providers for smaller deployments. They're pretty much the best bang for your buck you can get anywhere.
Sadly i have quite opposite experience.
Back in the day hetzner was my first dedicated server provider (after i grew up from shared hosting).
I had some issues with HDDs and they replaced faulty one with another that failed shortly after, then another heavily used hdd.
I migrated to OVH then.
After some years i bought another server from them and it was okay, apart from couple of downtimes.
Some time ago i got interested in k8s and decided to setup personal cluster. So i tried to register account with them again, because when you don't have any services they delete your account completely (really?)
I failed. They silently suspended my account after i made payment for verification. I tried to register again, they asked for scan of my passport, i sent it, and they suspended my account again. No replies from support regarding that issue too.
Funny, because i went through validation process 2 times already (when i rented dedicated servers from them) and now i can't create account to use their cloud offering (and dedicated servers too)
The account deletion when not having any services is a result of their privacy policy. Most European companies should do that but I think it's rare in practice. What they didn't do very well is that it's of course better to email you before deleting.
The HDD thing is (was?) definitely an issue with Hetzner. I had the same experience some years ago. Best option was to get hardware raid and SAS disks, those were datacenter quality instead of consumer and worked great.
Hetzner's backend billing system broke. They couldn't charge my credit card no matter how hard they tried. Eventually it ended up as a billing dispute, I paid the balance and terminated the servers and moved them to a different account that magically worked with the same card.
Got a letter in the mail a month ago that they had sent the broken account to a collections company in the US.
I "benchmarked" the network across a few big VPS providers, they all seem to do 400Mpbs+ within the continent and 100-200Mpbs when you go across the ocean (iperf3, I assume you also meant heavy sustained traffic). It also aligns with what the creator of PHP found out: https://toys.lerdorf.com/low-cost-vps-testing. Did you find a provider where the network would allow you to go 300Mbps+ across the ocean on a cheap VPS?
Re: CPU, if you were comparing them with the VMs from "the big 3", you should look under the "Dedicated vCPU" tab: https://www.hetzner.com/de/cloud
My 7 Euro hetzner cloud instance in Falkenstein easily does about 5Gbit (inter-europe).
From my experience, at Hetzner you are very rarely limited by their network. Usually only by poor peering from transits. Peerings with all the big names are great tho, so no reason to complain for me.
And from what I know from someone near Hetzner, they don't cheap out on peering at all.
Yeah, heavy sustained, and DO came up on top (even your link says so). Admittedly, I haven't tested across the pond. I've tested dcpu ones only AND only for my load (node.js with a worker). That's why "benchmarked" is in quotes. :)
Linode/Vultr/DigitalOcean/Hetzner/Terrahost
I can easily imagine cost savings on Hetzner could benefit certain loads, of course. It's always "depends".
I don't see Hetzner dedicated vCPU's on there? I did mention "dcpu" ones. But thank you for your benchmark, it's very useful for shared-cpu comparison.
How did I come to it? Measuring performance of my node.js service I need to run on it, iperf, speedtest.
Anyone know why dedicated is just not a thing in the US? I looked for a Hetzner alternative in the US and it's just crazy to me that it doesn't exist.
I think Hetzner's dedicated servers product started from the consumer demand for Counter-Strike servers, maybe it's because Counter-Strike wasn't as popular in the US? Not 100% that's why Hetzner has been succesful with dedicated servers but I don't have another explanation.
It's not as common, I agree, but I work for a data center company that does almost exclusively bare metal servers and colocation. We have a VPS part of our company but it's a different brand name and you won't find it on the main company's website.
I still have Hetzner for a dedicated server in Europe, and it's fine. I've had it for several years now with no issues.
Before getting this job, I noticed that most dedicated servers in America are from smaller companies that re-sell their dedicated servers or colocated servers from companies like my work's or other data centers (who mainly specialize in colocation services instead of selling their own hardware.
I suppose in America, the main people companies target are for simple cheap VPS servers or "cloud scaling" services. You can also make a lot more imo by stuffing as many customers onto 1 server instead of dedicated hardware for each customer. Also a lot less management and support needed. At work 99% of our maintenance and support is for individual dedicated servers. The VPS side of the company requires very little intervention on our end.
It’s definitely a thing, every major metro has tons of data centers doing it to various degrees of success. For the most part it’s for small/medium folks, as larger buy and staff their own equipment, and many smaller do VPS or cloud for various reasons.
Dedicated is starting to make more sense as cloud prices are kinda nuts + tooling is better now, so ‘roll your own cloud’ is becoming more feasible.
It just doesn’t get as much press as the big cloud offerings, and most of them are relatively local players specializing in their region (and associated network/property). OVH has several large DCs selling dedicated hardware for rent here in the US.
Do you have an example of being able to rent a single bare metal node, with the cost of the rack already priced in? I've never seen it, especially not like Hetzner where it can cost as little as $50/mo all inclusive for a full server.
I'm not sure what it is that you're not finding. I can just search 'dedicated server' and the first page has several examples of places that will rent you a single bare metal server in the US. There are dozens and dozens of options.
Leaseweb, they're a European company but with several US datacenter locations. They offer monthly priced dedicated servers, but are more expensive than Hetzner.
https://opennebula.io is worth looking at for that sort of thing (not that it's new). You don't have to do it that way, but its "edge" support for simple provisioning on bare metal providers doesn't include Hetzner; I think it assumes you can get instances on demand. That sort of solution isn't complex or expensive enough for my site, and doubtless others, though. Especially when you just want compute, I can't see the point in the pain (which surprised us) and expense of AWS et al.
I've started using Hetzner's Virginia cloud offering. So far their AMD processors are matching DigitalOcean's comparable product and I'm getting a lot more for my money. I've seen zero evidence of supposed crappy CPUs in Hetzner's cloud.
Also: Get one of their managed webhosting packages (starting from 1,90 €/month) and you'll get a domain with it. You can't get your own mail hosted for much cheaper.
That's cheaper indeed. I would hold against that the 24 €/year from Hetzner get you 10 GB of space for whatever. Personal website, small NextCloud, something like that. I think it's a good deal with a well known hoster.
Me too! You're in for a lot of pain, especially in the beginning, when trying to host your own mail server. Big players just dropping mails (Microsoft), local authorities refusing your mails (even on the postmaster inbox) and if your mail goes through it might still go straight to the spam folder. Granted, I'm using an .xyz domain, but I'm not sure how big of a role this really plays. Interestingly I didn't have problems with Google's mail servers so far.
I've not yet found this to be an issue with personal mails. If you're planning on setting up whatever system that needs to send _lots_ of mails, something else might be better indeed.
https://github.com/StarpTech/k-andy : "Zero friction Kubernetes stack on Hetzner Cloud", basically specialized k3s with some pieces customized for Hetzner, and automatic deployment on Hetzner cloud instances.
not to mention free DDoS protection. In Amazon I read in one hacker news article that it costs them around $6000 for the custom AWS DDoS team. Why people still use AWS? Other than that it is a corporate police. Honestly would like to know.
I always assume (to be clear: this has not happened to me) that Hetzner's margins being what they are, any customer that causes an undue support burden (regardless of culpability) is probably unlikely to remain a customer for very long.
I am extremely cautious of the sources of UGC I host publicly on my Hetzner machines. In addition to the fact that Germany lacks freedom of speech/publication (thus obligating a German organization like Hetzner to censor content that is legal to publish in most places, but not Germany), I imagine it wouldn't take many legitimate/normal UGC-related issues (e.g. properly-responded-to-by-the-box-customer DMCA takedowns) to make my customer relationship turn negative ROI for them.
I wouldn't, say, run a social media site open to the public on Hetzner, even if I responded to DMCA and other legally-mandated takedowns in single-digit minutes, 24/7/365. I just can't imagine they'd accept the overhead of such a customer.
That said, it's great for hosting big files that CloudFlare's TOS prohibits (video, podcasts, etc), just as long as you're certain nobody at Hetzner's going to get a call over one of your URLs.
Happy to provide an anecdote here. Had my home searched for user generated content on a Hetzner server around 2014. Some reports were also forwarded to me when the service was still alive. Hetzner provided a signed statement of me responding to abuse reports within a few hours of me asking.
No, you're not allowed to publish gory video games or racist literature[1] in Germany, which is clearly prior restraint. It's not that the US has some crazy absolutist freedom of expression (it does not), it's that Germany simply lacks it.
You can't have "mostly" free expression. It's either abridged or it isn't. Germany censors harmless digital art that the government deems inappropriate for adults to be able to see. It's a classic slippery slope (modern Germans defending their government's censorship and lack of free expression will usually cite Hitler/racist stuff, but that's not all that's banned).
It doesn't really matter what the constitution says, if in practice you don't have those rights. It's sort of like how the 2A in the USA says that the people have the right to keep and bear arms, but I don't suggest attempting to exercise that right in Central Park, because you don't actually have it. Same goes for free expression in Germany.
> Germany is one of the strictest censors of violence among the world’s video game consumers. Due to its history and a cohesive national opinion, the legislature limits content severely, much more severely than the surrounding European nations. This results in international developers choosing not to market to Germany, creating censored titles specifically for the German market, or finding themselves on a list of banned titles illegal to buy or sell.
[1]: there's also no indication that racist publications were responsible for WW2 (versus, say, Hitler himself), making this censorship-for-censorship's sake. Many other countries do not prohibit racist literature and have not committed a holocaust. So, of course, they banned violent video games too, because those don't cause violence either.
Because you asked, even though it's now way off-topic for this thread (and I will not respond further):
Because the text of 2A does not indicate that the right is contingent upon participation in a militia (and indeed 10 USC 246[1] legally defines the US militia as all able-bodied male citizens of ages 17 to 44 inclusive, as well as all female citizens who are members of the National Guard, even if it did), as 2A actually specifies RKBA as a right of the people (not "people of the militia", just "people").
Heh, and where was this interpretation when the Black Panthers were arming themselves in California? Even the NRA supported gun control back then.
> In contrast to the NRA’s rigid opposition to gun control in today’s America, the organization fought alongside the government for stricter gun regulations in the 1960s.
From 1930s to 2008 it was read differently. The words also don't say the mentally ill with violent uncontrollable ideations can't have them, but we constitutionally allow the disallowing of that presumably both out of common sense and because forcing allowing that has nothing to do with a well regulated militia.
Your information on video games is about a decade out of date, there were no video games put on List A or B for the past 2 years at least, and I haven't checked further. The last high profile ban was Wolfenstein, and that was for it's use of the Swastika.
I encourage you to read up on the J.E. Hoover and J. McCarthy on how much the US First Amendment is worth. Or maybe try running a pro ISIS webserver in the US, see how quickly you have the FBI knocking on your door. Let's not even talk about DCMA takedowns etc..
The US ranks lower than Germany in the FH and RWB freedom of the press indices [1], which while not quite the same is highly related to freedom of speech.
The First Amendment is stronger than any equivalent rule in any other modern country that I'm aware of. The fact that it's been undermined repeatedly, both in the past and the present (due to the recent wave of authoritarianism that has been sweeping US politics, which can be seen clearly on HN itself), doesn't have any bearing on its relative ranking - so, it can suck (or just be a little suboptimal), but still be better than everything else.
Moreover, how is DMCA relevant? Copyrighted works are outside the bounds of free speech.
Wait, what? How does the fact that it's undermined not have an impact on how useful a US right is to me?
DMCA, and in general the US legal system are extremely relevant to me as a user. If I have a theoretical right to free speech, but in practice any big US media company could kill it then I'm much better off in another country where maybe the theoretical right is 10% less but I can actually practically enjoy that right.
Those rankings just encode the political outlook of a certain leftist NGO.
In the USA, Neonazi publications are allowed, while in Germany they are illegal. Therefore, the US has a freer press than Germany. Freedom House would disagree, and they are simply wrong.
FYI - not according to the US constitution. It applies to everyone on US soil, and is supposed to bind the federal gov’t in general in how it acts everywhere.
De facto and de jure of course being completely different things.
> FYI - not according to the US constitution. It applies to everyone on US soil, and is supposed to bind the federal gov’t in general in how it acts everywhere.
Sure but that ship sailed years ago. Their constitution is also supposed to guarantee a right to a fair trial but obviously the legions of drone victims didn't get one.
An appropriate German name is one that is first recognised as a proper name. It cannot be associated with evil (e.g. Satan, Lucifer) or deemed religiously insensitive (e.g. Christus or Jesus). A name cannot be a product, brand, surname or a place name. Finally, German names have to indicate the child’s gender and they are not allowed to cross (one exception is Maria, which can be used as a boy’s second name). Neutral names (e.g. Alex, Kim) must be followed by a second name that indicates the child’s gender.
If your definition of freedom of speech means being able to name your kid whatever you want then obviously the US first amendment is not working since most states have rules regarding naming too. For example: https://www.theguardian.com/us-news/2015/apr/11/california-b...
California like several other states bans the use of diacritical marks on official documents. Last year, the state took up a bill that would have allowed diacritical marks, but it stalled out when a $10m price tag was attached
You're right that German naming rules are overbearing, but you'd have trouble naming a baby "X Æ A-Ⅻ" in a lot of places, including many US states, I believe.
> to censor content that is legal to publish in most places, but not Germany
We don't really like if somebody thinks Hitler was kind of a cool guy and that he should have continued "his work". Besides that you can say lots of things in Germany without getting too much trouble.
Naaaaah... The first five articles of the German rule of law are pretty clear. They explicitly forbid to discriminate based on race or alike. As the ideology of the Third Reich is based on this discrimination, "cheering" this ideology is not allowed.
Things have changed in recent years. Germany now has vaguely defined "hate speech" laws, meaning they can get you for any statement they don't like. It's not just about liking Hitler anymore.
It's not the same thing. It has been illegal to insult people or calling for their murder. The new laws in contrast are very vague and give the government much more room for interpretation.
For example, from Wikipedia, "agitation" is not allowed.
Merely disagreeing with a politician can be considered hate speech.
It does not take a lot of effort, either, just using certain words can be sufficient.
Germany now also seems to have the concept of "protected groups" that you can not critizize in any way, which I think is probably new. I am not a lawyer, though.
There are a few worrying cases. One person, for example, got his house raided for calling a politician a dick on Twitter [0]. There are also a few far-reaching laws, such as the NetzDG [1], which is being pretty harshly criticized. That being said, Germany has never been as pro-free speech as the US (holocaust denial, for example, is illegal for a long time already) and you can still state your opinion pretty openly, as long as you're not an extremist.
What they enforce is Nazi shit and pro child abuse publications being banned. Really not objectionable at all. If you must be angry, there are many countries that are significantly worse than this.
That was literally 12 years ago and is not in effect and doesn't even have a German wikipedia page and the federal court would have nicked. Ist alles von der Kunstfreiheit gedeckt!
You need to be more specific then, which law do you mean? Because the "Straftaten gegen die öffentliche Ordnung" is a whole section of the StGB which covers all sorts of things ranging from the prohibition of running an illegal online trading platform, to trespassing or leaving an accident. The closest is the Paragraph 131 which prohibits showing violence for the purpose of glorifying violence or to degrade the "Menschenwürde" (not sure what a good translation is).
Science disagrees on you with the harmless part. It is shown that violent video games desensitise people with regards to violence, just as violent movies do except there is a categorical difference interactiveness.
I don't like that fact either but to ignore facts because they don't mesh with my politics is bad.
? That article says there was an attempt to introduce a law banning violent video games. That attempt never was implemented, so what is this link supposed to prove?
So I take it your datacenter has never burned down?
Sorry, had to poke fun a little bit, obviously there's always a chance of having a bad experience. I've only had to interact with Hetzner support once, and it was positive, we determined that a consumer grade CPU was aggressively going into some sleep mode the Linux kernel wasn't waking up from, and the Hetzner support guy agreed that was the problem and determined it was possible to disable that feature in the BIOS, and went ahead and did that for us.
And of course the fact that that was necessary was on us for running our production on their consumer grade CPUs.
I believe Hetzner and maybe also OVH have a much bigger role to play in the deployments of the future. The big cloud players are overplaying their hands, and it's becoming more and more attractive to run on bare metal as devops tooling improves.
Fun story: Hetzner lost a colocation due to overheating like almost 20 years ago. Everyone has to learn over time that eventually adds up to deep domain knowledge. Both Hetzner and OVH have built custom solutions for buildings, climate and energy monitoring.
Not only that, but for some deployments it is actually preferable to limit the amount of money you can pay. In other words, better for the service to be down than for the company or person to be bankrupt. Last I checked, that simply wasn't possible with AWS (no, alerts are not the same thing).
Imo that's a must have feature for any cloud hosting provider - I don't want to bankrupt my company/myself because I misconfigure something or write a bug.
I recall there being discussion about it on HN before and a lot of people being confused as to why anyone needs it though.
I recently moved from OVH to Hetzner as I could get better hardware for the same price.
No issues so far, but OVH's network seems to be better, especially for people in other continents.
Edit to add:
- I'm using their Germany DCs, not the Finland one (which is cheaper but peering is worse).
- Using a CDN does improve the latency/speed for users outside Europe, but it still influences performance as CDN exit point often connect directly to the origin to fetch uncached content.
- I lost a VPS with OVH's DC fire, but I had backups somewhere else and fixed it quickly. A good thing about the lower prices is that I can have backups on multiple services (also cheap - Backblaze B2, for example) and still save money compared to AWS, Google Cloud, etc.
Currently have 10+ dedicated boxes at Hetzner and I'm taking time to write this comment because I like them that much. I only contacted support a few times during my 10+ years there but it was immediate response and to the point. I remember waiting 24h+ for OVH support with my service down (it was 5+ years ago though) and for Hetzner it was always in minutes.
Can you elaborate why do you think they are unskilled?
I cannot share that experience with the hetzner support. We currently do have some weird problems with an hcloud-dedicated joined vlan and the network topology and support is a bit weird there. I'm hoping that'll improve, because then hetzner might become a prod leg for us.
But besides that, both the cloud and dedicated support are very good. Last 10 - 20 HDDs we needed swapped were swapped in under 30 minutes each, and in some of the more complex issues, they were able to guide our engineer wherever they needed to be quickly. I've dealt with much, much worse hosters at multiples of that price.
We've been using Hetzner since the last article about their VA service appeared last month on HN and are truly impressed in all respects with the performance and cost. We plan to move all our servers over to Hetzner from AWS, however continue using S3 (We didn't see a comparable service on Hetzner for S3)
I also have 2 servers there, one virtual with FreeBSD and one root server as Linux hypervisor. I run the guests from a tmux session because I was too lazy to create any systemd job for this. The guest works like this already more reliably than an instance from more well-known providers.
Of course for larger deployments you'd have to take care of fail-over and all this, so it's not really an option unless you are up for setting this up all by yourself.
> Their vlan stuff being 1400 mtu is also annoying.
does this suggest that their networking gear is also dirt cheap or is this just an artifact of legacy compat and/or not wanting to wory about jumboframes?
Note that bandwidth is included for servers with gigabit uplink - for 10gb uplink "only" 30TB/month is included. How much is just 5TB/month egress on Amazon?
I've only had 1 HDD failure since 2013 which their support was quick to resolve. I haven't noticed any network interruptions myself personally during that time.
I've only had them for a few years but I've yet to experience any issues with their great value (e.g 2TB €9.90 /mo) storage box servers. https://www.hetzner.com/storage/storage-box
Only issue I have with them is the latency of their Germany DC's from the US, if they end up offering dedicated servers in a US DC I'll be moving over my existing Hetzner and AWS (non RDS linked) App servers over.
I am running dedicated servers on Hetzner since 2008 (max I had was 10 root servers at a time). Outages were quite common back in a day, both sudden server reboots and HDD failures. However, for last 5+ years I haven't got any single outage. Support is always very quick to react and you could usually get HDD replaced same day.
While having a long uptime sounds cool it is a signal that you don't patch that often. Maybe you patch your other stuff, but I would bet on even odds that you don't. So then that is the rest of the stack, eg systemd which has some mega flaws IIRC.
You're leaving yourself open to having something exploited. Have a look at your ssh logs where "people" are constantly trying to get in.
As someone who does full patches every couple of weeks on my servers and reboots every several months, I agree, however there's stuff that can live patch the kernel these days like kernelcare and livepatch by Canonical and more.
Another reason though to reboot every so often is for the server to do filesystem checks on the root partition(s).
I understand, and my uptime was just to show how reliable Hetzner has been. By “talk me into it” I meant please point out a real kernel security flaw that be exploited without already having access to the system. There very well might be some! I’m not well up on all of this.
Yes, I check my logs and see the constant stream of breakin attempts. Basic security precautions seem to keep them out.
Unfortunately, the firewall offering that Hetzner provides for their dedicated servers is IPv4 only.
So, if you're using software on the server which mucks around with firewall rules (eg using OS provided firewall on the server isn't good enough), then you're sad out of luck.
And their current IPv4 firewall has a 10 rule limit per server, which can't be raised. Mind boggling. :(
I've asked Hetzner if they have any plans to extend their firewall to include IPv6 support, or raise the # of firewall rules, but they have no plans to at this stage. :( :( :(
For this reason I've setup a OpnSense VM on my dedicated Hetzner server where all inbound and outbound IPv4/IPv6 traffic has to go through, it acts as a gateway for the host itself and my other VMs. OpnSense itself is a pretty powerful firewall with tons of other features.
Of course you'll lose access to your server if the OpnSense VM breaks or doesn't boot up for whatever reasons after an update or so, but after 2 years I haven't had any problems. But in case something goes wrong Hetzner offers some nice recovery options, even if you don't have internet access to you server you can access your volumes in some kind of VM and get access to it via a VNC like interface (I had to use this feature a few times during the initial setup which consisted of a lot of trial and error I locked myself out a few times).
I wouldn't run this setup for anything mission critical of course, it's way too hacky and an official firewall solution would be better, but for my personal purposes as a "home lab" like setup it works perfectly fine so far.
E.g. `docker run -p 8080:80 nginx` will expose the container's port 80 as port 8080 on the host. That port will be open whether or not the host has a firewall configured to block 8080.
You can do `docker run -p 127.0.0.1:8080:80 nginx` to only have the port on the host accessible on the loopback interface (for example if you have a reverse proxy on the host, proxying to 127.0.0.1:8080).
Not quite. It's easier to define one set of rules for the entire server group (Projects in Hetzner terminology) and forget about it than to manage OS firewalls individually.
What software that anyone actually uses does this, except for docker (which has well documented ways of using it properly that are tragically not the default)?
> well documented ways of using it properly that are tragically not the default
Docker is the stand out case from my point of view, and the ways it breaks networking seems to randomly change from version to version without warning nor consistency.
What are the well documented ways of using it properly, that avoid this weird behaviour from Docker?
Exposing ports breaks every security model in your firewall, which is why you should expose ports explicitly to localhost and if you want to expose them to the outside world you use a routing application (e.g. haproxy). I think that is the most obvious one. The fact that it messes with your iptables rules in a way that is frankly not very transparent is something that cannot be fixed, except by not using docker :)
> Exposing ports breaks every security model in your firewall
Of course. That's why we do only bind them to localhost, and tend to use nginx to proxy from that to the outside world (for web apps anyway).
That being said, I've still seen Docker do dumb stuff with port mapping, and seen it change behaviour between "minor" version updates. Though that was a while (few years) ago now.
> ... except by not using docker
Kind of stuck with it for now, and having to work around it's crappiness. ;)
I always think that too many web servers have IPv4 addresses. People don't seem to realise that CDNs - which everyone surely runs behind - will happily proxy IPv4 traffic to IPv6, so you don't need an IPv4 address to serve web traffic - only your CDN does.
Sure, but IPv4 just works. IPv6 mostly works, but isn't universally supported, is the less tested configuration, and "disable IPv6" seems to still be one of the best solutions for mysterious network problems. It just doesn't make sense to use anything other than IPv4 as long as you get a free IPv4 address with every server.
Which is why I'm very happy over this move by Hetzner. More monetary incentive to move away from IPv4 is exactly what we need to break the cycle of "nobody uses IPv6, so nothing supports it, so nobody uses it"
> and "disable IPv6" seems to still be one of the best solutions for mysterious network problems
It's not really a solution to the problem, it's usually just ignoring the problem and hiding the symptoms.
I'm surprised Hetzner is the first to do this, it's an obvious move with the sharp rise of IPv4 addresses. Most companies don't need IPv4 anyway, because their infrastructure usually ends up at a caching proxy or CDN regardless. Your backend API servers will usually also be talked to by other servers, which usually also run from a place with widespread IPv6 addresses.
I can see a (bleak) future where consumers are all om CG-NAT and everything but the frontend is running IPv6 as a cost cutting measure.
Most of the world works by hiding symptoms until something forces the issue. Often it never happens, so we’re all good (as it were). Sometimes it doesn’t, then everyone starts pointing fingers.
Many people are pretty adept at making sure the fingers don’t point at them.
Ah, I got excited and thought I could order a dedicated server - but choose if I wanted them to be on the same isle/shelf or on a different one! That would be cool! (This is still cool too, though!)
We ordered a new root server a few days before this, but are trying to primarily use IPv6 anyway. However the more I try to set things up, the more painful it becomes, with services like Github not being entirely available on IPv6, various ISPs not supporting IPv6 at all, so service proxying is needed etc.
Is there any best-practice for IPv6 VM host setup? I found the IPv6 First Guide [0], but I'm not entirely happy with the bridged networking to VMs being used.
It'll be implemented for cloud servers as well in the near future according to the official post on the forum.
> In addition to the continuous optimization of this solution, we are also already working on being able to offer cloud servers optionally with IPv6 only. Here we still ask you for a little patience!
If you have a few machines that are connected together, saving 1.70€ on each kinda means you could get a floating IP, or put towards a vswitch or whatever
You don't need a HTTP proxy, you could make use of existing 6to4 technologies. Cloud providers can probably offer those for free at reduced speed as part of their IPv6 package. There are also public 6to4 routers available today, ur I wouldn't trust my company's data to flow through those.
Well, if you have virtuals running on your dedicated server, you're probably going to need more IPs anyway and IPv4 subnets now cost a small fortune on Hetzner, while IPv6 is free and you get a whole /64 subnet.
> "We will continue to improve this new solution and are already working on an IPv6 only solution for cloud servers, too."
I'm eagerly waiting especially for this! The cloud servers are pretty cheap, but costs for IPv4 addresses make a significant part of the monthly cost. The Hetzner cloud server would be much more interesting if they weren't each tied to a public IPv4 address.
I just created an account and it got deactivated because of "inconsistencies". Wrote to support and they, again, told me it is because of inconsistencies, but will not tell me what those are.
They really do not give a f** about potential costumers.
I went with Server4you and they happily welcome me everything I book a new server.
The announcement doesn't indicate why I would want this. Is it for politically motivated people who want to help push IPv6 forward? Is it to simplify configuration?
I had to click through to the FAQ to read about additional cost for IPv4, but there the difference isn't specified, so it led to more questions, but I gave up.
Because if you want to make something there will be customers in the US.
And if you are making something interesting that utilizes the internets USP, it will have real-time communication between your customers.
And in the US if users from the east coast connect to something with the same latency as users from the west coast that evens out the advantages.
If you are making a static homepage then of course it doesn't matter because what you are making could also be a book or even a stone tablet.
The content always comes from the older medium until the new medium figures out it's own content at which point the old medium dies. See opera, theater, radio, television, youtube, twitch, etc. etc.
The final medium is the open 3D action MMO, be it in VR or not.
5 years ago i decided to never work on something that could not be sold globally, it's a good decision because it leans into the future.
You need one server in each region, but centrally in each region.
In europe there are maybe 20 actors for each of the many central countries (Germany, Belgium and Switzerland)... in the US there are only 2 (that I could find that allows me to remotely get a VPS): GCP in Omaha and IONOS in Kansas City!!!
AWS only has regions on the coasts!
You also only use these low latency "edge" servers for real-time (encrypted if you have private data) and keep the database and patch data on your own physical servers as they are not latency sensitive!
Clouds have very expensive networking and disk costs compared to buying hardware and hosting it on your home fiber! But you cannot live on all 3 major continents (EU, US, and Asia)!!!
Eventually people will exchange VPS on their home fiber, if you live in central US or Asia and want to exchange a VPS (preferably redundant 2x locations and on fiber with static IP and lead-acid power backup) for home hosted VPS in Sweden let me know!
As for building something on the internet for only people living in the UK that is meaningless.
Our customers in Scandinavia are asking us to stop using American cloud providers for their sensitive data, even though the datacentres are Scandinavia.
Central US in Iowa / North Central US in Illinois / South Central US in Texas / West Central US in Wyoming. (Plus all the normal locations on the coasts.)
(That's currently more Central US data centers than GCP which just has an Iowa and a Salt Lake City data center today. If you are keeping count.)
It's about having multiple continuous ranges of addresses.
Think of it in IPv4 terms, it's like having the range 192.168.0.0 to 192.168.0.255 (192.168.0.0/24) assigned to your host. 256 addresses should be plenty of addresses, but you can't cleanly segment them into multiple ranges, like you could with 192.168.0.0/16: because you can have 192.168.0.0/24, 192.168.1.0/24, 192.168.2.0/24.
By having multiple, complete blocks of /24, you can easily assign them to different classes of IP interfaces on your host.
Nothing in IPv6 says you have to stop dividing at the /64 level.
There has been some hardware that takes a bit of a performance hit when doing route lookups that are longer than /64 in the past, but if you're doing this all in software on an end host, that's not an issue.
Go ahead and divide up that /64 to smaller blocks for your classification purposes, you'll still have plenty.
The client obtains the network prefix from the RA, and then the client tries to generate unique host address.
""
The IPv6 stateless autoconfiguration mechanism requires no manual
configuration of hosts, minimal (if any) configuration of routers,
and no additional servers. The stateless mechanism allows a host to
generate its own addresses using a combination of locally available
information and information advertised by routers. Routers advertise
prefixes that identify the subnet(s) associated with a link, while
hosts generate an "interface identifier" that uniquely identifies an
interface on a subnet. An address is formed by combining the two.
In the absence of routers, a host can only generate link-local
addresses. However, link-local addresses are sufficient for allowing
communication among nodes attached to the same link.
""
Basically it explains that SLAAC RFC itself does not define the /64 limitation, but other RFCs that are relevant to network operation do.
"""
The addressing architecture [RFC4291] [RFC7136] sets the IID length
at 64 bits for all unicast addresses and therefore for all media
supporting SLAAC. An immediate effect of fixing the IID length at 64
bits is, of course, that it fixes the subnet prefix length also at 64
bits, regardless of the aggregate prefix assigned to the site
concerned, which in accordance with [RFC6177] should be /56 or
shorter.
"""
You can actually request a /56 per-server for a one-time fee of, IIRC, 60 EUR or so. Just talk to support since it's not listed anywhere in their docs for some reason.
Typically, a /64 in IPv6 is one subnet. You're not really supposed to subdivide networks beyond that point. The first 64 bits identify the network, the last 64 bits identify the host. Under this design, if you only assign a /64 per server, that would mean you couldn't have multiple networks on that VM (which you'd want for something like Docker.)
However, this principle isn't really enforced on a technical level. Only some standards such as SLAAC actually require the network to be a /64, and since SLAAC isn't really relevant for servers, most cloud providers have been relatively stingy with their IPv6 allocations - at least compared to what RFCs actually recommend. (Which would probably be a /48 per customer, per region.)
I appreciate the push to expand address allocations a bit, so that providers aren't charging a monthly fee for every /128 and the like. But /64 seems like ridiculous overkill. Even for a premises network connected to an ISP, the optimal thing to do is to continue NATting to hide information about your network. Privacy wise I'd rather have 2^16 separate /128's scattered throughout a provider's address space, than a single /64, which will inevitably be treated as a single address by the surveillance industry. Ultimately discrete public addresses are really only needed for services, and the services I want to share are few and enumerable. At this point I do more work making sure individual nodes aren't fully connected (eg Internet of Shit) than making services reachable.
The idea behind the /64 minimum was to make auto-configuration easier - you could just stick the layer 2 ID (i.e. the MAC address) in the second half and not have to worry about collisions or stateful assignment system like DHCP. Remember that IPv6 was designed in the mid-90's, so both of these decisions seem silly now - using the MAC would be a serious privacy issue (modern operating systems use a random ID which changes frequently) and DHCP is very mature.
Yeah I was going to touch upon that. I understand where they were coming from design wise (apenwarr's post is fantastic for that - https://apenwarr.ca/log/20170810). Privacy extensions mainly seems like doubling down on an unworkable idea. I can see the hypothetical benefit on a shared network, where say a coffee shop has a /64 and you can have every single app looking like a separate device on that network to an outside observer. I just foresee the inevitable future where IP surveillance databases contain information like "this /48 hands out /64 to each end user, so treat it as one entity", rendering those extra bits as a mere liability to be mitigated.
Sure, but as the ancestor comment mentions, SLAAC has no relevance for the use-cases people in this thread want more then 64-bits for (e.g. Docker sub-networks). Since your Docker containers don't have "MAC addresses", there's no reason that you need to use 64 bits specifically to configure them. Your container runtime is perfectly positioned to assign IP addresses and subnets however it chooses to. Assigning a /72 to each container is perfectly fine.
... giving each VM one IP out of the /64 is not a "dirty hack". I guess if you want to do complex networks between the VMs it'd help a bit? Still, /64 is IMHO a good default for single machines.
Traffic for any of the addresses in the /64 arrives at your server’s "external" network interface. Is there any non-hacky way to forward this traffic to an internal virtual network?
You just create a route for it, same as you would with any size network. Linux allows you to create routes for IPv6 prefixes longer than /64 without any issue. On AWS I have a VM with a /80 routed to it, which I have divided into multiple /96's internally. It works fine.
I know you are being funny. However, it does make me think hmmm... Is there any advantage/disadvantage to not handing out something like a /120? Or is there something else at play like in the way auto discovery is working? I am not familiar enough with it to say.
I have a /64 from my ISP and I want to run a few VLANs on my home network but I can't subdivide the /64 any further using my business class (TP-Link omada) router's controller. Maybe there are similar limitations in place here?
If an IPv6 spam filter is working on a per-address basis, it's never going to work. The smallest allocation you can get from a RIR is a /48. Even residential ISPs give at least a /64. You could use a different address for every email and never run out.
No, it's not enough because you end up needing Proxy NDP for your traffic to reach other subnets smaller than a /64 (e.g. a /80) if you have the /64 on your wan interface and carve it out. Normally, you'd have a /64 on your wan, then another /64 for your containers or multiple /64s for different container deployments or virtual machines. Then traffic would route properly between your networks with ipv6 forwarding enabled.
> is more than enough for all of your VMs/containers on the box
> you end up needing Proxy NDP for your traffic to reach other subnets smaller than a /64 (e.g. a /80) if you have the /64 on your wan interface and carve it out
If you're using proxy NDP then you don't have "more than enough" IP space for what you're doing. You have none and you're hacking up a workaround -- one that's inevitably going to have people coming away thinking "v6 is hard" rather than putting the blame where it deserves to be.
I monitor 3 other IPv6 locations, the monitoring server will very randomly throw alerts, and only from Hetzner. Yet, when I opened a ticket, I was told it was the fault of the other providers, despite the mtr traces showing otherwise, and not having issues outside Hetzner.
Hopefully more IPv6 users means that I won't be the only one impacted by those networking issues. I find IPv6 useful for servers that are not public-facing. They are firewalled of course, but it also means I can access them directly from home without hops or VPN (my home having a static IPv6 address).