Hacker News new | past | comments | ask | show | jobs | submit login
Ksplice: Automatic Rebootless Kernel Updates (2008) [pdf] (ksplice.com)
64 points by chicken_lady on Aug 15, 2014 | hide | past | favorite | 32 comments



The tragedy (from wikipedia):

On 21 July 2011, Oracle announced they acquired Ksplice, Inc. At the time the company was acquired, Ksplice, Inc. claimed to have over 700 companies using the service to protect over 100,000 servers. While the service had been available for multiple Linux distributions, it was stated at the time of acquisition that "Oracle believes it will be the only enterprise Linux provider that can offer zero downtime updates." More explicitly, "Oracle does not plan to support the use of Ksplice technology with Red Hat Enterprise Linux."[5] Existing legacy customers continue to be supported by Ksplice, but no new customers are being accepted for other platforms.[11]

https://en.wikipedia.org/wiki/Ksplice


Not exactly the case: Ubuntu, Fedora and Debian are supported for free, and RHEL for a fee:

Ksplice Uptrack is available for Oracle Linux, free of charge, for Oracle Linux customers with a Premier support subscription. Additionally, anyone can use Ksplice Uptrack for free on Ubuntu Desktop and Fedora.

http://www.ksplice.com/pricing


Thing is, while Oracle Linux is pretty much rebranded RHEL last time I checked, the KSplice commercial license won't let you use it with RHEL, and that won't be supported.


It is, you just have to buy an Oracle Premier Support License for your machine(s) and switch them from Red Hat to Oracle Linux (which requires switching yum repos, removing some packages, and installing a few others), but does not require a reboot - which means you can keep using the official Red Hat kernel (until you do reboot).

Disclaimer: I work there.


You forgot the part where you give Oracle your balls on a silver platter so they can hold them for you. We've all seen what Oracle likes to do with nice things.


Fairly sure Oracle Linux was at least in its initial incarnation a rebranded CentOS, which itself is of course a rebranded RHEL.


That "pricing" page is strangely devoid of pricing information.

EDIT: so, how much does it cost?


For Ubuntu, Fedora and Debian, nothing. For Oracle Linux, you need an Oracle support contract. Those don't come with a standard price tag.

Unfortunately, the website hasn't been fully updated since the acquisition, so there is some confusing verbiage. It's on my list (near the top, no less; sorry about the confusion).


I'm interested in CentOS.


I won't mince words, the cost of Ksplice these days is much higher than pre-acquisition, however you can get Ksplice for Enterprise Linux 5, 6 and 7, with the Redhat kernel without rebooting, as well as the Red Hat Compatible Kernel that Oracle recompiles, as well as Unbreakable Enterprise Kernel (UEK) r1, r2 and r3.

https://www.ksplice.com/uptrack/supported-kernels


Note that since Oracle basically closed ksplice, RedHat has developed kpatch and SuSE has developed kgraft, which are both similar ways of doing the same kind of thing, and there is some discussion going on about whether they can be merged or a better general purpose kernel patching method developed: http://wiki.linuxplumbersconf.org/2014:live_kernel_patching


The Uptrack service was obviously proprietary, but the Ksplice software itself (everything to actually make and install the patches) was open source. There are plenty of mirrors on github: https://github.com/linux-kernel-live-patching/ksplice

But there's also kgraft and kpatch.



Red Hat is working on ksplice and SuSE on kGraft. And unlike Ksplice they try to merge it into the mainline kernel.


dont worry, no reboot updates are thing of the past thanks to systemd


Care to elaborate?


There are a lot of competing implementations of live kernel patching, but work has started to agree on a common approach. As explained at http://www.linuxplumbersconf.org/2014/an-in-depth-look-live-...

"There has been a great deal of interest in live kernel patching (see http://lwn.net/Articles/597407/) over the past few months, with several different approaches proposed, including CRIU+kexec, kGraft, and kpatch, all in addition to ksplice. This microconference will host discussions on required infrastructure (including tracing, checkpoint/restart, kexec, and live patching), along with expositions and comparisons of the various approaches. The purpose, believe it or not, is to work towards a common implementation that everyone can live with. It should be a spirited discussion! For more details, please see http://wiki.linuxplumbersconf.org/2014:live_kernel_patching"


There are at least two reasons to restart something after you update it.

(1) the update could not be applied to the running instance, so you have to restart to put the update into service, and

(2) to demonstrate that the update did not break the ability of the thing to start.

This address #1. It does not address #2.

It is extremely annoying if something has run for a year with a couple dozen live updates applied without restarting and then you have to restart for some reason (power failure, perhaps), and then you discover that somewhere in the last year it lost the ability to start. I'd much rather discover that an update has introduced a problem with startup immediately after applying the update.

Restarting things should be a regular part of maintenance.


When you restart, you will restart into whichever kernel you have installed, which may be whatever you booted into previously or a new distro kernel with the updates compiled in. There are no Ksplice updates loaded until you tell them to load and Ksplice does not make any modifications to the actual Kernel binary, so #2 isn't really an issue. The real problem is there may be limitations on when you can reboot, leaving you in a vulnerable state. Sure, an ideal environment would let you reboot any system at any time, but those places are like unicorn ranches.

I completely agree with your last comment, though: systems suffer from weird bit-rot diseases. If you never reboot, you may find things challenging when you finally have to.

Disclosure: I work on the Ksplice team.


How do you feel about Oracles handling of the code?


I've worked with a lot of good teams over the years. I've worked with two great teams: The League of Legends team at Riot Games and the Ksplice team at Oracle. I know what many people think of when they think of an average engineer at a huge company such as Oracle, but this team is about as far from stereotypical as you can get.


I'm not seeing #2:

Step 1. Boot your version 1 kernel.

time passes, and version 2 is released

Step 2. Apply Ksplice updates to bring your kernel up to version 2.

Your system reboots (let's say due to a power failure):

Step 1. Boot your version 1 kernel. This boots because it's the same kernel you booted a year ago.

Step 2. (Before bringing up the network) Apply ksplice updates to bring your kernel to version 2. This works because you booted the same version 1 kernel, and the updates haven't changed either.


I used to reboot my Solaris workstation once a month. A friend mocked me--"Still in that Windows mentality". I replied, "No, I reboot it to make sure I still can".

When he rebooted his machine (after almost a year) and it failed to come up, I got to mock him back. "So can you remember what changes you made... in the last year".

On the other hand, even though I've never used it, ksplice must be a nifty way to apply urgent updates between regularly scheduled reboots.


Even with a dynamic software updating system in place, you shouldn't be deploying dynamic updates directly to production without testing them. It'd be pretty stupid to deploy an update by any mechanism that would render a service unstartable.


Anyone interested in this should also look up kGraft and kpatch, developed by SUSE and RedHat, respectively. Each has been submitted for inclusion in the mainline kernel.


Also kernelcare by the cloudlinux folk. Although that has a closed-source blob that it uses, it's available in production now, as opposed to kGraph and kpatch.


Ksplice is now owned by Oracle, meaning it's dead and buried (unless you give Oracle your moneyz).

Look at KPatch (included in RHEL7 as tech preview, will probably mature in subsequent minor version updates): http://rhelblog.redhat.com/2014/02/26/kpatch/


This seems like amazing technology, but I'm having a hard time seeing the use case for it. If you are sensitive to a single machine going down for a reboot, the occasional kernel upgrade is the least of your concerns. Those are at least predictable and rare. I'm sure there are environments where (say) the time to failover to a replica is business-critical, but what are they?


Most SME's usually don't have redundant everything (unlike large enterprises, and IT specific companies) in their environments. They will usually do a cost / benefit analysis and choose to skimp on certain redundancies (usually somewhat "stateless" stuff like routers, VPN endpoints and other network gear).

These companies are always hindered by the downtime incurred when one of these non-redundant devices needs restarting. In a couple of previous positions over the years I would have loved the ability to apply patches with no downtime.

There are environments where the time to failover to redundant infrastructure is critical (HVT, particle physics etc.) but the more common use case is probably going to be the companies that have no failover capability for certain parts of their infrastructure.


Full disk encryption or other situations where manual intervention is a hard requirement to get the system fully back online is one situation.

Any instance where you're running untrusted binaries on your server you'd want to update ASAP and not worry about getting pwned while waiting for scheduled downtime.

Before Oracle closed it to new customers, Ksplice was fairly cheap, ~$3-4/mo/server.


I'd love to see online patching on the free tier of Linux distros (Fedora, ubuntu, etc) as its the ideal test location.

To me this is the first step, the second is doing the same with system libraries. If zlib gets updated what do I have to restart to ensure running applications are using the right version? Right now the answer is a reboot.


Reminds me of Multics; it had online updates.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: