Put another way: The video patent cartels have fragmented due to greed, and so the cartel enablers at ISO have lost their purpose. In addition, organizations are tired of paying bridge tolls.
So we can finally have open standards for video that anyone can just use and implement. AV1 in particular seems to be on its way to replacing them: https://research.mozilla.org/av1-media-codecs/
I guess it wont mean that parents on video encoding won't exist, rather that now they wont be standardized. Not everyone can be sponsored by advertising monopoly, some might need their technology to bring revenue by itself. However yes, the pool broke by too greedy companies (IP based companies, not technology ones) and hevc is laughable. See it explained by mpeg chairman itself https://blog.chiariglione.org/a-crisis-the-causes-and-a-solu...
They will probably be standardized, but not by ISO. Currently it looks like AOMedia will take over, in part because unlike ISO they are willing to commit to having an open standard unencumbered by patents.
This is the classic chicken and egg problem with any multimedia encoding standards.
As someone who mostly uses video in the context of online systems, MP4 (H.264) is the closest we have to a universally supported format on the user's end and it has been for years. However, it's complicated, patent encumbered and very inefficient by modern standards. And even MP4 isn't 100% universally supported.
Once encoding and decoding at reasonable speeds is possible -- we're not there yet, but there has been a lot of progress already -- I see no reason that AV1 shouldn't render H.264 and similar legacy formats obsolete quite quickly. The new format doesn't need to be perfect at first, just good enough to use with mid-range to high-end devices. Then the openness and improved compression/quality will make it an attractive alternative, with legacy formats reduced to being just a fallback for less capable devices. One generation of devices later, almost everyone can play it. Two generations later, you probably have hardware support, but at that point it's more about efficiency things like reducing battery drain on the viewer's side and hardware costs on the encoding side, because the format itself has already won.
Yes, while the future is hard to predict, right now it looks like AV1 is on its way to fixing the world. If anyone can freely create and view videos without paying the endless tolls for permission, that would be a vast improvement.
Chariglione was the charismatic leader behind the MPEG standards, including those behind DVD (MPEG-2). The biggest challenge wasn’t technical but political: getting a bunch of companies to agree to pool their patents so licensing them could be streamlined.
For next-generation video beyond H.264 (AVC) and H.265 (HEVC), that consensus has broken-down and his prediction seems to be that open-source standards like Alliance for Open Media behind the AV1 video codec will prevail. It’s worth noting that H.264 and H.265 are telecom videoconferencing standards defined by the ITU-T, not consumer electronics standards defined by ISO, so I’d say the loss of relevance of MPEG happened 20 years ago, even if MPEG (part of ISO) was involved in both standards.
You're right, of course, the two notions are orthogonal, but in practice an open-source project like Mozilla cannot use royalty-encumbered codecs.
As for the royalty-free nature of AV1, it will only be known once all the patent lawsuits are litigated. The AOM patent litigation fund in itself is no guarantee, given the capricious nature of common-law courts.
I think the specifics matter here and I'm not in any way versed well enough in law to know so correct me if if I'm wrong but:
Mozilla could absolutely use and distribute x265 but that would also require them to have a license for the patents required for HVEC and would also require the users to have a license (which is of course unrealistic).
The only reason firefox can play h264 is because Cisco paid for an unlimited license with subdistribution and assigned it to mozilla.
So as h265/HVEC is concerned the problem for mozilla is that even if someone wanted to pay the bill like cisco did they would not be able to get a straight answer who and what they needed to pay for a license, right?
That's exactly right. While it is impossible to absolutely prove a negative (that no one will make a credible patent claim against AV1) -- it's worth noting that the line of codecs starting with VP3 that led up to AV1 go back to research done in the early 90's, and there has never been a serious patent claim against any of them.
This is at least partly by design: On2 had an explicit mission to create competitive codecs that avoided conflict with the MPEG patents. I'm pretty sure this was a major factor in their appeal to Google.
(Disclaimer: I was CTO &/or CEO of On2 up until 2003)
This is really a story of Open Source and shared intellectual property finally winning out over expensive, proprietary standards that tried to present themselves as something else.
Disclaimer: I was founder of on2 Technologies, acquired by Google and the basis for av1.
There really Is quite a bit to this story, starting in the early 90s and leading up to the situation today.
I used to work in the same building as on2 in Clifton Park. You could still make out the company name on the sign next to the main entrance. Many people are amazed to find out that the technology behind a lot of internet video has roots in this area. The building also housed many of the developers for company called Kitware, which are probably best known for cmake.
I always wondered happened to the people after Google purchased on2. It would make sense that Google would want to keep that knowledge and not just buy it for the patents and source. Thank you for clearing that up.
I suspect the source code would have had way less value without the engineers who created it.
Given they paid > $130M for the company, taking on 15 or 20 engineers who know the code inside & out, and are a proven functional team, would be a no-brainer for Google (they're always searching for talented tech ppl anyway).
The only sort of businesses who regularly buy IP without the team behind it are patent trolls and shitty companies who buy to kill competition.
As far as I know, the story has never really been covered by the tech press. Closest it's come is that some of the people involved may have inspired some of the characters and plot lines in Silicon Valley. Middle-out refers to b-frames, which were heavily patented, so we avoided them. Unfortunately that causes a popping artifact every time a keyframe comes along...
I think what might really be interesting was the decision process to open-source vp3, AKA Theora, the ancestor of vp8, vp9, and AV1. Even though it was only marginally competitive with mpeg-4, the existence of a free alternative, seemingly unencumbered by the sort of patent problems afflicting MPEG, put a check on MPEG becoming a virtual monopoly.
In 2010, Google took the baton with the acquisition of on2, and doubled down on the open-source approach.
I don't really understand what the point of EVC is. From my understanding it's really two standards, one royalty-free that's slightly worse than HEVC/H.265, and one patent-encumbered/licensed that's slightly better than H.265.
The post here says "EVC is promising because it provides a quality that is comparable with or better than AV1, although less than VVC. EVC may have a chance if a licence will be published. However, this has not happened yet." I can only assume, from those numbers above, that "comparable with or better than AV1" applies only to the encumbered/enhanced variant.
It's hard to see why anyone would bother to implement the "base" standard vs. the already widely-deployed AVC/H.264, nor the "enhanced" one which seems to be roughly comparable to AV1 but with licensing costs attached (and as the post points out, no certainty at all about what that licensing would actually entail). Apart from those companies that hold the relevant patents, of course.
The widely deployed AVC is still AVC Main Profile, which is what Youtube and many other Streaming / Broadcasting uses due to devices compatibility. EVC Baseline is expected ( or claims ) to be 20% better than AVC High Profile. So you should expect ~30% better than current AVC all while being royalty free.
The beauty of EVC baseline is that it is quite power efficient. Considering AVC already require the least computation in modern codec, EVC baseline offer 30% reduction in bitrate while requiring NO increase in decoding complexity and is actually 40 to 50% less in encoding complexity.
Note: None of these has been tested outside of its members as it does not provide a reference encoder due to "new" patents arrangement with this codec. So we have no way to verify those claims.
Edit: Will add the link to document with those claimed figures later.
My understanding is that YouTube uses at least some High Profile, and only Main for the lowest resolutions (though who knows how up to date that information is), along with of course VP9 and AV1 where they figure they can.
Anyway, if those claims bear out then it would seem like EVC has a possible future as an "AVC killer," though the computation complexity factors are complicated by the wide deployment of hardware acceleration for AVC encode/decode. Of course, hardware support could materialize for EVC as well, but it does rather complicate possible adoption for what we'll call a "sub-generational" improvement, in a world coming around right about now to AV1 hardware support.
Perhaps reduced computational demands could lead to a niche of usage in unaccelerated environments. I don't know if there's an example of significant uptake for an "in-between" step like this that one could point to as a possible model for success, though.
[1] Seemingly no source code available for AVS2 decoders, dubious claims of beating HEVC without independent tests, and China history of "independent" development by copy&paste notwithstanding
I would say that in this case, that's a reasonable decision.
If a physicist can't patent superconductivity (you can't patent discoveries and laws of nature), should it be possible to patent compression algorithms? The case for patents in software, as additional protection on top on copyright, is not clear to me.
Frankly neither copyright nor the patent law is actually suitable for software. Software deserves a tailor-made IP law that actually recognises the difference between compiled code and source code, the rights of the consumer to modify his device and keep it updated, etc. There should be an escrow-type system, where protection is granted if code is hosted in a public repository, and is made public after 20 years or so.
All executable should be treated in a standard way, and there should be no EULA outside of enterprise systems. A consumer person does not need to read EULA when they buy a book or a car, why should they do so when they buy a PC?
Until this happens, I feel developing nations shouldn't even recognise software patents - copyright is protection enough.
> should it be possible to patent compression algorithms?
I would say yes, at least for lossy compression, where the challenge is not about shuffling bits but finding semi-artistic graphical approximations of common video patterns.
I also approve of an industry-wide effort to patent-unencumbered solutions.
Am I reading it right ? The text says:
> But there is a big news: MPEG passed away on 2020/06/02T16:30 CEST.
Does that really mean MPEG is no more ? Or is it hyperbole about something that happened 2nd of june ?
This would be huge (and really sad).
I am not sure where to begin. But let's start with EVC.
EVC Baseline is expected to be 20% better than AVC High Profile, and EVC Main Profile is expected to be 30% better than HEVC. The Codec is backed by Huawei, Samsung and Qualcomm. If you are unfamiliar with the Smartphone market, The three would represent roughly 80% of marketshare excluding Apple.
Now on to VVC. I am surprised by the sentence
MC-IF has 31 members, 7 of which are licensing entities (i.e. a little less than ¼ of all members). The “industry” members account for just ½ of the HEVC patent holders. It is hardly believable that VVC will fare better than HEVC. It could very well fare worse because VVC adoption in broadcasting will take years.
Well Yes. That is because 7 of those licensing entities already covered most if not all of the industry. ( Excluding a few Open Media Alliance Member of course ) The MC-IF actually includes ALL of the current HEVC patents holders. That is HEVC Advance and Velos Media, which is basically Samsung and Qualcomm, along with many others that were not in any HEVC patent pool due to disagreement in the first place.
( And If you notice the removal of the infamous Technicolor, they sold their patents to another entity that is inside MC-IF, but I cant remember which one on top of my head )
VVC is expected to be 50% more efficient than HEVC. And judging from its reference encoder, this is the first time since AVC / H.264 era a video codec that might actually live up to its claims. ( Normally marketing likes to use unrealistic claims ) It really is the state of the art Video Codec, at a decoding complexity that is quite manageable. ( Lower than AV1 )
So what does all that means? Samsung, Huawei and Qualcomm are also inside MC-IF ( represented by different groups ). My guess is that EVC is basically a backup plan or a gesture to MC-IF, if the licensing deal can be agreed upon, they will go with EVC.
As a video codec enthusiast, I am extremely excited with both VVC and EVC.
As to MPEG ( Not to be confused with MPEG-LA ), I am not quite sure why he said it is dead. I reread the article a few times and still dont quite understand it. May be I am missing some context?
> It really is the state of the art Video Codec, at a decoding complexity that is quite manageable. ( Lower than AV1 )
I am afraid this has not been substantiated by any of the public decoder demonstrations I've seen. Please see the most recent VVC technical update presented at the MC-IF meeting on March 2nd of this year:
On slide 10 is a graph of VVC performance showing the VTM (VVC) decoder at 2.0x the complexity of HM (HEVC). On slide 12, the Ittiam production decoder boasts 1920x1080 @ 24fps on a 4-core Cortex A75 @ 2.5GHz.
Compare that with this recent study of dav1d (AV1) decoder complexity on a broad set of mobile SOCs, where 1920x1080 @ 24fps was easily reached by a Google Pixel 1 from 2016! Using just the two LITTLE cores! Even higher frame rates were achieved with more modern devices:
Yes. But I think it should be noted the HM and VTM are reference encoder and decoder, they are not meant to be used for production nor in any way optimised.
> But I think it should be noted the HM and VTM are reference encoder and decoder, they are not meant to be used for production nor in any way optimised.
Sure, but my comparison was to the Ittiam production decoder.
I think that is a very partial reading of the forces at play.
Apple may be 20% of unit sales, but an overwhelming majority of industry profits and has a lock on the best demographics (the rich and the young). AFAIK they haven’t decided, even if historically they tended to prefer ITU/MPEG standards (but boycotted Blu-Ray).
AV1 has Netflix and Google/YouTube behind it. Not sure about Amazon, but they own Twitch which is on the AV1 side. In an era where the content providers also control the app through which it is delivered, they have the control over which codec gets chosen, not the hardware manufacturer. They may tactically choose to use hardware-accelerated standards like HEVC when trading off the royalties they have to pay vs. using less battery life on their customers’ devices, but in the long term it seems to me AV1 will prevail
They want VVC. After all MC-IF was co-founded by Apple. But that doesn't mean MC-IF will succeed. Having everyone on the table does not necessarily mean positive results will come.
This is absolutely amazing, I had no idea this would come so soon. I wonder how much the vp% series of codecs played a part, or netflix/amazon/youtube controlling both ends of the channel?
The next device to get converted to the web is the smart tv, there is no reason it literally has be anything different then a webpage, same goes for Roku.
>It should be no surprise that the HEVC standard has some use in broadcasting, but its use on the web is estimated to be at 12%. If one considers that broadcasting is a rich but declining market and video on the web is constantly rising, one understands that ISO standards will be gradually relegated to a more and more marginal market.
True enough. But at least in the USA all the satellite companies, and their set top boxes, are being forced to switch to HEVC because their half of their physical frequency spectrum was stolen by telcos. So there'll be substantial amounts of people making hardware for quite a while. The web isn't everything yet.
The amount of data you can transmit over a radio link is proportional to the span of frequency you use. Your bandwidth. Satellite operators used to have almost all of C band's frequency span. Changes were recently pushed through to give half of this to mobile telcos. So now the satellite operators have to push the same amount of video/media with half the physical resource to do it. So they need to send half the data. The FCC has literally told the incumbent sat users to switch to HEVC to accomplish this.
Frequency gives you available bandwidth. Less bandwidth forces you to compress it more. So you either send worse quality, or compress it better with a newer/better codec.
It means you can force people who actually develops products to pay you money for using your totally novel and not at all vague idea that you wrote down on a piece of paper 15 years ago that is so unimaginably revolutionary that nobody else could ever come up with something remotely similar.
It means that you can basically levy tax on compatibility. Anyone, who wants to make device or application compatible with standard has to pay a fee to you.
It's more than that even. They lobbied governments to mandate use of this technology such that whenever you buy any device (e.g TV, Phone etc) you have to pay a tax for something even if you don't use it
I presume it means you can require royalties from anyone who wishes to implement them. If you build a DVD player, you pay someone a royalty for those IPs, if you build a BluRay player, it's the same.
Having the "standard" means having the lion's share of the market, which means more money.
In particular IP here ends up meaning patents. Patents are a particularly egregious right because though they were created with a specific trade in mind they were almost immediately perverted so as to keep only one side of the trade.
So the idea of patents is, if Alice invents something really fucking amazing she explains the invention, pays the government some money and they publicise her explanation but she is entitled to control who uses the invention for a period of time.
This is a trade because now Alice's invention is available to everybody, no chance Alice loses interest and it's lost for a thousand years before somebody else re-discovers it, but on the other hand Alice doesn't need to secure investors and risk the invention being a failure, she can demand terms like 10% of the profits from anyone taking those risks.
But, it soon turned out you don't need a complete working invention. Alice describes half an invention, and then nobody else can use it without the other half, but even if they figure out the other half on their own they owe Alice! Alice doesn't even need to invent the whole thing, she can just describe the easy half, the government signs off and then when somebody smarter finishes it Alice gets rich!
Today in most of the world people have found ways to argue that computer software, which is a work of literature and thus protected by Copyright (which has different problems we won't discuss here) can also count as an invention and be patented.
So the best outcome for an outfit like MPEG is that Bob invents a really clever technique that turns moving images into much less data than before, totally just as a single flash of inspiration somehow, and rather than keeping it secret, or using just in one MSDOS program in 1989 and then never again he publishes it as a patentable invention, then MPEG incorporates it in a video standard called, say, MPEG Bob. And maybe everybody in the world cheerfully pays Bob $100 each for this amazing invention. Hooray.
But very quickly the problem with MPEG is that Charlie gets a government patent for the invention of, say, dividing five into three equal integers and then hires a very good lawyer. Charlie's lawyer says it doesn't matter that this is nonsense and can't work, because $10M worth of lawyers say you owe Charlie for using this invention in MPEG Bob.
In between these extremes there are lots of problems less intrusive than Charlie. The Springfield Higher Institute of Technology gets a patent on an idea you, an expert in the field, have been telling people about for years, but you never wrote it down so you can't prove it. Did they really invent it, or just hear it second hand? Either way, they demand $1 each, but they are a university so maybe that's good? Although it is billions of dollars, and it was really your idea if it was anybody's...
Leonardo, the author of this piece and MPEG leader, is quite sure that patents are necessary, mostly because of the Bobs in this world though I suspect he'd be sympathetic to the Springfield Institute too - to him the existence of Charlie is an annoyance that we should all try to find some way past rather than a fatal flaw in the entire endeavour.
> Charlie gets a government patent for the invention of, say, dividing five into three equal integers
Hey, you just described S3TC patent! Funnily enough S3 ripped off Apple, copying verbatim Hoffert work at Apple Advanced Technology Group and consequent patents from 1990 (US5046119A) for Apple Video 'road pizza' codec, except their patent added "for texture compression" at the end. The patent is about dividing colorspace between two points by dividing colors by 3 :-) and is directly coped from Apple. Gotta love how wikipedia calls direct copy a "This mode operates similarly to mode 0xC0 of the original Apple Video codec".
Amazingly S3 even had the balls to try and sue Apple for S3TC royalties in 2010. Sadly Apple loves patents and didnt bother trying to invalidate, they were content winning on technicality. S3TC patents are expired now.
Summarizing your last point: everything is a remix. To people with a deep understanding of technology, it's clear that everything builds upon prior art and some of the best "innovation" is pulling together insights from many fields (many of the Mathematics).
Which is why it's so infuriating to read Leonardo's grandstanding "Those who have created a new intellectual object have to right to exploit it". What exactly did this "creation" build on? It was not produced in isolation.
A predatory business model based primarily on large companies exploiting legal technicalities to seek rent from innovators has been rendered obsolete because the large companies couldn't get their act together and the innovators did a better job anyway.
Efforts to co-ordinate those large companies in those exploits are probably now doomed, as are the organisations behind them.
The future almost certainly belongs to open standards and the community, though someone who has built their career around the old business model isn't happy about it.
It's a bit too quick to call propritary video codecs dead.
The billion dollar plus a year residual stream can pay for a lot of efforts to keep trying to refresh the stream.
The author of the post seems to hold the position that mpeg falling apart is because many companies have been too aggressive. But the lesson that some are going to walk away with it that they haven't been aggressive enough.
Proprietary video formats certainly aren't dead yet. MP4 is still the most common format by a long way in web work, for example.
I think it's clear which way the water is flowing, though, and I don't think any amount of distress on the part of the fading powers of yesteryear is going to change that. There is too much incentive and too much technical expertise on the other side, and that side also has enough big players with their own legal teams to block attempts at obstruction by questionable legal challenge.
But don't underestimate what a firehose of money can do, especially one in the hands of which is effectively a dying industry. (A small one, for sure, but one that was formerly extremely profitable)
The big players are can bring significant resources to bare to be sure, but they're also the least committed fundamentally. The right extremely beneficial licensing terms for say, HEVC, in exchange for dropping out of AOM would probably be extremely uhh "business smart" for more than a few of those big names.
One of the challenges in this space has always been that patent licensing for existing codec has zero marginal cost. Vendors can respond to the threat of open alternatives with licensing concessions. I think part of why we see the system collapsing a bit now for video is because they've managed to thicket themselves to the point where its become extremely politically difficult to negotiate in that manner.
So we can finally have open standards for video that anyone can just use and implement. AV1 in particular seems to be on its way to replacing them: https://research.mozilla.org/av1-media-codecs/