The important is, is your code good? We care about excellent code. We don't care who you are. Like maybe you're a dog. I don't care, right? I don't care where you come from. I need to look at your code. "Oh yeah, but I'm an engineer at this very large company in Italy, in Germany, in the US." We don't care. We care about the quality of your code, because this is what defines our community.
FFmpeg is probably one of the biggest CPU users in the world. Every cycle matters. We are talking about probably 3 billion devices which are going to decode video non-stop β because, for example, 30% of the video from Netflix are now in AV1, 50% of YouTube. This is what peak video codec should look like. 79.9% assembly, 90.6% C, and 0.5% other. And what's incredible is, with those tweets, which are factual, people get crazy. For the last two years they go crazy. "No, intrinsics is fine. The compiler auto-vectorization is fine. It's your fault. You don't understand." And we've tried that forever. For two years, two years later, showing hundreds of examples of handwritten assembly β "No no no, you're doing it wrong, the compiler can do this."
The intelligence agencies tried to ask, "Can you put a backdoor in VLC?" β "Yes, two of them." What do you say? No. No. I was a lot less polite. Basically saying hell no. Like if we had to compromise our software, we would shut it down. This is clear. Any tweets carrying your regret? No tweets regret. Or is it like the French song goes β regret nothing. Don't regret anything. No, it's because regrets are a tax on your mind.
(Lex) The following is a conversation all about FFmpeg and VLC with Jean-Baptiste Kempf and Kieran Kunhya. FFmpeg is an open-source software system that is the invisible backbone behind YouTube, Netflix, Chrome, VLC, Discord, and basically every platform that touches video or audio on the internet. It can decode, encode, transcode, stream, and play almost any video or audio format ever created. To me, it is one of the most incredible software systems ever developed. And it's all done by volunteers. VLC is also a legendary piece of software. It is an open-source media player that plays basically anything you throw at it β any format, any platform, no ads, no tracking. It has been downloaded over 6 billion times.
Jean-Baptiste is the president of VideoLAN and is a key figure behind VLC and FFmpeg. Kieran is a long-time codec engineer, FFmpeg contributor, and the man behind the now infamous FFmpeg account on X that I recommend everybody follow for the memes and for the unapologetic celebration of open source and great low-level software engineering.
(Lex) So, the legend goes VLC can open everything. What's the weirdest thing that you know that it can open? β (JB) There is a ton of people who are using VLC to record VHS videos. You plug it with a capture card and you can basically record VHS video. We support DVD-Audio lately β we spent the summer working on DVD-Audio support, and there is no one else making any DVD-Audio support. There is a custom encryption scheme. What about Lucasfilm? Weird codecs supported by FFmpeg β the game codec from the first Star Wars video game: someone has gone and implemented that and made sure it's bit-exact on one disc that existed at one time of one little sequence in the game.
(JB) At one VideoLAN conference, we made a competition to make the weirdest and most horrible file ever and see if VLC could play it. It was an MKV file made by Derek where each frame was changing resolution, aspect ratio, rotation. And it worked. There was another one where the whole video was actually animated subtitles (SSA), and on top of that there was a subtitle that was animated for each frame. There was a file that was a valid zip and a valid MP3 at the same time. So yeah, we had made a competition of stupid files. And it worked.
(JB on the cone logo) The logo of VLC is so iconic. We are a small team and the icon is known everywhere. I go to middle of nowhere in India or in China, people know the cone. 25% of the website traffic that comes to our main website is "cone player". So many people don't know VLC β they know the cone player. That's the thing they Google for. And once we tried to change it as a joke β we said it was going to be a caterpillar-like construction icon. During April 1st we had around 10,000 emails saying "No, don't change the logo."
(Lex) When we press play on a video player like VLC, what happens? How does it go from the file or the stream to the pixels on the screen and the sound on the speaker?
(JB) There are several stages. The first stage is to get from an address β the type of URL β to a stream of bytes. So this would be, for example, HTTP, file, DVD. You give the path to the media and get a stream of data.
(Kieran) The stream needs to be cut up by what's known as the container, the demultiplexer or demux. It needs to start demarcating video and audio frames. It just gets data from the operating system, blocks at a time, and needs to start cutting these frames up into compressed data. It then needs to start doing simple parsing of the video frames, mainly to figure out whether that codec is GPU-decodable or needs to fall back to software. We're very sort of used to assuming the GPU will play all of these things β there'll be hardware acceleration. I think it's up to 45% of files are not GPU-decodable. So these need to be probed, they need to be detected. There can be variants of a given codec, some of which are decodable on the GPU. Different vendors of GPU might have different capabilities.
(Kieran) If there's a software fallback, in the beginning is the first do de-entropy coding β so removing the mathematical coding of the bitstream. This uses capabilities such as Huffman coding or arithmetic coding to actually decompress the mathematical layer of the bitstream. We then need to start reading the syntax elements for intra prediction. Intra prediction are like still images of the video β your I-frames. This works and operates in the spatial domain. You have a residual because your prediction isn't quite matching that of reality. You've made a prediction, but then there's a little bit left, and that's what's known as the residual. This is stored in the frequency domain and these are quantized to compand their space. We then need to do the inverse transform to bring them back to the spatial domain and apply these residuals.
People don't realize how compressed we do. For audio, when you go from normal audio to MP3, you compress by 10Γ. When you move to video, you need 100Γ, 200Γ. So you need to remove all the details that you don't care about β because all the compression that we do, and this is very important, is to be viewed by humans. All the codecs, either for audio, mimic basically how your ear works. And same for your eyes. We don't work on RGB β everyone expects to work in RGB. We move to YUV: one is luminance (brightness), and the others are colors. This matches your eyes, where inside your eyes you have the cones and the rods.
(JB) Compression is not like a zip. A zip β you have data in, you get data out, and you try with all the zip compression to arrive within the limit. Here, we are degrading the signal. And so we need to degrade both the audio and the video signal in the best way possible. It involves first a lot of theoretical knowledge about how it works, how the eye works, but a lot of mathematical change, a lot of mathematical tricks. For example, when you move to RGB and you go to YUV, what we do very often is that we scale down the resolution of the color compared to the brightness. Just this, without compression, divides the size by two. But most people don't see it.
(JB) Then you go to very complex mathematical change β so of course Fourier transform, which de facto are not Fourier transforms, they are like discrete cosine transforms, but that's the same idea: frequency domain. We split the video by blocks β that's why when it's wrongly decoded, you see those blocks. And each generation of the codec is like 30% less for the same quality. This requires amounts of computational power that are huge β 30% better, but an order of magnitude perhaps even two orders of magnitude more compression power.
(Lex on containers) MP4, MOV, MKV β what are containers versus the thing that goes inside? (JB) The container is what we call the muxer. When I say demuxing, it means decontainerizing. Mux means multiplexer and demux means demultiplexer. A codec is actually coder-decoder. Containers are these collections of multiple tracks β what normal people call the file format. The most known one of course is MP4, but when I started, it was AVI from Microsoft. MOV, which became MP4, was a format from Apple. In the open-source community, Steve Lhomme started this Matroska format, which is a bit more complex and more feature-proof.
(Kieran) MP4 is understood to generally be H.264 plus AAC audio β 99% of the time that's it. The rest are de minimis. In reality, something like VLC has to deal with the fact that the file may say .mp4 but may be something completely different. The real world is a completely different place to a three-letter file format. (JB) In VLC and FFmpeg, we discard the file format β we look into the file to understand what's in it. We analyze in real time everything that we have and we don't trust the format. The file extension only bumps the priority of the module to try first.
(JB on FFmpeg as a DAW for video) For example, you can do things that you would do with Adobe After Effects in command line on FFmpeg. For images, there is not such tool. ImageMagick has a similar kind of approach, but it won't do the equivalent of Photoshop in command line. For video, you have FFmpeg in command line. The vision behind FFmpeg and VLC is to make everything that is very complex easy to use for normal people. People drop a file, they don't realize how complex the file is, but they play it. Or people put any type of thing inside FFmpeg with filters and it just works like magically.
(JB) For people who don't understand how open source is, my usual analogy is about a chocolate cheesecake. Usually when you want to buy your cheesecake, you go to a bakery, they give you the cheesecake. The other way of having a cheesecake is have your grandma give you a recipe of how to make it. When we do open source, we give you the chocolate cake AND we give you the recipe β we tell you how to build the oven and also how you're allowed to modify the recipe and sell it to someone else. Computers are not very clever. They go very, very fast. So a normal program has tens of billions of instructions instead of the tens you have in your chocolate recipe.
(JB on VideoLAN's origin) The story goes back to 1995. A professor at Γcole Centrale Paris had this idea: instead of having one satellite dish and a big decoder for each of the 1,500 students, what about you build an enormous dish, only one decoder, and you send the video directly on the network. Today it's obvious, but at the time it was the first to do video streaming. They built this project called "Network 2000". It's completely hacked, it crashes after 45 seconds β that's okay, the demo is 40 seconds. It leaks memory β that's okay. They put 64 megabytes of RAM instead of the 8 or 16 you had at the time. The demo should have stopped there. They had like amazing ATM network at 155 Mb/s, one of the best networks in Europe at that time.
(JB) Six months or a year later, two students arrive and say, "You know what? Maybe other people care about video streamed on a local network." And they create the VideoLAN project. It's not even open source yet. They spend around three years getting the school to agree to make it open source β the university wanted to monetize these MPEG-2 decoders. In 2001, the university agreed to make it open source. I joined the project in 2003 because that's when I joined the university. The first thing is β I'm not the one who created VLC. It just kind of naturally emerged from the VideoLAN project. I created the open-source non-profit organization VideoLAN and took everything out of the university to make it sustainable.
Along the way I got several offers β either to bundle horrible toolbars which were basically spyware, or change your web browser or your search engine, or put advertisement inside VLC. I didn't like that. I'm not against money β I created several startups. But I believe you need to make money ethically. There is a right way of doing it, and sneaky advertisement or stealing data is not the correct way. If Netflix had arrived at some point and said "we want to put Netflix inside VLC", probably the story would have been different. The only people who came to us were shady ad companies. If I had done that, I'd have a ton of money β and then three years later, the project is gone. The last offer I had was obscene. They said, "Yeah, but imagine with all that money, you could build something new open source." The mind trick was difficult. For me it was just like, "No, this doesn't work, this is not the right thing, so I don't do it." Selfishly, I need to go to bed at night and be happy about what I've done.
(JB on Linus Torvalds) Linus is one of a kind, and I would even go and say that what he did on Git is more interesting than what he did on the Linux kernel. He's very harsh, but what people don't see is usually when he's harsh, it's people who are maintainers of part of the kernel β so they know him. He's not harsh like that to everyone. The thing is what he created in his room is basically powering every server online. Even at Microsoft Cloud called Azure, I'm quite sure 70-80% of the servers are running Linux. All your Android phones are running Linux. We cannot compromise on quality β because in the end, the core community of VLC is five people. The core community of FFmpeg is 10 to 15. And we are the ones who are going to maintain your code. Out of 1,000 contributors, only 1% β so 10 β will come and stay. So we will maintain your code. It needs to be maintainable. It needs to be excellent.
The main one is that what we do in multimedia plays videos, and video is cool. We have so many people in the community who arrived because they loved watching anime. My advice when people ask "what should I work on in open source?" is always the same: work on something you love. I am working on VLC because I love movies. Second, technically β because we search for excellence β this is the best school ever of programming. If you're good in C, in FFmpeg, if you know how to write assembly, I assure you, you're going to be one of the best programmers ever. Andrew Kelley, who started Zig, was an FFmpeg developer. And third β you can be proud of it, because it's on the end of everyone. If you go see your grandma and say "I do this so that you can play video on your laptop", they understand.
(JB) All open source licensing uses the copyright law. It's de facto a copyright license contract that you give to the end user or to the developer. You have the first kind, which are basically very permissive β MIT, BSD. You give the code and you do whatever you want. This is popular for JavaScript and BSD operating systems. Then there's the other part β copyleft β where you need to give back to the community your modifications, with different strings attached. Some weak copyleft like Mozilla Public License, some stronger like GPL, or very strong like AGPL.
(JB) FFmpeg and VLC are mostly GPL or LGPL. The Linux kernel is GPL. Android is Apache. JavaScript frameworks mostly MIT. All the BSD kernels (OpenBSD, NetBSD) are of course BSD. It's a philosophical choice on how you want people to contribute back. I'd call it a social contract β because this is very important to understand.
(JB on changing libVLC from GPL to LGPL) You can always go from more permissive to less permissive β because of course those licenses are basically statements. So if you restrict, you can always restrict more. In a GPL project, you can take MIT code, but you cannot do the opposite. I changed the core of libVLC, the engine of VLC, from GPL to LGPL. There were two reasons. The first: so people can use the VLC engine in third-party applications. A lot of applications playing video on your phone or tablet are actually VLC engine inside, calling FFmpeg. That was one of the ways to create one of the companies I created, doing consulting and integration. With GPL, you couldn't do that β that means you needed to open source everything, and a lot of commercial companies don't want that.
(JB) The problem is: I'm a game developer, I want to play some videos, and I don't want to be forced to open source the entire game just to play those videos. That's where the libVLC LGPL allows you to do that. LGPL β "library GPL" β forces you to give back what you change on the component, the library. So you can use FFmpeg as LGPL in any type of application, even non-open-source, but you need to give back the modifications you did on FFmpeg, same on libVLC.
(JB) The second reason β a bit more obscure β is that the terms of conditions of the Apple App Store for iOS makes it very complex to have GPL applications, while it's easier to have LGPL ones. So VLC on Windows, Mac, Linux is GPL; the core is LGPL; but on iOS and Apple TV it's MPL.
Open source projects are what we call in US copyright law "joint works", or in civil law "collective works" or "collaborative works". The copyright is kept by all the individuals. Some open source projects force copyright assignment, but this is not what we do. Everyone has copyright on what they changed. So if you want to properly re-license, you need to find all the contributors. At that time, I had to contact more than 350 people. Sometimes it's just an email. I actually traveled to some places to go to their job and say, "Well, you licensed that, can you change from GPL to LGPL?" Most of the time, they don't even care. They wanted to help. But it also brought me to very complex situations. I arrived to the work of a person who was a factory worker. I said, "Well, I need you to sign that" β because it was his son who died who actually wrote the code. I had to explain all those open source meanings β no, I'm not a company trying to rip out the two lines of five lines that guy did. I was almost in tears. We're talking about life of people. We went to talk about the photo of his son. It's important to do it right.
(Kieran) Just to be clear, Google are one of the biggest supporters of open source out there. They have been for a long time. It's just I think some things kind of went a bit overboard this time. FFmpeg itself β and this is not a secret, it's on the homepage β processes untrusted data. There can be security issues when you parse untrusted data, that's very normal. But recently what changed was Google started using AI to create security reports on an open source project, FFmpeg. They provided very limited funding, and they even went to the media first announcing how good their AI was before the issues could be fixed. They provided a standard 90-day industry deadline, without really understanding the nature of volunteer-driven development. In addition, this vulnerability was one of the obscure 1990s game codecs.
It does seem that there's a portion of the security community that look at themselves a bit like building architects that never have to go to site. The language they use is extremely aggressive. They use very strong language like "You will get popped." For Joe public, "get popped" means something quite bad. For them, it means to get hacked. The way I would look at it personally is a little bit like the padlock on your home. The lock on your home is there to protect against the capabilities of what it's there to protect. It's not there to protect nuclear secrets. It's not there to protect Fort Knox. It could be looked at that they're using AI at a level of scale to go and pick those locks and then say, "Hey, your lock's not secure. You need to deal with this." Whereas actually, they're the ones with resources to be able to fix this.
(Kieran) The bug reports are very wordy. It's almost a denial-of-service by AI-generated bug reports on very niche codecs. And the other issue the security community has is everything is marked high priority. "You're going to need to deal with this, this is the most important thing in the world." High, high, high, vulnerable, scary, scary, scary on a game codec used on one disc in 1993. There was another quote-unquote vulnerability β it wasn't Google in this case β a filter could overflow, and one of your pixels could be the wrong color. This was marked HIGH, 7.5 severity in red. At some point the security industry needs to realize you can't keep crying wolf like this.
(Kieran) Google uses FFmpeg at a scale probably you or I couldn't even contemplate. Millions of CPU cores. Yes, they contribute in areas mostly regarding their own products β VP9, AV1 β but in a wider sense, there's a disproportionate level of contribution. Alex Strange, a former FFmpeg developer, posted on Hacker News: "The problem with security reports in general is security people are rampant self-promoters. Imagine your humble volunteer open source developer. If a security researcher finds a bug in your code, they're going to make up a cute name for it, start a website with a logo, Google is going to give them a million-dollar bounty, they're going to go to Def Con and get a prize, and I assume get some kind of secret security people orgy where everyone is dressed like they're in the Matrix. Nobody is going to do any of this for the open source developer."
(JB) Microsoft Teams posted on a bug tracker for volunteers that their issue is high priority. After politely requesting a support contract from Microsoft for long-term maintenance, they offered a one-time payment of a few thousand dollars instead. This is unacceptable. We didn't make it up. This is what Microsoft Teams actually did. (Kieran) They think a public bug tracker is actually a third-party vendor's Jira. It's there to report bugs.
(Kieran) The thing that made this particularly heinous was the name-dropping of Microsoft, the name-dropping of "this is a visible product". If this was a general bug report, it would have been a lot better. (Lex) I wonder what happens psychologically. They just think of FFmpeg as a vendor that Microsoft surely is paying a huge amount of money to. (JB) That person was just a manager on one project in Microsoft Teams. He had never really discussed with open source community. Usually there's what we call OSPOs β Open Source Program Offices β and they're supposed to discuss with open source communities, but they often don't explain it correctly internally.
(JB on VLC Android case) VLC on Android, because of a bug on the Play Store, the only way we got someone to answer was to put a very spicy tweet saying that we were going to stop distributing VLC for Android β and we have around 100 million people using that. Then someone from Android actually came and discussed with us. We had the same issue with Microsoft, saying we were going to stop distributing VLC on the Windows Store. Unfortunately, we are so small that the only very strong power we have to solve those issues is blaming on social network. VLC is probably one of the top 10 software used on Windows. I am not part of Microsoft ISV programs. I don't have a point of contact at Microsoft. I'm sure any other software β Adobe, Spotify β has a point of contact. I don't.
The way we looked at this is like it's a rap battle at the end of the day. We say stuff. They say stuff. But we can leave it on X β it's a perfect place for international rap battle. I say stuff about your mama, but it doesn't mean I'm going to have an actual personal issue with her. The teenager thing β that was a Google employee saying, "Hey, there are other ways to run an open source business. Go and just have a bit of fun." That's the point of this account. And if you can teach people about the ways of open source projects, assembly, etc., by doing that, I think there's a lot to be offered here. It's not dunking on people for dunking's sake.
(JB on David, the AV1 decoder) David is a decoder for the format that was done by Alliance for Open Media β AV1. When this format was launched, many people, especially from the Alliance for Open Media (Google, Netflix, Amazon, Mozilla), said "this format is so complex, it must be done in hardware." A few of us β mostly Ronald, Henrik, Martin and I β said we need an extremely good software decoder, because it's going to take time to have hardware. We wrote this project which is beyond insane. We are talking about 30,000 lines of C, but 240,000 lines of handwritten assembly.
(JB) FFmpeg has about 100,000 lines of assembly for all the codecs. David is one VideoLAN project, optimized to the maximum, because the motto when starting the project was "every cycle matters". David is used in VLC and in some software AV1 playback stacks. We are talking about probably 3 billion devices that will decode video non-stop β for example 30% of YouTube. With David we realized that with one or two cores you were able to decode 720p correctly.
When you write assembly code, you write using the instructions the actual processor is using directly. Most of the time you would write in C, and the compiler would generate assembly. There's a specific flavor of assembly we use in FFmpeg called SIMD β single instruction, multiple data. So if I want to add 5 to a number in scalar assembly, I use the add instruction. With SIMD I can have a whole vector of 16 different numbers, and one instruction sums all 16 elements at the same time. The key thing we do differently in FFmpeg is we don't use any abstractions like intrinsics. When we write SIMD we have a 10Γ to 50Γ speed improvement. That function is 62Γ faster than C.
(JB) We passed the time where hardware was going so much faster β we're at the end of Moore's Law. We have limitations for AI, for memory. You need to go down in the stack and optimize more to get more power from what you have. What people do is add more cores, but at some point you cannot add 250 cores. So what we do is take every inch of the machine. We abuse the machine. We use it in ways that the creator didn't expect. Sometimes we use a cryptography instruction in video processing to do nothing related. In David, we don't even use the function calling convention from the operating system β because we know we're going to be called from within our binary, so we can share data without saving all the registers in the common way.
(Kieran on calling conventions) David takes this even further. For speed reasons, it does its own calling convention within itself to bypass the rules of functions. The challenge in assembly is every operating system has its own calling conventions β Linux 32-bit, Windows 32-bit, Windows 64, Linux 64. One of the amazing things Loren Merritt did was create a very lightweight abstraction layer (x86inc), so you could write your assembly code once and it handled all the calling convention stuff for you. Which is always a problem because you had to manage four different variants. David takes it even further by doing its own.
(JB on reverse engineering β Kostya Shishkov) An amazing Ukrainian guy called Kostya, at the time living in Germany. He's one of those who are borderline geniuses, doing extremely complex codecs. We do a bit of engineering with Kieran but clearly not at this level. To reverse engineer a 1MB binary blob is probably an order of magnitude a month of work β and this guy is doing 20-30MB blobs. GoToMeeting was a big problem with VLC. At some point Kostya said, "Okay JB, I'm going to do it." In a matter of 2 months. He explained how β "I looked at the code, this looked like a DCT I used to see on WMV." The funniest part is the code has a ton of jokes inside.
(Kieran walking through reverse engineering GoToMeeting) First, there's a ton of other stuff there β the actual video conferencing client. It may not be easy to find the actual module doing the decompression. You need a way to actually dump the YUV data from the module. Often it involves opening in a disassembler, trying to guess where the hooks are. You need that as a point of comparison for when you actually do the reverse engineering β you'll need to be bit-exact or close to bit-exact. Then you open up your disassembler, use a lot of intuition to figure out where the DCT is, where's entropy coding. For a long time you don't see anything β you're debugging purely in memory.
FFmpeg is unique in the sense that it has been a winner-takes-all scenario. Browsers is a good analogy β they have to parse a lot of different content and render it. But there still are multiple browser engines: Firefox, Chrome, and some Japanese ones. That's not been the case in multimedia in general across a wide range of codecs. FFmpeg has won it all, I suppose, because every new codec added is actually worth more than the value of that codec itself, because it makes the whole thing better.
(Kieran on Rust) It has a very big Esperanto vibe about it. There's a lot of focus on the self-importance rather than solving real-world problems. It reminds me of the Sinclair C5 β Sir Clive Sinclair built a car and said "everyone will be traveling in one of these electric cars." The community doesn't quite understand that in order to get people to move, you have to build something that's as good as if not better than what you have now. People are doing Rust rewrites, but if they only do 85-90% of the feature set of what we need β like things like coreutils β that last 1% takes 99% of the time. To use Elon's famous quote: "Prototypes are easy." Rust isn't in that stage yet. I don't think anyone would object to seeing Rust code in FFmpeg, but it needs to work as well and support the same unit testing as everything else.
(Lex on Karpathy) (Kieran) Karpathy learned assembly because of the tweets. He went and figured out what's happening here. (JB) Like Tim Sweeney, Carmack, and a few others β very high-level people have raised the awareness on our X accounts, and that helped a lot.
(Kieran on asm-lessons) The way assembly is taught in books and online is very grammar-focused, and you don't in general learn a language from learning the grammar and the structure. You learn a language by asking someone what their name is, and you start from there. All the assembly books seem to be doing this β going through every instruction. I started a set of assembly lessons in the way it's done in FFmpeg. The requirements are quite simple: high school mathematics and C β actually not even C really, really just pointers. There's been contributions because of these lessons. It's really about trying to get this dying art to continue.
(JB on Kyber) If you start from where we used to be: you used to use FFmpeg to encode files. Then FFmpeg and VLC to encode in streaming services. Then you need to go lower and lower in latency. The question is, how low can we go? There are many use cases where you need to be fast β when you have feedback interaction. You're not just listening to something, you're actually controlling it: whether it's a drone flying, controlling humanoid robots from distance, controlling a rover, or playing a video game in the cloud. You push to the limit the network. You need to care, not about the quality like with H.264, but about latency. A millisecond is meaningful when you're controlling a car. Waymo β when Waymos don't work, even 1% of the time, there is someone basically remote-controlling that. This is exactly the stuff we're building.
What we do that is different from everyone else is we take only one socket, one connection, which is a QUIC protocol based on UDP β interesting because it's done for low latency. It doesn't have the TCP head-of-line problem. On the same wire, we send multiple streams: audio, video, but also the commands β mouse, keyboard, gamepad. We do that while maintaining coherence β synchronization. What people don't realize is that all the clocks actually drift. When you're controlling a robot, a robot is going to have 2 to 10 cameras, GPS, and sensors. If you want to train correctly your robotic AI model, you need all of that to be in sync. We account for clock drifting. My goal is 4 milliseconds glass-to-glass latency. So far we achieve 7 milliseconds Windows to Windows. There's around 3.5 ms inside the Nvidia hardware encoder and around 2 ms on the Intel decoder.
(JB) Kyber is also open source. It's a dual license β commercial and AGPL. If you want to use Kyber in your product, you must have your full product open source. If you want to use this amazing technology but not open source, you pay the commercial license. The hobbyists can use the technology and build something cool open source. A company gets support, all the IP, the right to modifications.
(JB on codec generations) AV1 is the codec done by the Alliance for Open Media β Google, Netflix, Amazon, Apple, VideoLAN β where we try to make a royalty-free very good codec. It's deployed now, but the codec was finished in 2018. A codec takes years to be used in wide scenarios. AV2 is the next generation, 30% better. We are going to do a David 2 β I call it "David" because "de" is two in French. David is an actual recursive acronym β D stands for David, an AV1 decoder. David is spelled with a 1; David 2 will be spelled with a 2. We did a demo at CES of VLC running the first demo of AV2.
(JB on the patent minefield) HEVC licensing was MPEG-LA plus another patent pool called HEVC Advance plus Nokia outside the patent pool. So it was impossible to license. HP decided several months ago that they were going to remove support for HEVC in their Windows laptops because the cost of those patents was increasing. There was uncapped patents β for YouTube or Netflix, we could talk about hundreds of millions of dollars of licensing per year. So they said "I don't want to give them that, I'll create my own codec." That's why we have the Alliance for Open Media that created AV1 and creates AV2. So the main difference would be that you need to work around the patents β VVC has all the patents of HEVC plus new ones, while AV2 tries to be as royalty-free as possible.
(Kieran on archives) One of the coolest communities in open source multimedia, mainly led by someone called Dave Rice from City University of New York, is the archiving community. They value open source β because yes, they lack budgets, but two, they see the fact that archiving video is important for the world. Being able to play that is a big problem. Famously in the UK, there was something called the New Domesday Book, and they archived lots of stuff on BBC microcomputers. Within 10 to 15 years, no one had the right software to play that. Someone had to go and reverse-engineer it.
One of the great things about FFmpeg is it's written in C. C is the closest to mathematics you're probably going to get, the closest to logic. Will we still have C compilers? Yes, we have languages that exist that haven't changed too much. We have mathematical notation that exists. It will be like Latin β a thing that you learn from the past, but it will still be usable in certain contexts. So the archiving community funded the development of the FFV1 codec β a lossless codec for archiving. They funded GPU encoding in FFmpeg to make FFV1 encode faster. There's a lot of great teams in archival groups across the world who've chosen FFmpeg and FFV1 as their archiving solution.
(JB on government pressure) Intelligence agencies tried to ask "Can you put a backdoor in VLC?" β basically saying hell no. Like if we had to compromise our software, we would shut it down. There is no code that gets into VLC that we don't control. The way we compile VLC, you would call me completely paranoid. We compile on boxes that are offline, where we start by compiling the compiler. We do everything offline on places that have never been connected to the internet. We do double signing β especially because we believe a governmental agency from not the Western world tried to push a fake binary into our own servers, and that scared us a lot. VideoLAN is open source. How can you kill it? I move to Malta, I move to Cayman Islands, I change the domain name and I start again. VLC is a tool.
(JB on staying zen) I used to stress about that a lot. I'm not anymore. What's the secret? I have a way of thinking about what is the worst-case scenario, always. Like a chess player. In the end, am I dead? Yes or no? That's how I do my startups. What is the worst case? Going bankrupt. That's life. A company lives, a company dies. That's okay. My moral way is always like: am I dying in the end? Am I hurting someone? If the answer is no, then too bad. Some lawyers are going to be unhappy. What are they going to do? Take all the money of VideoLAN? Wow, they're going to have $50,000. What are they going to do with that? The source code is out there. It's unstoppable. Also because what we do is good and it's done for everyone.
(Kieran on the future) Will VLC and FFmpeg be here 100 years from now? FFmpeg, yes. VLC, maybe. (JB) Today there are new codecs for what we call point cloud, or volumetric videos. There's a ton of research on RGBD β codecs for depth, useful for robotics and 3D. We already have a VR and XR version of VLC. On Kyber, we'd also like to do streaming of XR content for glasses that cannot have enough power, or inside Apple Vision or Quest. The community is open. Not everyone in the community sees that, but as Kieran and I, we are entrepreneurs, we know where it's going. Multimedia is a digital representation of streams for the human senses. We will do that. Imagine there's a way to have an odor sensor and diffuser. Of course your demuxer has a new track type that is basically odors. We already have a plugin for haptic β for 4D cinemas. It is a human sense, so it will get in.
(JB on the closing) Don't regret anything. Regrets are a tax on your mind. Learn from your mistakes, sure. But don't regret. Except if you have a time machine to go back in time, don't regret β it's going to tax your brain.
(Lex closing with a Linus quote) Most good programmers do programming not because they expect to get paid or get adulation by the public, but because it is fun to program.