Fundamentals

Video Codec: Everything You Need to Know

Summary: In this article, we talk about video codecs – what are they and how they work. We also provide an overview of the most popular video codecs, their applications, and tips on how to choose a video codec. 

While streaming videos or transferring large videos online, have you ever wondered how videos are compressed to a size that is small enough to travel through the internet to your device?

That’s where video codecs come in.

The word “video codec” is a portmanteau for “encoder – decoder” or “compress and decompress” and it is a critical component for delivering high quality videos efficiently over the internet. They compress and encode video into smaller file sizes for faster transmission, while retaining high video quality.

  • But how exactly do codecs work?
  • How many codecs are there?
  • And, how do you choose a codec for your application and use-case?

This guide explains the key technical concepts, applications and help you make the right choice for your use case.

Let’s get started!

This article will discuss the concepts behind how each of these quality metrics functions, with a focus on PSNR avg. MSE and PSNR avg. log compared to VMAF. Our opinions are based on real-world experiences from our internal experiments and those of more than 50 customers.

Without further ado, let’s dive into an overview of the three most common objective quality metrics, starting with PSNR.

What is a Video Codec And Why Is It Important for Video Streaming?

Let’s start with the absolute basics.

A software or hardware device that compresses and decompresses digital video data is called a video codec.

Codecs are required to reduce the enormous size of video files, making them easier to store, send via the Internet or on physical disks, and view using a player. Through algorithms that encode and decode video, effective compression without major data loss is achieved. Frankly speaking, video files would be difficult to share and view over the Internet without codecs due to their huge size.

But, you might ask, “why is it necessary to reduce the file size using a codec”, right? To answer that and make the concept clear, here are some key reasons why video codecs are so important:

  • Reducing Video Size: By compressing video files, codecs ensure smooth streaming, as they enable us to stream HD video content in a hassle-free and buffer-free manner without having to deal with a problematic pause/play button. This is particularly important at a time when streaming is revolutionizing the entertainment we watch.
  • Optimize for Distribution: Codecs encode parameters like resolution, bitrate, and frame rate tailored for target use – internet streaming, media storage, video conferencing, etc. Have you ever noticed that when you play back video on YouTube, you have multiple options (like 240p, 360, etc.?) Those multiple versions of the movie are produced by a codec.
  • Playback Performance: Hardware decoding built into devices plays encoded video smoothly by reducing processor load. And, in the absence of a hardware decoder, there are multiple software decoders as well that can be integrated onto a device (like the popular dav1d decoder for AV1 decoding).
  • Quality at Lower Bitrates: Advanced algorithms give better quality than older codecs at the same or lower bitrate, and this allows users to stream video on weaker internet connections.

I hope this explanation helped you understand why a video codec is important. Also, as a side note, we won’t be addressing any audio codec in this article (though most of the points apply to it as well).

Next, let’s look at the commonly used video codec formats and their characteristics.

How Video Codecs Work

Video codecs rely on complex algorithms to analyze and compress raw uncompressed video into much smaller encoded outputs.

They are created by large teams of engineers and scientists from organizations across the world to ensure that there is a common format and specification that clearly describes how the encoding and decoding processes take place. An example of such an organization is the MPEG group (Motion Picture Experts Group), which created the popular H.264/AVC, H.265/HEVC, and the latest VVC video codecs.

With this background, let’s see how a codec works.

Encoder and Decoder

A video codec consists of two components: an encoder and a decoder.

The encoder is responsible for compressing and encoding the source video based on specified parameters like bitrate, quality, speed and complexity. Once processed the output is stored for later use. The decoder on the receiving device unpacks and decodes the compressed video so that it can be played back.

Advanced codecs invest heavily in enhanced encoders and decoders to maximize performance. Optimized software and hardware decoders can even playback encoded videos smoothly in real-time on phones and web browsers.

And, there is a whole field of AI-driven video compression that is blooming, where CNNs and advanced ML techniques are being used for optimized video compression.

Compression (Encoding) Techniques

Video codecs use complex algorithms to “squeeze” common and redundant visual data from the video frames:

  • Spatial Redundancy – Nearby pixels within a frame tend to be similar and there is no point in transmitting all of these pixels (redundant information). So, the codec stores repetitive patterns just once and informs the decoder where the redundancy lies and what data needs to be filled in there.
  • Temporal Redundancy – Similar to spatial redundancy, consecutive frames have only small changes. Thus, codecs store only the differences between frames. If you think about it, in slow-moving sports like golf, or when an anchor is reading the news, a very small part of the video is changing between frames. So, a codec only needs to transmit the “difference” to the decoder.
  • Transform and Quantization – This are two advanced mathematical techniques where visual data in the pixel domain are converted to the frequency domain and certain data is discarded that is less perceivable to reduce file size.
  • Entropy Coding – This is one of the final stages where frequently occurring patterns get allocated shorter bit sequences. So, instead of transmitting one hundred “1”s, we could simply transmit the number 100 and tell the decoder to repeat the number “1” a hundred times.

Bitrate, Frame Rate and Resolution

Higher resolution, frame rate, and bitrate settings result in better quality, but also much larger file sizes. Codecs are tuned to give optimal quality at target bitrates and parameters.

  • Bitrate controls the amount of information, or detail, that is stored for each second of video. Higher bitrates result in less compression. More information means better picture quality, but also larger files.
  • Frame Rate is the number of individual images (frames) displayed per second of video. Typically, the frame rate will be 30 or 24 for film and 30 or 60 for video. Generally, higher frame rates produce smoother motion in the video, but they need to be heavily compressed for streaming since there is so much more information.
  • Resolution is the pixel dimensions of each frame, denoted as width x height in pixels, 640×480, 800×600, 1280×720 (720p), and 1920×1080 (1080p) are common dimensions. Higher resolutions will require higher bitrates to look as good and have no noticeable encoding artifacts.

Codecs play a crucial role in video technology, as they are responsible for encoding and decoding digital video data. By compressing video files, codecs make it possible to store and transmit video content efficiently. When a video is encoded, the codec reduces the file size by removing redundant data while striving to maintain acceptable quality.

Here is an example of two different implementations of the H.264/AVC video codec, and they produce vastly different results. On the left is a popular implementation that produces a file at 860 kbps, and on the right is Visionular’s Aurora4 codec that produces a higher video quality, at a much smaller bitrate! The video quality is measured using VMAF (go here to learn more about VMAF, PSNR, and SSIM video quality metrics).

This goes to show that different implementations of a video codec can produce very different results!

Video Codecs and Container Formats

Aside from codecs, video container formats and file types also influence how videos are compressed and stored. And, there is significant confusion between the two when spoken about in the industry.

While codecs handle the compression and decompression of video data, containers determine how the encoded video and audio streams are packaged together. This distinction is vital in ensuring compatibility and seamless playback across different devices.

For example, H.264/AVC, HEVC, and AV1 are video codecs, while MP4, MPEG-TS, AVI, and MOV are video container formats.

popular video codecs - H.264/AVC, HEVC, VP9, and AV1

Commonly Used Video Codecs for Streaming

Today’s most popular video codecs include H.264/AVC (Advanced Video Coding), HEVC (High-Efficiency Video Coding), VP9, and AV1 (amongst others). Each of these codecs offers something a little different, with each one targeting something different and with different technological advancements.

H.264/AVC

The most important and most widely used video codec today still remains AVC, also known as H.264. The codec provides a strong balance—nearly the best—between good compression efficiency, and a high-quality playback, making it useful for online streaming, broadcast, conferencing systems, and much more. Its widespread support across all sorts of devices and platforms makes this codec a very popular choice today.

H.265/HEVC

High Efficiency Video Coding, also known as H.265, is a significant advance over H.264. It is a much more efficient compression standard and is better suited to UHD content. HEVC offers much more efficient compression, enabling delivery of high-quality video at much lower (bit) rates, than was previously possible. This makes it ideal for UHD content delivery to fixed users or for mobile services. It is expected to be deployed more and more widely in the coming years, especially for delivery of 4K and 8K video.

VP9

What sets VP9 apart is that it is the first codec designed specifically with the web in mind and it’s targeted at web-based content. It offers competitive compression efficiency and is particularly well-suited to streaming high-definition video over the internet. It also offers support for adaptive streaming and hardware acceleration of video decoding, which makes it suitable for online video platforms. VP9 is an open and royalty free video coding format that was originally designed and released by Google.

AV1

AV1 (from the Alliance of Open Media), is actually one of the newest on the list of video codecs, just released in mid-2018. The codec itself is royalty-free. Its aim is deliver significant improvements in compression efficiency, especially for high-resolution and high-dynamic-range content. All of that is true, and there probably isn’t a knowledgeable encoder that would argue that point. The format is actually initially being pushed and advanced primarily by some of the internet’s most powerful companies: Amazon, Google, Cisco, Microsoft, Mozilla, and Netflix.

Uses and Applications of Video Codecs

While we know how a codec works, and the difference between a codec and a container, we still haven’t addressed the topic of understanding the applications of video codecs.

The way a video codec is configured, tuned, and deployed depends heavily on its target application – both in terms of the genre and the typical playback device.

Online Video Streaming (OTT, UGC, PGC)

For internet delivery to desktop/mobile devices, parameters are optimized for real-time performance at available viewer bandwidths:

  • Lower resolutions like 480p/720p
  • Moderate to high compression
  • Adaptive bitrate streaming

Popular web/OTT codecs include AV1, VP9, H.264/AVC and HEVC.

Blu-Ray/Media Distribution

For storing HD/UHD content on disks and digital media files, quality and resolution are maximized:

  • High bitrates (15-100+ Mbps)
  • High frame rates (60 fps)
  • Maximum resolutions (4K, 8K)
  • Light compression for minimal artifacts

Video Conferencing

Video chat apps need to encode/decode in real time over varying network conditions:

  • Tuned for very low latency
  • Dynamic adaptation to changing bandwidth
  • Fast performance on phone processors

Popular conferencing codecs include H.263, H.264/AVC, and more recently AV1.

As you can see, each application calls for a codec that’s tailored and optimized specifically for that use case.

Best Video Codec For Streaming?

This is a difficult topic because there are so many factors that need to be considered while choosing a codec – from both a business and technical angle. However, let us look at a key factors to consider when selecting the right video codec:

Bandwidth vs Quality Tradeoff

This is the most important trade-off to consider and it depends on your business model, the amount of viewership on your platform, the genres of your videos, and so many other critical factors.

Simply put, the lower the available bandwidth to your viewers, the more compression you need. But, to get the best compression, you need to pay a premium price for the best video codec, that can assure you great video quality, smaller bitrates, and the best CPU utilization possible.

Decoding Support

While the latest codecs may deliver better compression efficiency, it is important to evaluate the decoding support available across all the devices you expect your customers to use. For example, if 90% of your users use Android mobile phones, then you should choose a video codec that is supported on the Android eco-system (like H.264/AVC or AV1).

Licensing Costs

Proprietary codecs often have patent licensing fees (like H.264/AVC, HEVC, VVC, LC-EVC, etc.). On the other hand. options like AV1 and VP9 avoid royalty payments.

Ultimately, there is no one “best” codec. The choice depends on parameters like target resolution, available bandwidth, hardware/software decoding support, licensing costs and your specific needs.

Parting Thoughts

Video codecs play a crucial for streaming over the internet and storage on digital media. Understanding how they work provides key insights into delivering high quality video experiences online.

Since video consumption continues rising exponentially, codecs will evolve to be even smarter about compression and delivering visually lossless quality at lower file sizes. The next generation of video experiences over the internet will be enabled by codecs that are more powerful, efficient and intelligent.