Cross-platform audio/video processing library based on FFmpeg native libraries. Supports audio/video frames extraction (fast access to any frame by timestamp), reading file metadata and encoding media files from bitmap images and audio data.
FFMediaToolkit is a .NET library for creating and reading multimedia files. It uses native FFmpeg libraries by the FFmpeg.Autogen bindings.
Features
Decoding/encoding audio-video files in many formats supported by FFmpeg.
Extracting audio data as floating point arrays.
Access to any video frame by timestamp.
Creating videos from images with metadata, pixel format, bitrate, CRF, FPS, GoP, dimensions and other codec settings.
Supports reading multimedia chapters and metadata.
Code samples
Extract all video frames as PNG files
// Open video file
using var file = MediaFile.Open(@"D:\example\movie.mp4", new MediaOptions() { VideoPixelFormat = ImagePixelFormat.Rgba32 });
// Get pixel buffer from SkiaSharp bitmap
using var bitmap = new SKBitmap(file.Video.Info.FrameSize.Width, file.Video.Info.FrameSize.Height, SKColorType.Rgba8888, SKAlphaType.Unpremul);
var pixelBuffer = bitmap.GetPixelSpan();
int i = 0;
// Iterate over all frames in the video - decoded frame will be written to the buffer
while (file.Video.TryGetNextFrame(pixelBuffer))
{
// Save image as PNG file
using var fs = File.OpenWrite($@"D:\example\frame_{i++}.png");
bitmap.Encode(fs, SKEncodedImageFormat.Png, 100);
}
This example uses SkiaSharp to save decoded frames. See Usage details section for examples with other graphics libraries.
Video decoding
// Open a multimedia file
// You can use the MediaOptions properties to set decoder options
var file = MediaFile.Open(@"C:\videos\movie.mp4");
// Get the frame at 5th second of the video
var frame = file.Video.GetFrame(TimeSpan.FromSeconds(5));
// ...
// Frame should be disposed when no longer needed
frame.Dispose();
// Print information about the video stream
Console.WriteLine($"Bitrate: {file.Info.Bitrate / 1000.0} kb/s");
var info = file.Video.Info;
Console.WriteLine($"Duration: {info.Duration}");
Console.WriteLine($"Frames count: {info.NumberOfFrames ?? "N/A"}");
var frameRateInfo = info.IsVariableFrameRate ? "average" : "constant";
Console.WriteLine($"Frame rate: {info.AvgFrameRate} fps ({frameRateInfo})");
Console.WriteLine($"Frame size: {info.FrameSize}");
Console.WriteLine($"Pixel format: {info.PixelFormat}");
Console.WriteLine($"Codec: {info.CodecName}");
Console.WriteLine($"Is interlaced: {info.IsInterlaced}");
Encode video from images.
// You can set codec, bitrate, frame rate and many other options here
var settings = new VideoEncoderSettings(width: 1920, height: 1080, framerate: 30, codec: VideoCodec.H264) {
EncoderPreset = EncoderPreset.Fast,
CRF = 17,
};
// Create output file
using var file = MediaBuilder.CreateContainer(@"D:\example\video.mp4").WithVideo(settings).Create();
for(int i = 0; i < 300; i++)
{
// Load image using SkiaSharp (other libraries are also supported if provide access to pixel buffer)
using var bmp = SKBitmap.Decode($@"D:\example\frame_{i}.png");
// Encode the video frame
file.Video.AddFrame(new ImageData(bmp.GetPixelSpan(), ImagePixelFormat.Rgba32, bmp.Width, bmp.Height));
}
FFmpeg libraries are not included in the package. To use FFMediaToolkit, you need the FFmpeg shared build binaries: avcodec (v61), avformat (v61), avutil (v59), swresample (v5), swscale (v8).
Supported FFmpeg version: 7.x (shared build)
Windows - You can download it from the BtbN/FFmpeg-Builds or gyan.dev. You only need *.dll files from the .\bin directory (not .\lib) of the ZIP package. Place the binaries in .\runtimes\win-[x64\arm64]\native\ in the application output directory or set FFmpegLoader.FFmpegPath.
Linux - Download FFmpeg using your package manager. Default path is /usr/lib/*-linux-gnu
macOS, iOS, Android - Not supported.
You need to set FFmpegLoader.FFmpegPath with a full path to FFmpeg libraries.
In .NET Framework projects you have to disable the Build -> Prefer 32-bit option in Visual Studio project properties.
Usage details
FFMediaToolkit supports decoding video frames into pixel buffers which can be Span<byte>, byte[] or unmanaged memory. You can specify target pixel format by setting the MediaOptions.VideoPixelFormat property. The default format is Bgr24.
If you want to process or save the decoded frame, you can pass it to another graphics library, as shown in the examples below.
Video decoding
using var file = MediaFile.Open(@"D:\example\video.mp4", new MediaOptions() {
StreamsToLoad = MediaMode.Video,
VideoPixelFormat = ImagePixelFormat.Rgba32
});
using var bitmap = new SKBitmap(file.Video.Info.FrameSize.Width, file.Video.Info.FrameSize.Height, SKColorType.Rgba8888, SKAlphaType.Unpremul);
var buffer = bitmap.GetPixelSpan();
while(file.Video.TryGetNextFrame(buffer)) {
// do something
}
Video encoding
using var bmp = SKBitmap.Decode($@"D:\example\frame.png");
mediaFile.Video.AddFrame(new ImageData(bmp.GetPixelSpan(), ImagePixelFormat.Rgba32, bmp.Width, bmp.Height));
Video decoding
var buffer = new byte[file.Video.FrameByteCount];
var bmp = Image.WrapMemory<Bgr24>(buffer, file.Video.Info.FrameSize.Width, file.Video.Info.FrameSize.Height);
while(file.Video.TryGetNextFrame(buffer)) {
// do something
}
For GDI+ System.Drawing.Bitmap (Windows only):
Video decoding
// Create bitmap once
var rect = new Rectangle(Point.Empty, file.Video.Info.FrameSize);
var bitmap = new Bitmap(rect.Width, rect.Height, PixelFormat.Format24bppRgb);
// ...
// Read next frame
var bitLock = bitmap.LockBits(rect, ImageLockMode.WriteOnly, PixelFormat.Format24bppRgb);
file.Video.TryGetNextFrame(bitLock.Scan0, bitLock.Stride);
bitmap.UnlockBits(bitLock);
Video encoding
var rect = new Rectangle(Point.Empty, bitmap.Size);
var bitLock = bitmap.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format24bppRgb);
var bitmapData = ImageData.FromPointer(bitLock.Scan0, ImagePixelFormat.Bgr24, bitmap.Size);
mediaFile.Video.AddFrame(bitmapData); // Encode the frame
bitmap.UnlockBits(bitLock); // UnlockBits() must be called after encoding the frame
For desktop apps with WPF UI (Windows only):
Video decoding
using System.Windows.Media.Imaging;
// Create bitmap once
var bmp = new WriteableBitmap(media.Video.Info.FrameSize.Width, media.Video.Info.FrameSize.Height, 96, 96, PixelFormats.Bgr24, null);
// ...
// Read next frame
bmp.Lock();
var success = media.Video.TryGetNextFrame(bmp.BackBuffer, bmp.BackBufferStride);
if(success) {
bmp.AddDirtyRect(new Int32Rect(0, 0, media.Video.Info.FrameSize.Width, media.Video.Info.FrameSize.Height));
}
bmp.Unlock();
Video encoding
var bitmapSource = new BitmapImage(new Uri(@"D:\example\image.png"));
var wb = new WriteableBitmap(bitmap);
mediaFile.Video.AddFrame(ImageData.FromPointer(wb.BackBuffer, ImagePixelFormat.Bgra32, wb.PixelWidth, wb.PixelHeight));
Visual Basic usage
Writing decoded bitmap directly to the WPF WriteableBitmap buffer:
Dim file As FileStream = New FileStream("path to the video file", FileMode.Open, FileAccess.Read)
Dim media As MediaFile = MediaFile.Load(file)
Dim bmp As WriteableBimap = New WriteableBitmap(media.Video.Info.FrameSize.Width, media.Video.Info.FrameSize.Height, 96, 96, PixelFormats.Bgr24, Nothing)
bmp.Lock()
Dim decoded As Boolean = media.Video.TryGetFrame(TimeSpan.FromMinutes(1), bmp.BackBuffer, bmp.BackBufferStride)
If decoded Then
bmp.AddDirtyRect(New Int32Rect(0, 0, media.Video.Info.FrameSize.Width, media.Video.Info.FrameSize.Height))
End If
bmp.Unlock()
imageBox.Source = bmp
Converting ImageData to a byte array:
Dim data() As Byte = media.Video.GetNextFrame().Data.ToArray()
