OnnxStack.StableDiffusion - Onnx Stable Diffusion Library for .NET
OnnxStack.StableDiffusion is a library that provides access to Stable Diffusion processes in .NET.
It offers extensive support for features such as TextToImage, ImageToImage, VideoToVideo, ControlNet and more
Getting Started
OnnxStack.StableDiffusion can be found via the nuget package manager, download and install it.
PM> Install-Package OnnxStack.StableDiffusion
OnnxRuntime
Depending on the devices you have and the platform you are running on, you will want to install the Microsoft.ML.OnnxRuntime package that best suits your needs.
Run a simple Stable Diffusion process with a basic prompt
//Model:
//https://huggingface.co/runwayml/stable-diffusion-v1-5 (onnx branch)
// Create Pipeline
var pipeline = StableDiffusionPipeline.CreatePipeline("models\\stable-diffusion-v1-5");
// Set Prompt Options
var promptOptions = new PromptOptions { Prompt = "Photo of a cute dog." };
// Run Pipleine
var result = await pipeline.RunAsync(promptOptions);
// Save image result
var image = result.ToImage();
await image.SaveAsPngAsync("D:\\Results\\Image.png");
// Unload Pipleine
await pipeline.UnloadAsync();
Stable Diffusion Batch Example
Run Stable Diffusion process and return a batch of results
//Model:
//https://huggingface.co/runwayml/stable-diffusion-v1-5 (onnx branch)
// Create Pipeline
var pipeline = StableDiffusionPipeline.CreatePipeline("models\\stable-diffusion-v1-5");
// Prompt
var promptOptions = new PromptOptions{ Prompt = "Photo of a cat" };
// Batch Of 5 Images with unique seeds
var batchOptions = new BatchOptions
{
ValueTo = 5,
BatchType = BatchOptionType.Seed
};
// Run Pipleine
await foreach (var result in pipeline.RunBatchAsync(batchOptions, promptOptions))
{
// Save Image result
var image = result.ImageResult.ToImage();
await image.SaveAsPngAsync($"Output_Batch_{result.SchedulerOptions.Seed}.png");
}
// Unload Pipleine
await pipeline.UnloadAsync();
Stable Diffusion ImageToImage Example
Run Stable Diffusion process with an initial image as input
//Model:
//https://huggingface.co/runwayml/stable-diffusion-v1-5 (onnx branch)
// Create Pipeline
var pipeline = StableDiffusionPipeline.CreatePipeline("models\\stable-diffusion-v1-5");
// Load Input Image
var inputImage = await InputImage.FromFileAsync("Input.png");
// Set Prompt Options
var promptOptions = new PromptOptions
{
DiffuserType = DiffuserType.ImageToImage,
Prompt = "Photo of a cute dog.",
InputImage = inputImage
};
// Set Sheduler Options
var schedulerOptions = pipeline.DefaultSchedulerOptions with
{
// How much the output should look like the input
Strength = 0.8f
};
// Run Pipleine
var result = await pipeline.RunAsync(promptOptions, schedulerOptions);
// Save image result
var image = result.ToImage();
await image.SaveAsPngAsync("Output_ImageToImage.png");
// Unload Pipleine
await pipeline.UnloadAsync();
Input
Output
Stable Diffusion ControlNet Example
Run Stable Diffusion process with ControlNet depth
//Models:
//https://huggingface.co/axodoxian/controlnet_onnx
//https://huggingface.co/axodoxian/stable_diffusion_onnx
// Create Pipeline
var pipeline = StableDiffusionPipeline.CreatePipeline("models\\stable_diffusion_onnx", ModelType.ControlNet);
// Load ControlNet Model
var controlNet = ControlNetModel.Create("models\\controlnet_onnx\\controlnet\\depth.onnx");
// Load Control Image
var controlImage = await InputImage.FromFileAsync("Input_Depth.png");
// Set Prompt Options
var promptOptions = new PromptOptions
{
DiffuserType = DiffuserType.ControlNet,
Prompt = "Photo-realistic alien",
InputContolImage = controlImage
};
// Run Pipleine
var result = await pipeline.RunAsync(promptOptions, controlNet: controlNet);
// Save image result
var image = result.ToImage();
await image.SaveAsPngAsync("Output_ControlNet.png");
// Unload Pipleine
await pipeline.UnloadAsync();
Input
Output
Stable Diffusion VideoToVideo Example
Run Stable Diffusion process on a video frame by frame
//Model:
//https://huggingface.co/runwayml/stable-diffusion-v1-5 (onnx branch)
// Create Pipeline
var pipeline = StableDiffusionPipeline.CreatePipeline("models\\stable-diffusion-v1-5");
// Preload Models (optional)
await pipeline.LoadAsync();
// Load Video
var targetFPS = 15;
var videoInput = await VideoInput.FromFileAsync("Input.gif", targetFPS);
// Add text and video to prompt
var promptOptions = new PromptOptions
{
Prompt = "Elon Musk",
DiffuserType = DiffuserType.ImageToImage,
InputVideo = videoInput
};
// Run pipeline
var result = await pipeline.RunAsync(promptOptions, progressCallback: OutputHelpers.FrameProgressCallback);
// Save Video File
var outputFilename = Path.Combine(_outputDirectory, "Output_VideoToVideo.mp4");
await VideoInput.SaveFileAsync(result, outputFilename, targetFPS);
// Unload Pipleine
await pipeline.UnloadAsync();
