markhazleton/WebSpark.HttpClientUtilityv2.2.0

.NET 8.0.NET 9.0.NET 10.0

Drop-in HttpClient wrapper for .NET 8-10+ with Polly resilience, response caching, and OpenTelemetry. One-line setup eliminates boilerplate for retries, circuit breakers, correlation IDs, and structured logging. Perfect for microservices, APIs, and web scrapers. Includes authentication providers (Bearer, Basic, API Key), concurrent requests, fire-and-forget operations, and streaming support. MIT licensed, 252+ tests. For web crawling features, see WebSpark.HttpClientUtility.Crawler package.

markhazleton.github.io ↗

License

MIT

Deps

Install Size

—

Vulns

✓ 0

Published

Jan 3, 2026

Get Started

$ dotnet add package WebSpark.HttpClientUtility

Readme

WebSpark.HttpClientUtility

Drop-in HttpClient wrapper with Polly resilience, response caching, and OpenTelemetry for .NET 8-10+ APIs—configured in one line

Stop writing 50+ lines of HttpClient setup. Get enterprise-grade resilience (retries, circuit breakers), intelligent caching, structured logging with correlation IDs, and OpenTelemetry tracing in a single AddHttpClientUtility() call. Perfect for microservices, background workers, and web scrapers.

🚀 Why Choose WebSpark.HttpClientUtility?

Your HTTP setup in 1 line vs. 50+

Feature	WebSpark.HttpClientUtility	Raw HttpClient	RestSharp	Refit
Setup Complexity	⭐ One line	⭐⭐⭐ 50+ lines manual	⭐⭐ Low	⭐⭐ Low
Built-in Retry/Circuit Breaker	✅ Polly integrated	❌ Manual Polly setup	❌ Manual	❌ Manual
Response Caching	✅ Configurable, in-memory	❌ Manual	❌ Manual	❌ Manual
Correlation IDs	✅ Automatic	❌ Manual middleware	❌ Manual	❌ Manual
OpenTelemetry	✅ Built-in	❌ Manual ActivitySource	❌ Manual	❌ Manual
Structured Logging	✅ Rich context	❌ Manual ILogger	⭐⭐ Basic	⭐⭐ Basic
Web Crawling	✅ Separate package	❌ No	❌ No	❌ No
Production Trust	✅ 252+ tests, LTS support	✅ Microsoft-backed	✅ Popular (7M+ downloads)	✅ Popular (10M+ downloads)

When to use WebSpark:

✅ Building microservices with distributed tracing requirements
✅ Need resilience patterns without writing Polly boilerplate
✅ Want intelligent caching for API rate-limit compliance
✅ Building web scrapers or crawlers (with Crawler package)

When NOT to use WebSpark:

❌ You need declarative, type-safe API clients (use Refit)
❌ You want maximum control and minimal magic (use raw HttpClient)
❌ Legacy .NET Framework 4.x projects (WebSpark requires .NET 8+)

🛡️ Production Trust

Battle-Tested & Production-Ready

✅ 237+ unit tests (711 test runs across 3 frameworks) with 100% passing - tested on .NET 8, 9, and 10
✅ Source Link enabled - step-through debugging with symbol packages (.snupkg)
✅ Trimming & AOT ready - annotated for Native AOT and IL trimming compatibility
✅ Package validation - baseline validation ensures no breaking changes
✅ Zero-warning builds - strict code quality with TreatWarningsAsErrors=true
✅ Continuous Integration via GitHub Actions - every commit tested
✅ Semantic Versioning - predictable, safe upgrades
✅ Zero breaking changes within major versions - backward compatibility guaranteed
✅ Framework Support: .NET 8 LTS (supported until Nov 2026), .NET 9, .NET 10 (Preview)
✅ MIT Licensed - free for commercial use

Support & Maintenance

🔄 Active development - regular updates and improvements
📅 Long-term support - each major version supported for 18+ months
💬 Community support - GitHub Discussions for questions and best practices
📖 Comprehensive documentation - Full docs site

Breaking Change Commitment

We follow semantic versioning strictly:

Patch versions (2.0.x): Bug fixes only, zero breaking changes
Minor versions (2.x.0): New features, backward compatible
Major versions (x.0.0): Breaking changes with detailed migration guides

📦 v2.0 - Now in Two Focused Packages!

Starting with v2.0, the library is split into two packages:

Package	Purpose	Size	Use When
WebSpark.HttpClientUtility	Core HTTP features	163 KB	You need HTTP client utilities (authentication, caching, resilience, telemetry)
WebSpark.HttpClientUtility.Crawler	Web crawling extension	75 KB	You need web crawling, robots.txt parsing, sitemap generation

Upgrading from v1.x? Most users need no code changes! See Migration Guide.

📚 Documentation

View Full Documentation →

The complete documentation site includes:

Getting started guide
Feature documentation
API reference
Code examples
Best practices

⚡ 30-Second Quick Start

Install

dotnet add package WebSpark.HttpClientUtility

Minimal Example (Absolute Minimum)

// Program.cs
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHttpClientUtility();
var app = builder.Build();

app.MapGet("/weather", async (IHttpRequestResultService http) =>
{
    var request = new HttpRequestResult<WeatherData>
    {
        RequestPath = "https://api.weather.com/forecast?city=Seattle",
        RequestMethod = HttpMethod.Get
    };
    var result = await http.HttpSendRequestResultAsync(request);
    return result.IsSuccessStatusCode ? Results.Ok(result.ResponseResults) : Results.Problem();
});

app.Run();

record WeatherData(string City, int Temp);

That's it! You now have:

✅ Automatic correlation IDs for tracing
✅ Structured logging with request/response details
✅ Request timing telemetry
✅ Proper error handling and exception management
✅ Support for .NET 8 LTS, .NET 9, and .NET 10 (Preview)

📖 Show more: Service-based pattern with error handling

// Program.cs
builder.Services.AddHttpClientUtility(options =>
{
    options.EnableCaching = true;      // Cache responses
    options.EnableResilience = true;   // Retry on failure
});

// WeatherService.cs
public class WeatherService
{
    private readonly IHttpRequestResultService _http;
    private readonly ILogger<WeatherService> _logger;

    public WeatherService(
        IHttpRequestResultService http,
        ILogger<WeatherService> logger)
    {
        _http = http;
        _logger = logger;
    }

    public async Task<WeatherData?> GetWeatherAsync(string city)
    {
        var request = new HttpRequestResult<WeatherData>
        {
            RequestPath = $"https://api.weather.com/forecast?city={city}",
            RequestMethod = HttpMethod.Get,
            CacheDurationMinutes = 10  // Cache for 10 minutes
        };

        var result = await _http.HttpSendRequestResultAsync(request);

        if (!result.IsSuccessStatusCode)
        {
            _logger.LogError(
                "Weather API failed: {StatusCode} - {Error}",
                result.StatusCode,
                result.ErrorDetails
            );
            return null;
        }

        return result.ResponseResults;
    }
}

📖 Show more: Full-featured with auth and observability

// Program.cs - Advanced configuration
builder.Services.AddHttpClientUtility(options =>
{
    options.EnableCaching = true;
    options.EnableResilience = true;
    options.ResilienceOptions.MaxRetryAttempts = 3;
    options.ResilienceOptions.RetryDelay = TimeSpan.FromSeconds(2);
    options.DefaultTimeout = TimeSpan.FromSeconds(30);
});

// WeatherService.cs - Advanced usage
public async Task<WeatherData?> GetWeatherWithAuthAsync(string city, string apiKey)
{
    var request = new HttpRequestResult<WeatherData>
    {
        RequestPath = $"https://api.weather.com/forecast?city={city}",
        RequestMethod = HttpMethod.Get,
        CacheDurationMinutes = 10,
        Headers = new Dictionary<string, string>
        {
            ["X-API-Key"] = apiKey,
            ["Accept"] = "application/json"
        }
    };

    var result = await _http.HttpSendRequestResultAsync(request);

    // Correlation ID is automatically logged and propagated
    _logger.LogInformation(
        "Weather request completed in {Duration}ms with correlation {CorrelationId}",
        result.RequestDuration,
        result.CorrelationId
    );

    return result.IsSuccessStatusCode ? result.ResponseResults : null;
}

Web Crawling Features (Crawler Package)

Install Both Packages

dotnet add package WebSpark.HttpClientUtility
dotnet add package WebSpark.HttpClientUtility.Crawler

Register Services

// Program.cs
builder.Services.AddHttpClientUtility();
builder.Services.AddHttpClientCrawler();  // Adds crawler features

Use Crawler

public class SiteAnalyzer
{
    private readonly ISiteCrawler _crawler;
    
    public SiteAnalyzer(ISiteCrawler crawler) => _crawler = crawler;

    public async Task<CrawlResult> AnalyzeSiteAsync(string url)
    {
        var options = new CrawlerOptions
        {
            MaxDepth = 3,
            MaxPages = 100,
            RespectRobotsTxt = true
        };
        
        return await _crawler.CrawlAsync(url, options);
    }
}

🚀 Features

Base Package Features

Simple API - Intuitive request/response model
Authentication - Bearer token, Basic auth, API key providers
Correlation IDs - Automatic tracking across distributed systems
Structured Logging - Rich context in all log messages
Telemetry - Request timing and performance metrics
Error Handling - Standardized exception processing
Type-Safe - Strongly-typed request and response models
Caching - In-memory response caching (optional)
Resilience - Polly retry and circuit breaker policies (optional)
Concurrent Requests - Parallel request processing
Fire-and-Forget - Background request execution
Streaming - Efficient handling of large responses
OpenTelemetry - Full observability integration (optional)
CURL Export - Generate CURL commands for debugging
Source Link - Step-through debugging with symbol packages
Trimming/AOT Ready - Compatible with Native AOT and IL trimming
Package Validation - Baseline validation ensures stability

Crawler Package Features

Site Crawling - Full website crawling with depth control
Robots.txt - Automatic compliance with robots.txt rules
Sitemap Generation - Create XML sitemaps from crawl results
HTML Parsing - Extract links and metadata with HtmlAgilityPack
SignalR Progress - Real-time crawl progress updates
CSV Export - Export crawl results to CSV files
Performance Tracking - Monitor crawl speed and efficiency

📚 Common Scenarios

Enable Caching

builder.Services.AddHttpClientUtility(options =>
{
    options.EnableCaching = true;
});

// In your service
var request = new HttpRequestResult<Product>
{
    RequestPath = "https://api.example.com/products/123",
    RequestMethod = HttpMethod.Get,
    CacheDurationMinutes = 10  // Cache for 10 minutes
};

Add Resilience (Retry + Circuit Breaker)

builder.Services.AddHttpClientUtility(options =>
{
    options.EnableResilience = true;
  options.ResilienceOptions.MaxRetryAttempts = 3;
    options.ResilienceOptions.RetryDelay = TimeSpan.FromSeconds(2);
});

All Features Enabled

builder.Services.AddHttpClientUtilityWithAllFeatures();

🔄 Upgrading from v1.x

If You DON'T Use Web Crawling

No code changes required! Simply upgrade:

dotnet add package WebSpark.HttpClientUtility --version 2.0.0

Your existing code continues to work exactly as before. All core HTTP features (authentication, caching, resilience, telemetry, etc.) are still in the base package with the same API.

If You DO Use Web Crawling

Three simple steps to migrate:

Step 1: Install the crawler package

dotnet add package WebSpark.HttpClientUtility.Crawler --version 2.0.0

Step 2: Add using directive

using WebSpark.HttpClientUtility.Crawler;

Step 3: Update service registration

// v1.x (old)
services.AddHttpClientUtility();

// v2.0 (new)
services.AddHttpClientUtility();
services.AddHttpClientCrawler();  // Add this line

That's it! Your crawler code (ISiteCrawler, SiteCrawler, SimpleSiteCrawler, etc.) works identically after these changes.

Need Help? See the detailed migration guide or open an issue.

📖 Documentation

Getting Started Guide - Complete walkthrough
Configuration Options - All settings explained
Caching Guide - Response caching strategies
Resilience Guide - Retry and circuit breaker patterns
Web Crawling - Site crawler features
Migration Guide - From raw HttpClient
API Reference - Complete API documentation

🎓 Sample Projects

Explore working examples in the samples directory:

BasicUsage - Simple GET/POST requests
WithCaching - Response caching implementation
WithResilience - Retry and circuit breaker patterns
ConcurrentRequests - Parallel request processing
WebCrawler - Site crawling example

🤝 Contributing

Contributions are welcome! See our Contributing Guide for details.

Fork the repository
Create a feature branch
Add tests for your changes
Ensure all tests pass
Submit a pull request

📊 Project Stats

252+ Unit Tests - 100% passing
Supports .NET 8 LTS, .NET 9, & .NET 10 (Preview)
MIT Licensed - Free for commercial use
Active Maintenance - Regular updates

📦 Related Packages

Package	Purpose	Status
WebSpark.HttpClientUtility.Testing	Test helpers & fakes for unit testing	✅ Available (v2.1.0+)

Testing Package Features:

FakeHttpResponseHandler - Mock HTTP responses without network calls
Fluent API - Easy test setup with ForRequest().RespondWith()
Sequential Responses - Test retry behavior with multiple responses
Request Verification - Assert requests were made correctly
Latency Simulation - Test timeout scenarios

dotnet add package WebSpark.HttpClientUtility.Testing

See the Testing documentation for examples.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🔗 Links

Questions or Issues? Open an issue or start a discussion!