Rayneforge.Stratus.RemoteIndexer

A .NET library for semantic code indexing with hybrid search capabilities, supporting OpenAI and Azure OpenAI embeddings.

Features

• Text Extraction: Intelligent file classification and text extraction with configurable size limits
• Smart Chunking: Token-aware chunking with overlap policies for optimal context preservation
• Embeddings: Support for OpenAI and Azure OpenAI embedding models with caching and batching
• Hybrid Search: Combined vector and lexical (FTS) search with configurable scoring
• Draft/Commit Workflow: Transactional indexing with rollback support via SQLite
• Clustering: K-means clustering with AI-powered labeling for semantic grouping

Installation

dotnet add package Rayneforge.Stratus.RemoteIndexer

Quick Start

Configuration

{
  "Indexer": {
    "WorkspaceDirectory": ".workspace",
    "Sqlite": {
      "ConnectionString": "Data Source=indexer.db"
    },
    "Indexing": {
      "MaxFileBytes": 1000000,
      "Chunking": {
        "MaxCharsPerChunk": 2000,
        "OverlapChars": 200
      }
    },
    "AI": {
      "Profile": "Lite",
      "Provider": "OpenAI",
      "ApiKey": "your-api-key"
    }
  }
}

Basic Usage

using Rayneforge.Stratus.RemoteIndexer.Storage;
using Rayneforge.Stratus.RemoteIndexer.Search;
using Rayneforge.Stratus.RemoteIndexer.Options;

// Configure services
services.Configure<IndexerOptions>(configuration.GetSection("Indexer"));
services.AddSingleton<IIndexStore, SqliteIndexStore>();
services.AddSingleton<HybridSearch>();

// Index files
var store = serviceProvider.GetRequiredService<IIndexStore>();
await store.UpsertFileAsync(new IndexRecord
{
    RelativePath = "src/Program.cs",
    Content = fileContent,
    Chunks = chunks,
    State = "draft"
});

// Commit changes
var commitId = await store.CommitAllAsync();

// Search
var search = serviceProvider.GetRequiredService<HybridSearch>();
var results = await search.SearchAsync("authentication logic", 
    new HybridSearchOptions { MaxResults = 10 });

Architecture

The library is deterministic by design - the same file always produces the same chunks, embeddings, and hashes. This ensures consistency between local and remote indexing operations.

Core Components

• Extraction: File type detection, text extraction with size limits
• Chunking: Token-aware chunkers with overlap for context preservation
• : OpenAI/Azure OpenAI clients with batching and caching