Background Workers

Three production-ready background workers for async operations, auto-tuning, and index maintenance. Built with PostgreSQL background worker framework and tenant isolation.

neuranq - Async Job Queue

Asynchronous job queue executor with SKIP LOCKED, rate limits, retries, and poison job handling. Perfect for batch embedding generation, model inference, and long-running operations.

Features

SKIP LOCKED Queuing

Concurrent job processing without lock contention

Rate Limiting

Per-tenant QPS and token budgets

Auto Retry

Exponential backoff for transient failures

Poison Jobs

Dead letter queue for failed jobs

Configuration

# postgresql.conf
neurondb.neuranq_enabled = on
neurondb.neuranq_naptime = 1000        # Check queue every 1 second
neurondb.neuranq_batch_size = 100      # Process 100 jobs per cycle
neurondb.neuranq_max_retries = 3       # Retry failed jobs 3 times

neuranmon - Auto-Tuner

Automatically tunes index parameters based on query performance and SLO targets. Adjusts ef_search, rotates caches, and tracks recall@k metrics.

What It Does

Monitors query latency and adjusts ef_search for HNSW indexes
Tracks recall@10 and recall@100 to measure search quality
Rotates embedding and model caches based on access patterns
Records performance metrics for trend analysis

# postgresql.conf
neurondb.neuranmon_enabled = on
neurondb.neuranmon_naptime = 60000     # Check every 60 seconds
neurondb.neuranmon_target_latency_ms = 10  # Target latency SLO
neurondb.neuranmon_min_recall = 0.95   # Minimum 95% recall

neurandefrag - Index Maintenance

Automatic index maintenance: compacts HNSW graphs, re-levels layers, prunes tombstones, and schedules rebuilds for optimal performance.

Maintenance Tasks

Graph Compaction

Removes fragmentation from HNSW graphs after deletes and updates

Layer Re-leveling

Rebalances hierarchical layers for optimal search performance

Tombstone Pruning

Removes deleted vector markers to reclaim space

Rebuild Scheduling

Automatically rebuilds indexes when fragmentation exceeds threshold

# postgresql.conf
neurondb.neurandefrag_enabled = on
neurondb.neurandefrag_naptime = 300000     # Check every 5 minutes
neurondb.neurandefrag_fragmentation_threshold = 0.30  # Rebuild at 30%

Monitor Workers

-- View all worker status
SELECT * FROM neurondb_worker_status();

-- Returns:
--  worker_name  | status  |      last_run       | jobs_processed | avg_runtime_ms
-- --------------+---------+---------------------+----------------+---------------
--  neuranq      | running | 2025-11-03 12:30:15 |      427       |      12.3
--  neuranmon    | running | 2025-11-03 12:30:10 |       89       |      45.7
--  neurandefrag | running | 2025-11-03 12:28:00 |       23       |     234.8