Local inference with centralized control
Model execution and customer knowledge remain at the edge, while a separate management plane provides fleet visibility and controlled lifecycle operations.
Production AI and Private Infrastructure
VaultAI - Private Enterprise AI and Knowledge Platform
A decentralized private-AI platform combining a customized llama.cpp/CUDA runtime, RAG, dynamic agents, secure WebRTC connectivity, and centrally managed edge inference.
Private AIInference runtimesDistributed edge systemsSecure connectivityKnowledge and agent systems
The system
VaultAI is a decentralized private and edge AI platform built for organizations that need inference and enterprise knowledge workflows inside customer-controlled environments. Inference nodes can operate in offices, branches, homes, remote sites, or other private locations while a central console provides controlled administration across the distributed fleet.
Each edge appliance combines local NVIDIA compute, a customized inference runtime, a TypeScript application layer, dynamic agents, retrieval-augmented generation, and local vector search. Users interact with the private node through an application path designed for direct, encrypted communication rather than exposing the inference appliance as a public Internet service.
The architecture separates the user-facing data path from the management plane. Interactive AI traffic reaches the selected edge environment through an end-to-end encrypted WebRTC DataChannel, while machine administration uses mutually authenticated device-management channels. This allows local inference resources to remain behind ordinary private networking and NAT while still being securely reachable and centrally managed.
Engineering relationship
OPTIME designed and developed the product platform from scratch. Approximately 10 engineers contributed across the broader program, covering native C++ and CUDA runtime engineering, distributed systems, backend services, real-time connectivity, AI applications, knowledge retrieval, security, and device operations.
The work went substantially beyond integrating an existing model API. OPTIME modified llama.cpp, implemented custom CUDA kernels and runtime controls, built the AI proxy and agent infrastructure, created secure user-to-device and machine-management paths, and integrated the software with NVIDIA DGX Spark and NVIDIA Jetson Thor T5000 deployment targets.
The product has been delivered to the customer. The customer is now deploying it with pilot customers; the case does not claim broad adoption or commercial scale.
Engineering constraints
- Run useful language-model workloads on distributed edge appliances with finite GPU memory and different local demand patterns.
- Maintain responsive multi-session behavior while coordinating KV caches, batching, memory, scheduling, LoRAs, and runtime state.
- Reach private devices behind ordinary NAT without requiring the inference runtime to be exposed directly to the public Internet.
- Keep user-to-device content protected end to end while authenticating machines independently for centralized administration.
- Connect enterprise knowledge and agent workflows to local inference without making RAG the entire product architecture.
- Operate a fleet of private nodes with controlled configuration, runtime management, and lifecycle ownership.
What OPTIME engineered
- A modified llama.cpp runtime in C++ with additional REST APIs for KV-cache management, sessions, LoRAs, runtime control, and operational integration.
- Custom CUDA kernels plus GPU execution work focused on memory use, batching, cache behavior, session scheduling, and reliable concurrent operation.
- LoRA loading and management so specialized adapters can be controlled without treating each adaptation as a separate appliance stack.
- A custom TypeScript AI proxy coordinating the application layer, runtime sessions, model access, dynamic agent construction, and retrieval workflows.
- RAG pipelines backed by pgvector for private knowledge retrieval and evidence-grounded responses inside the customer-controlled environment.
- Direct user-to-edge communication over WebRTC DataChannel with end-to-end encrypted application traffic.
- mTLS-authenticated machine-to-machine management between the central console and distributed edge nodes.
- Deployment integration for NVIDIA DGX Spark and NVIDIA Jetson Thor T5000 compute environments.
Architecture
User application
The enterprise-facing application selects and interacts with a customer-controlled edge AI environment.
Secure WebRTC DataChannel
End-to-end encrypted interactive traffic reaches the edge node without requiring the inference appliance to be directly exposed.
Edge AI appliance
A privately deployed DGX Spark or Jetson Thor T5000 node provides local compute and storage.
TypeScript AI proxy
The proxy coordinates application requests, agents, sessions, retrieval, and runtime-control functions.
Dynamic agents and RAG
Agent workflows retrieve controlled enterprise context from pgvector-backed knowledge stores.
Modified llama.cpp runtime
C++ runtime extensions manage sessions, KV caches, LoRAs, batching, and inference execution.
CUDA and NVIDIA compute
Custom kernels and GPU-aware runtime behavior execute model workloads on the selected NVIDIA platform.
Separate management plane
The central console manages distributed nodes through mTLS-authenticated machine channels, independently of user content.
Key engineering decisions
Direct application-to-device communication
WebRTC DataChannel provides an interactive path to a node behind private networking and NAT without publishing the inference runtime as a conventional public endpoint.
Runtime optimization as a product layer
KV-cache behavior, memory batching, session scheduling, LoRA management, and runtime control were engineered together instead of being left to an unmodified serving binary.
Separate user and machine trust paths
User content is protected end to end, while mTLS authenticates devices for central management. The two paths have different responsibilities and security boundaries.
Platform evolution
The runtime and application layers were designed so models, LoRAs, agents, knowledge collections, and edge hardware can evolve without turning every node into a one-off deployment.
Central management supports a distributed operating model while inference and customer data remain within the chosen private environment.
Current capability
OPTIME delivered a decentralized private-AI platform built around a customized C++/CUDA inference runtime, centrally managed edge nodes, secure user-to-device connectivity, RAG, and dynamically constructed AI agents.
The customer is currently deploying the platform with pilot customers. No claim is made about broad production adoption, customer count, or commercial scale.
Verified result
The product platform was completed and delivered to the customer for pilot-customer deployment.
Technology & Engineering Role
- C++
- Performance-sensitive inference runtime, session, cache, and control engineering.
- CUDA
- Custom GPU kernels and hardware-aware execution optimization.
- llama.cpp
- Customized model runtime with additional session, KV-cache, LoRA, and REST control.
- TypeScript
- AI proxy, runtime orchestration, dynamic agents, and application integration.
- WebRTC DataChannel
- Direct interactive user-to-edge connectivity through private networking and NAT.
- pgvector
- Local vector retrieval for private RAG and enterprise knowledge workflows.
- mTLS
- Machine authentication for central management of distributed edge nodes.
- NVIDIA DGX Spark
- Private inference compute option for higher-capacity edge environments.
- NVIDIA Jetson Thor T5000
- Compact edge inference target for distributed private deployments.
- LoRA
- Managed model adaptation within the shared runtime environment.
- RAG
- Evidence-grounded access to customer-controlled knowledge.