1. The Modern Ingestion Crisis: Why AI Pipelines are Breaking
In the current push for Enterprise Generative AI, secure document federation has emerged as the primary bottleneck for production-grade Retrieval-Augmented Generation (RAG). The "ingestion crisis" is not merely a technical challenge of data movement; it is a fundamental risk to enterprise data integrity. When document pipelines are built as fragmented, proprietary silos, they often strip away the complex security context required to keep data safe.
The most catastrophic failure point in modern RAG systems is the leakage of Access Control Lists (ACLs). Traditional ingestion methods frequently ignore the native permissions of source systems like SharePoint or enterprise content management (ECM) platforms. This results in AI agents surfacing sensitive information to unauthorized users, effectively bypassing decades of established security protocols. Furthermore, the industry is currently hamstrung by proprietary vendor lock-in, where organizations are forced into rigid, non-portable ecosystems that lack the flexibility to adapt to new vector databases or emerging embedding models.
The Failure of Traditional Ingestion
Proprietary Silo Fragility: Incompatible connector architectures create brittle pipelines that are difficult to monitor and impossible to port.
ACL Mapping Gaps: A failure to translate source permissions into search-engine security filters, leading to significant data leakage risks.
Thread-Per-Request Scaling Limits: Legacy I/O models rely on physical OS threads, which incur high memory costs and cannot scale to handle the concurrent demands of modern document streams.
Zero-Trust Deficiencies: A lack of built-in authority translation, making it impossible to maintain a zero-trust posture across distributed AI environments.
Solving these systemic failures requires more than just better software; it requires a new, open-source standard for how data and security context move through the AI stack.
2. The Open Ingestion Standard (OIS): A Manifesto for Data Sovereignty
We are introducing the Open Ingestion Standard (OIS) as a formal specification to decouple content sources from AI destinations. By standardizing the packaging of documents, metadata, and permissions, OIS ensures that data movement is no longer a proprietary black box, thereby eliminating vendor lock-in and returning data sovereignty to the enterprise.
The OIS architecture is grounded in three strategic pillars:
Zero-Trust Security Mapping: OIS mandates the synchronization of native ACLs. By packaging source permissions into a unified format, it ensures that search engines and LLMs strictly enforce document visibility based on the querying user's explicit authorization.
Vendor-Neutrality: The standard formalizes best practices for cross-vendor exchange, ensuring that connectors and configurations remain compatible regardless of the underlying platform.
Event-Decoupled Architecture: Using an asynchronous, message-based approach, OIS allows ingestion to scale horizontally and independently, protecting source systems from performance degradation during massive crawl operations.
The standard defines a Unified Document Payload Schema, focusing on Security SIDs (Security Identifiers) and inheritance rules to maintain security posture. This is paired with a Unified Job Configuration Schema (YAML/JSON), which allows architects to declare schedule execution rates, crawling parameters, and search endpoint mappings in a platform-agnostic way.
Conceptual OIS Payload Example (sample-document.json):
{
"id": "doc-554032-sp",
"source": {
"type": "sharepoint",
"instance": "https://enterprise.sharepoint.com/sites/finance"
},
"content": {
"mimeType": "application/pdf",
"text": "Extracted text content regarding fiscal strategy..."
},
"metadata": {
"title": "Fiscal Strategy 2026",
"author": "Financial Planning Dept",
"version": "1.2"
},
"security": {
"inheritanceEnabled": true,
"permissions": [
{ "sid": "S-1-5-21-3623811015-3361044348-30300820-1013", "allow": true },
{ "sid": "finance-exec-group", "allow": true }
]
}
}A standard is only as effective as its implementation; OpenCrawling serves as the high-performance reference engine for the OIS orchestration logic.
3. OpenCrawling: The High-Performance Java 25 Reference Engine
OpenCrawling is a cloud-native ingestion framework designed for the most demanding enterprise environments. We selected a tech stack comprised of Java 25 and Spring Boot 4 to ensure the engine provides the non-blocking, high-throughput pipeline required for modern RAG orchestration.
The decision to utilize Java 25 was driven by the need for Virtual Threads and Structured Concurrency. These features allow OpenCrawling to maintain massive I/O throughput with a minimal resource footprint. By moving away from expensive physical OS threads, the engine can process thousands of concurrent document streams without the high memory costs associated with traditional thread-per-request models. This ensures that complex ingestion tasks remain observable, resilient, and highly scalable.
By leveraging Spring AI and Spring Boot 4, OpenCrawling provides a modular backend that funnels raw performance into a sophisticated architectural pattern: the Kafka Claim Check.
4. The Ingestion Journey: Deep Dive into the Kafka Claim Check Pattern
The Kafka Claim Check Pattern is critical to our architecture. It prevents message broker congestion by ensuring heavy document payloads never circulate through Kafka topics. Instead, only lightweight references and metadata the "claim checks" are passed between services.
The Technical Ingestion Lifecycle:
Metadata Scanning: Repository connectors scan the target source (e.g., SharePoint or S3).
The Claim Check Record: The crawler publishes a lightweight
IngestionMessageto theopencrawling-ingestiontopic. This message contains a URI to the file on shared storage, preserving broker performance.Text Extraction: The
IngestionConsumerpulls the reference, retrieves the file, and utilizes Apache Tika for deep content and metadata extraction.Semantic Chunking & Embedding: Extracted text is split into chunks and sent to the
opencrawling-chunkstopic. TheEmbeddingConsumerthen calls Ollama (utilizing models likemxbai-embed-largeornomic-embed-text) to generate vectors.Vector Persistence: Embedded chunks are published to the
opencrawling-embeddedtopic. TheVectorStoreWriterConsumerthen persists them into pgvector (Port 5432).
To maximize architectural efficiency, the engine automatically routes embeddings to dimension-specific tables in pgvector, such as vector_store_1024, vector_store_768, and vector_store_384. For massive scale-out, the EmbeddingConsumer can be scaled horizontally and pointed toward a load-balanced cluster of Ollama instances.
5. Advanced Capabilities: Pluggable SPIs and the Secure MCP Server
OpenCrawling is built with an extensibility-first philosophy. Developers can utilize the Pluggable SPI Architecture to implement RepositoryConnector, TransformationConnector, and OutputConnector interfaces, allowing the framework to reach any proprietary data store or specialized vector engine.
A core innovation of this framework is the Secure Model Context Protocol (MCP) Server. This component exposes security-filtered similarity search directly to LLMs. By enforcing user principals and roles at the server-side, the MCP server provides a robust defense against Prompt Injection and Data Leakage. It ensures that AI agents can only retrieve documents that the specific user context is authorized to access, maintaining the zero-trust posture established at the start of the pipeline.
Technical Reference: Infrastructure Ports
PostgreSQL / pgvector: 5432
Redis (Cache & Session): 6379
Ollama (Local Embeddings): 11434
Apache Kafka (Broker): 9092
6. Getting Involved: Building the Future of Open Ingestion
OIS and OpenCrawling are community-driven initiatives licensed under the Apache License 2.0. We believe that secure, high-performance ingestion is the foundation of trustworthy enterprise AI, and that foundation must be open.
We invite architects and developers to join us in defining the next phase of document federation:
Join the Discussion: Contribute to RFCs and the connector roadmap (SharePoint, S3, Qdrant, etc.) on our GitHub Discussions.
Star the Repositories: Support the
opencrawlingandopen-ingestion-standardorganizations on GitHub.Experiment Locally: Use the Docker Compose quickstart to spin up the entire stack including Kafka, pgvector, and Ollama with a single command:
docker compose up -d.
By providing a vendor neutral, zero-trust foundation, OIS and OpenCrawling resolve the modern ingestion crisis, enabling the next generation of enterprise AI to be as secure as it is powerful.
