DocGen: Deterministic Neuro-Symbolic AST-RAG Engine

Executive Overview

DocGen (developed as an MSc Thesis project at Széchenyi István University) is an enterprise API documentation generation platform designed to eliminate documentation drift in continuous software delivery.

Traditional approaches to API documentation fail in two distinct ways:

1. Manual Documentation: Hand-written specifications (OpenAPI/Swagger, Postman collections) drift out of sync with production code within weeks. Industry surveys show that over 67% of software teams cite limited time as the primary bottleneck to keeping documentation updated.
2. Pure Generative / Standard RAG: Feeding raw files or naive text chunks into Large Language Models (LLMs) triggers severe context window overflow, semantic horizon blindness (inability to see cross-file service calls), and structural hallucinations in parameter types and HTTP return codes.

DocGen resolves this tension using a Neuro-Symbolic architecture. It couples the deterministic, grammar-aware precision of Tree-sitter Abstract Syntax Trees (ASTs) with the semantic reasoning capabilities of modern LLMs, orchestrated through Haystack 2.0, Weaviate, and an asynchronous Celery/Redis task queue.

Figure 1: High-level architectural overview of the Main DocGen service, showcasing the separation between client access layers, ingestion filters, symbolic analysis, relational graph orchestration, and Weaviate persistence.

---

Comparative Architectural Analysis

To understand the necessity of this hybrid paradigm, consider how DocGen contrasts with existing industry baselines:

Capability	Manual Practices (Swagger / Postman)	Pure LLM / Standard RAG (e.g. LRASGen)	DocGen (Neuro-Symbolic)
Boundary Detection	Human-defined (Manual)	Probabilistic (Prone to hallucination)	Deterministic (Tree-sitter AST S-expressions)
Update Complexity	O(N) Manual overhead	O(N) Full re-indexing	O(Δ) Blast Radius Logic
Large Codebase Scalability	Low (High maintenance friction)	Low (Context window overflow)	High (Atomic Endpoint Routing)
Execution Flow Awareness	None	Semantic Guesswork	Absolute (Relational BFS Traversal)
Security Auditing	External SAST tooling required	Generic prompt instructions	Integrated OWASP SAST during call stack walk

---

Core System Architecture: The 7-Stage Documentation Pipeline

The primary documentation engine runs through seven strictly bounded, deterministic-to-neural processing stages.

Figure 2: Execution lifecycle of the core 7-stage documentation pipeline.

+----------------------------------------------------------------------------------------------------+
|                                    DocGen 7-Stage Pipeline                                         |
|                                                                                                    |
|  [ Repository Ingestion ] ---> [ FileHasher: SHA-256 ] ──(Unchanged? Skip)                         |
|                                       │                                                            |
|                                       ▼ (Modified Files: Δ)                                        |
|                          [ Tree-sitter SCM Queries ] ──(Strategy Pattern: TS, Java, C#)            |
|                                       │                                                            |
|                      ┌────────────────┴────────────────┐                                           |
|                      ▼                                 ▼                                           |
|          [ Controller Extractor ]             [ FilesAnalyzer (AST Chunks) ]                       |
|          (Routes, Methods, Tags)              (Constructor Injection, DTOs, Calls)                 |
|                      │                                 │                                           |
|                      ▼                                 ▼                                           |
|          [ WeaviateCodeWriter ]              [ SQLite: dependencies.db ]                           |
|          (NoSQL Document Storage)             (Relational Call-Graph Edges)                        |
|                      │                                 │                                           |
|                      └────────────────┬────────────────┘                                           |
|                                       ▼                                                            |
|                       [ Recursive BFS Graph Traversal ]                                            |
|                       (Resolves Controller → Service → Repo → DTO)                                 |
|                                       │                                                            |
|                                       ▼                                                            |
|                       [ DocumentationCreator (LLM) ]                                               |
|                       (Endpoint-by-Endpoint + OWASP SAST)                                          |
|                                       │                                                            |
|                                       ▼                                                            |
|                       [ Vector K-Means & Semantic Naming ]                                         |
|                       (Davies-Bouldin Auto-K Clustering)                                           |
|                                       │                                                            |
|                                       ▼                                                            |
|                       [ SwaggerBuilder & WeaviateDocWriter ]                                       |
|                       (RFC-Compliant OpenAPI 3.0 + Natural Language Search)                        |
+----------------------------------------------------------------------------------------------------+

Stage 1: Repository Ingestion & The O(Δ) Update Paradigm

When code is submitted via CLI, Webhook, or SaaS, the SourceHandler clones or syncs the repository into an isolated workspace. The LanguageFinder determines the language profile (.ts, .java, .cs).

Before any costly parsing occurs, the FileHasher calculates cryptographic hashes of every file, comparing them against the local SQLite database.

Figure 3: Ingestion Pipeline architecture detailing source retrieval and cryptographic change detection.

[!NOTE] Let Δ represent the strictly isolated subset of files modified, added, or deleted in a given commit (Δ ⊆ N). By computing the exact “Blast Radius” of a commit, DocGen skips up to 95%+ of AST parsing and LLM operations, reducing re-indexing time on production codebases from 120s down to under 2 seconds.

Stage 2: The Symbolic Layer (Tree-Sitter Analysis Pipeline)

Only files identified in Δ enter the AnalysisPipeline. Here, the concrete syntax is converted into Abstract Syntax Trees using Tree-sitter.

Figure 4: Analysis Pipeline architecture illustrating deterministic AST extraction and neural dependency mapping.

1. SCM Query Controller Extraction: Language-specific Tree-sitter Scheme (.scm) queries isolate API entry points without text pattern matching:

(method_definition
  decorator: (decorator 
    (call_expression 
      function: (identifier) @http.method
      arguments: (arguments (string) @http.route)
    )
  )
  name: (property_identifier) @function.name
)

2. Strategy Pattern Extensibility: Extraction is decoupled from language syntax via the Strategy Pattern (TypeScriptStrategy, JavaStrategy, CSharpStrategy), isolating framework-specific annotations (e.g., @GetMapping vs [HttpGet] vs @Get).
3. ASTCodeSplitter: Unlike naive RAG chunkers that split code arbitrarily by character count (often severing a loop or function mid-syntax), ASTCodeSplitter slices exclusively along intact semantic nodes (class_declaration, method_definition).
4. FilesAnalyzer: The LLM inspects bounded AST chunks to identify injected dependencies, service instantiations, and method delegations, generating strict JSON payloads validated by LLMJsonHandler.

Stage 3: Dual-Storage Architecture (Decoupled Relational & Vector)

To optimize data retrieval without synchronization locks, storage duties are split across two dedicated systems:

• Unstructured Code Storage (WeaviateCodeWriter): Ingests raw code chunks into Weaviate purely as high-throughput NoSQL document payloads—bypassing embedding generation entirely at this stage to save significant compute.
• Relational Call Graph (dependencies.db): An isolated SQLite database records directional edges connecting controllers, services, repositories, and DTOs.

Stage 4: Recursive Dependency Resolution (Cross-Referencing BFS)

Standard RAG suffers from the Semantic Horizon: an endpoint in UserController.getUser might delegate database access and permission checks through three layers of services and interfaces. Top-K semantic vector search often retrieves unrelated snippets that merely share variable names.

DocGen eliminates this guesswork by running a Breadth-First Search (BFS) across the SQLite relational graph, using exact node IDs to pull complete code blocks directly from Weaviate:

def resolve_endpoint_context(root_id: str, sqlite_db, weaviate_client) -> list[str]:
    queue = [root_id]
    visited = set()
    aggregated_context = []

    while queue:
        current_id = queue.pop(0)
        if current_id in visited:
            continue
        visited.add(current_id)

        # 1. Fetch exact deterministic code chunk by ID
        chunk = weaviate_client.fetch_by_id(current_id)
        if chunk:
            aggregated_context.append(chunk.source_code)

        # 2. Resolve structural dependencies from SQLite
        dependencies = sqlite_db.get_edges(current_id)
        for dep_id in dependencies:
            queue.append(dep_id)

    return aggregated_context

Stage 5: Neural Documentation Synthesis & Automated Security Auditing

The aggregated call stack is passed to the DocumentationCreator.

• Atomic Endpoint-by-Endpoint Generation: Rather than feeding an entire controller into the LLM, each route is synthesized independently. This preserves context focus, eliminates token overflow, and mathematically prevents parameter contamination across routes.
• Strict Anti-Hallucination Boundaries: Prompt directives explicitly forbid the model from inventing missing query parameters or conjecturing response schemas not grounded in the code context.
• Integrated OWASP SAST Auditing: As the LLM inspects the execution chain, it actively scans for API vulnerabilities (such as OWASP API Top 10 risks: missing authentication guards, exposed secrets, SQL/NoSQL injection risks, or unvalidated parameters), embedding severity-graded alerts (HIGH, MEDIUM, LOW) directly into the route’s OpenAPI documentation field.

Stage 6: Mathematical Clustering & Semantic Naming

Naive documentation generators group endpoints solely based on directory paths or URL prefixes. In enterprise codebases, this replicates messy legacy folder structures and separates related microservice features across repositories.

DocGen implements an automated mathematical grouping pipeline:

1. Vector-Based K-Means: Endpoint summary embeddings are pulled from Weaviate and clustered using Scikit-Learn’s K-Means algorithm.
2. Davies-Bouldin Auto-K Optimization: The system dynamically selects the optimal number of clusters K by minimizing the Davies-Bouldin Index score, ensuring maximum intra-cluster cohesion and inter-cluster separation.
3. Semantic Naming via LLM: The model reads the clustered endpoint summaries and generates a unified human-readable category name (e.g., grouping /users/auth and /billing/customer under “Account Management & Billing”).

Stage 7: Final Assembly, SwaggerBuilder, & Semantic Indexing

The fragmented endpoint specifications are assembled into a master OpenAPI 3.0 document.

Figure 5: Architecture of the Indexing Pipeline, displaying final assembly, Swagger building, and semantic indexing.

• SwaggerBuilder: Programmatic utilities (add_path, add_schema, add_parameter) merge JSON fragments without LLM formatting flaws, deduplicating shared DTO schemas and guaranteeing syntax compliance with OpenAPI 3.0 standards.
• Semantic Indexing (WeaviateDocWriter): The finalized documentation is embedded into Weaviate using Haystack’s DocumentWriter with asynchronous gRPC batching.
• Transactional Commit (FileHashSaver): Git hashes are updated in SQLite only after successful completion, ensuring resilience against pipeline interruptions.

---

Supplementary Architectural Pipelines

Beyond the primary generation flow, DocGen introduces three specialized pipelines that make the platform self-expanding and developer-centric.

1. The SCM Query Generation Pipeline (Autonomous Self-Expansion)

Adding support for a new programming language or framework usually requires manually authoring complex Tree-sitter SCM queries. Because Tree-sitter syntax is extremely sensitive to exact AST node nomenclature, LLMs frequently hallucinate query syntax.

DocGen overcomes this using an automated Micro-Snippet Test-Driven Development (TDD) Loop:

Figure 6: Architecture of the Query Generation Pipeline featuring the self-correcting TDD repair loop.

1. MicroSnippetGenerator: The LLM generates a minimal synthetic code snippet of the target framework pattern (e.g., a bare Java @GetMapping method).
2. ASTQueryExtractor: The snippet is parsed into an S-expression representation.
3. QueryGenerator: The LLM drafts an SCM query bounded by strict grammar constraints and reference templates.
4. QueryValidator & Semantic Match Loop: The validator compiles and executes the candidate query against the synthetic AST. If it fails to compile or returns zero captured nodes, the compiler error is returned to QueryRepair for iterative correction (up to 3 retries).
5. QueryWriter: The verified SCM query is saved into the production library, enabling DocGen to autonomously learn new languages.

2. The Information Retrieval Pipeline (Hybrid Search Engine)

DocGen allows developers to query the API documentation using natural language (e.g., “How do I reset an expired user password?”).

Figure 7: Architecture of the Semantic Query Pipeline combining dense vector similarity with sparse lexical matching.

• Hybrid Search Architecture: Queries are vectorized via TextEmbedder. Weaviate executes concurrent dense vector search (semantic intent) and sparse BM25 retrieval (exact technical terms like HTTP status codes or path slugs), balanced by an alpha tuning parameter.
• Metadata Confinement: Strict repository isolation filters guarantee zero leakage across distinct project workspaces.

3. On-Demand Example Generation Pipeline

Generating multi-language code snippets (cURL, JavaScript Fetch, Python Requests) for every endpoint during batch indexing wastes significant compute.

DocGen delegates this to an on-demand service:

• Operates strictly when a developer selects an endpoint in the interactive portal.
• The FetchExampleGenerator pulls the validated OpenAPI schema directly from Weaviate, prompts the LLM to populate realistic mock values matching the parameter types, and formats the output into syntax-highlighted snippets via LLMJsonHandler.

---

Empirical Benchmarks & Quantitative Evaluation

DocGen was evaluated against standard RealWorld (Conduit) benchmark repositories spanning TypeScript (NestJS, Express), Java (Spring Boot), and C# (ASP.NET Core) in monolithic and microservice configurations. Ground truth was established against the official RealWorld OpenAPI specification.

1. Extraction Precision and Recall Across Models

We measured Path Recall (R, ratio of discovered endpoints to expected endpoints) and Path Precision (P, ratio of valid discovered endpoints to total generated endpoints, measuring hallucination resistance).

Figure 8: Consolidated metrics for Path and Method Precision and Recall across multiple LLMs.

Figure 9: Method Precision and Recall distribution across TypeScript, Java, and C#.

Granular Metric Breakdowns by Model

The deterministic AST boundary guarantees near-identical precision across both small and large models, as shown in the individual parameter breakdowns below:

2. The Efficiency Breakthrough: Small Model Superiority

A central empirical finding of this research is that lightweight models (such as Gemini-2.5-Flash-Lite and Qwen-2.5-Coder) achieve 90–100% precision—matching the performance of Gemini-2.5-Pro while running 8 to 10 times faster.

Figure 10: Execution time analysis for full repository indexing across different frameworks and models.

Because the symbolic AST layer deterministically isolates the exact code execution path, the LLM only needs to perform semantic translation rather than broad architectural reasoning, eliminating the need for expensive high-parameter models.

3. Pipeline Scalability vs. Codebase Size

We evaluated indexing latency across variable repository sizes (file counts ranging from 25 to 70+ files):

Figure 11: Scalability analysis showing execution time and effectiveness vs. codebase size.

The data indicates that execution time remains dominated by LLM inference latency rather than codebase volume. Thanks to the O(Δ) update architecture, incremental maintenance runs remain flat regardless of total repository size.

---

Enterprise Platform Implementation & Deployment

DocGen is engineered for enterprise production environments and supports three primary operational modalities:

1. Local Terminal CLI (Privacy-First): Executes natively on developer workstations using local Tree-sitter binaries and a containerized Weaviate instance, ensuring proprietary intellectual property never leaves the network.
2. Asynchronous SaaS Platform (Celery + Redis + PostgreSQL):
   • Non-Blocking REST API: FastAPI handles authentication, team administration, and task dispatching.
   • Horizontal Celery Pool: Distributed workers handle AST analysis and vector indexing asynchronously without blocking user traffic.
   • Real-Time WebSocket Feedback: Progress states stream live to the React dashboard.
3. Automated CI/CD Integration: Native GitHub Action triggers incremental documentation updates on every merged pull request.

# 1. Spin up the distributed enterprise stack
docker compose up -d

# 2. Access endpoints and observability:
# - FastAPI Swagger Interface: http://localhost:8000/docs
# - Phoenix OpenTelemetry Tracing: http://localhost:6006
# - Weaviate Vector Console: http://localhost:8080/v1/meta