DocGen: OpenAPI from REST source with Tree-sitter and Haystack

What it is

DocGen is my MSc project at Széchenyi István University. It reads a REST API codebase and writes an OpenAPI 3.0 spec plus searchable endpoint docs.

Hand-written Swagger drifts. Dumping whole files into an LLM overflows the context window and invents parameters. DocGen splits the job: Tree-sitter finds controllers and methods; an LLM reads those files to list injected services and calls; a BFS over that graph gathers the code for one endpoint; another LLM turn writes that endpoint’s OpenAPI fragment; a SwaggerBuilder merges fragments so the model is not responsible for valid YAML.

It ships three ways: a CLI (DocGen-RAG), a multi-tenant FastAPI + React console (DocGen-API / DocGen-UI), and a GitHub Action. Code lives under the DocGen-io org.

Runtime topology Figure 1. React UI talks to FastAPI. FastAPI enqueues a Celery task. The worker runs Haystack pipelines against Weaviate, Postgres, Redis, and SQLite.

Packages

Package	Role
`DocGen-RAG`	Haystack pipelines, Tree-sitter extractors, CLI, evaluation
`DocGen-API`	JWT SaaS API, teams, jobs, revisions, WebSocket logs
`DocGen-API/worker`	Celery: `run_documentation_pipeline` plus search, cluster, examples
`DocGen-UI`	Vite + React console
`DocGen-Action`	GitHub Action wrapping the RAG Docker image

Runtime: UI :5173, API :8000, Postgres :5432, Redis :6379 (Celery broker + log pub/sub), Weaviate :8080, Phoenix :6006. RAG’s compose starts Weaviate. API’s compose starts Postgres, Redis, the API, and the worker. JWT is HS256.

The API never imports DocumentationPipeline on the request path. JobService.submit_job writes a row in Postgres and celery_client.send_task("worker.tasks.run_documentation_pipeline", …). Git credentials stay on the job record. Worker logs go to Redis pub/sub logs:{job_id} and the console tails them over ws://…/ws/logs/{job_id}?token=JWT.

Celery config: JSON serializer, Redis as broker and result backend, task_time_limit=3600. A failed generate task sets the job FAILED, deletes output/<project>/ if it can, and retry(max_retries=0). Search, clustering, examples, and listing endpoints are separate tasks so they do not re-run generate.

Console

The Vite + React console is the job UI. Sidebar: dashboard, jobs, revisions, traces, config, endpoints, prompts, teams. Roles are ADMIN / MAINTAINER / EDITOR / VIEWER. The gallery at the top of this page is a run against AliSaleemHasan/Dartsee with gemini-2.5-flash-lite — one completed job in 56s. Opening a job shows source URL, duration, artifact counts, and the pipeline log stream.

How a documentation job runs

The orchestrator is three Haystack pipelines. The Celery generate task constructs a PipelineContext (project_name, job_id, team_id, user_id) and calls DocumentationPipeline.run.

Source (git or local)
        │
        ▼
 IngestionPipeline     SourceHandler → FileHasher (git hash-object vs SQLite)
        │  only changed files
        ▼
 AnalysisPipeline      ControllerExtractor (Tree-sitter)
                       ASTCodeSplitter + FilesAnalyzer (LLM on whole files)
        │
        ▼
 IndexingPipeline      WeaviateCodeWriter
                       EndpointGraphManager (BFS + SQLite edges)
                       DocumentationCreator (LLM per endpoint graph)
                       DocumentationMerger / SwaggerBuilder
                       EndpointClusterer (K-Means)
                       WeaviateDocWriter (embedded docs)
                       FileHashSaver (commit hashes last)

Figure 2. Ingestion: clone/copy, hash, keep Δ.

Ingest

SourceHandler clones or copies the repo. FileHasher runs git hash-object per file and compares against output/<project>/dependencies.db. Unchanged files never enter analysis. Hashes are not written until FileHashSaver after a successful merge, so a crash does not skip work next time. If hashing a file fails, that file is processed anyway; if the SQLite lookup itself fails, the hasher returns every file.

That is file-level skip. Changing a service file does not automatically re-document every controller that calls it.

Analyze

Tree-sitter .scm queries pick HTTP methods and routes. Strategies isolate syntax:

TypeScript — NestJS @Controller / @Get and Express-style captures
Java — Spring Boot
C# — ASP.NET

LanguageFinder can name Python/Go/PHP from extensions. There are no production .scm extractors for those languages. New queries can be drafted by the SCM pipeline below.

ASTCodeSplitter cuts on semantic nodes rather than character windows, and skips methods already captured as REST endpoints. FilesAnalyzer then sends numbered whole files to the LLM (thread pool, default 5 workers) and asks for injected dependencies, instantiations, and calls as JSON (LLMJsonHandler).

Analysis Figure 3. Tree-sitter controllers plus LLM dependency mapping.

AST vs LLM Figure 4. What the grammar sees versus what the model is asked to name.

Dual store and BFS

Weaviate (WeaviateCodeWriter) stores code documents without embeddings at this stage (compose sets DEFAULT_VECTORIZER_MODULE: none).
SQLite dependencies.db stores file_hashes and directed edges (endpoint_id, caller, target).

EndpointGraphManager starts at each controller endpoint, queue.pop(0), walks analyzer edges, writes the graph. DocumentationCreator then fetch_by_node_id for those nodes. Doc generation is ID lookup, not top-k vector search.

Endpoint docs and merge

Each route is documented on its own graph so one controller’s parameters do not leak into another. Prompts tell the model not to invent query params or response fields, to inline schemas, and to copy method and path exactly. They also ask for HIGH/MEDIUM/LOW security notes in the description when the code shows a real concern.

DocumentationMerger / SwaggerBuilder (openapi: 3.0.3) assemble the spec and drop requestBody on GET/HEAD/DELETE.

Indexing Figure 5. Weaviate write, graphs, per-endpoint docs, merge, hash commit.

Grouping

Endpoint summary embeddings → scikit-learn K-Means. K is chosen by minimizing the Davies–Bouldin index. An LLM names the cluster. Indexing always wires EndpointClusterer; the console can also fire run_clustering_task (GET /endpoints/{project}/clusters). Groups persist as ProjectGrouping in Postgres.

Grouping Figure 6. Vector clustering of documented endpoints.

On the Dartsee run the clusterer named GREETINGS, GAME RETRIEVAL, and GAME STATISTICS. The gallery search box is QueryPipeline (dense + BM25), not the cluster step.

Other pipelines

SCM query generation

Adding a framework means writing Tree-sitter queries. The CLI loop (query-gen):

LLM writes a tiny snippet of the target pattern
Parse to S-expression
LLM drafts an .scm query
Validator compiles it and runs it on the snippet; compiler errors go back for up to 3 repairs
QueryWriter saves into queries/

That is how new languages get extractors. It is not on the main documentation path.

SCM query generation Figure 7. Draft → compile → repair loop for .scm files.

Search

QueryPipeline embeds the question, runs Weaviate dense retrieval and BM25 (top_k default 2), merges by (path, method), filters on team_id / project_name / job_id / user_id. It concatenates two retrievers; it is not Weaviate native hybrid with an alpha. The console hits GET /endpoints/{project}/query?q=…, which enqueues run_semantic_search_task.

Figure 8. Dense + BM25 over endpoint_documentation.

Examples

FetchExampleGenerator runs when someone opens an endpoint (POST /endpoints/{project}/examples), not during indexing. It reads the stored OpenAPI fragment and emits JS fetch, Python requests, and cURL.

Example generator Figure 9. On-demand snippets from a stored schema.

Proposed schema edits go through Artifact Revisions (POST /{team_id}/docs/propose). Approve / reject require a maintainer. Approve patches the Weaviate chunk via update_weaviate_documentation_chunk. The Dartsee revision in the gallery is a summary-string tweak (its ID → its Game ID).

Traces in the sidebar talk to Phoenix (VITE_PHOENIX_HOST, default :6006). The Celery parent process launches Phoenix; each worker child instruments Haystack.

Generated spec vs Dartsee’s own Swagger

Dartsee already has a hand-written OpenAPI. Same six GET paths. The generated file is a flat default tag; the hand-written spec groups Games / Heatmap / Statistics. Summaries differ.

GET /games is the close-up. Hand-written query params: page (default 1), type, limit (default 20). Generated adds skip (default 0) and sets limit default 10. The long description walks GamesController → GamesService.findAll / count → PaginatedResponseDto. Response examples invent payload data (Cyberpunk 2077, Geralt of Rivia).

Evaluation (RealWorld / Conduit)

Structural eval lives in DocGen-RAG/evaluation/. Each row clones a git URL, runs DocumentationPipeline, and compares output/<repo>/swagger.json to the official RealWorld OpenAPI (12 paths, 19 methods). Matching is suffix-based after normalizing {param} / :param. Validity is openapi-spec-validator.

The Gemini table (evaluation/data/evaluation_results.csv) is 8 repos × 3 models. File counts processed: 27–69.

Language	Framework	Repo
TypeScript	Express & Firestore	marcusmonteirodesouza/typescript-firestore-terraform-realworld-backend
TypeScript	Express + PostgreSQL + Drizzle	AhmedFawzy31/realworld-express-postgresql-drizzle
TypeScript	NestJS + Neo4j	neo4j-examples/nestjs-neo4j-realworld-example
TypeScript	NestJS + TypeORM	lujakob/nestjs-realworld-example-app
Java	Spring Boot + MyBatis	joyheros/realworld
Java	Spring Boot + JPA + PostgreSQL	marcusmonteirodesouza/realworld-backend-spring-boot-java-keycloak-postgresql
C#	ASP.NET Core example 1	gothinkster/aspnetcore-realworld-example-app
C#	ASP.NET Core example 2	TomislavVinkovic/realworld-api-asp-net-10

Averages:

Model	Avg method F1	Avg path F1	Avg time (s)	Valid OpenAPI	Completeness (gen/expected methods)
gemini-2.5-flash	0.755	0.792	415	100%	0.96
gemini-2.5-flash-lite	0.818	0.836	110	88%	1.09
gemini-2.5-pro	0.821	0.846	862	88%	1.09

Flash-lite matches Pro on F1 and is 862 / 110 ≈ 7.8× faster on this set. Completeness above 1 means extra methods versus the 19-method ground truth (Spring MyBatis is the usual over-generator). Express + Drizzle is the consistent miss: path precision 0.33, recall 0.25 on flash-lite and flash (pro: 0.40 / 0.33).

Precision / recall by language Figure 10. Method and path P/R by language on the 8-repo set.

F1 by model Figure 11. Method F1. Lite ≈ Pro; Flash a bit lower.

Execution time Figure 12. Wall time is dominated by the model, not file count.

Scalability Figure 13. 27–69 files. Time still tracks the LLM, not N.

OpenAPI validity Figure 14. Spec validity after SwaggerBuilder merge.

GitHub Action

DocGen-Action is a Docker wrapper around ghcr.io/DocGen-io/docgen-rag-action:v1. It wants a persistent Weaviate URL (localhost is rejected) and a Gemini key or Vertex/Ollama. Default model is gemini-2.5-flash-lite. fetch-depth: 0 so incremental hash checks work.

- uses: DocGen-io/docgen-action@main
  with:
    path: 'src/api'
    weaviate-url: ${{ secrets.WEAVIATE_URL }}
    gemini-api-key: ${{ secrets.GEMINI_API_KEY }}

Run it locally

Weaviate (from DocGen-RAG):

cd DocGen-RAG
docker compose up -d
uv sync --extra cli
uv run docgen
uv run python evaluation/evaluate.py

SaaS API + worker (from DocGen-API; Postgres :5432, Redis :6379, API :8000):

cd DocGen-API
cp .env.example .env
docker compose up --build
# or without Docker:
uv sync
uv run alembic upgrade head
uv run uvicorn api.main:app --reload --port 8000

UI: cd DocGen-UI && pnpm install && pnpm dev → http://localhost:5173 (VITE_API_URL defaults to http://localhost:8000/api/v1). Phoenix UI :6006. API docs :8000/docs.