ADR-071: Documentation Pipeline Architecture

Status

Proposed

Date

2025-12-27

Context

Documentation is often treated as an afterthought - generated from code or created separately. This leads to:

1. Inconsistency - Different styles, formats, levels of detail
2. Staleness - Docs fall behind as code evolves
3. Duplication - Same info in multiple places (code comments, wiki, KB)
4. Manual effort - Human-intensive doc creation and maintenance

We need a documentation-first pipeline where:

• Docs are PRIMARY - Not derived from code, but the source of truth
• AI assists - LLM generates drafts from context
• Humans approve - Review before publishing
• RAG is DERIVED - Embeddings generated from approved docs

This ADR focuses on the documentation generation pipeline. Related:

• ADR-070 - Multi-domain RAG
• ADR-072 - Qute Web docs server
• ADR-073 - Security layer

Decision

1. Documentation Types

2. Pipeline Flow

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│  TRIGGER    │────▶│   GENERATE  │────▶│   APPROVE   │
│             │     │             │     │             │
│ • Manual    │     │ • AI draft  │     │ • Review    │
│ • Schedule  │     │ • Qute tmpl │     │ • Edit      │
│ • AD change │     │ • JSON data │     │ • Publish   │
└─────────────┘     └─────────────┘     └─────────────┘
                                               │
                                               ▼
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   INGEST    │◀────│   COMMIT    │◀────│   FORMAT    │
│             │     │             │     │             │
│ • Embed     │     │ • Git push  │     │ • Markdown  │
│ • Index     │     │ • Version   │     │ • HTML      │
│ • Publish   │     │ • Audit     │     │ • JSON      │
└─────────────┘     └─────────────┘     └─────────────┘

3. Approval Queue

Documents requiring human review are placed in an approval queue:

CREATE TABLE doc_approval_queue (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    entity_type VARCHAR(50) NOT NULL,    -- 'ad_table', 'process', 'guide'
    entity_id VARCHAR(255) NOT NULL,     -- 'C_Order', 'Complete Document'
    domain VARCHAR(50) NOT NULL,         -- 'idempiere', 'cloudempiere'
    generated_content JSONB NOT NULL,    -- AI-generated draft
    confidence_score DECIMAL(3,2),       -- 0.00-1.00
    status VARCHAR(20) DEFAULT 'PENDING',-- PENDING, APPROVED, REJECTED
    reviewer_id UUID,
    reviewer_notes TEXT,
    created_at TIMESTAMP DEFAULT now(),
    reviewed_at TIMESTAMP
);

Auto-approval rules:

• confidence_score >= 0.85 AND entity_type = 'ad_table' → AUTO_APPROVED
• All others → PENDING (require human review)

4. Qute Templates

Templates define the structure of generated documentation:

src/main/resources/templates/
├── docs/
│   ├── table.qute.md        # AD_Table documentation
│   ├── process.qute.md      # Process documentation
│   ├── callout.qute.md      # Callout documentation
│   └── validator.qute.md    # Validator documentation
└── web/
    ├── tableView.html       # Web rendering template
    └── processList.html     # Web list template

Template example (table.qute.md):

# {tableName}

{description}

## Columns

| Column | Type | Description |
|--------|------|-------------|
{#for col in columns}
| {col.name} | {col.type} | {col.description} |
{/for}

## Related

- Window: [{windowName}](/docs/windows/{windowName})
- Processes: {#for p in processes}[{p.name}](/docs/processes/{p.name}){/for}

5. Generation Service

@ApplicationScoped
public class DocGenerationService {

    @Inject
    TemplateInstance tableTemplate;

    @Inject
    LlmService llmService;

    public GeneratedDoc generateTableDoc(String tableName, String language) {
        // 1. Fetch AD metadata
        TableDoc metadata = tableDocGenerator.generate(tableName, language);

        // 2. Generate descriptions via LLM if missing
        if (metadata.description() == null) {
            String aiDescription = llmService.generateDescription(
                "table", tableName, metadata.columns()
            );
            metadata = metadata.withDescription(aiDescription);
        }

        // 3. Render template
        String content = tableTemplate
            .data("doc", metadata)
            .render();

        // 4. Calculate confidence
        double confidence = calculateConfidence(metadata);

        return new GeneratedDoc(content, confidence);
    }
}

6. Workspace Structure

Documentation organized by domain and type:

docs/cloudempiere-docs-template/
├── ad-reference/
│   ├── tables/          # Generated from AD_Table
│   ├── windows/         # Generated from AD_Window
│   ├── processes/       # Generated from AD_Process
│   ├── callouts/        # Hybrid (AI + human)
│   └── validators/      # Hybrid (AI + human)
├── help-guides/
│   ├── sales/           # Human authored
│   ├── purchasing/      # Human authored
│   └── inventory/       # Human authored
├── architecture/
│   ├── decisions/       # ADRs
│   └── patterns/        # Design patterns
└── support-kb/
    └── [K_Entry exports] # From iDempiere K_Entry table

Implementation Status

Consequences

Benefits

• Consistency - All docs follow templates
• Quality - Human approval ensures accuracy
• Efficiency - AI generates drafts
• Traceability - Git history, approval audit

Drawbacks

• Latency - Approval adds delay
• Complexity - Multiple stages to manage
• Dependency - Requires LLM availability

Next Steps

1. Implement DocGenerationService
2. Create approval UI in Chat API
3. Add Git commit integration
4. Implement auto-approval rules
5. Wire into RAG ingestion pipeline

References

• Source: docs/rag-complex-for-quarkus/documentation-pipeline-architecture.md
• Source: docs/rag-complex-for-quarkus/llm-direct-documentation.md
• Parent: ADR-068