ADR-063: Hub Storage Adapter Service

<!-- MADR 3.0 Template - Markdown Any Decision Records -->

Status

Proposed

Date

2025-12-19

Deciders

• Norbert Bede

Context and Problem Statement

iDempiere Hub needs persistent storage for various services beyond just centralized ID management. Currently, each service implements its own storage mechanism, leading to:

• Duplicated connection/configuration code
• Inconsistent error handling
• No flexibility to change storage backends
• Tight coupling between domain logic and storage implementation

Services requiring persistent storage include:

• Centralized ID Management (ADR-062) - Entity type sequences and registry
• Configuration Management - User preferences, credentials, settings
• Cache Layer - Query results, RAG embeddings cache
• State Management - Wizard state, session data

A generic storage adapter would provide a unified interface for all these use cases while allowing pluggable backends (PostgreSQL, AWS S3/DynamoDB, Redis, etc.).

Decision Drivers

• Separation of Concerns: Domain services should not know about storage implementation details
• Flexibility: Support multiple backends for different deployment scenarios (local dev, AWS, on-premise)
• Reusability: Single adapter used by multiple services
• Testability: Easy to mock storage in unit tests
• Cloud-Native: Support serverless storage options (S3, DynamoDB) for CloudEmpiere deployment
• Simplicity: Keep interface minimal - not a full ORM

Considered Options

1. Service-Specific Storage - Each service manages its own storage
2. Generic Key-Value Adapter - Unified interface with pluggable backends
3. Full ORM Integration - Use Hibernate/Panache for everything
4. External Service - Separate microservice for storage

Decision Outcome

Chosen option: "Generic Key-Value Adapter", because it provides the right level of abstraction for Hub's needs without the complexity of a full ORM or the overhead of a separate service.

Confirmation

• [ ] Interface defined with key-value and sequence operations
• [ ] PostgreSQL adapter implemented (default)
• [ ] Configuration via environment variables
• [ ] CentralizedIdService refactored to use adapter
• [ ] Unit tests with in-memory adapter

Pros and Cons of the Options

Option 1: Service-Specific Storage

Each service implements its own storage mechanism.

• Good, because no new abstraction to learn
• Good, because each service optimized for its needs
• Bad, because code duplication
• Bad, because inconsistent error handling
• Bad, because hard to change backends

Option 2: Generic Key-Value Adapter (Chosen)

Unified interface with pluggable backends.

• Good, because single point of configuration
• Good, because easy to swap backends
• Good, because testable with in-memory adapter
• Good, because atomic operations for sequences
• Neutral, because requires some abstraction
• Bad, because may not fit all use cases perfectly

Option 3: Full ORM Integration

Use Hibernate/Panache for all storage.

• Good, because proven technology
• Good, because rich querying capabilities
• Bad, because overkill for key-value needs
• Bad, because tight coupling to JPA
• Bad, because harder to use non-SQL backends

Option 4: External Service

Separate microservice for storage.

• Good, because fully decoupled
• Good, because can be shared across systems
• Bad, because network overhead
• Bad, because deployment complexity
• Bad, because overkill for Hub's needs

Architecture

Interface Design

/**
 * Generic storage adapter for Hub services.
 * Provides key-value operations with namespace isolation.
 */
public interface StorageAdapter {

    // ========== Key-Value Operations ==========

    /**
     * Store a value.
     * @param namespace Logical grouping (e.g., "ids", "config", "cache")
     * @param key Unique key within namespace
     * @param value Value to store (serialized to JSON)
     */
    void put(String namespace, String key, Object value);

    /**
     * Retrieve a value.
     * @return Value or empty if not found
     */
    <T> Optional<T> get(String namespace, String key, Class<T> type);

    /**
     * Delete a value.
     * @return true if value existed
     */
    boolean delete(String namespace, String key);

    /**
     * Check if key exists.
     */
    boolean exists(String namespace, String key);

    /**
     * Get all keys in namespace.
     */
    Set<String> keys(String namespace);

    // ========== Atomic Sequence Operations ==========

    /**
     * Atomically increment and return new value.
     * Creates sequence starting at initialValue if not exists.
     * @param namespace Namespace for isolation
     * @param sequenceName Sequence identifier
     * @param initialValue Starting value if sequence doesn't exist
     * @return New value after increment
     */
    long incrementAndGet(String namespace, String sequenceName, long initialValue);

    /**
     * Get current sequence value without incrementing.
     */
    long getCurrentValue(String namespace, String sequenceName);

    /**
     * Set sequence to specific value (use with caution).
     */
    void setSequenceValue(String namespace, String sequenceName, long value);

    // ========== Batch Operations ==========

    /**
     * Get all entries in namespace.
     */
    <T> Map<String, T> getAll(String namespace, Class<T> type);

    /**
     * Delete all entries in namespace.
     */
    void clearNamespace(String namespace);

    // ========== Health & Info ==========

    /**
     * Check if storage is available.
     */
    boolean isHealthy();

    /**
     * Get storage backend type.
     */
    String getBackendType();
}

Namespace Convention

Backend Implementations

PostgreSQL Adapter (Default)

-- Schema for PostgreSQL adapter
CREATE TABLE hub_storage (
    namespace VARCHAR(100) NOT NULL,
    key VARCHAR(255) NOT NULL,
    value JSONB NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (namespace, key)
);

CREATE TABLE hub_sequences (
    namespace VARCHAR(100) NOT NULL,
    sequence_name VARCHAR(255) NOT NULL,
    current_value BIGINT NOT NULL DEFAULT 0,
    PRIMARY KEY (namespace, sequence_name)
);

-- Index for namespace queries
CREATE INDEX idx_hub_storage_namespace ON hub_storage(namespace);

AWS DynamoDB Adapter

Table: hub-storage
  Partition Key: namespace (String)
  Sort Key: key (String)
  Attributes: value (Map), updated_at (Number)

Table: hub-sequences
  Partition Key: namespace (String)
  Sort Key: sequence_name (String)
  Attributes: current_value (Number)

  -- Uses DynamoDB atomic counter for incrementAndGet

AWS S3 Adapter

Bucket: cloudempiere-hub-storage
  /{namespace}/{key}.json

  -- For sequences: /{namespace}/_sequences/{sequence_name}.json
  -- Note: S3 doesn't support atomic increment, use DynamoDB for sequences

In-Memory Adapter (Testing)

@Alternative
@Priority(1)
public class InMemoryStorageAdapter implements StorageAdapter {
    private final Map<String, Map<String, Object>> storage = new ConcurrentHashMap<>();
    private final Map<String, Map<String, AtomicLong>> sequences = new ConcurrentHashMap<>();
    // ... implementation
}

Configuration

# Storage adapter configuration
hub.storage.backend=${HUB_STORAGE_BACKEND:postgresql}

# PostgreSQL (default)
hub.storage.postgresql.datasource=default

# AWS DynamoDB
hub.storage.dynamodb.region=${AWS_REGION:us-east-1}
hub.storage.dynamodb.table-prefix=hub-

# AWS S3
hub.storage.s3.bucket=${HUB_S3_BUCKET:cloudempiere-hub-storage}
hub.storage.s3.region=${AWS_REGION:us-east-1}

# In-Memory (testing only)
hub.storage.inmemory.enabled=false

Integration with CentralizedIdService

@ApplicationScoped
public class CentralizedIdServiceImpl implements CentralizedIdService {

    @Inject
    StorageAdapter storage;

    @Inject
    CentralizedIdConfig config;

    @Override
    public int allocateId(String tableName, String entityType, String comment, boolean localOverride) {
        IdAllocationMode mode = detectMode(entityType, localOverride);

        return switch (mode) {
            case LOCAL_SYSTEM -> generateLocalId(tableName);

            case CENTRALIZED_REQUIRED -> {
                // Check scope: internal (CE_*) vs community
                if (isCloudEmpiereEntityType(entityType)) {
                    // Use StorageAdapter for internal IDs
                    yield allocateFromStorage(tableName, entityType);
                } else {
                    // Use HTTP client for community server
                    yield allocateFromCommunityServer(tableName, entityType, comment);
                }
            }

            case LOCAL_OVERRIDE -> generateLocalId(tableName);
        };
    }

    private int allocateFromStorage(String tableName, String entityType) {
        String sequenceKey = tableName + ":" + entityType;
        long nextId = storage.incrementAndGet("ids:sequences", sequenceKey, 2000000L);
        LOG.infof("Allocated ID %d from StorageAdapter for %s", nextId, sequenceKey);
        return (int) nextId;
    }

    private boolean isCloudEmpiereEntityType(String entityType) {
        return entityType != null && entityType.startsWith("CE");
    }
}

Implementation Plan

Phase 1: Core Interface & PostgreSQL Adapter (1-2 days)

• [ ] Define StorageAdapter interface
• [ ] Implement PostgreSqlStorageAdapter
• [ ] Create database schema migration
• [ ] Add configuration properties
• [ ] Unit tests with in-memory adapter

Phase 2: CentralizedIdService Integration (1 day)

• [ ] Refactor CentralizedIdServiceImpl to use StorageAdapter
• [ ] Add scope detection for CE_* entity types
• [ ] Integration tests

Phase 3: AWS Adapters (Future)

• [ ] Implement DynamoDbStorageAdapter
• [ ] Implement S3StorageAdapter
• [ ] CloudFormation/Terraform templates

Phase 4: Additional Services (Future)

• [ ] ConfigService using StorageAdapter
• [ ] CacheService using StorageAdapter
• [ ] State management for wizards

Usage Examples

Centralized IDs (CloudEmpiere Internal)

// Entity type CE_* uses StorageAdapter
storage.incrementAndGet("ids:sequences", "AD_Table:CE", 2000000L);
// Returns: 2000001, 2000002, 2000003, ...

// Register entity type
storage.put("ids:registry", "CE", Map.of(
    "owner", "cloudempiere",
    "description", "CloudEmpiere internal",
    "created", Instant.now()
));

Configuration Storage

// Store hub configuration
storage.put("config:hub", "default_entity_type", "U");
storage.put("config:hub", "api_timeout", 30000);

// Retrieve
Optional<String> entityType = storage.get("config:hub", "default_entity_type", String.class);

Query Cache

// Cache query result
String cacheKey = DigestUtils.sha256Hex(sqlQuery);
storage.put("cache:query", cacheKey, Map.of(
    "result", queryResult,
    "cached_at", Instant.now(),
    "ttl_seconds", 300
));

Security Considerations

1. Namespace Isolation: Each namespace is logically isolated
2. No Direct SQL: Adapter prevents SQL injection via parameterized queries
3. Credential Storage: Sensitive values should use separate secrets management
4. AWS IAM: DynamoDB/S3 adapters use IAM roles, not embedded credentials

Related ADRs

• ADR-062: Centralized ID Management - Primary consumer
• ADR-054: AI Tool Architecture Clarity - Service patterns
• ADR-055: Tool Ecosystem Review - Shared services

References

• Quarkus Configuration Reference
• AWS DynamoDB Developer Guide
• AWS S3 Developer Guide

Note: This ADR defines a generic storage abstraction. Domain-specific logic (ID allocation rules, entity type validation) remains in the consuming services.