KaryaFlow: A Distributed Autonomous Task Orchestration System

Problem & Constraints

Traditional task managers rely on manual updates and human discipline; tasks routinely sit idle without intervention. KaryaFlow replaces this with continuous monitoring, deterministic operational enforcement, mandatory interaction loops, and optional context-aware reasoning. The Discord layer is only an interface adapter — the core is an interface-agnostic backend designed to also integrate with Slack, email, internal chat, web dashboards, or custom enterprise software.

Constraints: scheduling correctness, escalation, and tenant isolation must not depend on LLM availability or retrieval quality. The system must remain operationally safe under broker duplication, Redis outage, AI provider timeout, and uneven per-tenant traffic.

Task Ownership & Lifecycle

Tasks flow through Pending → Processing → Review → Closed. Only active states (Pending, Processing, Review) are monitored by the agent. Operational conditions are modeled separately from the primary workflow state as flags/timestamps rather than peer states: is_blocked, escalated_at, and needs_manual_review. This keeps the lifecycle linear and the operational flags composable.

• blocked — task cannot currently progress.
• escalated — accountability escalation has occurred.
• needs_manual_review — the agent could not safely classify or resolve the latest interaction.

Target-State vs Adoption Path

• v1 — deterministic control loop: hybrid scheduler, rules-based state transitions, explicit escalation policy, no AI dependency in the hot path.
• v2 — richer operational policy: config-driven thresholds, stronger reporting and operator controls, more nuanced rule engine.
• v3 — intelligent augmentation: LLM-assisted response classification, retrieval-informed prompting, behavioral context for adaptive follow-up.

Architecture Overview

Initial deployment is a distributed monolith — all services live in one binary and communicate in-process, but are structured as independent modules. Kafka/NATS is wired in from day one so extracting a module later requires only a deployment change.

Discord Adapter (Bot / Discord SDK)        ↓API Gateway (Go — JWT-authenticated)        ↓------------------------------------------Task Service   │ Auth ServiceAgent Service  │ Retrieval ServiceNotification   │ AI ServiceService        │------------------------------------------        ↓Message Broker (Kafka / NATS)        ↓Worker Pool (Go routines)        ↓Data Layer (Postgres + Redis + pgvector / Weaviate)        ↓Observability (Prometheus + OpenTelemetry + Zap)

Agent Scheduler — Hybrid Model

The Agent Service is intentionally deterministic in the hot path. Scheduling, escalation, and state progression are correct without AI or retrieval; intelligent services improve interpretation and prompt quality but are not required for operational safety.

• Primary path — event-triggered: every task state change or user response emits an event. The Agent consumes it, evaluates the task, and writes next_agent_check_at to the task record.
• Safety net — cron sweep: a background goroutine runs every 5 minutes querying for tasks where next_agent_check_at ≤ now AND workflow_state IN ('pending','processing','review'). Catches any task whose event was missed due to broker failure or restart.
• Combined, this guarantees at-least-once evaluation without relying solely on event delivery.

Agent Decision Tree

Task enters evaluation        │        ▼Has the user responded to the last prompt?   ├── YES → Parse response intent (rules first, AI optional)   │            ├── "start" / "progress"  → update state, schedule next check   │            │                            based on commitment   │            ├── "blocked"              → set blocked flag, notify manager   │            ├── "delay: N hours"       → next_agent_check_at = now + N hours   │            └── unclassified           → mark needs_manual_review,   │                                          fall back to default follow-up   │   └── NO → How many consecutive missed responses?               ├── 1 miss  → send reminder               ├── 2 misses → send re-check with higher urgency               └── 3+ misses → trigger escalation                               ├── notify manager (RBAC: manager role)                               ├── attach context summary if available                               └── set escalated_at

Deterministic classification path: explicit rules and keyword extraction cover start, progress, blocked, done, and simple time commitments. Ambiguous text optionally calls the AI Service; if AI is unavailable or times out, the task is marked needs_manual_review and falls back to a default reminder path. Retrieval is consulted before sending a prompt — if a repeat-delay pattern is detected for the user, commitment windows are shortened and follow-up frequency is increased. If retrieval is unavailable, the Agent continues with default enforcement policy.

if user.delay_count_last_30_days >= 3:    max_commitment = min(user_requested_hours, 2)else:    max_commitment = user_requested_hours

In production these values are config-driven rather than hardcoded constants. Review ownership is policy-driven but assignment is explicit: when a task enters review, the system writes a concrete reviewer_id and reminders/escalations target that stored assignment rather than re-deriving ownership on each evaluation.

Key Design Decisions

Data Model

• All core tables carry tenant_id (and division_id where relevant) for row-level isolation.
• Soft deletes via deleted_at — no hard deletes on user or task records.
• Append-only audit_logs for all state-changing operations.
• Migrations managed with Ent + Atlas versioned migrations.

CREATE TABLE tasks (    id                    UUID PRIMARY KEY DEFAULT gen_random_uuid(),    tenant_id             UUID NOT NULL REFERENCES tenants(id),    division_id           UUID NOT NULL REFERENCES divisions(id),    assignee_id           UUID NOT NULL REFERENCES users(id),    creator_id            UUID NOT NULL REFERENCES users(id),    reviewer_id           UUID REFERENCES users(id),    title                 TEXT NOT NULL,    description           TEXT,    workflow_state        TEXT NOT NULL DEFAULT 'pending',    -- pending | processing | review | closed    priority              TEXT NOT NULL DEFAULT 'medium',    due_at                TIMESTAMPTZ,    committed_until       TIMESTAMPTZ,    next_agent_check_at   TIMESTAMPTZ,    miss_count            INT NOT NULL DEFAULT 0,    is_blocked            BOOLEAN NOT NULL DEFAULT false,    blocked_at            TIMESTAMPTZ,    needs_manual_review   BOOLEAN NOT NULL DEFAULT false,    manual_review_at      TIMESTAMPTZ,    escalated_at          TIMESTAMPTZ,    embedding_id          UUID,    created_at            TIMESTAMPTZ NOT NULL DEFAULT now(),    updated_at            TIMESTAMPTZ NOT NULL DEFAULT now(),    deleted_at            TIMESTAMPTZ);-- Primary lookup path for the cron sweep — partial to stay fast at scaleCREATE INDEX idx_tasks_next_check ON tasks(next_agent_check_at)    WHERE workflow_state IN ('pending','processing','review')      AND deleted_at IS NULL;

CREATE TABLE processed_events (    id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),    tenant_id     UUID NOT NULL REFERENCES tenants(id),    consumer_name TEXT NOT NULL,    event_id      UUID NOT NULL,    event_type    TEXT,    aggregate_id  UUID,    processed_at  TIMESTAMPTZ NOT NULL DEFAULT now());CREATE UNIQUE INDEX uq_processed_events_consumer_event    ON processed_events(consumer_name, event_id);

Auth, RBAC & Tenant Boundary

KaryaFlow does not own user credentials. External identity providers (Discord, Slack) are the source of truth for authentication. On first interaction the Auth Service auto-provisions an internal user record, then issues a short-lived JWT (15 min, RS256) containing user_id, tenant_id, division_id, and role. Refresh tokens (7d) are stored in Redis as refresh:{user_id} for server-side revocation.

• Token layer: JWT carries tenant_id; no cross-tenant token can be issued.
• API layer: every handler injects tenant_id from the token, never from request parameters.
• Database layer: all core tables carry tenant_id; PostgreSQL row-level security enforces tenant_id = current_setting('app.tenant_id') as defense-in-depth.

Embedding & Retrieval Pipeline

This subsystem is a later-phase augmentation layer. It improves classification quality, prompt personalization, and behavioral policy tuning, but the system remains operationally correct when retrieval is disabled.

{  "query": "<task title + description>",  "filters": {    "tenant_id": "...",    "user_id": "...",    "doc_type": ["task_summary", "user_behavior_log"]  },  "top_k": 5,  "min_score": 0.75}

min_score = 0.75 cosine similarity. Below threshold the Agent proceeds with no retrieved context rather than low-quality context. Retrieved documents are injected as structured context blocks, not raw text, so the model can clearly distinguish historical patterns from the current task. If retrieval latency breaches its budget or the vector store is unavailable, the Agent continues with default policy — retrieval is allowed to fail closed into deterministic behavior.

Reliability & Failure Modes

• Broker failure: the cron sweep remains the correctness safety net for missed evaluations; idempotent consumers absorb replay safely.
• Redis failure: an optimization layer for caches and hot scheduling data; correctness-critical flows degrade to PostgreSQL-backed reads rather than stopping task progression.
• AI timeout or provider failure: classification is bypassed; ambiguous responses fall back to rule-based parsing or needs_manual_review. Core scheduler timing and escalation continue unaffected.
• Retrieval / vector-store failure: the Agent omits behavioral context and uses default commitment and escalation policy.
• Bounded exponential-backoff retries; after the threshold, events move to a DLQ with depth and age as operational signals.

Operational Scenarios

Time-based commitment

Task pending for an extended period. Agent prompts: 'How much time do you need?' User says 'Need 2 hours'. Deterministic parsing extracts delay = 2h; AI is not required. next_agent_check_at and committed_until are set to now + 2h. After 2 hours the cron sweep picks up the task → Agent re-evaluates → Task moves to Processing. No unnecessary reminders; commitment enforced with precision.

Silent failure & escalation

Task in processing for 48h with no response. Agent prompt → no response in 4h → miss_count = 1 → reminder. No response in 2h → miss_count = 2 → urgent re-check. Still no response → miss_count = 3 → escalation. Agent sets escalated_at, publishes AGENT_ESCALATED, and Notification Service pings the manager with task context and optional behavior summary.

Repeated delay pattern

User with delay_count_30d = 4 requests 'I need 8 hours'. The configured cap applies: max_commitment = min(8, 2) = 2 hours. next_agent_check_at is set to now + 2h, overriding the request. Retrieval context confirms the pattern → Agent increases follow-up frequency. Nightly batch regenerates the user behavior embedding with updated stats.

AI timeout during classification

User responds ambiguously: 'trying, maybe later if blockers clear'. Rules cannot confidently classify. Agent calls AI Service; the request times out. Circuit breaker records the failure; the Agent marks needs_manual_review = true. next_agent_check_at is still scheduled using default reminder policy — no escalation or transition is lost. AI failure degrades decision quality, not system correctness.

Tradeoffs Summary

• Distributed monolith vs microservices: faster development, with a defined operator-driven extraction path when per-tenant load or service CPU thresholds are sustained.
• Event-driven vs synchronous calls: event-driven for cross-module communication; synchronous gRPC for real-time reads. Synchronous paths are bounded and explicit.
• RAG vs traditional querying: complementary — SQL for structured queries, vector retrieval for behavioral patterns. Retrieval remains optional and the control loop runs without it.
• LLM usage: only for response classification and prompt generation; scheduling and escalation remain deterministic. Graceful degradation under model outage.
• Ent + Atlas vs raw SQL migrations: type-safe queries, automatic migration diffing, and schema drift detection in CI.
• Reliability complexity vs operational safety: explicit idempotency, retries, DLQs, circuit breakers, and tenant-aware rate limiting from the beginning.