ZoopFlow Architecture Overview

This document provides a comprehensive overview of the ZoopFlow architecture, including its core components, data flow, integration points, and deployment models.

High-Level System Overview

ZoopFlow is a distributed workflow orchestration framework designed to provide a robust, scalable, and flexible solution for business process automation. It enables the creation of reusable workflow steps that can be orchestrated into complex flows, with built-in support for error handling, state management, and distributed execution.

At its core, ZoopFlow separates workflow definition from execution, allowing for a clean separation of concerns. The framework integrates with Temporal.io for workflow orchestration, providing durability, fault tolerance, and distributed execution capabilities while maintaining a developer-friendly API.

┌─────────────────────────────────────────────────────────────────┐
│                      ZoopFlow Framework                          │
│                                                                 │
│  ┌───────────────┐    ┌────────────────┐    ┌────────────────┐  │
│  │ Step Definition│    │Flow Definition │    │ Registry System│  │
│  │    Layer      │───▶│     Layer      │───▶│                │  │
│  └───────────────┘    └────────────────┘    └────────────────┘  │
│          │                     │                     │           │
│          │                     ▼                     │           │
│          │             ┌────────────────┐            │           │
│          │             │Flow Interpreter│            │           │
│          │             │     System     │            │           │
│          │             └────────────────┘            │           │
│          │                     │                     │           │
│          ▼                     ▼                     ▼           │
│  ┌───────────────┐    ┌────────────────┐    ┌────────────────┐  │
│  │ Execution     │    │  Temporal      │    │  Runtime       │  │
│  │ Context       │◀──▶│  Bridge        │◀──▶│  Environment   │  │
│  └───────────────┘    └────────────────┘    └────────────────┘  │
│                              │                                   │
└──────────────────────────────┼───────────────────────────────────┘
                               │
                               ▼
                      ┌─────────────────┐
                      │   Temporal.io   │
                      │  Workflow Engine│
                      └─────────────────┘

Core Components and Their Responsibilities

ZoopFlow consists of the following core components, each with distinct responsibilities:

1. Step Definition Layer

The Step Definition Layer provides the API for defining reusable workflow steps:

Responsibilities:
- Define discrete units of work with well-defined inputs and outputs
- Specify validation schemas for inputs and outputs
- Configure timeout and retry behavior
- Provide step metadata for discovery and documentation

Key Interfaces:

interface StepDefinition<TInput, TOutput> {
  id: string;
  version: string;
  description?: string;
  category?: string;
  input_schema: JSONSchema;
  output_schema: JSONSchema;
  execute: (input: TInput, context: StepContext) => Promise<TOutput>;
  timeout_seconds?: number;
  retry_config?: RetryConfig;
}

2. Flow Definition Layer

The Flow Definition Layer provides the API for orchestrating steps into workflows:

Responsibilities:
- Define the structure and sequence of workflow steps
- Manage data flow between steps
- Handle conditional logic and branching
- Support parallel execution and join patterns
- Enable error handling and recovery

Key Interfaces:

interface FlowDefinition<TInput, TOutput> {
  id: string;
  version: string;
  description?: string;
  input_schema: JSONSchema;
  output_schema: JSONSchema;
  steps: (input: TInput, context: FlowContext) => Promise<TOutput>;
}

3. Flow Interpreter System

The Flow Interpreter System handles the loading, validation, and execution of flow definitions:

Responsibilities:
- Load and validate flow definitions
- Orchestrate execution of nodes within a flow
- Manage execution context and state
- Handle errors and timeouts during execution
- Provide metrics and execution information
- Resolve node handlers and step implementations

Key Components:

interface IFlowInterpreter {
  loadFlow<TInput, TOutput>(flowDefinition): FlowGraphDefinition<TInput, TOutput>;
  validateFlow(flowDefinition): { valid: boolean; errors?: string[] };
}
 
interface IFlowExecutionEngine {
  executeFlow<TInput, TOutput>(flow, input, options): Promise<FlowExecutionResult<TOutput>>;
}

4. Temporal Bridge

The Temporal Bridge integrates ZoopFlow with Temporal's workflow engine:

Responsibilities:
- Register ZoopFlow steps as Temporal activities
- Create Temporal workflows from ZoopFlow flows
- Map ZoopFlow retry configurations to Temporal retry policies
- Handle workflow lifecycle management
- Support signal processing for dynamic workflow control

Implementation Details:

class TemporalBridge {
  registerStepAsActivity(step: StepDefinition<any, any>): void;
  registerFlowAsWorkflow(flow: FlowDefinition<any, any>): void;
  executeFlow(flowId: string, input: any, options?: any): Promise<any>;
}

4. Registry System

The Registry System manages metadata for steps and flows:

Responsibilities:
- Store and retrieve step/flow definitions
- Manage versioning of components
- Support discovery and introspection
- Facilitate marketplace integration
- Enforce access control policies

Implementation Details:

class StepRegistry {
  registerStep(step: StepDefinition<any, any>): void;
  getStep(id: string): StepMetadata | undefined;
  listSteps(): StepMetadata[];
}

5. Runtime Environment

The Runtime Environment manages the execution of steps and flows:

Responsibilities:
- Provision and manage worker processes
- Balance execution load across workers
- Monitor execution health and performance
- Scale resources based on demand
- Support resource isolation and security

6. Execution Context

The Execution Context provides utilities and services during execution:

Responsibilities:
- Provide logging capabilities
- Manage workflow state
- Enable service discovery and access
- Supply execution metadata
- Support checkpointing and recovery
- Track execution history and status
- Manage lifecycle transitions

Key Interfaces:

interface StepContext {
  // Execution metadata
  info: ExecutionInfo;
  stepMetadata: StepMetadata;
  readonly executionMetadata: StepExecutionMetadata;
  readonly history: StepHistoryEntry[];
  
  // State management
  getState<T>(key: string): Promise<T | undefined>;
  setState<T>(key: string, value: T): Promise<void>;
  getStateObject(): Promise<Record<string, unknown>>;
  updateState(stateUpdates: Record<string, unknown>): Promise<void>;
  
  // Checkpointing and recovery
  createCheckpoint(metadata?: Record<string, unknown>): Promise<string>;
  restoreFromCheckpoint(checkpointId: string): Promise<void>;
  getCheckpointInfo(checkpointId: string): Promise<CheckpointInfo | undefined>;
  getAllCheckpoints(): Promise<CheckpointInfo[]>;
  
  // Logging and history
  log(message: string, metadata?: Record<string, unknown>): void;
  addHistoryEntry(type: 'custom', details: Record<string, unknown>): void;
  
  // Lifecycle management
  complete(details?: Record<string, unknown>): Promise<void>;
  fail(reason: string, details?: Record<string, unknown>): Promise<void>;
  
  // Service access
  getService<T>(name: string, options?: Record<string, unknown>): T;
}

7. Workflow Wrapper

The Workflow Wrapper encapsulates flow execution within Temporal workflows:

Responsibilities:
- Manage workflow execution lifecycle
- Handle activity invocation based on flow definition
- Process signals for flow control
- Maintain workflow state
- Support error handling and recovery

Data Flow and Execution Model

The execution of a ZoopFlow workflow follows a three-phase lifecycle:

1. Definition Phase

During the definition phase, developers create reusable workflow components:

Step Definition:
- Steps are defined with input/output schemas
- Each step implements a single unit of functionality
- Steps can include retry configurations and timeouts
Flow Definition:
- Flows orchestrate steps into a coherent workflow
- Flows define the data flow between steps
- Control flow constructs (conditionals, loops) shape execution paths
Registration:
- Steps and flows are registered with the registry
- Metadata is extracted for discovery and introspection
- Components are made available for execution

2. Deployment Phase

During the deployment phase, the workflow components are prepared for execution:

Validation:
- CLI tools discover and validate step/flow definitions
- Schema validators ensure correct structure and types
- Linters check for best practices and potential issues
Worker Deployment:
- Workers are deployed with registered steps and flows
- Resources are allocated based on expected workload
- Monitoring and observability tooling is configured

3. Execution Phase

During the execution phase, workflows are executed:

Workflow Initialization:
- Flows are triggered via API or events
- Input validation ensures data integrity
- Execution context is established
Step Execution:
- Temporal executes the workflow, calling steps as needed
- Each step receives input data and execution context
- Step outputs are validated against schemas
- Retries are handled automatically for transient failures
State Management:
- Workflow state is persisted between execution steps
- Checkpoints enable recovery from failures
- Queries allow external inspection of workflow state
Completion:
- Results are validated against output schema
- Final results are returned to the caller
- Execution metrics are collected and reported

Integration with Temporal

ZoopFlow integrates deeply with Temporal.io to leverage its workflow engine capabilities:

Core Integration Points

Steps → Activities:
- ZoopFlow steps are mapped to Temporal activities
- Retry configurations are translated to Temporal retry policies
- Timeouts are enforced via Temporal activity options
Flows → Workflows:
- ZoopFlow flows become Temporal workflows
- Flow execution is managed through Temporal's workflow engine
- Workflow history provides durability and fault tolerance
Contexts:
- Step Context wraps Temporal's Activity Context
- Flow Context wraps Temporal's Workflow Context

Flow Definition Adapter

The Flow Definition Adapter translates JSON flow definitions into Temporal-compatible workflow definitions:

Converts flow nodes to workflow activities
Maps flow edges to workflow control flow
Translates flow configuration to workflow options
Handles type conversions between Flow and Temporal

Temporal Integration Components

The Temporal integration components serve as the connection points between the Flow system and Temporal:

Register flows and steps with the Temporal service
Create and manage Temporal clients and workers
Execute workflows based on flow definitions
Process signals through the SignalManager for dynamic workflow control
Provide adapters for Temporal-specific contexts

State Management

State management is implemented through Temporal's built-in persistence mechanisms:

Workflow state includes execution status, node information, and data
Checkpoints are created at key execution points
State is automatically recovered during workflow replay
Queries provide external access to workflow state

Distributed Execution Architecture

ZoopFlow's distributed execution architecture separates workflow orchestration from step execution, supporting various deployment models:

Key Architectural Principles

Separation of Flow Execution from Data Storage:
- Flow and Step definitions reside with ZoopFlow
- Client data remains in client-controlled environments
- Execution happens at ZoopFlow but operates on client data
Centralized Marketplace with Private Extensions:
- Central repository of steps and flows (ZoopFlow Marketplace)
- Organization-specific enablement of marketplace components
- Private registries for enterprise clients' custom components
Human Task Integration:
- Support for UI-driven human interactions in workflows
- Shareable workflow links for end-user interaction
- Progressive enhancement from ZoopFlow UI controls to client-customized interfaces

Primary Components

ZOOP SaaS Platform:
- Executor Service: Manages flow and step definitions
- Definition Database: Stores all flow and step definitions
- Execution Runtime: Workflow engine built on Temporal
- Task UI Service: Hosts workflow UI components for human tasks
- Temporal Bridge: Coordinates with Temporal.io for workflow orchestration
Client Environment:
- Client Data Stores: Contains all client data
- Client SDK: Registers flows and steps with ZoopFlow
- Private Registry (Optional): Client-hosted repository of custom components

Distributed Execution Diagram

Marketplace and Service Architecture

System Boundaries and Integration Points

ZoopFlow defines clear system boundaries and integration points:

External Integration Points

Client Applications:
- REST APIs for workflow management
- SDKs for programmatic access
- Webhooks for event notifications
Data Stores:
- Database connectors for structured data
- File system adapters for unstructured data
- API clients for external services
External Services:
- Authentication providers
- Notification services
- Third-party API integrations

Internal Integration Points

Component Registration:
- Registry interfaces for steps and flows
- Validation pipelines for component integrity
- Versioning mechanisms for compatibility
Execution Control:
- Signal interfaces for workflow control
- Query interfaces for state inspection
- Metrics interfaces for monitoring
Extension Points:
- Middleware hooks for cross-cutting concerns
- Plugin interfaces for custom functionality
- Tag handlers for customized data processing

Security Model

ZoopFlow's security model is designed to protect both the platform and customer data:

Data Security

Data Residency:
- Client data never stored in ZoopFlow
- Encryption in transit for all communications
- Secure access to client data stores
Credentials Management:
- Secure storage of connection credentials
- Just-in-time access to sensitive resources
- Rotation policies for access tokens
Encryption:
- TLS for all communications
- Field-level encryption for sensitive data
- Support for client-managed encryption keys

Access Control

Authentication:
- Multi-tenant identity management
- Integration with enterprise identity providers
- Support for OAuth, SAML, and API keys
Authorization:
- Role-based access control (RBAC)
- Fine-grained permissions for platform features
- Resource-level access policies
Audit and Compliance:
- Comprehensive audit logging
- Tamper-proof activity history
- Compliance reporting tools

Performance Characteristics

ZoopFlow is designed for high performance across various workloads:

Execution Performance

Step Execution:
- Low-latency activity dispatch
- Configurable resource allocation
- Optimized for both short and long-running steps
Workflow Throughput:
- Parallel execution of compatible steps
- Batching of related operations
- Efficient state management to minimize overhead
State Management:
- Incremental checkpointing to reduce storage overhead
- Lazy loading of workflow history
- Optimized replay for long-running workflows

Metrics and Monitoring

Performance Metrics:
- Execution time per step and flow
- Resource utilization (CPU, memory, I/O)
- Throughput and latency statistics
Health Indicators:
- Worker availability and responsiveness
- Queue depth and processing rates
- Error rates and recovery success
Observability:
- Distributed tracing across execution stages
- Detailed logging with structured metadata
- Real-time dashboards for system health

Scalability Considerations

ZoopFlow is built to scale from small deployments to enterprise-level workloads:

Horizontal Scaling

Worker Scaling:
- Dynamically adjust worker pool size
- Balance load across available workers
- Support for worker specialization by task type
Task Queue Partitioning:
- Distribute workload across task queues
- Route related workflows to the same queue
- Prioritize critical workflows
Service Tiers:
- Dedicated resources for priority clients
- Shared resources for standard workloads
- Isolation for performance-sensitive operations

Capacity Planning

Resource Estimation:
- Workflow complexity assessment
- Throughput requirements analysis
- Peak load planning
Growth Management:
- Gradual resource allocation
- Auto-scaling based on demand
- Reserved capacity for critical operations

Deployment Models

ZoopFlow supports multiple deployment models to meet diverse customer needs:

1. SaaS with Data Residency (Primary Model)

In this model:

ZoopFlow manages all flow/step definitions and their execution
Flow execution metadata is stored in ZoopFlow's database
All client data remains in client-controlled environments
ZoopFlow connects to client data stores to read/write data during execution

Benefits:

Rapid deployment without infrastructure requirements
Data security and compliance with data residency requirements
ZoopFlow management of execution without direct data access

┌─────────────────────────────┐           ┌─────────────────────────────┐
│      ZoopFlow Platform      │           │      Client Environment     │
│                             │           │                             │
│  ┌───────────┐ ┌─────────┐  │           │  ┌───────────┐              │
│  │  Registry │ │Execution│  │           │  │Client Data│              │
│  │  System   │ │ Runtime │  │◄─────────►│  │  Stores   │              │
│  └───────────┘ └─────────┘  │           │  └───────────┘              │
│                             │           │                             │
│  ┌───────────┐ ┌─────────┐  │           │  ┌───────────┐              │
│  │   Task    │ │Temporal │  │           │  │Client SDK │              │
│  │    UI     │ │ Bridge  │  │◄─────────►│  │           │              │
│  └───────────┘ └─────────┘  │           │  └───────────┘              │
└─────────────────────────────┘           └─────────────────────────────┘

2. Hybrid Model (Future Extension)

In this model:

Flow execution in ZoopFlow, step execution on client premises
Registry synchronization between environments

Benefits:

Greater control over sensitive operations
Reduced data movement
Lower latency for data-intensive steps

3. On-Premises Model (Future Extension)

In this model:

Complete deployment of ZoopFlow within client environment
Only marketplace and updates from ZoopFlow cloud

Benefits:

Maximum control over entire platform
Full data residency compliance
Integration with on-premises systems

Conclusion

ZoopFlow provides a robust, scalable, and flexible framework for workflow orchestration. Its modular architecture, integration with Temporal.io, and support for distributed execution make it suitable for a wide range of automation scenarios from simple business processes to complex enterprise workflows.

The separation of workflow definition from execution, combined with strong typing and validation, enables developers to create reliable and maintainable workflows while leveraging the power of a distributed execution engine.

Overview Flow Definition System