Task Engine

The Task Engine is the core component of the EigenLayer-AVS system responsible for managing, compiling, and executing tasks.

The Task Engine is the core component of the EigenLayer-AVS system responsible for managing, compiling, and executing tasks. It provides a flexible execution environment that processes tasks as directed acyclic graphs (DAGs) of operations, allowing complex workflows to be defined and executed in a deterministic manner.

Purpose and Scope#

This document covers the architecture, components, and functionality of the Task Engine, including:

  • Engine component for task management and coordination
  • Virtual Machine (VM) for task compilation and execution
  • Task execution flow and state management
  • Supported task node types and processors

For information about task data structures and modeling, see Task Model. For details on individual processors, see VM Processors.

Task Engine Architecture#

The Task Engine consists of two main components: the Engine and the Virtual Machine (VM). The Engine manages task coordination, scheduling, and lifecycle, while the VM handles the actual compilation and execution of tasks.

Sources:

Engine Component#

The Engine component is responsible for:

  1. Managing task lifecycle (creation, execution, completion)
  2. Coordinating with operators for task triggers
  3. Interacting with storage to persist task data
  4. Handling task execution scheduling through the queue system

The Engine exposes methods for creating, triggering, and monitoring tasks, as well as managing secrets that can be used within tasks.

Sources:

Virtual Machine (VM)#

The VM is responsible for:

  1. Compiling tasks into executable plans
  2. Executing task plans node by node
  3. Managing execution state and variables
  4. Processing template expressions in task definitions
  5. Dispatching node execution to appropriate processors

The VM has a lifecycle with distinct states (initialize, compiled, ready, executing, completed) and maintains execution context through the entire task run.

Sources:

Task Execution Flow#

Task execution follows a well-defined flow from creation to completion. The Engine coordinates with operators for task triggers, while the VM handles the actual execution.

Sources:

Task States and Transitions#

Tasks and VM executions have well-defined states that track progress through the lifecycle:

EntityStateDescription
TaskActiveTask is active and can be triggered
TaskExecutingTask is currently executing
TaskCompletedTask has completed all executions
TaskFailedTask has failed
TaskCanceledTask has been canceled
VMVMStateInitializeVM is initializing
VMVMStateCompiledTask has been compiled into execution plan
VMVMStateReadyVM is ready for execution
VMVMStateExecutingVM is currently executing
VMVMStateCompletedVM has completed execution

Sources:

Task Compilation and Execution#

Task Compilation#

The VM compiles tasks into execution plans by analyzing the edges between nodes and identifying the entry point. The compilation process:

  1. Creates Step objects for each node in the task
  2. Builds a map of nodes to their next steps based on edges
  3. Identifies the entry point based on the trigger node
  4. Sets the VM state to Ready when compilation is complete

Sources:

Task Execution#

The VM executes tasks node by node, following the execution plan created during compilation:

  1. VM starts with the entry point node
  2. For each node, it dispatches execution to the appropriate processor
  3. Results from each node are stored in the VM's variable map
  4. VM follows the Next pointers to determine the next node to execute
  5. Execution continues until there are no more nodes to execute

The VM supports various node types through specialized processors:

Node TypeProcessorDescription
RestApiRestProcessorHTTP API calls
ContractReadContractReadProcessorSmart contract read operations
ContractWriteContractWriteProcessorSmart contract write operations
GraphQLQueryGraphQLQueryProcessorGraphQL API queries
CustomCodeJSProcessorCustom JavaScript code execution
BranchBranchProcessorConditional branching logic

Sources:

Template Processing#

The VM includes a template processing system that allows for dynamic content in task definitions. Templates use {{ expression }} syntax and can reference:

  1. Variables from previous node outputs
  2. Trigger data
  3. User-defined secrets

The preprocessText method evaluates these expressions using a JavaScript engine:

Sources:

Common Patterns and Use Cases#

Workflow Definition#

Tasks are defined as directed acyclic graphs (DAGs) with nodes representing operations and edges representing execution flow:

Tasks can be triggered manually or automatically via events, blocks, or time-based triggers.

Sources:

Variable Sharing Between Nodes#

Outputs from each node are stored in the VM's variable map and can be referenced by subsequent nodes:

This allows for data passing between nodes in the execution flow.

Sources:

Integration with Other Components#

Storage Integration#

The Task Engine uses BadgerDB for persistent storage of:

  1. Task definitions
  2. Execution results
  3. User secrets
  4. Smart wallet configurations

The storage layer is accessed through the db field in both Engine and VM.

Sources:

Operator Integration#

The Task Engine interacts with operators through:

  1. StreamCheckToOperator: Streams task definitions to operators
  2. AggregateChecksResult: Processes trigger notifications from operators

Operators monitor for trigger conditions and notify the Engine when conditions are met.

Sources:

Queue Integration#

The Task Engine uses a queue system (apqueue) for asynchronous task execution:

  1. When a task is triggered, it's enqueued for execution
  2. The engine creates an execution record in the "queued" state
  3. The queue system processes the task asynchronously
  4. The execution record is updated with the final result

Sources:

Key APIs and Usage#

Task Creation#

CreateTask(user *model.User, taskPayload *avsproto.CreateTaskReq) (*model.Task, error)

Creates a new task with the specified nodes, edges, and trigger.

Sources:

Task Triggering#

TriggerTask(user *model.User, payload *avsproto.UserTriggerTaskReq) (*avsproto.UserTriggerTaskResp, error)

Manually triggers a task execution. Can be run in blocking or non-blocking mode.

Sources:

Task Listing#

ListTasksByUser(user *model.User, payload *avsproto.ListTasksReq) (*avsproto.ListTasksResp, error)

Lists tasks belonging to a user, with pagination support.

Sources:

Execution Listing#

ListExecutions(user *model.User, payload *avsproto.ListExecutionsReq) (*avsproto.ListExecutionsResp, error)

Lists executions for a given task or set of tasks, with pagination support.

Sources:

Advanced Features#

Secret Management#

The Task Engine provides APIs for managing secrets that can be used in tasks:

CreateSecret(user *model.User, payload *avsproto.CreateOrUpdateSecretReq) (bool, error)
UpdateSecret(user *model.User, payload *avsproto.CreateOrUpdateSecretReq) (bool, error)
ListSecrets(user *model.User, payload *avsproto.ListSecretsReq) (*avsproto.ListSecretsResp, error)
DeleteSecret(user *model.User, payload *avsproto.DeleteSecretReq) (bool, error)

Secrets are stored securely and can be referenced in tasks using {{ apContext.configVars.secretName }}.

Sources:

Smart Wallet Integration#

The Task Engine integrates with smart wallets for:

  1. Task ownership verification
  2. Contract interactions
  3. Execution permissions

Smart wallet configurations are passed to the VM for use in contract interactions.

Sources:

Conclusion#

The Task Engine provides a powerful, flexible system for defining and executing complex workflows. It combines the management capabilities of the Engine with the execution capabilities of the VM to create a comprehensive task execution environment. The various processors support a wide range of operations, from HTTP calls to smart contract interactions, allowing for sophisticated automations and integrations.

Feature this wiki to auto refresh weekly

Try DeepWiki on your private codebase with Devin