Task Engine
The Task Engine is the core component of the EigenLayer-AVS system responsible for managing, compiling, and executing tasks.
The Task Engine is the core component of the EigenLayer-AVS system responsible for managing, compiling, and executing tasks. It provides a flexible execution environment that processes tasks as directed acyclic graphs (DAGs) of operations, allowing complex workflows to be defined and executed in a deterministic manner.
Purpose and Scope#
This document covers the architecture, components, and functionality of the Task Engine, including:
- Engine component for task management and coordination
- Virtual Machine (VM) for task compilation and execution
- Task execution flow and state management
- Supported task node types and processors
For information about task data structures and modeling, see Task Model. For details on individual processors, see VM Processors.
Task Engine Architecture#
The Task Engine consists of two main components: the Engine and the Virtual Machine (VM). The Engine manages task coordination, scheduling, and lifecycle, while the VM handles the actual compilation and execution of tasks.
Sources:
Engine Component#
The Engine component is responsible for:
- Managing task lifecycle (creation, execution, completion)
- Coordinating with operators for task triggers
- Interacting with storage to persist task data
- Handling task execution scheduling through the queue system
The Engine exposes methods for creating, triggering, and monitoring tasks, as well as managing secrets that can be used within tasks.
Sources:
Virtual Machine (VM)#
The VM is responsible for:
- Compiling tasks into executable plans
- Executing task plans node by node
- Managing execution state and variables
- Processing template expressions in task definitions
- Dispatching node execution to appropriate processors
The VM has a lifecycle with distinct states (initialize, compiled, ready, executing, completed) and maintains execution context through the entire task run.
Sources:
- core/taskengine/vm.go27-50
- core/taskengine/vm.go105-143
- core/taskengine/vm.go367-400
- core/taskengine/vm.go453-485
Task Execution Flow#
Task execution follows a well-defined flow from creation to completion. The Engine coordinates with operators for task triggers, while the VM handles the actual execution.
Sources:
- core/taskengine/engine.go288-320
- core/taskengine/engine.go569-626
- core/taskengine/vm.go410-451
- core/taskengine/vm.go453-485
Task States and Transitions#
Tasks and VM executions have well-defined states that track progress through the lifecycle:
Entity | State | Description |
---|---|---|
Task | Active | Task is active and can be triggered |
Task | Executing | Task is currently executing |
Task | Completed | Task has completed all executions |
Task | Failed | Task has failed |
Task | Canceled | Task has been canceled |
VM | VMStateInitialize | VM is initializing |
VM | VMStateCompiled | Task has been compiled into execution plan |
VM | VMStateReady | VM is ready for execution |
VM | VMStateExecuting | VM is currently executing |
VM | VMStateCompleted | VM has completed execution |
Sources:
Task Compilation and Execution#
Task Compilation#
The VM compiles tasks into execution plans by analyzing the edges between nodes and identifying the entry point. The compilation process:
- Creates Step objects for each node in the task
- Builds a map of nodes to their next steps based on edges
- Identifies the entry point based on the trigger node
- Sets the VM state to Ready when compilation is complete
Sources:
Task Execution#
The VM executes tasks node by node, following the execution plan created during compilation:
- VM starts with the entry point node
- For each node, it dispatches execution to the appropriate processor
- Results from each node are stored in the VM's variable map
- VM follows the Next pointers to determine the next node to execute
- Execution continues until there are no more nodes to execute
The VM supports various node types through specialized processors:
Node Type | Processor | Description |
---|---|---|
RestApi | RestProcessor | HTTP API calls |
ContractRead | ContractReadProcessor | Smart contract read operations |
ContractWrite | ContractWriteProcessor | Smart contract write operations |
GraphQLQuery | GraphQLQueryProcessor | GraphQL API queries |
CustomCode | JSProcessor | Custom JavaScript code execution |
Branch | BranchProcessor | Conditional branching logic |
Sources:
Template Processing#
The VM includes a template processing system that allows for dynamic content in task definitions. Templates use {{ expression }}
syntax and can reference:
- Variables from previous node outputs
- Trigger data
- User-defined secrets
The preprocessText
method evaluates these expressions using a JavaScript engine:
Sources:
Common Patterns and Use Cases#
Workflow Definition#
Tasks are defined as directed acyclic graphs (DAGs) with nodes representing operations and edges representing execution flow:
Tasks can be triggered manually or automatically via events, blocks, or time-based triggers.
Sources:
Variable Sharing Between Nodes#
Outputs from each node are stored in the VM's variable map and can be referenced by subsequent nodes:
This allows for data passing between nodes in the execution flow.
Sources:
Integration with Other Components#
Storage Integration#
The Task Engine uses BadgerDB for persistent storage of:
- Task definitions
- Execution results
- User secrets
- Smart wallet configurations
The storage layer is accessed through the db
field in both Engine and VM.
Sources:
Operator Integration#
The Task Engine interacts with operators through:
StreamCheckToOperator
: Streams task definitions to operatorsAggregateChecksResult
: Processes trigger notifications from operators
Operators monitor for trigger conditions and notify the Engine when conditions are met.
Sources:
Queue Integration#
The Task Engine uses a queue system (apqueue
) for asynchronous task execution:
- When a task is triggered, it's enqueued for execution
- The engine creates an execution record in the "queued" state
- The queue system processes the task asynchronously
- The execution record is updated with the final result
Sources:
Key APIs and Usage#
Task Creation#
CreateTask(user *model.User, taskPayload *avsproto.CreateTaskReq) (*model.Task, error)
Creates a new task with the specified nodes, edges, and trigger.
Sources:
Task Triggering#
TriggerTask(user *model.User, payload *avsproto.UserTriggerTaskReq) (*avsproto.UserTriggerTaskResp, error)
Manually triggers a task execution. Can be run in blocking or non-blocking mode.
Sources:
Task Listing#
ListTasksByUser(user *model.User, payload *avsproto.ListTasksReq) (*avsproto.ListTasksResp, error)
Lists tasks belonging to a user, with pagination support.
Sources:
Execution Listing#
ListExecutions(user *model.User, payload *avsproto.ListExecutionsReq) (*avsproto.ListExecutionsResp, error)
Lists executions for a given task or set of tasks, with pagination support.
Sources:
Advanced Features#
Secret Management#
The Task Engine provides APIs for managing secrets that can be used in tasks:
CreateSecret(user *model.User, payload *avsproto.CreateOrUpdateSecretReq) (bool, error)
UpdateSecret(user *model.User, payload *avsproto.CreateOrUpdateSecretReq) (bool, error)
ListSecrets(user *model.User, payload *avsproto.ListSecretsReq) (*avsproto.ListSecretsResp, error)
DeleteSecret(user *model.User, payload *avsproto.DeleteSecretReq) (bool, error)
Secrets are stored securely and can be referenced in tasks using {{ apContext.configVars.secretName }}
.
Sources:
Smart Wallet Integration#
The Task Engine integrates with smart wallets for:
- Task ownership verification
- Contract interactions
- Execution permissions
Smart wallet configurations are passed to the VM for use in contract interactions.
Sources:
Conclusion#
The Task Engine provides a powerful, flexible system for defining and executing complex workflows. It combines the management capabilities of the Engine with the execution capabilities of the VM to create a comprehensive task execution environment. The various processors support a wide range of operations, from HTTP calls to smart contract interactions, allowing for sophisticated automations and integrations.
Feature this wiki to auto refresh weekly