System Frames

Reference for SystemFrame types: pipeline lifecycle, interruptions, speaking state, input, and diagnostics

SystemFrames have higher priority than DataFrames and ControlFrames and are never cancelled during user interruptions. They are queued and processed in order with other SystemFrames. They carry signals that must always be delivered: pipeline startup and teardown, error notifications, user input, and speaking state changes. See the frames overview for base class details, mixin fields, and frame properties common to all frames.

Pipeline Lifecycle

StartFrame

The first frame pushed into a pipeline, marking the point at which frames start flowing. Every processor receives this before any DataFrames or ControlFrames arrive; a processor holds anything that reaches it earlier and delivers it, in arrival order, once the StartFrame has passed.

Pipeline configuration reaches processors through FrameProcessorSetup in setup(), which runs before this frame.

Reading audio_in_sample_rate, audio_out_sample_rate, enable_metrics, enable_tracing, enable_usage_metrics, report_only_initial_ttfb, or tracing_context off a StartFrame is deprecated since v1.8.0 and warns. These fields are removed in 2.0.0 — read them from the FrameProcessorSetup passed to setup() instead:

async def setup(self, setup: FrameProcessorSetup):
    await super().setup(setup)
    self._sample_rate = setup.audio_out_sample_rate

CancelFrame

Stops the pipeline immediately, skipping any queued non-SystemFrames. Use this when you need to abort without waiting for pending work to drain. For example, when the user has left the session.

reason · Any | None · default: "None"

Optional reason for the cancellation.

Errors

ErrorFrame

Carries an error notification, typically pushed upstream so earlier processors can react.

error · str · required

Human-readable error message.

category · ErrorCategory | None · default: "None"

What kind of failure this was — a rejected API key (AUTHENTICATION) versus a provider outage (SERVER), for example. See Error Handling.

processor · FrameProcessor | None · default: "None"

The processor that raised the error. Read processor.is_usable to tell an error the processor can carry on from apart from one that has finished it.

exception · Exception | None · default: "None"

The underlying exception, if one was caught.

fatal · bool · default: "False" · deprecated

Deprecated in v1.8.0. Will be removed in 2.0.0. Check processor.is_usable instead, and set the pipeline’s processor_unusable_policy to decide what an unusable processor does to the run.

FatalErrorFrame

Deprecated in v1.8.0, removed in 2.0.0. Report the error with push_error(..., force_treat_as_permanent=True) when it leaves its processor unable to work, or push an EndWorkerFrame after a plain ErrorFrame when the pipeline should stop for a reason that isn’t about a processor’s state. See Migrating from fatal errors.

An unrecoverable error requiring the bot to shut down. The fatal field is always True.

Inherits from ErrorFrame.

Processor Pause/Resume

Resuming always travels the high-priority input queue, so a paused processor is reached regardless of what has queued up behind the pause. Pausing comes in two forms: FrameProcessorPauseFrame is a ControlFrame that takes effect in order, after the frames ahead of it, while FrameProcessorPauseUrgentFrame takes effect immediately.

FrameProcessorResumeFrame

Resumes a previously paused processor, releasing all buffered frames for processing in the order they arrived.

processor · FrameProcessor

The processor to resume.

FrameProcessorPauseUrgentFrame

Pauses a processor immediately, without waiting for queued frames to drain first.

processor · FrameProcessor

The processor to pause.

FrameProcessorResumeUrgentFrame

Equivalent to FrameProcessorResumeFrame. Either resumes a paused processor and releases its buffered frames.

processor · FrameProcessor

The processor to resume.

Interruptions

InterruptionFrame

Interrupts the pipeline, discarding pending DataFrames and ControlFrames. Typically triggered when the user starts speaking during a bot response.

User Speaking State

UserStartedSpeakingFrame

Indicates that a user turn has begun. By this point, transcriptions are usually already flowing through the pipeline.

UserStoppedSpeakingFrame

Marks the end of a user turn. The bot’s response is triggered separately by the turn detection system.

ProposedUserStartedSpeakingFrame

Proposes that a user turn has started. Emitted by a component with its own turn detection, typically an STT or realtime LLM service whose provider reports speech boundaries. It is a proposal rather than a decision: an ExternalUserTurnStartStrategy resolves it into a UserStartedSpeakingFrame and broadcasts the interruption.

Its end-of-turn counterpart, ProposedUserStoppedSpeakingFrame, is a ControlFrame. Resolving a start proposal broadcasts an interruption, which has to reach the pipeline ahead of whatever is queued; resolving a stop proposal has to stay ordered against the final TranscriptionFrame, since the strategy needs that text in hand to close the turn on.

UserSpeakingFrame

Emitted by the VAD processor while the user is actively speaking. Useful for UI feedback or suppressing idle timeouts.

UserTurnInferenceCompletedFrame

Indicates that the user turn is semantically complete. Emitted by any component that can judge conversational turn completeness — for example an LLM with turn-completion markers, an STT service with built-in turn detection, or a dedicated end-of-turn classifier. Stop strategies that gate the user-turn-stop event on an external completeness signal (e.g. LLMTurnCompletionUserTurnStopStrategy) consume this frame to finalize the turn. An absence of this frame means the turn is not yet considered complete.

UserMuteStartedFrame

Broadcast when one or more user mute strategies activate. User mute temporarily suppresses user input while the bot is speaking to prevent interruptions. While muted, the LLMUserAggregator drops incoming user frames (InputAudioRawFrame, TranscriptionFrame, InterimTranscriptionFrame, UserStartedSpeakingFrame, UserStoppedSpeakingFrame, ProposedUserStartedSpeakingFrame, ProposedUserStoppedSpeakingFrame, VAD signals, and InterruptionFrame). Lifecycle frames (StartFrame, EndFrame, CancelFrame) are never muted.

UserMuteStoppedFrame

Broadcast when all active user mute strategies deactivate, allowing user input to be processed again.

VAD Events

These frames are emitted directly by the Voice Activity Detection (VAD) processor and carry timing metadata. Higher-level speaking-state frames (UserStartedSpeakingFrame, UserStoppedSpeakingFrame) are derived from these.

VADUserStartedSpeakingFrame

VAD confirmed that speech has started.

start_secs · float · default: "0.0"

Timestamp in seconds when speech onset was detected.

timestamp · float · default: "time.time()"

Wall-clock time when the frame was created.

VADUserStoppedSpeakingFrame

VAD confirmed that speech has ended.

stop_secs · float · default: "0.0"

Timestamp in seconds when speech ended.

timestamp · float · default: "time.time()"

Wall-clock time when the frame was created.

SpeechControlParamsFrame

Notifies processors that VAD or turn detection parameters have changed at runtime.

vad_params · VADParams | None · default: "None"

Updated VAD parameters.

turn_params · BaseTurnParams | None · default: "None"

Updated turn detection parameters.

Bot Speaking State

BotStartedSpeakingFrame

Emitted by the output transport when the bot begins speaking. Broadcast in both directions so processors on either side of the transport can react.

BotStoppedSpeakingFrame

Emitted by the output transport when the bot finishes speaking. Also broadcast in both directions.

BotSpeakingFrame

Emitted continuously while the bot is speaking. Processors can use this to suppress idle timeouts or drive visual indicators.

Connection Status

BotConnectedFrame

The bot has joined the transport room. Only relevant for SFU-based transports: Daily, LiveKit, HeyGen, and Tavus.

ClientConnectedFrame

A client or participant has connected to the transport.

Input Frames

Input frames carry raw data from transport sources into the pipeline. As SystemFrames, they are never discarded during interruptions. Incoming user data must always be processed.

InputAudioRawFrame

Raw audio received from the transport. Inherits the audio, sample_rate, num_channels, and num_frames fields from the AudioRawFrame mixin.

Inherits from AudioRawFrame.

UserAudioRawFrame

Audio from a specific user in a multi-participant session.

Inherits from InputAudioRawFrame.

user_id · str · default: "\"\""

Identifier for the user who produced this audio.

InputImageRawFrame

Raw image received from the transport. Inherits image, size, and format from the ImageRawFrame mixin.

Inherits from ImageRawFrame.

UserImageRawFrame

An image from a specific user, optionally tied to a pending image request.

Inherits from InputImageRawFrame.

user_id · str · default: "\"\""

Identifier for the user who produced this image.

text · str | None · default: "None"

Optional text associated with the image.

append_to_context · bool | None · default: "None"

Whether to append this image to the LLM context.

request · UserImageRequestFrame | None · default: "None"

The original request frame that triggered this image capture.

InputTextRawFrame

Text received from the transport, such as a user typing in a chat interface. Inherits the text field from TextFrame.

Inherits from TextFrame.

DTMF Input

InputDTMFFrame

A DTMF keypress received from the transport. Inherits the button field from the DTMFFrame mixin.

Inherits from DTMFFrame.

OutputDTMFUrgentFrame

A DTMF keypress for immediate output, bypassing the normal frame queue.

Inherits from DTMFFrame.

Transport Messages

InputTransportMessageFrame

A message received from an external transport. The message format is transport-specific.

message · Any · required

The transport message payload.

OutputTransportMessageUrgentFrame

An outbound transport message that bypasses the normal queue for immediate delivery.

message · Any · required

The transport message payload.

Function Calling

FunctionCallsStartedFrame

Signals that one or more function calls are about to begin executing.

function_calls · Sequence[FunctionCallFromLLM] · required

Sequence of function calls that will be executed.

FunctionCallCancelFrame

Signals that a function call was cancelled, typically due to user interruption when the function’s cancel_on_interruption flag is set.

function_name · str · required

Name of the function that was cancelled.

tool_call_id · str · required

Unique identifier for the cancelled function call.

User Interaction

UserImageRequestFrame

Requests an image from a specific user, typically to capture a camera frame for vision processing.

user_id · str · required

Identifier for the user to capture from.

text · str | None · default: "None"

Optional text prompt associated with the image request.

append_to_context · bool | None · default: "None"

Whether to append the resulting image to the LLM context.

video_source · str | None · default: "None"

Specific video source to capture from.

function_name · str | None · default: "None"

Function name if this request originated from a tool call.

tool_call_id · str | None · default: "None"

Tool call identifier if this request originated from a tool call.

result_callback · Any | None · default: "None"

Callback to invoke with the captured image result.

STTMuteFrame

Mutes or unmutes the STT service. While muted, incoming audio is not sent to the STT provider.

mute · bool · required

True to mute, False to unmute.

UserIdleTimeoutUpdateFrame

Updates the user idle timeout at runtime. Setting the timeout to 0 disables idle detection, and setting a positive value enables it. Updates apply immediately: a running idle timer restarts with the new duration, and if the bot is waiting for the user to speak, the timer is armed right away.

timeout · float · required

New idle timeout in seconds. 0 disables detection.

Diagnostics

MetricsFrame

Performance metrics collected from processors. Emitted when metrics reporting is enabled in the pipeline configuration.

data · List[MetricsData] · required

List of metrics data entries.

Service Metadata

ServiceMetadataFrame

Base metadata frame broadcast by services at startup, providing information about service capabilities and configuration.

service_name · str · required

Name of the service that emitted this metadata.

user_turn_strategies · UserTurnStrategies | None · default: "None"

The turn strategies the service recommends, when it has an opinion — a service with its own turn detection uses this to ask for ExternalUserTurnStrategies.

STTMetadataFrame

Metadata from an STT service, including latency characteristics used for turn detection tuning.

Inherits from ServiceMetadataFrame.

ttfs_p99_latency · float · required

P99 latency in seconds for time-to-final-segment. Used by turn detectors to calibrate wait times.

LLMServiceMetadataFrame

Broadcast at pipeline start by an LLM service. Downstream processors — notably LLMContextAggregatorPair — read is_realtime_service to tell whether a realtime (speech-to-speech) service is in the pipeline and configure themselves accordingly. See Realtime (Speech-to-Speech) Services.

Inherits from ServiceMetadataFrame.

is_realtime_service · bool · default: "False"

Whether the broadcasting service is a realtime (speech-to-speech) LLM service.

RTVI

Frames for the Real-Time Voice Interface (RTVI) protocol, which bridges clients and the pipeline. These frames handle custom messaging between the client and server.

RTVIServerMessageFrame

Sends a server message to the connected client.

data · Any · required

The message data to send to the client.

RTVIClientMessageFrame

A message received from the client, expecting a server response via RTVIServerResponseFrame.

msg_id · str · required

Unique identifier for the client message.

type · str · required

The message type.

data · Any | None · default: "None"

Optional message data from the client.

RTVIServerResponseFrame

Responds to an RTVIClientMessageFrame. Include the original client message frame to ensure the response is properly correlated. Set the error field to respond with an error instead of a normal response.

client_msg · RTVIClientMessageFrame · required

The original client message this response is for.

data · Any | None · default: "None"

Response data to send to the client.

error · str | None · default: "None"

Error message. When set, the client receives an error-response instead of a server-response.

Pipeline Worker Frames

Pipeline worker frames provide a system-priority mechanism for requesting pipeline actions from outside the normal frame flow. They are converted into their corresponding standard frames when processed.

WorkerSystemFrame

Base class for system-priority worker frames.

CancelWorkerFrame

Requests immediate pipeline cancellation. Converted to a CancelFrame when processed by the pipeline.

Inherits from WorkerSystemFrame.

reason · Any | None · default: "None"

Optional reason for the cancellation request.

InterruptionWorkerFrame

Requests a pipeline interruption. Converted to an InterruptionFrame when processed.

Inherits from WorkerSystemFrame.