API Reference

Metadata that all persisted resources must have.

The kaspr version. Defaults to the latest.

The number of desired instances.

The container image to use. Defaults to the corresponding image of the version configuration.

Bootstrap server names to connect to. This should be given as a comma separated list of <hostname>:<port> pairs.

TLS configuration. Provide an empty entry {} to enable TLS or exclude the field to disable TLS.

Authentication configuration for Kafka.

Compute Resources required by each instance on the application.

Disk storage required by each instance of the application.

Python package management configuration for automatic package installation via init containers.

Templates for Kubernetes resources created for the application, including the ServiceAccount, Pod, Service, Kaspr container and more.

Template for the ServiceAccount resource created for the application.

Template for the application Pod.

Template for the Service resource created for the application.

Template for the main Kaspr container.

Template for the Python packages init container.

Labels added to the Kubernetes resource as key-value pairs.

Annotations added to the Kubernetes resource as key-value pairs.

Metadata applied to the ServiceAccount resource.

Metadata applied to the Pod resource.

List of references to secrets in the same namespace to use for pulling any of the images used by this Pod.

Configures pod-level security attributes and common container settings.

The grace period in seconds between sending the termination signal and forcefully stopping the Pod. Value must be a non-negative integer.

Node selector to use for scheduling the Pod. If not specified, the Pod will be scheduled on any available node.

The Pod's affinity rules.

The Pod's tolerations.

The Pod's topology spread constraints.

The name of the priority class used to assign priority to the Pod.

The name of the scheduler used to dispatch this Pod. If not specified, the default scheduler is used.

Optional list of hosts and IPs injected into the Pod's hosts file.

Indicates whether information about services should be injected into Pod environment variables.

Additional volumes that can be mounted to the Pod.

Metadata applied to the Service resource.

Environment variables applied to the container. Supports literal value or valueFrom using secretKeyRef, configMapKeyRef, or fieldRef.

Security context for the container.

Additional volume mounts applied to the container.

Environment variables applied to the init container. Supports literal value or valueFrom using secretKeyRef, configMapKeyRef, or fieldRef.

Security context for the init container.

Additional volume mounts applied to the init container.

Environment variable name.

Literal value of the environment variable. Must be set exclusively with valueFrom.

Source for the environment variable value. Must be set exclusively with value, and must define exactly one of secretKeyRef, configMapKeyRef, or fieldRef.

Selects a value from a Kubernetes Secret key.

Selects a value from a Kubernetes ConfigMap key.

Selects a value from Pod fields via the Kubernetes downward API.

Path of the Pod field to select (for example, metadata.name).

API version of the schema the fieldPath is written against.

The default replication factor for topics created by the application.

Default number of partitions for topics created by the application.

This setting controls whether or not creation of topics is allowed.

This setting controls whether or not the message scheduler is enabled.

This setting controls whether or not cron scheduling support is enabled. When enabled, schedulerEnabled is also automatically enabled.

This setting controls whether or not scheduler debug statistics are printed to log.

Default number of partitions for internal scheduler related topics. Defaults to general topic partition configuration if not set.

How often we save checkpoint to storage (and to changelog topic)

Number of days the dispatcher will look back from current date to seek starting a point for dispatching messages when an checkpoint is not found. This is mostly used during initial app deployment.

How often we checkpoint the dispacher's location in the timetable (in seconds).

How often we checkpoint the janitor's location in the timetable (in seconds).

How often the janitor attempts to clean.

Number of seconds the janitor trails current from the highwater timetable location.

How often the cron ticker evaluates registry entries, in seconds.

Extra lookahead buffer, in seconds, beyond the tick interval for materializing cron fires.

Minimum allowed interval, in seconds, between consecutive cron fires. Cron expressions that fire more frequently will be rejected.

Default serializer for message keys across the app. Must be one of raw, json, pickle, or binary.

Default serializer for message values across the app. Must be one of raw, json, pickle, or binary.

This is the maximum write buffer size. It represents the amount of data to build up in memory before converting to a sorted on-disk file. The default is 64 MB.

Maximum number of write buffers (memtables) that can be built in memory at the same time.

Target size for files at level-1 in the LSM tree. Used to determine the size of the SST (Sorted String Table) files that RocksDB generates during compactions. The default is 64 MB.

Size for caching uncompressed data. Defauls to about 30% of application's total memory budget

Size for caching compressed data. Defaults to 254MB.

A Bloom filter in RocksDB is used to quickly check whether a key might be in an SST (Sorted String Table) file without actually reading the file, which can significantly improve read performance.

If set to true, index and filter blocks will be stored in block cache, together with all other data blocks.

Base HTTP path for serving web requests.

Port number between 1024 and 65535 to use for the web server.

Metadata that all persisted resources must have. An agent must declare a label kaspr.io/app with the value of the app name it will be run on.

Specification of the desired behavior of the KasprAgent.

A short description of the agent for documentation purpose.

If true, the agent runs one worker per assigned input topic partition instead of using the default shared-consumer mode. This is useful when processing depends on partition-local ordering or state.

Input configuration for the agent.

Output configuration for the agent. Processed events are sent out to the optional output.

Sequence of operations that handle input events. Processors are optional and can be used to modify or act on the input events before they are sent to the output.

If true, the agent will create the input topic/channel if it does not exist.

Buffering configuration. This allows the agent to collect multiple events before processing them.

Input topic configuration; mutually exclusive with channel.

Input channel configuration; mutually exclusive with topic.

Topic names to subscribe to. This can be a single topic or a comma-separated list of topics. Required, but mutually exclusive with pattern.

Regex pattern of topic names to subscribe to. Required, but mutually exclusive with name.

Serializer for the key portion of the kafka message. Must be one of raw, json, pickle, or binary.

Serializer for the value portion of the kafka message. Must be one of raw, json, pickle, or binary.

Name of the channel to subscribe to

Maximum number of events to buffer before processing them.

Time window for buffering events. Events will be processed after this duration, even if the max limit has not been reached. Examples: 5s, 1m, 1h.

List of topics to publish to.

Topic name

If true, the topic is declared lazily before the first publish.

Wait for broker acknoledgement before considering the message sent.

Serializer for the key portion of the kafka message. Must be one of raw, json, pickle, or binary.

Serializer for the value portion of the kafka message. Must be one of raw, json, pickle, or binary.

Number of partitions to use when declaring the topic.

Retention period for the topic in seconds when declaring it.

Whether log compaction is enabled when declaring the topic.

Whether log deletion is enabled when declaring the topic.

Number of replicas to use when declaring the topic.

Additional broker topic configuration applied when declaring the topic.

Custom code to be used to select the key of the message. The code must define a function that takes a single argument and returns a value.

Custom code to be used to select the value of the message. The code must define a function that takes a single argument and returns a value.

Custom code to be used to select the partition number of the message. The code must define a function that takes a single argument and returns an integer.

Custom code to be used to select the headers of the message. The code must define a function that takes a single argument and returns a key and value map.

Custom code to be used to select the messages to be sent. The code must define a function that takes a single argument and returns a boolean value. Predicates that return true will be sent to the output topic, while those that return false will be discarded.

Name of the function that is invoked. If code defines multiple functions, the first function is used as the entrypoint. To avoid ambiguity, it is recommended to explicilty specify the entrypoint.

Python code to be executed. The code must define a valid Python function to use as an entrypoint.

Sequence of operations to run.

Initialization code to run.

Chain of operations to run on input events.

A unique name for the operation. This is used to identify the operation in the pipeline.

List of tables which are made available in the operation's function.

Takes an input value and returns an output value which is passed down to the next operation or the agent's output .

Takes an input value and determines if the event is eligible to be passed down to the next operation or the agent's output. If the filter function returns true, the event is passed down, otherwise it is discarded.

Send a message to a Kafka topic as part of the operation.

The name of the KasprTable resource.

The name of the parameter in which the table is made available in the operation's function.

Topic name. Required unless nameSelector is provided.

Custom code to dynamically determine the topic name. Required unless name is provided.

Use the input value to the topicSend operation as the output value for the next pipeline step. If ack is also true, the downstream value becomes (originalValue, ackMetadata) when a send occurs.

Wait for broker acknoledgement before considering the message sent.

Serializer for the key portion of the kafka message. Must be one of raw, json, pickle, or binary.

Serializer for the value portion of the kafka message. Must be one of raw, json, pickle, or binary.

Custom code to be used to select the key of the message. The code must define a function that takes the current pipeline value and returns a value.

Custom code to be used to select the value of the message. The code must define a function that takes the current pipeline value and returns a value.

Custom code to be used to select the partition number of the message. The code must define a function that takes the current pipeline value and returns an integer.

Custom code to be used to select the headers of the message. The code must define a function that takes the current pipeline value and returns a key and value map.

Custom code to be used to determine if the message should be sent. Predicates that return true are sent, while those that return false skip the send. Predicate does not discard the pipeline value; use filter operations when you need to stop the pipeline branch.

Metadata that all persisted resources must have. A join must declare a label kaspr.io/app with the value of the app name it belongs to.

Specification of the desired behavior of the KasprJoin.

A short description of the join for documentation purposes.

Name of the left-side KasprTable resource. This is the "driving" table whose changes trigger foreign key extraction.

Name of the right-side KasprTable resource (the "lookup" table). Must belong to the same KasprApp.

Python function that extracts the join key from left-table values. The returned value is used to look up the corresponding record in the right table.

Join semantics. inner skips emission when the right side has no match. left emits with right=null when no match exists. Allowed values: inner, left.

Name of the output channel for joined results. KasprAgents reference this via input.channel.name.

Status of the referenced KasprApp. Contains name (string) and status (AppFound or AppNotFound).

Status of the referenced left KasprTable. Contains name (string) and status (TableFound or TableNotFound).

Status of the referenced right KasprTable. Contains name (string) and status (TableFound or TableNotFound).

Name of the ConfigMap containing the serialized join spec.

Hash of the current join configuration.

Metadata that all persisted resources must have.

Specification of the desired table configuration.

The name of the table.

A short description of the table.

If true, the table is global. Global tables have a full copy of the data on every node.

Python code that returns a default value when a key is not found in the table.

Serializer for the table key. Must be raw or json.

Serializer for the table value. Must be raw or json.

Number of partitions in the table's changelog topic.

Additional Kafka topic configurations applied when creating the table's changelog topic (e.g., retention.ms, cleanup.policy).

Advanced table configuration options.

Window configuration for the table. Enables tumbling or hopping windowed aggregation.

Tumbling window configuration. Windows are fixed-size, non-overlapping time intervals.

Hopping window configuration. Windows are fixed-size but advance by a step interval, so they may overlap.

How events are assigned to windows. Must be stream (event timestamp), now (wall-clock time), or custom (user-defined).

Python function that extracts a timestamp from the event for window assignment. Only used when relativeTo is custom.

The window duration (e.g., 30s, 1m, 1h, 1d).

How long to retain data for each window after it closes (e.g., 30s, 1m, 1h, 1d).

The window duration (e.g., 30s, 1m, 1h, 1d).

The interval at which new windows are created (e.g., 30s, 1m, 1h, 1d).

How long to retain data for each window after it closes (e.g., 30s, 1m, 1h, 1d).

Metadata that all persisted resources must have. A task must declare a label kaspr.io/app with the value of the app name it belongs to.

Specification of the desired behavior of the KasprTask.

The name of the task.

A short description of what this task does.

Whether to run this task only on the leader instance.

Optional recurring schedule. Omit to run once when the app becomes ready. Use cron or interval for recurring execution.

Sequence of operations that run when the task fires.

Cron expression defining the schedule.

Interval duration (e.g., 10s, 5m, 1h).

Sequence of operations to run.

Initialization code to run before task operations are invoked.

Chain of operations to run when the task fires.

A unique name for the operation. This is used to identify the operation in the pipeline.

List of tables which are made available in the operation's function.

Takes the current task value and returns the next value passed to the rest of the pipeline.

Takes the current task value and determines if execution should continue. If the filter returns true, the value is passed forward; otherwise it is discarded.

Send a message to a Kafka topic as part of the operation.

The name of the KasprTable resource.

The name of the parameter in which the table is made available in the operation's function.

Topic name. Required unless nameSelector is provided.

Custom code to dynamically determine the topic name. Required unless name is provided.

Use the input value to the topicSend operation as the output value for the next pipeline step. If ack is also true, the downstream value becomes (originalValue, ackMetadata) when a send occurs.

Wait for broker acknoledgement before considering the message sent.

Serializer for the key portion of the kafka message. Must be one of raw, json, pickle, or binary.

Serializer for the value portion of the kafka message. Must be one of raw, json, pickle, or binary.

Custom code to be used to select the key of the message. The code must define a function that takes a single argument and returns a value.

Custom code to be used to select the value of the message. The code must define a function that takes a single argument and returns a value.

Custom code to be used to select the partition number of the message. The code must define a function that takes a single argument and returns an integer.

Custom code to be used to select the headers of the message. The code must define a function that takes a single argument and returns a key and value map.

Custom code to be used to determine if the message should be sent. Predicates that return true are sent, while those that return false skip the send. Predicate does not discard the pipeline value; use filter operations when you need to stop the pipeline branch.

Name of the web view interface

A short description of the web view for documentation purpose.

Describes the request endpoint listener.

Describes the web view's response output.

Sequence of operations that run while handling the HTTP request.

HTTP method to use for the request. Must be one of GET, POST, PUT, DELETE, PATCH, HEAD, OPTIONS

Web path to listen on.

Sequence of operations to run.

Initialization code to run before web view operations are invoked.

Chain of operations to run while handling the request.

A unique name for the operation. This is used to identify the operation in the pipeline.

List of tables which are made available in the operation's function.

Takes the current request value and returns the next value passed to the rest of the pipeline.

Takes the current request value and determines if execution should continue. If the filter returns true, the value is passed forward; otherwise it is discarded.

Send a message to a Kafka topic as part of the operation.

The name of the KasprTable resource.

The name of the parameter in which the table is made available in the operation's function.

Topic name. Required unless nameSelector is provided.

Custom code to dynamically determine the topic name. Required unless name is provided.

Use the input value to the topicSend operation as the output value for the next pipeline step. If ack is also true, the downstream value becomes (originalValue, ackMetadata) when a send occurs.

Wait for broker acknoledgement before considering the message sent.

Serializer for the key portion of the kafka message. Must be one of raw, json, pickle, or binary.

Serializer for the value portion of the kafka message. Must be one of raw, json, pickle, or binary.

Custom code to be used to select the key of the message. The code must define a function that takes a single argument and returns a value.

Custom code to be used to select the value of the message. The code must define a function that takes a single argument and returns a value.

Custom code to be used to select the partition number of the message. The code must define a function that takes a single argument and returns an integer.

Custom code to be used to select the headers of the message. The code must define a function that takes a single argument and returns a key and value map.

Custom code to be used to determine if the message should be sent. Predicates that return true are sent, while those that return false skip the send. Predicate does not discard the pipeline value; use filter operations when you need to stop the pipeline branch.

Response content type. Must be one of text/html, application/json, text/plain, application/octet-stream

HTTP status code to return on success.

Headers as key and value pairs to return in the response

Custom function to transform the response body on success or error.

Custom function to determine the status code that is returned in response.

Custom function to determine the headers returned in response.

Function to transform response on success.

Function to transform response on error.

The type of authentication to use. Must be one of scram-sha-256, scram-sha-512, or plain

Username used for authentication.

Details of the kubernetes secret where the authentication password is stored.

The name of the Kubernetes secret resource containting the password.

The name of the key in the Secret under which the password is stored.

Storage type, must be ephemeral or persistent-claim

The storage class to use for dynamic volume allocation.

Specifies if the persistent volume claim has to be deleted when the app is undeployed.

List of Python packages to install (e.g., ["requests", "pandas>=2.0.0"]).

Base URL of the Python Package Index (e.g., https://pypi.org/simple).

Additional package index URLs to search.

Hosts to trust for SSL verification (skip certificate checks).

Credentials for private PyPI registry authentication.

Cache configuration for storing installed packages across restarts.

Installation behavior policy (retries, timeout, failure handling).

Resource requirements for the init container that installs packages.

Cache backend type. Must be pvc or gcs. When set to pvc, uses a PersistentVolumeClaim (requires RWX storage class). When set to gcs, uses Google Cloud Storage.

Enable PVC-based caching. When false (default), uses ephemeral emptyDir storage per pod. Only applies when type is pvc.

The storage class to use for the packages cache PVC.

Size of the packages cache PVC (e.g., 256Mi, 1Gi).

Access mode for the cache PVC. Only ReadWriteMany is currently supported for shared package cache.

Whether to delete the PVC when the app is undeployed.

GCS cache configuration. Required when type is gcs.

GCS bucket name for storing cached package archives.

Key prefix for cached archives in the bucket.

Maximum archive size to upload to GCS. Archives exceeding this size are skipped.

Reference to a Secret containing a GCS service account key JSON file.

Name of the Secret containing the service account key.

Key within the Secret containing the JSON file.

Number of retry attempts for package installation.

Timeout in seconds for package installation. Minimum value is 60.

Behavior on installation failure. block prevents the pod from starting. allow lets the pod start without the packages.

Reference to a Secret containing PyPI registry credentials.

Name of the Secret containing credentials.

Key in the Secret for the username.

Key in the Secret for the password.