Vertical Pod Autoscaling

Overview

A KasprApp can ask the operator for a VerticalPodAutoscaler (VPA), which sets the resources of the app’s Kaspr container from their observed usage. It is opt-in: the operator creates nothing unless verticalPodAutoscaler.enabled is true.

When to use it

Vertical autoscaling suits apps whose event flow is volatile and unpredictable: bursts of many events that take long to process, in apps with large partitioned tables. Such apps are best spread across many instances, so that a burst is processed by all of them. But traffic is not always heavy, and instances sized for the bursts leave pods and CPU idle the rest of the time. Adding and removing instances as the traffic changes is costly too: every partition that moves to another instance must first replay its tables’ changelogs there to rebuild its state, which for large tables is slow and wasteful.

A VerticalPodAutoscaler keeps such an app scaled out, and adjusts the CPU its instances request to their utilization instead. Instances are resized in place: they keep running and keep their partitions, so neither the app nor its event flow is disrupted.

To set it up:

• Keep replicas at the number of instances the bursts need (at most the number of partitions).
• Set the CPU limit in resources to what an instance may use in a burst. With the default controlledValues: RequestsOnly, it stays the ceiling.
• Enable the VerticalPodAutoscaler with the default updateMode: InPlaceOrRecreate. It follows usage over time rather than single spikes: the limit lets instances burst above their request while their node has CPU to spare, and the VPA keeps the CPU they reserve close to what they use.

Quick Start

apiVersion: kaspr.io/v1alpha1
kind: KasprApp
metadata:
  name: orders
spec:
  replicas: 3
  bootstrapServers: kafka:9092
  resources:                  # what pods start with
    requests: {cpu: 500m, memory: 1Gi}
    limits: {cpu: "2", memory: 2Gi}
  verticalPodAutoscaler:
    enabled: true
    minAllowed: {cpu: 100m}
    maxAllowed: {cpu: 1500m}
  storage:
    type: persistent-claim
    size: 1Gi

The operator creates a VerticalPodAutoscaler named orders-app-vpa, owned by the app (it goes away with the app, or when enabled is turned off). By default it sets only the CPU requests of the Kaspr container (controlledValues: RequestsOnly, so the limits stay ceilings), and applies them to running pods in place (updateMode: InPlaceOrRecreate). All settings are in the API reference.

kubectl get kasprapp orders -o jsonpath='{.status.conditions[?(@.type=="VerticalPodAutoscalerReady")]}'
kubectl describe vpa orders-app-vpa     # recommendations

Will pods restart?

The VPA never edits the KasprApp or its StatefulSet. It sets the recommended requests on each pod as it is created, and changes running pods whose requests drift out of the recommended range:

Changing spec.resources yourself restarts every pod, one at a time, as with any pod template change.

CPU and memory

CPU resizes take effect at once: the app does not need to notice them. A Kaspr worker runs its agents, serialization and Kafka client in one asyncio event loop under Python’s GIL, so it uses about one core for them; more CPU only helps work outside that loop (RocksDB background threads, native libraries that release the GIL, child processes). Set maxAllowed near what the app can use, and add replicas for more throughput.

Memory is not autoscaled by default. Kaspr sizes its RocksDB block cache from the container’s memory limit when it starts, so a memory limit changed in place would go unnoticed until the next restart. With controlledResources: [cpu, memory], the operator makes the container restart on memory resizes (resizePolicy), which changes the pod template and restarts the app’s pods once.

Status

None of these keep the app from running. Changes of the condition are also posted as events on the app.

The scale subresource

KasprApp has a scale subresource, so kubectl scale kasprapp orders --replicas=5 works and other autoscalers can target a KasprApp. A VerticalPodAutoscaler you write yourself must target the KasprApp, not its StatefulSet:

spec:
  targetRef:
    apiVersion: kaspr.io/v1alpha1
    kind: KasprApp
    name: orders