Kafka Configuration

This page describes the configuration surface for Kafka clusters in Surge. Understanding these fields helps you choose a cluster shape that matches your workload and avoid settings that cannot be changed later.

General Settings

Display Label: The human-readable name shown in the Viduli console. You can change it later.

Resource Name: The unique identifier for the Kafka resource inside the project. This name cannot be changed after the resource is created.

Engine and Version

Engine: Surge currently supports Kafka as the broker engine.

Kafka Version: The Kafka version used by the cluster.

Both the engine and version are fixed after creation.

Node Layout

Kafka clusters support two node layouts.

Combined: Controller and broker roles run on the same nodes. This is the simpler layout and is suitable for smaller environments. Combined clusters use a single node pool with a total node count between 1 and 9.

Independent: Controller nodes and broker nodes run in separate pools. This layout is the better choice when you want clearer separation between quorum and data-plane capacity. Controller count can be set between 1 and 5. Broker count can be set between 1 and 15.

The node layout cannot be changed after the cluster is created. If you expect to adjust broker capacity later, choose the independent layout at creation time.

Compute Resources

Kafka nodes use dedicated compute.

CPU: Per-node CPU allocation. Supported range: 125 millicores to 16 vCPU.

Memory: Per-node RAM allocation. Supported range: 512 MiB to 32 GiB.

Disk: Per-node disk allocation. Supported range: 5 GiB to 4 TiB.

In the independent layout, controller and broker pools each have their own CPU, memory, disk, and node-count settings.

Replication Settings

Mode: Kafka clusters currently use single-region replication mode.

Region: The deployment region for the cluster.

Replication mode and region are fixed after creation.

Topic Defaults

Topic defaults let you define baseline settings for topics created on the cluster. Surge exposes the following defaults in the configuration view:

Partitions: Default partition count.

Replication Factor: Default replica count for new topics. This can be a number or auto.

Min In-Sync Replicas: Default min.insync.replicas value. This can be a number or auto.

Cleanup Policy: Default log cleanup behavior.

Compression Type: Default compression used for new topics.

Timestamp Type: Default message timestamp mode.

Kafka also maintains internal topic defaults for consumer offsets and transaction logs. Those values are part of the cluster configuration and are resolved automatically when auto is used.

Process Settings

Process settings affect Kafka broker internals.

Network Threads: Number of network threads per listener.

IO Threads: Number of I/O threads used by the broker.

Heap Size (MB): JVM heap allocation per broker.

These fields support explicit numeric values or auto.

Leader Settings

Leader settings control how Kafka handles partition leadership.

Auto Rebalance: Whether the cluster automatically rebalances leaders.

Unclean Election: Whether Kafka may elect an out-of-sync replica as leader.

Imbalance Check Interval: How often Kafka checks for leader imbalance.

Changing Configuration After Creation

The Configuration tab lets you update supported settings on an existing cluster, but some fields are immutable.

The following fields cannot be changed after creation:

• engine
• Kafka version
• deployment region
• replication mode
• node layout (combined or independent)

For ongoing topic work, use the separate topic management page.