Client Configuration

This pages explains how to configure your Apache Kafka client with WarpStream.

Don't forget to review our documentation for tuning your Kafka client for maximum performance with WarpStream once you're done. A few small changes in client configuration can result in 10-20x higher throughput when using WarpStream.

WarpStream provides API-compatibility with Apache Kafka, so you can just connect your clients to the WarpStream agents by setting the WarpStream Application Bootstrap URL (obtained from the WarpStream console) as the value in the Kafka bootstrap settings. For example, using the librdkafka client in Go:

var (
    // Not explicitly required, but will eliminate inter-zone networking
    // in multi-zone deployments.
    availabilityZone = lookupAZ()
    // Not explicitly required, but may help with debugging to isolate
    // individual clients in logs and telemetry.
    sessionID = uuid.New().String()
    // The string you would normally use as your client ID with regular
    // Apache Kafka.
    applicationID = "application-foo"
    clientID = fmt.Sprintf("%s,ws_si=%s,ws_az=%s", sessionID, availabilityZone)
    bootstrapServer = "api-80ba097c-d4ef-4e0b-8e86-d05b80fee6ed.kafka.discoveryv2.prod-z.us-east-1.warpstream.com:9092"
)

producerConfig := map[string]kafka.ConfigValue{
	// Not developing locally? In K8s you should use the Agent
	// service from Agent chart. If you're not using K8s, then
	// you can use our convenience hosted bootstrap URL which
	// you can find in the "Connect" tab of the virtual cluster
	// view in the WarpStream console.
	"bootstrap.servers": "localhost:9092",
	"broker.address.family": "v4",
	"log.connection.close":  "false",
	"client.id": clientID,
}

config := kafka.ConfigMap(producerConfig)
producer, err := kafka.NewProducer(&config)
if err != nil {
	return fmt.Errorf("error initialiazing kafka producer: %w", err)
}

Client IDs

If you just want to get up and running quickly, you can use a regular client ID and skip encoding your application's availability zone in the client ID. However, beware that you may incur higher costs due to inter-zone networking.

You can read more about the WarpStream service discovery system in our Service Discovery reference documentation, but if you want to take advantage of WarpStream's zone-aware service discovery system and achieve good load balancing, you must encode your application's availability zone in your Kafka client's client ID using the format in the code sample above.

To learn more about the WarpStream-specific client ID features (like ws_si and ws_az) check out our documentation on configuring Kafka Client ID features.

Follow this documentation to learn how to determine your application's availability zone in all major cloud environments and properly template your client ID.