Rolling Restarts and Upgrades
This page explains how rolling restarts and upgrades work in WarpStream.
Last updated
Was this helpful?
This page explains how rolling restarts and upgrades work in WarpStream.
Last updated
Was this helpful?
The WarpStream Agents perform a graceful shutdown routine when they receive a sigterm signal. The graceful shutdown routine strives to minimize disruption to the cluster. By default, this graceful shutdown process takes 80 seconds to complete. However, many container orchestration frameworks will not wait that long for a container to shutdown gracefully.
For example, in Kubernetes the default graceful termination window is 30s. We recommend increasing this value to 5 minutes. In Kubernetes, the configuration value for this is called terminationGracePeriodSeconds.
In addition, for this graceful shutdown to work smoothly with your application, we recommend setting the metadata refresh interval on your client to 1 minute. See our for more details.
If you're unable to update your clients to refresh metadata more frequently, you can instead tune the Agents to spend longer than 80 seconds performing their clean shutdown routine by increasing the value of the -gracefulShutdownDuration flag or WARPSTREAM_GRACEFUL_SHUTDOWN_DURATION environment variable to a value higher than your client's configured Metadata refresh interval (5m by default for most clients). If you do this, you may also need to update the graceful termination window in Kubernetes or whatever software you're using to schedule your containers as described above.
Rolling restarts are handled by WarpStream gracefully as long as the Agent containers are allowed to complete their graceful shutdown process, and clients are appropriately tuned to refresh their cluster metadata frequently as describe in the section above.
Rolling upgrades behave just like rolling restarts and leverage the same graceful shutdown mechanism. The only difference is that the Agent docker image tag is changed. Upgrades are always backwards compatible and no manual upgrade steps are required except deploying the new Docker image version in an incremental manner.