## Release notes for kOps 1.29 series

# Significant changes

* Add support to --emit-per-nodegroup-metrics on the cluster autoscaler addon (emitPerNodegroupMetrics) [PR#16693](https://github.com/kubernetes/kops/pull/16693/)

## Deferred deletion / pruning phase

Some infrastructure changes are potentially disruptive to the continued
operation of the cluster.  For the most disruptive operations, particularly
those that break rolling-update of the cluster, we have started to use deferred
deletion to minimize the impact.  For example, on AWS we create a second NLB
during the `kops update` phase when we cannot change the NLB directly.
`kops update` will report that a `--prune` is needed.  To minimize disruption,
we recommend you perform this after a rolling-update, for example:

```
kops update $MYCLUSTER --yes --admin
kops rolling-update $MYCLUSTER --yes
kops update $MYCLUSTER --yes --admin --prune # NEW!
```

Deferred deletion is currently used to safely introduce security groups for NLBs on AWS,
and to move to an internal load balancer for `kops-controller` on GCP.

## Initial OpenTelemetry Support

We are starting to add (experimental) support for OpenTelemetry,
in particular Tracing support.  Setting `OTEL_EXPORTER_OTLP_TRACES_FILE`
will write a trace file which can then be read by the traceserver program.
More information and options are described in [docs/opentelemetry.md](/docs/opentelemetry.md).
The tracing data is not expected to be particularly useful for end-users in
this release; the (non-standard) recording approach is instead intended to
work well with our Prow end-to-end testing system so that developers can
optimize kOps.

Please note: this is *not* telemetry in the "phone-home" sense.
The kOps project does not collect data from your machine.  As an
open-source project we do not even want to collect any of your data.
Currently the only OpenTelemetry backend supported is writing to a
filesystem (and it is opt-in).  In future you will be able to configure
other OpenTelemetry backends, but this data will only be sent if
you enable OpenTelemetry, and only sent to where you configure.

## AWS

* Network Load Balancers in front of the Kubernetes API and bastion hosts now
have a security group attached. These security groups are used for security group rules
allowing incoming traffic to the NLBs as well as traffic between the NLBs and their target
instances.

* Posts event data to URL upon instance interruption action in aws-node-termination-handler with `WEBHOOK_URL`.

## GCP

* As of Kubernetes version 1.29, credentials for private GCR/AR repositories will be handled by the out-of-tree credential provider. This is an additional binary that each instance downloads from the assets repository.
* Two additional `StorageClasses` are created on GCP clusters. These are called `balanced-csi` and `ssd-csi` and utilize the GCP Balanced and SSD Persistent Disk volume types respectively.
* **Breaking Change** - the default `StorageClass` has been changed from `standard-csi` to `balanced-csi`.

* We now use a private load-balancer for in-cluster traffic on GCP, which allows us
  to use network tags to restrict access only to the cluster nodes.

# Breaking changes

## Other breaking changes

* `kops toolbox dump` limits the number of nodes dumped to 500 by default. Use `--max-nodes` to override.

* Support for Kubernetes version 1.23 has been removed.

# Known Issues

* The Amazon VPC CNI is now compatible with Ubuntu 22.04. Fix applied via [kubernetes/kops#16313](https://github.com/kubernetes/kops/issues/16313).

# Deprecations

* Support for Kubernetes version 1.24 is deprecated and will be removed in kOps 1.30.

* Support for Kubernetes version 1.25 is deprecated and will be removed in kOps 1.31.

* Support for AWS Classic Load Balancer for API is deprecated and should not be used for newly created clusters.

* All legacy addons (under `/addons`) are deprecated in favor of managed addons, including the [metrics server addon](https://github.com/kubernetes/kops/tree/master/addons/metrics-server) and the [autoscaler addon](https://github.com/kubernetes/kops/tree/master/addons/cluster-autoscaler).
