Longhorn v1.6.3-rc2

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Feature

• [BACKPORT][v1.6.3][FEATURE] Add additional monitoring settings to ServiceMonitor resource. 8984 - @ejweber @chriscchien

Improvement

• [BACKPORT][v1.6.3][IMPROVEMENT] Fix contradicting node status events 9327 - @ejweber @roger-ryao
• [BACKPORT][v1.6.3][IMPROVEMENT] Always update the built-in installed system packages when building component images 8722 - @yangchiu @c3y1huang
• [BACKPORT][v1.6.3][IMPROVEMENT] Update sizes in Engine and Volume resources less frequently 8684 - @ejweber @roger-ryao
• [BACKPORT][v1.6.3][IMPROVEMENT] Longhor Manager Flood with "Failed to get engine proxy of ... cannot get client for engine" Message 8729 - @derekbit @roger-ryao
• [BACKPORT][v1.6.3][IMPROVEMENT] Restore Latest Backup should be applied with BackingImage name value 8671 - @a110605 @roger-ryao
• [BACKPORT][v1.6.3][IMPROVEMENT] Improve and simplify chart values.yaml 8636 - @ChanYiLin @chriscchien
• [BACKPORT][v1.6.3][IMPROVEMENT] BackingImage UI improvement 8655 - @a110605 @roger-ryao
• [BACKPORT][v1.6.3][IMPROVEMENT] Saving Settings page changes 8602 - @a110605 @roger-ryao
• [BACKPORT][v1.6.3][IMPROVEMENT] The client-go rest client rate limit inside the csi sidecar component might be too small (csi-provisioner, csi-attacjer. csi-snappshotter, csi-attacher) 8726 - @PhanLe1010
• [BACKPORT][v1.6.3][IMPROVEMENT] Add setting to configure support bundle timeout for node bundle collection 8624 - @c3y1huang @chriscchien
• [BACKPORT][v1.6.3][IMPROVEMENT] Problems mounting XFS volume clones / restored snapshots 8797 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.6.3][IMPROVEMENT] Cannot expand a volume created by Longhorn UI 8828 - @mantissahz
• [BACKPORT][v1.6.3][IMPROVEMENT] environment_check.sh should check for the iscsi_tcp kernel module 8720 - @tserong @roger-ryao
• [BACKPORT][v1.6.3][IMPROVEMENT] toomanysnapshots UI element not prominent enough to prevent runaway snapshots 8672 - @a110605 @roger-ryao

Bug

• [BUG] Regression in 1.6.x-head, significant increase in execution time 9439 - @ChanYiLin @roger-ryao
• [BUG] test case test_recurring_job the backup recurring job's retain is not working on v1.6.x-head for amd64 9454 - @mantissahz @chriscchien
• [BACKPORT][v1.6.3][BUG] Canceling expansion results in a volume expansion error 9469 - @derekbit
• [BUG] test case test_support_bundle_should_not_timeout timeout on v1.6.x-head for amd64 9452 - @yangchiu @derekbit
• [BACKPORT][v1.6.3][BUG] Replica Auto Balance options under General Setting and under Volume section should have similar case 8786 - @yangchiu @a110605
• [BACKPORT][v1.6.3][BUG] kubectl drain node is blocked by unexpected orphan engine processes 9446 - @ejweber @chriscchien @roger-ryao
• [BUG][UI][v1.6.x] blank dropdown menu in update volume property modals 9465 - @a110605
• [BACKPORT][v1.6.3][BUG] Instance manager missing required selector labels after manager crash 9472 - @c3y1huang @chriscchien
• [BACKPORT][v1.6.3][BUG] Engine Upgrade to 1.7.1 fails on volumes with strict-local data locality 9447 - @james-munson @chriscchien
• [BUG][v1.6.x] Abnormal snapshot missing status field 9438 - @yangchiu @derekbit
• [BACKPORT][v1.6.3][BUG] Accidentally encountered a single replica volume backup stuck at progress 17% indefinitely after a node rebooted 9399 - @yangchiu @ChanYiLin
• [BUG] Security issues in longhorn 1.6.2 version images 9132 - @c3y1huang
• [BACKPORT][v1.6.3][BUG] Longhorn did not close and open encrypted volumes correctly when the service k3s-agent restarted for a while 9386 - @mantissahz @roger-ryao
• [BACKPORT][v1.6.3][BUG] Non-existing block device results in longhorn-manager to be in Crashloopbackoff state 9074 - @yangchiu @derekbit
• [BACKPORT][v1.6.3][BUG] Longhorn keeps resetting my storageClass 9395 - @mantissahz @roger-ryao
• [BACKPORT][v1.6.3][BUG] Volume failed to create healthy replica after data locality and replica count changed and got stuck in degraded state forever 8561 - @ejweber @chriscchien @roger-ryao
• [BACKPORT][v1.6.3][BUG] [Backupstore] Need to close the reader after downloading files for the Azure backup store driver. 9283 - @yangchiu @mantissahz
• [BACKPORT][v1.6.3][BUG] LH fails silently when node has attached volumes 9211 - @yangchiu @ejweber
• [BACKPORT][v1.6.3][BUG] Fix longhorn-manager TestCleanupRedundantInstanceManagers 8670 - @derekbit @roger-ryao
• [BACKPORT][v1.6.3][BUG] Longhorn thinks node is unschedulable 9052 - @c3y1huang @roger-ryao
• [BACKPORT][v1.6.3][BUG] Volume stuck in degraded 9295 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.6.3][BUG] test case test_system_backup_and_restore_volume_with_backingimage failed on sle-micro ARM64 9227 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.6.3][BUG] System Backup Fails and DR Volume Enters Attach-Detach Loop When Volume Backup Policy is Set to Always 9339 - @c3y1huang @roger-ryao
• [BACKPORT][v1.6.3][BUG] v1 volume replica rebuld fail after upgade from v1.7.0 to v1.7.1-rc1 9336 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.6.3][BUG] instance-manager is stuck at starting state 8678 - @derekbit
• [BACKPORT][v1.6.3][BUG] instance-manager pod for v2 volume is killed due to a failed liveness probe. 8808 - @derekbit @chriscchien
• [BACKPORT][v1.6.3][BUG] Pod auto-deletion may cause thousands of logs 9020 - @ejweber @roger-ryao
• [BACKPORT][v1.6.3][BUG] Share manager controller reconciles tens of thousands of times 9088 - @ejweber @roger-ryao
• [BACKPORT][v1.6.3][BUG] Scale replica snapsots warning 8851 - @ejweber
• [BACKPORT][v1.6.3][BUG] Longhorn can no longer create XFS volumes smaller than 300 MiB 8560 - @ejweber @chriscchien
• [BACKPORT][v1.6.3][BUG]filesystem trim RecurringJob times out (volumes where files are frequently created and deleted) 9048 - @c3y1huang @chriscchien
• [BACKPORT][v1.6.3][BUG] Can not revert V2 volume snapshot after upgrade from v1.6.2 to v1.7.0-dev 9066 - @chriscchien @DamiaSan
• [BACKPORT][v1.6.3][BUG] Rebuilding Replica fails on larger volumes 8949 -
• [BACKPORT][v1.6.3][BUG] Orphan longhorn-engine-manager and longhorn-replica-manager services 8858 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.6.3][BUG] When revision counter is disabled, the engine might choose a replica with a smaller head size to be the source of truth for auto-salvage 8661 - @PhanLe1010
• [BACKPORT][v1.6.3][BUG] toomanysnapshots UI message displays incorrect snapshot count 8700 - @ejweber
• [BACKPORT][v1.6.3][BUG] Uninstallation will fail if invalid backuptarget is set. 8793 - @mantissahz @chriscchien

Misc

• [BACKPORT][v1.6.3][TASK] Update the best practice page to mention these broken kernels 8882 - @PhanLe1010

Contributors

• @ChanYiLin
• @DamiaSan
• @PhanLe1010
• @a110605
• @c3y1huang
• @chriscchien
• @derekbit
• @ejweber
• @innobead
• @james-munson
• @mantissahz
• @roger-ryao
• @tserong
• @yangchiu

Longhorn v1.6.3-rc1

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Feature

• [BACKPORT][v1.6.3][FEATURE] Add additional monitoring settings to ServiceMonitor resource. 8984 - @ejweber @chriscchien

Improvement

• [BACKPORT][v1.6.3][IMPROVEMENT] Fix contradicting node status events 9327 - @ejweber @roger-ryao
• [BACKPORT][v1.6.3][IMPROVEMENT] Always update the built-in installed system packages when building component images 8722 - @yangchiu @c3y1huang
• [BACKPORT][v1.6.3][IMPROVEMENT] Update sizes in Engine and Volume resources less frequently 8684 - @ejweber @roger-ryao
• [BACKPORT][v1.6.3][IMPROVEMENT] Longhor Manager Flood with "Failed to get engine proxy of ... cannot get client for engine" Message 8729 - @derekbit @roger-ryao
• [BACKPORT][v1.6.3][IMPROVEMENT] Restore Latest Backup should be applied with BackingImage name value 8671 - @a110605 @roger-ryao
• [BACKPORT][v1.6.3][IMPROVEMENT] Improve and simplify chart values.yaml 8636 - @ChanYiLin @chriscchien
• [BACKPORT][v1.6.3][IMPROVEMENT] BackingImage UI improvement 8655 - @a110605 @roger-ryao
• [BACKPORT][v1.6.3][IMPROVEMENT] Saving Settings page changes 8602 - @a110605 @roger-ryao
• [BACKPORT][v1.6.3][IMPROVEMENT] The client-go rest client rate limit inside the csi sidecar component might be too small (csi-provisioner, csi-attacjer. csi-snappshotter, csi-attacher) 8726 - @PhanLe1010
• [BACKPORT][v1.6.3][IMPROVEMENT] Add setting to configure support bundle timeout for node bundle collection 8624 - @c3y1huang @chriscchien
• [BACKPORT][v1.6.3][IMPROVEMENT] Problems mounting XFS volume clones / restored snapshots 8797 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.6.3][IMPROVEMENT] Cannot expand a volume created by Longhorn UI 8828 - @mantissahz
• [BACKPORT][v1.6.3][IMPROVEMENT] environment_check.sh should check for the iscsi_tcp kernel module 8720 - @tserong @roger-ryao
• [BACKPORT][v1.6.3][IMPROVEMENT] toomanysnapshots UI element not prominent enough to prevent runaway snapshots 8672 - @a110605 @roger-ryao

Bug

• [BACKPORT][v1.6.3][BUG] Non-existing block device results in longhorn-manager to be in Crashloopbackoff state 9074 - @yangchiu @derekbit
• [BACKPORT][v1.6.3][BUG] Accidentally encountered a single replica volume backup stuck at progress 17% indefinitely after a node rebooted 9399 - @yangchiu @ChanYiLin
• [BACKPORT][v1.6.3][BUG] Replica Auto Balance options under General Setting and under Volume section should have similar case 8786 - @yangchiu @a110605
• [BACKPORT][v1.6.3][IMPROVEMENT] System restore unable to restore volume with backing image 8629 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.6.3][BUG] Longhorn did not close and open encrypted volumes correctly when the service k3s-agent restarted for a while 9386 - @mantissahz @roger-ryao
• [BACKPORT][v1.6.3][BUG] Longhorn keeps resetting my storageClass 9395 - @mantissahz @roger-ryao
• [BACKPORT][v1.6.3][BUG] Volume failed to create healthy replica after data locality and replica count changed and got stuck in degraded state forever 8561 - @ejweber @chriscchien @roger-ryao
• Security issues in longhorn 1.6.2 version images 9132 - @c3y1huang
• [BACKPORT][v1.6.3][BUG] [Backupstore] Need to close the reader after downloading files for the Azure backup store driver. 9283 - @yangchiu @mantissahz
• [BACKPORT][v1.6.3][BUG] LH fails silently when node has attached volumes 9211 - @yangchiu @ejweber
• [BACKPORT][v1.6.3][BUG] Fix longhorn-manager TestCleanupRedundantInstanceManagers 8670 - @derekbit @roger-ryao
• [BACKPORT][v1.6.3][BUG] Longhorn thinks node is unschedulable 9052 - @c3y1huang @roger-ryao
• [BACKPORT][v1.6.3][BUG] Volume stuck in degraded 9295 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.6.3][BUG] test case test_system_backup_and_restore_volume_with_backingimage failed on sle-micro ARM64 9227 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.6.3][BUG] System Backup Fails and DR Volume Enters Attach-Detach Loop When Volume Backup Policy is Set to Always 9339 - @c3y1huang @roger-ryao
• [BACKPORT][v1.6.3][BUG] v1 volume replica rebuld fail after upgade from v1.7.0 to v1.7.1-rc1 9336 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.6.3][BUG] instance-manager is stuck at starting state 8678 - @derekbit
• [BACKPORT][v1.6.3][BUG] instance-manager pod for v2 volume is killed due to a failed liveness probe. 8808 - @derekbit @chriscchien
• [BACKPORT][v1.6.3][BUG] Pod auto-deletion may cause thousands of logs 9020 - @ejweber @roger-ryao
• [BACKPORT][v1.6.3][BUG] Share manager controller reconciles tens of thousands of times 9088 - @ejweber @roger-ryao
• [BACKPORT][v1.6.3][BUG] Scale replica snapsots warning 8851 - @ejweber
• [BACKPORT][v1.6.3][BUG] Longhorn can no longer create XFS volumes smaller than 300 MiB 8560 - @ejweber @chriscchien
• [BACKPORT][v1.6.3][BUG]filesystem trim RecurringJob times out (volumes where files are frequently created and deleted) 9048 - @c3y1huang @chriscchien
• [BACKPORT][v1.6.3][BUG] Can not revert V2 volume snapshot after upgrade from v1.6.2 to v1.7.0-dev 9066 - @chriscchien @DamiaSan
• [BACKPORT][v1.6.3][BUG] Rebuilding Replica fails on larger volumes 8949 -
• [BACKPORT][v1.6.3][BUG] Orphan longhorn-engine-manager and longhorn-replica-manager services 8858 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.6.3][BUG] When revision counter is disabled, the engine might choose a replica with a smaller head size to be the source of truth for auto-salvage 8661 - @PhanLe1010
• [BACKPORT][v1.6.3][BUG] toomanysnapshots UI message displays incorrect snapshot count 8700 - @ejweber
• [BACKPORT][v1.6.3][BUG] Uninstallation will fail if invalid backuptarget is set. 8793 - @mantissahz @chriscchien

Misc

• [BACKPORT][v1.6.3][TASK] Update the best practice page to mention these broken kernels 8882 - @PhanLe1010

Contributors

• @ChanYiLin
• @DamiaSan
• @PhanLe1010
• @a110605
• @c3y1huang
• @chriscchien
• @derekbit
• @ejweber
• @innobead
• @mantissahz
• @roger-ryao
• @tserong
• @yangchiu

Longhorn v1.7.1

Longhorn v1.7.1 Release Notes

Longhorn 1.7.1 introduces several improvements and bug fixes that are intended to improve system quality, resilience, and stability.

The Longhorn team appreciates your contributions and expects to receive feedback regarding this release.

Note

For more information about release-related terminology, see Releases.

Installation

Important

Ensure that your cluster is running Kubernetes v1.21 or later before installing Longhorn v1.7.1.

You can install Longhorn using a variety of tools, including Rancher, Kubectl, and Helm. For more information about installation methods and requirements, see Quick Installation in the Longhorn documentation.

Upgrade

Important

Ensure that your cluster is running Kubernetes v1.21 or later before upgrading from Longhorn v1.6.x or v1.7.x (< v1.7.0) to v1.7.1.

Longhorn only allows upgrades from supported versions. For more information about upgrade paths and procedures, see Upgrade in the Longhorn documentation.

Deprecation & Incompatibilities

The functionality of the environment check script overlaps with that of the Longhorn CLI, which is available starting with v1.7.0. Because of this, the script is deprecated in v1.7.0 and is scheduled for removal in v1.8.0.

For information about important changes, including feature incompatibility, deprecation, and removal, see Important Notes in the Longhorn documentation.

Post-Release Known Issues

For information about issues identified after this release, see Release-Known-Issues.

Resolved Issues

Improvement

• [BACKPORT][v1.7.1][IMPROVEMENT] Longhorn CLI should install cryptsetup 9316 - @mantissahz @roger-ryao
• [BACKPORT][v1.7.1][IMPROVEMENT] Resilience handling for the last replica timeout 9275 - @ejweber @chriscchien
• [BACKPORT][v1.7.1][IMPROVEMENT] Check kernel module dm_crypt on host machines 9310 - @mantissahz

Bug

• [BACKPORT][v1.7.1][BUG] Fix security issues in v1.7.1 RC images 9363 - @c3y1huang
• [BACKPORT][v1.7.1][BUG] should set backing image minNumberOfCopies to 1 when upgrading Longhorn 9353 - @ChanYiLin @chriscchien
• [BACKPORT][v1.7.1][BUG] System Backup Fails and DR Volume Enters Attach-Detach Loop When Volume Backup Policy is Set to Always 9333 - @c3y1huang @roger-ryao
• [BACKPORT][v1.7.1][BUG] [Backupstore] Need to close the reader after downloading files for the Azure backup store driver. 9282 - @yangchiu @mantissahz
• [BACKPORT][v1.7.1][BUG] Unable to access azurite backup store by DNS hostname 9341 - @mantissahz
• [BACKPORT][v1.7.1][BUG] v1 volume replica rebuild fail after upgrade from v1.7.0 to v1.7.1-rc1 9332 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.7.1][BUG] v2-data-engine setting validator doesn't take disabled nodes into account when checking hugepages 9320 - @tserong @roger-ryao
• [BACKPORT][v1.7.1][BUG] Some volumes stuck in "Attaching" state after upgrade to 1.7.0 9270 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.7.1][BUG] error logs appeared in uninstallation job 9304 - @ChanYiLin @chriscchien
• [BACKPORT][v1.7.1][BUG] Incorrect NFS endpoint after enable/disable storage network for RWX volume 9273 - @Vicente-Cheng @roger-ryao
• [BACKPORT][v1.7.1][BUG] Volume stuck in degraded 9285 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.7.1][BUG] LH fails silently when node has attached volumes 9210 - @ejweber @chriscchien

Contributors

• @ChanYiLin
• @PhanLe1010
• @Vicente-Cheng
• @c3y1huang
• @chriscchien
• @ejweber
• @innobead
• @mantissahz
• @roger-ryao
• @tserong
• @yangchiu
• @jhkrug
• @jillian-maroket
• @rebeccazzzz
• @forbesguthrie
• @asettle

Longhorn v1.7.1-rc3

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Improvement

• [BACKPORT][v1.7.1][IMPROVEMENT] Longhorn CLI should install cryptsetup 9316 - @mantissahz @roger-ryao
• [BACKPORT][v1.7.1][IMPROVEMENT] Resilience handling for the last replica timeout 9275 - @ejweber @chriscchien
• [BACKPORT][v1.7.1][IMPROVEMENT] Check kernel module dm_crypt on host machines 9310 - @mantissahz

Bug

• [BACKPORT][v1.7.1][BUG] Fix security issues in v1.7.1 RC images 9363 - @c3y1huang
• [BACKPORT][v1.7.1][BUG] should set backing image minNumberOfCopies to 1 when upgrading Longhorn 9353 - @ChanYiLin @chriscchien
• [BACKPORT][v1.7.1][BUG] System Backup Fails and DR Volume Enters Attach-Detach Loop When Volume Backup Policy is Set to Always 9333 - @c3y1huang @roger-ryao
• [BACKPORT][v1.7.1][BUG] [Backupstore] Need to close the reader after downloading files for the Azure backup store driver. 9282 - @yangchiu @mantissahz
• [BACKPORT][v1.7.1][BUG] Unable to access azurite backup store by DNS hostname 9341 - @mantissahz
• [BACKPORT][v1.7.1][BUG] v1 volume replica rebuld fail after upgade from v1.7.0 to v1.7.1-rc1 9332 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.7.1][BUG] v2-data-engine setting validator doesn't take disabled nodes into account when checking hugepages 9320 - @tserong @roger-ryao
• [BACKPORT][v1.7.1][BUG] Some volumes stuck in "Attaching" state after upgrade to 1.7.0 9270 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.7.1][BUG] error logs appeared in uninstallation job 9304 - @ChanYiLin @chriscchien
• [BACKPORT][v1.7.1][BUG] Incorrect NFS endpoint after enable/disable storage network for RWX volume 9273 - @Vicente-Cheng @roger-ryao
• [BACKPORT][v1.7.1][BUG] Volume stuck in degraded 9285 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.7.1][BUG] LH fails silently when node has attached volumes 9210 - @ejweber @chriscchien

Contributors

• @ChanYiLin
• @PhanLe1010
• @Vicente-Cheng
• @c3y1huang
• @chriscchien
• @ejweber
• @innobead
• @mantissahz
• @roger-ryao
• @tserong
• @yangchiu

Longhorn v1.7.1-rc2

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Improvement

• [BACKPORT][v1.7.1][IMPROVEMENT] Longhorn CLI should install cryptsetup 9316 - @mantissahz @roger-ryao
• [BACKPORT][v1.7.1][IMPROVEMENT] Resilience handling for the last replica timeout 9275 - @ejweber @chriscchien
• [BACKPORT][v1.7.1][IMPROVEMENT] Check kernel module dm_crypt on host machines 9310 - @mantissahz

Bug

• [BACKPORT][v1.7.1][BUG] [Backupstore] Need to close the reader after downloading files for the Azure backup store driver. 9282 - @yangchiu @mantissahz
• [BACKPORT][v1.7.1][BUG] Unable to access azurite backup store by DNS hostname 9341 - @mantissahz
• [BACKPORT][v1.7.1][BUG] System Backup Fails and DR Volume Enters Attach-Detach Loop When Volume Backup Policy is Set to Always 9333 - @c3y1huang @roger-ryao
• [BACKPORT][v1.7.1][BUG] v1 volume replica rebuld fail after upgade from v1.7.0 to v1.7.1-rc1 9332 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.7.1][BUG] v2-data-engine setting validator doesn't take disabled nodes into account when checking hugepages 9320 - @tserong @roger-ryao
• [BACKPORT][v1.7.1][BUG] Some volumes stuck in "Attaching" state after upgrade to 1.7.0 9270 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.7.1][BUG] error logs appeared in uninstallation job 9304 - @ChanYiLin @chriscchien
• [BACKPORT][v1.7.1][BUG] Incorrect NFS endpoint after enable/disable storage network for RWX volume 9273 - @Vicente-Cheng @roger-ryao
• [BACKPORT][v1.7.1][BUG] Volume stuck in degraded 9285 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.7.1][BUG] LH fails silently when node has attached volumes 9210 - @ejweber @chriscchien

Contributors

• @ChanYiLin
• @PhanLe1010
• @Vicente-Cheng
• @c3y1huang
• @chriscchien
• @ejweber
• @innobead
• @mantissahz
• @roger-ryao
• @tserong
• @yangchiu

Longhorn v1.7.1-rc1

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Improvement

• [BACKPORT][v1.7.1][IMPROVEMENT] Resilience handling for the last replica timeout 9275 - @ejweber @chriscchien
• [BACKPORT][v1.7.1][IMPROVEMENT] Longhorn CLI should install cryptsetup 9316 - @mantissahz @roger-ryao
• [BACKPORT][v1.7.1][IMPROVEMENT] Check kernel module dm_crypt on host machines 9310 - @mantissahz

Bug

• [BACKPORT][v1.7.1][BUG] v2-data-engine setting validator doesn't take disabled nodes into account when checking hugepages 9320 - @tserong @roger-ryao
• [BACKPORT][v1.7.1][BUG] Some volumes stuck in "Attaching" state after upgrade to 1.7.0 9270 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.7.1][BUG] error logs appeared in uninstallation job 9304 - @ChanYiLin @chriscchien
• [BACKPORT][v1.7.1][BUG] Incorrect NFS endpoint after enable/disable storage network for RWX volume 9273 - @Vicente-Cheng @roger-ryao
• [BACKPORT][v1.7.1][BUG] Volume stuck in degraded 9285 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.7.1][BUG] [Backupstore] Need to close the reader after downloading files for the Azure backup store driver. 9282 - @yangchiu @mantissahz
• [BACKPORT][v1.7.1][BUG] LH fails silently when node has attached volumes 9210 - @ejweber @chriscchien

Contributors

• @ChanYiLin
• @PhanLe1010
• @Vicente-Cheng
• @chriscchien
• @ejweber
• @innobead
• @mantissahz
• @roger-ryao
• @tserong
• @yangchiu

Longhorn v1.7.0

Longhorn v1.7.0 Release Notes

This latest version of Longhorn introduces several features, enhancements, and bug fixes that are intended to improve system quality and the overall user experience. Highlights include new V2 Data Engine features, platform-agnostic deployment, high availability, and improvements to data protection, stability, performance, and resilience.

The Longhorn team appreciates your contributions and anticipates receiving feedback regarding this release.

For more information about release-related terminology, see Releases.

Warning

The Longhorn team has identified a critical issue that affects volume attachment in Longhorn v1.7.0. We are currently working on releasing v1.7.1, which will be available soon. If your Longhorn cluster contains engine resources with names in the format <volume name>-e-<8-char random id> that are created before v1.5.2 and v1.4.4, please put v1.7.0 upgrade on hold until v1.7.1 is available.

You can use the following command to check if it is safe to upgrade your Longhorn cluster to v1.7.0:

[ $(kubectl -n longhorn-system get engines.longhorn.io -o name | grep -E '\-e\-[a-z0-9]{8}$' | wc -l) -gt 0 ] && echo "Please hold off on upgrading to v1.7.0 until v1.7.1 is available." || echo "Safe to upgrade to v1.7.0."

Deprecation & Incompatibilities

The functionality of the environment check script overlaps with that of the Longhorn CLI, which is available starting with v1.7.0. Because of this, the script is deprecated in v1.7.0 and is scheduled for removal in v1.8.0.

For information about important changes, including feature incompatibility, deprecation, and removal, see Important Notes in the Longhorn documentation.

Primary Highlights

New V2 Data Engine Features

Although the V2 Data Engine is still considered a preview feature in this release, the core functions have been significantly enhanced.

• Online replica rebuilding: Allows Longhorn to rebuild replicas while the volume is running without any IO interruptions.
• Filesystem trim: Reclaims unused space more effectively.
• Block-type disk support for SPDK AIO, NVMe and VirtIO Bdev drivers: Broadens the range of compatible hardware and improves performance in high-demand environments.
• V2 volume support for data plane live upgrade: Enables upgrading of the spdk_tgt process, which handles IO operations, without any downtime. Coverage of the live upgrade function will be extended to the control plane in a future release.

The Longhorn team will continue to develop features for the V1 Data Engine and to prepare the V2 Data Engine for use in all types of environments.

Platform-Agnostic Deployment

Longhorn is designed to seamlessly operate on general-purpose Linux distributions and certain container-optimized operating systems. v1.7.0 provides robust and efficient persistent storage solutions for Kubernetes clusters running on Container-Optimized OS (COS).

High Availability

v1.7.0 introduces features that enhance system resilience and address potential single points of failure.

• High availability for backing images: Mitigates risks associated with image failures.
• RWX volumes fast failover: Enables rapid detection of and response to Share Manager pod failures (independent of the timing and sequence of Kubernetes node failures).

Data Protection

Starting with v1.7.0, Longhorn supports periodic and on-demand full backups that help reduce the likelihood of backup corruption and enhance the overall reliability of the backup process.

Scheduling

The replica auto-balancing feature was enhanced to address disk space pressure from growing volumes. These enhancements reduce manual intervention by automatically rebalancing replicas under disk pressure and improve performance with faster replica rebuilding through local file copying.

Storage Network

The storage network now supports RWX volumes with network segregation, enabling dedicated traffic lanes for storage operations.

Longhorn CLI

The Longhorn CLI, which is the official Longhorn command line tool, is introduced in v1.7.0. This tool interacts with Longhorn by creating Kubernetes custom resources (CRs) and executing commands inside a dedicated pod for in-cluster and host operations. Usage scenarios include installation, operations such as exporting replicas, and troubleshooting.

Installation

Important

Ensure that your cluster is running Kubernetes v1.21 or later before installing Longhorn v1.7.0.

You can install Longhorn using a variety of tools, including Rancher, Kubectl, and Helm. For more information about installation methods and requirements, see Quick Installation in the Longhorn documentation.

Upgrade

Important

Ensure that your cluster is running Kubernetes v1.21 or later before upgrading from Longhorn v1.6.x to v1.7.0.

Longhorn only allows upgrades from supported versions. For more information about upgrade paths and procedures, see Upgrade in the Longhorn documentation.

Post-Release Known Issues

For information about issues identified after this release, see Release-Known-Issues.

Resolved Issues

Highlights

• [FEATURE] Longhorn block-type disks supports different SPDK disk bdev drivers 7672 - @derekbit @roger-ryao
• [FEATURE] Volume cloning enhancement 2907 - @yangchiu @PhanLe1010
• [FEATURE] Share manager HA - Experimental 6205 - @PhanLe1010 @james-munson @roger-ryao
• [FEATURE] v2 volume replica online rebuilding 7199 - @shuo-wu @chriscchien
• [FEATURE] v2 volume supports live upgrade for data plane 6001 - @derekbit @chriscchien
• [IMPROVEMENT] Use fsfreeze instead of sync before snapshot 2187 - @ejweber @chriscchien
• [FEATURE] Auto-balancing volumes between nodes & disks 4105 - @yangchiu @c3y1huang
• [FEATURE] HA backing image 2856 - @ChanYiLin @chriscchien
• [FEATURE] Create Longhorn CLI focusing on non CR resource operations 7927 - @c3y1huang @chriscchien
• [FEATURE] Support periodic or on-demand full backups to enhance backup reliability 7070 - @ChanYiLin @chriscchien
• [IMPROVEMENT] Improve rebuilding is canceled if it takes longer than 24 hours 2765 - @PhanLe1010 @chriscchien
• [UI][FEATURE] Support volume cloning using UI 8741 - @a110605 @chriscchien
• [UI][FEATURE] Add backing image encryption and clone support 8789 - @a110605 @mantissahz
• [FEATURE] Support volume encryption for (encrypted) backing image volumes 7051 - @ChanYiLin @roger-ryao
• [FEATURE] Container-Optimized OS support 6165 - @yangchiu @c3y1huang

Features

• [FEATURE] v2 volume supports filesystem trim 7534 - @derekbit @chriscchien
• [FEATURE] Add nodeSelector as a parameter to VolumeSnapshotClass for backing images 6526 - @ChanYiLin @chriscchien
• [FEATURE] Add additional monitoring settings to ServiceMonitor resource. 8142 - @yangchiu @ejweber @kejrak
• [FEATURE] Support storage network for RWX volumes 8184 - @c3y1huang @chriscchien
• [FEATURE] Add BackupBackingImage UI 7541 - @a110605 @mantissahz
• [FEATURE][UI] Add parameters support to the Backup and RecurringJob 8291 - @a110605 @chriscchien
• [UI][FEATURE] Add Fields to Backing Image Creation and Update 8485 - @a110605 @chriscchien
• [FEATURE] Change parent of a logical volume for volume rebuild 5866 - @DamiaSan

Improvements

• [IMPROVEMENT] Restore Latest Backup should be applied with BackingImage name value 7560 - @a110605 @roger-ryao
• [IMPROVEMENT] Remove Offline Rebuilding from UI on master and v1.7.0 9090 - @a110605 @derekbit @chriscchien
• [IMPROVEMENT] Add a label and selector for webhooks to longhorn-manager pod manifest. [8803](https:/...

Longhorn v1.7.0-rc4

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

• [FEATURE] Longhorn block-type disks supports different SPDK disk bdev drivers 7672 - @derekbit @roger-ryao
• [FEATURE] Volume cloning enhancement 2907 - @yangchiu @PhanLe1010
• [FEATURE] Share manager HA - Experimental 6205 - @PhanLe1010 @james-munson @roger-ryao
• [FEATURE] v2 volume replica online rebuilding 7199 - @shuo-wu @chriscchien
• [FEATURE] v2 volume supports live upgrade for data plane 6001 - @derekbit @chriscchien
• [IMPROVEMENT] Use fsfreeze instead of sync before snapshot 2187 - @ejweber @chriscchien
• [FEATURE] Auto-balancing volumes between nodes & disks 4105 - @yangchiu @c3y1huang
• [FEATURE] HA backing image 2856 - @ChanYiLin @chriscchien
• [FEATURE] Create Longhorn CLI focusing on non CR resource operations 7927 - @c3y1huang @chriscchien
• [FEATURE] Support periodic or on-demand full backups to enhance backup reliability 7070 - @ChanYiLin @chriscchien
• [IMPROVEMENT] Improve rebuilding is canceled if it takes longer than 24 hours 2765 - @PhanLe1010 @chriscchien
• [UI][FEATURE] Support volume cloning using UI 8741 - @a110605 @chriscchien
• [UI][FEATURE] Add backing image encryption and clone support 8789 - @a110605 @mantissahz
• [FEATURE] Support volume encryption for (encrypted) backing image volumes 7051 - @ChanYiLin @roger-ryao
• [FEATURE] Container-Optimized OS support 6165 - @yangchiu @c3y1huang

Feature

• [FEATURE] v2 volume supports filesystem trim 7534 - @derekbit @chriscchien
• [FEATURE] Add nodeSelector as a parameter to VolumeSnapshotClass for backing images 6526 - @ChanYiLin @chriscchien
• [FEATURE] Add additional monitoring settings to ServiceMonitor resource. 8142 - @yangchiu @ejweber @kejrak
• [FEATURE] Support storage network for RWX volumes 8184 - @c3y1huang @chriscchien
• [FEATURE] Add BackupBackingImage UI 7541 - @a110605 @mantissahz
• [FEATURE][UI] Add parameters support to the Backup and RecurringJob 8291 - @a110605 @chriscchien
• [UI][FEATURE] Add Fields to Backing Image Creation and Update 8485 - @a110605 @chriscchien
• [FEATURE] Change parent of a logical volume for volume rebuild 5866 - @DamiaSan

Improvement

• [IMPROVEMENT] Restore Latest Backup should be applied with BackingImage name value 7560 - @a110605 @roger-ryao
• [IMPROVEMENT] Remove Offline Rebuilding from UI on master and v1.7.0 9090 - @a110605 @derekbit @chriscchien
• [IMPROVEMENT] Add a label and selector for webhooks to longhorn-manager pod manifest. 8803 - @james-munson @roger-ryao
• [IMPROVEMENT] Fix outdated environment check in longhorn manager 7571 - @mantissahz @roger-ryao
• [IMPROVEMENT] v2 volume snapshot supports UserCreated flag 7578 - @DamiaSan @roger-ryao
• [IMPROVEMENT] Undocumented traffic on port 8503 9076 - @derekbit
• [IMPROVEMENT] Cannot expand a volume created by Longhorn UI 6446 - @mantissahz @chriscchien
• [IMPROVEMENT] Minor log improvement 6396 - @Vicente-Cheng
• [IMPROVEMENT] Refactor the usage of timer reset in SPDK JSON-RPC read method 7297 - @shuo-wu
• [IMPROVEMENT] Update sizes in Engine and Volume resources less frequently 8076 - @ejweber @chriscchien
• [IMPROVEMENT] Docker build of instance-manager should error out if the execution of SPDK pkgdep.sh fails 8063 - @derekbit @roger-ryao
• [IMPROVEMENT] Enable spdk_tgt debug log for helping debug 7939 - @derekbit @chriscchien
• [BUG] Do not terminate nfs-ganesha in share-manager pod after failing to access recovery backend 8345 - @derekbit @chriscchien
• [IMPROVEMENT] Fall back to a running instance-manager if a default is not available 8464 - @derekbit @chriscchien
• [IMPROVEMENT] Only sync log settings to running instance manager pod 8466 - @derekbit
• [IMPROVEMENT] Expose local logical volume and attach it to local node 8551 - @derekbit @chriscchien
• [IMPROVEMENT] Update nfs-ganesha to v5.9 9008 - @derekbit @chriscchien
• [IMPROVEMENT] Pre-pull images (share-manager image and instance-manager image) on each Longhorn node 8376 - @mantissahz @chriscchien
• [IMPROVEMENT] Add websocket for backup backing image 8849 - @ChanYiLin @roger-ryao
• [IMPROVEMENT] Allow users to request backup volume update 7982 - @mantissahz @chriscchien
• [IMPROVEMENT] Problems mounting XFS volume clones / restored snapshots 8796 - @PhanLe1010 @chriscchien
• [IMPROVEMENT] Do not start any instance-manager pods for the v2 data engine on a node if one is already running 8456 - @derekbit @roger-ryao
• [IMPROVEMENT] Auto cleanup related Snapshot when removing Backup 8365 - @FrankYang0529 @roger-ryao
• [IMPROVEMENT] Improve environment_check script for NFS protocol bug and the host system self diagnosis 7931 - @james-munson @roger-ryao
• [UI][IMPROVEMENT] Tweak some minor UI issues 8646 - @a110605 @roger-ryao
• [IMPROVEMENT] environment_check.sh should check for the iscsi_tcp kernel module 8697 - @tserong @roger-ryao
• [IMPROVEMENT] Improve and simplify chart values.yaml 5089 - @yangchiu @ChanYiLin
• [IMPROVEMENT] The client-go rest client rate limit inside the csi sidecar component might be too small (csi-provisioner, csi-attacjer. csi-snappshotter, csi-attacher) 8699 - @PhanLe1010 @chriscchien
• [IMPROVEMENT] Always update the built-in installed system packages when building component images 8721 - @c3y1huang @roger-ryao
• [UI][IMPROVEMENT] Improve the UX of updating danger zone settings 8071 - @a110605 @roger-ryao
• [IMPROVEMENT] Prevent unnecessary updates of instanceManager status 8420 - @derekbit @chriscchien
• [IMPROVEMENT] Longhor Manager Flood with "Failed to get engine proxy of ... cannot get client for engine" Message 8266 - @derekbit @chriscchien
• [IMPROVEMENT] Add setting value limits in SettingDefinition 7441 - @ChanYiLin @roger-ryao
• [IMPROVEMENT] Investigate performance bottleneck in v1 data path 8436 - @PhanLe1010 @roger-ryao
• [IMPROVEMENT] Disable revision counter by default 8563 - @PhanLe1010 @chriscchien
• [IMPROVEMENT] BackingImage UI improvement 7293 - @a110605 @roger-ryao
• [IMPROVEMENT] Avoid misleading log messages in longhorn manager while syncing danger zone settings 7797 - @mantissahz @chriscchien
• [IMPROVEMENT] Add setting to configure support bundle timeout for node bundle collection 8623 - @c3y1huang @roger-ryao
• [IMPROVEMENT] RWX volume scheduling 7872 - @derekbit @chriscchien
• [IMPROVEMENT] Expose virtual size of qcow2 backing images 7923 - @tserong @roger-ryao
• [IMPROVEMENT] Saving Settings page changes 7497 - @a110605 @roger-ryao
• [IMPROVEMENT] UI does not find credentials for Backblaze B2 s3-compatible ...

Longhorn v1.7.0-rc3

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

• [FEATURE] Share manager HA - Experimental 6205 - @PhanLe1010 @james-munson @roger-ryao
• [FEATURE] v2 volume replica online rebuilding 7199 - @shuo-wu @chriscchien
• [FEATURE] Volume cloning enhancement 2907 - @yangchiu @PhanLe1010
• [IMPROVEMENT] Use fsfreeze instead of sync before snapshot 2187 - @ejweber @chriscchien
• [FEATURE] Auto-balancing volumes between nodes & disks 4105 - @yangchiu @c3y1huang
• [FEATURE] HA backing image 2856 - @ChanYiLin @chriscchien
• [FEATURE] Create Longhorn CLI focusing on non CR resource operations 7927 - @c3y1huang @chriscchien
• [FEATURE] Support periodic or on-demand full backups to enhance backup reliability 7070 - @ChanYiLin @chriscchien
• [IMPROVEMENT] Improve rebuilding is canceled if it takes longer than 24 hours 2765 - @PhanLe1010 @chriscchien
• [FEATURE] Longhorn block-type disks supports different SPDK disk bdev drivers 7672 - @derekbit @roger-ryao
• [FEATURE] v2 volume supports live upgrade for data plane 6001 - @derekbit @chriscchien
• [UI][FEATURE] Support volume cloning using UI 8741 - @a110605 @chriscchien
• [UI][FEATURE] Add backing image encryption and clone support 8789 - @a110605 @mantissahz
• [FEATURE] Support volume encryption for (encrypted) backing image volumes 7051 - @ChanYiLin @roger-ryao
• [FEATURE] Container-Optimized OS support 6165 - @yangchiu @c3y1huang

Feature

• [FEATURE] v2 volume supports filesystem trim 7534 - @derekbit @chriscchien
• [FEATURE] Add nodeSelector as a parameter to VolumeSnapshotClass for backing images 6526 - @ChanYiLin @chriscchien
• [FEATURE] Add additional monitoring settings to ServiceMonitor resource. 8142 - @yangchiu @ejweber @kejrak
• [FEATURE] Support storage network for RWX volumes 8184 - @c3y1huang @chriscchien
• [FEATURE] Add BackupBackingImage UI 7541 - @a110605 @mantissahz
• [FEATURE][UI] Add parameters support to the Backup and RecurringJob 8291 - @a110605 @chriscchien
• [UI][FEATURE] Add Fields to Backing Image Creation and Update 8485 - @a110605 @chriscchien
• [FEATURE] Change parent of a logical volume for volume rebuild 5866 - @DamiaSan

Improvement

• [IMPROVEMENT] Remove Offline Rebuilding from UI on master and v1.7.0 9090 - @a110605 @derekbit @chriscchien
• [IMPROVEMENT] Add a label and selector for webhooks to longhorn-manager pod manifest. 8803 - @james-munson @roger-ryao
• [IMPROVEMENT] Fix outdated environment check in longhorn manager 7571 - @mantissahz @roger-ryao
• [IMPROVEMENT] v2 volume snapshot supports UserCreated flag 7578 - @DamiaSan @roger-ryao
• [IMPROVEMENT] Undocumented traffic on port 8503 9076 - @derekbit
• [IMPROVEMENT] Cannot expand a volume created by Longhorn UI 6446 - @mantissahz @chriscchien
• [IMPROVEMENT] Minor log improvement 6396 - @Vicente-Cheng
• [IMPROVEMENT] Refactor the usage of timer reset in SPDK JSON-RPC read method 7297 - @shuo-wu
• [IMPROVEMENT] Update sizes in Engine and Volume resources less frequently 8076 - @ejweber @chriscchien
• [IMPROVEMENT] Docker build of instance-manager should error out if the execution of SPDK pkgdep.sh fails 8063 - @derekbit @roger-ryao
• [IMPROVEMENT] Enable spdk_tgt debug log for helping debug 7939 - @derekbit @chriscchien
• [BUG] Do not terminate nfs-ganesha in share-manager pod after failing to access recovery backend 8345 - @derekbit @chriscchien
• [IMPROVEMENT] Fall back to a running instance-manager if a default is not available 8464 - @derekbit @chriscchien
• [IMPROVEMENT] Only sync log settings to running instance manager pod 8466 - @derekbit
• [IMPROVEMENT] Expose local logical volume and attach it to local node 8551 - @derekbit @chriscchien
• [IMPROVEMENT] Update nfs-ganesha to v5.9 9008 - @derekbit @chriscchien
• [IMPROVEMENT] Pre-pull images (share-manager image and instance-manager image) on each Longhorn node 8376 - @mantissahz @chriscchien
• [IMPROVEMENT] Add websocket for backup backing image 8849 - @ChanYiLin @roger-ryao
• [IMPROVEMENT] Restore Latest Backup should be applied with BackingImage name value 7560 - @a110605 @roger-ryao
• [IMPROVEMENT] Allow users to request backup volume update 7982 - @mantissahz @chriscchien
• [IMPROVEMENT] Problems mounting XFS volume clones / restored snapshots 8796 - @PhanLe1010 @chriscchien
• [IMPROVEMENT] Do not start any instance-manager pods for the v2 data engine on a node if one is already running 8456 - @derekbit @roger-ryao
• [IMPROVEMENT] Auto cleanup related Snapshot when removing Backup 8365 - @FrankYang0529 @roger-ryao
• [IMPROVEMENT] Improve environment_check script for NFS protocol bug and the host system self diagnosis 7931 - @james-munson @roger-ryao
• [UI][IMPROVEMENT] Tweak some minor UI issues 8646 - @a110605 @roger-ryao
• [IMPROVEMENT] environment_check.sh should check for the iscsi_tcp kernel module 8697 - @tserong @roger-ryao
• [IMPROVEMENT] Improve and simplify chart values.yaml 5089 - @yangchiu @ChanYiLin
• [IMPROVEMENT] The client-go rest client rate limit inside the csi sidecar component might be too small (csi-provisioner, csi-attacjer. csi-snappshotter, csi-attacher) 8699 - @PhanLe1010 @chriscchien
• [IMPROVEMENT] Always update the built-in installed system packages when building component images 8721 - @c3y1huang @roger-ryao
• [UI][IMPROVEMENT] Improve the UX of updating danger zone settings 8071 - @a110605 @roger-ryao
• [IMPROVEMENT] Prevent unnecessary updates of instanceManager status 8420 - @derekbit @chriscchien
• [IMPROVEMENT] Longhor Manager Flood with "Failed to get engine proxy of ... cannot get client for engine" Message 8266 - @derekbit @chriscchien
• [IMPROVEMENT] Add setting value limits in SettingDefinition 7441 - @ChanYiLin @roger-ryao
• [IMPROVEMENT] Investigate performance bottleneck in v1 data path 8436 - @PhanLe1010 @roger-ryao
• [IMPROVEMENT] Disable revision counter by default 8563 - @PhanLe1010 @chriscchien
• [IMPROVEMENT] BackingImage UI improvement 7293 - @a110605 @roger-ryao
• [IMPROVEMENT] Avoid misleading log messages in longhorn manager while syncing danger zone settings 7797 - @mantissahz @chriscchien
• [IMPROVEMENT] Add setting to configure support bundle timeout for node bundle collection 8623 - @c3y1huang @roger-ryao
• [IMPROVEMENT] RWX volume scheduling 7872 - @derekbit @chriscchien
• [IMPROVEMENT] Expose virtual size of qcow2 backing images 7923 - @tserong @roger-ryao
• [IMPROVEMENT] Saving Settings page changes 7497 - @a110605 @roger-ryao
• [IMPROVEMENT] UI does not find credentials for Backblaze B2 s3-compatible ...

Longhorn v1.7.0-rc2

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

• [FEATURE] Create Longhorn CLI focusing on non CR resource operations 7927 - @c3y1huang @chriscchien
• [FEATURE] Support periodic or on-demand full backups to enhance backup reliability 7070 - @ChanYiLin @chriscchien
• [FEATURE] HA backing image 2856 - @ChanYiLin @chriscchien
• [FEATURE] Share manager HA - Experimental 6205 - @PhanLe1010 @james-munson @roger-ryao
• [IMPROVEMENT] Improve rebuilding is canceled if it takes longer than 24 hours 2765 - @PhanLe1010 @chriscchien
• [FEATURE] v2 volume replica online rebuilding 7199 - @shuo-wu @chriscchien
• [FEATURE] Longhorn block-type disks supports different SPDK disk bdev drivers 7672 - @derekbit @roger-ryao
• [FEATURE] v2 volume supports live upgrade for data plane 6001 - @derekbit @chriscchien
• [UI][FEATURE] Support volume cloning using UI 8741 - @a110605 @chriscchien
• [UI][FEATURE] Add backing image encryption and clone support 8789 - @a110605 @mantissahz
• [FEATURE] Auto-balancing volumes between nodes & disks 4105 - @yangchiu @c3y1huang
• [FEATURE] Support volume encryption for (encrypted) backing image volumes 7051 - @ChanYiLin @roger-ryao
• [IMPROVEMENT] Use fsfreeze instead of sync before snapshot 2187 - @ejweber @chriscchien
• [FEATURE] Volume cloning enhancement 2907 - @yangchiu @PhanLe1010
• [FEATURE] Container-Optimized OS support 6165 - @yangchiu @c3y1huang

Feature

• [FEATURE] v2 volume supports filesystem trim 7534 - @derekbit @chriscchien
• [FEATURE] Add nodeSelector as a parameter to VolumeSnapshotClass for backing images 6526 - @ChanYiLin @chriscchien
• [FEATURE] Add additional monitoring settings to ServiceMonitor resource. 8142 - @yangchiu @ejweber @kejrak
• [FEATURE] Support storage network for RWX volumes 8184 - @c3y1huang @chriscchien
• [FEATURE] Add BackupBackingImage UI 7541 - @a110605 @mantissahz
• [FEATURE][UI] Add parameters support to the Backup and RecurringJob 8291 - @a110605 @chriscchien
• [UI][FEATURE] Add Fields to Backing Image Creation and Update 8485 - @a110605 @chriscchien
• [FEATURE] Change parent of a logical volume for volume rebuild 5866 - @DamiaSan

Improvement

• [IMPROVEMENT] Undocumented traffic on port 8503 9076 - @derekbit
• [IMPROVEMENT] Cannot expand a volume created by Longhorn UI 6446 - @mantissahz @chriscchien
• [IMPROVEMENT] Minor log improvement 6396 - @Vicente-Cheng
• [IMPROVEMENT] Refactor the usage of timer reset in SPDK JSON-RPC read method 7297 - @shuo-wu
• [IMPROVEMENT] Update sizes in Engine and Volume resources less frequently 8076 - @ejweber @chriscchien
• [IMPROVEMENT] Docker build of instance-manager should error out if the execution of SPDK pkgdep.sh fails 8063 - @derekbit @roger-ryao
• [IMPROVEMENT] Enable spdk_tgt debug log for helping debug 7939 - @derekbit @chriscchien
• [BUG] Do not terminate nfs-ganesha in share-manager pod after failing to access recovery backend 8345 - @derekbit @chriscchien
• [IMPROVEMENT] Fall back to a running instance-manager if a default is not available 8464 - @derekbit @chriscchien
• [IMPROVEMENT] Only sync log settings to running instance manager pod 8466 - @derekbit
• [IMPROVEMENT] Expose local logical volume and attach it to local node 8551 - @derekbit @chriscchien
• [IMPROVEMENT] Update nfs-ganesha to v5.9 9008 - @derekbit @chriscchien
• [IMPROVEMENT] v2 volume snapshot supports UserCreated flag 7578 - @DamiaSan @roger-ryao
• [IMPROVEMENT] Add a label and selector for webhooks to longhorn-manager pod manifest. 8803 - @james-munson @roger-ryao
• [IMPROVEMENT] Fix outdated environment check in longhorn manager 7571 - @mantissahz @roger-ryao
• [IMPROVEMENT] Pre-pull images (share-manager image and instance-manager image) on each Longhorn node 8376 - @mantissahz @chriscchien
• [IMPROVEMENT] Add websocket for backup backing image 8849 - @ChanYiLin @roger-ryao
• [IMPROVEMENT] Restore Latest Backup should be applied with BackingImage name value 7560 - @a110605 @roger-ryao
• [IMPROVEMENT] Allow users to request backup volume update 7982 - @mantissahz @chriscchien
• [IMPROVEMENT] Problems mounting XFS volume clones / restored snapshots 8796 - @PhanLe1010 @chriscchien
• [IMPROVEMENT] Do not start any instance-manager pods for the v2 data engine on a node if one is already running 8456 - @derekbit @roger-ryao
• [IMPROVEMENT] Auto cleanup related Snapshot when removing Backup 8365 - @FrankYang0529 @roger-ryao
• [IMPROVEMENT] Improve environment_check script for NFS protocol bug and the host system self diagnosis 7931 - @james-munson @roger-ryao
• [UI][IMPROVEMENT] Tweak some minor UI issues 8646 - @a110605 @roger-ryao
• [IMPROVEMENT] environment_check.sh should check for the iscsi_tcp kernel module 8697 - @tserong @roger-ryao
• [IMPROVEMENT] Improve and simplify chart values.yaml 5089 - @yangchiu @ChanYiLin
• [IMPROVEMENT] The client-go rest client rate limit inside the csi sidecar component might be too small (csi-provisioner, csi-attacjer. csi-snappshotter, csi-attacher) 8699 - @PhanLe1010 @chriscchien
• [IMPROVEMENT] Always update the built-in installed system packages when building component images 8721 - @c3y1huang @roger-ryao
• [UI][IMPROVEMENT] Improve the UX of updating danger zone settings 8071 - @a110605 @roger-ryao
• [IMPROVEMENT] Prevent unnecessary updates of instanceManager status 8420 - @derekbit @chriscchien
• [IMPROVEMENT] Longhor Manager Flood with "Failed to get engine proxy of ... cannot get client for engine" Message 8266 - @derekbit @chriscchien
• [IMPROVEMENT] Add setting value limits in SettingDefinition 7441 - @ChanYiLin @roger-ryao
• [IMPROVEMENT] Investigate performance bottleneck in v1 data path 8436 - @PhanLe1010 @roger-ryao
• [IMPROVEMENT] Disable revision counter by default 8563 - @PhanLe1010 @chriscchien
• [IMPROVEMENT] BackingImage UI improvement 7293 - @a110605 @roger-ryao
• [IMPROVEMENT] Avoid misleading log messages in longhorn manager while syncing danger zone settings 7797 - @mantissahz @chriscchien
• [IMPROVEMENT] Add setting to configure support bundle timeout for node bundle collection 8623 - @c3y1huang @roger-ryao
• [IMPROVEMENT] RWX volume scheduling 7872 - @derekbit @chriscchien
• [IMPROVEMENT] Expose virtual size of qcow2 backing images 7923 - @tserong @roger-ryao
• [IMPROVEMENT] Saving Settings page changes 7497 - @a110605 @roger-ryao
• [IMPROVEMENT] UI does not find credentials for Backblaze B2 s3-compatible storage for backup 8462 - @mantissahz
• [IMPROVEMENT] Add documents for Longhorn deployed with GitOps [6948](h...