Skip to main content

OSISM 7

Instructions for the upgrade can be found in the Upgrade Guide.

ReleaseRelease Date
7.0.47. May 2024
7.0.33. May 2024
7.0.217. April 2024
7.0.127. March 2024
7.0.020. March 2024
warning

7.0.0a, 7.0.0b, 7.0.0c, 7.0.0d are pre-releases. Do not use these releases.

7.0.4 (20240507)​

Release date: 7. May 2024

Due to an error in osism/sbom repository, the Nova and Octavia images were not updated as originally stated in the release notes for OSISM 7.0.3. Therefore it was unfortunately necessary to release a 7.0.4 which contains the correct SBOM. In 7.0.4 nothing has changed except the SBOM. The correct images for Nova and Octavia are now included in 7.0.4 and the bugs listed below have been fixed. An update of the Manager Service must be done and then, if required, an update of Nova and Octavia can be updated as originally planned for OSISM 7.0.3. We apologise for the mistake and the resulting effort.

  • The OpenStack service images for Octavia and Nova have been rebuilt. Upgrades of the Octavia and Nova services are recommended. No upgrades of other OpenStack and associated infrastructure services such as MariaDB or RabbitMQ are required.

    • The Nova images have been rebuilt to add packages to be able to use vTMP (osism/issues#1008) If this is not relevant, the upgrade can be skipped.

    • The octavia images have been rebuilt to resolve an issue when creating a LB + a listener with an allowed_cidr with the fully-populated API (osism/issues#980)

    • When upgrading the Nova and Octavia API services, there is a short downtime of the APIs. This downtime is usually less than 1 minute.

7.0.3 (20240503)​

Release date: 3. May 2024

  • The Ceph service images have not been rebuilt. No upgrade of Ceph is required.

  • The OpenStack service images have not been rebuilt. No upgrade of OpenStack is required.

  • During the preparation of the upgrades of the regions of the PCO a bug (osism/issues#973) has been noticed which leads to a delay of up to 2 minutes between the necessary container stops and starts. This is due to a bug in the service units of all Kolla services. The bug is fixed in the current release. To avoid the delay during an upgrade, a fix must be applied in advance for all service units from Kolla.

    osism apply fix-gh973

  • The Kubernetes Cluster API image for the Kubernetes (K8s) 1.30 series is available. The images are now provided directly with osism manage image clusterapi. This means that K8s Cluster API images are now available for K8s series 1.27, 1.28, 1.29 and 1.30.

  • All Ansible collections have been prepared for use with Ubuntu 24.04. It is currently not recommended to upgrade existing clusters to Ubuntu 24.04 or to install new clusters with Ubuntu 24.04. There will be a note in future release notes that announces the full support of Ubuntu 24.04. At the moment everything related to Ubuntu 24.04 is a technical preview.

  • A new role tempest has been added to the Ansible collection osism.validations. This makes it possible to perform significantly more tests than with the previously used osism.validations.refstack role. The new role will be used in the testbed in future to significantly increase the number of tests performed in the CI.

  • New documentation for the project manager and the simple stress.

  • When using the reboot play, it is now possible to wait for the reboot to be completed (osism/issues#758).

    osism apply reboot \
      -e reboot_wait=yes \
      -e ireallymeanit=yes

  • Monitoring services are now activated by default for the internal Kubernetes cluster.

  • The openstack_cacert parameter used by Kolla is now set to "/etc/ssl/certs/ca-certificates.crt" by default. The previous default was "". If this is not wanted, the parameter must be overwritten in environments/kolla/configuration.yml.

  • The ironic_agent_files_directory parameter used by Kolla is now set to /share/ironic by default. The previous default was "{{ node_custom_config }}". If this is not wanted, the parameter must be overwritten in environments/kolla/configuration.yml. This is in preparation for the Ironic IPA images no longer being stored in the configuration repository but within the manager service at runtime.

  • The Ironic IPA images are now downloaded from ironic play by default. If this is not wanted, enable_ironic_agent_download_images: false must be set in environments/kolla/configuration.yml. The Ironic IPA images can now also be downloaded independently of the ironic play with osism apply ironic-download-ipa-images.

  • The ceph_cluster_fsid parameter is now generated automatically. It can be removed from environments/configuration.yml. The automatically generated ceph_clusterfs_fsid parameter is set to the value of the fsid parameter from environments/ceph/configuration.yml.

  • You can now use your own hook scripts in osism/cfg-cookiecutter. These are placed in the directory {{cookiecutter.project_name}}/scripts.d/ directory. The scripts are executed in alphabetical order. The scripts must be executable.

  • Versions not yet pinned in the manager environment of the configuration repository (Ansible collections, osism/cfg-generics, ..) are now automatically pinned during synchronisation with gilt overlay. This also applies to the osism update manager script.

  • The Docker version and the Docker CLI version can now also be managed via osism/cfg-generics. It is recommended to pin the Docker version in environments/configuration.yml.

    docker_version: '5:24.0.9'

  • When using the single service plays for Ceph (ceph-osds, ceph-mons, ..), no service restarts are executed for other services. This can be adjusted with the ceph_handler_*_restart parameters. The default value is false.

    ceph_handler_crash_restart
    ceph_handler_mdss_restart
    ceph_handler_mgrs_restart
    ceph_handler_mons_restart
    ceph_handler_osds_restart
    ceph_handler_rbdmirrors_restart
    ceph_handler_rgws_restart

    The following example now makes it possible when adding new OSDs to really only start the OSDs that have been newly added and to avoid a restart of all OSDs in the cluster or on a node.

    osism apply ceph-osds -e ceph_handler_osds_restart=False

  • The restart of Ceph Services is now throttled. By default, only one Ceph service is restarted at a time. This can be adjusted with the ceph_handler_*_restart_throttle parameters. The default value is 1.

    ceph_handler_crash_restart_throttle
    ceph_handler_mdss_restart_throttle
    ceph_handler_mgrs_restart_throttle
    ceph_handler_mons_restart_throttle
    ceph_handler_osds_restart_throttle
    ceph_handler_rbdmirrors_restart_throttle
    ceph_handler_rgws_restart_throttle

  • OVN container images will be built without the -march=broadwell parameter in the future. The OVN images have not been updated with this release, this only serves as an announcement for the future. By removing the parameter, it is possible to use the OVN container images on older CPUs. It is planned to provide different variants of the OVN and OVS container images in the future to enable parameters for modern CPUs in order to improve performance in particular.

  • This has nothing to do with the release itself and is just a notice. The build of the OpenStack Zed images has been disabled.

7.0.2 (20240407)​

Release date: 17. April 2024

  • The Ceph & OpenStack service images have not been rebuilt. No upgrades of Ceph & OpenStack and associated infrastructure services such as MariaDB or RabbitMQ are required.

  • Properties for device type vrfs are now supported in the osism.commons.network role.

  • Debian support in osism.commons & osism.services Ansible collection.

    • The roles of the osism.commons collection are now usable with Debian. The roles have been tested with Debian Bookworm.
    • The roles of the osism.services collection are now usable with Debian. The roles have been tested with Debian Bookworm.

  • If the OSISM CLI is executed as root user there is now an informal warning message.

  • Use a single network for Ceph frontend & backend in the cookiecutter. More information in the Ceph network configuration reference.

  • When synchronising the configuration repository, it is now ensured that you are on the correct branch.

  • Senlin images available again as rolling tag for OpenStack 2023.2.

  • Zun images available again as rolling tag for OpenStack 2023.2 & 2024.1.

  • New FRR configuration template for loadbalancers with an external uplink in the osism.services.frr role.

  • scs:name-v1 and scs:name-v2 extra specs are now set via the OpenStack flavor manager.

  • Task ID is now displayed in the output of the OSISM CLI for prepared tasks.

  • New Makefile target in the OSISM Testbed to fetch the Wireguard configuration file: vpn-wireguard-config.

  • With the play noop it is now possible to run a play with noop. This is useful for testing purposes of the manager service.

  • The stable repository is now used as the default for the Netdata service in the osism.services.netdata role. This avoids package conflicts in future.

7.0.1 (20240327)​

Release date: 27. March 2024

  • It's now possible to also use custom plays in the kolla and in the ceph environments. As for the other environments, custom plays can now be stored there in a e.g. playbook-hello-world.yml file and run by using e.g. osism apply -e kolla hello-world.

  • The version of k3s has been updated to 1.29.2. If Kubernetes is used, upgrade with osism apply kubernetes.

  • There is a new parameter ceph_custom_keys in the copy-ceph-keys play. This makes it possible to copy the keys from custom Ceph pools.

    ceph_custom_keys:
      - src: ceph.client.manila1.keyring
        dest: "{{ configuration_directory }}/environments/kolla/files/overlays/manila/ceph.client.manila1.keyring"

  • There is a new parameter with_keycloak in the cookiecutter. This can now be used to select whether the keycloak integration should be prepared or not.

  • An error in the ceph-pools play has been fixed that prevented the keys from being created for the pools.

  • The ceph-iscsigws play has been removed. The iSCSI gateway is in maintenance as of November 2022. This means that it is no longer in active development and will not be updated to add new features.

  • With ceph_serial it is now possible to define how many hosts Ansible should manage at a single time in the Ceph plays.

    osism apply ceph-mgrs -e ceph_serial=1

  • With ANSIBLE_VERSION it's now possible to overwrite the use Ansible version when working with the run.sh script inside the manager environment.

  • The osism.commons.known_hosts role has been completely revised.

    • avoid duplicate entries in the destination file

    • avoid comments in the destination file

    • make use of static entries possible

      It's now possible to add a known_hosts parameter to the host_vars to set static known hosts entries for a specific host. When this parameter is set ssh-keygen will not be used to generate the known hosts entries on the fly.

      known_hosts:
        - ssh-rsa AAAAB3NzaC1y...
        - ecdsa-sha2-nistp256 AAAAE2VjZHN...
        - ssh-ed25519 AAAAC3NzaC1...

    • make use of extra entries possible

      It's now possible to add a known_hosts_extra parameter to the configuration repository to set extra known hosts entries.

      known_hosts_extra:
        - testbed-node-1.testbed.osism.xyz ssh-rsa AAAAB3Nza...
        - testbed-node-2.testbed.osism.xyz ssh-rsa AAAAB3Nza...

    • The Octavia images have been updated. If Octavia is used, an upgrade must be done with osism apply -a upgrade octavia. We addressed the following issues.

7.0.0 (20240320)​

Release date: 20. March 2024

  • Shortly before the release, gilt made a major release which led to breaking changes. It is therefore important for the moment to install python-gilt < 2 when synchronising the configuration repository against the generics. In the CI and within the container images, we currently use python-gilt == 1.2.3. This is also the version that's currently installed in the container images and that's set in the requirements.txt.

Removals & Deprecations​

  • The role osism.services.tang for deploying the Tang service is deprecated in preparation for removal as it is currently not in a usable state. An attempt will be made to make the role usable until the next release. The same applies to Clevis integration via the osism.commons.clevis role.

  • The role osism.services.openldap for deploying the OpenLDAP service has been removed.

  • The manager plays to control the Vault service (seal, unseal, ..) have been removed as these tasks will be realized directly via the OSISM CLI (osism set vault password, ..) in the future.

  • The role osism.services.bird for deploying the Bird service has been removed.

New features​

  • With the osism manage image octavia command it is possible to rotate the Octavia Amphora image, which is rebuilt daily. Older images are deactivated. The old images must be removed manually after rotating the amphorae instances.

  • With the osism manage image clusterapi command it is possible to import all currently stable Cluster API images (v1.27, v1.28, and v1.29). As soon as new minor or major versions are available, these are also imported. Old and no longer used versions must currently be removed manually.

  • The persistence feature in Octavia can enabled with the new enable_octavia_jobboard parameter. The jobboard in Octavia is an experimental feature. It is not recommended to use it in production.

    environments/kolla/configuration.yml
    enable_octavia_jobboard: "yes"

    This requires an additional database, which is only created when Octavia play is run in bootstrap mode first.

    osism apply -a bootstrap octavia

    The secret octavia_persistence_database_password must be added to environments/kolla/secrets.yml before.

    environments/kolla/secrets.yml
    octavia_persistence_database_password:  # generate with: pwgen 32 1

  • In preparation for the migration to Rook, the Rook operator is deployable on the internal Kubernetes cluster with osism apply rook. The Rook operator is not yet used for the Ceph deployment. For the deployment of Ceph we still use the ceph-ansible project. For the next release a tool called rookify is planned to migrate the Ceph deployment from ceph-ansible to Rook.

  • CentOS 9 support in osism.commons`` & osism.services`` Ansible collection.

    • The roles of the `osism.commons`` collection are now usable with CentOS 9. The roles have been tested with CentOS 9.
    • The roles of the `osism.services`` collection are now usable with CentOS 9. The roles have been tested with CentOS 9.

  • With the openstack-resource-manager it is now possible to clean up orphaned amphora instances of Octavia or volumes that are stuck in the DELETING state.

  • Kubernetes can now be deployed with k3s-ansible on the management plane and the control plane.

  • It is now possible to manage the Ceph pools independently of ceph-osds`` play using the ceph-pools` play.

Upgrade notes​

  • The switch from classic queue mirroring and durable queues to quorum queues in RabbitMQ has not yet been tested and documented. This is planned for the next release. The switch is not yet recommended.

  • The hosts_interface parameter is now set to internal_interface by default.

  • The Keycloak deployment via Docker Compose, which was previously included as a technical preview, has been completely revised and is now deployed on Kubernetes. No migration from the old deployment via Docker Compose to the new deployment via Kubernetes has been prepared. If you are currently using the Keycloak service, do not upgrade the Keycloak service and contact us in advance.

  • The Keystone role service is required by a number of OpenStack services. The role has been created automatically with new deployments for some time now. It is possible that this role is not yet available on older deployments and must be created once in preparation for the upgrade. You can check whether the role is available in the output of openstack --os-cloud admin role list. If it does not exist, it can be created with openstack --os-cloud admin role create service.

  • The use of ProxySQL for MariaDB is now possible and it is possible to switch to it as part of the upgrade. It is not mandatory and there is no recommendation. The parameter enable_proxysql is added to environments/kolla/configuration.yml for this purpose.

    environments/kolla/configuration.yml
    enable_proxysql: yes

    The secrets listed below (proxysql_admin_password, proxysql_stats_password, mariadb_monitor_password) must also be added or changed.

    When migrating to ProxySQL, it is important to upgrade MariaDB first.

    When migrating to ProxySQL, it is important to perform the loadbalancer upgrade before all OpenStack service upgrades. To make sure that the OpenStack services continue to work after the upgrade when ProxySQL is enabled as part of the upgrade, the ProxySQL service must have been deployed first. The ProxySQL service is deployed with the loadbalancer play.

    It is possible that connectivity with the database may be interrupted for a short time during the migration. It is therefore recommended to carry out extensive tests on the staging environment in advance.

  • The following secrets must be added in environments/kolla/secrets.yml:

    environments/kolla/secrets.yml
    octavia_persistence_database_password:  # generate with: pwgen 32 1
    prometheus_bcrypt_salt:                 # generate with: pwgen 22 1 <-- there's a 22
    prometheus_grafana_password:            # generate with: pwgen 32 1
    prometheus_password:                    # generate with: pwgen 32 1
    proxysql_admin_password:                # generate with: pwgen 32 1
    proxysql_stats_password:                # generate with: pwgen 32 1

  • The parameter mariadb_monitoring_password in environments/kolla/secrets.yml has to be renamed to mariadb_monitor_password. If the parameter is not present, it is added.

    environments/kolla/secrets.yml
    mariadb_monitor_password:     # generate with: pwgen 32 1

  • The following parameters must be removed from the configuration repository from environments/kolla/configuration.yml:

    environments/kolla/configuration.yml
    ceph_nova_user: nova
    ceph_nova_keyring: ceph.client.nova.keyring

  • Parameters for the Netbox service in environments/infrastructure/configuration.yml or secrets.yml must now also be added in environments/manager/configuration.yml or secrets.yml. In an upcoming release, the parameters can be removed from the infrastructure environment.

  • The Ansible callback plugin osism.commons.still_alive is now available to avoid timeouts for long-running tasks. This currently has to be explicitly enabled in the Ansible configuration. This is done in the environments/ansible.cfg file in the configuration repository. The callback plugin is enabled by default in the future. After this change has been made, the update of the manager must be performed. A manager with a version before OSISM 7.0.0 cannot be longer used if this plugin is set in environments/ansible.cfg.

    environments/ansible.cfg
    [defaults]
    ...
    stdout_callback = osism.commons.still_alive

  • In the inventory, the nova_backend parameter must be added to the host vars of compute nodes where local storage is used.

    nova_backend: default

  • The SSL certificate file haproxy.pem is now available in a different location in the haproxy container. Previously it was stored under /etc/haproxy/haproxy.pem. From now on it is stored under /etc/haproxy/certificates/haproxy.pem. If you have customised the configuration for the haproxy service or use overlays for this, adjust the locations of the SSL certificate as required.

  • Due to the upgrade from Fluentd to version 5, some directory names within the container image for Fluentd have changed. If you have worked with overlay files in the Fluentd service, check these in advance. Currently we know that /var/run/td-agent is now available as /var/run/fluentd (check osism/issues#864 for details). We assume that other directory names have changed similarly.

  • If a Keystone domain with LDAP backend is used, it is important to add the tls_cacertfile parameter in the [ldap] section of the corresponding domain configuration.

    [ldap]
    ...
    tls_cacertfile = /etc/ssl/certs/ca-certificates.crt

Known issues​

  • If error Couldn't fetch the key client.bootstrap-rbd at /var/lib/ceph/bootstrap-rbd/." occurs when updating Ceph in task create potentially missing keys (rbd and rbd-mirror), create directory /var/lib/ceph/bootstrap-rbd/ on the 1st control node used for Ceph. Use the UID 64045 and the GID 64045. Set 0755 as permissions.

  • There are currently problems when using a custom CA in combination with Octavia (osism/issues#890). A bugfix for this will be made available soon.

  • There is another problem when using Octavia in combination with OVN which leads to a leakage of ports when deleting load balancers (osism/issues#921). A bugfix for this is also expected to be available soon.

  • The manager service is updated via osism update manager. If this command is not yet available, you can use osism-update-manager as an alternative.

    osism: 'update manager' is not an osism command. See 'osism --help'.

Other & References​

Refstack 2022.11 results:

======
Totals
======
Ran: 356 tests in 1221.9879 sec.
  - Passed: 353
  - Skipped: 3
  - Expected Fail: 0
  - Unexpected Success: 0
  - Failed: 0
Sum of execute time for each test: 715.6658 sec.

OpenStack 2023.2 press announcement: https://www.openstack.org/software/openstack-bobcat

OpenStack 2023.2 release notes: https://releases.openstack.org/bobcat/index.html

Release notes for each OpenStack service: