Testing Checklist

Installer tests - 22.04.4 (Jammy Jellyfish)

Release date: - 23 February 2024
Final Freeze: - -
Beta release: - -
RC reached at: - -

IMPORTANT: All of these tests must be re-done if an upload of Calamares or settings is done after a pre-release freeze.

  • calamares: 3.2.61-0ubuntu0.1
  • calamares-settings-lubuntu: 1:22.04.4.3
  • calamares-settings-ubuntu-common: 1:22.04.4.3
  • lubuntu-default-settings: 22.04.2
  • casper 1.470.3 (ISO of 20230216 [& possibly earlier] used 1.470.2)

NOTE: Currently updated for the jammy media [20240125]

https://iso.qa.ubuntu.com/qatracker/milestones/429/builds

Full disk install tests

Type Encryption Firmware Internet Swap (Y/N) Completed Date Tester ISO box.detail paste.for.tracker
Full disk install no BIOS yes y 20240221 @arraybolt3 20240216 dell_opt_9020 testcase: full disk, no-encryption, BIOS, internet, swap
Full disk install yes BIOS yes n 20240219 @guiverc 20240216 dc7700 testcase: full disk, encryption, BIOS, internet, no-swap
Full disk install no BIOS no y 20240222 @arraybolt3 20240216 hp_elitebook_8570p testcase: full disk, no-encryption, BIOS, no-internet, swap
Full disk install yes BIOS no n 20240222 @guiverc 20240216 hp_dc7700 testcase: full disk, encryption, BIOS, no-internet, no-swap
Full disk install no EFI yes y 20240222 @arraybolt3 20240216 hp_elitebook_8570p testcase: full disk, no-encryption, EFI, internet, swap
Full disk install yes EFI yes y 20240219 @guiverc 20240216 svp11216cgb testcase: full disk, encryption, EFI, internet, swap
Full disk install no EFI no n 20240221 @arraybolt3 20240216 dell_opt_9020 testcase: full disk, no-encryption, EFI, no-internet, no-swap
Full disk install yes EFI no n 20240221 @arraybolt3 20240216 dell_opt_9020 testcase: full disk, encryption, EFI, no-internet, no-swap
Full disk install no EFI + secure boot yes y 20240221 @arraybolt3 20240216 dell_opt_9020 testcase: full disk, no-encryption, secure-EFI, internet, swap
Full disk install yes EFI + secure boot yes n 20240221 @arraybolt3 20240216 dell_opt_9020 testcase: full disk, encryption, secure-EFI, internet, no-swap
Full disk install no EFI + secure boot no y 20240218 @leok 20240216 dell_lat_7280 testcase: full disk, no-encryption, secure-EFI, no-internet, swap
Full disk install yes EFI + secure boot no n 20240218 @leok 20240216 lenV14ILL testcase: full disk, encryption, secure-EFI, no-internet, no-swap

Please try and ensure one of SWAP (Y) & NO-SWAP (N) exist for each test type (ie. BIOS, uEFI or Secure-uEFI) for releases of hirsute or later.

Other Installs

Install Type Completed date Tester ISO box.detail paste.for.tracker
Install alongside 20240221 @arraybolt3 20240216 hp-elitebook-8570p testcase: install alongside
Replace partition 20240221 @arraybolt3 20240216 hp-elitebook-8570p testcase: replace partition
Install using existing partition 20240222 @arraybolt3 20240216 hp-elitebook-8570p testcase: install using existing partition
Custom partitioning with separate /home 20240218 @leok 22240216 dell_opt_7280 testcase: custom partitioning with separate /home
Custom partitioning on btrfs 20240221 @arraybolt3 20240216 dell_opt_9020 testcase: btrfs install
Custom partitioning on xfs 20240218 @leok 20240216 asrock hm410 testcase: xfs install
Install using another language 20240218 @leok 20240216 dell_lat_7280
Auto-login after install 20240218 @leok 20240216 dell_lat_7280

The install using another language, and/or auto-login can be combined with another install, ie. a single QA-test can be used to refresh the details of up to two lines on this checklist (eg. auto-install or install with another language)

Other test items

Test Type Completed Date Tester box.detail
GUI upgrade¹ from prior release na na na
TUI upgrade² from prior release na na na
GUI upgrade¹ from LTS to next LTS release na na na
TUI upgrade² from LTS to next LTS release na na na

¹ do-release-upgrade -d -m desktop -f DistUpgradeViewKDE
² do-release-upgrade -d

  • The purpose of the “Optional box” field is to allow re-tests to be done on a different box to prior time (if possible)
  • na or - is short for not-applicable

For help in understanding this checklist, please view Testing Checklist - understanding the testcases

6 Likes

To cope with all the newer install options we have available for noble & up (ie. 22.04 LTS), we maybe need to reduce some of our testing.

Maybe restrict to two only for each class of machines (instead of our current four), eg.

For BIOS/legacy =

  • testcase: full disk, no-encryption, BIOS, internet, swap
  • testcase: full disk, encryption, BIOS, no-internet, no-swap

For uEFI -

  • testcase: full disk, no-encryption, EFI, no-internet, no-swap
  • testcase: full disk, encryption, EFI, internet, swap

For Secure-uEFI

  • testcase: full disk, no-encryption, secure-EFI, internet, swap
  • testcase: full disk, encryption, secure-EFI, no-internet, no-swap

(ie. two options, one with another without encryption,

Dropping recent 12 tests to 6

We then test extra tests for

  • full installation
  • normal installation
  • minimal installation

The extra options are maybe best mixed in with the full disk install options, ie.

  • download & install updates (for most internet connection installs)
  • install third party (again most internet)
  • element
  • thunderbird
  • virtual machine manager
  • krita

(the last four apps; element-krita can be tacked onto an existing install type where internet is used; alas with 6 only full disk installs we’re left only with 3 to use… so we can add another of the recent EFI tests maybe)

  • testcase: full disk, no-encryption, EFI, internet, swap

Alternatively we keep the 12 full disk install tests we have now, and add options to cover

  • testcase: full disk, no-encryption, BIOS, internet, swap, updates, no-third-party
  • testcase: full disk, encryption, BIOS, internet, no-swap, no-updates, third-party
  • testcase: full disk, no-encryption, BIOS, no-internet, swap, no-updates, no-third-party
  • testcase: full disk, encryption, BIOS, no-internet, no-swap, no-updates, no-third-party

  • testcase: full disk, no-encryption, EFI, internet, swap, no-updates, third-party, element
  • testcase: full disk, encryption, EFI, internet, swap updates, third-party, thunderbird
  • testcase: full disk, no-encryption, EFI, no-internet, no-swap, no-updates, no-third-party
  • testcase: full disk, encryption, EFI, no-internet, no-swap, no-updates, no-third-party

  • testcase: full disk, no-encryption, secure-EFI, internet, swap, updates, third-party, virtual-machine-manager
  • testcase: full disk, encryption, secure-EFI, internet,swap, no-updates, no-third-party, krita
  • testcase: full disk, no-encryption, secure-EFI, no-internet, swap, no-updates, no-third-party
  • testcase: full disk, encryption, secure-EFI, no-internet, no-swap, no-updates, no-third-party

My major concern is complexity and width, both on sites like phab where we had it, and here on discourse, let alone on testing iso.qa.ubuntu.com

ie. my last iso.qa checklist item I was never happy with (due complexity), eg.

<dt>Select the "Erase disk" and options that match one of the following install types"
    <dd> "testcase: full disk, no-encryption, BIOS, internet, swap" </dd>
    <dd> "testcase: full disk, encryption, BIOS, internet, no-swap" </dd>
    <dd> "testcase: full disk, no-encryption, BIOS, no-internet, swap" </dd>
    <dd> "testcase: full disk, encryption, BIOS, no-internet, no-swap" </dd>
    <dd>Please include a brief description of your hardware in the first comment line on iso.qa.ubuntu.com</dd>
    <dd>Please copy and paste your chosen testcase into the second comment line on iso.qa.ubuntu.com</dd>
    <dd>Please ensure you see BIOS listed besides your drive. If you see EFI it's covered in another testcase, meaning there is a problem with the installer (please file a bug report) OR your hardware doesn't match this install type</dd>
    <dd>A partition layout should be shown which uses all disk space, and the bootloader should be installed to the correct disk</dd>
    <dd>Please select drop-down to match your chosen swap/no-swap option</dd>
    <dd>Please also select encryption if chosen, and enter passphrase as requiredtwice</ddt>
    <dd>Confirm Boot Loader location is correct</dd>
    <dd>If "Erase disk" does not show, your target device has a mounted partition. Please close installer, umount your partitions (possibly a swap partition that was mounted on boot), and return to the 'click the "Install FAMILY XX.YY" icon step'</dd>

(which was the BIOS testcase) but the additional tests as listed here could be tacked onto the BIOS, uEFI & Secure-uEFI testcases easily. Any thoughts?

2 Likes

Thanks @guiverc - agree this needs a good think! With the extra options the possibilities are very many.
Then we also have the combinations of file systems etc.

1 Like

I didn’t mention the BTRFS/XFS intentionally, as I didn’t want to change them. My latest (as of months/cycles? ago) testcase combined them into a single test like part of the BIOS I pasted into what I typed… and I was expecting/intending for the other items in this “checklist” to remain the same.

2 Likes