Testing Checklist

Development
,
1

Installer tests - 22.04.4 (Jammy Jellyfish)

Release date: - 23 February 2024
Final Freeze: - -
Beta release: - -
RC reached at: - -

IMPORTANT: All of these tests must be re-done if an upload of Calamares or settings is done after a pre-release freeze.

  • calamares: 3.2.61-0ubuntu0.1
  • calamares-settings-lubuntu: 1:22.04.4.3
  • calamares-settings-ubuntu-common: 1:22.04.4.3
  • lubuntu-default-settings: 22.04.2
  • casper 1.470.3 (ISO of 20230216 [& possibly earlier] used 1.470.2)

NOTE: Currently updated for the jammy media [20240125]

https://iso.qa.ubuntu.com/qatracker/milestones/429/builds

Full disk install tests

Type Encryption Firmware Internet Swap (Y/N) Completed Date Tester ISO box.detail paste.for.tracker
Full disk install no BIOS yes y 20240221 @arraybolt3 20240216 dell_opt_9020 testcase: full disk, no-encryption, BIOS, internet, swap
Full disk install yes BIOS yes n 20240219 @guiverc 20240216 dc7700 testcase: full disk, encryption, BIOS, internet, no-swap
Full disk install no BIOS no y 20240222 @arraybolt3 20240216 hp_elitebook_8570p testcase: full disk, no-encryption, BIOS, no-internet, swap
Full disk install yes BIOS no n 20240222 @guiverc 20240216 hp_dc7700 testcase: full disk, encryption, BIOS, no-internet, no-swap
Full disk install no EFI yes y 20240222 @arraybolt3 20240216 hp_elitebook_8570p testcase: full disk, no-encryption, EFI, internet, swap
Full disk install yes EFI yes y 20240219 @guiverc 20240216 svp11216cgb testcase: full disk, encryption, EFI, internet, swap
Full disk install no EFI no n 20240221 @arraybolt3 20240216 dell_opt_9020 testcase: full disk, no-encryption, EFI, no-internet, no-swap
Full disk install yes EFI no n 20240221 @arraybolt3 20240216 dell_opt_9020 testcase: full disk, encryption, EFI, no-internet, no-swap
Full disk install no EFI + secure boot yes y 20240221 @arraybolt3 20240216 dell_opt_9020 testcase: full disk, no-encryption, secure-EFI, internet, swap
Full disk install yes EFI + secure boot yes n 20240221 @arraybolt3 20240216 dell_opt_9020 testcase: full disk, encryption, secure-EFI, internet, no-swap
Full disk install no EFI + secure boot no y 20240218 @leok 20240216 dell_lat_7280 testcase: full disk, no-encryption, secure-EFI, no-internet, swap
Full disk install yes EFI + secure boot no n 20240218 @leok 20240216 lenV14ILL testcase: full disk, encryption, secure-EFI, no-internet, no-swap

Please try and ensure one of SWAP (Y) & NO-SWAP (N) exist for each test type (ie. BIOS, uEFI or Secure-uEFI) for releases of hirsute or later.

Other Installs

Install Type Completed date Tester ISO box.detail paste.for.tracker
Install alongside 20240221 @arraybolt3 20240216 hp-elitebook-8570p testcase: install alongside
Replace partition 20240221 @arraybolt3 20240216 hp-elitebook-8570p testcase: replace partition
Install using existing partition 20240222 @arraybolt3 20240216 hp-elitebook-8570p testcase: install using existing partition
Custom partitioning with separate /home 20240218 @leok 22240216 dell_opt_7280 testcase: custom partitioning with separate /home
Custom partitioning on btrfs 20240221 @arraybolt3 20240216 dell_opt_9020 testcase: btrfs install
Custom partitioning on xfs 20240218 @leok 20240216 asrock hm410 testcase: xfs install
Install using another language 20240218 @leok 20240216 dell_lat_7280
Auto-login after install 20240218 @leok 20240216 dell_lat_7280

The install using another language, and/or auto-login can be combined with another install, ie. a single QA-test can be used to refresh the details of up to two lines on this checklist (eg. auto-install or install with another language)

Other test items

Test Type Completed Date Tester box.detail
GUI upgradeĀ¹ from prior release na na na
TUI upgradeĀ² from prior release na na na
GUI upgradeĀ¹ from LTS to next LTS release na na na
TUI upgradeĀ² from LTS to next LTS release na na na

Ā¹ do-release-upgrade -d -m desktop -f DistUpgradeViewKDE
Ā² do-release-upgrade -d

  • The purpose of the ā€œOptional boxā€ field is to allow re-tests to be done on a different box to prior time (if possible)
  • na or - is short for not-applicable

For help in understanding this checklist, please view Testing Checklist - understanding the testcases

6 Likes
2

To cope with all the newer install options we have available for noble & up (ie. 22.04 LTS), we maybe need to reduce some of our testing.

Maybe restrict to two only for each class of machines (instead of our current four), eg.

For BIOS/legacy =

  • testcase: full disk, no-encryption, BIOS, internet, swap
  • testcase: full disk, encryption, BIOS, no-internet, no-swap

For uEFI -

  • testcase: full disk, no-encryption, EFI, no-internet, no-swap
  • testcase: full disk, encryption, EFI, internet, swap

For Secure-uEFI

  • testcase: full disk, no-encryption, secure-EFI, internet, swap
  • testcase: full disk, encryption, secure-EFI, no-internet, no-swap

(ie. two options, one with another without encryption,

Dropping recent 12 tests to 6

We then test extra tests for

  • full installation
  • normal installation
  • minimal installation

The extra options are maybe best mixed in with the full disk install options, ie.

  • download & install updates (for most internet connection installs)
  • install third party (again most internet)
  • element
  • thunderbird
  • virtual machine manager
  • krita

(the last four apps; element-krita can be tacked onto an existing install type where internet is used; alas with 6 only full disk installs weā€™re left only with 3 to useā€¦ so we can add another of the recent EFI tests maybe)

  • testcase: full disk, no-encryption, EFI, internet, swap

Alternatively we keep the 12 full disk install tests we have now, and add options to cover

  • testcase: full disk, no-encryption, BIOS, internet, swap, updates, no-third-party
  • testcase: full disk, encryption, BIOS, internet, no-swap, no-updates, third-party
  • testcase: full disk, no-encryption, BIOS, no-internet, swap, no-updates, no-third-party
  • testcase: full disk, encryption, BIOS, no-internet, no-swap, no-updates, no-third-party
  • testcase: full disk, no-encryption, EFI, internet, swap, no-updates, third-party, element
  • testcase: full disk, encryption, EFI, internet, swap updates, third-party, thunderbird
  • testcase: full disk, no-encryption, EFI, no-internet, no-swap, no-updates, no-third-party
  • testcase: full disk, encryption, EFI, no-internet, no-swap, no-updates, no-third-party
  • testcase: full disk, no-encryption, secure-EFI, internet, swap, updates, third-party, virtual-machine-manager
  • testcase: full disk, encryption, secure-EFI, internet,swap, no-updates, no-third-party, krita
  • testcase: full disk, no-encryption, secure-EFI, no-internet, swap, no-updates, no-third-party
  • testcase: full disk, encryption, secure-EFI, no-internet, no-swap, no-updates, no-third-party

My major concern is complexity and width, both on sites like phab where we had it, and here on discourse, let alone on testing iso.qa.ubuntu.comā€¦

ie. my last iso.qa checklist item I was never happy with (due complexity), eg.

<dt>Select the "Erase disk" and options that match one of the following install types"
    <dd> "testcase: full disk, no-encryption, BIOS, internet, swap" </dd>
    <dd> "testcase: full disk, encryption, BIOS, internet, no-swap" </dd>
    <dd> "testcase: full disk, no-encryption, BIOS, no-internet, swap" </dd>
    <dd> "testcase: full disk, encryption, BIOS, no-internet, no-swap" </dd>
    <dd>Please include a brief description of your hardware in the first comment line on iso.qa.ubuntu.com</dd>
    <dd>Please copy and paste your chosen testcase into the second comment line on iso.qa.ubuntu.com</dd>
    <dd>Please ensure you see BIOS listed besides your drive. If you see EFI it's covered in another testcase, meaning there is a problem with the installer (please file a bug report) OR your hardware doesn't match this install type</dd>
    <dd>A partition layout should be shown which uses all disk space, and the bootloader should be installed to the correct disk</dd>
    <dd>Please select drop-down to match your chosen swap/no-swap option</dd>
    <dd>Please also select encryption if chosen, and enter passphrase as requiredtwice</ddt>
    <dd>Confirm Boot Loader location is correct</dd>
    <dd>If "Erase disk" does not show, your target device has a mounted partition. Please close installer, umount your partitions (possibly a swap partition that was mounted on boot), and return to the 'click the "Install FAMILY XX.YY" icon step'</dd>

(which was the BIOS testcase) but the additional tests as listed here could be tacked onto the BIOS, uEFI & Secure-uEFI testcases easily. Any thoughts?

2 Likes
3

Thanks @guiverc - agree this needs a good think! With the extra options the possibilities are very many.
Then we also have the combinations of file systems etc.

1 Like
4

I didnā€™t mention the BTRFS/XFS intentionally, as I didnā€™t want to change them. My latest (as of months/cycles? ago) testcase combined them into a single test like part of the BIOS I pasted into what I typedā€¦ and I was expecting/intending for the other items in this ā€œchecklistā€ to remain the same.

2 Likes