Testing Checklist

guiverc · November 29, 2023, 4:16am

Installer tests - 22.04.4 (Jammy Jellyfish)

Release date: - 23 February 2024
Final Freeze: - -
Beta release: - -
RC reached at: - -

IMPORTANT: All of these tests must be re-done if an upload of Calamares or settings is done after a pre-release freeze.

calamares: 3.2.61-0ubuntu0.1
calamares-settings-lubuntu: 1:22.04.4.3
calamares-settings-ubuntu-common: 1:22.04.4.3
lubuntu-default-settings: 22.04.2
casper 1.470.3 (ISO of 20230216 [& possibly earlier] used 1.470.2)

NOTE: Currently updated for the jammy media [20240125]

https://iso.qa.ubuntu.com/qatracker/milestones/429/builds

Ubuntu Release tracker - Jammy Jellyfish (22.04.4 LTS) Point-Release Status Tracking
Ubuntu Release Schedule - Jammy Jellyfish Release Schedule - Release - Ubuntu Community Hub
Ubuntu Release Notes - Jammy Jellyfish Release Notes - Release - Ubuntu Community Hub
Lubuntu issue tracker - (nothing so far to track; nothing expected either)

Full disk install tests

Type	Encryption	Firmware	Internet	Swap (Y/N)	Completed Date	Tester	ISO	box.detail	paste.for.tracker
Full disk install	no	BIOS	yes	y	20240221	@arraybolt3	20240216	dell_opt_9020	testcase: full disk, no-encryption, BIOS, internet, swap
Full disk install	yes	BIOS	yes	n	20240219	@guiverc	20240216	dc7700	testcase: full disk, encryption, BIOS, internet, no-swap
Full disk install	no	BIOS	no	y	20240222	@arraybolt3	20240216	hp_elitebook_8570p	testcase: full disk, no-encryption, BIOS, no-internet, swap
Full disk install	yes	BIOS	no	n	20240222	@guiverc	20240216	hp_dc7700	testcase: full disk, encryption, BIOS, no-internet, no-swap
Full disk install	no	EFI	yes	y	20240222	@arraybolt3	20240216	hp_elitebook_8570p	testcase: full disk, no-encryption, EFI, internet, swap
Full disk install	yes	EFI	yes	y	20240219	@guiverc	20240216	svp11216cgb	testcase: full disk, encryption, EFI, internet, swap
Full disk install	no	EFI	no	n	20240221	@arraybolt3	20240216	dell_opt_9020	testcase: full disk, no-encryption, EFI, no-internet, no-swap
Full disk install	yes	EFI	no	n	20240221	@arraybolt3	20240216	dell_opt_9020	testcase: full disk, encryption, EFI, no-internet, no-swap
Full disk install	no	EFI + secure boot	yes	y	20240221	@arraybolt3	20240216	dell_opt_9020	testcase: full disk, no-encryption, secure-EFI, internet, swap
Full disk install	yes	EFI + secure boot	yes	n	20240221	@arraybolt3	20240216	dell_opt_9020	testcase: full disk, encryption, secure-EFI, internet, no-swap
Full disk install	no	EFI + secure boot	no	y	20240218	@leok	20240216	dell_lat_7280	testcase: full disk, no-encryption, secure-EFI, no-internet, swap
Full disk install	yes	EFI + secure boot	no	n	20240218	@leok	20240216	lenV14ILL	testcase: full disk, encryption, secure-EFI, no-internet, no-swap

Please try and ensure one of SWAP (Y) & NO-SWAP (N) exist for each test type (ie. BIOS, uEFI or Secure-uEFI) for releases of hirsute or later.

Other Installs

Install Type	Completed date	Tester	ISO	box.detail	paste.for.tracker
Install alongside	20240221	@arraybolt3	20240216	hp-elitebook-8570p	testcase: install alongside
Replace partition	20240221	@arraybolt3	20240216	hp-elitebook-8570p	testcase: replace partition
Install using existing partition	20240222	@arraybolt3	20240216	hp-elitebook-8570p	testcase: install using existing partition
Custom partitioning with separate /home	20240218	@leok	22240216	dell_opt_7280	testcase: custom partitioning with separate /home
Custom partitioning on btrfs	20240221	@arraybolt3	20240216	dell_opt_9020	testcase: btrfs install
Custom partitioning on xfs	20240218	@leok	20240216	asrock hm410	testcase: xfs install
Install using another language	20240218	@leok	20240216	dell_lat_7280
Auto-login after install	20240218	@leok	20240216	dell_lat_7280

The install using another language, and/or auto-login can be combined with another install, ie. a single QA-test can be used to refresh the details of up to two lines on this checklist (eg. auto-install or install with another language)

Other test items

Test Type	Completed Date	Tester	box.detail
GUI upgrade¹ from prior release	na	na	na
TUI upgrade² from prior release	na	na	na
GUI upgrade¹ from LTS to next LTS release	na	na	na
TUI upgrade² from LTS to next LTS release	na	na	na

¹ do-release-upgrade -d -m desktop -f DistUpgradeViewKDE
² do-release-upgrade -d

The purpose of the “Optional box” field is to allow re-tests to be done on a different box to prior time (if possible)
na or - is short for not-applicable

For help in understanding this checklist, please view Testing Checklist - understanding the testcases

guiverc · December 12, 2023, 4:44am

To cope with all the newer install options we have available for noble & up (ie. 22.04 LTS), we maybe need to reduce some of our testing.

Maybe restrict to two only for each class of machines (instead of our current four), eg.

For BIOS/legacy =

testcase: full disk, no-encryption, BIOS, internet, swap
testcase: full disk, encryption, BIOS, no-internet, no-swap

For uEFI -

testcase: full disk, no-encryption, EFI, no-internet, no-swap
testcase: full disk, encryption, EFI, internet, swap

For Secure-uEFI

testcase: full disk, no-encryption, secure-EFI, internet, swap
testcase: full disk, encryption, secure-EFI, no-internet, no-swap

(ie. two options, one with another without encryption,

Dropping recent 12 tests to 6

We then test extra tests for

full installation
normal installation
minimal installation

The extra options are maybe best mixed in with the full disk install options, ie.

download & install updates (for most internet connection installs)
install third party (again most internet)
element
thunderbird
virtual machine manager
krita

(the last four apps; element-krita can be tacked onto an existing install type where internet is used; alas with 6 only full disk installs we’re left only with 3 to use… so we can add another of the recent EFI tests maybe)

testcase: full disk, no-encryption, EFI, internet, swap

Alternatively we keep the 12 full disk install tests we have now, and add options to cover

testcase: full disk, no-encryption, BIOS, internet, swap, updates, no-third-party
testcase: full disk, encryption, BIOS, internet, no-swap, no-updates, third-party
testcase: full disk, no-encryption, BIOS, no-internet, swap, no-updates, no-third-party
testcase: full disk, encryption, BIOS, no-internet, no-swap, no-updates, no-third-party

testcase: full disk, no-encryption, EFI, internet, swap, no-updates, third-party, element
testcase: full disk, encryption, EFI, internet, swap updates, third-party, thunderbird
testcase: full disk, no-encryption, EFI, no-internet, no-swap, no-updates, no-third-party
testcase: full disk, encryption, EFI, no-internet, no-swap, no-updates, no-third-party

testcase: full disk, no-encryption, secure-EFI, internet, swap, updates, third-party, virtual-machine-manager
testcase: full disk, encryption, secure-EFI, internet,swap, no-updates, no-third-party, krita
testcase: full disk, no-encryption, secure-EFI, no-internet, swap, no-updates, no-third-party
testcase: full disk, encryption, secure-EFI, no-internet, no-swap, no-updates, no-third-party

My major concern is complexity and width, both on sites like phab where we had it, and here on discourse, let alone on testing iso.qa.ubuntu.com…

ie. my last iso.qa checklist item I was never happy with (due complexity), eg.

<dt>Select the "Erase disk" and options that match one of the following install types"
    <dd> "testcase: full disk, no-encryption, BIOS, internet, swap" </dd>
    <dd> "testcase: full disk, encryption, BIOS, internet, no-swap" </dd>
    <dd> "testcase: full disk, no-encryption, BIOS, no-internet, swap" </dd>
    <dd> "testcase: full disk, encryption, BIOS, no-internet, no-swap" </dd>
    <dd>Please include a brief description of your hardware in the first comment line on iso.qa.ubuntu.com</dd>
    <dd>Please copy and paste your chosen testcase into the second comment line on iso.qa.ubuntu.com</dd>
    <dd>Please ensure you see BIOS listed besides your drive. If you see EFI it's covered in another testcase, meaning there is a problem with the installer (please file a bug report) OR your hardware doesn't match this install type</dd>
    <dd>A partition layout should be shown which uses all disk space, and the bootloader should be installed to the correct disk</dd>
    <dd>Please select drop-down to match your chosen swap/no-swap option</dd>
    <dd>Please also select encryption if chosen, and enter passphrase as requiredtwice</ddt>
    <dd>Confirm Boot Loader location is correct</dd>
    <dd>If "Erase disk" does not show, your target device has a mounted partition. Please close installer, umount your partitions (possibly a swap partition that was mounted on boot), and return to the 'click the "Install FAMILY XX.YY" icon step'</dd>

(which was the BIOS testcase) but the additional tests as listed here could be tacked onto the BIOS, uEFI & Secure-uEFI testcases easily. Any thoughts?

leok · December 12, 2023, 8:54am

Thanks @guiverc - agree this needs a good think! With the extra options the possibilities are very many.
Then we also have the combinations of file systems etc.

guiverc · December 12, 2023, 9:01am

I didn’t mention the BTRFS/XFS intentionally, as I didn’t want to change them. My latest (as of months/cycles? ago) testcase combined them into a single test like part of the BIOS I pasted into what I typed… and I was expecting/intending for the other items in this “checklist” to remain the same.