Quality Assurance testing: Recording Results

Our Lubuntu member David (KGIII/uninvolved) has written about his own beta testing here on linux-tips and it’s worth a read.

In Lubuntu we have created many testcases (alas still not uploaded to the iso.qa.tracker) which are covered in this thread titled Testing Checklist - understanding the testcases. This will help explain the Testing Checklist we ensure is filled prior to every release (inc. re-release/point-release) of Lubuntu

Recording the results

To record your testing, you’ll need a Launchpad/Ubuntu One login. Go here to create one if you haven’t already got one.

To look at all teams testsuites for jammy, though our install testcases can be found here, and the primary ISO and QA testing tracker can be found here.

Our current behavior when we run a QA-test install is to use the “Install using Calamares (entire disk)” testcase, where in the Comments section we use the first line to briefly list hardware used, the second line has an entry describing the testcase tested, like

Lenovo YG SL 7 14ITL05 i5-1135G7,16GB,Intel Iris Xe,WiFi, 512GB M2NVMe SSD
Testcase:UEFI+secure boot,no encryption,full disk with swap file,no Internet

which highlights it was a Secure uEFI install, no encryption, full disk with swap enabled, and no internet connected; all of which matches an testcase found on our checklist, this test performed by Lubuntu Council member Leó Kolbeinsson. (The second line can be copy/pasted from the testing checklist if helpful)

Another example is from Lubuntu member David/KGIII

MSI Modern 15 (Intel Core i5-10210U, Intel UHD, 32 GB RAM) .

How you describe your hardware is up to you, I get details from sudo lshw roughly in format

“Make model (cpu, ram, gpu)”

so my lenovo thinkpad x201 will show as

lenovo thinkpad x201 (i5-m520, 4gb, i915)

If you need help making sense of the testing checklist, then this wiki page may be helpful.

Reporting bugs

Follow the directions in the testsuite. If it does all of those things, great! If it doesn’t, we do need your help. Please file a bug on your testing box, and record the bug ID in the Bug (or Critical Bug) section in the iso.qa.ubuntu.com page when complete.

Follow the directions in the bug report, and be as specific as you can. This is a key step - an unreported bug will be an unfixed bug. For details on reporting bugs please refer to our wiki page and don’t forget to include the bug report number in the iso.qa.ubuntu.com test report.

Questions?

If you have any questions, appropriate places to ask are of course, our discourse site or for faster responses please ask on IRC (#lubuntu-devel) or via our telegram channel

Testers can also chat live on IRC (#ubuntu-quality) or telegram (Ubuntu Testers).

Please note tests that complete okay with only minor issues should still be PASSED, but the bug ID reported in the “Bugs” section. If you consider it a show-stopper bug, record it in the “Critical bugs” section (failing the test if you believe absolutely necessary).

ISO testcases can take as little as 30 minutes to complete (including filing a report on the iso.qa tracker), and whilst we have only a single testcase for installs on the iso.qa site, we have an expanded checklist you can see at https://phab.lubuntu.me/w/release-team/testing-checklist/

4 Likes