Sometimes, I have free time remaining after I’ve tested the live .iso. So, I install it and play around with it for a few minutes, checking to see that everything works and whatnot.
This is almost invariably on the same hardware - I’m not testing on multiple pieces of hardware for the purposes of this question. I presume that testing on multiple bits of hardware would result in two test results, so that’s not the question.
The question is, when I’ve tested the live .iso and then gone ahead and installed it, do I fill out a second test result for the “Install using Calamares (entire disk)”? More accurately, is it of any value if I do?
Each installation deserves its own test. From what I’m hearing, though, what you’re doing is an installation plus a general once over. There should be a test case for those separately. If not, that’s something we need to add. We’d love your help with it!
On this page, there is both testing for the live environment and one for installation. I normally do just the first one. Sometimes, after that’s done, I do an installation (using the whole disk, actually) and play around for a bit longer, just because I had the time to do so.
But, it’s run live once - and then (sometimes) installed once. In my head, that’s two test results - but I’m not really sure what value the second ‘test’ has as it has already been tested in the live environment.
I’m more than happy to fill out a second form on days when I do both, for the record. I give the system 2 to 5 hours per week, but it turns out that I’ve had quite a bit more free time than that. The pandemic has its upsides!
You know this topic has me considering something: testing the live environment might be silly. I mean, sure, it’s probably good to make sure it works perfectly for people just checking to see if it works, but really, what matters is if the installed system works correctly. Maybe we should have a third test case, or perhaps just replace the live one. Thoughts?
In order to do the install you have to first do a live session…the big difference is that the requirements for the live session on the testing tracker site require that in addition to the booting “live” you also test the applications.Of course you could also do this after an install…I sometimes like to test printing,LAN connections,Bluetooth,scanning etc after an install …in addition to doing this on the “live session only”.
I think replacing the “live case” with an “install plus case” would be a good way forward and a more comprehensive approach - i.e. test apps and some of the above mentioned items - at the discretion of the tester of course.
What are the odds that the live system has different behavior than the installed system? If those odds are greater than zero, it may be worth keeping both.
And…
That seems most viable, though there’s the cost vs. benefit where ‘cost’ is equal to time invested.
Kinda a level above what the bare basic ‘install it and see if it boots’ for the “Install using Calamares (entire disk)” test? I can dig it.
I don’t have the time to commit to doing all the additional things, like testing printers and external devices. Well, if I did have the time, I’d still be reluctant to commit to doing it consistently. I’m currently tossing an hour+ at it per day, which is fine by me. I do like the idea however. In fact, there may even be days when I can commit to more.
It’s greater than zero, but not by much. An example where the live system is most certainly different than the installed system: there’s no installer in the latter.
Right. That’s why I’m kind of uncertain about the value in doing two testcases that ultimately test the same thing, especially for something as extensive. I would argue it’s probably quicker to do an install than to run through and test everything.
I don’t think this is essential. It is essential that the applications for those aspects open without unexpected errors and that behavior is appropriate. In other words, you can open the printer app and you might not see printers because you have none, but it works correctly. Does that make sense?
Having given it some more thought, if there is a new test case then it should be “Install Plus” - meaning you install, maybe not using the entire partition, that you test some hardware configurations, that you go above and beyond the ‘install with the full disk’ test that already exists.
Leok and guiverc tend to do those types of test already.
Which leads me back to the value for the time. It’s not my time that I’d be spending. Though I’ll probably help more with those sorts of things as I acclimate. I’ve already got the live testing down to under 30 minutes from start to finish - unless something glitches. Then I poke around a bit longer.
‘live’ tests test the software we’ve included on the system. ‘live’ tests do not involve the installation, thus the ‘install’ is a separate test, as I would fill out two reports. The ‘live’ covering all you did in ‘live’, which the test case correctly requires you open & close the programs in the menu, thus it’s more than just booting.
testcases/image/Auto login (+31/-0)
testcases/image/Custom partitioning on BTRFS (+30/-0)
testcases/image/Custom partitioning on XFS (+30/-0)
testcases/image/Custom partitioning with seperate home (+30/-0)
testcases/image/Full disk install encryption BIOS internet (+32/-0)
testcases/image/Full disk install encryption BIOS offline (+39/-0)
testcases/image/Full disk install encryption EFI internet (+32/-0)
testcases/image/Full disk install encryption EFI offline (+39/-0)
testcases/image/Full disk install encryption SecureEFI internet (+32/-0)
testcases/image/Full disk install encryption SecureEFI offline (+39/-0)
testcases/image/Full disk install no-encryption BIOS internet (+32/-0)
testcases/image/Full disk install no-encryption BIOS offline (+37/-0)
testcases/image/Full disk install no-encryption EFI internet (+32/-0)
testcases/image/Full disk install no-encryption EFI offline (+37/-0)
testcases/image/Full disk install no-encryption SecureEFI internet (+32/-0)
testcases/image/Full disk install no-encryption SecureEFI offline (+37/-0)
testcases/image/Install using another language (+29/-0)
testcases/image/Install with existing partition (+32/-0)
testcases/image/Modern Live (+20/-0)
testcases/image/Replace a partition (+30/-0)
testcases/image/Upgrade using GUI (+36/-0)
testcases/image/Upgrade using TUI (+33/-0)
Personally I feel that number of test options is rather excessive, and potentially scary (for new testers). The code is there if anyone wants to pick it up (still stuck in review) I passed ownership to Lubuntu QA long ago, so if a new direction is chosen, it maybe a useful resource anyway.
We have had issues a number of times were bugs occur on live systems, but not installed systems, plus the reverse too (ie. live is flawless, but install has issues). It’s not common, but it does occur.
I’ve figured out what I’m going to do - for the time being, unless things change (which they probably needn’t do).
If I test the live ISO, I’ll file the test.
If I go ahead and install it, and then play around with it, then I’ll fill out a second test form for the installation category.
I mostly am still in that phase where I’m trying to not take time from people. I don’t want to fill in multiple test results unless they’re actually of some value. I figure the dev time is pretty valuable and I can appreciate the cost of time.
Could tests maybe be automated? I see 20+ tests just for the installer, which might be something for Terraform. What other things are tested? Could they be tested in an automated matter, to ease the work for testers? And also be very consistent in results.
This is really not a question for Lubuntu, but a question for the Ubuntu project as a whole. Manual testing has always been a part of what has been done. Even with Ubuntu Server, where there isautomated testing, they still insist on doing manual testing. That said, we could probably set something up, but it seems like a fair amount of effort for a small benefit.
In a great coincidence, I’ve been reading about automated testing over the past week or so. I’ve only managed to get in a few hours here and a few hours there, as my blocks of time are still scarce.
I am currently hoping to get this installed and looked at this week:
AutoKey is a way to automate input in a way, but it is not necessarily meant to do automated testing. You’d have to build up a whole framework to evaluate results.
My thinking is that I may be able to use it to input all the things that I normally input during a test of the live distro. Thus saving me oodles of clicking.
Though I’ll probably help more with those sorts of things as I acclimate. I’ve already got the live testing down to under 30 minutes from start to finish - unless something glitches. Then I poke around a bit longer.