=== yofel_ is now known as yofel === trijntje_ is now known as trijntje === hggdh is now known as hggdh_AFK === ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Current Session: Introduction to QA / Testing class - Instructors: balloons [19:00] Logs for this session will be available at http://irclogs.ubuntu.com/2013/01/15/%23ubuntu-classroom.html following the conclusion of the session. [19:01] Hello and welcome to this intro to ubuntu quality session. This is the first in a series of sessions the quality team is presenting over the next several weeks. [19:01] This session is intended to introduce you to what the quality team does, and some of the sites and tools utilized by the team, as well as how you can join and participate. [19:02] You may ask questions at any point.. Just be sure to utilize the #ubuntu-classroom-chat channel [19:02] Everyone ready? [19:02] I'm going to give a brief overview of what QA is and how the team works, then we'll dive into some of the tools we use [19:02] yes [19:02] Finally, we'll talk about how you can get involved and then do a Q & A [19:03] So to start off, let’s take a look at the wiki page for the team [19:03] http://wiki.ubuntu.com/QATeam [19:04] On the page you can see some of the teams goals and purpose [19:04] simply put, we help ensure everyone's work in ubuntu is presented in the best possible way [19:05] from designing good process, to testing, to making sure things 'just work', we want the culmination of work that results in the ubuntu image to be the best it can be [19:06] So where do we hang out? [19:06] Many different places actually :-) [19:06] The quality team is a diverse group of folks with different skillsets [19:06] however, we do have some official communication channels should you need to reach us [19:07] right here on IRC, #ubuntu-quality [19:07] on our mailing list: https://lists.ubuntu.com/mailman/listinfo/ubuntu-quality [19:07] in the forums; http://ubuntuforums.org/forumdisplay.php?f=427 [19:07] on askubuntu; http://askubuntu.com/questions/tagged/quality [19:08] and even on social sites like facebook, twitter and google [19:09] So let's talk about some of the sites that we utilize and there purpose [19:09] First up is http://errors.ubuntu.com [19:10] errors is driven by whoopsie-daisy, the brainchild of evan.. If anyone was able to make it to the last UDS, you may have seen his presentation on the behind the scenes details for this tool and it's growth [19:11] what errors show us is the error rates for our current ubuntu installs, linking them to bug reports, sorting by frequency and showing the average over time [19:12] although as a team we are typically focused more on the developing version of ubuntu, we can utilize this data to see what might need more testing or new regressions tests if a package has a high occurrence of crtiical bugs [19:12] Next up is the QA dashboard [19:12] http://reports.qa.ubuntu.com/ [19:13] dkessel asked: what is the scale of that graph? 0.1 errors per day can't be right? [19:14] good question dkessel.. I believe that is showing the errors per day in comparison to a benchmark rate, in this case 12.04 [19:14] I am willing to be corrected on that, but that's my understanding :-) [19:15] Ok, back to the dashboard :-) [19:15] This is currently utilized to show the automated testing results for our daily images [19:16] by default your looking at the current development release, and todays builds sorted descending by date [19:17] we can utilize this to follow-up with further manual testing on images that repeatedly show as broken. In addition, we withhold publishing known broken images for manual testing until they meet a baseline criteria for installation via automated testing [19:17] Next is the QATracker [19:18] There are several instances, but I will briefly link to one, the iso tracker [19:18] http://iso.qa.ubuntu.com [19:18] The tracker is where we submit results and get testing information.. Our testing workload is managed through the different qatracker sites [19:20] Ok, I'd like to talk about some tools we utilize now [19:20] In conjunction, we'll chat about some of the activities we do in QA [19:21] As we just spoke about, the QATracker is a big part of enabling us to work together as a team [19:21] https://wiki.ubuntu.com/Testing/QATracker [19:22] chilicuil asked: what's the difference between http://reports.qa.ubuntu.com/sru/ and http://people.canonical.com/~ubuntu-archive/pending-sru.html ? [19:22] Pausing to answer this for a sec :-) [19:22] chilicuil, the difference is that the reports site is showing testing for kernel sru [19:22] the sru pending report is showing all sru's and there status [19:23] I trust that answers the question [19:23] Ok, back to the tracker quickly [19:23] the wiki page has more details about how it works and the workflow.. [19:24] we'll come back in a moment to look at the walkthroughs [19:24] for now, let's move on [19:24] We mentioned the isotracker and linked everyone to it.. iso.qa.ubuntu.com [19:25] So, you can see there are images for raring and precise ready for testing [19:25] iso testing is one activity you can do as part of the team, as is a great way to start [19:25] one of the tools you can utilize to help with this is called testdrive [19:25] This page describes the tool in a bit more detail: https://wiki.ubuntu.com/UsingDevelopmentReleases [19:26] In addition, the walkthrough uses testdrive to teach you how to undergo your first iso test: https://wiki.ubuntu.com/Testing/ISO/Walkthrough [19:27] In short, testdrive will help you download the daily iso of your choosing for testing, and ease you into doing an instillation test by starting up a vm for you [19:28] Ok, so that's one facet of 'manual testing' [19:29] there's a few more.. We also test packages.. packages.qa.ubuntu.com. This typically invovles installing software from a ppa and running through a set of tests. [19:29] Finally, phillw in the next session will be covering writing tests for manual testcases, so I won't cover that here.. [19:29] On the 'automated testing' side of things we utilize a few tools as well to help [19:30] The first is called Autopilot, https://launchpad.net/autopilot [19:30] as a team we've adopted this tool this cycle to help us automate some of our manual testing that is better suited to for automation [19:31] We are having several hackathons where you can learn more about writing autopilot tests, as well as autopkg tests [19:31] Autopkg tests are the other automated testing we as a team participate in [19:31] http://developer.ubuntu.com/packaging/html/auto-pkg-test.html [19:32] The difference between the two is the scope of the test [19:32] autopkg tests are aimed at testing integration, low-level and build time testing [19:32] it's more in line with system testing, ensuring a package works as expected on the installation target [19:33] for autopilot testing, we are focused on functional testing.. this means we are looking at testing from a user or gui perspective [19:34] If you'd like to know more about contributing autopilot or autopkg tests, see these answers and join us at the hackathons [19:34] http://askubuntu.com/questions/236202/how-do-i-contribute-an-autopkg-test-to-ubuntu [19:34] http://askubuntu.com/questions/233219/how-do-i-contribute-an-autopilot-test [19:35] I'd like to touch on another activity that we as a team do in testing.. Please feel free to ask questions if I'm glazing over something, or something didn't make sense [19:35] We'll stop at the end for a general q & a as well [19:36] Hardware testing is a growing area for us as a team [19:36] The laptop testing team has been around for awhile doing excellent work to ensure ubuntu works on a wide range of hardware [19:36] https://wiki.ubuntu.com/Testing/Laptop [19:37] Currently, we are looking for more folks to help push forward a testing hw database using ubuntu friendly, hexr, and the work the laptop team continues to do.. So it's an exciting time if your interested in that type of testing [19:38] Ok, so perhaps I've enticed you to join us in our quality endeavors :-) Getting involved in quality is a great way to start contributing to ubuntu [19:39] you will be exposed to many different teams and people.. And the work and skillsets required are always interesting [19:40] The next steps for joining the team are quite simple. It's an open membership. You simply need a ubuntu SSO account [19:40] https://login.launchpad.net/+new_account, if you don't have one [19:41] That will allow you to contribute results to the tracker. In addition you should join our launchpad team and mailing list.. And then leave us a message and say hello! We're happy to help you get started and guide you through an area you'd like to help in [19:41] LP team: https://launchpad.net/~ubuntu-testing [19:41] mailing list: https://lists.ubuntu.com/mailman/listinfo/ubuntu-quality [19:41] The mailing list will keep you informed about testing events we are having, new initiatives, and allow you to ask questions and get help [19:42] Finally, our #ubuntu-quality channel has folks from around the world hanging out at many different hours of the day.. Do stop by and say hello there as well.. [19:42] Ok, That's the end of my spiel :-) I'm happy to field any questions you may have now [19:45] ripper asked: What's the size of the QA team (internal/community)? Do you develop all of your QA tools on your own? [19:47] ripper, good question. As far as size is concerned, you can look and see the number of folks on our launchpad team.. currently sitting at 183.. Not everyone is active all the time (nor expected to be :-) ), but in general we have a nice size of folks around and willing to help and engage. As far as tools are concerned, we generally utilize things built by others (typically within the ubuntu community), although some tools come from within. [19:49] For instance, autopilot comes to us from the unity team. Autopkg is a debian standard, we're just using it. Testdrive was built by the ubuntu community I believe, but virtualbox, lvm, etc are from somewhere else. The qatracker was built by folks internally for the purposes of doing testing work [19:49] In general, I always love to see more development minded folks write some nifty tools we all can utilize [19:50] JoseeAntonioR asked: Are there any QA tools that are still in development, but mature enough for the team to use, that we should know about? [19:50] There are 10 minutes remaining in the current session. [19:50] Hmm.. there are certainly other tools for QA [19:51] xpressor and sikuli for instance are both also functional testing tools.. mago / ldtp have been used in times past [19:51] checkbox is utilized by ubuntu friendly, has been utilized by our team in the past, and may again be utilized to do the hardware database [19:53] Any other questions? I hope this has been a good introduction for all of you. I would encourage you to pick out some sessions the other community members are leading on specific topics [19:53] there's some great stuff being covered, and it will help ensure you don't get lost or confused in trying to help [19:54] and you get to be instructed by someone other than me :-p.. so that might be a bonus for you [19:54] adamw asked: What's the process for testing updates to stable releases? I didn't see that covered, unless I missed it [19:54] adamw, we skimmed it briefly, but didn't talk about the process [19:54] we call these SRU's [19:55] that means, stable release updates [19:55] There are 5 minutes remaining in the current session. [19:55] The process is tracked via launchpad typically, and invovles the developer who fixed the bug, the SRU team, and (hopefully) the bug submitter and/or affected folks [19:56] The package is uploaded to a proposed queue where the aforementioned people will test and ensure it fixes the issue before being released to the general repository for everyone [19:57] Thanks for coming out everyone! I hope to see you around again. Feel free to contact me anytime [19:57] you have my IRC nick now.. I'l link my lp page [19:57] https://launchpad.net/~nskaggs [19:57] Finally, happy testing! [19:58] thanks a lot, balloons, this has been a great session. [19:59] coming up, we have an Introduction to Manual Test Cases, with phillw, in about an hour. Stay tuned! [20:00] Logs for this session will be available at http://irclogs.ubuntu.com/2013/01/15/%23ubuntu-classroom.html === ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || No Sessions Currently in Progress [20:26] Atlantic777 (: [20:27] hey :D === ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Current Session: Introduction to Manual Test Cases class - Instructors: phillw [21:00] Logs for this session will be available at http://irclogs.ubuntu.com/2013/01/15/%23ubuntu-classroom.html following the conclusion of the session. === adamw is now known as absolutely-not-a === absolutely-not-a is now known as adamw [21:03] let's wait a few seconds while phillw gets ready :) [21:04] Hi folks, I'm phillw [21:05] https://wiki.ubuntu.com/phillw [21:05] I'm going to give an introduction to manual test cases, please feel free to ask questions as I go along. If I don't immediately reply in here it is because I will be covering that question in this session. [21:05] so, to begin. [21:05] Whilst the growth and development of the automated test systems is a massive step forward, there are times when a human is needed :) [21:06] in some ways, think of the millions of dollars that NASA spent on designing a pen that would in space... The USSR simply used pencils. [21:07] As part of the evolution of testing, the manual test cases are being re-written into a new format which is easier to follow. [21:08] The test cases, which are linked to the iso tracker, are important as they give a consistant set of steps to follow. [21:09] When ever thinking about a manual test case, you must follow one simple rule... [21:09] would my mum understand what you are asking the person to do. [21:10] They are click by click instructions... Do this [21:10] expect this result [21:10] https://wiki.ubuntu.com/Testing/TestCaseFormat [21:12] even a long, involved, test case such as http://iso.qa.ubuntu.com/qatracker/testcases/1439/info [21:12] should be easy to follow [21:13] Test cases used to look (and a lot still do) look like http://testcases.qa.ubuntu.com/Applications/Calculator [21:13] The important things are written down, and the steps are there. [21:14] !q [21:14] !y [21:15] Ade asked: Are test cases written to a use case or Software Requirements Specification? [21:15] there are two types of test cases [21:15] generic (which apply across all flavours) [21:16] specific (which are application / flavour / architecture specific [21:19] there are two areas for test cases, those on the iso-tracker at http://iso.qa.ubuntu.com/qatracker/milestones/243/builds [21:20] such as http://iso.qa.ubuntu.com/qatracker/testcases/1300/info [21:20] and those for applications [21:21] which are held at http://packages.qa.ubuntu.com/ [21:22] an example being http://packages.qa.ubuntu.com/qatracker/testcases/1448/info [21:23] there is a 3rd manual test system, used for smoke testing which has a slightly different format for setting up. [21:24] As part of the push for getting things transferred to the new format, there is a page area on the QATeam area. https://wiki.ubuntu.com/QATeam/TestcaseUpdates [21:25] It may look like nothing has happened, but balloons has been tidying up the page to remove those areas that have been fully completed :) [21:28] but, what is needed is more people to assist. [21:29] as stated on https://wiki.ubuntu.com/Testing/TestCaseFormat getting involved is a good way to logically think out solutions to questions. [21:31] So, going back to http://testcases.qa.ubuntu.com/Applications/Calculator [21:31] we would begin with the standard header.... [21:33]
    [21:33]
  1. Goto Applications->Accessories->Calculator
    2. [21:33]
[21:33] at which point, we expect it to start [21:33] so, the expected result would be written === jreznik is now known as jreznik_afk [21:34]
  • Calculator starts
    • [21:35] from there on, you would state [21:36]
  • Do this
    • [21:36]
  • Expect this
  • [21:39] I've not had chance to do the calc one, give me a moment to pull up the details of an exisiting one. [21:40] http://iso.qa.ubuntu.com/qatracker/testcases/1301/info is written as [21:41] http://pastebin.com/JiZf3quS [21:42] whilst it is somewhat involved, you can see the steps for "Do This" and "Expect This". [21:43] JoseeAntonioR asked: Shouldn't the last one be '
  • Expect this
    • '? [21:43] JoseeAntonioR: indeed it should, my fault for missing the close tag. [21:44]
  • .....
    • in order to keep the format correct, my typo. [21:45] which is also no end of fun for me to debug when scratching my head :) [21:47] I hope that this introduction to writing manual test cases has not frightened people away, as stated on https://wiki.ubuntu.com/QATeam/TestcaseUpdates please feel free to contact us! [21:47] I'll leave the remaining time for any questions. [21:48] kparal asked: what is your opinion on having generic test cases (people have to figure out some stuff by themselves, or it's intentionally fuzzy to get different results from different testers) compared to very detailed test cases ("click on button X, then on button Y", which need to be updated regularly as the software changes)? do you use both approaches, or prefer just the detailed test cases? [21:49] hi kparal one of the advantages of having the test cases split between 'generic', such as those on the iso tracker and the others on the application area is that a test case can be updated for a change in the software (application). [21:50] There are 10 minutes remaining in the current session. [21:50] so, we do use both approaches. [21:51] all test cases should be in the format, Do 'A', exepect 'B'. [21:52] *expect* [21:54] as smoke tests are different to normal testing. for the record of this session I'll copy over a chat. [21:54] I would add I think the difference your describing is smoke tests versus testcases [21:54] smoke tests do have test cases as well? [21:55] meaning, during smoke testing we'll ask a series of genericish questions, to which the tester(s) can reply and find issues [21:55] (21:53:19) balloons: yes, here's some we have right now this week for pulseaudio [21:55] (21:53:37) balloons: http://packages.qa.ubuntu.com/qatracker/milestones/251/builds/34454/testcases/1336/results [21:55] (21:54:07) balloons: this type of testing is useful, but in s different way than a detailed testcase [21:55] There are 5 minutes remaining in the current session. [21:55] Ade asked: Where is the best place to view test cases that need to be written/updated and once complete then how would you submit them? [21:56] Ade, if you follow https://wiki.ubuntu.com/QATeam/TestcaseUpdates you can contact any of the team at https://launchpad.net/~ubuntu-testcase [21:57] or simply ask on the QATeam mailing list [21:57] https://wiki.ubuntu.com/QATeam/Contact [21:58] kparal asked: in which part of the cycle do you do smoke testing, and in which part do you do application test cases testing? is that separated or simultaneous? [21:59] I'm not overly familiar with smoke testing, balloons would be better person to ask about that subject. We'll make an effort to update the Wiki to answer your question (If he hasn't already done it :) ) [22:00] Logs for this session will be available at http://irclogs.ubuntu.com/2013/01/15/%23ubuntu-classroom.html [22:00] Thanks everyone. remember, you can always ask questions on #ubuntu-quality === ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || No Sessions Currently in Progress [22:01] thanks a lot, phillw! stay tuned to the calendar and the blog to see upcoming sessions. see you there!