=== yofel_ is now known as yofel
=== trijntje_ is now known as trijntje
=== hggdh is now known as hggdh_AFK
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Current Session: Introduction to QA / Testing class - Instructors: balloons
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2013/01/15/%23ubuntu-classroom.html following the conclusion of the session.19:00
balloonsHello and welcome to this intro to ubuntu quality session. This is the first in a series of sessions the quality team is presenting over the next several weeks.19:01
balloonsThis session is intended to introduce you to what the quality team does, and some of the sites and tools utilized by the team, as well as how you can join and participate.19:01
balloonsYou may ask questions at any point.. Just be sure to utilize the #ubuntu-classroom-chat channel19:02
balloonsEveryone ready?19:02
balloonsI'm going to give a brief overview of what QA is and how the team works, then we'll dive into some of the tools we use19:02
balloonsFinally, we'll talk about how you can get involved and then do a Q & A19:02
balloonsSo to start off, let’s take a look at the wiki page for the team19:03
balloonsOn the page you can see some of the teams goals and purpose19:04
balloonssimply put, we help ensure everyone's work in ubuntu is presented in the best possible way19:04
balloonsfrom designing good process, to testing, to making sure things 'just work', we want the culmination of work that results in the ubuntu image to be the best it can be19:05
balloonsSo where do we hang out?19:06
balloonsMany different places actually :-)19:06
balloonsThe quality team is a diverse group of folks with different skillsets19:06
balloonshowever, we do have some official communication channels should you need to reach us19:06
balloonsright here on IRC, #ubuntu-quality19:07
balloonson our mailing list:     https://lists.ubuntu.com/mailman/listinfo/ubuntu-quality19:07
balloonsin the forums;     http://ubuntuforums.org/forumdisplay.php?f=42719:07
balloonson askubuntu;     http://askubuntu.com/questions/tagged/quality19:07
balloonsand even on social sites like facebook, twitter and google19:08
balloonsSo let's talk about some of the sites that we utilize and there purpose19:09
balloonsFirst up is  http://errors.ubuntu.com19:09
balloonserrors is driven by whoopsie-daisy, the brainchild of evan.. If anyone was able to make it to the last UDS, you may have seen his presentation on the behind the scenes details for this tool and it's growth19:10
balloonswhat errors show us is the error rates for our current ubuntu installs, linking them to bug reports, sorting by frequency and showing the average over time19:11
balloonsalthough as a team we are typically focused more on the developing version of ubuntu, we can utilize this data to see what might need more testing or new regressions tests if a package has a high occurrence of crtiical bugs19:12
balloonsNext up is the QA dashboard19:12
ClassBotdkessel asked: what is the scale of that graph? 0.1 errors per day can't be right?19:13
balloonsgood question dkessel.. I believe that is showing the errors per day in comparison to a benchmark rate, in this case 12.0419:14
balloonsI am willing to be corrected on that, but that's my understanding :-)19:14
balloonsOk, back to the dashboard :-)19:15
balloonsThis is currently utilized to show the automated testing results for our daily images19:15
balloonsby default your looking at the current development release, and todays builds sorted descending by date19:16
balloonswe can utilize this to follow-up with further manual testing on images that repeatedly show as broken. In addition, we withhold publishing known broken images for manual testing until they meet a baseline criteria for installation via automated testing19:17
balloonsNext is the QATracker19:17
balloonsThere are several instances, but I will briefly link to one, the iso tracker19:18
balloonsThe tracker is where we submit results and get testing information.. Our testing workload is managed through the different qatracker sites19:18
balloonsOk, I'd like to talk about some tools we utilize now19:20
balloonsIn conjunction, we'll chat about some of the activities we do in QA19:20
balloonsAs we just spoke about, the QATracker is a big part of enabling us to work together as a team19:21
balloons        https://wiki.ubuntu.com/Testing/QATracker19:21
ClassBotchilicuil asked: what's the difference between http://reports.qa.ubuntu.com/sru/ and http://people.canonical.com/~ubuntu-archive/pending-sru.html ?19:22
balloonsPausing to answer this for a sec :-)19:22
balloonschilicuil, the difference is that the reports site is showing testing for kernel sru19:22
balloonsthe sru pending report is showing all sru's and there status19:22
balloonsI trust that answers the question19:23
balloonsOk, back to the tracker quickly19:23
balloonsthe wiki page has more details about how it works and the workflow..19:23
balloonswe'll come back in a moment to look at the walkthroughs19:24
balloonsfor now, let's move on19:24
balloonsWe mentioned the isotracker and linked everyone to it.. iso.qa.ubuntu.com19:24
balloonsSo, you can see there are images for raring and precise ready for testing19:25
balloonsiso testing is one activity you can do as part of the team, as is a great way to start19:25
balloonsone of the tools you can utilize to help with this is called testdrive19:25
balloonsThis page describes the tool in a bit more detail: https://wiki.ubuntu.com/UsingDevelopmentReleases19:25
balloonsIn addition, the walkthrough uses testdrive to teach you how to undergo your first iso test: https://wiki.ubuntu.com/Testing/ISO/Walkthrough19:26
balloonsIn short, testdrive will help you download the daily iso of your choosing for testing, and ease you into doing an instillation test by starting up a vm for you19:27
balloonsOk, so that's one facet of 'manual testing'19:28
balloonsthere's a few more.. We also test packages.. packages.qa.ubuntu.com. This typically invovles installing software from a ppa and running through a set of tests.19:29
balloonsFinally, phillw in the next session will be covering writing tests for manual testcases, so I won't cover that here..19:29
balloonsOn the 'automated testing' side of things we utilize a few tools as well to help19:29
balloonsThe first is called Autopilot, https://launchpad.net/autopilot19:30
balloonsas a team we've adopted this tool this cycle to help us automate some of our manual testing that is better suited to for automation19:30
balloonsWe are having several hackathons where you can learn more about writing autopilot tests, as well as autopkg tests19:31
balloonsAutopkg tests are the other automated testing we as a team participate in19:31
balloonsThe difference between the two is the scope of the test19:32
balloonsautopkg tests are aimed at testing integration, low-level and build time testing19:32
balloonsit's more in line with system testing, ensuring a package works as expected on the installation target19:32
balloonsfor autopilot testing, we are focused on functional testing.. this means we are looking at testing from a user or gui perspective19:33
balloonsIf you'd like to know more about contributing autopilot or autopkg tests, see these answers and join us at the hackathons19:34
balloonsI'd like to touch on another activity that we as a team do in testing.. Please feel free to ask questions if I'm glazing over something, or something didn't make sense19:35
balloonsWe'll stop at the end for a general q & a as well19:35
balloonsHardware testing is a growing area for us as a team19:36
balloonsThe laptop testing team has been around for awhile doing excellent work to ensure ubuntu works on a wide range of hardware19:36
balloonsCurrently, we are looking for more folks to help push forward a testing hw database using ubuntu friendly, hexr, and the work the laptop team continues to do.. So it's an exciting time if your interested in that type of testing19:37
balloonsOk, so perhaps I've enticed you to join us in our quality endeavors :-) Getting involved in quality is a great way to start contributing to ubuntu19:38
balloonsyou will be exposed to many different teams and people.. And the work and skillsets required are always interesting19:39
balloonsThe next steps for joining the team are quite simple. It's an open membership. You simply need a ubuntu SSO account19:40
balloonshttps://login.launchpad.net/+new_account, if you don't have one19:40
balloonsThat will allow you to contribute results to the tracker. In addition you should join our launchpad team and mailing list.. And then leave us a message and say hello! We're happy to help you get started and guide you through an area you'd like to help in19:41
balloonsLP team: https://launchpad.net/~ubuntu-testing19:41
balloonsmailing list:  https://lists.ubuntu.com/mailman/listinfo/ubuntu-quality19:41
balloonsThe mailing list will keep you informed about testing events we are having, new initiatives, and allow you to ask questions and get help19:41
balloonsFinally, our #ubuntu-quality channel has folks from around the world hanging out at many different hours of the day.. Do stop by and say hello there as well..19:42
balloonsOk, That's the end of my spiel :-) I'm happy to field any questions you may have now19:42
ClassBotripper asked: What's the size of the QA team (internal/community)? Do you develop all of your QA tools on your own?19:45
balloonsripper, good question. As far as size is concerned, you can look and see the number of folks on our launchpad team.. currently sitting at 183.. Not everyone is active all the time (nor expected to be :-) ), but in general we have a nice size of folks around and willing to help and engage. As far as tools are concerned, we generally utilize things built by others (typically within the ubuntu community), although some tools come from within.19:47
balloonsFor instance, autopilot comes to us from the unity team. Autopkg is a debian standard, we're just using it. Testdrive was built by the ubuntu community I believe, but virtualbox, lvm, etc are from somewhere else. The qatracker was built by folks internally for the purposes of doing testing work19:49
balloonsIn general, I always love to see more development minded folks write some nifty tools we all can utilize19:49
ClassBotJoseeAntonioR asked: Are there any QA tools that are still in development, but mature enough for the team to use, that we should know about?19:50
ClassBotThere are 10 minutes remaining in the current session.19:50
balloonsHmm.. there are certainly other tools for QA19:50
balloonsxpressor and sikuli for instance are both also functional testing tools.. mago / ldtp have been used in times past19:51
balloonscheckbox is utilized by ubuntu friendly, has been utilized by our team in the past, and may again be utilized to do the hardware database19:51
balloonsAny other questions? I hope this has been a good introduction for all of you. I would encourage you to pick out some sessions the other community members are leading on specific topics19:53
balloonsthere's some great stuff being covered, and it will help ensure you don't get lost or confused in trying to help19:53
balloonsand you get to be instructed by someone other than me :-p.. so that might be a bonus for you19:54
ClassBotadamw asked: What's the process for testing updates to stable releases? I didn't see that covered, unless I missed it19:54
balloonsadamw, we skimmed it briefly, but didn't talk about the process19:54
balloonswe call these SRU's19:54
balloonsthat means, stable release updates19:55
ClassBotThere are 5 minutes remaining in the current session.19:55
balloonsThe process is tracked via launchpad typically, and invovles the developer who fixed the bug, the SRU team, and (hopefully) the bug submitter and/or affected folks19:55
balloonsThe package is uploaded to a proposed queue where the aforementioned people will test and ensure it fixes the issue before being released to the general repository for everyone19:56
balloonsThanks for coming out everyone! I hope to see you around again. Feel free to contact me anytime19:57
balloonsyou have my IRC nick now.. I'l link my lp page19:57
balloonsFinally, happy testing!19:57
JoseeAntonioRthanks a lot, balloons, this has been a great session.19:58
JoseeAntonioRcoming up, we have an Introduction to Manual Test Cases, with phillw, in about an hour. Stay tuned!19:59
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2013/01/15/%23ubuntu-classroom.html20:00
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || No Sessions Currently in Progress
tockitjAtlantic777 (:20:26
Atlantic777hey :D20:27
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Current Session: Introduction to Manual Test Cases class - Instructors: phillw
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2013/01/15/%23ubuntu-classroom.html following the conclusion of the session.21:00
=== adamw is now known as absolutely-not-a
=== absolutely-not-a is now known as adamw
JoseeAntonioRlet's wait a few seconds while phillw gets ready :)21:03
phillwHi folks, I'm phillw21:04
phillwI'm going to give an introduction to manual test cases, please feel free to ask questions as I go along. If I don't immediately reply in here it is because I will be covering that question in this session.21:05
phillwso, to begin.21:05
phillwWhilst the growth and development of the automated test systems is a massive step forward, there are times when a human is needed :)21:05
phillwin some ways, think of the millions of dollars that NASA spent on designing a pen that would in space... The USSR simply used pencils.21:06
phillwAs part of the evolution of testing, the manual test cases are being re-written into a new format which is easier to follow.21:07
phillwThe test cases, which are linked to the iso tracker, are important as they give a consistant set of steps to follow.21:08
phillwWhen ever thinking about a manual test case, you must follow one simple rule...21:09
phillwwould my mum understand what you are asking the person to do.21:09
phillwThey are click by click instructions... Do this21:10
phillwexpect this result21:10
phillweven a long, involved, test case such as http://iso.qa.ubuntu.com/qatracker/testcases/1439/info21:12
phillwshould be easy to follow21:12
phillwTest cases used to look (and a lot still do) look like http://testcases.qa.ubuntu.com/Applications/Calculator21:13
phillwThe important things are written down, and the steps are there.21:13
ClassBotAde asked: Are test cases written to a use case or Software Requirements Specification?21:15
phillwthere are two types of test cases21:15
phillwgeneric (which apply across all flavours)21:15
phillwspecific (which are application / flavour / architecture specific21:16
phillwthere are two areas for test cases, those on the iso-tracker at http://iso.qa.ubuntu.com/qatracker/milestones/243/builds21:19
phillwsuch as http://iso.qa.ubuntu.com/qatracker/testcases/1300/info21:20
phillwand those for applications21:20
phillwwhich are held at http://packages.qa.ubuntu.com/21:21
phillwan example being http://packages.qa.ubuntu.com/qatracker/testcases/1448/info21:22
phillwthere is a 3rd manual test system, used for smoke testing which has a slightly different format for setting up.21:23
phillwAs part of the push for getting things transferred to the new format, there is a page area on the QATeam area. https://wiki.ubuntu.com/QATeam/TestcaseUpdates21:24
phillwIt may look like nothing has happened, but balloons has been tidying up the page to remove those areas that have been fully completed :)21:25
phillwbut, what is needed is more people to assist.21:28
phillwas stated on https://wiki.ubuntu.com/Testing/TestCaseFormat getting involved is a good way to logically think out solutions to questions.21:29
phillwSo, going back to http://testcases.qa.ubuntu.com/Applications/Calculator21:31
phillwwe would begin with the standard header....21:31
phillw<li>Goto Applications->Accessories->Calculator</li>21:33
phillwat which point, we expect it to start21:33
phillwso, the expected result would be written21:33
=== jreznik is now known as jreznik_afk
phillw<li title="expected_result">Calculator starts</li>21:34
phillwfrom there on, you would state21:35
phillw<li title = "action"> Do this </li>21:36
phillw<li title = "expected result">Expect this <li>21:36
phillwI've not had chance to do the calc one, give me a moment to pull up the details of an exisiting one.21:39
phillwhttp://iso.qa.ubuntu.com/qatracker/testcases/1301/info is written as21:40
phillwwhilst it is somewhat involved, you can see the steps for "Do This" and "Expect This".21:42
ClassBotJoseeAntonioR asked: Shouldn't the last one be '<li title = "expected result">Expect this </li>'?21:43
phillwJoseeAntonioR: indeed it should, my fault for missing the close tag.21:43
phillw<li> ..... </li> in order to keep the format correct, my typo.21:44
phillwwhich is also no end of fun for me to debug when scratching my head :)21:45
phillwI hope that this introduction to writing manual test cases has not frightened people away, as stated on https://wiki.ubuntu.com/QATeam/TestcaseUpdates please feel free to contact us!21:47
phillwI'll leave the remaining time for any questions.21:47
ClassBotkparal asked: what is your opinion on having generic test cases (people have to figure out some stuff by themselves, or it's intentionally fuzzy to get different results from different testers) compared to very detailed test cases ("click on button X, then on button Y", which need to be updated regularly as the software changes)? do you use both approaches, or prefer just the detailed test cases?21:48
phillwhi kparal one of the advantages of having the test cases split between 'generic', such as those on the iso tracker and the others on the application area is that a test case can be updated for a change in the software (application).21:49
ClassBotThere are 10 minutes remaining in the current session.21:50
phillwso, we do use both approaches.21:50
phillwall test cases should be in the format, Do 'A', exepect 'B'.21:51
phillwas smoke tests are different to normal testing. for the record of this session I'll copy over a chat.21:54
phillwI would add I think the difference your describing is smoke tests versus testcases21:54
phillwsmoke tests do have test cases as well?21:54
phillwmeaning, during smoke testing we'll ask a series of genericish questions, to which the tester(s) can reply and find issues21:55
phillw(21:53:19) balloons: yes, here's some we have right now this week for pulseaudio21:55
phillw(21:53:37) balloons: http://packages.qa.ubuntu.com/qatracker/milestones/251/builds/34454/testcases/1336/results21:55
phillw(21:54:07) balloons: this type of testing is useful, but in s different way than a detailed testcase21:55
ClassBotThere are 5 minutes remaining in the current session.21:55
ClassBotAde asked: Where is the best place to view test cases that need to be written/updated and once complete then how would you submit them?21:55
phillwAde, if you follow https://wiki.ubuntu.com/QATeam/TestcaseUpdates you can contact any of the team at https://launchpad.net/~ubuntu-testcase21:56
phillwor simply ask on the QATeam mailing list21:57
ClassBotkparal asked: in which part of the cycle do you do smoke testing, and in which part do you do application test cases testing? is that separated or simultaneous?21:58
phillwI'm not overly familiar with smoke testing, balloons would be better person to ask about that subject. We'll make an effort to update the Wiki to answer your question (If he hasn't already done it :) )21:59
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2013/01/15/%23ubuntu-classroom.html22:00
phillwThanks everyone. remember, you can always ask questions on #ubuntu-quality22:00
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || No Sessions Currently in Progress
JoseeAntonioRthanks a lot, phillw! stay tuned to the calendar and the blog to see upcoming sessions. see you there!22:01

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!