=== asac_ is now known as asac
[08:42] <davmor2> Morning Everybody :)
=== apachelogger_ is now known as apachelogger
[12:55] <etienne> hi
[12:56] <etienne> i've just upgrade my laptop to intrepid and now gconf-d take mostly of cpu time during the fist ten minutes
[12:57] <etienne> cannot find any thing about it on launchpad
[12:57] <etienne> i wondered if i was alone :)
[14:49] <cr3> schwuk: dude, I noticed that the /builds page no longer works following your latest changes in production
[14:49] <cr3> schwuk: the problem is that the changes to the views are now reporting duplicate builds
[14:49] <davmor2> cgregan: morning Dude :)
[14:50] <cr3> schwuk: I'm currently working on making builds a top-level object which should make this easier to manage
[14:51] <cgregan> davmor2: Hey! How was your weekend?
[14:52] <davmor2> Fine thanks.  How were things your end of the pond?
[15:45] <davmor2> cgregan: should this be one test or one per section it's not a very clear testcase? https://wiki.ubuntu.com/Testing/Cases/UMEdesktop-power
[15:47] <cgregan> davmor2: One table per section
[15:48] <davmor2> cgregan: okay cool they all seemed to be linked so wasn't as clear cut as others :)
[15:53] <cr3> schwuk: dude, stgraber has raised an interesting concern that we might now want to separate the concept of builds and disks
[15:53] <cr3> s/now/not
[15:53] <cr3> stgraber: ideally, I would like to have feedback from other members of the qa team regarding your concern
[15:55] <stgraber> cr3: ok
[15:55] <cr3> it's still early for the portland possy though
[15:56] <stgraber> they should be around soon (30min or so I'd guess)
[15:56] <cr3> stgraber: at very least, since a build is done for all architectures at once, there could be build and build_architecture tables. would that make sense to you?
[15:57] <cr3> stgraber: do you remember the /builds page? didn't you find that more useful than having all builds in a single list such as on iso.qa.u.c?
[16:01] <stgraber> well, that page looks good but won't work in some cases. Let's say we have "Ubuntu alternate 20080817" that's ready for release and "Kubuntu alternate 20080817" that needs to be rebuilt for some reason. We'll then have in the same "build" a valid image and one that's been rebuilt and is then deprecated
[16:01] <stgraber> from an ISO tester point of view, what I want to see is the builds that are candidate for the release and have the other in some archive part of the page
[16:02] <cr3> stgraber: how can I identify a build that is a candidate? how is this marked somewhere somehow?
[16:02] <stgraber> AFAIK you can't
[16:02] <cr3> stgraber: I want this to be identifiable programmatically, having to flag stuff on the certification is not an option :(
[16:03] <stgraber> images are usually many built during the week before a release
[16:03] <stgraber> so usually all images generated during the week are candidate and the latest is the current
[16:03] <stgraber> but we saw with alpha-3 that the latest isn't always the candidate
[16:03] <stgraber> (as we generated some test images that had an higher version number than the candidate one)
[16:04] <stgraber> currently it's done by hand by the release manager as he's the only one to know what images are candidates for the release
[16:05] <cr3> candidates are identified by being the ones appearing under http://cdimage.ubuntu.com/releases/8.10/alpha-4/, right?
[16:07] <cr3> anyways, it is clear that we have a new concept here which is a set of builds
[16:07] <cr3> which we'll call "milestone"
[16:09] <cr3> so, I suggest we trash the concept of build and introduce the concept of milestone which is another animal
[16:09] <stgraber> cr3: no, candidates are in /daily and /daily-live
[16:09] <stgraber> only the release is moved to /alpha-4
[16:10] <stgraber> hehe, I had a concept of milestone in the QA-Tracker so my design wasn't that bad :)
[16:10] <cr3> stgraber: so, currently under http://cdimage.ubuntu.com/daily/, we have the candidates  20080817 and  20080818, right?
[16:11] <stgraber> well a definition of "candidate builds" could be: Daily images generated once the cdimage builders are put to manual
[16:12] <cr3> "put to manual"?
[16:12] <stgraber> so as the current daily are generated automatically and we don't have a schedule release this week, they aren't candidate according to the above definition
[16:12] <stgraber> yes, they usually generate daily builds at 7am or something like that, on release week they don't and the release managers have to manually trigger the CD builder
[16:13] <stgraber> best would be to ask slangasek or pitti, they know that a lot better than I do
[16:14] <cr3> crap, pitti seems to be on vacation for the next couple weeks
[16:15] <slangasek> stgraber: fwiw, sometimes I don't turn off the cronjobs until after I've let them build images that I'm going to use as candidates
[16:18] <cr3> slangasek: so, I would like to discuss what you might consider useful reporting of test results for dailies, candidates and milestones which seem to be the three types of images built, right?
[16:19] <slangasek> hrm, that's not how I would divide them up
[16:19] <slangasek> the difference between a candidate and a milestone is whether it's been released
[16:19] <slangasek> a milestone is always a candidate first...
[16:20] <cr3> slangasek: ok, an image set as a candidate can be a milestone too, but a candidate might not result in a milestone. hence the distinction
[16:20] <slangasek> ok
[16:21] <cr3> slangasek: what kind of separation might you have had in mind?
[16:22] <slangasek> in terms of the testing we do, I would tend to distinguish between "dailies", "alphas", "betas", "RC/release"
[16:22] <slangasek> I'm not sure there's any testing that I need feedback on that would happen /after/ a milestone is released, as opposed to while it's still a candidate
[16:23] <cr3> ok, so you need the concept of a candidate which is the state of an image before is might become an "alpha", "beta" or "rc/release"
[16:24] <davmor2> slangasek: wouldn't the next daily become the next candidate after a milestone?
[16:25] <slangasek> davmor2: the only images that are candidates are the ones published the week of a milestone
[16:26] <davmor2> :) right :)
[16:26] <cr3> slangasek: so, when an image is a candidate, I would like to provide to you a report of the test results for those images to provide somekind of reassurance to flag the candidate as a milestone. would that be useful?
[16:26] <slangasek> cr3: I'm not sure I follow you.  If you mean for there to be a "candidates" bucket into which candidates for all milestones are lumped, then I don't think that's accurate in terms of our test coverage
[16:27] <slangasek> cr3: yes; but to what extent does this overlap with iso.qa.ubuntu.com?
[16:27] <cr3> slangasek: the point is to replace iso.qa.u.c by leveraging the automation fo certification.c.c, which will be made public gradually
[16:28] <cr3> slangasek: you will therefore potentially have test results within the hour of publishing a new image
[16:29] <slangasek> automated testing is not a replacement for the test coverage we currently get via iso.qa.u.c, though
[16:30] <cr3> slangasek: the test suite also supports manual tests, I'm just saying you'll get instant results from automated testing and manual results will trickle in later in a similar faschion as iso.qa.u.c
[16:31] <slangasek> ok; and who will have access to define the test cases, etc?
[16:31] <cr3> slangasek: the community will be able to define these test cases on a structured wiki, testcases.wiki.ubuntu.com me thinks
[16:32] <cr3> slangasek: does that sound acceptable so far?
[16:33] <slangasek> and defining a test case on the wiki will make it available to report results for?
[16:33] <cr3> exactly
[16:34] <slangasek> hmm, that seems a bit odd, but ok
[16:35] <cr3> we have provisions for scaling this model when there will be many test cases, such as separating them topically so that people interested in networking can only be exposed to relevant tests. there are also other ideas but we initially would like to provide the necessary means for the community to contribute testing
[16:36] <slangasek> what about the mechanism for marking an image as a candidate?
[16:36] <slangasek> currently this is done manually through the iso.qa web interface; my ultimate goal is to be able to update a manifest via cdimage.u.c that handles this
[16:36] <cr3> ok, that could be detected automatically from cdimage.u.c
[16:36] <slangasek> no, it can't
[16:37] <slangasek> the information about which images are candidates is *not* stored on cdimage.u.c, it's stored in the RM's wetware and published via iso.qa by hand
[16:37] <cr3> slangasek: my understanding is that if it's got a date, such as http://cdimage.ubuntu.com/daily/20080817/, it's a candidate
[16:37] <slangasek> no
[16:37] <cr3> what is RM's wetware?
[16:37] <slangasek> the candidates are whichever images I say are candidates at that time
[16:37] <slangasek> cr3: "my brain"? :)
[16:37] <cr3> slangasek: we need an xml-rpc interface to your brain then
[16:38] <slangasek> I need to be able to continue doing test rebuilds of images at the same time that an image is posted that's a candidate and is being tested by the community
[16:38] <slangasek> the new one doesn't become a candidate for the milestone except by explicit decision by the release team
[16:38] <cr3> slangasek: ok, so I would be very receptive to storing this information in the manifest then. I already crawl cdimage.u.c, so reading the manifest would not be difficult at this point
[16:38] <slangasek> right; however, currently the manifest doesn't exist
[16:39] <cr3> slangasek: ok, that would be a good first step. how can we make it happen?
[16:39] <slangasek> it would have to wait until I have free time to implement the stuff floating in my brain
[16:40] <slangasek> the manifest needs to be used for several different things
[16:41] <slangasek> if in the short term you can implement the same functionality that's present on iso.qa, namely that the release team and QA folks have access to mark images as candidates and subsequently invalidate them, that would at least not be a regression
[16:43] <davmor2> cgregan: none of this is to be tested? or is it that there is no specific test on any one device? https://wiki.ubuntu.com/Testing/Cases/UMEdesktop-drivers
[16:43] <cr3> slangasek: "invalidate them"? if an image was once a candidate for a particular milestone, I wouldn't want to lose this information. would you mind if we simply made it clear which was the latest candidate for a particular milestone?
[16:44] <slangasek> cr3: "invalidate" meaning "indicating to testers that this is not an active target for testing so they shouldn't waste time downloading it, and removing it from whatever milestone 'overview' we have"
[16:45] <cr3> slangasek: aha! so it might be possible that an image is invalidated before it is superceeded by another image.
[16:46] <slangasek> ... I guess I'm assuming that there'll be an overview page that shows me a summary of test status for all active candidate images for the current milestone, since having to click all over the place to get an overview is totally impractical
[16:46] <slangasek> cr3: well, we normally invalidate candidate images before we have their replacements ready
[16:46] <cr3> slangasek: agreed, regarding the overview report
[16:47] <slangasek> since we want to stop testers from continuing to test them when we already know they have to be replaced
[16:47] <davmor2> slangasek: And we love you for it :)
[16:48] <cr3> slangasek: interesting, thanks for sharing this process, you've given me plenty to think about
[16:48] <slangasek> cr3: sorry :-)
[16:57] <davmor2> cgregan: do you want the table that was at the top of each page knocking on the head now I've put them in place?
[16:57] <cr3> slangasek: by the way, do you happen to use launchpad for candidate notifications? for example, everyone part of some team could be notified when an image is marked as candidate or invalidated?
[16:58] <slangasek> iso.qa uses its own mail system, with per-image-type subscriptions; you could send to a team email address as well as to an individual, I guess, but I don't think this is done currently
[16:58] <slangasek> the per-image-type subscriptions are because testers usually have assigned images
[16:59] <cr3> are they assigned to a particular image type, like kubuntu-i386 or for a particular date as well, kubuntu-i386-20080818
[16:59] <cr3> perhaps having a group in launchpad for each image type might be useful
[17:00] <cr3> stgraber: what do you think ^^^
[17:02] <davmor2> cgregan: or do you want them leaving there for reference just incase things go wrong?
[17:02] <cr3> slangasek: just to make sure, it wouldn't be particularly relevant for you to see all test results for images built on 20080818, right?
[17:03] <slangasek> cr3: right, that's much less relevant than seeing the results for the set of images marked as candidates
[17:03] <slangasek> cr3: and wrt assignments, the assigning is by image type; so kubuntu-i386-alternate or kubuntu-amd64-dvd
[17:04] <cr3> hm, but others might find it useful for dailies to be able to track the number of passes and fails while working up to a milestone
[17:05] <cr3> so I need both the concept of builds and candidates (which include milestones). ok, time to work now
[17:07] <cgregan> davmor2: The one at the top can be removed
[17:07] <davmor2> cgregan: np's I'll either fly through that tonight or in the morning :)
[17:08] <cgregan> davmor2: cool
[17:08] <davmor2> cgregan: all the pages that had them are now updated (crosses fingers).
[17:09] <davmor2> anyway teatime ttyl :)
[17:10] <cgregan> thanks
[17:57] <stgraber> cr3: we planned to have some mailinglist ubuntu-testing@l.u.c which would receive all build notifications
[17:57] <stgraber> cr3: we currently have some people (mainly canonical employees) subscribed to some test cases who receive e-mail notification only for those testcases
[18:00] <cr3> stgraber: I was thinking about something along those lines, but do you think that a launchpad team might be preferable?
[18:05] <stgraber> nope, we are currently trying to get rid of teams like "Kubuntu testers", "Xubuntu testers", ... because most of us test more than one distro
[18:06] <stgraber> best would be testcase subscription as we currently have so someone can say: I will test all Ubuntu alternate i386 and will then be notified by mail when we have that disk available
[18:08] <cr3> hm, testcase subscription, something else to think about
=== thekorn__ is now known as thekorn