[08:08] <jamespage> jibel, gema: OK my seed update yesterday broken the server tomcat ISO tests...
[08:09] <jamespage> it did not behave exactly how I thought it was going to
[08:09] <gema> jamespage: do you mean that there is a server failure due to your change today?
[08:10] <jamespage> gema, yep
[08:10] <gema> jamespage: can you point me to the failing job?
[08:10] <jamespage> gema, https://jenkins.qa.ubuntu.com/view/Quantal/view/ISO%20Testing%20Dashboard/job/quantal-server-amd64_tomcat-server/45/
[08:10] <gema> https://jenkins.qa.ubuntu.com/view/Quantal/view/ISO%20Testing%20Dashboard/job/quantal-server-amd64_tomcat-server/ <- this?
[08:10] <jamespage> snap
[08:10] <gema> haha, got it, thanks
[08:11] <gema> jamespage: do you want us to raise a bug?
[08:12] <gema> jamespage: I'd like to attach a bug to the failure , for consistency
[08:12] <jamespage> gema, I need to check with cjwatson - but I think its blocked on bug 1009579
[08:13] <gema> jamespage: so the failure is also due to that problem?
[08:14] <jamespage> gema, like I said I'm not sure
[08:14] <jamespage> I believed that the seed change should just work and the MIR could follow
[08:14] <jamespage> but that does not appear to be the case
[08:14] <gema> jamespage: ok, who is on the case?
[08:15] <jamespage> gema, I am
[08:15] <gema> jamespage: ok, keep us posted
[08:15] <gema> jamespage: jibel is on holidays, babyface_ and I are watching jenkins
[08:15] <gema> jamespage: I will attach that bug to the job until we know more, that way I can track progress
[08:16] <gema> jamespage: and everyone knows we are working on it
[08:16] <jamespage> gema, sure
[08:16] <jamespage> I can always revert the seed change if need be and restore tomcat6
[08:17] <gema> jamespage: ok, let me know if you do that, so that we can rerun
[08:17] <gema> jamespage: we aim at fixing these problems within the hour, the only caveat being that it is not always easy
[08:18] <gema> jamespage: if reverting the change will give us a clue on whether the problem is on the test or on tomcat, I'd go for it
[08:18] <jamespage> gema, the issue is that the tomcat7 packages are not on the ISO - so my seed change broke it
[08:18] <jamespage> the test is giving the correct result - tomcat is not installed
[08:19] <gema> jamespage: and tomcat should be on the iso, right?
[08:19] <jamespage> gema, yep!
[08:19] <gema> jamespage: ok, I am getting there, hold on x) does bug 1009579 mention that tomcat is not there?
[08:20] <gema> jamespage: because that didn't strike me as what it says when I read it
[08:20] <jamespage> gema, no but the fact that the tomcat7 packages are not yet in main is *I think* the reason they did not get on the ISO
[08:21] <jamespage> if that is the case completing that MIR will fix the issue
[08:21] <jamespage> but that might take longer than one hour
[08:52] <gema> jamespage: indeed, I have added a comment to the bug
[08:53] <gema> jamespage: we seem to have lost all the care for the current iso being sound
[08:53] <gema> jamespage: not because of this, but there are all sorts of problems ongoing
[08:53] <jamespage> gema, ah - I see
[08:53] <gema> jamespage: I need to talk to pgraner about it :)
[08:54] <gema> jamespage: wasn't there a way to leave the old tomcat there until the new one is ready?
[08:54] <jamespage> gema, yeah - not update the seed I think
[08:54] <jamespage> but like I said this was not the behaviour I expected
[08:55] <gema> jamespage: ok
[09:11] <jamespage> gema, choices
[09:11] <jamespage> 1) I can back the seed change out and we can respin (but I would consider that overkill - it breaks one small part of server only)
[09:12] <jamespage> 2) I can wait until jdstrand comes online and see if he can review today - in which case it could be OK tomorrow/or with a respin once promoted
[09:23] <gema> jamespage: let's go for option 2 and if it cannot happen today or tomorrow, we go for option 1?
[09:24] <jamespage> gema, sounds OK to me
[09:24] <gema> jamespage: ack
[14:08] <balloons> hmm
[19:10] <phillw> balloons: ping
[19:10] <balloons> hey phillw
[19:11] <phillw> balloons: do I have permission to 'up' bug 1007394 to critical?
[19:11] <phillw> as complete show stopper, it does fall in that classification
[19:14] <phillw> If a 2nd person is required to confirm Lars' work, I'll try to grab an image overnight, but my broadband is suffering from intermittent drop outs.
[19:16] <balloons> phillw, was this in the release notes for alpha 2?
[19:16] <phillw> we did not have an alpha2 for ppc-alternate because of the complete failure.
[19:17] <balloons> right, but I wonder if it made the release notes
[19:17] <balloons> at any rate, I was going to bring it up this week with the release team
[19:17] <balloons> but of course, we should get it confirmed first
[19:17] <phillw> I doubt it would have done. I think only those that were released made it. let me go check.
[19:18] <balloons> once confirmed, it's definitely a  showstopper
[19:19] <phillw> No, it was not included.
[19:20] <phillw> I'll ask on the ppc forum to see if someone can confirm. I'll also try downloading an iso myself and un packing the old ppc that lives here.
[19:21] <stgraber> commented
[19:21] <stgraber> as it stands we have no clue whether it's even an mdadm issue, hopefully the lsof or ls output will help clarify that
[19:22] <balloons> stgraber, thanks :-)
[19:22] <phillw> stgraber: thanks
[19:22] <stgraber> it looks like it's just stuck in infinite I/O wait which can be caused by a lot of different things, usually hardware/kernel related
[19:22] <phillw> I'm sure lars is more than happy to provide anything that help you that he is able to.
[19:32] <phillw> although of course, it may be faster for me to write the entire iso down in binary format with the way my link is :(
[19:36] <balloons> no zsync?
[19:36] <balloons> I don't have ppc hardware or I'd help
[19:46] <phillw> balloons: the problem is that the adsl link keeps resetting, this is annoying normally but pretty fatal for downloads of isos !
[19:46] <balloons> wget -c ;-)
[19:46] <balloons> but yes
[19:46] <balloons> and I wish you luck
[19:46] <phillw> I wget's it on to the SII server.
[19:48] <phillw> it is faster to download from there than the ubuntu servers. The *really* odd thing is that the server gets a really fast link into the iso backbone of ~ 2MB/s, and can then furnish more quickly to me.
[19:49] <phillw> I'm using my 3G device now. It hammers my data allowance, but from memory my 28 days is about up (I have PAYG) so it will need a top up come weekend anyway and I'm not going away this weekend.