[00:23] hi rickspencer3! [00:23] hi rickspencer3 [00:23] hi chrisccoulson, micahg [00:23] my servers seem to have language packs installed [00:23] so, spamaps says there's no lang packs on a server by default [00:23] so, our bug case is an exception [00:23] at least for the servers [00:23] so, my concern is with blocking the firefox upgrade is that we also need to block all of the language packs too [00:24] as the language pack upgrades depend on having the firefox language packs in the archive at the time of upgrade [00:24] else the user loses all of their firefox translations [00:24] hmm, I wonder if update-manager will install the langpack when it becomes available [00:24] i'm not sure what the process of blocking packages is (do we rm them, or chmod 0 them?), but pitti isn't around tomorrow (the only person who can respin language packs) [00:25] micahg, no [00:25] hi SpamapS thanks for joining again [00:25] they will only get installed during the upgrade where the recommends is added [00:25] they definitely need to exist during this upgrade, or the user misses out entirely [00:25] mdeslaur: if the langpacks aren't on the server by default, do we have to worry about it? [00:25] that's what we need to establish, i guess [00:26] There are plenty of reasons to add a language pack to a server.. [00:26] language-pack-es-base: /usr/share/locale-langpack/es/LC_MESSAGES/wget.mo [00:26] micahg: I'm installing a new server now to see [00:26] all my current servers have language packs installed that I don't remember manually installing [00:26] chrisccoulson: we can push the next round of updates as well [00:27] m'eh, if arm wasn't so slow, this wouldn't be such an issue [00:27] is anyone running arm servers? [00:27] can't we just publish the i386 and amd64 builds as soon as they are done, and just skip armel for this build entirely? [00:27] let us assume for a moment that 100% of natty servers have a lang back isntalled [00:27] in such a case, do we *still* need to block anything [00:27] ? [00:28] well, mdeslaur is about to test this [00:28] is this disruptive enough to users to bring down the system, or should we rather wait? [00:28] micahg, what I'm saying is, we no that >0 server users have a lang pack installed [00:28] rickspencer3, if we could publish the new firefox build in the next couple of hours (ie, when the i386 and amd64 builds are finished), then i don't think we need to block anything [00:28] but if we wait the 19 hours for the armel build to finish, then it might be a different story [00:29] that's why i'm wondering if we can just skip waiting for armel entirely (assuming this is just a server issue) [00:29] is there anyway that users will succumb to this issue if they don't specifically use apt-get distupgrade? [00:29] rickspencer3: unattended-upgrades possible [00:30] ok, so who can verify if unattended-upgrades use distupgrade or just upgrade? [00:30] so, the amd64 and i386 should be done in less that 3 hours [00:30] SpamapS, do you know? [00:30] micahg, did you turn off the test suite? [00:30] they should be less than 1.5 hours in that case [00:30] chrisccoulson: ah, no, kees said not to [00:30] oh :( [00:30] that's a pain [00:30] that doubles the build time [00:30] I agree with kees though [00:31] rickspencer3: I don't, not versed in unattended [00:31] I think adding risk on top of of risk is not a good idea atm [00:31] rickspencer3, well, we don't really do anything with the test-suite results yet ;) [00:31] cloud servers seem to not have language packs.. but servers installed via the installer do. [00:31] (which is why i suggested disabling it for this build) [00:31] uh [00:32] chrisccoulson, ok, that's good to know! [00:32] * rickspencer3 adds to list [00:32] ;) [00:32] heh [00:32] so, can we agree on whether we need to wait the 19 hours for the armel build to finish, as that could be the dealbreaker? [00:32] ugh, I forgot about that... [00:33] I can reupload somewhere else [00:33] SpamapS, is anyone using armel server builds? ;) [00:33] chrisccoulson: I'm sure there is *a* user out there somewhere. [00:34] right [00:34] SpamapS, sure. is *a* user enough to block an upgrade for the other 19,999,999 users though? ;) [00:34] but that user would hae to run dist-upgrade ... not see that firefox will be isntsalled, and install nayway [00:35] chrisccoulson, could we not let the update out asap for non-ARM users, and then release it to the ARM users when it's down building? [00:35] rickspencer3, we can do the former (publish amd64 and i386 when they are built) [00:35] the issue is that we don't have a mechanism for publishing the armel builds later on when they are finished [00:36] we would need to do another upload with a new version number [00:36] rickspencer3: ATM no, LP is working on fixing that in the future [00:36] ok ... [00:36] i really think we should just publish i386 and amd64 as soon as they are finished, and we can worry about armel tomorrow [00:37] chrisccoulson, why not just upload the bug fix (done) and then just let evertyhing build and propegate as normal? [00:37] it's not clear to me that this requires that the update be released asap [00:37] rickspencer3, because they won't be published to -security until tomorrow night [00:37] micahg: uhm...what version of language pack is affected? [00:38] i guess it depends on how serious we think the issue is [00:38] ie: my newly installed natty server did a dist-upgrade and nothing happened [00:38] 1:11.04+20110607 [00:38] chrisccoulson, that's what I'm asking? [00:38] mdeslaur: ^^ [00:38] mdeslaur, holds the key! [00:39] I've got language-pack-en-base installed at 1:11.04+20110607 [00:39] and it didn't pull in any firefox packages [00:39] mdeslaur, how did you upgrade? [00:39] chrisccoulson: apt-get dist-upgrade [00:39] oh [00:39] chrisccoulson: is it a Recommends? [00:40] recommends on the firefox-locale package [00:40] mdeslaur, yeah. language-pack-en-base recommends firefox-locale-en, which depends on firefox [00:40] that should have pulled it in :/ [00:40] apt-get dist-upgrade should not be pulling in new recommends [00:40] mdeslaur, is the server pulling in recommends by default? [00:40] ah, right, SpamapS ^^ [00:40] chrisccoulson: yes, but that only happens on install, not on dist-upgrade [00:40] the desktop does, and apt-get dist-upgrade does the right thing there [00:41] yes [00:41] mdeslaur, apt-get distr-upgrade pulls in recommends on upgrade too :) [00:41] servers do pull in recommends by default [00:41] (apt-get upgrade doesn't) [00:41] ok [00:41] i'm confused now then [00:41] on install [00:41] ah [00:41] SpamapS: but on upgrades, not? [00:41] SpamapS, only on install? [00:42] then how do we explain bug 800857? [00:42] Launchpad bug 800857 in firefox "language packs pull in Firefox on upgrade" [High,In progress] https://launchpad.net/bugs/800857 [00:42] "under no circumstances are currently installed packages removed, or packages not already installed retrieved and installed." [00:42] from man apt-get regarding 'upgrade' [00:42] SpamapS: what about dist-upgrade [00:42] SpamapS, yeah, that's true for upgrade. but dist-upgrade is different [00:43] mdeslaur: I would expect dist-upgrade to pull in new things, yes [00:43] but its not as clear in the man page [00:43] unattended upgrade will install new kernels, so I assume it's a dist-upgrade. [00:43] that's a good point [00:43] kees: my dist-upgrade is not pulling in new recommends [00:43] kees: I seem to recall it will only install recommends on "install", but not on "dist-upgrade" [00:43] mdeslaur: correct, but I thought this was a Depend: not a Recommends: [00:43] Oh for recommends.. hmm.. yeah dist-upgrade may not pick those up [00:43] kees: could that be accurate [00:43] it's only a recommends [00:44] dist-upgrade should be picking up recommends though. we're depending on this behaviour for pulling in the new firefox translations on upgrade, and it's been tested extensively on the desktop :/ [00:44] i'm not sure why it didn't work in mdeslaur's case :/ [00:45] I would not expect apt to behave much differently on servers and desktops.. though I haven't verified my expectations. [00:45] which package is it that is triggering the problem? [00:45] (binary package) [00:45] kees: language-pack-en-base IIRC [00:46] kees: do you have a natty server? [00:46] kees, language-pack-xx-base now recommends firefox-locale-xx [00:46] (which depends on firefox) [00:46] heh. I don't have that installed on my server. ;) [00:46] so, the server test should be, install the release version of language-pack-en-base, dist-upgrade and see what's offered [00:46] kees: no? it's installed by default... [00:47] $ apt-cache policy language-pack-en-base | grep Installed [00:47] Installed: (none) [00:47] * kees shrugs [00:47] whoa! apt-get install language-pack-en wants to pull in all of X [00:48] right [00:48] I did a dpkg -P language-pack-en language-pack-en-base [00:48] ok, let me try to install the release, and do a dist-upgrade again [00:49] chrisccoulson: we still have the langpacks in proposed, we can copy them again [00:49] mdeslaur: yeah, apt-get upgrade doesn't pull X. apt-get dist-upgrade does. [00:49] (tested in natty chroot with apt-get install language-pack-en-base=1:11.04+20110421 language-pack-en=1:11.04+20110421) [00:49] that makes sense [00:49] ok, my dist-upgrade now tried to pull in X [00:49] crap [00:50] mdeslaur: so, I guess we need to pull it? [00:50] let me try and trigger unattended-updates [00:50] k [00:52] setting up auto updates, and running /etc/cron.daily/apt manually seems to not do anything.... [00:53] perhaps we should simply use the release twitter account to tell server users not to dist-upgrade until tomorrow? [00:53] micahg, hae you set up an incident report yet? [00:54] rickspencer3, yeah, we could do. who has access to that twitter account? [00:54] rickspencer3: no, not yet [00:54] let me look [00:55] kees: how are you setting it up? I can't seem to get it to run [00:56] mdeslaur: hm, it seems to want dbus... one sec [00:56] in /etc/apt/apt.conf.d/10periodic I added [00:56] APT::Periodic::Verbose "1"; [00:57] and changed RandomSleep to "1" instead of 1800 [00:57] from https://help.ubuntu.com/community/AutomaticSecurityUpdates [00:59] and I'm doing rm /var/lib/apt/periodic/update-stamp /var/lib/apt/periodic/upgrade-stamp /var/lib/apt/periodic/download-upgradeable-stamp /var/lib/apt/periodic/autoclean-stamp [00:59] rickspencer3: did you find the twitter info, I have it [00:59] micahg, I did not [00:59] could you go ahead and use it [01:00] I'll set up the incident report [01:00] mdeslaur: now set Verbose to "3"... [01:00] kees: ok, I managed to get it to run [01:01] yeah, it's doing a dist-upgrade... [01:01] I downgraded libxml2, and ran it, and it only updated libxml2, it didn't try and update the language pack [01:01] kees: it is? [01:01] Allowed origins are: ['o=Ubuntu,a=natty-security'] [01:01] Checking: language-pack-en-base ([""]) [01:01] pkg 'apt-xapian-index' not in allowed origin [01:01] sanity check failed [01:01] * kees scratches his head [01:02] kees: I don't have a 10periodic file, don,t know where you got that from [01:02] mdeslaur: the docs from https://help.ubuntu.com/community/AutomaticSecurityUpdates [01:02] yeah, those don't make sense to me [01:02] but set 1800 to "1", and add Verbose [01:02] ah-ha [01:02] I think it's saying that it is refusing to do the upgrade because the new packages aren't in the -security pockey [01:02] *pocket [01:03] thats how I read it [01:03] kees: they should be though [01:03] well, is that if you have -updates + -security enabled or just -security [01:04] micahg: right, I'm trying the most common auto-update config first (-security only) [01:04] orly? why would you get a package not in -security then? [01:04] micahg: because it would be a new depends [01:05] ah, but so are kernels, aren't they? [01:05] oh! it's failing the sanity check because it's trying to pull in aspell, which isn't in the -security pocket in my case [01:05] so, automatic updates will block the update if -updates is not included in /etc/apt/apt.conf.d/50unattended-upgrades [01:05] mdeslaur: correct. it's checking every single package. [01:06] and (for natty) if I add -updates to /etc/apt/apt.conf.d/50unattended-upgrades it will _still_ fail because some of the stuff from that X stack isn't in -updates _either_. [01:07] yep, I fail the sanity check on aspell because it isn't in -updates [01:07] ok, so we may be lucky with most users [01:07] micahg, https://wiki.ubuntu.com/IncidentReports/2011-06-22-Dist-upgrade-pulls-firefox [01:07] so there is no natty autoupdate configuration that will install this update [01:07] rickspencer3: thanks, will start filling out [01:07] agreed [01:08] thank $DEITY [01:08] ok, so i feel better than i did 15 minutes ago :) [01:08] kees: mdeslaur, so the consensus is not to pull it then? [01:08] micahg: agreed [01:09] micahg: that would be my opinion, yes. put a note on twitter about avoiding dist-upgrade for the next few hours, and that should be enough [01:09] ok, second point of business would then be, do we push i386/amd64 right away or wait for armel (16 hour delay) [01:09] 16 hours.... *whistle* [01:09] I think we should push i386/amd64 right away, IMHO [01:10] *nod* [01:10] my first impulse is to push i386/amd64 ahead of armel too [01:10] so, do we forget about armel/powerpc then? [01:10] those should be ready in the next 1.5 - 2 hours [01:10] we can launch a second rebuild, and push it out again for armel/powerpc [01:11] into -updates? [01:11] we double the update frequency if we do the double-build [01:11] yep [01:11] we can call it "firefox 6" [01:11] :) [01:11] haha [01:11] lol [01:11] mdeslaur, why not 5.0.1? ;) [01:12] k, and a USN for each update then? [01:12] or 5.1? [01:12] micahg: I would do just the first with the regression fix notification. [01:12] micahg: the second is effectively a no-change rebuild [01:13] ok, I can add that in the changelog for the .3 upoad [01:13] *upload [01:13] 2 hours + 17 hours will give us ample time to think about it and decide what to do [01:13] how does this look for the identi.ca note: dist-upgrade on Natty will pull in Firefox in some cases, fix is uploaded, watch this bug: http://pad.lv/800857 [01:13] ? [01:13] rickspencer3: s/Natty/Natty server/ [01:14] mdeslaur, well, if you have desktop and don't have firefox installed, it will do the same thing [01:14] oh, right [01:14] though (even server) [01:14] I should add that [01:14] yeah, that's a good way to put it [01:15] how about: "dist-upgrade on Natty (even server) will incorrectly install Firefox if not already installed. fix uploaded, see http://pad.lv/800857" [01:15] is that 140 char? :P [01:15] how about this [01:15] dist-upgrade on Natty (even Server) will pull in Firefox in some cases. fix is uploaded, upgrade works as expected http://pad.lv/800857 [01:16] dang. we can spend all night massagign this into 140 characters ;) [01:16] s/will pull in Firefox in some cases/may pull in Firefox/ [01:16] shorter [01:16] the builds will be ready by the you've got it in to 140 characters ;) [01:16] **by the time [01:16] hehe [01:17] hehe [01:17] lol [01:17] just doing my job [01:17] heh :-) [01:17] dist-upgrade on Natty (even Server) may install Firefox in some cases. Fix uploaded, upgrade works fine http://pad.lv/800857 [01:17] someone should update the bug also [01:17] i think the buildd's need a turbo button [01:18] s/in some cases// [01:18] but yeah, looks fine [01:18] apt-get dist-upgrade on Natty (even Server) may install Firefox. Fix uploaded, apt-get upgrade works fine http://pad.lv/800857 [01:18] done and done [01:19] I can add a comment to the bug if nobody's currently doing it [01:20] mdeslaur: please do [01:21] done [01:21] I'm not sure this really falls into the category of "very serious problem for a very large number of users" [01:21] but I guess it's best to over communicate rather than under communicate [01:22] * mdeslaur wished launchpad had a "fix typos" button to edit comments after they're posted... [01:23] mdeslaur, i need one of those for the whole internet [01:23] rickspencer3: hehe :) [01:24] micahg: is there anything I can do to help at this point, or is everything under control? [01:25] * kees was just going to ask the same. :) have to head to dinner. I will check back. [01:25] i think we're ok now aren't we? we just need to ensure that someone is around who can copy the new builds to security in a couple of hours, don't we? [01:27] * rickspencer3 eyes kees [01:27] chrisccoulson: these are building in the mozilla PPA? [01:30] mdeslaur, yeah [01:31] sorry, went to look for some food [01:31] :-) [01:31] hmm...yeah, I think an AA is required for that...micahg, is that accurate, or can we unembargo directly from the PPA now? [01:37] mdeslaur: we've been able to unembargo from the PPA for a while (overrides were missing, but I think that's been fixed as well now) [01:38] I just don't know if a partial copy will work, but I'll try it as soon as the binaries are ready [01:38] rickspencer3: is it worth modifying the /topic in -devel [01:38] micahg: ok, cool...we need to update the wiki page, it's out of date [01:38] micahg, I don't believe so, but, as you wish [01:40] right, time for some more thunderbird messaging menu hacking before i call it a night [01:42] ugh, just realized my changelog description sucked [01:42] * micahg adds info to the bug [01:49] I don't have archive admin super-powers, unfortunately. we'll need jdstrand or slangasek as most timezone sane [01:53] kees: we shouldn't need it [01:55] chrisccoulson: BTW, on the commit to trunk to fix the issue, I think it should only break on a new upstream version [01:55] how come? [01:55] a new upstream version of firefox, or the language pack? [01:55] well, if we do packaging changes, that shouldn't affect the integration of the langpacks [01:56] new upstream of firefox [01:56] the breaks is designed to catch both (and it needs to) [01:56] why do we need to worry about packaging changes (i.e. ubuntu1 -> ubuntu2)? [01:57] micahg, we don't. the current versioning is to avoid this lintian error - http://lintian.debian.org/tags/not-binnmuable-all-depends-any.html [01:57] (although, that only triggers on a Depends, but the same issue would affect Breaks too) [01:59] chrisccoulson: right, so why not use ${upstream:Version} as that solves the issue with that tag (I think that's the right way to call it) [02:00] micahg, i'd rather they were tighter than that. what if i add a patch which introduces a new string, for example? [02:02] so, that string would be untranslated, is there any other side effect? [02:02] chrisccoulson, micahg, I need to step away [02:02] I'll have my cell phone with me if you guys need anything [02:02] laters [02:03] micahg, no, the UI would break. the string has to be in every single language pack translated or not [02:03] else it breaks [02:03] rickspencer3: ok, thanks, I'm still working on the incident report [02:03] chrisccoulson: ugh, ok then :) === asac_ is now known as asac [02:22] chrisccoulson: also, if we would have shipped the langpacks separately, we could've released much faster :) [02:22] micahg, that's not a reason to split them. they should be in the same source package, like with everything else in our archive ;) [02:22] they are completely tied to each other, so it makes no sense to have them in separate packages [02:23] and i need a firefox source tree to build them [02:25] yeah, it's one of those catch 22s [02:40] chrisccoulson: I have the globalmenu disappearing thing again, anything I can do to debug? [02:51] * jdstrand is here [02:51] can someone fill me on on what is happening-- I see bug #800857 [02:51] Launchpad bug 800857 in firefox "language packs pull in Firefox on upgrade" [High,In progress] https://launchpad.net/bugs/800857 [02:52] micahg: ^ [02:54] jdstrand: ure [02:54] *sure [02:55] jdstrand: https://wiki.ubuntu.com/IncidentReports/2011-06-22-Dist-upgrade-pulls-firefox, update building in security PPA, will push when i386/amd64 is ready [02:56] reading [02:57] btw, one could just run unattended-upgrades and see... [02:57] well, we ended up doing that in the end and it failed (luckily) [02:58] btw, I am not sure this is as crisis-y as it is being made out (esp with unattended-upgrades not being a factor) [02:58] jdstrand: right, well, that's why we didn't pull it [02:58] imho, it is the right choice to push i386/amd64 [02:58] armel would be at least 12 hours later [02:59] (if not closer to 14) [02:59] so, I've got a watch on the upload checking every 5 minutes for publication, I'll smoke test the builds (and verify dist-upgrade no longer pulls in firefox) and push it out [02:59] then send out a USN [03:00] micahg: sounds fine. there seemed to be a need for an AA-- why is that? to disable the affected pacakges? [03:00] jdstrand: no, I don't think there is a need for an AA [03:02] micahg: so the issue is that firefox-locale-* Depends on firefox, and the new langpacks Recommends firefox-locale-* [03:02] right [03:02] neat [03:02] and no one caught it with a week in proposed [03:02] that was not intuitive [03:02] well, most server users aren't running with -proposed [03:03] right, but I would've expected a kubuntu user to test it [03:03] true [03:08] jdstrand: should I add about my original call for testing to the incident report? [03:09] micahg: yeah, but not in the timeline. maybe as 'Background' in the 'Incident Description' [03:09] jdstrand: k [04:38] k, amd64 is published, i386 just finished building [04:39] I'm pulling amd64 as we speak to do a full test run [04:40] well, I'll skip sun-java [04:41] \o/ woohoo, dist-upgrade seems to be fixed with the new upload [04:46] \o/ [04:49] jdstrand: do I need to test the dist-upgrade on amd64 as well? [04:51] no [04:51] micahg: ^ [04:51] k [04:52] I did that here for just regular users and it went fine [04:52] k, installing the QRT depends [04:52] we can assume the packaging changes went through (and I can verify they are there) [04:54] actually the lanpacks come from the i386 build anyway, so we can assume it is all good since the upgrade worked fine here [04:58] I don't know what all the fuss is about really. I mean server users would get a way better browser experience with firefox than w3m anyway [05:02] * micahg didn't know that firefox had a cli mode :P :) [05:03] new in 5.0 [05:03] hooray for version bumps :) [05:04] er, that should be ncurses mode, but you know what I mean :) [05:14] micahg: amd64 is good to go [05:14] jdstrand: k, running through i386 now, should be done in 10-15 min [05:15] micahg: are you doing the full run? [05:15] jdstrand: yeah I figured why not [05:15] k [05:15] good cal [05:15] call [05:15] we have time before the next publisher run [05:16] micahg: so, at this point, I think I'm gonna had out [05:16] head [05:16] jdstrand: k, thanks, I'm hoping the script lets me copy, if not, I'll get someone else to do it [05:17] micahg: afaic, once you unembargo, you can wait til the morning for the USN [05:17] micahg: if you want [05:17] jdstrand: k, thanks [05:18] micahg: the unembargo script will copy from that ppa. the overrides won't be right, but I can adjust those in the morning [05:18] micahg: you'll have archive admins coming online soon too [05:18] jdstrand: k, yep [05:18] (other ones) [05:18] micahg: good night. sorry this all fell on you, but good job handling it [05:19] jdstrand: no problem, thanks, good night to you too, and thanks for helping with the testing [05:19] sure thing [05:44] micahg: yeah, excellent work on this. [05:44] kees: thanks :) [05:48] kees: and thanks to you and mdeslaur for the server testing [05:49] micahg: sure thing! I'm glad I got to learn more of the auto-update internals. :) [13:36] hmmm, chromebug time [13:42] uh? [13:43] you mean breakpad time? ;) [13:50] heh :) [13:52] lol, i see i've finally taken over nearly all of the i386 builders! [14:04] damn, i thought it was already christmas :P === m_conley_away is now known as m_conley [14:17] m_conley, http://bazaar.launchpad.net/~chrisccoulson/messagingmenu-extension/me-fixes/revision/54 \o/ [14:17] we can turn the indicator on and off now :) [14:18] chrisccoulson: hooray! Thanks! :) [14:18] chrisccoulson: almost ready for that pull request? [14:18] m_conley, yeah, pretty much [14:19] i wanted to move the preferences in to the main pref window as well, but i guess that can wait for now [15:21] finally - https://launchpad.net/~mozillateam/+archive/firefox-stable/ \o/ [15:29] m_conley, https://code.launchpad.net/~chrisccoulson/messagingmenu-extension/me-fixes/+merge/65678 [15:29] there's quite a bit to review ;) [15:41] chrisccoulson: i've had worse. ;) [15:41] heh :) [15:41] chrisccoulson: I like the logging business you did. That's cool. Will remember. [15:41] m_conley, i copied that from the AddonManager code ;) [15:42] it's quite useful [15:42] chrisccoulson: no doubt [15:42] chrisccoulson: no more dump statements for me. :) [15:42] heh [15:42] i think the AddonLogging code uses dump(), but it adds a prefix to the message as well [15:43] and you don't have to worry about log levels [15:44] * m_conley nods [16:19]