| === Ursinha is now known as Ursinha-afk | ||
| === Ursinha-afk is now known as Ursinha | ||
| robru | is there a problem with the bot that syncs packages from ci train silos to the archive? I published some stuff hours ago but it's not in -proposed or even UNAPPROVED | 01:54 |
|---|---|---|
| robru | or rather "not in the upload queue" | 01:55 |
| Mirv | I wonder the same as robru, no sight of qtcreator-plugin-go, nuntium webbrowser-app, unity-weappa-qml anywhere | 05:06 |
| infinity | Mirv: I assume this bot has logs or something? This isn't the place to ask about it. | 06:17 |
| Mirv | infinity: from what we can see the copy to archives should have happened. I'm not sure if didrocks can check anything more, but cjwatson has sometimes digged logs from somewhere deeper on what has happened regarding the copy when it doesn't appear anywhere (like rejection because no binaries found or such) | 06:24 |
| didrocks | Mirv: I'm looking at the logs of the copy machine which was asked to be on snakefruit | 06:25 |
| didrocks | so not you or robru have access to those logs (same for launchpad logs) | 06:25 |
| infinity | Mirv: If the bot doing the copy had an email address anyone checked... | 06:25 |
| Mirv | ok | 06:25 |
| didrocks | infinity: well, we can say the same when the copy to launchpad doesn't work | 06:26 |
| didrocks | infinity: there is no central logs for them | 06:26 |
| didrocks | and I don't keep ranting about it, being quite rare | 06:26 |
| infinity | We can go digging in LP logs, but that's not the right answer when a copy fails and WE TELL YOU ABOUT IT, but you don't read it. | 06:26 |
| didrocks | infinity: exactly the same in that case | 06:26 |
| infinity | didrocks: When copies fail, the uploader is sent an email. | 06:26 |
| didrocks | infinity: not when they fail, I'm telling when they go to limbo | 06:26 |
| infinity | didrocks: (ie: the bot) | 06:26 |
| infinity | didrocks: There's no such place as "limbo". | 06:27 |
| didrocks | which already happened more than once, but rarely, as told | 06:27 |
| didrocks | infinity: ah, definitively there was | 06:27 |
| infinity | didrocks: Either the API call failed (immediate response from the API), or the copy job failed (you get an email with an OOPS) | 06:27 |
| didrocks | Mirv: the rsync progress didn't timeout | 06:27 |
| didrocks | infinity: not with asynchronous copy | 06:27 |
| infinity | didrocks: Yes, really. | 06:27 |
| infinity | didrocks: I do this all the time. | 06:27 |
| infinity | didrocks: And get emails occasionally. | 06:27 |
| didrocks | never saw the OOPSS with the bot's emails | 06:27 |
| didrocks | in that case which happened twice in 2 years and half, so acceptable | 06:28 |
| didrocks | Mirv: ok, next run should copy it now | 06:28 |
| didrocks | Mirv: not sure why --timeout didn't work in that case for rsync to exit | 06:28 |
| Mirv | \o/ | 06:35 |
| cjwatson | didrocks: There's definitely a problem if the bot has a working e-mail address but you aren't getting mails about async copy job failures. As infinity says, normally the person who requested a copy gets a mail if the async copy fails. | 08:17 |
| cjwatson | didrocks: Do I infer correctly from what you said above that the bot has a real mailbox somewhere and occasionally somebody checks it? | 08:18 |
| Laney | 2 | 08:18 |
| didrocks | cjwatson: the CI team did. But I remember twice when apart from the launchpad logs, there was no trace of the copy being rejected (and it wasn't) you had to look at launchpad logs | 08:19 |
| cjwatson | didrocks: And that's what I'm saying, *all* such cases should also have been mailed to the copier | 08:19 |
| didrocks | cjwatson: but in that case, as told, it's the rsync progress being stuck which caused the issue on snakefruit (even with the --timeout, which is weird) | 08:19 |
| cjwatson | didrocks: I remember those cases as well, and I just assumed that there wasn't a real mailbox behind the bot | 08:19 |
| cjwatson | didrocks: Ah, I didn't gather that, OK | 08:19 |
| cjwatson | didrocks: I only had to look at Launchpad logs in those cases because nobody seemed to know how to find the bot's incoming mail | 08:20 |
| didrocks | cjwatson: I remember about the first time, we were at a sprint and we had nothing in the email logs (sitting next to each other at a meeting) | 08:20 |
| cjwatson | Hm, that's obviously escaped my memory | 08:21 |
| didrocks | but as told, those cases were rare (2 times in 2 years and half), so more than acceptable IMHO | 08:21 |
| didrocks | but yeah, the bot has a valid email address. Unfortunately, only the CI team has access to it AFAIK | 08:21 |
| cjwatson | Still, I'd like to chase down the cause if possible ... | 08:21 |
| cjwatson | didrocks: Would it be possible for the train to declare that it's sponsoring the real human requester in its copy requests? | 08:22 |
| cjwatson | if you see what I mean | 08:22 |
| infinity | That would be nice. I'd love to have better blame mechanics than "a bot did it". | 08:22 |
| didrocks | cjwatson: yeah, would be completely possible, (I guess the person pushing the publish button) | 08:22 |
| didrocks | that was discussed some months ago with stgraber, but he told that some changes on LP side was needed | 08:23 |
| cjwatson | He's right, but they're trivial | 08:23 |
| didrocks | (at the core sprint in january/february) | 08:23 |
| infinity | Hrm. Why am I still voiced? | 08:23 |
| cjwatson | didrocks: So, if the bot used sponsored=, then it's at least possible for LP to know whom to mail | 08:23 |
| apw | you have been since release | 08:23 |
| didrocks | cjwatson: however, not sure how much the CI team would be willing to work on it apart if this becomes a top priority, they are supposed to move to the airline in a matter of weeks now | 08:24 |
| didrocks | (and as you probably know, I can work on it on the train, still, even if I'm moving to other duties) | 08:24 |
| infinity | apw: Yeah, I'm obviously just oblivious to little plus signs. | 08:24 |
| cjwatson | didrocks: Well, the airline ought to do the same thing, I assume it's not desperately hard to fix in either place? | 08:24 |
| cjwatson | (possibly incorrectly ...) | 08:24 |
| apw | they have been annyoing me for ages :) as they are yellow for me | 08:24 |
| cjwatson | And the Launchpad fix is the same in either place | 08:25 |
| didrocks | cjwatson: yeah, I guess the longest is just to get the right info from the jenkins env from the job runner | 08:25 |
| didrocks | so if you think the launchpad changes can be done easily, I can work with sil2100 on the train side | 08:26 |
| cjwatson | It's just this method: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/soyuz/model/packagecopyjob.py#L247 | 08:26 |
| cjwatson | I'll prepare a fix this morning | 08:26 |
| infinity | There. | 08:27 |
| didrocks | cjwatson: sponsored would be a launchpad login? | 08:27 |
| cjwatson | You pass a Person object | 08:27 |
| didrocks | ah ok | 08:27 |
| didrocks | good | 08:27 |
| sil2100 | uh oh? | 08:28 |
| bzoltan1 | seb128: ping | 08:28 |
| seb128 | bzoltan1, hey | 08:28 |
| didrocks | cjwatson: of course, it doesn't seem that jenkins is sending the user running the job info in its env… :/ | 08:29 |
| bzoltan1 | seb128: yo man ... I would like to ask your help and time :) | 08:29 |
| didrocks | cjwatson: will try to dig a little bit in that whenever I have some time for that | 08:29 |
| bzoltan1 | seb128: In the https://launchpad.net/ubuntu/utopic/+queue?queue_state=0&queue_text= the qtcreator-plugin-go package needs an eye | 08:29 |
| cjwatson | didrocks: Ah, always a hitch | 08:30 |
| bzoltan1 | seb128: this is the QtCreator plugin to support Go and Go-QML app development. | 08:30 |
| infinity | Ahh, such patience people have these days. | 08:30 |
| cjwatson | didrocks: I'm filing an LP bug for this, so shall I add an ubuntu-ci-services-itself task? | 08:30 |
| cjwatson | Or is it cupstream2distro? | 08:30 |
| didrocks | cjwatson: yeah, and cupstream2distro as well | 08:30 |
| didrocks | one for the airline, one for the train | 08:30 |
| cjwatson | Both? OK. | 08:30 |
| didrocks | thanks :) | 08:30 |
| seb128 | bzoltan1, let me have a look | 08:31 |
| bzoltan1 | seb128: thanks | 08:31 |
| seb128 | yw! | 08:32 |
| cjwatson | didrocks: The other bonus is that the sponsored person will be tracked and show up in the LP UI, so it'll be easy to see who published something | 08:32 |
| cjwatson | (well, in theory, I think that works ...) | 08:32 |
| infinity | Not sure how true that is for copies. | 08:33 |
| infinity | For example, https://launchpad.net/ubuntu/+source/systemd-ui/3-2 was copied by ~adconrad, sponsored for ~logan, but neither of us shows up on the page. | 08:34 |
| cjwatson | ubuntu -> ubuntu/trusty | 08:34 |
| cjwatson | and I think it shows up then | 08:34 |
| cjwatson | err, utopic | 08:34 |
| infinity | Ahh, so it does. | 08:35 |
| infinity | I hate that page. | 08:35 |
| cjwatson | probably in the publishing history too | 08:35 |
| didrocks | cjwatson: yeah, so, there is a plugin (of course)… However, of course, it will say "Didier Roche" for instance in this env var. So, people.find(text=) can find it, not sure about duplication then… | 08:35 |
| cjwatson | so unnecessarily hard to find, but it's there | 08:35 |
| seb128 | those ubuntu ubuntu/<series> difference are confusing | 08:35 |
| infinity | Or, rather, I hate that there are two pages with differing but similar info, and one has a crap layout. :P | 08:35 |
| seb128 | it's annoying that the serie-specific one doesn't have the diffs (especially that it's the one in the -changes emails) | 08:36 |
| didrocks | I guess an additional field in case there are multiple matches (and so, we fail by default)… | 08:36 |
| cjwatson | mm, I guess this requires Jenkins hacking to pass through the Launchpad user name then | 08:37 |
| cjwatson | Searching on a person's full name isn't sensible :) | 08:37 |
| cjwatson | didrocks: https://bugs.launchpad.net/launchpad/+bug/1319704 | 08:38 |
| didrocks | cjwatson: thanks, will see if sil2100 wants to be tutored on that one | 08:38 |
| didrocks | cjwatson: yeah, jenkins hacking… we can add a mandatory field as a first step, but needs people to be aware of it | 08:38 |
| infinity | Well, all the Jenkins instances have been switch to SSO, right? So it should, in theory, not be crazy hard to determine the LP ID of the requestor and pass it through. | 08:39 |
| infinity | (In theory... I realise this is Jenkins, and nothing about that screams "easy") | 08:39 |
| cjwatson | didrocks: Yeah, certainly shouldn't need an extra user-visible field since it (in theory) has that information | 08:39 |
| didrocks | infinity: right, the question is how much time we should investigate into this knowing that: 1. I'm transitioning to other duties 2. the CI team is supposed to deliver something equivalent to CI Train in less than a month | 08:40 |
| didrocks | so not sure if the jenkins part needs to be heavily invested in | 08:41 |
| infinity | Yeah, that's fair. | 08:42 |
| cjwatson | If the result ends up being that we transition to the airline before this is fixed and wontfix it in the train, I don't mind | 08:43 |
| cjwatson | Presumably it's a trade-off against potential time spent investigating confusion before then | 08:43 |
| cjwatson | didrocks: I think your estimate of number of times this has come up is a bit low, FWIW - there have been a few times when you weren't around | 08:44 |
| didrocks | cjwatson: it would have been good to have it raised a little bit more at the CI (even daily-release) UDS sessions so that I prioritized it or on the ML | 08:45 |
| didrocks | anyway, I'll see what I can do | 08:46 |
| cjwatson | Well, can't prioritise *everything* :) | 08:46 |
| didrocks | the real issue is only this jenkins -> cu2d part | 08:46 |
| cjwatson | Thanks | 08:46 |
| sil2100 | cjwatson, infinity: hi guys! Can I just quickly ask you for something? We have released a new unity8 yesterday and the faux package needs bumping for it to move out of -proposed :) | 08:56 |
| cjwatson | sil2100: Laney said in #ubuntu-ci-eng that he'd already done so | 08:57 |
| sil2100 | Ah :) Thanks Laney! /me was in a hangout | 08:57 |
| didrocks | cjwatson: ok, I've done the support for it in cu2d. (was able to find in some way through jenkins what should be the launchpad username). However, will need testing and sil2100 to convince webops to install that plugin | 09:11 |
| sil2100 | didrocks: the RT is filled, now I poke webops | 09:11 |
| didrocks | in addition, there will be an additional field, like if I sponsor for sil2100 for instance | 09:12 |
| didrocks | so that people can collect names, will make MOTU approval and tracking easier | 09:12 |
| cjwatson | didrocks: OK, but you can only pass one sponsored person to Launchpad | 09:12 |
| didrocks | right | 09:12 |
| didrocks | that sounds good to me, one packager is supposed to look at the silo | 09:13 |
| cjwatson | So I assume you're resolving to the most relevant person | 09:13 |
| cjwatson | I've filed an MP for the Launchpad part, anyhow | 09:13 |
| didrocks | yeah | 09:14 |
| didrocks | sil2100: changes in trunk | 09:20 |
| didrocks | don't deploy before the plugin is there | 09:20 |
| sil2100 | didrocks: \o/ thanks! No answer from webops, they seem to be busy with something else right now | 09:20 |
| didrocks | (it will need the jenkins template to be deployed as well for the record) | 09:20 |
| didrocks | sil2100: well, actually, we can deploy, but the field will then be mandatory | 09:20 |
| cjwatson | sil2100: There's no EMEA webops vanguard coverage this week | 09:21 |
| mlankhorst | can any archive admin remove xorg-lts-transitional from utopic? | 11:31 |
| infinity | mlankhorst: Sure. | 11:37 |
| Saviq | hey all, is there anything we could do in unity8 to lift the need for manual approval of each unity8 migration? | 11:41 |
| cjwatson | Saviq: I've been thinking about it, but it's tricky | 11:46 |
| cjwatson | (other than "port unity8 to all architectures", which is mostly blocked on platform-api ...) | 11:47 |
| cjwatson | Saviq: I do still consider this my problem to fix though, since our current solution is crap | 11:48 |
| cjwatson | I might be able to come up with something that at least doesn't put the crap quite so much on the surface | 11:48 |
| Saviq | cjwatson, I thought it was possible to have a package only on a certain set of architectures, would that not be good enough for now? | 11:51 |
| cjwatson | unity8 *is* only on a certain set of architectures, but the problem is that indicator-network depends on it and indicator-network has a festering mess of reverse-dependencies which are hard to disentangle | 11:52 |
| cjwatson | The alternative to this is a hideous pile of architecture limitations everywhere | 11:53 |
| cjwatson | (manually) | 11:53 |
| Saviq | heh :| | 11:53 |
| cjwatson | So we pretend that unity8 is actually available on all architectures to make the problem go away at least somewhat gracefully | 11:53 |
| cjwatson | And this is the result | 11:53 |
| Saviq | cjwatson, ok, please let me know if we can do anything to help | 11:53 |
| xnox | make src:unity8 build an empty unity8 package on unsupported arches?! | 11:57 |
| cjwatson | xnox: I think I prefer fixing proposed-migration. | 12:01 |
| cjwatson | It actually shouldn't be *too* hard, as it has the existing Packages files as input | 12:02 |
| === psivaa is now known as psivaa-afk | ||
| cjwatson | Saviq: OK, I believe it's fixed now - shouldn't require manual bumps on unity8 uploads any more | 12:18 |
| cjwatson | Just took me a while to figure out which angle to approach it from, but easy enough in the end | 12:18 |
| Saviq | cjwatson, awesome, thanks! | 12:21 |
| infinity | cjwatson: Do we not want to use that substitution for all the entries in FauxPackages? | 12:22 |
| cjwatson | Probably, it was just less urgent for the others | 12:23 |
| cjwatson | Be my guest if you're feeling keen :) | 12:23 |
| infinity | Oh, I guess some don't appear to really have the issue. | 12:23 |
| cjwatson | Indeed | 12:23 |
| infinity | Like wine seems to not care about the version. | 12:23 |
| infinity | Or maybe it only doesn't care about the 1.4 version. :P | 12:24 |
| * infinity shrugs. | 12:24 | |
| infinity | I'll care if it causes an issue later. | 12:24 |
| infinity | I'm sort of a zombie right now. | 12:24 |
| === jhodapp|afk is now known as jhodapp | ||
| bzoltan1 | seb128: do you have any news about the qtcreator-plugin-go? | 12:53 |
| seb128 | bzoltan1, no, still on my list | 12:54 |
| bzoltan1 | seb128: OK | 12:55 |
| === doko_ is now known as doko | ||
| === psivaa-afk is now known as psivaa | ||
| === pete-woods is now known as pete-woods-lunch | ||
| === pete-woods-lunch is now known as pete-woods | ||
| === jhodapp is now known as jhodapp|lunch | ||
| === beidl_ is now known as beidl | ||
| robru | hey guys, another poke about that snakefruit rsync thing. seems to be off again, can somebody take a look at it? | 18:20 |
| === jhodapp|lunch is now known as jhodapp | ||
| bdmurray | slangasek: Could you merge this? https://code.launchpad.net/~brian-murray/ubuntu-archive-tools/set-phased-update-percentage/+merge/219444 | 19:37 |
| cjwatson | robru: it *seems* current; incoming/ has one file (packagelist_rsync_landing-014-utopic) timestamped less than a minute ago | 20:06 |
| cjwatson | I can only guess here though | 20:06 |
| infinity | cjwatson: I assumed impatience, based on "New source: golang-udm" appearing right after he asked. :) | 20:06 |
| infinity | Oh, except that's not a sync. | 20:07 |
| cjwatson | he asked about it on #ubuntu-ci-eng 15 minutes later too, so I'm guessing not that one | 20:08 |
| cjwatson | I don't know how often this is supposed to run, which syncs are expected to arrive and are missing, nor where the logs are, though, so flying pretty blind :) | 20:08 |
| cjwatson | didrocks just said this morning that rsync was getting stuck rather than honouring its --timeout, so I'm guessing that he fixed it by killing a stale rsync process | 20:11 |
| cjwatson | but there are no such on snakefruit right now - so hopefully it timed out by itself and is resolved now | 20:11 |
| robru | cjwatson, nah -- http://people.canonical.com/~rbpark/citrain/ I've got 9 packages in "no known space and time", published as early as 2 hours ago. typically I see these things get into -proposed within minutes of publishing, it's very strange and bad for it to be this stuck. something is definitely wrong | 20:14 |
| cjwatson | robru: I think it needs somebody who knows more about how our end of it is supposed to work than I do, I'm afraid | 20:17 |
| robru | cjwatson, do you have any idea who? this issue is extremely opaque to me | 20:18 |
| cjwatson | robru: didrocks | 20:19 |
| cjwatson | (I'm afraid) | 20:19 |
| cjwatson | robru: FWIW there's no mention of address-book-app in the relevant LP logs, so it's not that the copy failed, at least not in that case | 20:19 |
| cjwatson | (or rather no relevant mention - there are some MPs that show up in the same logs) | 20:20 |
| robru | cjwatson, yeah, that's what I'm trying to get at. it's not that the copy failed... it's that whatever component is responsible for the copying isn't doing them | 20:20 |
| cjwatson | hmm | 20:20 |
| cjwatson | ./incoming/packagelist_rsync_landing-001-utopic:ci-train-ppa-service/landing-001 Release utopic Proposed utopic address-book-app 0.2+14.10.20140514-0ubuntu1 0.2+14.10.20140512-0ubuntu1 mathieu-tl | 20:20 |
| robru | cjwatson, that's one of the ones I'm expecting to see uploaded... | 20:21 |
| cjwatson | oh, it's a regression from didrocks' patch earlier today | 20:21 |
| cjwatson | 2014-05-15 20:21:23,967 INFO Found packagelist_rsync_landing-017-utopic | 20:21 |
| cjwatson | Traceback (most recent call last): | 20:21 |
| cjwatson | File "/home/ubuntu-archive/cu2d/cupstream2distro//copy2distro", line 51, in <module> | 20:21 |
| cjwatson | sponsored = launchpadmanager.get_person(sponsored_name) | 20:21 |
| cjwatson | AttributeError: 'module' object has no attribute 'get_person' | 20:21 |
| slangasek | bdmurray: merged | 20:21 |
| cjwatson | looks like it's fixed in cupstream2distro trunk, so I've pulled | 20:22 |
| robru | cjwatson, ugh, any way to just revert that until didrocks gets back to fix it? literally all citrain publishing are blocked on this | 20:22 |
| cjwatson | like I say, I pulled what should've been the fix | 20:23 |
| cjwatson | give it a few minutes for the next run | 20:23 |
| robru | cjwatson, ok thanks | 20:23 |
| cjwatson | (my son demands attention) | 20:23 |
| cjwatson | robru: looks good now | 20:26 |
| robru | cjwatson, thanks | 20:26 |
| === beidl_ is now known as beidl | ||
| === Ursinha is now known as Ursinha-afk | ||
| === Ursinha-afk is now known as Ursinha | ||
| === jhodapp is now known as jhodapp|afk | ||
| bdmurray | arges: Does Tuesday work for you for SRU duty? | 23:03 |
| bdmurray | arges: if so maybe you could update https://wiki.ubuntu.com/StableReleaseUpdates#Publishing | 23:03 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!