| kenvandine | robru, ack'd an publishing | 01:06 |
|---|---|---|
| robru | kenvandine: tanks | 01:09 |
| kenvandine | robru, np | 01:09 |
| robru | bbl | 01:20 |
| robru | Mirv: the fix I did that made uncommitted branches go away broke dual silos | 04:12 |
| robru | Mirv: the watch phase in a dual silo watches the wily build twice | 04:13 |
| robru | Mirv: I'm working on a proper fix but won't be able to land it today | 04:13 |
| robru | Mirv: so for your shift you should double check vivid builds before publishing | 04:13 |
| === chihchun_afk is now known as chihchun | ||
| Mirv | robru: ok, thanks for telling, I'll do manual checks. | 04:53 |
| robru | Mirv: actually it's only 10, I may roll out tonight... just working on unit tests. | 04:54 |
| robru | the trick is that if I rollout I need to stay up at least long enough to confirm nothing exploes. | 04:55 |
| Mirv | heh, ok, we'll manage either way | 04:55 |
| robru | nah I better not roll out so late, I'm getting sleepy | 05:23 |
| robru | Mirv: yeah I'll sleep on this and roll out first thing tomorrow. | 05:24 |
| Mirv | robru: sounds like a good idea | 05:51 |
| robru | Mirv: maybe warn people that it's possible that vivid builds fail but train reports only wily status | 06:04 |
| robru | Only for today though, my fix is good i think | 06:04 |
| robru | Mirv: oh actually, what would you prefer? I could revert the thing that introduced this bug, but then we're back to "merge failed due to uncommitted changes" when merging silos | 06:05 |
| Mirv | robru: yes, I'll look a bit what people are doing and tell them | 06:11 |
| Mirv | robru: I think this is better, we can glance the vivid builds a bit more closely | 06:12 |
| robru | Mirv: OK, yeah the uncommitted changes thing required a lot of futzing in train internals when it arose. | 06:13 |
| Mirv | pstolowski: hey! so they thing is that the branch did already go through QA and was landed on Monday. but since this x86-only webbrowser unit test flakiness was found out, it'd be good to understand if we're truly safe with it or not. the silo 054 is only there to test how the revert would look like, if something odd is spotted. | 07:25 |
| Mirv | so that's why I ran all the unit tests for packages I could think of doing network tests, in addition to the autopilot tests and manual tests done during the landing earlier. | 07:27 |
| pstolowski | Mirv, otp | 07:34 |
| dbarth_ | robru: ack, will review silos this morning | 07:45 |
| robru | dbarth_: thanks! | 07:50 |
| Mirv | sil2100: so, we need to keep extra eye on vivid+dual silos for today. the watch_only currently watches over wily twice, so there's a possibility of missing a vivid build failure. robert will push the fix in the evening, but I thought it's better to live with that than with the "merge failed due to uncommitted changes" problem (a fix for which brought this regression) | 08:21 |
| sil2100 | Mirv: ok, good to know, not sure if there will be many dual landing silos with the wily final freeze | 08:22 |
| === sil2100 changed the topic of #ubuntu-ci-eng to: Train trouble? ping trainguards | CI problems? ping cihelp | Train: http://bit.ly/1hGZsfS | QA Signoffs: http://bit.ly/1qMAKYd | Known Issues: wily final freeze and OTA-7 string/feature freeze now in effect | ||
| Mirv | sil2100: well final beta freeze will be over in <12h | 08:23 |
| Mirv | sil2100: not final freeze, final beta freeze :) | 08:23 |
| Mirv | final freeze Oct 15th | 08:24 |
| sil2100 | Oh, right, indeed ;) | 08:24 |
| === sil2100 changed the topic of #ubuntu-ci-eng to: Train trouble? ping trainguards | CI problems? ping cihelp | Train: http://bit.ly/1hGZsfS | QA Signoffs: http://bit.ly/1qMAKYd | Known Issues: wily final beta freeze and OTA-7 string/feature freeze now in effect | ||
| sil2100 | Now that sounds better | 08:24 |
| dbarth_ | hey o/ can a ci admin help me upload oxide-qt 1.9.3 to silo 21 ? | 08:34 |
| dbarth_ | it is now available in the phablet-team ppa, courtesy of oSoMon | 08:35 |
| pstolowski | Mirv, hey, I'm not sure what to do, as far as arm is concerned all looks fine with latest rc-proposed images. I guess it would be worth checking x86, but that would require unity8 environment on the desktop | 08:35 |
| dbarth_ | we need a full source rebuild, to catch the right media-hub dependency which is in the overlay ppa | 08:35 |
| sil2100 | dbarth_: let help you | 08:41 |
| Mirv | pstolowski: right, it was just an idea if you can come up with anything extra to check or run in a loop, and also as a FYI that something has changed. it'd be nice of course to have a test case to trigger that error message sometimes seen without this bug fix. | 08:41 |
| sil2100 | dbarth_: will need to re-build the source package, this will take a while | 08:42 |
| dbarth_ | sil2100: oh yes, it will take time, but that's really required now | 08:42 |
| dbarth_ | thanks | 08:42 |
| sil2100 | dbarth_: the rebuild too, but I meant that before uploading it to the PPA I need to re-build the source to get it a new version number, which usually takes a while on my PC - but it's in progress | 08:43 |
| Mirv | sil2100: if you didn't notice, the Monday's qtbase landing makes one webbrowser unit test flaky on x86 only. oSoMoN is trying to understand it. the Monday's fix is a further fix to another fix which upstream urged everyone to ship on May (the previous fix is already in OTA-6). https://codereview.qt-project.org/#/c/120738 | 08:50 |
| Mirv | sil2100: I rebuilt all scopes, unity8 etc to have unit test results in addition to AP results, but only that one test on x86 seems to become flaky. if something is noted, there's a revert silo 054 which can be tested with. | 08:51 |
| dbarth_ | sil2100: ah, i thought you could do a ppa copy? is there something we can do next time for the version number? | 08:51 |
| Mirv | sil2100: but the fix could/should also fix random network failures, and at least pstolowski had seen the symtom in scope logs earlier when multiple requests were queued (which this new landed patch fixes), so as long as everything seems perfect on armhf I'd vote to keep the patch in | 08:52 |
| sil2100 | dbarth_: we can't do a PPA copy because since it's a rebuild, we need to have a different version number | 08:53 |
| dbarth_ | hmm ok, so we could have spared the re-upload to phablet; good to know anyway | 08:53 |
| sil2100 | Mirv: I would also opt for keeping the patch released in this case, only considering reverting it if the flaky test seems unfixable for some reason | 08:54 |
| Saviq | psivaa, hey, so I've been trying to get the dependencies fixed... unfortunately there's a lot of alternatives between unity8 and qtmir, and when upgrading just unity8... apt decides it's better to remove qtmir-android and ubuntu-touch instead of upgrading them... | 09:09 |
| psivaa | Saviq: this is with image 191? | 09:11 |
| Saviq | http://pastebin.ubuntu.com/12540313/ | 09:11 |
| Saviq | psivaa, 183 on mako, same thing | 09:11 |
| Saviq | it's basically because qtdeclarative5-qtmir-plugin has qtmir-desktop | qtmir-android | 09:12 |
| psivaa | Saviq: yea, I do not have an idea as to how to proceed with this. testing a devel package on an older image doesn't look very suitable | 09:13 |
| Saviq | and apt apparently decides the first one is better than the second, even if it causes a removal of some packages instead of just upgrades | 09:13 |
| Saviq | psivaa, well, it *should* work, that's why we have the elaborate dependency chains... | 09:13 |
| Saviq | psivaa, one thing that comes to mind | 09:14 |
| Saviq | psivaa, is we could add "ubuntu-touch" to the list of packages for apt-get install, that makes apt choose the right solution... | 09:15 |
| Saviq | and we always want ubuntu-touch don't we... only adverse effect might be its otherwise unnecessary upgrade | 09:15 |
| psivaa | Saviq: ok, we could try adding ubuntu-touch to the apt-get install list, obviously i'm not entirely sure of its effects on other package tests | 09:25 |
| Saviq | psivaa, IIUC that boottest only runs on touch, right? so ubuntu-touch is always there... the only point where it could be a problem is when there's a new ubuntu-touch package | 09:27 |
| psivaa | Saviq: right, it only runs on touch. | 09:28 |
| psivaa | Saviq: and yes, when there is a new ubuntu-touch and there is another package to test, we'd actually be testing an environment of both upgraded | 09:29 |
| alan_g | cihelp I'm not sure why, but we've just had a couple of new jobs added to Mir CI (mir-wily-i386-ci and mir-mediumtests-builder-wily-armhf). They are failing consistently and blocking work - can we disable them please? (We will want to re-enable once we've resolved the problems with those targets.) | 09:38 |
| psivaa | alan_g: aiui, this was a request based on https://code.launchpad.net/~fginther/cupstream2distro-config/add-mir-priority-jobs/+merge/272138 | 09:40 |
| alan_g | psivaa: that request was for them to be added as without their failures affecting the CI PASS. That isn't what I see. | 09:42 |
| psivaa | alan_g: do you have a job that these are affecting | 09:44 |
| psivaa | a link i mean | 09:45 |
| alan_g | https://code.launchpad.net/~raof/mir/fix-ftbfs-against-mesa-11/+merge/272203 | 09:45 |
| alan_g | Although... there's another failure too. | 09:45 |
| alan_g | I guess it could be clearer which build results affect the overall result. | 09:47 |
| psivaa | alan_g: yea, since these are specifically marked nonfatal, it should be the other failure that's causing the MP to fail | 09:48 |
| alan_g | psivaa: thanks for the help. | 09:49 |
| Saviq | psivaa, just noticed one thing though, the apt-get command has a -t ${release-proposed}, ubuntu-touch might not be in proposed... but would apt-get install a package from release pocket to satisfy a dependency of a proposed package when -t ${release}-proposed is used? | 10:03 |
| Saviq | psivaa, FWIW, http://paste.ubuntu.com/12540662/ made the boottest pass for me with silo 38 | 10:10 |
| psivaa | Saviq: looking | 10:11 |
| psivaa | Saviq: curious how this would have installed the actually required packages from -proposed? | 10:12 |
| Saviq | psivaa, you mean without -t wily-proposed? | 10:12 |
| psivaa | Saviq: yes | 10:13 |
| Saviq | psivaa, -t is just prioritizing | 10:13 |
| psivaa | Saviq: ahh ack, so the needs_install.packages were actually installed from wily-proposed | 10:13 |
| Saviq | psivaa, from silo 38 in fact | 10:14 |
| Saviq | psivaa, I only removed it because I was testing with the PPA, it should be fine with it when the packages are in proposed | 10:14 |
| * Saviq tries with something that's actually in proposed | 10:14 | |
| Saviq | only question is if apt-get will be fine with ubuntu-touch not being in proposed when -t wily-proposed is passed | 10:15 |
| Saviq | trainguards, mir from https://requests.ci-train.ubuntu.com/#/ticket/365 is UNAPPROVED queue, is that because of FF? is there a plan to deal with that these days? | 10:22 |
| * Saviq starts thinking we should start a wily overlay.. | 10:23 | |
| Mirv | Saviq: yes, because of that, the final beta freeze should be over later today | 10:24 |
| ogra_ | Saviq, no need to "start" anything ... the existing one could serve more than one release | 10:25 |
| Mirv | Saviq: a given idea for the eventual wily -> wily + 1 delay from slangasek was configuring dual landings to be both to the overlay (although train does not support such at the moment) | 10:28 |
| Mirv | but that's not needed for this ending-very-soon freeze, more like when the final freeze hits in on Oct 15th | 10:28 |
| Mirv | and then when wily + 1 opens the overlay/wily can be copied there | 10:29 |
| Saviq | ogra_, d'oh, of course | 10:30 |
| Saviq | psivaa, so, while I couldn't properly verify because of the beta freeze, IMO http://paste.ubuntu.com/12540898/ should work | 10:49 |
| psivaa | Saviq: thanks for this. I'll make a note of it and give it a try, may be when the freeze is over | 10:51 |
| === chihchun is now known as chihchun_afk | ||
| Saviq | psivaa, btw, can't see any boottest results in excuses, have they been disabled? | 10:57 |
| Saviq | also... shouldn't nm have been blocked in migration when it broke the boottest? :) | 10:58 |
| psivaa | Saviq: yea, i dont see them either, disabling them is not on our side, may be the release team did? | 10:59 |
| psivaa | Saviq: as per how nm slipped in, the boottesting is done after provisioning + and then installing the package. | 10:59 |
| psivaa | so the nm would not have broken boottest | 10:59 |
| Saviq | right | 10:59 |
| psivaa | sil2100: do you know if boottesting is disabled in the britney side? | 11:01 |
| sil2100 | psivaa: it shouldn't, I only saw some propositions but nothing definite | 11:24 |
| psivaa | Saviq: not sure if you've got VPN setup, http://d-jenkins.ubuntu-ci:8080/job/psivaa-wily-boottest-unity8/9/artifact/results/log/*view*/ is with your suggestion to add ubuntu-touch, but on Krilllin though | 11:56 |
| psivaa | Saviq: 'ERROR: timed out waiting for Unity greeter' | 11:57 |
| === psivaa is now known as psivaa-lunch | ||
| Saviq | psivaa-lunch, unity8 and qtmir dependencies are not fixed yet (they are in silo 38) | 12:02 |
| === alan_g is now known as alan_g|lunch | ||
| Saviq | psivaa-lunch, so it only upgraded unity8 without the required qtmir upgrade | 12:02 |
| === psivaa-lunch is now known as psivaa | ||
| psivaa | Saviq: ahh ack, understand. | 13:20 |
| === fginther` is now known as fginther | ||
| camako | trainguards, silo 57 says "Migration: mir is in the UNAPPROVED queue". What does this mean? | 14:02 |
| sil2100 | camako: that's normal, wily is still in final beta freeze so the archive for most packages is frozen | 14:04 |
| camako | sil2100, so when is it going to migrate? | 14:04 |
| sil2100 | camako: any seeded package, as per infinity's announcement, now requires an archive admin approval before migrating - and I think mir is seeded in desktop | 14:04 |
| sil2100 | s/archive admin/release team | 14:06 |
| camako | sil2100, ok I see. Do I need to do anything? | 14:06 |
| sil2100 | camako: not sure, I suppose we could poke the release team to check it and push it through later today, as the beta images should be ready around later today | 14:06 |
| camako | sil2100, and it's dual landing. Would this block the vivid landing too? | 14:07 |
| camako | kgunn ^^ | 14:07 |
| sil2100 | camako: no, the vivid landing is already released | 14:08 |
| sil2100 | It's just the wily part that will be blocked :) | 14:08 |
| camako | thanks sil2100 | 14:08 |
| dbarth | sil2100: is there an apparent reason why oxide-qt failed to buikd on armf? https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-021/+build/7935628 | 14:10 |
| dbarth | there's no build log or indication of why it failed | 14:10 |
| kgunn | camako: cool | 14:11 |
| kgunn | well...not cool, but thanks for telling me :) | 14:12 |
| cjwatson | dbarth: I've retried, that was around the time of a buildd upgrade so could've been related to that | 14:12 |
| cjwatson | (that should have just caused builds to be retried automatically, but ...) | 14:12 |
| dbarth | ah | 14:13 |
| dbarth | cjwatson: so you retired the armhf one specifically already? or should i do something in the interface? | 14:14 |
| dbarth | retried, even... ;) | 14:14 |
| cjwatson | 2015-09-24 11:20:45+0000 [QueryProtocol,client] Scanning kishi19 failed with: Attempted to start a build on a known-bad builder. | 14:14 |
| cjwatson | 2015-09-24 11:20:45+0000 [QueryProtocol,client] Judged builder kishi19 (1 failures) with job PACKAGEBUILD-7935628 (1 failures): False, False | 14:14 |
| cjwatson | 2015-09-24 11:20:45+0000 [QueryProtocol,client] Failing job PACKAGEBUILD-7935628. | 14:14 |
| cjwatson | I think the sysadmin doing the upgrade may not have quite followed the prescribed ordering | 14:14 |
| kgunn | sil2100: yeah, so altho it landed in vivid....since it was dual landed, that does mean we have to wait for any new (dual) landings in those projects right ? | 14:14 |
| dbarth | hmm, welll that explains then | 14:14 |
| cjwatson | dbarth: I retried it in Launchpad. If CI Train thinks the build has failed, you should do a *watch-only* build | 14:15 |
| cjwatson | armhf specifically, yes | 14:15 |
| dbarth | ok, thanks for that | 14:15 |
| cjwatson | Also checked, no other instances of this AFAICS | 14:15 |
| dbarth | i'll do a watch build in a bit once the ppa task is completed | 14:15 |
| cjwatson | Well, you can do it now and let the computer remember it for you :) | 14:16 |
| cjwatson | It'll wait for the build to complete | 14:16 |
| dbarth | even better ! | 14:16 |
| dbarth | thank you sir | 14:16 |
| cjwatson | np | 14:16 |
| rvr | jamesh: Approving silo 19 | 14:17 |
| sil2100 | kgunn: theoretically yes, but if you're certain that these packages are migrated we can force-merge it | 14:20 |
| dbarth | hmm, the watch actually didn't go so well | 14:21 |
| cjwatson | ok, you need a trainguard for that | 14:21 |
| cyphermox | oi | 14:21 |
| sil2100 | dbarth: what's up? | 14:21 |
| dbarth | yup | 14:21 |
| cjwatson | although: I did say "if CI Train thinks the build has failed" | 14:22 |
| dbarth | sil2100: trying to watch build silo 21, now that cjwatson kindly restarted the armhf build which had failed (see log above) | 14:22 |
| cjwatson | AFAICS, it did not think that the build had failed | 14:22 |
| cjwatson | the last run of the build job was nine days ago ... | 14:22 |
| dbarth | uh | 14:22 |
| cjwatson | unless I entirely fail to understand https://ci-train.ubuntu.com/job/ubuntu-landing-021-1-build/build | 14:22 |
| dbarth | it was a manual source upload | 14:23 |
| cjwatson | AIUI, a watch-only build is only necessary if the train thinks the build has failed | 14:23 |
| cjwatson | which is why I had that "if" in my instruction | 14:23 |
| sil2100 | dbarth: strange, there's no sync in the silo even | 14:24 |
| sil2100 | dbarth: I think that's some bug in the train code | 14:26 |
| sil2100 | Ah | 14:27 |
| sil2100 | dbarth: no, wait, it's not a bug | 14:27 |
| sil2100 | dbarth: ok, fixed, now running watch_only, seems to work | 14:28 |
| dbarth | ok, great | 14:28 |
| dbarth | thanks all | 14:29 |
| kgunn | sil2100: what do yo mean "certain packages are migrated" ? | 14:29 |
| kgunn | in the end yes, it'd be nice to have a "force merge" button..knowing that they _would_ migrate after freeze is over | 14:30 |
| kgunn | this way, the merge back happens, and we can continue landing new stuff, and the "new" pkg should just overwrite what's in the queue right ? | 14:30 |
| robru | kgunn: depends what queue you're talking about. that's already possible with Proposed pocket, but if you force merge something in UNAPPROVED, it gets deleted forever | 14:58 |
| kgunn | robru: ok, that i don't know "where it is"....it was mir/u-s-c from silo 57 y;day | 14:59 |
| robru | kgunn: I don't know either but it would have said either 'Migration: foo is in the Proposed pocket' or 'Migration: foo is in the UNAPPROVED queue' | 15:00 |
| robru | kgunn: basically: pockets are safe places for packages to hang out even after the silo is freed. queues aren't. | 15:00 |
| robru | kgunn: just read some scrollback, it was UNAPPROVED | 15:04 |
| robru | which is expected due to the freeze | 15:05 |
| kgunn | robru: assuming they create their beta today, it could at least make it into proposed ? | 15:05 |
| kgunn | at least i would hope that's how this works | 15:05 |
| robru | kgunn: yes once the freeze is lifted it would go into proposed | 15:06 |
| kgunn | tat | 15:07 |
| kgunn | ta even | 15:07 |
| robru | you're welcome | 15:14 |
| === vrruiz_ is now known as rvr | ||
| ogra_ | sil2100, i need to skip today | 15:46 |
| popey | sil2100: https://code.launchpad.net/~zsombi/ubuntu-ui-toolkit/pickerProblems/+merge/272062 we need that in _before_ OTA-7 for notes app - date picker is broken | 15:52 |
| popey | sil2100: I assume zsombi needs to land it, once reviewed. | 15:52 |
| sil2100 | ogra_: ACK | 15:56 |
| sil2100 | popey: do you know if it's targetted for any current landing? | 15:56 |
| sil2100 | bzoltan: ^ ? | 15:56 |
| popey | I don't | 15:56 |
| robru | sil2100: meeting? | 16:00 |
| bzoltan_ | sil2100: popey: that MR is a work in progress | 16:47 |
| sil2100 | bzoltan_: you think it would be ready for OTA-7? | 16:47 |
| bzoltan_ | popey: sil2100: it need to be landed first to the staging | 16:47 |
| bzoltan_ | sil2100: when OTA7 gates close? | 16:47 |
| sil2100 | bzoltan_: the final freeze will be (we hope) in 2 weeks | 16:47 |
| sil2100 | The dates aren't final yet since there was some confusion with those that Pat set | 16:48 |
| bzoltan_ | sil2100: popey: no problem... it will make it to that | 16:48 |
| sil2100 | bzoltan_: would it be possible to get that next week ASAP? :) | 16:48 |
| bzoltan_ | sil2100: i will do a landing next week for sure. So it is very much possible... but first that MR need to be approved and merged tothe staging | 16:49 |
| === karni is now known as karni-away | ||
| === alan_g|lunch is now known as alan_g|EOD | ||
| robru | bfiller: let me know if you need help with silo 48, looks like your packaging is wrong. | 18:06 |
| bfiller | robru: which package is broken? renatu ^^ | 18:07 |
| robru | bfiller: renatu address-book-app has broken build depends line | 18:08 |
| robru | Version should come before arches | 18:08 |
| renatu | let me see | 18:08 |
| renatu | bfiller, this is the Kaleo's branches | 18:09 |
| robru | Kaleo: in address-book-app you've got "foo [arches] (> version)" but it should be "foo (> version) [arches]" | 18:10 |
| Kaleo | kenvandine, ^ | 18:11 |
| Kaleo | robru, thanks | 18:11 |
| robru | Kaleo: lol did ken make that mistake? He should know better... | 18:12 |
| Kaleo | robru, yeah, one beer :) | 18:13 |
| robru | Heh | 18:14 |
| robru | Kaleo: nah, it's from this MP: https://code.launchpad.net/~fboucault/address-book-app/converged_bottom_edge/+merge/270693 blame fboucault | 18:21 |
| robru | bfiller: ^ | 18:22 |
| Kaleo | robru, nah I merged ken's MR that creates the issue into that one :) | 18:22 |
| Kaleo | kenvandine, youa around? | 18:22 |
| robru | Kaleo: oh, you are fboucault, lol | 18:23 |
| Kaleo | robru, funny heh | 18:23 |
| robru | Kaleo: nice to meet you ;-) | 18:23 |
| kenvandine | Kaleo, yeah, what's up? | 18:45 |
| Kaleo | kenvandine, in address-book-app you've got "foo [arches] (> version)" but it should be "foo (> version) [arches]" | 18:45 |
| Kaleo | kenvandine, ;) | 18:45 |
| Kaleo | kenvandine, I fixed it locally in my branch that merges yours | 18:46 |
| kenvandine | ugh | 18:46 |
| kenvandine | ok | 18:46 |
| kenvandine | cool | 18:46 |
| kenvandine | thx :) | 18:46 |
| robru | kenvandine: hey while you're around can you publish a couple silos? https://requests.ci-train.ubuntu.com/#/publishable ;-) | 18:49 |
| kenvandine | robru, i'll take a look | 18:50 |
| robru | kenvandine: thanks | 18:51 |
| robru | blam | 19:03 |
| kenvandine | robru, done ;) | 19:03 |
| camako | sil2100, are the beta images done? Should we talk to the release team to approve silo 57 migration (Mir 0.16)? | 19:39 |
| oSoMoN | trainguards: can the i386 and amd64 builds for vivid be retried in silo 46 please? (https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-046/+build/7936554 and https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-046/+build/7936551) | 20:08 |
| oSoMoN | trainguards: and on a related note, would it be reasonable to give permission to the person who owns a silo to retry a failed build for one given arch? | 20:09 |
| oSoMoN | (reasonable and doable, of course) | 20:10 |
| sil2100 | ogra_: on it | 20:47 |
| sil2100 | I mean | 20:48 |
| sil2100 | ogra_: scratch that | 20:48 |
| sil2100 | oSoMoN: on it (even though he's gone) | 20:48 |
| robru | sil2100: oh i tried already, dias it need it again? | 20:49 |
| sil2100 | They were failed, so probably they fail constantly... | 20:57 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!