| robru | Train going down in 15! | 00:44 |
|---|---|---|
| robru | it's alive! | 02:05 |
| === robru changed the topic of #ubuntu-ci-eng to: Train trouble? ping trainguards | CI problems? Use JenkaaS: http://bit.ly/jenkins-docs | Train: http://bit.ly/1hGZsfS | QA Signoffs: http://bit.ly/1qMAKYd | Known issues: diffs missing, robru is regenerating | ||
| === chihchun_afk is now known as chihchun | ||
| === robru changed the topic of #ubuntu-ci-eng to: Train trouble? ping trainguards | CI problems? Use JenkaaS: http://bit.ly/jenkins-docs | Train: http://bit.ly/1hGZsfS | QA Signoffs: http://bit.ly/1qMAKYd | Known issues: if your ticket says "diffs mising", that should fix itself next build | ||
| robru | Trevinho: congrats on being the only tester and the first person to do a build in production ;-) | 03:33 |
| Trevinho | robru: yeah, I noticed that... Pretty nice :-) | 03:35 |
| Trevinho | robru: I noticed just one small thing: I just added to my silo a new MP which affects a new compnent that wasn't listed before... Then when I did the build thing it wasn't listed | 03:36 |
| === chihchun is now known as chihchun_afk | ||
| robru | Trevinho: yeah, that's expected. that list is based off the sources list which wouldn't be updated until after the build. the problem is that identifying source names from MPs is quite slow so it's really not feasible to detect that right away | 03:37 |
| Trevinho | robru: like: the silo had unity and compiz MPs. I added a new BAMF MP and the did the build thing, But bamf checkbox wwasn't there in the build page | 03:37 |
| robru | Trevinho: if you add bamf to the list of sources by hand, it'd show that option. or run a diff | 03:37 |
| Trevinho | robru: I suppose yes | 03:38 |
| Trevinho | robru: not a big deal though... | 03:38 |
| Trevinho | robru: very nice stuff! | 03:38 |
| robru | Trevinho: thanks! I'm noticing other small issues, I'll have to push some fixes ;-) | 03:38 |
| Trevinho | robru: indeed there will be somehting but overall the experiecne is nice | 03:39 |
| robru | Trevinho: thanks! | 03:40 |
| Trevinho | :) | 03:40 |
| Trevinho | robru: I actually was wondering that things had been switched because of the new favicon :) | 03:41 |
| robru | Trevinho: yes, as we phase out the "train" we had to do away with train favicon, there's also a new error image but hopefully you don't see that too soon;-) | 03:42 |
| Trevinho | robru: i actually saw it :) | 03:44 |
| Trevinho | robru: not sure how it happened but while loading the page... | 03:44 |
| robru | Hmmm | 03:44 |
| Trevinho | err, when opnenig the ticket from the build page | 03:44 |
| === chihchun_afk is now known as chihchun | ||
| robru | Trevinho: that's odd, when did www-data start contributing to this project? http://bazaar.launchpad.net/~ci-train-bot/compiz/compiz-ubuntu-yakkety-landing-012/revision/4050#debian/changelog | 04:30 |
| robru | (le sigh) | 04:30 |
| === chihchun is now known as chihchun_afk | ||
| robru | Trevinho: this is happening because smspillaz' bzr is misconfigured: https://code.launchpad.net/~smspillaz/compiz/compiz.animationaddon-returns/+merge/295497 (he somehow doesn't have angle brackets around his email in the commit logs, so parsing is failing) | 04:37 |
| robru | Trevinho: I've pushed an experimental fix, try again in 20 minutes and see if that helps (should at least use the raw name rather than www-data) | 04:41 |
| === chihchun_afk is now known as chihchun | ||
| tvoss | robru, you still around? | 06:44 |
| robru | tvoss: hi | 06:44 |
| tvoss | robru, hey there :) I'm trying to build https://requests.ci-train.ubuntu.com/log/1449/build/ but fails due to lack of permissions. I'm logged in to the train, though | 06:45 |
| robru | Wat | 06:47 |
| robru | That's not | 06:47 |
| robru | No | 06:47 |
| tvoss | robru, ? :) | 06:48 |
| robru | tvoss: is your email address set in launchpad? That is some kind of lp error, nothing to do with permissions as far as bileto is concerned | 06:49 |
| tvoss | robru, yup, set | 06:50 |
| robru | tvoss: no it isn't? https://launchpad.net/~thomas-voss | 06:51 |
| robru | tvoss https://www.irccloud.com/pastebin/CX93BshO/ | 06:53 |
| tvoss | robru, well, it's set but hidden :) | 06:53 |
| tvoss | robru, try again | 06:54 |
| robru | tvoss: can you make it public? I'm not sure how that would have worked before. it needs to know that so that it knows what to put in the changelog message | 06:54 |
| tvoss | robru, done | 06:54 |
| robru | tvoss: ok, looks good now. try the job again (I already built it but if you want your name in the changelog build it again0 | 06:54 |
| tvoss | robru, ack | 06:56 |
| robru | tvoss: http://bazaar.launchpad.net/~phablet-team/location-service/trunk/view/head:/debian/changelog#L10 A HA! Apparently old code fell back on default email address since your address was hidden all this time. Let me know if you want me to restore that or if you prefer having your address in changelogs now ;-) | 07:01 |
| tvoss | robru, I'm fine with my mail address being in the changelog | 07:02 |
| robru | great, less work for me, yippee! | 07:02 |
| marcustomlinson | trainguards: Something odd about silo 31, it's been "running" autopkgtest for 24 hours now | 07:07 |
| robru | marcustomlinson: hmm indeed something looks goofy there. please ping pitti to dig into that | 07:08 |
| robru | no mention of landing-031 at http://autopkgtest.ubuntu.com/running.shtml | 07:09 |
| marcustomlinson | robru: will do thanks | 07:10 |
| robru | you're welcome | 07:12 |
| robru | OK, getting sleepy... | 07:16 |
| tvoss | trainguards, trying to kick a rebuild for https://requests.ci-train.ubuntu.com/#/ticket/1449 never ends in the silo | 09:11 |
| tvoss | robru, ^ :) | 09:22 |
| pstolowski | robru, hey, getting python exception in the silo - http://pastebin.ubuntu.com/16886214/ | 09:30 |
| sil2100 | eh | 09:37 |
| sil2100 | Ok, I got an exception while trying to rebuild a silo as well | 09:41 |
| Mirv | hmm | 09:46 |
| pete-woods | so it seems that building has changed hugely in citrain | 10:50 |
| pete-woods | I've added a new package to my silo | 10:51 |
| pete-woods | but it's not available in the list of checkboxes | 10:51 |
| tvoss | Mirv, sil2100 any more insight into the build failures? | 10:51 |
| pete-woods | manually added to the source package list helped.. | 10:51 |
| pete-woods | great, I seem to no longer have permission to build the silo | 10:51 |
| pete-woods | trainguards: http://paste.ubuntu.com/16887307/ | 10:52 |
| pete-woods | am I the first to start seeing this one? | 10:53 |
| pete-woods | oh | 10:57 |
| pete-woods | I see it's a lot of people | 10:57 |
| pete-woods | silly me didn't read bak | 10:57 |
| pete-woods | back | 10:57 |
| Mirv | tvoss: the best guess is the big rollout as announced by robert today in an e-mail | 11:09 |
| Mirv | robru noticed a problem regarding the "0replaceme" functionality, but this permission problem is apparently another | 11:10 |
| rvr | Mirv: Did you tell me that silo 9 was going to be reworked or did I imagine it? | 11:22 |
| Mirv | rvr: yes I did, and it's built, I was under impression mzanetti was still going to think a bit about it though. it did fix the problem to at least some extent, he was wondering why needs two app restarts instead of just one when the problem happens. | 11:23 |
| mzanetti | yeah, didn't make much progress, but I haven't managed to see a .lock file since | 11:25 |
| mzanetti | rvr, when you test it, keep on running this: | 11:25 |
| mzanetti | rvr, find ~ -name *.lock | 11:25 |
| mzanetti | rvr, before restarting the app. if you find a .lock file in e.g. ~/config/com.ubuntu.music/ and then on the next start the music app works fine, we're good | 11:26 |
| rvr | mzanetti: What I did was to create the lock file manually to see whether music app would start or not, and it did not. | 11:27 |
| mzanetti | rvr, did you try a second restart? | 11:27 |
| rvr | Yes | 11:28 |
| mzanetti | if I created the lock file with touch, that silo would get over it on the second run only - which is what I don't understand properly yet | 11:28 |
| rvr | Hmm | 11:29 |
| alf_ | trainguards: Hi! Some time ago there were discussions about supporting MPs for git branches in ci-train requests. Are there any news about this? | 11:44 |
| Mirv | alf_: robert will answer you in a few hours but I believe that's still some time off even though this big rollout of features brings it closer | 11:56 |
| alf_ | Mirv: thanks | 11:56 |
| === seb128_ is now known as seb128 | ||
| === _salem is now known as salem_ | ||
| === chihchun is now known as chihchun_afk | ||
| alf_ | trainguards: Hi! I am not able to rebuild packages in ci-train PPA. Only way I have found to rebuild is to abandon request and create a new one. Any idea what's going, or what I am doing wrong? See https://requests.ci-train.ubuntu.com/#/ticket/1487 | 13:33 |
| Mirv | alf_: probably fallout from the big rollout of new train code as announced by robert. he'll answer questions and start fixing in a few hours. | 13:34 |
| Mirv | alf_: some discussion a few hours ago on this channel above | 13:34 |
| alf_ | Mirv: ok, I will just fall back to the new request method for now | 13:35 |
| dobey | oh fun | 13:35 |
| dobey | https://requests.ci-train.ubuntu.com/log/1435/build/latest | 13:36 |
| dobey | alf_: you mean rebuild individual binary packages? | 13:36 |
| alf_ | dobey: I haven't tried individual packages, but a full rebuild doesn't work for me. ci-train doesn't complain is just that no new source packages reach the PPA to get built. | 13:38 |
| dobey | alf_: it tells you why though | 13:39 |
| Trevinho | robru: hey, it seems I can't force rebuild and it doesn't go... Weird, it was working tonight | 13:39 |
| dobey | Trevinho: you mean on the new build page with the one big button? | 13:40 |
| alf_ | dobey: Perhaps the problem is that rebuilds don't get proper new package versions (in the past they got .1,.2 etc suffixes), so the PPA ignores them? | 13:40 |
| Trevinho | dobey: yep | 13:40 |
| Trevinho | dobey: it used to work tonight, as I've been the first user of it according to robru :). And I also did a rebuild | 13:40 |
| dobey | Trevinho: yeah i just got an error with that anyway. but if you go to jenkins and find the build job for your silo #, you can do it the "old" way there | 13:41 |
| Trevinho | might be... it maybe doesn't get the .X thing | 13:41 |
| Trevinho | oh, thanks | 13:41 |
| dobey | alf_: well, one problem is that there was a manual upload to yakkety for u-s-c, and you need to merge those changes back to trunk for it | 13:41 |
| alf_ | dobey: thanks, I will fix that... | 13:42 |
| dobey | alf_: and repowerd failed to build: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-013/+packages | 13:43 |
| alf_ | dobey: Yes, that's why I want to rebuild after making some updates to the MP in the ci-train, but that doesn't work | 13:43 |
| dobey | alf_: ah, it seems to perhaps not be adding the .N | 13:45 |
| dobey | alf_: build from https://ci-train.ubuntu.com/job/ubuntu-landing-013-1-build/build instead perhaps | 13:46 |
| alf_ | dobey: thanks, I will try that | 13:47 |
| Trevinho | the thing is that if you tried to build with the new system, then you've to force a rebuild in jenkins since.... The system thinks ti has already built the last src | 13:51 |
| robru | pstolowski: make your email public | 14:10 |
| pstolowski | robru, oh.. how/where? | 14:10 |
| robru | pstolowski: in lp. The error is that it doesn't have permission to access your email address for the changelog entry | 14:14 |
| robru | dobey: also^^ | 14:14 |
| robru | Previously it would just default to its own email address, I had no idea so many people had private emails | 14:15 |
| robru | OK I'll investigate this other build issue in a sec, breakfast first | 14:15 |
| dobey | robru: fix it to work for people with private e-mails. :) | 14:16 |
| pstolowski | robru, 'hide my email addresses from other LP users', is that it? | 14:19 |
| dobey | pstolowski: yes | 14:19 |
| pstolowski | dobey, hmm i just changed it and it still fails | 14:19 |
| dobey | but it's all or none | 14:19 |
| dobey | and it shouldn't matter if it's private or not | 14:20 |
| === chihchun_afk is now known as chihchun | ||
| dobey | i think what it needs here, is the openid needs to have a way to allow you to select which e-mail address to make visible to the citrain site, and citrain should use that e-mail address if it must use the address of the person who clicked 'build' | 14:21 |
| robru | dobey: right, good call. I can do that. but since this issue has a workaround let me fix this .X thing first | 14:24 |
| dobey | well, building via the jenkins job is a workaround for both :) | 14:24 |
| robru | pstolowski: hmm? tvoss had the same issue and he fixed it by going public | 14:25 |
| tvoss | robru, help would also be appreciated for: https://requests.ci-train.ubuntu.com/#/ticket/1449 | 14:26 |
| robru | tvoss: yep, i have an idea about that, will have to write some code though. | 14:27 |
| dobey | hmm, though it complains about diff missing | 14:28 |
| pstolowski | robru, not sure what's going on.. i got the exception, but it's building | 14:31 |
| robru | pstolowski: log | 14:32 |
| robru | ? | 14:32 |
| pstolowski | robru, i closed it, sorry. it was the same as before afaict | 14:33 |
| robru | pstolowski: what ticket? | 14:33 |
| pstolowski | robru, https://requests.ci-train.ubuntu.com/#/ticket/979 | 14:34 |
| robru | pstolowski: I dont' see the exception I just see it working: http://bazaar.launchpad.net/~ci-train-bot/unity-scopes-shell/unity-scopes-shell-ubuntu-yakkety-landing-065/revision/327 | 14:35 |
| pstolowski | robru, weird.. browser caching error? | 14:39 |
| robru | pstolowski: ah, could be | 14:40 |
| pstolowski | anyway..nvm if it works then good :) | 14:40 |
| rvr | Saviq: Silo 59 approved | 14:42 |
| Saviq | w00t | 14:43 |
| Saviq | rvr, you really wanted to prove us wrong, didn't you ;) | 14:44 |
| Saviq | mterry, please do the honors :) ↑↑ | 14:44 |
| mterry | Saviq, ooh | 14:45 |
| mterry | Saviq, let me go through all branches and confirm approval then press button | 14:45 |
| Saviq | mterry, I don't think train or rvr would have let it through unapproved | 14:46 |
| mterry | Saviq, that's a check made at publish time I believe | 14:46 |
| mterry | Saviq, (I know that one or two are unapproved, but trivially, will chase people to give final thumbs) | 14:47 |
| Saviq | mterry, ack | 14:47 |
| robru | mterry: Saviq if MPs were anything other than Work In Progress or Approved, train would say 'Bad Merges' state, and when you click publish it also checks that everything is Approved for you | 14:47 |
| mterry | robru, is that... is that a golden ticket for the web page icon? :) | 14:50 |
| robru | mterry: yes! ;-) | 14:50 |
| mterry | rvr, thanks for the silo 59 review, I know it was a lot of tiny things to check | 14:55 |
| rvr | mterry: Yeah, I prefer smaller ones. | 14:55 |
| mterry | rvr, we're in a big backlog right now | 14:55 |
| rvr | Saviq: I tried to prove you wrong but had no luck :D | 14:57 |
| dobey | robru: why would ticket say "diff missing" if it shows the diffs in the artifacts list? | 15:01 |
| robru | dobey: because it doesn't check the artifacts list, it checks swift directly. sometimes swift can be inaccessible and that can come back 'diffs missing'. Also I just changed the naming scheme of the diffs so it's possible you have old-style diffs (though unlikely if you just built) | 15:02 |
| robru | dobey: which ticket? | 15:02 |
| robru | alf_: pstolowski: dobey: tvoss: ok, the issue with the versions not incrementing .X should be resolved, please try your builds again | 15:03 |
| pstolowski | ok | 15:03 |
| dobey | robru: https://requests.ci-train.ubuntu.com/#/ticket/1435 | 15:03 |
| robru | dobey: yeah those are old-style diffs. series/source_name/content.diff should be series_source_name_content.diff | 15:04 |
| robru | dobey: I'm just testing the fix for the email then in a few minutes you should be able to use the new jobs again | 15:05 |
| renatu | robru, hey could you help me to understand what is happening on silo 73: https://requests.ci-train.ubuntu.com/#/ticket/1037 | 15:16 |
| renatu | robru, I updated the changes but it is not pushing it to ppa | 15:16 |
| robru | renatu: you mean "destination version missing from changelog"? | 15:17 |
| renatu | robru, fail to build | 15:17 |
| renatu | robru, I fixed the build, but it is not pushing the changes to ppa | 15:17 |
| renatu | robru, the ppa still sayint that it failed 1h ago | 15:18 |
| robru | renatu: oh, I *just* fixed that, please try again | 15:18 |
| renatu | robru, nice, thanks | 15:18 |
| robru | renatu: you're welcome | 15:18 |
| robru | renatu: sorry for the disruption! | 15:18 |
| renatu | robru, np, the new build interface is much better now | 15:19 |
| robru | renatu: thanks! yeah no more auto-log-out-after-10-seconds ;-) | 15:19 |
| robru | tvoss: dobey: pstolowski: ok bileto is now grabbing emails from your SSO session rather than from LP, so if you set your emails private it should still succeed in putting your email in the changelog. | 15:26 |
| pstolowski | robru, cool, tx | 15:27 |
| robru | pstolowski: you're welcome | 15:27 |
| robru | tvoss: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_39a8dbb93caf4ec889f8a1b7f69885db/bileto-1449/2016-06-01_09:01:38/vivid_net-cpp_packaging_changes.diff this diff is kinda funny | 16:04 |
| robru | tvoss: first, it seems you're clobbering a release that's already in vivid overlay, and it seems it has the same/similar changelog message. so I guess this is a duplicate release? | 16:05 |
| robru | tvoss: second, you shouldn't use bullet points in the Commit Message field on the MP. only the first line is used and it's passed to dch which prepends the bullet point for you, so you're getting duplicate bullet points there. | 16:05 |
| robru | tvoss: third, you should use the lp feature of linking bugs to the branches, those will be included in the message automatically. | 16:06 |
| dobey | robru: but it can only use the e-mail that is set as the "default" in lp right? | 16:21 |
| robru | dobey: well it can only use the email that is given to it by SSO, so and SSO (at least for me) only shows one email address, so presumably that is only the default, yes | 16:22 |
| robru | dobey: log out and log back in to see which email that is. or do a build and see what shows up in the changelog ;-) | 16:23 |
| dobey | oh, so it's not even the lp e-mail | 16:23 |
| dobey | it's the one you use to log in to sso | 16:23 |
| robru | dobey: right, because I don't have magical admin powers to access your private emails in your lp profile; this way is the only way that I can get an email from you | 16:25 |
| dobey | robru: well, still, what e-mail i use to log in to sso, and which one should appear in the changelog entries, aren't the same. | 16:27 |
| robru | dobey: I'm not really sure what you're wanting from me here. | 16:27 |
| dobey | robru: well, when i figure out if it's possible, i'll let you know | 16:27 |
| robru | dobey: ok. I guess you want it to query lp first and then fall back on SSO if that fails, but you still need to make your lp email public for that to work | 16:28 |
| dobey | no, that's not what i want. that would still present the same issue, because i'm pretty sure you'd only grab the "preferred" address from LP | 16:29 |
| robru | dobey: well, yeah. why is your preferred address not the one you prefer? | 16:32 |
| dobey | robru: well it is the one i prefer. however, the one i prefer for me in general, and the one that is preferred for canonical projects, are not the same. | 16:33 |
| robru | dobey: I'm not sure where else I'd get your email from. Maybe make a new SSO using the email you want, and then use that account exclusively for bileto builds? | 16:34 |
| robru | dobey: in the old system you would have just been "Rodney Dawes <ci-train-bot@canonical.com>" so surely your SSO email address is an improvement there. | 16:36 |
| dobey | robru: well, it seems that the openid bits can't do what i was hoping it could do. so i'm asking to get requests.ci-train.u.c added to a special list where sso will always return the foo.bar@canonical.com address for accounts that have one defined, regardless of what e-mail was used to log in, for the bileto site | 16:40 |
| robru | dobey: well if that works, then go for it. does that special list already exist? | 16:40 |
| robru | dobey: while you're at it, also get bileto.ubuntu.com added to that list, we'll start using that domain eventually | 16:41 |
| dobey | robru: seems so. | 16:41 |
| Trevinho | robru: internal server errror :o | 16:47 |
| Trevinho | robru: on requesting a buld | 16:48 |
| robru | Trevinho: where? | 16:48 |
| Trevinho | https://requests.ci-train.ubuntu.com/log/1481/build/ check compiz, build | 16:48 |
| Trevinho | robru: I've commented a MP in the list, but it shouldn't matter, right? | 16:48 |
| robru | Trevinho: yes theoretically commented MPs are ignored correctly | 16:49 |
| robru | well, a bunch of "diff missing" implies that swift is down ^, possibly other services busted | 16:49 |
| robru | Trevinho: https://requests.ci-train.ubuntu.com/log/1481/build/latest working now, momentary hiccup I guess? | 16:50 |
| robru | Trevinho: wait, what? this is some unicode bs | 16:51 |
| robru | Trevinho: https://pastebin.canonical.com/157824/ (\xf1 is your funny n) | 16:52 |
| robru | Trevinho: but this makes no sense because a) it used to work and b) that shouldn't be forcing ascii at all wtf | 16:52 |
| robru | Trevinho: eh, I have an idea. apparently the thing I did to fix emails now broke unicode characters in environment variables, fun times. | 16:55 |
| === alan_g is now known as alan_g|EOD | ||
| Saviq | robru, getting 500 when trying to build this https://requests.ci-train.ubuntu.com/log/1483/build/latest :/ | 17:02 |
| robru | Saviq: try again, I just pushed a fix | 17:03 |
| Saviq | robru, I *just* tried, and tried again | 17:03 |
| Saviq | 599 | 17:03 |
| Saviq | 500, even | 17:03 |
| robru | Saviq: ok, it's because bileto is trying to store your name in an env var and for some reason it's not happy with the unicode. | 17:05 |
| robru | Saviq: I kicked the build for you for now, trying to work out a proper fix | 17:05 |
| Saviq | ack, tx | 17:05 |
| robru | stupid python defaulting to ascii env vars IN TWENTY SIXTEEN | 17:05 |
| robru | Saviq: Trevinho: ok I have a fix confirmed working in staging, just waiting for #webops to roll it out (guy's on lunch, could take a while), in the mean time just ping me if you need any builds done | 17:14 |
| Saviq | that's just unicodism I tell you | 17:15 |
| Saviq | it's just like racism, but about unicode chars | 17:15 |
| robru | Saviq: ok well production has gone and shat itself now | 17:23 |
| Saviq | sounds like fuyn | 17:24 |
| robru | Saviq: HTTP500 on some endpoints but not all, and nothing in the logs. great. also can't reproduce in staging. | 17:24 |
| Trevinho | robru: new gif for me too :) | 17:27 |
| Saviq | robru, when it's back, if you could please rebuild https://requests.ci-train.ubuntu.com/#/ticket/1483 again, thanks | 17:38 |
| Saviq | cimi, actually it might be !dead for you, since it seems to be robru's unicodism kicking in again | 17:39 |
| cimi | Saviq, sorry in other words why is only dead for me? | 17:39 |
| robru | Saviq: yeah i dunno i asked them to restart it to get some new environment vars and now the whole thing's dead, and guy went on lunch | 17:39 |
| cimi | ah ok, well I go afk a bit, will be back later to check | 17:41 |
| cimi | thank you | 17:41 |
| robru | great I can't even log in now | 17:47 |
| robru | Trevinho: Saviq: just for shits and giggles can I get one of you to run this staging job: https://requests.ci-train.staging.ubuntu.com/log/1186/build/ just to confirm the fix really works | 18:00 |
| Trevinho | robru: done | 18:01 |
| robru | Trevinho: thanks | 18:01 |
| Saviq | robru, https://requests.ci-train.staging.ubuntu.com/log/1186/build/latest seems to be running | 18:01 |
| robru | Saviq: Trevinho: ok, looks really good, thanks guys. I have no idea why production exploded, just waiting for #webops guy to come back and reboot it basically, exact same code is working perfect in staging. | 18:02 |
| Trevinho | yay nice when it happens | 18:02 |
| robru | Saviq: apologies, seems to be running now: https://requests.ci-train.ubuntu.com/log/1483/build/4 | 18:42 |
| robru | literally just rebooted it. no idea what went wrong | 18:42 |
| robru | cimi: we're back up, please retry | 18:51 |
| robru | bblunch | 18:58 |
| robru | lawl | 19:47 |
| === chihchun is now known as chihchun_afk | ||
| robru | Much diff missing. Wow. | 22:01 |
| === salem_ is now known as _salem | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!