=== doko_ is now known as doko
=== psivaa is now known as psivaa-brb
=== psivaa-brb is now known as psivaa
=== greyback is now known as greyback|away
=== Pici` is now known as Pici
=== wickedpuppy2 is now known as wickedpuppy
=== greyback|away is now known as greyback
* barry waves15:03
* sithlord48 waves15:03
=== bhuey_ is now known as bhuey
xnoxlooks like slangasek nor cjwatson are around =)15:05
xnoxwe could all just skive off..... =)15:05
jodhcjwatson is out this afternoon.15:05
robrudamnit, I prepped my list for once!15:05
jodhpartaay y'all!15:05
bhueyslangasek: was up really late last night15:06
bhueyI'm starting anyways15:07
meetingologyMeeting started Thu Jul 31 15:07:43 2014 UTC.  The chair is xnox. Information about MeetBot at http://wiki.ubuntu.com/meetingology.15:07
meetingologyAvailable commands: action commands idea info link nick15:07
bhueydon't know what the procedure is...15:07
xnox#topic Lightning round15:07
=== meetingology changed the topic of #ubuntu-meeting to: Lightning round
bhueygo ?15:07
bhueyLast week15:08
bhuey-change direction and abandoned the security update to 7u55-2.4.715:08
bhuey-merged debian package 7u65-2.5.1 to ubuntu utopic, merged differences into this package15:08
bhuey-backported it to 12.04, 14.04 and utopic, this went well except for 12.0415:08
bhueyThis week15:08
bhuey-posted test differences between various openjdk versions for regression analysis15:08
bhuey-struggling to deal with 'quotes' in a call to 'configure' in the build/ directory and wondering why the build system is reverting Matthias's patches15:08
bhuey-continued to debug the above15:08
bhuey-get familiar with debian version 3 packaging15:08
xnox$ shuf -e jodh robru sil2100 xnox barry bmurray doko stgraber15:08
xnoxnot as fancy.15:08
xnox* usb-creator - formating fixes uploaded15:09
xnox* mega-transition helping out tie up loose ends15:09
xnox* llvm 3.5 - fixed powerpc build (committed upstream) and fixed15:09
xnox  ppc64el packaging (accepted in debian). Will switch default to 3.515:09
xnox  after alpha2 freeze block is lifted. That will make temporary add15:09
xnox  3.5 & 3.4 in main, until next mesa update.15:09
* jodh wonders who 'not as fancy' is :)15:09
xnox* uploaded ubuntukylin-meta seating in NEW, but proper further seed15:09
xnox  setup is still needed.15:09
xnox* exit interview, and HRy things.15:09
xnox* working on porting more lazr.* things to python315:09
xnox* sent rebased apt-ftpmaster generate contents only patch to mvo on15:09
xnox  BTS. (To enable for it to be used, will need further patch to cronscripts)15:09
xnox* complete more systemd/startpar tasks15:09
xnox* complete lazr.*/launchpadlib ports15:09
xnoxbarry: =)15:09
xnoxjodh: oh just a comment. as in it's not a one liner as it usually pasted around here.15:09
jodhxnox: yeah, I'm just being silly.15:09
barryphone: system-image 2.3.1.  smoke test crash investigation (LP: #1349478).  discussing jani's branches.  other stuff.15:09
ubottuLaunchpad bug 1349478 in Ubuntu system image "/usr/sbin/system-image-dbus:sqlite3.OperationalError:_check_for_update:emit_signal:UpdateAvailableStatus:__init__:__enter__:_cursor" [High,In progress] https://launchpad.net/bugs/134947815:09
barrydebuntu: zope stack (all squared away now!).  gunicorn py3 package support (debian bug #756057).  testing pyvenv/virtualenv from trusty PPA.15:09
ubottuDebian bug 756057 in gunicorn "gunicorn: Support Python 3" [Normal,Open] http://bugs.debian.org/75605715:09
xnoxjodh: and it didn't suffly much, did it?!15:09
barryother: dealing w/various hardware issues.  other discussions on various other topics.15:10
jodh* upstart15:10
barryjodh: =)15:10
jodh*** upstart 1.13.1-0ubuntu2 upload to fix cgmanager tests.15:10
jodh* system-image15:10
jodh  * Spent most of the week learning about how system image builds15:10
jodh    (ongoing :)15:10
jodh  * Working with mvo, cjwatson and stgraber on setting up new images.15:10
xnoxbarry: gunicorn python3 \o/15:11
xnoxthe all prepared robru is next =)15:11
barryxnox: yep, have github pull req pending for debian maintainer.  should hopefully land soonish15:11
robru* landings, landings, landings, landings!15:11
robru* fixed up qtcompositor landing in ubuntu-system-settings15:11
robru* learned all about Jasmine unittesting framework for JS -- very nice!15:11
robru* wrote vast amounts of unit test code for both CI Train Silo Dashboard and NFSS Web UI15:11
robru* attended GUADEC -- saw many talks and took lots of notes15:11
robru* Several CI Vanguard shifts15:11
robru* stopped some bits of CI Train from hard-coding spreadsheet column indexes, allowing greater15:11
robruflexibility to change the spreadsheet in the future (in preparation for RTM)15:11
robru* vast simplification of CI Train spreadsheet, dropping all stupid "silo tabs" and streamlinin15:11
robrug the testing:done? setting into the pending tab, which will make the spreadsheet much easier15:11
robruto navigate when RTM doubles the number of silos we have.15:11
robru* various minor pre-RTM preparatory fixes & changes in CI Train15:11
robru* 99 commits against the CI Train silo dashboard page. yikes! (many many many small tweaks and15:11
robru iterations, and increases in test coverage).15:11
xnoxrobru: sil2100: has ci-train been tested / dry-run against rtm on dogfood?15:12
sil2100xnox: doing it all the time ;)15:12
xnoxsil2100: awesome.15:12
xnoxsil2100: you are next! =)15:13
* slangasek waves belatedly15:13
sil2100xnox: just got blocked by some firewall blockings, but now I'm unblocked again, webops helped out15:13
robruxnox, sil2100 has taken charge of the ci train core. I'm mostly working on the periphery, like the dashboard and queuebot.15:13
xnox#chair slangasek15:13
meetingologyCurrent chairs: slangasek xnox15:13
sil2100- Landing team work, landing e-mails, landing coordination - standard stuff15:13
sil2100- Pushing on promotion and TRAINCON-0 issues15:13
sil2100- CI Train maintenance and features:15:13
sil2100  * Decoupling prod from preprod, now for testing preprod custom branches can be used15:13
sil2100  * Small enablements related to the modified spreadsheet15:13
sil2100  * Continuing work on enabling other distributions (ubuntu-rtm)15:13
sil2100    - More firewall holes needed in our prodstack instance15:13
sil2100    - More complex silo configuration handling15:13
sil2100    - Many many corner cases where ubuntu was still considered instead of selected distro15:13
sil2100  * Test landing of indicator-location to ubuntu-rtm (in progress)15:13
sil2100  * Prepared branch with 'retry failed jobs'15:13
sil2100  * Add additional twin upload projects15:13
sil2100  * No time to finish work on auto-merge-clean15:13
sil2100- Work on defining TRAINCON-0 formal rules15:13
sil2100- Packaging advice for some upstream developers15:14
sil2100- Documenting the TRAINCON-0 incident15:14
sil2100- Playing around with some hardware15:14
bdmurraythis covers about 2 weeks for me since I was on holiday last thursday15:14
bdmurrayupdated errors for assets.ubuntu.com to r48615:14
bdmurraytesting of errors frontend change to filter on pkg_arch15:14
bdmurraysubmitted RT to have errors frontends updated and errors_static_url modified15:14
bdmurraysetup logrotation for the daisy, errors frontends and pushed to the daisy and errors charms15:14
bdmurrayupdated daisy retracer to keep core dumps when retracing and save crash files in certain situations15:14
bdmurraysubmitted rt to have retracers updated to r50415:14
bdmurrayupdated RT 72977 regarding errors log rotation15:14
bdmurraysubmitted RT 73492 regarding updating the daisy frontends to r49815:14
bdmurrayupdated daisy bucket code to pass architecture to bucketversionscount15:14
bdmurraypush bzr changes for daisy to increment arch counters15:15
bdmurraycreated some armhf retrace success failures graphs in graphite15:15
bdmurraysubmitted RT to add more retracers for errors15:15
bdmurraypinged a webop to run import-user-packages cron job (they said it worked but still nothing in the ColumnFamily)15:15
bdmurraymanually ran import_user_packages to the temp DSE ring database15:15
bdmurrayresearch into recoverable problem bucket grouping upstart and url-dispatcher issues15:15
bdmurraydiscovered and reported apport bug 134957915:15
ubottubug 1349579 in apport (Ubuntu) "whoopsie-upload-all uses an incorrect assumption regarding what to upload" [Undecided,Fix released] https://launchpad.net/bugs/134957915:15
bdmurraysubmitted merge proposal fixing apport bug 132952015:15
ubottubug 1329520 in apport (Ubuntu) "whoopsie-upload-all crashes while processing crash file" [High,Fix released] https://launchpad.net/bugs/132952015:15
bdmurrayinvestigation into SystemImageInfo not appearing in apport .crash files on the phone15:15
bdmurrayuploaded new version of apport to utopic which will properly gather SystemImageInfo15:15
bdmurrayuploaded new version of whoopsie to upotic that will send SystemImageInfo to errors15:15
bdmurraytested whoopsie bug 1320988 regarding online / offline connectivity15:15
ubottubug 1320988 in whoopsie (Ubuntu) "whoopsie did not become on-line after connecting to wifi" [High,Confirmed] https://launchpad.net/bugs/132098815:15
bdmurrayinvestigation into whoopsie bug 134060415:15
ubottubug 1340604 in whoopsie (Ubuntu) "[phone] crash files are only uploaded on boot when not running in the foreground" [Undecided,New] https://launchpad.net/bugs/134060415:15
bdmurrayreported apport bug 1347009 regarding retraced crashes missing stacktrace15:15
ubottubug 1347009 in Daisy "apport-retrace occassionally creates a retraced report without a stacktrace" [Low,Triaged] https://launchpad.net/bugs/134700915:15
bdmurrayresearch into duplicates for ubuntu-release-upgrader bug 134772115:15
ubottubug 1347721 in apt (Ubuntu Trusty) "Saucy -> Trusty upgrade failed: procps fails to configure" [High,Triaged] https://launchpad.net/bugs/134772115:15
bdmurray✔ done15:15
doko- GCC default set to 4.915:15
doko- update of cross toolchains15:15
doko- openjdk-7 mentoring and fixes15:15
doko- clean up component mismatches, dep-waits, ftbfs in main, three days of nagging and fixing15:15
doko- packaging review of some third party software15:15
doko- updated tightvnc, updated tigervnc (at least it built for arm64 and ppc64el)15:15
doko- twisted transition (ubuntu-sso-client still unfixed)15:16
doko- arm64 toolchain discussion15:16
stgraberWas on vacation last week.15:17
stgraberE-mail and IRC catchup on Monday mostly.15:17
stgraberHelped with some code reviews, discussions, ... wrt ubuntu core system-image.15:17
stgraberGot an initial system-image published for ubuntu-core.15:17
stgraberDiscussed partitioning plan for new touch devices.15:17
stgraberFixed some LXC CI issues.15:17
stgraberRecorded a video on running GUI apps inside LXC: https://www.youtube.com/watch?v=QYsj9LEqxXk15:17
stgraberPoked some more at running Unity8 inside LXC.15:17
stgraberSome more LXC-related discussions (conference planning, ...)15:17
stgraberDiscussed some of our NetworkManager patches.15:17
stgraberFixed a couple of configuration issues with the ISO tracker related to alpha-2, 14.04.1 and
xnoxslangasek: your turn =) and take over chairing =)15:18
slangasekhmm :)15:18
slangasek * working with bhuey on prepping openjdk security update15:20
slangasek * discussions around the 'init' package and systemd-sysv to unblock new images using systemd from the start15:20
slangasek * tracking crash retracing success on the phones; working with bdmurray et al. to get any blocking issues fixed in advance of RTM15:20
slangasek * performance reviews15:20
slangasek * helped move forward some HWE SRUs related to a server engagement15:20
slangasek * helped with getting out of TRAINCON-0 on Monday15:20
slangasek * next week: joining a cloud team sprint (as is Colin), so expect limited availability15:20
slangasek * the week after: on vacation15:20
slangasekbdmurray: on the errors.u.c side, when do you expect we'll have the per-image view available?15:21
bdmurrayslangasek: what was the final answer for the counter?15:22
slangasekbdmurray: ah, in scrollback, let's iron that out after the meeting?15:22
bdmurrayslangasek: okay, it shouldn't take too long to add15:23
slangasekany other questions over status?15:23
slangasek[TOPIC] Upstart cgroup support15:24
=== meetingology changed the topic of #ubuntu-meeting to: Upstart cgroup support
slangasekforgot to do an in-depth topic for last week's meeting... remembered this week :)15:24
jodhThanks Steve. Everyone sitting comfortably?15:25
slangasekso jodh will talk a bit about the work he did to implement cgroup support into upstart15:25
jodhToday, I'm going to give a brief [1] talk about cgroup support in upstart15:25
jodhand some of the challenges we faced. This may go some way in explaining15:25
jodhthe seemingly never-ending upstart async branch updates I've given in15:25
jodhthe past in this meeting :-)15:25
jodh= Intro =15:25
jodhAs of version 1.13, Upstart supports cgroups. By "support", I mean "has15:25
jodhthe ability to place job processes into one or more cgroups" (for15:25
jodhservice resource control). It does _not_ mean that Upstart uses cgroups15:25
jodhto mop up the mess if a service ends badly (process supervision).15:25
jodh= The cgroup Stanza =15:25
jodhAfter thrashing out the design with stgraber, slangasek and hallyn15:25
jodh(http://upstart.ubuntu.com/wiki/Cgroup), we added support to parse a new15:25
jodh"cgroup" stanza that a job can specify. The final syntax is extremely15:25
jodhclean and praise goes to stgraber for realising how simple we could make15:25
jodhit!) Here's a summary of the behaviour:15:25
jodh- If not specified, the job processes are not placed into (any new)15:25
jodh  cgroups.15:25
jodh- If a job specifies a cgroup stanza, that job cannot legitimately start15:26
jodh  until the cgroup manager itself is running. To handle this, we added a new15:26
jodh  initctl command ("notify-cgroup-manager-address") which the15:26
jodh  cgmanager.conf job itself calls in post-start to notify upstart where15:26
jodh  to find cgmanager :-)15:26
jodh- If specified, *all* job processes are put into the specified cgroup(s).15:26
jodh- If specified as "cgroup <controller>" ("cgroup cpuset" for example),15:26
jodh  Upstart will add the job to a job-specific cgroup whose value will be15:26
jodh- If specified as "cgroup cpuset foo 12", Upstart will place the job15:26
jodh  processes into the implicit job-specific cpuset cgroup and set15:26
jodh  "foo=bar" in that group.15:26
jodh- If specified as "cgroup cpuset hello foo 12", Upstart will place the15:26
jodh  job into a the cpuset cgroup called "hello" and set "foo=12" in that15:26
jodh  group. If "hello" does not exist, it will be created. This allows15:26
jodh  multiple different jobs to enter the same cgroup if desired.15:26
jodh- The cgroup name ("hello" in the example above) can also contain variables:15:26
jodh  "cgroup cpuset db/$foo/$bar-$baz".15:26
jodh- You can also get at the cgroup that Upstart would create on behalf of15:26
jodh  the job using the magic $UPSTART_CGROUP variable (note that this is15:26
jodh  _not_ an environment variable and is only valid within a cgroup15:26
jodh  stanza). For example: "cgroup cpuset db/$UPSTART_CGROUP".15:26
jodhSo far so good.15:26
jodhSince there is already an excellent cgroup manager available, and since15:26
jodhwe try to avoid adding extra complexity to PID 1, we opted to avoid15:26
* slangasek waits for the footnote to resolve15:26
jodhre-inventing the wheel by having cgmanager(8) handle the actual cgroup15:26
jodhoperations. So, when Upstart starts a job that specified a cgroup15:26
jodhstanza, it needs to do the following:15:26
jodh- Connect to cgmanager.15:26
jodh- Ask cgmanager to create the cgroup(s).15:26
jodh- Ask cgmanager to move the specified process(es) into a cgroup.15:27
jodh- Ask cgmanager to set apply a particular setting to a cgroup.15:27
jodhHowever, there's a problem with the above. What if cgmanager hung?15:27
jodhWe'll come back to this, but first I need to explain how Upstart spawns15:27
jodha job.15:27
jodh= Async spawning =15:27
jodh== Historical synchronous spawning ==15:27
jodhUpstart used to do the following when wishing to start a new job process:15:27
jodh1) Create a pipe.15:27
jodh2) fork itself.15:27
jodh3) Have the child do all necessary setup such as dropping privileges,15:27
jodh   closing fds, switching apparmor profiles, etc. Then:15:27
jodh   - If the child finished its setup successfully it simply exec'd the15:27
jodh   relevant program specified in the job .conf file for the job process15:27
jodh   in question).15:27
jodh   - But if the setup failed, the child wrote a status message back up to15:27
jodh     the parent (PID 1) explaining what went wrong, and then exited.15:27
jodh4) All the time the child wass doing setup, PID 1 was doing a _blocking15:27
jodh   read_ on its end of the pipe. This implies that no operation in the15:27
jodh   child setup phase could block, since if it could block, it would also15:27
jodh   block PID 1, and thus DoS the system.15:27
jodhAs a result, we couldn't call cgmanager from PID 1 directly, since that15:27
jodhcould lead to a DoS, but we couldn't call it from the child _either_.15:27
jodh== Brave New World ==15:27
jodhThe solution was to change the way in which Upstart spawns. In the new15:27
jodhworld, Upstart still creates the pipe but doesn't do a blocking read; it15:27
jodhjust adds the fd for the read end of the pipe to a queue and waits for15:27
jodhsome notification from the child. So we now have asynchronous child15:27
jodhspawning. To achieve this, we had to increase the number of states the15:27
jodhjob can be, since there is now a distinction between:15:27
jodh- "the job process has been spawned successfully"15:28
jodh- "the job process is _being_ spawned" (but we don't know what the outcome is yet).15:28
jodhHere is the old state transition diagram:15:28
jodhAnd here's the new one:15:28
jodhHowever, that only solved half the problem - since Upstart was now15:28
jodhspawning asynchronously, the design meant that the order in which child15:28
jodhnotifications could arrive became non-deterministic since either of the15:28
jodhfollowing could happen "first":15:28
jodh- child exits and Upstart is notified via waitid().15:28
jodh- child pipe closes or has data written to it and is notified by select().15:28
jodhThis needed careful handling since *both* those operations could update15:28
jodhthe job state, but we didn't want the state to be "double-bumped".15:28
jodhIn summary, adding the new async spawning feature was quite a challenge15:28
jodhwith xnox and I gaining a few grey hairs in the process (his don't show!15:28
jodhErrm... ;-)15:28
jodh= Test Suite =15:28
jodhSince the new states and the new async nature of spawning meant that a15:28
jodhlarge chunk of the (large!) set of Upstart tests suddently broke. Hence,15:28
jodhit took lots of careful reviewing of both the code and all the required15:28
jodhtest changes to resolve this new feature.15:28
jodh= Stateful re-exec =15:28
jodhThis needed updating to handle the new cgroup stanza data. But we also15:28
jodhneeded to consider scenarios like this:15:28
jodh- PID 1 starts a job process asynchronously.15:28
jodh- child takes "a long time" to setup.15:28
jodh- PID 1 is restarted.15:28
jodhPost-re-exec, PID 1 needs to know to keep track of the outstanding child15:28
jodhsetup operations it is (asynchronously) waiting on. To handle this, we15:28
jodhadded a new JobProcessData object to store the transitory child setup15:29
jodhmeta-data (which gets discarded once the child has either died or15:29
jodhresponded down the pipe).15:29
jodh= Cgroup Operations =15:29
jodhWith the advent of async spawning, the child now makes all necessary15:29
jodhcalls on the cgmanager with PID 1 being completely immune to any issues15:29
jodhthat that may entail. In fact, aside from storing the parsed cgroup15:29
jodhstanza data, all PID 1 does is store the address of cgmanager!15:29
jodh= Conclusion =15:29
jodhThe final result is an extremely clean and safe design. By introducing15:29
jodhasync spawning we were also able to make Upstart fully immune to the15:29
jodhchild blocking PID 1 (there used to be a couple of areas that15:29
jodhtheoretically could cause issues on a mis-configured system).15:29
jodh[1] - FTR, I'm using ev's definition of 'brief' :-)15:29
* jodh grumbles over whitespace damage....15:29
jodhA non-garbled version: http://paste.ubuntu.com/7915372/15:29
slangasekjodh: so, http://people.canonical.com/~jhunt/upstart/upstart-states-new.png is the state diagram for what's implemented now in 1.13?15:30
jodhslangasek: yep - I need to refresh the cookbook with that.15:31
xnox(also pretified the graph in graphviz for vertical top/down layout)15:32
xnoxvs previously "optimal" graph15:32
jodhxnox: yeah, that's much improved, thanks! :)15:32
slangasekstill just the single error path, though; couldn't you have made it more complicated? ;)15:32
barryjodh: very interesting!  is there a timeout after which the child is just considered hung, and does upstart do anything about that state?15:33
jodhbarry: no - no timeout.15:33
xnoxbarry: it just gets stuck in e.g. "starting/pre-starting" state.15:34
jodhbarry: if the child hangs, the state will reflect that if you run 'initctl status $job'.15:34
xnox(or some such, can't remember exact names of the spawning state"15:34
* slangasek nods15:35
slangasekif the job never gets around to starting, it's not init's job to fix it ;)15:35
slangasekjodh: do we have people using the cgroup support in anger yet?15:36
jodhbarry: If a job does hang though, you may get something useful in cgmanagers log (assuming you've set cgmanager_opts= in /etc/init/cgmanager.conf).15:36
jodhslangasek: I don't think so actually. I checked on a recent touch image and I can't see any evidence of it being used yet. We need to poke ted! :)15:37
slangasekjodh: has ted been poked about this yet?15:37
slangasekif not, then yes, yes you do ;)15:37
jodhslangasek: not by me directly. I added him to https://code.launchpad.net/~jamesodhunt/ubuntu/utopic/cgmanager/enable-upstart-cgroup-support/+merge/227209 so he should be aware of it.15:38
xnoxslangasek: i'll poke ted about it.15:38
slangasekxnox: ok, thanks15:39
jodhlet's all poke ted. err...15:39
slangasekany other questions about cgroups in upstart?15:40
slangasekjodh: thanks for presenting!15:41
slangasek[TOPIC] AOB15:41
=== meetingology changed the topic of #ubuntu-meeting to: AOB
bdmurrayslangasek: I was thinking maybe you and stgraber could discuss bug 1314616?15:42
ubottubug 1314616 in bitcoin (Ubuntu) "[SRU] bitcoin to be maintained upstream in PPA: Replace distro archive "bitcoin" bitcoin with an empty dummy package" [Undecided,Confirmed] https://launchpad.net/bugs/131461615:42
slangasekoh no15:42
stgraberoh, that again?15:42
slangasekstgraber: I had explicitly told them to submit an SRU to disable the daemon on upgrade15:43
slangasekstgraber: and you are apparently not happy with the proposed solution15:43
stgraberslangasek: I'm not?15:43
slangasekstgraber: so yes, we should talk, but probably not during the meeting :-)15:43
slangasekstgraber: that's what I heard!15:43
bdmurraythere was an email to ubuntu-devel about it earlier this month15:44
slangasekanyway, maybe we talk on #ubuntu-devel after the meeting?15:44
stgraberslangasek: I think I was just unhappy that the reporter tried to get us to do things without going through the proper SRU process15:44
slangasekah :-)15:44
stgraberslangasek: I don't care about the package itself and am perfectly happy to have it die one way or another :)15:44
slangasekok then!15:45
slangasekanything else to discuss on this fine summer day?15:45
stgraberI mostly complained to the reporter when he started nagging me as the current patch pilot to do something which needed discussion with the SRU team. Now that the SRU team has clearly been informed of it, someone should just sponsor a debdiff and be done with it.15:45
* slangasek nods15:46
=== meetingology changed the topic of #ubuntu-meeting to: Ubuntu Meeting Grounds | Calendar/Scheduled meetings: http://fridge.ubuntu.com/calendar | Logs: https://wiki.ubuntu.com/MeetingLogs | Meetingology documentation: https://wiki.ubuntu.com/meetingology
meetingologyMeeting ended Thu Jul 31 15:47:40 2014 UTC.15:47
meetingologyMinutes:        http://ubottu.com/meetingology/logs/ubuntu-meeting/2014/ubuntu-meeting.2014-07-31-15.07.moin.txt15:47
slangasekthanks, everyone!15:47
=== psivaa is now known as psivaa-bbl
=== psivaa-bbl is now known as psivaa
geniiI wonder how many people have "repaired" their own brakes instead of getting a mechanic, then ended up careening off a cliff16:58
geniiWrong channel, sorry :)16:58
=== DalekSec_ is now known as DalekSec

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!