sergiusens | rsalveti: I need to download the full diff, but yeah, we need a systemd unit to do something on boot | 00:22 |
---|---|---|
rsalveti | sergiusens: right, this change is just something post update | 00:23 |
sergiusens | mvo: mterry helpers to me feels like a drop zone for things that have no home yet; same for handler, helpers and handlerhelpers... but yeah, I wouldn't mind moving it but I don't want to affect Chipaca's big refactor | 00:28 |
sergiusens | affect or be affected | 00:28 |
rsalveti | sergiusens: do we have a way to get systemd units that would only be executed at first boot/after updates | 00:53 |
rsalveti | ? | 00:53 |
sergiusens | rsalveti: we would have to do the flag trick (like snappy firstboot) | 00:57 |
rsalveti | right | 00:58 |
rsalveti | seems to be the only remaining bug we have | 00:59 |
rsalveti | jdstrand: sergiusens: Chipaca: hm, installed image 78 (15.04/edge), clean, then webdm and tried to install a few snaps, but they are all failing | 05:11 |
rsalveti | Jun 5 05:08:14 localhost kernel: [ 39.097383] audit: type=1400 audit(1433480894.752:22): apparmor="DENIED" operation="mkdir" profile="/usr/bin/ubuntu-core-launcher" name="/tmp/snaps/" pid=1024 comm="ubuntu-core-lau" requested_mask="c" denied_mask="c" fsuid=0 ouid=0 | 05:11 |
rsalveti | Jun 5 05:08:14 localhost kernel: [ 39.117912] audit: type=1400 audit(1433480894.772:23): apparmor="DENIED" operation="capable" profile="snake.mectors_snake_0.0.5" pid=1024 comm="snakeweb" capability=12 capname="net_admin" | 05:12 |
rsalveti | xkcd-webserver.canonical gave package not found | 05:12 |
rsalveti | system-status.victor: failed to install: package name with namespace not supported | 05:12 |
rsalveti | xkcd-webserver installed fine via cmdline though | 05:14 |
rsalveti | this is not bug 1460152 since I didn't do any image update | 05:14 |
ubottu | bug 1460152 in Snappy 15.04 "apparmor cache not updated when apparmor.d rules change (breaks 15.04/stable -> 15.04/edge updates)" [Critical,In progress] https://launchpad.net/bugs/1460152 | 05:14 |
miken | rsalveti: it's not related to bug 1460517 is it? (I created that last week, trying to get someone to check whether it's webdm or snappy itself setting those perms) | 05:31 |
ubottu | bug 1460517 in webdm "Cannot run other snaps after first installing webdm" [Undecided,New] https://launchpad.net/bugs/1460517 | 05:31 |
rsalveti | miken: yeah, looks like it | 05:31 |
miken | Although, that was with stable. | 05:31 |
rsalveti | hm, will investigate a bit more tomorrow | 05:32 |
dholbach | good morning | 06:39 |
fgimenez | good morning | 07:01 |
zyga | good morning | 07:24 |
dholbach | is it possible to change channels within a core instance? | 07:29 |
elopio | rsalveti: sergiusens: I would like to get that snappy-merge-integration-tests merged first. | 07:58 |
elopio | the only thing that was missing from my review like two weeks ago was that it wanted to sign the deb. | 07:58 |
elopio | any idea how to get out from emergency mode? | 08:22 |
elopio | or how did I get into it | 08:22 |
elopio | I see: Welcome to emergency mode! After logging in, typroot@localhost:~# | 08:22 |
Chipaca | elopio: where? wha? | 08:59 |
JamesTait | Good morning, people! Happy Friday, and happy World Environment Day! 😃 | 09:02 |
elopio | Chipaca: my bbb was working two days ago. But today it went nuts. | 09:06 |
Chipaca | elopio: I don't think we have an emergency boot thing; that sounds like it's booting the debian | 09:06 |
Chipaca | our emergency boot thing is "boot the other one" | 09:06 |
elopio | Chipaca: without the sd card it boots to debian without problems. | 09:07 |
Chipaca | elopio: reflash the sd? | 09:07 |
Chipaca | elopio: get a new sd? | 09:07 |
elopio | Chipaca: I'm reflashing. | 09:07 |
Chipaca | elopio: have it exorcised? | 09:07 |
Chipaca | :) | 09:07 |
elopio | I see messages like: | 09:08 |
elopio | [ 24.948063] FAT-fs (mmcblk0p1): IO charset iso8859-1 not found | 09:08 |
elopio | I'll just start all over again. | 09:08 |
=== dholbach_ is now known as dholbach | ||
ogra_ | hmm | 09:57 |
Chipaca | ogra_: hmm? | 10:15 |
ogra_ | Chipaca, i was wondering about http://paste.ubuntu.com/11584783/ | 10:21 |
ogra_ | (which i see on a really old RPi image here) | 10:21 |
Chipaca | ogra_: people have been getting those, and we looked into it a bit and got nowhere and blamed the sd card | 10:22 |
Chipaca | ogra_: tbh reflashing seems to fix it | 10:22 |
ogra_ | yeah, i should stop using the expensive branded ones i guess :P | 10:22 |
Chipaca | ogra_: if it *isn't* the sd card feeling sorry for itself, i don't know | 10:22 |
ogra_ | it doesnt seem to do any harm for a normal boot ... i'm juzst wondering what happens on upgrades if it goes readonly | 10:23 |
Chipaca | ogra_: only one way to find out | 10:24 |
ogra_ | well, not really ... at least not on a RPi ... not sure the BBB exposes the same thing | 10:24 |
fgimenez | elopio, rsalveti i've just received the boards =) | 10:40 |
elopio | fgimenez: nice :) | 10:40 |
elopio | I've just ran the failed update on my beagle. Works fine here. | 10:41 |
fgimenez | elopio, good, on kvm all seems to be fine too for rev 78 | 10:45 |
fgimenez | elopio, i'm finishing some changes in the script to allow the delta upgrade from edge -1 to edge that mentioned sergiusens | 10:46 |
sergiusens | Chipaca: elopio ogra_ I was hoping that my fix to only right to the sdcard when necessary would alleviate this problem | 11:13 |
sergiusens | oh, the charset issue, there was an initramfs missmatch somewhere along the lines | 11:14 |
mattyw | afternoon everyone | 11:21 |
fgimenez | sergiusens, about the delta upgrades, they are only available from rev -1 to rev, right? | 11:22 |
ogra_ | sergiusens, the above inst a charset issue ... it is actually bad filesystem blocks ... but as i said, the install is ancient | 11:26 |
fgimenez | elopio, the full upgrade path stable -> edge -1 (using script) -> edge (using snappy update) works fine too with the latest image in kvm | 11:29 |
sergiusens | ogra_: oh, that can be it | 11:32 |
sergiusens | fgimenez: rev -1 from the system running snappy's point of view is $version, so you should only be able to update to $version+n | 11:33 |
fgimenez | sergiusens, ok, i've added already the check to prevent updating to previous versions, i've also added code to download delta files if available for the current_version -> requested_version transition | 11:39 |
fgimenez | sergiusens, let me know what do you think https://code.launchpad.net/~fgimenez/+junk/system_upgrader | 11:39 |
=== tvoss is now known as tvoss|test | ||
=== tvoss|test is now known as tvoss | ||
=== tvoss is now known as tvoss|test | ||
=== tvoss|test is now known as tvoss | ||
jdstrand | rsalveti: capability net_admin is not allowed in any policy, so that is expected | 13:12 |
jdstrand | rsalveti: oh-- the core launcher policy has: /tmp/snap.*/ w, | 13:14 |
jdstrand | rsalveti: that does *not* match /tmp/snaps | 13:14 |
jdstrand | rsalveti: I believe miken is right about bug #1460517 and so I added information there | 13:21 |
nothal | Bug #1460517: Cannot run other snaps after first installing webdm <Snappy:New> <Snappy Launcher:New> <http://launchpad.net/bugs/1460517> | 13:21 |
ubottu | bug 1460517 in Snappy Launcher "Cannot run other snaps after first installing webdm" [Undecided,New] https://launchpad.net/bugs/1460517 | 13:21 |
* jdstrand updated the description for bug #1460517 | 13:21 | |
mterry | How do I switch channels for snappy? Like, say I wanted to go from rolling to 15.04/edge | 13:40 |
mterry | I would have assumed it's system-image-cli, but I recall hearing that doesn't work the same on snappy | 13:42 |
* mterry tries anyway | 13:42 | |
kyrofa | beowulf, ping | 14:00 |
beowulf | kyrofa: pong | 14:01 |
kyrofa | beowulf, in order to test webdm, I used ubuntu-device-flash to create a rolling image | 14:01 |
kyrofa | beowulf, is it somehow automatically updated? Webdm, at least? | 14:02 |
beowulf | kyrofa: you'll get the latest released one from the store, or you can checkout and build it and get tip | 14:03 |
kyrofa | So when you release a new version to the store, I automatically get it? | 14:03 |
beowulf | kyrofa: good question, i'll have to refer you to sergiusens or someone who knows more | 14:04 |
beowulf | kyrofa: (i don't know about automatic updates) | 14:04 |
kyrofa | beowulf, the reason I ask: this morning I'm not able to get webdm to install anything | 14:04 |
beowulf | kyrofa: what was the error? | 14:04 |
kyrofa | beowulf: DEBUG: [/usr/bin/snappy internal-unpack /tmp/snaps/webdm/0.8/tmp/xkcd-webserver811585830 /apps/xkcd-webserver.canonical/0.5 /] failed: operation not supported | 14:04 |
kyrofa | beowulf, and webdm returns a 500 | 14:05 |
kyrofa | beowulf, but it was working fine yesterday... so I'm confused | 14:05 |
rsalveti | jdstrand: nice, that seems to be it | 14:06 |
kyrofa | beowulf, the web interface seems to break too | 14:06 |
kyrofa | beowulf, stuck at installing 100% | 14:06 |
beowulf | kyrofa: yeah, that's a bug | 14:06 |
kyrofa | beowulf, then it prints "ERROR: xkcd-webserver.canonical failed to install: unpack /tmp/snaps/webdm/0.8/tmp/xkcd-webserver298198856 to /apps/xkcd-webserver.canonical/0.5 failed with exit status 1 " | 14:06 |
beowulf | kyrofa: the 500 causes that | 14:06 |
kyrofa | beowulf, you handle it better than the scope does :P | 14:07 |
beowulf | kyrofa: the bug is the install behaviour does nothing after the 500, so remains in whatever state it was before the 500, which is install progress 100% (which means the download happened and then we died) | 14:07 |
kyrofa | beowulf, can you verify that webdm is broken? | 14:08 |
beowulf | snappy tells webdm the error and webdm does the popup, but then should reset the button or something... basically i'm not sure what to do so have yet to fix it | 14:08 |
sergiusens | mterry: change channels.ini, the qa doc explains how to do it step by step | 14:08 |
beowulf | kyrofa: give me a sec to get a new image | 14:08 |
sergiusens | kyrofa: beowulf as soon as something updates you get the update | 14:09 |
kyrofa | sergiusens, good deal, so I'm not going insane | 14:09 |
sergiusens | rsalveti: I think your webdm broken on edge scenario is related to the core launcher changes | 14:09 |
rsalveti | right | 14:10 |
beowulf | kyrofa: so there you go, it was his fault *points at random person* | 14:10 |
sergiusens | rsalveti: jdstrand we are at a point where any change we make needs to work everywhere | 14:10 |
mterry | sergiusens, sorry but do you know where that qa doc lives? | 14:10 |
sergiusens | beowulf: kyrofa I'm guessing it's the latest changes to the core launcher that landed | 14:10 |
sergiusens | mterry: I will try and search my drive :-P | 14:11 |
sergiusens | mterry: yay, wasn't so hard https://docs.google.com/document/d/1R_Tw0N0QbEpjFeYf9XnVV8Gp8ldT2Ig0PO6MfR-kuSM/edit | 14:11 |
mterry | sergiusens, thank you! | 14:11 |
sergiusens | u r welcome! | 14:12 |
mterry | sergiusens, I ran "sudo system-image-cli --switch ubuntu-core/15.04/edge" which sat there for a while and finally exited with no message. And didn't do anything | 14:12 |
mterry | sergiusens, seems like that should have worked :-/ | 14:12 |
kyrofa | sergiusens, alright, good to know. I'm happy to help in any way I can! | 14:12 |
mterry | or at least errored out | 14:12 |
jdstrand | sergiusens: that's fine... I think I am missing context. the rule change I gave in the bug doesn't remove anything, only add /tmp/snaps/ | 14:18 |
jdstrand | sergiusens: therefore it should meet the criteria you just gave | 14:18 |
fgimenez | elopio, while trying to get the current installed version on kvm in my first attempt i copied code from ubuntu-ota-tests, the dbus client | 14:25 |
fgimenez | elopio, it fails trying to find com.canonical.SystemImage, do you know why? | 14:26 |
elopio | fgimenez: I don't. barry is your friend. | 14:27 |
longsleep | Hi folks, when running Â`snappy update` it fails with `Unable to determine bootloder`. Any help? I am new to snappy, managed to build my own image though. | 14:27 |
sergiusens | mterry: you want to change channels.ini and then run snappy update | 14:27 |
* barry is everyone's friend | 14:28 | |
mterry | sergiusens, yeah I figured it out and it's working now | 14:28 |
mterry | sergiusens, not friendly experience, having to remount and all that jazz | 14:28 |
* mterry wants s-i to work | 14:28 | |
ogra_ | heh, what for, it is going away | 14:28 |
sergiusens | jdstrand: good then, I was just going down the path something incompatible changing, but it seems the rule is incorrect; I'll propose an MP | 14:28 |
mterry | ogra_, well then I want its replacement to work :) | 14:29 |
sergiusens | mterry: it's not friendly at all; but this is all influx now (channels and os/kernel updates) | 14:29 |
mterry | sergiusens, yeah I know. humph | 14:29 |
sergiusens | mterry: the reason for the document to exist is its unfriendliness :-) | 14:29 |
ogra_ | mterry, the replacement wont know about channels anymore :) | 14:29 |
mterry | ogra_, oh really? What's the replacement? | 14:30 |
mterry | I heard s-i was on the outs but didn't hear what's new | 14:30 |
ogra_ | mterry, snaps are the replacement | 14:30 |
fgimenez | elopio, ok :) barry, it seems that part of the code that we used in ubuntu-ota-tests doesn't work on snappy running in kvm, seems to be unable to find com.canonical.SystemImage | 14:30 |
ogra_ | mterry, the whole image will come from the store in the future | 14:30 |
mterry | ogra_, ... but channels still exist with snaps | 14:30 |
ogra_ | assembled from snaps | 14:31 |
mterry | hm | 14:31 |
barry | fgimenez: you mean, it can't find the dbus service? | 14:31 |
fgimenez | barry, yes, wait, i'll paste the error | 14:31 |
sergiusens | longsleep: you are missing either a proper entry in the gadget snap (called oem in current releases), valid content in /boot/uboot or /boot/grub ... if you built your image manually make sure you have everything correctly | 14:31 |
sergiusens | ogra_: channels will be per snap; in a sense, we are splitting the concept of channel and release (and moving away from customization channel as well) | 14:32 |
longsleep | sergiusens: Thanks. Is there any way to debug this - i have put my image builder up at https://github.com/longsleep/snappy-odroidc - i would like to see it working 100% | 14:33 |
barry | fgimenez: i bet i know why, but do pastebin the error | 14:33 |
ogra_ | sergiusens, right, so there wont be a way to switch | 14:33 |
ogra_ | what you build from is your "channel" | 14:33 |
sergiusens | longsleep: I'm working on release activities so I can't help today, but maybe ogra_ can ;-) | 14:33 |
mterry | Where is the docker snappy packaging? | 14:33 |
* sergiusens tries to deflect ogra_ from asking hard questions | 14:33 | |
sergiusens | or weird statements :-P | 14:34 |
longsleep | sergiusens: all right - thanks for the hints though :) | 14:34 |
ogra_ | lol ... well, i'm up to my ears in the RPi2 image currently | 14:34 |
sergiusens | mterry: it's in snappy-hub somewhere | 14:34 |
mterry | awesome | 14:34 |
sergiusens | mterry: https://code.launchpad.net/~snappy-dev/snappy-hub/docker | 14:34 |
mterry | yup, thanks | 14:34 |
* mterry hugs sergiusens | 14:34 | |
fgimenez | barry, here it is http://paste.ubuntu.com/11589316/ | 14:35 |
* sergiusens hugs back | 14:35 | |
longsleep | If anyone wants to take a look, https://github.com/longsleep/snappy-odroidc/blob/master/oem/meta/package.yaml, https://github.com/longsleep/snappy-odroidc/blob/master/device/hardware.yaml - feedback highly appreciated | 14:35 |
barry | fgimenez: does /usr/sbin/system-image-dbus exist? | 14:36 |
fgimenez | barry, nope, no system-image-* under /usr/sbin/ | 14:37 |
barry | fgimenez: okay, so here's what i think is going on... | 14:37 |
barry | fgimenez: while system-image 3.0 was working its slow way to landing in the archive, mvo forked the project for snappy. but he didn't need the dbus api so i believe the fork does not install system-image-dbus. the -cli gained some additional functionality so that mvo wouldn't have to call into the dbus service, which he didn't like (he can give you rationale for that). what i think needs to happen is to *un*fork si 3.0 for snappy, | 14:40 |
barry | and then install both the system-image-cli and system-image-dbus (and of course system-image-common) packages into the base os. i think mvo agrees, at least with the unforking, but it hasn't been high enough on his list to get to yet. si 3.0 is in wily now so there should be no blockers other than round tuits to doing the unfork | 14:40 |
longsleep | sergiusens, ogra_ : So valid content in /boot/uboot .. looks good to me "a b boot.ini snappy-system.txt" | 14:41 |
barry | well, maybe if you need si 3.0 in another channel, but it should be backportable | 14:42 |
ogra_ | longsleep, http://people.canonical.com/~ogra/snappy/odroidc/ thats very old but i think the file contents in the device tarball should still be fine | 14:43 |
fgimenez | barry, ok thanks a lot, for now i think that we can go ahead with the current state, but we'll probably need si 3.0 sooner or later, what do you think elopio? | 14:45 |
barry | fgimenez: you'll definitely want si 3.0. i guess when mvo gets back online we can talk about a plan to unfork | 14:46 |
longsleep | ogra_: Thanks - only difference to mine i see is that i am not using flashtool-assets and uEnv.txt (shipping boot.ini instead). Booting works, maybe the snappy tool checks on uEnv.txt on update or something? | 14:47 |
ogra_ | longsleep, snapopy requites uEnv.txt in the right place (at least it did once) to recognize the system as booted from uBoot | 14:48 |
longsleep | ogra_: well it does not require it to boot - but maybe some tools check for it. | 14:50 |
ogra_ | thats what i meant | 14:50 |
ogra_ | snappy expects it | 14:50 |
longsleep | ogra_: So you think its worth a shot to ship an empty uEnv.txt and update might work? | 14:50 |
ogra_ | longsleep, update of what exactly ? | 14:51 |
ogra_ | snappy update ubuntu.-core cant work ... unless the image is on the official system-image server (i think at least) | 14:51 |
longsleep | ogra_: sorry - running 'snappy update' fails with Unable to determin bootloader | 14:51 |
ogra_ | try: touch /boot/uboot/uEnv.txt | 14:52 |
ogra_ | (well, with sudo) | 14:52 |
barry | fgimenez: maybe send an email to the relevant parties and we can discuss an unfork plan/schedule | 14:52 |
longsleep | ogra_: yeah that does something now | 14:52 |
longsleep | ogra_: soe somewhere seems to be a check for uEnv.txt | 14:53 |
ogra_ | yes, as i said above | 14:53 |
longsleep | ogra_: Thanks for the help - i will add an empty one to the image builder for now. | 14:54 |
ogra_ | <ogra_> longsleep, snapopy requites uEnv.txt in the right place (at least it did once) to recognize the system as booted from uBoot | 14:54 |
ogra_ | ;) | 14:54 |
longsleep | ogra_: yeah - i got it now - you meant the 'snappy tool' - for me snappy was the whole system. Misunderstanding on my side sorry. | 14:55 |
ogra_ | longsleep, lolo, no, bad naming on our side :) | 14:55 |
ogra_ | *lol even | 14:55 |
ogra_ | we just call everything snappy nowadays :) | 14:55 |
fgimenez | barry, ack, thanks! | 14:57 |
beuno | my dog responds to snappy now as well | 14:57 |
longsleep | ogra_: hehe i think things could be worse than that. Snappy is a nice word too - i like it. | 14:57 |
longsleep | ogra_: its at Applying update now - so things look good! | 14:57 |
ogra_ | yay | 14:58 |
longsleep | ogra_: lets see if it reboots though :) | 14:58 |
ogra_ | it should just ignore the uEnv.txt | 14:58 |
longsleep | ogra_: It booted just fine - awesome! | 14:59 |
ogra_ | :D | 14:59 |
mattyw | dholbach, ping? | 15:02 |
dholbach | mattyw, pong | 15:05 |
mattyw | dholbach, hey there, just looking at your email about improving the snappy docs, I was about to submit a bug for a nitcpick, but I thought I'd ping instead | 15:07 |
dholbach | either way you like it | 15:12 |
mattyw | dholbach, the instructions for snappy-remote on this page https://developer.ubuntu.com/en/snappy/tutorials/build-snaps/ states 8022 for the port to use in ssh, but that's only the port used when doing the redirecting described using the kvm image. I'd be tempted to state explicitly that the port would just be 22 if you were using it directly | 15:12 |
dholbach | ok, I'll file a bug | 15:12 |
dholbach | thanks for reporting | 15:12 |
dholbach | bug 1462408 | 15:13 |
ubottu | bug 1462408 in Ubuntu Developer Portal "Correct snappy-remote port" [Undecided,New] https://launchpad.net/bugs/1462408 | 15:13 |
mattyw | dholbach, it seems obvious, but there's a lot of new stuffy in snappy, it's nice to know that there are some things that aren't new :) | 15:13 |
mattyw | dholbach, thanks very much for listening :) | 15:13 |
dholbach | yeah... thanks a bunch for spotting it :) | 15:14 |
sergiusens | mattyw: dholbach more to the point, not putting a port defaults to 22 | 15:20 |
elopio | fgimenez: yes, please send the email. | 15:20 |
elopio | fgimenez: sounds like remerging is important. And if mvo has reasons not to use the dbus api, maybe on the tests we should use the cli too. Lets wait for his reply about why... | 15:21 |
elopio | fgimenez: did you report any bugs from the exploratory testing? | 15:21 |
sergiusens | elopio: fgimenez no, we don't use the dbus api, we took great lengths to remove it | 15:21 |
sergiusens | elopio: fgimenez sorry, I forgot to mention that in the standup | 15:22 |
fgimenez | sergiusens, me too :) then, what should we use instead? | 15:23 |
sergiusens | fgimenez: system-image cli in json writer mode | 15:23 |
sergiusens | fgimenez: system-image-cli --machine-readable | 15:24 |
sergiusens | fgimenez: but I'm not sure what you want to do either | 15:24 |
fgimenez | sergiusens, yep, i've seen that through the code | 15:25 |
fgimenez | sergiusens, we have code we used for testing ota on touch, that relies in the dbus api for querying and calling methods | 15:26 |
fgimenez | sergiusens, if we want to reuse some of that we need to adapt the calls to system-image-cli | 15:26 |
sergiusens | fgimenez: yes, I think you want that; in any case, I'm more worried about the core upgrader logic more than the s-i side of things | 15:27 |
sergiusens | fgimenez: as in testing the core upgrader (which just cares about the files being in the right place) | 15:27 |
sergiusens | fgimenez: also, if you use dbus, you won't be testing with what snappy uses at all | 15:28 |
fgimenez | sergiusens, ok thanks, of course the tests should exercise the real system as is, doesn't make sense use dbus then | 15:37 |
fgimenez | elopio, no more bugs from the exploratory testing, now i'm with the "write snappy app" document | 15:38 |
kyrofa | sergiusens, ping | 15:42 |
sergiusens | kyrofa: | pong | 15:42 |
rsalveti | fgimenez: great (just saw the message you got the boards) | 15:43 |
kyrofa | sergiusens, currently the webdm API doesn't contain any information regarding snap cost, so my scope currently has "FREE" hard-coded in. Does the store support non-free snaps? | 15:43 |
elopio | sergiusens: so, we are making the release today? | 15:44 |
sergiusens | kyrofa: non free snaps for snappy doesn't exist yet, we don't have an implementation in snappy (client) itself either | 15:44 |
sergiusens | kyrofa: we know what needs to be done though | 15:44 |
sergiusens | kyrofa: just needs to happen | 15:44 |
kyrofa | sergiusens, good deal, I just wanted to touch base on it | 15:45 |
fgimenez | rsalveti, yep :) i haven't put hands on them yet | 15:45 |
sergiusens | kyrofa: I guess that rest api needs some concept (planned at least to deal with this) | 15:46 |
kyrofa | sergiusens, good call | 15:47 |
elopio | fgimenez: /me tries the webcam appliance builder. | 15:49 |
fgimenez | elopio, ok, you may hit the apparmor issue about the /tmp/snaps/ directory | 15:52 |
rsalveti | elopio: we currently have 3 known blocking things for the release | 15:58 |
rsalveti | let me find the links | 15:59 |
rsalveti | https://bugs.launchpad.net/ubuntu-core-launcher/+bug/1460517 | 15:59 |
ubottu | Ubuntu bug 1460517 in Snappy Launcher 15.04 "ubuntu-core-launcher apparmor denial when creating /tmp/snaps" [Undecided,New] | 15:59 |
rsalveti | https://bugs.launchpad.net/snappy/+bug/1460152 | 15:59 |
ubottu | Ubuntu bug 1460152 in Snappy 15.04 "apparmor cache not updated when apparmor.d rules change (breaks 15.04/stable -> 15.04/edge updates)" [Critical,In progress] | 15:59 |
rsalveti | and investigate if we can land https://trello.com/c/YkdrYyX6/79-rootdelay-300-for-azure-images | 16:00 |
rsalveti | fgimenez: did you find any other major issue with latest image? | 16:00 |
fgimenez | rsalveti, nope so far | 16:00 |
rsalveti | fgimenez: I also built 2 extra armhf ones yesterday to align the image number, just to make it easier to identify the version | 16:00 |
rsalveti | great | 16:01 |
elopio | rsalveti: thanks, that's useful. | 16:02 |
elopio | rsalveti: do we need to do some testing on raspberrypi? | 16:02 |
rsalveti | elopio: not yet, we're waiting ogra_ to produce a newer image first | 16:02 |
fgimenez | leaving, have a nice weekend everyone o/ | 16:08 |
sergiusens | rsalveti: given that https://code.launchpad.net/~mvo/ubuntu/vivid/ubuntu-core-config/lp1460152-workaround/+merge/261179 is an ubuntu package I will need your help | 16:15 |
sergiusens | rsalveti: but I approved it as it's working as expected | 16:15 |
rsalveti | sergiusens: sure | 16:15 |
rsalveti | sergiusens: I can take care of uploading that to our ppa | 16:15 |
rsalveti | and rolling/wily | 16:15 |
sergiusens | rsalveti: not sure what the procedure is here; since it is an ubuntu package does it require an SRU to merge? | 16:16 |
rsalveti | sergiusens: yeah, this is the udd branch | 16:16 |
rsalveti | I'll just upload to wily, then to our ppa and have another wi for next week to do SRUs for our PPA changes | 16:17 |
sergiusens | rsalveti: the origin is vivid but the target is trunk so it's all so confusing :-) | 16:17 |
Chipaca | back! sorry for this. How can I help? | 16:17 |
* Chipaca reads a bit of backlog | 16:17 | |
sergiusens | Chipaca: sorry for what? | 16:17 |
Chipaca | sergiusens: being late back | 16:17 |
sergiusens | Chipaca: oh, not a problem :-) | 16:17 |
rsalveti | there are 2 main things now | 16:17 |
rsalveti | bug https://bugs.launchpad.net/ubuntu-core-launcher/+bug/1460517 | 16:17 |
ubottu | Ubuntu bug 1460517 in Snappy Launcher 15.04 "ubuntu-core-launcher apparmor denial when creating /tmp/snaps" [Undecided,New] | 16:18 |
rsalveti | and https://trello.com/c/YkdrYyX6/79-rootdelay-300-for-azure-images | 16:18 |
sergiusens | rsalveti: yeah, going to propose the fix for the core launcher now | 16:18 |
rsalveti | if you flash latest, then install webdm and try to install snaps it will eventually fail | 16:18 |
rsalveti | great | 16:18 |
sergiusens | rsalveti: I bet eventually means after rebooting? | 16:18 |
rsalveti | sergiusens: don't remember if I rebooted or not | 16:19 |
sergiusens | Chipaca: want to take a look at the trello one? | 16:19 |
rsalveti | but can easily test again | 16:19 |
rsalveti | let me upload the other package and will give it a try | 16:19 |
sergiusens | rsalveti: I bet it's an ordering issue | 16:19 |
Chipaca | already looking | 16:19 |
rsalveti | right | 16:19 |
Chipaca | sergiusens: doesn't 15.04 have private /tmp with this update? | 16:19 |
Chipaca | sergiusens: https://bugs.launchpad.net/ubuntu-core-launcher/+bug/1460517 is about webdm creating /tmp/snaps without -m 01777, right? | 16:20 |
ubottu | Ubuntu bug 1460517 in Snappy Launcher 15.04 "ubuntu-core-launcher apparmor denial when creating /tmp/snaps" [Undecided,New] | 16:20 |
=== tvoss is now known as tvoss|test | ||
=== tvoss|test is now known as tvoss | ||
Chipaca | gah, dunno | 16:21 |
Chipaca | anyway, rootdelay. sure. | 16:21 |
Chipaca | rootdelay=300 according to the docs ask the kernel to wait 5 minutes before mounting root | 16:22 |
Chipaca | that seems excessive | 16:22 |
Chipaca | rootdelay=[KNL] Delay (in seconds) to pause before attempting to | 16:23 |
Chipaca | mount the root filesystem | 16:23 |
sergiusens | rsalveti: with the new way for tmpdirs to be created, webdm and any other app should have independent paths | 16:23 |
sergiusens | Chipaca: yeah, we need to see if that can be added only for azure images | 16:23 |
sergiusens | Chipaca: either in u-d-f or in livecd-rootfs | 16:23 |
Chipaca | people really think waiting 5 minutes for a cloud image is reasonable? | 16:23 |
* Chipaca boggles | 16:23 | |
rsalveti | utlemming: ^? | 16:23 |
rsalveti | 300 is indeed a lot | 16:24 |
Chipaca | rsalveti: you know you suck at this "taking the day off" thing, right? | 16:24 |
rsalveti | Chipaca: I'm doing that, not being a manager today :P | 16:25 |
rsalveti | but in reality just waiting to get off for lunch | 16:25 |
Chipaca | :) | 16:26 |
rsalveti | sergiusens: comparing core-launcher from wily and from our ppa, the only extra thing we have at wily is "Allow executing from /frameworks" | 16:27 |
rsalveti | sergiusens: is that wily only? | 16:27 |
sergiusens | rsalveti: that is wily only | 16:27 |
rsalveti | great | 16:27 |
sergiusens | rsalveti: until we decide to do a big backport ;-) | 16:28 |
rsalveti | right :-) | 16:28 |
Chipaca | jdstrand: once /tmp is a (private) bind mount, do apparmor rules apply to the path as the process "sees" it? | 16:30 |
Chipaca | if so, /tmp/ should be "do whatever" | 16:30 |
Chipaca | although you're talking about the launcher rules | 16:31 |
Chipaca | and the launcher is not the one mkdir'ing /tmp/snaps/ | 16:31 |
Chipaca | jdstrand: either I am confused, or you are, or a linear combination of both | 16:31 |
jdstrand | Chipaca: this is for the laucnher | 16:32 |
Chipaca | jdstrand: right; the launcher doesn't mkdir /tmp/snaps; it mkdir's /tmp/snap.%d_%s_XXXXXX | 16:32 |
Chipaca | the "permission denied" for /tmp/snaps/ is for a pre-private-/tmp | 16:33 |
jdstrand | the launcher is trying to make /tmp/snaps | 16:33 |
Chipaca | is not :) | 16:33 |
jdstrand | the launcher that is causing that denial that is | 16:33 |
jdstrand | I don't know what launcher that is | 16:33 |
Chipaca | jdstrand: the thing that would mkdir /tmp/snaps was the wrapper, not the launcher | 16:34 |
Chipaca | unless you're saying there were actualy apparmor denies, it's just regular unix permissions | 16:35 |
Chipaca | because webdm would mkdir /tmp/snaps/ without setting it to 01777 | 16:35 |
Chipaca | then anything after that would fail | 16:35 |
jdstrand | https://bugs.launchpad.net/snappy/+bug/1460517 | 16:36 |
ubottu | Ubuntu bug 1460517 in Snappy Launcher 15.04 "ubuntu-core-launcher apparmor denial when creating /tmp/snaps" [Undecided,New] | 16:36 |
sergiusens | Chipaca: people are complaining that the latest webdm has these issues though | 16:36 |
sergiusens | Chipaca: and the latest webdm has 01777 in it | 16:36 |
jdstrand | meh, the denial isn't in the bug | 16:36 |
* jdstrand hunts it down | 16:36 | |
Chipaca | sergiusens: if they didn't reboot after updating, the bug will still be there | 16:37 |
sergiusens | Chipaca: right, that could be it | 16:37 |
Chipaca | sergiusens: because (i presume) webdm doesn't clean up the old bug, just stops creating it | 16:37 |
jdstrand | this is the reported denial: | 16:37 |
jdstrand | Jun 3 11:20:10 localhost kernel: [ 134.805380] audit: type=1400 | 16:37 |
jdstrand | audit(1433330410.595:12): apparmor="DENIED" operation="mkdir" | 16:37 |
jdstrand | profile="/usr/bin/ubuntu-core-launcher" name="/tmp/snaps/" pid=895 | 16:37 |
jdstrand | comm="ubuntu-core-lau" requested_mask="c" denied_mask="c" fsuid=0 ouid=0 | 16:37 |
Chipaca | however | 16:37 |
jdstrand | profile="/usr/bin/ubuntu-core-launcher" | 16:37 |
Chipaca | the new launcher, with private /tmp/, should make it all just go away | 16:37 |
rsalveti | failing | 16:38 |
rsalveti | <rsalveti> Jun 5 05:08:14 localhost kernel: [ 39.097383] audit: type=1400 audit(1433480894.752:22): apparmor="DENIED" operation="mkdir" profile="/usr/bin/ubuntu-core-launcher" name="/tmp/snaps/" pid=1024 comm="ubuntu-core-lau" requested_mask="c" denied_mask="c" fsuid=0 ouid=0 | 16:38 |
rsalveti | what I had earlier today | 16:38 |
jdstrand | also, profile="/usr/bin/ubuntu-core-launcher" | 16:38 |
sergiusens | Chipaca: jdstrand on edge I don't see this issue and my tmp's are /tmp/snap.0_hello-dbus-fwk_JuFKMH snap.0_snake.mectors_Dfz11P | 16:38 |
sergiusens | no /tmp/snaps/ at all | 16:39 |
rsalveti | the bug I had was on a clean image | 16:39 |
rsalveti | just installed some snaps, then installed webdm, then tried to install additional snaps from the webdm interface | 16:39 |
rsalveti | I did reboot once, just don't remember exactly when | 16:39 |
sergiusens | rsalveti: are you perhaps mixing up bugs? | 16:39 |
Chipaca | rsalveti: what versions of everything? | 16:40 |
rsalveti | let me try to reproduce again | 16:40 |
sergiusens | rsalveti: there is no /tmp/snaps on edge | 16:40 |
rsalveti | image 78 | 16:40 |
rsalveti | that was the denial I had | 16:41 |
rsalveti | but anyway, let me try reproducing it | 16:41 |
Chipaca | sergiusens: well, well. well. | 16:42 |
sergiusens | Chipaca: what did I do now? :( | 16:42 |
Chipaca | there *is* a /tmp/snaps, *if* apparmor sees it as from "inside" | 16:43 |
Chipaca | that is: the launcher mounts a private /tmp/ and then creates /tmp/snaps inside that | 16:43 |
Chipaca | this is after dropping privs, fwiw | 16:43 |
Chipaca | so if we're seeing DENIEDs with the new code, that's why | 16:44 |
Chipaca | (face. desk. etc.) | 16:44 |
sergiusens | Chipaca: but that's up to the profile of the snappy package itself, not the core launcher | 16:44 |
Chipaca | no, this is still in the launcher | 16:44 |
sergiusens | Chipaca: so we need to allow for both | 16:44 |
Chipaca | before the aa_change_onexec call even | 16:45 |
Chipaca | sergiusens: which is what jdstrand said | 16:45 |
Chipaca | man, he's so ahead of us, he should buy us beer | 16:45 |
sergiusens | Chipaca: so this is good /tmp/snap{s,.*}/ w, | 16:45 |
jdstrand | wait, if I'm ahead, why am I buying beer? | 16:45 |
jdstrand | :P | 16:45 |
Chipaca | jdstrand: you arrived at the bar first | 16:46 |
rsalveti | lol | 16:46 |
jdstrand | ah | 16:46 |
jdstrand | I figured you'd buy my beer since I patiently waited for you | 16:46 |
jdstrand | :) | 16:46 |
rsalveti | 32.40 KB/s from the store | 16:46 |
ogra_ | did someone mention free beer ? | 16:46 |
rsalveti | annoying | 16:46 |
Chipaca | ogra_: that's what i heard | 16:46 |
* ogra_ thought he got a highlight | 16:46 | |
Chipaca | sergiusens: in case an answer was necessary: yes, that should be perfect | 16:47 |
sergiusens | Chipaca: https://code.launchpad.net/~sergiusens/ubuntu-core-launcher/tmpmor/+merge/261252 | 16:49 |
sergiusens | there then | 16:49 |
jdstrand | sergiusens, Chipaca: thanks for taking care of this | 16:49 |
Chipaca | sergiusens: there ya go | 16:50 |
* rsalveti kicks the store | 16:50 | |
sergiusens | Chipaca: heh, tarmac chose to go nuts :/ | 16:51 |
jdstrand | sergiusens: with that merge, what is TMPDIR set to? and, are apps able to write to it? | 16:52 |
jdstrand | sergiusens: I'm thinking 'no' and there will need to be a corresponding ubuntu-core-security update | 16:53 |
sergiusens | jdstrand: http://paste.ubuntu.com/11591737/ | 16:54 |
sergiusens | jdstrand: wrt to this whole movement, Chipaca and tyhicks where managing it | 16:54 |
Chipaca | jdstrand: i think u-c-security still has /tmp/snaps/app.origin/tmp/ | 16:55 |
Chipaca | jdstrand: and AIUI that will work | 16:55 |
jdstrand | Chipaca: it for sure does | 16:55 |
Chipaca | jdstrand: we could/should drop it to all of /tmp/ later down th eline | 16:55 |
Chipaca | jdstrand: but, baby steps | 16:55 |
jdstrand | ah | 16:55 |
jdstrand | that was the bit I was missing | 16:55 |
Chipaca | the whole idea of making the old tmp was to not have to do the update in lockstep | 16:55 |
jdstrand | ok, no I am catching up :) | 16:55 |
jdstrand | see, this little game we are playing is fun | 16:56 |
jdstrand | gotcha | 16:56 |
* Chipaca never likes it when security people talk about "game" and "fun" | 16:56 | |
jdstrand | so, we need a policy update whenever we set TMPDIR to /tmp | 16:56 |
Chipaca | s/whenever/before/ | 16:57 |
Chipaca | :) | 16:57 |
jdstrand | yeah | 16:57 |
jdstrand | :) | 16:57 |
jdstrand | let me know when you are working on that :) | 16:57 |
* Chipaca 'll try | 16:57 | |
elopio | ok, I'm tired. I'll EOD. | 16:58 |
rsalveti | Jun 5 16:58:21 localhost kernel: [ 865.945352] audit: type=1400 audit(1433523501.755:19): apparmor="DENIED" operation="mkdir" profile="/usr/bin/ubuntu-core-launcher" name="/tmp/snaps/" pid=4059 comm="ubuntu-core-lau" requested_mask="c" denied_mask="c" fsuid=0 ouid=0 | 16:58 |
elopio | will check back later, and continue testing on monday. | 16:58 |
rsalveti | jdstrand: Chipaca: sergiusens: when trying to install snake from webdm | 16:58 |
elopio | enjoy your day. | 16:58 |
jdstrand | yes | 16:58 |
jdstrand | rsalveti: yes | 16:58 |
rsalveti | first boot -> installed webdm -> from webdm tried to install snake | 16:59 |
rsalveti | but it installed fine | 16:59 |
jdstrand | rsalveti: the launcher is creating /tmp/snaps inside the private mount before the change profile | 16:59 |
rsalveti | right | 16:59 |
jdstrand | rsalveti: so we need the fix. sergiusens did an MP and it is moving forward now | 16:59 |
rsalveti | yeah, makes sense now | 17:02 |
* rsalveti is still getting used with all this | 17:02 | |
rsalveti | but this is so freaking awesome | 17:02 |
rsalveti | elopio: have a nice weekend! | 17:03 |
rsalveti | Jun 5 17:03:38 localhost ubuntu-core-launcher[703]: 2015-06-05 17:03:38 ERROR snappy logger.go:199 system-status.victor failed to install: package name with namespace not supported | 17:04 |
elopio | you too. | 17:04 |
rsalveti | namespace? | 17:04 |
sergiusens | rsalveti: if it's a framework, it's not allowed | 17:05 |
sergiusens | if not, not sure | 17:05 |
rsalveti | it's not a framework | 17:05 |
sergiusens | rsalveti: that package has a dot in it's package name most likely | 17:05 |
rsalveti | hm, right | 17:05 |
jdstrand | mterry: fyi, I don't recall if I mentioned it, but inotify_* calls were add to wily and 15.04/edge some time ago | 17:05 |
rsalveti | wonder if we should have any sort of store validation | 17:06 |
rsalveti | to see if the user can at least install it | 17:06 |
mterry | jdstrand, oh nice... | 17:06 |
rsalveti | having broken snaps in there is kind of annoying | 17:06 |
sergiusens | rsalveti: it shouldn't be there if so, maybe beuno knows | 17:06 |
jdstrand | note, the new review tools are going to show an error if there is a dot in the yaml 'name' field | 17:07 |
jdstrand | of course, I got totally sidetracked from them this week (unfortunately) | 17:07 |
sergiusens | rsalveti: care to do some dputting for https://code.launchpad.net/~snappy-dev/ubuntu-core-launcher/trunk ? | 17:08 |
rsalveti | sergiusens: sure | 17:08 |
* sergiusens takes advantage of rsalveti's dput service :-) | 17:09 | |
beuno | rsalveti, sergiusens, what what? | 17:13 |
rsalveti | beuno: system-status.victor snap is failing to even install | 17:14 |
rsalveti | but it's available in the store | 17:14 |
rsalveti | so was wondering about ways to validate that before even making it available in there | 17:14 |
beuno | rsalveti, I don't think you get to control what crazy stuff people put in the store | 17:15 |
beuno | :) | 17:15 |
rsalveti | beuno: that's fine, but we can do some sort of validation | 17:15 |
rsalveti | beuno: if it fails to even install | 17:15 |
rsalveti | why should we publish it? | 17:15 |
beuno | rsalveti, we just run the review scripts | 17:15 |
beuno | those don't try to install | 17:15 |
rsalveti | right, then what jdstrand said should fix it at least | 17:15 |
rsalveti | for this case | 17:15 |
beuno | that's a heavy-weight process, I think, trying to install | 17:16 |
beuno | we can, and I'm sure we will | 17:16 |
beuno | but it's not trivial | 17:16 |
rsalveti | right, later as part of CI and so on | 17:16 |
beuno | no, not CI at all | 17:16 |
beuno | people will upload random broken stuff that they won't pay for CI | 17:16 |
rsalveti | sure, in this case, yes | 17:16 |
rsalveti | guess channels will at least help | 17:16 |
beuno | so you probably want ratings and reviews as part of this soon | 17:17 |
beuno | so people can give feedback to the developer | 17:17 |
rsalveti | yup | 17:17 |
beuno | and hit at others that it's broken | 17:17 |
rsalveti | sergiusens: will you propose an MR for https://code.launchpad.net/~snappy-dev/ubuntu-core-launcher/15.04 as well? | 17:19 |
rsalveti | ubuntu-core-config and ubuntu-core-launcher uploaded to wily | 17:20 |
rsalveti | core-config also uploaded to our ppa | 17:20 |
rsalveti | just missing core-launcher | 17:20 |
sergiusens | rsalveti: ah, got it | 17:21 |
rsalveti | sergiusens: I can create it | 17:25 |
rsalveti | the mr | 17:25 |
rsalveti | sergiusens: Chipaca: https://code.launchpad.net/~rsalveti/ubuntu-core-launcher/15.04-tmpmor/+merge/261254 | 17:27 |
rsalveti | similar to https://code.launchpad.net/~sergiusens/ubuntu-core-launcher/tmpmor/+merge/261252 | 17:27 |
rsalveti | if all good I can upload that and trigger a new image | 17:27 |
sergiusens | rsalveti: oh, I just created the same :-P | 17:28 |
Chipaca | rsalveti: I approve. | 17:28 |
Chipaca | sergiusens: go have lunch | 17:28 |
sergiusens | yay | 17:28 |
rsalveti | yeah, thought you were having lunch already :-) | 17:28 |
* sergiusens will go have lunch | 17:28 | |
rsalveti | Chipaca: thanks | 17:28 |
rsalveti | Chipaca: do we have auto-merge and so on for this guy? | 17:29 |
sergiusens | rsalveti: Chipaca I just set it up | 17:29 |
rsalveti | great | 17:30 |
jdstrand | rsalveti: heh, I was going through irc trying to see where I wrote up my suggestions on a tutorial for creating/debugging snaps and I only just now saw you asked me to dump the info in https://wiki.ubuntu.com/Snappy/ somewhere. I did so now: https://wiki.ubuntu.com/Snappy/Debugging | 17:35 |
jdstrand | rsalveti: the other day I wanted to give this to dholbach and the docs team, but now thinking it needs a card. what is your opinion? | 17:36 |
zyga | can snapps publish systemd units (e.g. a timer unit) | 17:44 |
zyga | or in other words, how do I write a simple snap that does something every $interval | 17:44 |
zyga | just loop manually | 17:44 |
zyga | or is there a better way? | 17:44 |
Chipaca | zyga: we don't expose an easy way for that | 17:46 |
Chipaca | zyga: the systemd unit we create for services has some hooks, but nothing that you could use as a timer afaik | 17:46 |
zyga | Chipaca: ok, so if I write a program that just waits by itself | 17:46 |
jdstrand | rsalveti: oh, I didn't realize you were off-- please ignore me here and check your inbox on monday :) | 17:47 |
zyga | Chipaca: and it crashes | 17:47 |
zyga | Chipaca: does snappy restart it via the unit? | 17:47 |
Chipaca | zyga: if it's a service, systemd should take care of that yes | 17:47 |
zyga | rsalveti: https://git.launchpad.net/~zyga/+git/pmr/tree/process-merge-requests | 17:47 |
zyga | Chipaca: cool, thanks | 17:47 |
Chipaca | zyga: what does a timer unit look like? | 17:47 |
zyga | rsalveti: very early stuff, tarmac will be my target next week | 17:47 |
Chipaca | maybe we want to expose that | 17:47 |
zyga | Chipaca: it's a cron replacement unit, let me give you a link | 17:47 |
zyga | Chipaca: http://www.freedesktop.org/software/systemd/man/systemd.timer.html | 17:48 |
zyga | Chipaca: it has some nice features | 17:48 |
Chipaca | ah, so systemd.timer(5) | 17:48 |
Chipaca | nice | 17:48 |
zyga | Chipaca: e.g. I'd love to be able to wake the machine up from suspend to do nightly maintenance | 17:48 |
Chipaca | zyga: that'd be cool, there are lots of things that could use this | 17:50 |
Chipaca | zyga: something for our architects :) | 17:50 |
zyga | yep | 17:50 |
* Chipaca resisting to ad-hoc it now that there are people in charge of not ad-hocing stuff | 17:50 | |
Chipaca | zyga: want to drop a card in the "incoming proposals" on the trello? | 17:52 |
Chipaca | zyga: https://trello.com/b/4PQViyUQ/snappy-core-stakeholders-backlog | 17:52 |
zyga | Chipaca: wow, gladly | 17:52 |
Chipaca | personal will be needing this anyway | 17:53 |
Chipaca | i've got to go make dinner; sergiusens, will bbl to carry on with the webcam thing | 18:01 |
zyga | Chipaca: I cannot add anything to the board there, sorry | 18:12 |
sergiusens | Chipaca: fwiw, snappy autopilot is a systemd timer unit | 18:42 |
sergiusens | rsalveti: did you trigger a build? | 19:25 |
hokkos | where is the package list of ubuntu snappy core ? | 20:26 |
kgunn | question on uploading a snap to the store....the country thing is confusing, it says pick countries you don't want to distribute in, pick countries to only distribute in... | 20:28 |
kgunn | does that mean there's a default if you don't select anythnig ? and what is that default ? | 20:28 |
kgunn | beuno: ^ | 20:29 |
beuno | kgunn, default, I think, is everything | 21:04 |
kgunn | k | 21:05 |
kgunn | mterry: you still about ? | 21:42 |
kgunn | was just curious about the 2 yaml files in mir, i get the mir-demo one...but i guess the second has me wondering | 21:46 |
kgunn | chat on monday i suppose | 21:46 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!