| mup | PR snapcraft#2795 opened: manifest: track and annotate `primed-stage-packages` <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/2795> | 00:25 |
|---|---|---|
| mwhudson | hm core refreshed and now google-play-music-desktop-player broke? | 01:49 |
| mborzecki | morning | 06:35 |
| zyga | Hey mborzecki | 06:38 |
| zyga | Please read the review I got from Jamie | 06:38 |
| zyga | We need to discuss this | 06:39 |
| mborzecki | zyga: hey hey, i've landed the fix for master yesterday, so please update your PRs if needed | 06:39 |
| zyga | Perhaps at 8:30? | 06:39 |
| zyga | mborzecki: will do | 06:39 |
| mborzecki | zyga: yeah, 830 sounds fine | 06:39 |
| mborzecki | zyga: duh, epel8 build failed on s390x :/ | 06:47 |
| zyga | mborzecki: order one to debug at home ;D | 06:50 |
| mborzecki | haha | 06:50 |
| zyga | mborzecki: I rebased and pushed on https://github.com/snapcore/snapd/pull/7726 | 06:51 |
| mup | PR #7726: RFC: change how snapd tracks processes <Created by zyga> <https://github.com/snapcore/snapd/pull/7726> | 06:51 |
| zyga | the other two can wait, we need to discuss them first | 06:51 |
| zyga | mborzecki: I also enabled f31 locally and confirmed 7726 *passes* there | 06:52 |
| mborzecki | zyga: nothing we can fix though https://kojipkgs.fedoraproject.org//work/tasks/4109/38804109/root.log Eighth_Doctor probably knows if this is being addressed | 06:53 |
| zyga | yeah, let's ignore that | 06:54 |
| zyga | brb | 07:06 |
| mborzecki | hmm mvo isn't around yet | 07:18 |
| * zyga adds test for https://github.com/snapcore/snapd/pull/7724 | 07:19 | |
| mup | PR #7724: cmd/snap-confine: tracking processes with classic confinement <Created by zyga> <https://github.com/snapcore/snapd/pull/7724> | 07:20 |
| zyga | mborzecki: I have a swap day for Friday | 07:20 |
| zyga | mborzecki: I'll get a swap day for Monday at the sprint | 07:20 |
| zyga | mborzecki: and two days for travel | 07:20 |
| zyga | mborzecki: but I have a feeling I wont ever take them :/ | 07:20 |
| zyga | that's nearly a swap week now | 07:20 |
| mborzecki | unlimited company vacation? :) | 07:20 |
| mborzecki | zyga: btw. with your tweek from yday, it looks like you're prepping up fro the 4-day working week, tryign to squeeze 5 days worth of work into 4 days :P | 07:21 |
| mborzecki | and the productivity is up :) | 07:21 |
| zyga | :D | 07:22 |
| zyga | I wish I had 4 day week | 07:22 |
| zyga | or that 5 hour workday | 07:22 |
| zyga | I feel tired lately | 07:22 |
| mborzecki | mvo: morning | 07:31 |
| zyga | hey mvo | 07:31 |
| mvo | hey mborzecki and zyga | 07:32 |
| zyga | mvo: it's been a long evening | 07:32 |
| mborzecki | mvo: pinged you in https://github.com/snapcore/snapd/pull/7721 could use your input there | 07:32 |
| mup | PR #7721: gadget: add support for hybrid partitioning schemas <Simple 😃> <⛔ Blocked> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7721> | 07:32 |
| zyga | https://github.com/snapcore/snapd/pull/7722 :) | 07:32 |
| mup | PR #7722: cmd/snap-confine: add sc_join_sub_group <Created by zyga> <https://github.com/snapcore/snapd/pull/7722> | 07:32 |
| mvo | mborzecki: thanks, I have a look | 07:32 |
| mup | PR snapd#7652 closed: o/ifacestate,interfaces,interfaces/policy: slots-per-plug: * <Needs Samuele review> <Priority 🏇> <Created by pedronis> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7652> | 07:42 |
| * zyga -> breakfast | 07:46 | |
| * zyga pushed a patch to https://github.com/snapcore/snapd/pull/7724 and now _really_ does breakfast | 07:54 | |
| mup | PR #7724: cmd/snap-confine: tracking processes with classic confinement <Created by zyga> <https://github.com/snapcore/snapd/pull/7724> | 07:54 |
| === pstolowski|afk is now known as pstolowski | ||
| pstolowski | mornings | 08:07 |
| mborzecki | pstolowski: hey | 08:08 |
| * zyga breaks to rest for a while | 08:50 | |
| * dot-tobias hey all | 09:06 | |
| zyga | hey dot-tobias :) | 09:08 |
| ackk | hi, I'm getting an error when running snap commands (and in the snap service as well) about meta/snap.yaml not being found. but the file is obviously there | 09:10 |
| ackk | this is from a "snap try" | 09:14 |
| zyga | ackk: is it visible from the preserved mount namespace? | 09:16 |
| zyga | ackk: to find out: sudo nsenter -m/run/snapd/ns/maas.mnt | 09:16 |
| zyga | ackk: stat /snap/maas/current/meta/snap.yaml | 09:16 |
| ackk | zyga, it's not maas this time :) | 09:18 |
| zyga | ackk: /o\ :D | 09:18 |
| ackk | zyga, stat fails | 09:18 |
| ackk | zyga, (it's canonical-rbac) | 09:18 |
| zyga | now you can explore what is there | 09:18 |
| ackk | zyga, https://paste.ubuntu.com/p/V9mn5GZztg/ | 09:19 |
| zyga | check what is mounted there | 09:20 |
| ackk | zyga, from within the ns or outside? | 09:20 |
| ackk | zyga, from outside I see nsfs on /run/snapd/ns/canonical-rbac.mnt type nsfs (rw) | 09:21 |
| zyga | ackk: inside, where it is broken please | 09:22 |
| dot-tobias | mvo: core 2.42.1 has been in candidate since Monday, will it get promoted to stable this week or did someone discover issues? Asking for an anxious stakeholder 😊 | 09:22 |
| ackk | zyga, https://paste.ubuntu.com/p/FG4g8mBRNX/ | 09:23 |
| mup | PR snapd#7727 opened: tests: improve TestDoPrereqRetryWhenBaseInFlight to fix occasional flakiness <Test Robustness> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7727> | 09:24 |
| zyga | ackk: and on the host? | 09:25 |
| ackk | zyga, https://paste.ubuntu.com/p/6GqMpSKWVt/ | 09:25 |
| ackk | zyga, note, this is in a bionic container | 09:25 |
| ackk | (on disco as host) | 09:25 |
| zyga | right | 09:26 |
| zyga | ideally compare what is mounted where you see the snap.yaml | 09:26 |
| zyga | and what is mounted where you don't | 09:26 |
| ackk | zyga, so the /snap directory inside is empty apart from those 2 files, which seems broken? | 09:29 |
| mvo | dot-tobias: it was planned to go to stable yesterday, I need to check what happend, worst case is that it gets promoted this monday | 09:35 |
| zyga | mborzecki: so | 09:45 |
| zyga | mborzecki, mvo: that branch can be scrapped | 09:46 |
| zyga | but we can do something else still that's supported by systemd | 09:46 |
| popey | ogra: https://snapcraft.io/beebeep - you gonna update that? looks like a new release in edge? | 09:46 |
| zyga | but snap-confine will depend on dbus | 09:46 |
| zyga | and will require to use dbus to talk to systemd to run anything that is not a service | 09:46 |
| zyga | and will need to know if something is a service or not | 09:46 |
| zyga | that's the status quo | 09:46 |
| mborzecki | zyga: hm reading #fedora-devel now | 09:47 |
| mborzecki | zyga: ehh, so dbus and scope then? | 09:47 |
| zyga | yes | 09:47 |
| zyga | no other way | 09:47 |
| mborzecki | damn, that sucks | 09:47 |
| zyga | I'm trying to pull some documentation that's better than just reading the source | 09:48 |
| mborzecki | zyga: so if i'm reading this right, fire up transient scope, wait for it, then move the process to that scope right? | 09:48 |
| zyga | yes | 09:48 |
| zyga | StartTransientUnit and AttachProcessesToUnit | 09:48 |
| zyga | and it should be a scope, not a slice, for the reasons I gave on #fedora-devel | 09:49 |
| mborzecki | hm not exactly something we can simulate as separate steps in systemd-run | 09:49 |
| zyga | mborzecki: wanna jump into this? :) | 09:49 |
| mborzecki | zyga: fixing up the f31 pr, but let's sync afterwards maybe? | 09:50 |
| zyga | sure | 09:50 |
| zyga | I was supposed to rest anyway | 09:50 |
| popey | ogra: in fact there's an even newer upstream release. 5.8.2 - you should setup automatic builds :D | 09:50 |
| mborzecki | zyga: btw. one scope per user right? | 09:50 |
| diddledan | if only there was a service for automated snap builds | 09:51 |
| mup | PR pc-amd64-gadget#23 opened: Add missing partitions and improve grub.cfg-recovery <Created by mvo5> <https://github.com/snapcore/pc-amd64-gadget/pull/23> | 09:51 |
| zyga | mborzecki: I don't know yet | 09:51 |
| zyga | mborzecki: I think ... not | 09:51 |
| zyga | mborzecki: I thik it's just one scope if I'm right | 09:51 |
| zyga | but let me write a prototype first | 09:51 |
| mborzecki | zyga: https://gist.github.com/bboozzoo/76b1535c93686a27bb7fdbaad0f560f7#user-process some notes from before | 09:51 |
| mborzecki | zyga: there's even a note about separate scope for snaps | 09:52 |
| dot-tobias | mvo: Thanks for the update! (re Core update release) | 10:07 |
| Chipaca | mvo: btw i've been puzzling over this 'have two grub.cfg' and i'm not sure i understand it | 10:53 |
| pstolowski | doh, "{\"error_list\":[{\"code\":null,\"message\":\"Nonce is missing or | 10:53 |
| Chipaca | pstolowski: where was that? | 10:54 |
| pstolowski | Chipaca: https://api.travis-ci.org/v3/job/608640768/log.txt | 10:54 |
| pstolowski | Chipaca: google:fedora-30-64:tests/main/selinux-lxd | 10:54 |
| Chipaca | pstolowski: log seems truncated | 10:55 |
| Chipaca | anyway, i guess we need to retry those :-/ | 10:55 |
| pstolowski | Chipaca: ah i restarted the job assuming .txt would be kept | 10:55 |
| Chipaca | no worries | 10:56 |
| ogra | popey, yeah, will update it... i asked upstream about taking it over but he didnt reply (we actually had an active conversation that somehow died from his side) | 11:01 |
| mvo | Chipaca: in a meeting right now | 11:02 |
| mvo | Chipaca: but happy to talk to talk after | 11:02 |
| popey | ogra: cool. i wanted to pimp it on the snapcraft social account | 11:02 |
| popey | but don't want to do that if it's not up to date | 11:02 |
| ogra | great, i'll test the edge version today and will promote it if it works | 11:02 |
| Chipaca | popey: ogra: maybe pimping it is the push needed for them to take it on? | 11:03 |
| ogra | yeah, perhaps | 11:03 |
| popey | well, once we pimp it, the numbers will rise | 11:03 |
| popey | it already has quite a few installs, but I think we can probably double it | 11:03 |
| ogra | the numbers are pretty high already ... i think its my most downloaded snap atm | 11:03 |
| popey | I had never heard of it before today | 11:03 |
| popey | did you know you can tell if it's your most downloaded by going to https://snapcraft.io/snaps | 11:04 |
| ogra | (an it still has one prob ... it doesnt beep :P ... ) | 11:04 |
| popey | it shows a graph at the top | 11:04 |
| ogra | yeah | 11:04 |
| popey | https://usercontent.irccloud-cdn.com/file/IQu0dsu0/Screenshot%20from%202019-11-07%2011-04-27.png | 11:04 |
| ogra | 90k ? | 11:05 |
| ogra | whats that ? | 11:05 |
| Chipaca | ogra: popey: maybe update the description so it doesn't say you need to unzip it? | 11:05 |
| popey | yeah :) | 11:05 |
| ogra | one of yours | 11:05 |
| ogra | Chipaca, lol, thanks, will do | 11:05 |
| popey | no, not mine. just one i have access to | 11:05 |
| popey | I bet diddledan has a nice looking graph on his page :) | 11:06 |
| ogra | (that snap was actually the result of an ubuntu-users ML discussion... i only invested like 30min into it to prove how quick one can make a snap out of 3rd party SW) | 11:06 |
| popey | (higher than any of mine) | 11:06 |
| popey | nice | 11:06 |
| popey | put another 30 mins in :) | 11:06 |
| ogra | nah, rather 1h or two | 11:06 |
| ogra | it never really got polished | 11:06 |
| popey | is it a pulseaudio problem? | 11:06 |
| popey | ok, I won't pimp it till you get audio and the theme looking a bit less.... Windows 95 | 11:07 |
| ogra | it uses some KDE audio lib | 11:07 |
| popey | maybe use the kde frameworks snap | 11:07 |
| ogra | some outdated thing ... | 11:07 |
| popey | to re-use their libs | 11:07 |
| popey | oh, arts? | 11:07 |
| ogra | no, the thing after arts | 11:07 |
| ogra | something with q or p ... i forgot the name | 11:07 |
| ogra | phonon ! | 11:08 |
| popey | ah yes | 11:08 |
| zyga | jdstrand: some news | 11:08 |
| zyga | jdstrand: bad news: we cannot do what I did in that PR | 11:08 |
| popey | See, this is one thing I love about snaps, is preserving this old crap :) | 11:08 |
| ogra | haha | 11:08 |
| popey | see also: mosaic | 11:08 |
| zyga | jdstrand: good news: we can ask systemd to do exactly that for us | 11:08 |
| zyga | jdstrand: bad news: it requires us to talk to systemd over dbus | 11:09 |
| zyga | jdstrand: good news: it really works | 11:09 |
| popey | mosaic has nearly 300 installs. Lunatics :) | 11:09 |
| zyga | jdstrand: good news: it does not change the snapd sice of the task much, the only difference is that the location is a scope so it looks like snap.foo.bar.scope | 11:09 |
| zyga | jdstrand: bad news it's somewhat racy, we need to create a transient unit with the pid of the process we want to have | 11:10 |
| zyga | jdstrand: and if that fails we instead need to attach process to the scope | 11:10 |
| zyga | jdstrand: if that fails we need to retry creating the unit | 11:10 |
| ogra | popey, well, it is one of these things you install, try out once and forget | 11:11 |
| popey | true | 11:11 |
| * popey schedules a tweet about mosaic for 7th January, when it will be 23 years old | 11:12 | |
| ogra | i'm sure i have an idling mosaic install on one of my machines that i only tried once | 11:12 |
| * popey looks at Chipaca | 11:13 | |
| * popey looks at the wikipedia page for netscape navigator | 11:13 | |
| * popey looks back at Chipaca | 11:13 | |
| Chipaca | popey: I know! | 11:14 |
| popey | :D | 11:14 |
| Chipaca | popey: but the on-disk filesystem seems to be incompatible :-( | 11:14 |
| Chipaca | I need to give it a weekend | 11:14 |
| Chipaca | popey: something changed _on disk_ since the libc5 days | 11:14 |
| Chipaca | at least that's what the errors tell me | 11:14 |
| Chipaca | popey: HOWever, mosaic isn't the oldest software in the store :) | 11:14 |
| popey | the birthday is 21s February, so you have until then :) | 11:14 |
| popey | oh? | 11:15 |
| Chipaca | popey: mosaic is from 1993 | 11:15 |
| Chipaca | popey: simcity is from 1989 | 11:15 |
| popey | is simcity in the store? | 11:15 |
| Chipaca | obvs | 11:15 |
| popey | i didnt find it | 11:15 |
| Chipaca | popey: 'micropolis' | 11:16 |
| popey | oh! | 11:16 |
| Chipaca | popey: simcity is (tm) ea | 11:16 |
| Chipaca | popey: micropolis is a build from the now-free-software simcity code | 11:16 |
| popey | clever | 11:16 |
| * popey hugs diddledan | 11:16 | |
| * popey gets that on the hype train | 11:16 | |
| Chipaca | one of the nicest things about these old-as-hat things is that they're tiny :) | 11:17 |
| ogra | bah | 11:17 |
| ogra | $ beebeep | 11:17 |
| ogra | ln: failed to create symbolic link '/home/ogra/snap/beebeep/13/.config/gtk-2.0/gtkfilechooser.ini': File exists | 11:17 |
| ogra | /snap/beebeep/13/beebeep: error while loading shared libraries: libxcb.so.1: wrong ELF class: ELFCLASS64 | 11:18 |
| ogra | so that requires some actual work it seems | 11:18 |
| Chipaca | ogra: the elves class war! sounds like a socialist dystopia d&d thing | 11:19 |
| mborzecki | man, spread suite on f31 is messy | 11:20 |
| mborzecki | it's like the stuff breaks in most akward ways | 11:20 |
| mborzecki | quick errand, back in 30 | 11:24 |
| * zyga lunch | 11:40 | |
| mvo | Chipaca: back now, if you want we can have a quick HO (now or later) about the two grubs/uc20 etc | 11:41 |
| mvo | Chipaca: s/now/in 5min/ :) | 11:41 |
| Chipaca | mvo: digging into some bugs atm | 11:42 |
| Chipaca | mvo: didn't we land a fix to allow snapd in a model? | 11:42 |
| Chipaca | in fact this model here has snapd | 11:42 |
| ogra | crap .. the new beepeep will need core18 ... | 11:47 |
| Chipaca | ogra: everybody's on core18 these days | 11:48 |
| pstolowski | doh #2, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" | 11:48 |
| Chipaca | pstolowski: _where_ are those coming from?! i got the too, a few days ago | 11:49 |
| pstolowski | Chipaca: https://api.travis-ci.org/v3/job/608640768/log.txt (not restarting it this time ;)) | 11:49 |
| pstolowski | Chipaca: seems that those debian-sid tests failed because of that | 11:49 |
| Chipaca | yeah | 11:50 |
| Chipaca | pstolowski: i've raised it with the store, but please let me know if you see it again (and in what distro) | 11:51 |
| Chipaca | it might be a broken CA wotsit on debian, for all i know | 11:52 |
| pstolowski | Chipaca: uhmm. going to restart it again | 11:53 |
| Chipaca | pstolowski: k | 11:53 |
| mup | PR snapd#7470 closed: DRAFT: core20 snap install <Created by cmatsuoka> <Closed by cmatsuoka> <https://github.com/snapcore/snapd/pull/7470> | 11:54 |
| Chipaca | cmatsuoka: 👋 | 11:56 |
| cmatsuoka | Chipaca: hello | 11:56 |
| mvo | Chipaca: yeah, snapd should work, I think we are at the point now where the next missing piece is the extraction of the kernel | 11:56 |
| Chipaca | cmatsuoka: do you know if, as well as the kernel, we need to have the initrd extracted for secure boot? | 11:56 |
| mvo | hey cmatsuoka ! good morning :) | 11:56 |
| cmatsuoka | mvo: hi. good morning. I just closed #7470 because all parts are handled elsewhere, I hope you don't mind | 11:57 |
| mup | PR #7470: DRAFT: core20 snap install <Created by cmatsuoka> <Closed by cmatsuoka> <https://github.com/snapcore/snapd/pull/7470> | 11:57 |
| mvo | Chipaca: my understanding is ( cmatsuoka please correct me if I'm wrong) that the uc20 kernel will be a single file with an embedded initrmafs | 11:57 |
| Chipaca | ahhhh | 11:57 |
| mvo | cmatsuoka: yeah, thats great | 11:58 |
| Chipaca | i was just thinking / remembering the days of single-blob kernel+initramfs (on a floppy) | 11:58 |
| mvo | Chipaca: I'm not sure this is the case today though, I need to check | 11:58 |
| mvo | Chipaca: haha - it all comes back (what is a floppy btw?) | 11:58 |
| Chipaca | mvo: that thing that looks like a vending machine | 11:58 |
| mvo | Chipaca: lol | 11:59 |
| cmatsuoka | Chipaca, mvo: yes, I think the idea is still to have a single file | 11:59 |
| Chipaca | mvo: that was in reference to this, btw: https://mobile.twitter.com/fea0er/status/1160099135569063936 | 11:59 |
| mvo | Chipaca, cmatsuoka I just extracted the pc-kernel snap and it looks like at least today there is still an initramfs :/ | 12:00 |
| cmatsuoka | Chipaca, mvo: at that time I was probably rebuilding my kernels to avoid using initrds | 12:00 |
| Chipaca | cmatsuoka: dhcp in the kernel, but tinyX in the initrd | 12:01 |
| Chipaca | hm, let me try something :-) | 12:03 |
| cmatsuoka | mvo, Chipaca: I didn't check the latest kernel snap but a few weeks ago it was quite outdated, I had to inject newer versions to run those entropy and crash tests | 12:06 |
| mvo | cmatsuoka: thanks, thats good to know | 12:12 |
| Eighth_Doctor | zyga: I filed a bug report about it: https://pagure.io/releng/issue/8977 | 12:18 |
| Chipaca | ok, i'm going out for a bit | 12:19 |
| * zyga goes to implement the new thing | 12:20 | |
| Chipaca | mvo: looks like having an embedded initramfs means recompiling the kernel every time you need to change the initramfs | 12:24 |
| Chipaca | it's not just appending one to t'other | 12:24 |
| mvo | Chipaca: uh, that will make testing very inconvenient | 12:24 |
| Chipaca | yes | 12:24 |
| Chipaca | maybe there's a workaround, none of this is too well documented | 12:25 |
| Chipaca | anyway, i really need to step out for a bit :) | 12:26 |
| * Chipaca goes, now | 12:26 | |
| zyga | Eighth_Doctor: thanks! | 12:26 |
| mborzecki | re | 12:27 |
| mborzecki | Eighth_Doctor: so releng is the right place for problems like this? | 12:28 |
| Eighth_Doctor | usually, yes | 12:28 |
| Eighth_Doctor | it means something went wrong when the packages were imported from the Red Hat CDN into Koji | 12:29 |
| mborzecki | Eighth_Doctor: got it! | 12:29 |
| mvo | Chipaca: silly question, I thought the uefi grub has regexp build-in, I boot with -bios ...OVMF.fd but I dont't get regexp in my grub. did I misundersttood? | 12:38 |
| mvo | Chipaca: anyway, not urgent, need to get lunch first :) | 12:38 |
| ogra | hmm ... diddledan do you know anything about the alsa-libs part not working on build.s.io ? seems it stumbles over the ftp url | 12:39 |
| ogra | works fine locally | 12:39 |
| * pstolowski lunch | 12:42 | |
| ogra | ok, here we go ... switching to http download helps :) | 12:54 |
| zyga | mborzecki: back? | 12:54 |
| mborzecki | zyga: yes | 12:54 |
| zyga | mborzecki: I added a hack and it works | 12:55 |
| zyga | mborzecki: using the dbus api now | 12:55 |
| zyga | I'll run a spread pass now | 12:55 |
| zyga | mborzecki: I need to check instances as I'm worried that _ is invalid in scopes | 12:55 |
| zyga | but that's fine, we can escape if needed | 12:56 |
| mborzecki | zyga: _ ? | 12:56 |
| zyga | yes | 12:56 |
| mborzecki | zyga: does systemd complain or just mishebaves? | 12:56 |
| zyga | mborzecki: I didn't try it yet | 12:56 |
| mborzecki | zyga: ah, ok, so it's written down somewhere, tha's fine then | 12:57 |
| zyga | yeah, whatever it is it's okay | 12:57 |
| zyga | I _think_ the situation is somewhat okay now | 12:57 |
| zyga | but | 12:57 |
| zyga | I need to check one last thing | 12:57 |
| zyga | is using delegate breaking services now | 12:57 |
| zyga | or not | 12:57 |
| zyga | given that we tell systemd what we did | 12:57 |
| zyga | (the suspense continues) | 12:57 |
| ogra | popey, the beebeep snap is updated and works so far (it even beeps now!!) but i have a hard time convincing my system to be english for an updated screenshot | 13:12 |
| ogra | popey, mind installing it from --edge and do a screenshot of the window ? | 13:12 |
| popey | sho thang | 13:12 |
| ogra | gracias ! | 13:12 |
| popey | does it use bonjour? | 13:12 |
| ogra | yeah ... | 13:13 |
| ogra | needs two machines if you actually want to chat ... and on first start it offers to create a username, pick different ones on different machines | 13:13 |
| zyga | jdstrand: around? | 13:13 |
| ogra | voice messages dont work (yet) btw | 13:14 |
| popey | I once had a train using my phone as an insecure hotspot. Some rando connected to it, and I saw them show up via bonjour in pidgin. I pinged them and said "Yes, it is okay to use my hotspot" :D | 13:14 |
| zyga | mborzecki: I think the next step is to implement that in C | 13:14 |
| popey | bonjour is awesome. | 13:14 |
| zyga | mborzecki: but apart from that... not terrible? | 13:14 |
| ogra | yep ... | 13:14 |
| ogra | too bad android doesnt support it at all :( | 13:15 |
| ogra | (all my core installs around the house use it (using the avahi snap) but my phone always needs the IP anyway ... | 13:15 |
| ogra | my first snap using "audio-playback" !! | 13:16 |
| * ogra knows jamie will like that | 13:17 | |
| ogra | funnily pulse is still autoconnecting ... not sure why | 13:17 |
| popey | ogra: sent you some via telegram | 13:21 |
| zyga | oh drat, I think jamie is off | 13:22 |
| zyga | oh well | 13:22 |
| === ricab is now known as ricab|lunch | ||
| zyga | Chipaca: it's odd that one cannot install a classic snap on core with --jailmode | 13:23 |
| ogra | bah, since when do we have these silly minimal size restrictions for screenshots ! | 13:24 |
| popey | use convert to scale it up :D | 13:24 |
| zyga | brb | 13:28 |
| mborzecki | cachio: hey, do we tweak the umask in fedora images for our spread tests? | 13:30 |
| ogra | popey, i used gimp to add a transparent frame :P | 13:34 |
| ogra | popey, anyway, website updated ... i'll promote it to stable now and we should be done | 13:35 |
| popey | that looks much better! | 13:35 |
| popey | thanks ogra <3 | 13:35 |
| popey | will line up a social post | 13:35 |
| ogra | bah and now chromium is stuck :P | 13:36 |
| ogra | aaand ... promoted | 13:37 |
| dot-tobias | ogra: Is there a way to build a Core image in which only *one* snap is not from latest/stable? E.g. I want all snaps from the normal stable channel, but one snap from other-track/stable. | 13:42 |
| dot-tobias | required-snaps in the model assertion complains about invalid snap names if I use the notation that's allowed in build-snaps, i.e. `snap-name/track/channel` | 13:43 |
| zyga | dot-tobias: AFAIK no, known limitation | 13:43 |
| ogra | dot-tobias, not sure, i know there was a request to support that a while ago, but i dont know if anything was implemented yet | 13:44 |
| Chipaca | zyga: you were working on #1851480, yes? | 13:51 |
| mup | Bug #1851480: Hooks are not included in slot/plug label expressions <snapd:New> <https://launchpad.net/bugs/1851480> | 13:51 |
| zyga | Chipaca: no, dot-tobias is | 13:51 |
| Chipaca | orly | 13:52 |
| Chipaca | dot-tobias: should I set the bug to in progress? | 13:52 |
| dot-tobias | Chipaca: Not really, haven't had the time to start yet. Will ping here once there's something, have to dig into Go + snapd inner workings first | 13:53 |
| Chipaca | dot-tobias: should I assign the bug to you? | 13:54 |
| zyga | Chipaca: if you set it to in progress then please assign it to someone (dot-tobias) | 13:54 |
| dot-tobias | Chipaca: Yes please, unless there's someone idling around craving to fix this, I'll try to get to it until next week | 13:54 |
| Chipaca | dot-tobias: what's your lp nick? | 13:55 |
| zyga | dot-tobias: oh, btw, I'll be in US west coast timezone all of next week | 13:55 |
| zyga | dot-tobias: and typically busy | 13:56 |
| zyga | I'm always on IRC but if I'm not responding this is probably why | 13:56 |
| dot-tobias | Chipaca: https://launchpad.net/~glancr → or you mean the display name? | 13:56 |
| dot-tobias | zyga: Good to know, thanks for the heads-up 😊 | 13:56 |
| dot-tobias | zyga, ogra: Thanks for the feedback re ubuntu-image, I could've had a quick look at the ubuntu-image bug tracker before stealing your time: https://bugs.launchpad.net/ubuntu-image/+bug/1815580 → seems to me like --snap my-snap:channel is supported, though one needs to remember to juggle required-snaps in the assertion + --snap args for ubuntu-image | 13:57 |
| mup | Bug #1815580: Support for snap prepare-image --snap=<snap>=<channel> <Ubuntu Image:Fix Committed by sil2100> <ubuntu-image (Ubuntu):Fix Released> <https://launchpad.net/bugs/1815580> | 13:57 |
| ogra | dot-tobias, yeah ... thats a workaround/hack to overcome the current limitation ;) | 13:57 |
| mvo | dot-tobias: just got clarification that core is actually promoted to stable, its just not at 100% rollout yet | 14:02 |
| dot-tobias | mvo: Ah, great! Helps a lot, thank you for inquiring on my behalf. Much appreciated | 14:03 |
| cachio | mborzecki, no, we don't | 14:06 |
| ogra | roadmr, so i got this graph at the top of https://snapcraft.io/snaps ... any idea why my most used snap is not in the list/graph at all ? | 14:07 |
| ogra | (mjpeg-streamer has way more users than any other snap i own but is not listed at all in that overview ... on its own metrics page it is fine though) | 14:08 |
| mborzecki | cachio: pushed the last batch of fixes to #7702, the suite should be able to run succesfully on f31 now | 14:10 |
| mup | PR #7702: tests: adding fedora 31 to google-unstable backend <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7702> | 14:10 |
| mvo | dot-tobias: you're welcome | 14:10 |
| cachio | mborzecki, thanks a lot | 14:15 |
| === ricab|lunch is now known as ricab | ||
| zyga | mvo: can we restrict refresh-app-awareness to 18.04+ | 14:27 |
| zyga | mvo: if so I think this can be done | 14:28 |
| zyga | OR can we backport systemd ;_) | 14:28 |
| mvo | zyga: sounds like a question for foundations | 14:28 |
| zyga | mvo: then I can rephrase my question as | 14:28 |
| mvo | Chipaca: how do you run qemu/kvm to get the uefi grub? or is there a way to know which one was run? | 14:28 |
| zyga | mvo: can I really require proper systemd version for this and not implement any hacks | 14:29 |
| zyga | mvo: then if so I will focus on that | 14:29 |
| zyga | mvo: and we can discuss backport as an option for foundations | 14:29 |
| zyga | mvo: but I personally would not cry if it was not on 16.04 | 14:29 |
| mvo | zyga: I think its a conversation - chances are not that great but its worth a shoot | 14:29 |
| mvo | zyga: maybe the relevant bits can simply be backported | 14:29 |
| zyga | mvo: the feature is coupled to systemd running as a user session manager | 14:29 |
| Chipaca | mvo: i go by the splash screen, but i think you can ask grub, but i'd need to get the efi-able box up to confirm (and i can't access it rn) | 14:30 |
| zyga | mvo: aka 16.04 not having that, when that feature was implemented the APIs were created to support that | 14:30 |
| Chipaca | mvo: basically, enter grub's commandline and see what "echo $prefix" prints | 14:30 |
| diddledan | supporting such an old release as 14.04 was gonna be painful eventually :-) | 14:30 |
| Chipaca | if it says EFI, it isn't EFI | 14:30 |
| mvo | Chipaca: ok, I will poke around - I tried "regexp" iirc and that gave me an error | 14:30 |
| Chipaca | mvo: ah, then you're not in EFI | 14:30 |
| Chipaca | mvo: you need to copy regexp.mod in as well for that to work | 14:30 |
| Chipaca | mvo: (EFI grub has regexp built in) | 14:30 |
| mvo | Chipaca: I'm sure I'm holding it wrong, echo $prefix says (hd0,gpt2)/EFI/ubuntu | 14:31 |
| Chipaca | mvo: it goes in EFI/ubuntu/i386-pc/regexp.mod | 14:31 |
| mvo | Chipaca: but *maybe* I have the uc18 grub | 14:31 |
| mvo | Chipaca: let me try to download the focal one | 14:31 |
| Chipaca | mvo: the grub on the machine you run ubuntu-image impacts the grub on the image? | 14:32 |
| Chipaca | mvo: note it's booting in i386-pc mode afaict (the error should mention it) | 14:32 |
| Chipaca | mvo: so you'll need to find that mod, not the amd64 one you probably have | 14:32 |
| mvo | Chipaca: I had to build my own pc gadget and I suspect it build it against core18 | 14:32 |
| mvo | Chipaca: again, thanks, that is good to know | 14:33 |
| mvo | Chipaca: ha! I have grub 2.02 but focal has 2.04 so its all my fault | 14:33 |
| zyga | mborzecki: check this out | 14:35 |
| zyga | mborzecki: the test passes on 64 bit 16.04 | 14:35 |
| zyga | mborzecki: fails on 32 bit 16.04 | 14:35 |
| zyga | ?!?!? | 14:35 |
| zyga | on the 32bit it doesn't have AttachProcessToUnit | 14:35 |
| zyga | Unknown method 'AttachProcessesToUnit' or interface 'org.freedesktop.systemd1.Manager'. | 14:35 |
| zyga | but it really really works on 64bit | 14:36 |
| diddledan | zyga: just do it twice on 32bit so you have 64 :-p | 14:36 |
| * zyga spawns shell to confirm this | 14:36 | |
| zyga | but | 14:36 |
| zyga | mvo: ^ this might indicate that we're not in hot water | 14:36 |
| zyga | as 16.04 x86_64 is where most of the sweet spot for support starts | 14:36 |
| zyga | or perhaps our 32bit images are old | 14:37 |
| zyga | I'll know soon (tm) | 14:37 |
| diddledan | I imagine the 64bit has a newer version somehow. It certainly seems odd for an API to not exist in a supposedly same release with different bittiness | 14:38 |
| zyga | diddledan: I suspect it's just a fluke in our test image | 14:40 |
| zyga | cachio: are 32 bit and 64 bit images for ubuntu 16.04 in sync? | 14:40 |
| diddledan | "computer says no" | 14:40 |
| cachio | zyga, in sync? | 14:40 |
| zyga | cachio: I suspect systemd version is different between them | 14:40 |
| zyga | cachio: are the packages updated equally | 14:41 |
| cachio | zyga, on Monday both were updated | 14:41 |
| zyga | interesting | 14:41 |
| zyga | thanks, I'll know the details soon | 14:41 |
| cachio | zyga, I don't have the details about that update | 14:42 |
| cachio | I know we update && upgrade | 14:42 |
| cachio | and install all the snapd dependencies | 14:42 |
| mup | PR snapd#7683 closed: overlord/ifacestate: remove automatic connections if plug/slot missing <Created by stolowski> <Merged by stolowski> <https://github.com/snapcore/snapd/pull/7683> | 14:58 |
| zyga | mborzecki: I went towards uuid named scope | 15:06 |
| zyga | mborzecki: this works on 16.04+ | 15:06 |
| zyga | mvo: is https://bugs.launchpad.net/snapd/+bug/1848567 fixed in 2.42.1? | 15:12 |
| mup | Bug #1848567: autogenerated per-snap snap-update-ns apparmor profile may contain many duplicate mount rules causing excessive parser memory usage <aa-parser> <AppArmor:New> <snapd:Fix Committed by zyga> <https://launchpad.net/bugs/1848567> | 15:12 |
| zyga | did you cherry pick that apparmor parser memory fix? | 15:12 |
| mvo | zyga: it is | 15:14 |
| zyga | ta | 15:15 |
| mvo | zyga: but we need someone from foudnations to actually approve the SRU :/ | 15:15 |
| mborzecki | cmatsuoka: posted some comments under #7723 | 15:17 |
| mup | PR #7723: snap-bootstrap: create encrypted partition <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/7723> | 15:18 |
| jdstrand | zyga: I am here now | 15:18 |
| cmatsuoka | mborzecki: thanks! | 15:18 |
| zyga | jdstrand: hey! | 15:18 |
| zyga | jdstrand: so ... so much progress | 15:18 |
| zyga | :D | 15:18 |
| jdstrand | zyga: that is good news :) | 15:18 |
| zyga | jdstrand: where do I start, I think the news is indeed good | 15:18 |
| zyga | jdstrand: I sent an update to the PR | 15:18 |
| jdstrand | ah, I only read backscroll | 15:19 |
| zyga | jdstrand: I wanted to ask you how you feel about dbus | 15:19 |
| zyga | https://github.com/snapcore/snapd/pull/7722#issuecomment-551086170 | 15:19 |
| * cachio lunch | 15:19 | |
| mup | PR #7722: cmd/snap-confine: add sc_join_sub_group <Created by zyga> <https://github.com/snapcore/snapd/pull/7722> | 15:19 |
| jdstrand | zyga: let me read. otoh I have a comment, but let me read what you have | 15:19 |
| zyga | jdstrand: tl;dr; we cannot do the move but we can ask systemd to do the move by calling a dbus methodf | 15:20 |
| zyga | that's all, I'll let you read the details | 15:20 |
| jdstrand | zyga: right | 15:21 |
| jdstrand | zyga: oh, heh, that comment is basically backscroll :) | 15:22 |
| mvo | xnox: can I build a new pc gadget with core20 ? I think you did a build against core18 but that means we have grub 2.02 without regexp in there which will not work with the latest grub.cfg that john added | 15:22 |
| * jdstrand reads patch | 15:23 | |
| zyga | jdstrand: the API exists on 16.04+ | 15:23 |
| zyga | not on 14.04 | 15:23 |
| zyga | jdstrand: the patch in the -wip branch does the two-way dance (with attach as a fallback) but I since reworked it | 15:24 |
| jdstrand | I think that is ok with how we defined 14.04 support | 15:24 |
| zyga | jdstrand: to spawn a new scope each time | 15:24 |
| zyga | jdstrand: with uuid inside | 15:24 |
| jdstrand | but let me keep reading | 15:24 |
| zyga | jdstrand: yes, I'm happy with that, 14.04 is ok | 15:24 |
| zyga | ok | 15:24 |
| mvo | Chipaca: I get a uc20 that boots now \o/ with your grub.cfg. it hangs in initrd but thats ok :) | 15:25 |
| zyga | jdstrand: for the dbus call I was thinking to use sd-dbus C API, since we link to libudev which is a differently-named version of systemd it is not a new dependency | 15:26 |
| jdstrand | zyga: hehe, busctl | 15:27 |
| zyga | yes :D | 15:27 |
| zyga | take that security | 15:27 |
| zyga | let's system() out to busctl | 15:27 |
| zyga | :D | 15:27 |
| jdstrand | \o/ | 15:27 |
| zyga | but it helped to prototype this | 15:27 |
| jdstrand | ok, more seriously | 15:27 |
| jdstrand | of course :) | 15:27 |
| jdstrand | so, it is super validating that my interpretation of that page was reasonable and my highlevel idea is what upstream recommends, even if I didn't precisely know how to do it ;P | 15:28 |
| jdstrand | this isn't options 1-3 though | 15:29 |
| jdstrand | anyhoo | 15:29 |
| jdstrand | cool that there is a way with systemd | 15:29 |
| mvo | sil2100: hey, I noticed its your SRU day, can I ask you to review the 2.42.1 SRU in unapproved please :) ? | 15:30 |
| jdstrand | (it is sorta 3 I guess) | 15:30 |
| jdstrand | ok | 15:30 |
| * zyga checks the 1-3 list again | 15:30 | |
| jdstrand | zyga: it doesn't really matter. what I had forgotten and you reminded me today and yesterday is dbus is required | 15:31 |
| zyga | jdstrand: there's one more cool thing there that I could mention | 15:31 |
| jdstrand | please do | 15:31 |
| zyga | jdstrand: systemd has a concept of a controller | 15:31 |
| zyga | jdstrand: I'm not 100% sure how this is meant to be used | 15:32 |
| zyga | but | 15:32 |
| zyga | jdstrand: you can have a unit that a controller watches over | 15:32 |
| zyga | jdstrand: and it will tell you when it wants to wrap it up cause the job died | 15:32 |
| zyga | jdstrand: and I was wondering if that's like notification for us | 15:32 |
| jdstrand | oh | 15:32 |
| jdstrand | yeah | 15:32 |
| jdstrand | interesting | 15:32 |
| zyga | jdstrand: but before I would go there I'd have to check more | 15:32 |
| zyga | jdstrand: it's not documented, I read most of what I found by reading systemd directly | 15:32 |
| * jdstrand nods | 15:32 | |
| zyga | jdstrand: there's a property on a scope | 15:32 |
| zyga | jdstrand: controller | 15:32 |
| zyga | jdstrand: it's not a cgroup controller | 15:33 |
| zyga | jdstrand: it's a dbus path of the place you can call | 15:33 |
| zyga | jdstrand: and it calls it with one method, RequestStop or something like that | 15:33 |
| zyga | jdstrand: it _smells_ like something reaching into a container via a container manager | 15:33 |
| zyga | jdstrand: but I think we don't need to rely on that | 15:33 |
| zyga | jdstrand: just found it interesting | 15:33 |
| zyga | jdstrand: to answer your earlier question | 15:34 |
| zyga | jdstrand: this is mode 1 a | 15:34 |
| jdstrand | zyga: it makes sense> you put everything in places so systemd could track. it has a way to tell you about what it is tracking. that would obviate the need for the parsing code | 15:34 |
| zyga | jdstrand: which is the :-) (happy face) variant | 15:34 |
| zyga | jdstrand: I was thinking we _might_ keep the parsing code | 15:34 |
| mborzecki | Chipaca: you're interested in nonce logs? | 15:34 |
| zyga | jdstrand: until snap-confine can be told it's starting a service | 15:35 |
| Chipaca | mborzecki: sure | 15:35 |
| zyga | jdstrand: then we could only do the scope for user commands | 15:35 |
| zyga | jdstrand: and not for services where it is ... weird | 15:35 |
| jdstrand | zyga: maybe that is what nspawn uses </guess> | 15:35 |
| mborzecki | Chipaca: https://paste.ubuntu.com/p/wMr9Wq66BB/ | 15:35 |
| zyga | jdstrand: and I think this makes sense in more ways actually | 15:35 |
| zyga | jdstrand: because it means all kinds of workloads are managed by systemd now | 15:35 |
| zyga | jdstrand: services, user services, non-service apps and hooks | 15:35 |
| zyga | jdstrand: each one is understood by systemd as a thing | 15:36 |
| zyga | jdstrand: we could even feed it with unit level meta-data | 15:36 |
| zyga | jdstrand: like Description | 15:36 |
| zyga | jdstrand: man page references | 15:36 |
| zyga | jdstrand: perhaps we could even reference snaps or whatever, I'd have to check | 15:36 |
| zyga | jdstrand: since we make a dbus call and we can populate anything that a scope unit can model | 15:36 |
| jdstrand | zyga: ok, so as exciting as this is, I have some reservations about dbus, but I think we may be able to mitigate those concerns. hear me out (I'll type .. when done) | 15:37 |
| zyga | jdstrand: understood | 15:37 |
| jdstrand | a) calling dbus from snap-confine is going to bring a significant attack surface into snap-confine in terms of addressable apis in the address space | 15:38 |
| jdstrand | b) calling dbus is going to bring some likely undesired rules into the apparmor profile | 15:38 |
| xnox | mvo: so the pc-gadget in launchpad is built againt focal, with focal binaries. I can retrigger that, download, unsquashfs, sed core18 to core20 and release into the store | 15:38 |
| xnox | mvo: despite declaring core18, all binaries inside it are from focal | 15:39 |
| jdstrand | c) the combination of a and b is likely going to weaken the security posture of our hardening (primarily apparmor, but see address space) | 15:39 |
| ogra | xnox, cheater !!! :) | 15:39 |
| jdstrand | d) having a dbus call in the list of things we do is going to really slow things down. not a big deal for a service, but for a non-daemon, yikes | 15:40 |
| xnox | ogra: i only am blocked on IS to deploy my launchpad-buildd changes, and for my launchpad UI branches to be merged and deployed by webops. | 15:40 |
| jdstrand | (now to the mitigations) | 15:40 |
| xnox | ogra: it's not like i didn't do everything to fix our infra to do the right thing properly first. | 15:40 |
| * ogra hugs xnox ... i wasnt serious :) | 15:40 | |
| zyga | jdstrand: as a remark, a, b and c *might* not be needed if I can make it work from snap-run before setuid happens | 15:41 |
| jdstrand | reading the patch, I was happy to see that we first create the scope and then slice based on the security label | 15:41 |
| jdstrand | if that is the full label with command name, potentially not as nice | 15:42 |
| zyga | scope/slice? there's only scope, the slice is what we got originally | 15:42 |
| zyga | (as in given to us by systemd) | 15:42 |
| jdstrand | I was happy cause I thought, ok, the first command invocation was slow, but then we can just add stuff ourselves, which is fast | 15:43 |
| jdstrand | but then you said to add stuff, we need the second api | 15:43 |
| jdstrand | zyga: I said slice trying to use systemd parlance, but might've mispoke. I meant the scope ends up being a branch and the security label a leaf | 15:44 |
| zyga | yes, that's correct | 15:44 |
| jdstrand | (back to soliliquy) | 15:44 |
| jdstrand | and the second api requires a dbus call, so every snap command invocation needs the calls | 15:45 |
| jdstrand | so (I | 15:45 |
| jdstrand | 'm getting there) | 15:45 |
| jdstrand | the first thought is, have a snap-dbus-helper in go that snap-confine calls out to | 15:46 |
| jdstrand | we can profile transition to it and alleviate the setuidness/address space/expanding policy issues | 15:46 |
| jdstrand | (a-c) | 15:46 |
| jdstrand | if we fork/exec this we don't have to wait on it. we let it do its thing and perform the move any time later | 15:47 |
| jdstrand | (d) | 15:47 |
| xnox | mvo: i think i will take your PR and push it into the store | 15:48 |
| jdstrand | the snap run idea is interesting as well. we can discuss the merits of that | 15:48 |
| jdstrand | .. | 15:48 |
| zyga | so I think that's the thing to investigate now | 15:48 |
| zyga | my thinking is as follows: | 15:48 |
| zyga | snap run checks if a service is being started | 15:49 |
| zyga | if so, nothing new needs to happen | 15:49 |
| zyga | we can use existing systemd cgroup machinery for services to find what we need | 15:49 |
| zyga | if this is not a service or it is a hook then we call the dbus api from go and move snap run to a new scope with UUID + snap security tag in the name | 15:49 |
| zyga | snapd side we can differentiate those and scan /sys/fs/cgroup/{,systemd} | 15:50 |
| zyga | in both cases holding a lock we can build the security-tag -> definitely-not-running data set | 15:50 |
| zyga | (and if possibly running we know the PIDs that existed at the time) | 15:51 |
| zyga | .. | 15:51 |
| zyga | oh and perhaps as a side note, I would not go and optimize the dbus part until we know how much it really costs | 15:51 |
| jdstrand | zyga: it is 100 going to cost. feel free to measure it | 15:51 |
| jdstrand | 100% | 15:52 |
| jdstrand | how much, I don't know | 15:52 |
| jdstrand | but, in theory, this can happen asynchronyously | 15:52 |
| jdstrand | it easily measurable, but don't do the measuring with busctl | 15:53 |
| jdstrand | it's* | 15:53 |
| jdstrand | so, does this api work with user sessions? | 15:53 |
| jdstrand | zyga: I would be surprised that a non-root process could move itself to a new leaf | 15:54 |
| zyga | zyga@eoan:~/go/src/github.com/snapcore/snapd$ time busctl --user call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager StartTransientUnit 'ssa(sv)a(sa(sv))' testing.$(cat /proc/sys/kernel/random/uuid).scope fail 1 PIDs au 1 $BASHPID 0 | 15:54 |
| zyga | o "/org/freedesktop/systemd1/job/787" | 15:54 |
| zyga | real0m0,010s | 15:54 |
| zyga | user0m0,006s | 15:54 |
| zyga | sys0m0,003s | 15:54 |
| jdstrand | which would be required with snap run | 15:54 |
| zyga | it can if all the permissions match | 15:54 |
| zyga | the pid is the same as the invoker and owner of the unit | 15:54 |
| jdstrand | zyga: the permissions being the uid/gid on the leaf? | 15:54 |
| zyga | (since you can move to a service as well) | 15:54 |
| zyga | not on the leaf, on the unit | 15:55 |
| zyga | scopes have no oner | 15:55 |
| zyga | since they are crated dynamically and are injected relative to where you are | 15:55 |
| zyga | after I invoked that call I got move tod: | 15:55 |
| zyga | 0::/user.slice/user-1000.slice/user@1000.service/testing.ba4454d6-0242-49ac-864e-e6ab1d6c2fb4.scope | 15:55 |
| zyga | *moved to | 15:55 |
| zyga | try it | 15:55 |
| jdstrand | zyga: ok, so that would mean the scope would need to be per command, and per command per uid owner, no? | 15:55 |
| zyga | no root needed | 15:55 |
| zyga | there's a per command scope, yes | 15:56 |
| zyga | my point is that the owner of each scope matches the invoker because that's the only way to have a scope | 15:56 |
| zyga | there's no unit for a scope file on disk | 15:56 |
| jdstrand | zyga: how do you calculate the uuid? | 15:56 |
| zyga | I ask the kernel for one, look at the command above | 15:56 |
| jdstrand | that's another thing that would block in low entropy scenarios | 15:57 |
| jdstrand | could* | 15:57 |
| zyga | yes, alternative is to just use a counter | 15:57 |
| zyga | jdstrand: but I wasn't sure what is doing the counting | 15:57 |
| jdstrand | well, not block, but I guess this isn't a security mechanism | 15:57 |
| zyga | for things like sessions or systemd-run | 15:57 |
| zyga | a counter would be preferred if we can count reliably | 15:57 |
| zyga | but that might imply we need to maintain the counter in snap confine | 15:58 |
| zyga | alternatively perhaps we could open a tmp file on /run/snapd/counters | 15:58 |
| zyga | and the inode of that file is the value | 15:58 |
| zyga | but ... | 15:58 |
| zyga | no ... that is broken | 15:58 |
| zyga | anyway, I think uuid is "good enough" to proceed | 15:58 |
| jdstrand | zyga: can you mockup the series of 0::s from the /proc/pid/cgroup for when a snap daemon starts, then a snap command is started twice by the same user and then a snap command is started once by a different user? | 15:59 |
| zyga | yeah, hold on | 16:00 |
| * jdstrand wants to visualize the end result to make sure we are on the same page | 16:00 | |
| zyga | note that it depends on "where" the user is | 16:00 |
| zyga | in particular it differs if I run firefox from the dock | 16:00 |
| zyga | or run it from the terminal :) | 16:00 |
| zyga | I'll show you | 16:00 |
| jdstrand | zyga: sure, mix two calls from the dock and one from the terminal in | 16:00 |
| jdstrand | zyga: (this would likely turn into a requested code comment ;) | 16:01 |
| jdstrand | zyga: while you are doing that, it means that dbus must be up and running before a snap can be started (unless we fork off). that is perhaps fine since systemd probably needs it itself early on, but a consideration nonetheless | 16:03 |
| zyga | jdstrand: indeed, I think there is a way to depend on it though | 16:03 |
| zyga | I mean | 16:03 |
| zyga | it's socket activated | 16:03 |
| zyga | so no need to really | 16:04 |
| * jdstrand is trying to think through side effects | 16:04 | |
| jdstrand | potentially will slow down the first thing that is started, but shouldn't be a concern | 16:04 |
| jdstrand | zyga: I'll also say, this is probably not something that we can land in master before the sprint due to time constraints, measuring, etc. however, I think that mvo would be able to speak to this well. eg, "lots of investigative work was done and the probably is now fully understood. there were systemd interactions that complicated things, and we wanted to ensure we implemented something robust. until | 16:08 |
| jdstrand | last week, everything was brittle, but now we know the methodology to use (confirmed with upstream) and we are working through the most performant and robust design. we should have that in the next week or so. I | 16:08 |
| jdstrand | While we missed the roadmap item, I | 16:09 |
| jdstrand | 'm pleased with the work and the solid direction" | 16:09 |
| jdstrand | or similar | 16:09 |
| jdstrand | :) | 16:09 |
| jdstrand | if it were me, I would handle it that way. it isn't my call though | 16:09 |
| jdstrand | what is my call is letting you know that I'm concerned about rushing this. we should get Samuele signoff imho | 16:10 |
| jdstrand | mvo: fyi ^ | 16:10 |
| zyga | firefox started from gnome shell: 0::/user.slice/user-1000.slice/user@1000.service/testing.cbdc667d-d3e6-4246-8ecb-7461ba346f51.scope | 16:11 |
| jdstrand | s/probably/problem/ | 16:11 |
| zyga | various programs started from the shell: | 16:11 |
| zyga | 0::/user.slice/user-1000.slice/user@1000.service/testing.cd8f04c3-56d2-425a-a74a-0693440a78a3.scope | 16:11 |
| zyga | 0::/user.slice/user-1000.slice/user@1000.service/testing.dd2b526f-379f-465c-a704-10eb243fa991.scope | 16:11 |
| zyga | (all as my user) | 16:11 |
| zyga | odd, this is fedora 31, I think I noticed that it used to have some more variance before | 16:12 |
| jdstrand | (please show the leaves as well | 16:12 |
| jdstrand | ) | 16:12 |
| jdstrand | zyga: this might be better understood in a pastebin | 16:12 |
| zyga | yeah, I'll collect it, sorry for pasting in-the-middle work | 16:12 |
| mvo | jdstrand: in a meeting, will read backlog | 16:12 |
| zyga | the leaves are owned my my user | 16:12 |
| zyga | [zyga@fedora31 ~]$ ls -ld /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/testing.dd2b526f-379f-465c-a704-10eb243fa991.scope | 16:13 |
| zyga | drwxr-xr-x. 2 zyga zyga 0 11-07 17:01 /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/testing.dd2b526f-379f-465c-a704-10eb243fa991.scope | 16:13 |
| jdstrand | no worries. I want to be able to see all of it so we can discuss by referencing line numbers | 16:13 |
| zyga | ok | 16:13 |
| jdstrand | zyga: ftr, I'm *very* encouraged by this as an overall concept. how we iron out the details of the concept is all we're talking about | 16:14 |
| jdstrand | (ie, the scope/delegate/dbus bits) | 16:14 |
| jdstrand | (the snap run bits are the details :) | 16:14 |
| jdstrand | (or snap-confine, etc, etc...) | 16:15 |
| zyga | 0::/system.slice/sleep.service <- a regular service | 16:15 |
| * zyga collects everything | 16:15 | |
| jdstrand | thanks | 16:29 |
| ijohnson | hey marcustomlinson, is it okay if I adopt your patch at https://github.com/snapcore/snapd/commit/49114d576feb371a27fa4c0d5074dfa487aab358 to merge into snapd? I will need to refactor it a bit before we can merge it likely, and we can't use the 3rd party dep you have there, but this is great work and I would like to see it merged into snapd (as well as use it more for snap startup performance investigations) | 16:30 |
| ijohnson | if you'd like I can just add commits on top of your commit in a branch, or I could just make a new commit with your changes there and provide attribution in the commit message/PR description | 16:31 |
| marcustomlinson | ijohnson: either is fine, I don't mind :) | 16:31 |
| marcustomlinson | ijohnson: and yes please! go ahead | 16:31 |
| zyga | jdstrand: so I have a little bit | 16:34 |
| zyga | https://www.irccloud.com/pastebin/mlAOGSGg/ | 16:34 |
| jdstrand | zyga: fyi, my comment in the pr without putting any time constraints on you: https://github.com/snapcore/snapd/pull/7722#issuecomment-551159049 | 16:36 |
| mup | PR #7722: cmd/snap-confine: add sc_join_sub_group <Created by zyga> <https://github.com/snapcore/snapd/pull/7722> | 16:36 |
| zyga | jdstrand: I'm nearly convinced this doesn't need to be in snap-confine | 16:36 |
| zyga | jdstrand: I'll try to move this to snap run | 16:37 |
| zyga | jdstrand: with uuid and stuff | 16:37 |
| zyga | jdstrand: 0 changes to snap confine would be a sweet deal out of this work | 16:37 |
| jdstrand | zyga: ok, so with this you are skipping the security label | 16:39 |
| zyga | jdstrand: yeah, I think so | 16:39 |
| zyga | jdstrand: well, | 16:39 |
| jdstrand | zyga: that is one of the things I wanted to get on the same page about. I was still thinking there would be a leaf | 16:39 |
| zyga | jdstrand: given where we've been so far I think it's hard to say so for sure ;-) | 16:39 |
| * jdstrand nods | 16:39 | |
| zyga | jdstrand: the leaf is the scope here, no/ | 16:40 |
| jdstrand | zyga: what is happening at line 15? the ls seems to contain the same thing as the new scope | 16:42 |
| zyga | in line 15 I just check the ownership of the scope | 16:42 |
| ijohnson | marcustomlinson: great, thanks! | 16:42 |
| zyga | in line 12 I read it | 16:42 |
| jdstrand | oh, I missed -d | 16:42 |
| zyga | it's printed on line 13 | 16:42 |
| jdstrand | sorry | 16:43 |
| zyga | no worries :) | 16:43 |
| jdstrand | zyga: remind me, StartTransient unit with Delegate is enough to keep systemd happy and for us to track? | 16:47 |
| zyga | without delegate | 16:47 |
| zyga | delegate was needed to be able to move a process there later | 16:47 |
| jdstrand | I was trying to read backscroll and lost track of that | 16:47 |
| jdstrand | ah right | 16:47 |
| zyga | but given that it dependso n more recent systemd and using an uuid is good enough I don't think we need to use that | 16:47 |
| jdstrand | this is purely for tracking and don't need to move | 16:47 |
| zyga | so just starting a transient unit of type scope with snap-run PID | 16:47 |
| zyga | yes | 16:48 |
| zyga | we could even leverage that in hooks | 16:48 |
| zyga | where we "try" to kill hooks that are long running | 16:48 |
| zyga | but now we can really actually stop them | 16:48 |
| zyga | reliably | 16:48 |
| jdstrand | zyga: and the real thing my do s/testing/snap-run/ (or something) | 16:49 |
| jdstrand | might* | 16:49 |
| jdstrand | (for the scope name) | 16:49 |
| zyga | are you asking about the scope name to use for real? | 16:49 |
| jdstrand | yes | 16:49 |
| jdstrand | or were you thinkg the security label? | 16:50 |
| zyga | I think it should be f($SNAP_SECURITY_TAG), something like snap.PKG{,.hook}.NAME.$UUID | 16:50 |
| jdstrand | ok | 16:50 |
| zyga | we can shuffle those around to look nice | 16:50 |
| zyga | but I think it's a good start | 16:50 |
| zyga | and as I mentioned earlier, only for non-service apps and hooks | 16:50 |
| zyga | services are good to go as is | 16:51 |
| jdstrand | zyga: with this, every command invocation gets a new uuid. when are these cleaned up? will systemd be able to handle potentially thousands of these? | 16:51 |
| zyga | they are automatically cleaned by systemd | 16:51 |
| zyga | I can try how it scales | 16:51 |
| jdstrand | I think we'd need to understand that with the uuid approach | 16:52 |
| jdstrand | zyga: not saying we need to be thinking like this, but, it would seem ok if the best implementation used newer systemd features. we could downgrade from one to another but more I was thinking perhaps systems without a new enough systemd don't get the feature | 16:53 |
| jdstrand | zyga: thinking being, this is mostly handy for desktop, and desktop tends to keep moving forward | 16:53 |
| mvo | zyga, jdstrand still in a meeting - but if we have dbus in the hot path of starting snaps we should probably get some sense how fast/slow this is, i.e. if it makes starting snaps slower | 16:53 |
| jdstrand | anyway, just something for back of mind | 16:53 |
| zyga | given that this is supported in 16.04 I think it's a good idea as-is | 16:54 |
| jdstrand | zyga: I was more talking if the uuid approach wasn't as great as it could be, but we aren't there yet | 16:54 |
| zyga | jdstrand: ah | 16:54 |
| zyga | I see | 16:54 |
| zyga | jdstrand: I ran 1000 commands this way and they are not in sysfs anymore | 16:55 |
| jdstrand | mvo: yes. we discussed that and agree it needs to be measured. there are options that might allow us to fork and move on and let the child do the dbus stuff asynchronously | 16:55 |
| jdstrand | mvo: not sure how that would work with the current 'as of this minute transient unit via snap run' thinking, but yes, we agree it must be looked at | 16:56 |
| zyga | jdstrand: we could send the method and perhaps not wait for the reply | 16:57 |
| jdstrand | zyga: those were command that exited though. how about 1000 sleep 600 commands? | 16:57 |
| zyga | jdstrand: but something of this kind, it's really a message dispatch | 16:57 |
| zyga | jdstrand: you read my mind :) | 16:57 |
| jdstrand | zyga: I | 16:57 |
| jdstrand | meh | 16:57 |
| zyga | jdstrand: though I did a sleep $i instead ;) | 16:57 |
| jdstrand | (different keyboard) | 16:57 |
| jdstrand | zyga: I'm assuming they exited that is | 16:57 |
| zyga | jdstrand: yep | 16:58 |
| zyga | jdstrand: while they are running you can see them | 16:58 |
| zyga | https://www.irccloud.com/pastebin/KdUgFNrZ/ | 16:58 |
| jdstrand | zyga: yeah, send and forget is perhaps good enough. I wonder how that command can fail (and we therefore lose track of a pid) | 16:58 |
| zyga | they even can be checked | 16:58 |
| zyga | jdstrand: well, it's either that or wait :) | 16:59 |
| jdstrand | zyga: well, if we fork, the child can wait and retry | 16:59 |
| zyga | and if that fails? | 16:59 |
| jdstrand | then we lose it | 17:00 |
| zyga | sure but at some point: 1) you are failing but the app is running 2) the app is running untracked | 17:00 |
| jdstrand | it could be seen as best effort in that regard | 17:00 |
| zyga | yep | 17:00 |
| zyga | though I would not go there initially, we need to see how it works still | 17:00 |
| mup | PR snapcraft#2782 closed: snapcraft: introduce click-based YAML configuration file support <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2782> | 17:00 |
| jdstrand | which by far most of the time would be great | 17:00 |
| zyga | jdstrand: re cgroups, they are cleaned up as the scopes become empty | 17:00 |
| jdstrand | zyga: right, so the thing exits, systemd sees the last one left, it removes the scope. makes sense | 17:01 |
| zyga | jdstrand: there's one reason we might want to use a delegate though | 17:01 |
| zyga | jdstrand: using a delegate we could have another leaf level | 17:02 |
| zyga | jdstrand: and attach eBPF programs there | 17:02 |
| zyga | jdstrand: perhaps we will have to for device cgroup | 17:02 |
| jdstrand | zyga: I wonder how well that works with name=systemd since, aiui, they woul dhave to poll the cgroup.proc file rather than listen on cgroup.events in v2 | 17:02 |
| jdstrand | zyga: also for the cpu/mem discussion yesterday | 17:03 |
| zyga | jdstrand: for cpu/mem it is relevant | 17:03 |
| zyga | jdstrand: we cannot do that according to systemd rules | 17:03 |
| zyga | jdstrand: you cannot write to anything that you are not a delegate of | 17:03 |
| zyga | jdstrand: you can ask systemd to change properties of existing scopes/services | 17:03 |
| jdstrand | zyga: but maybe we just use systemd apis to control systemd's cpu/mem handling? | 17:03 |
| zyga | jdstrand: but not externally by poking at sysfs directly | 17:03 |
| jdstrand | right | 17:04 |
| zyga | yes, I think that's good if we can do that | 17:04 |
| * jdstrand nods | 17:04 | |
| zyga | because the alternative is that we do use delegates for services | 17:04 |
| jdstrand | exactly | 17:04 |
| zyga | i.e. have a slice (not a scope) | 17:04 |
| * jdstrand nods | 17:04 | |
| zyga | and put Delegate=snaps.slice | 17:04 |
| zyga | and then inside that be system manager and handle things | 17:04 |
| zyga | but I think ... that's systemd's job | 17:04 |
| zyga | s/system/service/ | 17:04 |
| jdstrand | right. we are trying to play nicely with it and leverage it, so why not for this too? | 17:05 |
| jdstrand | it does mean we are probably forever coupled to systemd | 17:05 |
| jdstrand | it is imaginable for an alternative init to be used with what we have now (service management, timers, etc) | 17:06 |
| jdstrand | even if that is a heap of work | 17:06 |
| jdstrand | this talks us a bit farther down the systemd path | 17:06 |
| jdstrand | takes* | 17:06 |
| zyga | jdstrand: yes, but perhaps to good effect, | 17:06 |
| jdstrand | I may not have slept enough last night ;) | 17:06 |
| zyga | jdstrand: for instance killing hooks was sys-v init style | 17:07 |
| jdstrand | zyga: oh for sure, just calling it out. not a blocker | 17:07 |
| zyga | I think I'll wrap up for now | 17:08 |
| jdstrand | zyga: a property I like about snap run is that the correct user is doing the dbus call | 17:08 |
| zyga | it's 18:00 and I want to not repeat yesterday | 17:08 |
| zyga | and you are right this won't land before the sprint | 17:08 |
| jdstrand | zyga: a property I like about snap-confine is it is in a position to organize things rigidly | 17:08 |
| jdstrand | zyga: food for thought | 17:09 |
| zyga | hmm, now that I think of it | 17:09 |
| jdstrand | zyga: this was *super* interesting and fruitful | 17:09 |
| zyga | I think as a user I got a polkit prompt from the shell | 17:09 |
| jdstrand | ohh.... | 17:09 |
| zyga | need to double check that | 17:09 |
| jdstrand | that is important. that is cached | 17:09 |
| zyga | wait, I'm dumb | 17:10 |
| jdstrand | zyga: if snap-confine is back on the table, I think I feel quite strongly we need a helper and that be written in go | 17:10 |
| jdstrand | (not saying it is) | 17:10 |
| zyga | no prompt | 17:11 |
| zyga | I was passing --system :D | 17:11 |
| jdstrand | just for back of mind | 17:11 |
| zyga | https://www.irccloud.com/pastebin/8wRREiYV/ | 17:11 |
| zyga | I'll double check that on ubuntu with v1 | 17:11 |
| zyga | same | 17:11 |
| * jdstrand nods | 17:11 | |
| zyga | no prompt | 17:11 |
| jdstrand | yeah, come to think of it | 17:11 |
| zyga | so it can safely live in snap-run | 17:12 |
| jdstrand | you wouldn | 17:12 |
| jdstrand | t need polkit for --user (duh) and it is also ok for the unconfined user to call the apis that systemd exports. moving this down doesn't affect the properties of the resource limits... | 17:12 |
| jdstrand | and snap run is pre confinement | 17:13 |
| jdstrand | well, except when it isn't (snap calling snap) | 17:13 |
| zyga | jdstrand: snap calling snap is semi-supported | 17:13 |
| jdstrand | there might be more to investigate there | 17:13 |
| zyga | jdstrand: I don't know what actually relies on it nowadays | 17:13 |
| jdstrand | yeah | 17:13 |
| zyga | jdstrand: and apart from that, it only works in devmode | 17:13 |
| jdstrand | but we should understand the implications | 17:13 |
| zyga | jdstrand: as otherwise snap-confine cannot reassociate with pid-1 mount ns | 17:14 |
| zyga | jdstrand: so in strict mode it is refused anywa | 17:14 |
| jdstrand | also, make sure that classic doesn't have any gotchas | 17:14 |
| zyga | *anyway | 17:14 |
| zyga | jdstrand: I _think_ it doesn't | 17:14 |
| zyga | I mean, classic is not affecting this in any way as far as I can see | 17:14 |
| jdstrand | zyga: yep, thanks so much for the work and discussion. we'll definitely pick this up, next week if not sooner | 17:14 |
| zyga | jdstrand: tomorrow I'll focus on making snap-run do this via go-dbus | 17:15 |
| jdstrand | zyga: sure. a classic snap can call sddbus itself. anyway, just thinking through pain points and things to document | 17:15 |
| zyga | jdstrand: both in a standalone PR and in a wip PR with the snapd side using it | 17:15 |
| jdstrand | zyga: have nice evening :) have a beer or do something fun. I think we cracked the overall approach :) | 17:16 |
| zyga | I think so | 17:16 |
| zyga | I don't know if I have any at home :) | 17:16 |
| zyga | but I plan to see what the family is up to upstairs | 17:16 |
| === pstolowski is now known as pstolowski|afk | ||
| jdstrand | this might be cause to leave the cave | 17:17 |
| zyga | thank you for your focus, attention and wisdom :) | 17:17 |
| zyga | haha | 17:17 |
| zyga | yes | 17:17 |
| jdstrand | zyga: back at you :) | 17:17 |
| jdstrand | (and thanks!) | 17:17 |
| zyga | see you tomorrow or if we miss each other somehow at the sprint next week :) | 17:17 |
| jdstrand | sounds good. take care | 17:17 |
| zyga | o/ | 17:17 |
| * zyga EODs | 17:17 | |
| zyga | jdstrand, mvo: I wrote down the implementation plan on https://github.com/snapcore/snapd/pull/7722#issuecomment-551179377 | 17:23 |
| mup | PR #7722: cmd/snap-confine: add sc_join_sub_group <Created by zyga> <https://github.com/snapcore/snapd/pull/7722> | 17:23 |
| zyga | I didn't include things like testing and measurement because I want to EOD but I agree those will happen | 17:24 |
| jdstrand | zyga: for when you come back online. when thinking about the benefits of snap run, lets also consider what is required for device (and freezer) cgroup (equivalents) going forward. ie, perhaps that means snap-confine calling a helper is better to have a uniform approach going forward (or not, just don't want you to do snap run only to realize we maybe didn't think that all the way through with future work) | 17:24 |
| zyga | just wanted to write it down while fresh | 17:24 |
| jdstrand | ah, heh | 17:24 |
| zyga | ha | 17:24 |
| zyga | freezer can go away at some point via new mount api | 17:24 |
| zyga | in pure v2 we should just really assume you have matching kernel and use new syscalls | 17:24 |
| zyga | devices are more complex because they require (helper or not) eBPF attached to cgroup | 17:25 |
| zyga | we cannot do that unless we ask for delegate unit | 17:25 |
| jdstrand | that perhaps makes sense | 17:25 |
| zyga | there's some special handling that happens then | 17:25 |
| jdstrand | right | 17:25 |
| zyga | because the eBPF programs systemd itself uses are added with different mode | 17:25 |
| mvo | zyga: yeah, lets discuss more tomorrow, get some rest :) | 17:25 |
| zyga | so that all programs run (aka multi mode) | 17:25 |
| jdstrand | my ony point is if we know that, and we know what that will sorta look like, maybe we look at tracking through that lens | 17:25 |
| jdstrand | anyway, enjoy your evening :) | 17:26 |
| zyga | I think we can use helpers or bake it into snap* but it needs the delegate "hop" at some point | 17:26 |
| zyga | but that's a good observation that we can align on this for other features | 17:26 |
| zyga | mvo: yeah, :) | 17:26 |
| * zyga really EOD now | 17:26 | |
| jdstrand | mvo: are you out of your meeting? | 17:26 |
| mvo | jdstrand: yes | 17:29 |
| jdstrand | mvo: hey, so that is a lot of backscroll. let me get you the most important thing for next week (outside of the comments in the PR we made in the last day) | 17:34 |
| mvo | jdstrand: thats super nice of you, thank you! | 17:35 |
| mvo | jdstrand: I will have dinner now but you can mail or /msg me the important bits if that is ok | 17:36 |
| jdstrand | mvo: a) I think we want pedronis to review this and b) https://paste.ubuntu.com/p/7HdCp9p9K5/ | 17:37 |
| jdstrand | mvo: that is fine. enjoy dinner | 17:37 |
| mvo | jdstrand: thank you! | 17:37 |
| mvo | jdstrand: yeah, thanks for the pastebin, I think thats a reasonable answer. will you be in vancouver? | 17:37 |
| jdstrand | mvo: meh, in the paste: /the probably/the problem/ | 17:38 |
| jdstrand | mvo: I will | 17:38 |
| jdstrand | mvo: I have limited availability tomorrow (similar to today; front loaded the week with work) | 17:38 |
| jdstrand | mvo: but in Vancouver next week and zy ga and I will continue to discuss this | 17:39 |
| mvo | jdstrand: excellent! | 17:40 |
| * Chipaca EODs | 18:37 | |
| * cachio afk | 19:19 | |
| mup | PR snapd#7724 closed: cmd/snap-confine: tracking processes with classic confinement <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/7724> | 19:33 |
| blake_r | having issue where calculating the version of the maas snap is failing now with snapcraft | 22:46 |
| blake_r | i now see in the logs | 22:46 |
| blake_r | fatal: not a git repository (or any of the parent directories): .git | 22:47 |
| blake_r | fatal: not a git repository (or any of the parent directories): .git | 22:47 |
| blake_r | some reason the .git directory is no longer being copied into the VM | 22:47 |
| blake_r | using just source: . | 22:47 |
| blake_r | with no source-type defined | 22:47 |
| blake_r | any ideas on why that is occurring now, when it didn't happen before | 22:47 |
| blake_r | sergiusens: ^ | 22:48 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
