[04:02] <pilil__> good morning.
[04:02] <pilil__> I have a question about system wrappers. After installation all apps and services should be started only through wrappers, but when one wrapper trying to launch another wrapper I receive errors like this:
[04:02] <pilil__> $sudo /usr/bin/ubuntu-core-launcher myapp myapp_service_0.0.1 /apps/myapp/0.0.1/bin/myapp-wrapper
[04:02] <pilil__> ubuntu-core-launcher: error while loading shared libraries: cannot apply additional memory protection after relocation: Permission denied
[04:03] <pilil__> even with frameworks
[04:05] <pilil__> without ubuntu-core-launcher - everything ok
[04:07] <pilil__> Is there proper way to launch one file from another?
[07:00] <dholbach> good morning
[07:03] <fgimenez> good morning
[07:18] <elopio> fgimenez: crazy day here today, but all your branches are +1-ed. Sorry for the delay.
[07:18] <fgimenez> elopio, hey, np thanks :)
[07:19] <fgimenez> elopio, i'll apply your suggestion to the config testbed one, i like the boolean value option
[07:20] <elopio> fgimenez: as you prefer. The change would be pretty simple, so if it passes for you also feel free to top-approve.
[07:20] <elopio> I'm leaving now. See you soon.
[07:21] <fgimenez> elopio, ok see you
[08:17] <ogra_> pilil, this sounds like a question for the security team, either wait til the US gets up or write a mail to the mailing list
[08:20] <longsleep> Moin folks, so is there a place where i can add feature requests for snapcraft?
[08:20] <ogra_> file a whishlist bug
[08:20] <ogra_> und moin :)
[08:20] <longsleep> whishlist bug ok - let me try that
[08:22] <longsleep> haha bug #50788
[08:22] <nothal> Bug #50788: We don't need "Wishlist"  <http://launchpad.net/bugs/50788>
[08:22] <ogra_> lol
[08:23] <ogra_> reported 2006-06-23 :)
[08:23] <longsleep> yeah
[08:23] <longsleep> and marked invalid
[08:23] <longsleep> just was the first hit
[08:23] <ogra_> man, these names bring back memories :)
[08:23] <pilil> ogra_, who should I ask it from security team?
[08:24] <ogra_> pilil, try jdstrand or tyhicks (i'm not sure who exactly could help here)
[08:25] <pilil> ogra_, I got it, thanks
[08:27] <pilil> ogra_, there is another question. How can we add new extrauser to Snappy, cause tools like passwd or useradd still works with /etc/passwd instead of /var/lib/extrausers?
[08:29] <ogra_> pilil, i think that is fixed in the rolling release, it is quite a change so it was not backported afaik
[08:29] <longsleep> ogra_: bug #1480144 added, i tagged it with wishlist, not sure if that is the correct way
[08:29] <nothal> Bug #1480144: Snapcraft should be able to run in clean environment with pbuilder/cowbuilder <Snappy:New> <http://launchpad.net/bugs/1480144>
[08:31] <pilil> ogra_, thanks
[08:35] <ogra_> longsleep, i know cross building is planned, but it will likely still take a bit before it happens
[08:35] <ogra_> (not on top of the list)
[08:40] <tasdomas> hi
[08:41] <tasdomas> why does the raspberry pi2 snappy image contain an .ssh/authorized_keys entry for ogra@anubis?
[08:45] <ogra_> tsthats a fake key ... ubuntu-device-flash needs a valid key if you enable --develper-mode during build
[08:45] <ogra_> tasdomas, ^^^
[08:46] <ogra_> tasdomas, i hope to finish a new image today and will make that clearer (calling it dummy@dummy or some such) in that build
[08:49] <longsleep> ogra_: yeah - in the meantime i might just create a small tool "debsto
[08:49] <longsleep> err
[08:50] <longsleep> ogra_: debs2snap or something
[08:50] <ogra_> haha
[08:50] <longsleep> that way i can just use the existing gear plus one extra step
[08:56] <biezpal> ogra_, question about systemd and snaps. In the package.yaml we can specify the type of service (dbus or not), is there a plans to implement other types of services, like forking or other?
[09:00] <ogra_> biezpal, hmm, not sure what you mean by type of service ... do you mean the bus-name filed for framework snaps ?
[09:00] <ogra_> https://developer.ubuntu.com/en/snappy/guides/package-metadata/
[09:02] <JamesTait> Good morning all; happy Friday, and happy System Administrator Appreciation Day! ð
[09:03] <ogra_> wasnt that yesterday ?
[09:04] <ogra_> oh, wasn't :P
[09:14] <biezpal> ogra_, I'm talking about systemd service unit Type
[09:15] <biezpal> [Unit]
[09:15] <biezpal> Description=swamp services management service
[09:15] <biezpal> After=syslog.target
[09:15] <biezpal> [Service]
[09:16] <ogra_> hmm, then i dont see the relation to package.yaml here
[09:16] <biezpal> Type=forking
[09:16] <biezpal> we want to specify type of service described in package.yaml
[09:16] <ogra_> but package.yaml doesnt offer that (at least currently)
[09:17] <biezpal> we can describe service from package.yaml, but not the type of it?
[09:18] <ogra_> you can put into the description what you want ... but there is no "type" field or anything that would do anything meaningful with it
[09:19] <ogra_> (see the linked documentation above)
[09:20] <biezpal> forked service is being killed by systemd because systemd is thinking that process is stopped
[09:21] <biezpal> now, to get rid of it we are manually edit systemd unit and specify Type = forking
[09:21] <ogra_> righ
[09:21] <ogra_> t
[09:22] <ogra_> and where does the package.yaml come into play here ? thats the bit i dont understand
[09:22] <biezpal> we want Type of service to be taken from package.yaml
[09:23] <ogra_> oh, so this is a feature request ?
[09:23] <biezpal> it's just a question to find the way
[09:23] <ogra_> (to extend package.yamnl ?)
[09:24] <biezpal> to build unit automatically with additional options
[09:25] <c-lob> sorry if I stick my nose in this discussion, but isn't it possible to pack the .service file into the snap?
[09:26] <ogra_> right, that doesnt sound like something supported yet ... i'd start a thread on the mailing list
[09:26] <ogra_> c-lob, indeed you can do that
[09:27] <biezpal> c-lob, where we can read about that option?
[09:27] <c-lob> biezpal, well I'm just thinking about now :)
[09:27] <c-lob> biezpal, well I'm just thinking about it now :)
[09:27] <biezpal> lol
[09:28] <biezpal> ogra_, maybe you know?)
[09:28]  * ogra_ takes a look at webdm
[09:32] <ogra_> hmm, so i dont see a way, it uses the generated unit too (which fires up a script that cares for all teh rest). i guess you should try starting a thread on the ML
[09:33] <ogra_> (there is probably no way to change the type of the unit, but surely a way to achieve what you wanted to do with setting it via some other mechanism)
[09:38] <vmayoral> ppisati: just e-mailed you, i'm experiencing some issues with the compiled kernels when booting with the Snappy FS, apparently the "system-boot" partition fails to get mounted. Would be great getting your input here
[09:40] <ppisati> vmayoral: i'm looking
[09:40] <vmayoral> ppisati: thanks
[09:40] <ppisati> mount: wrong fs type, bad o
[09:40] <ppisati> missing codepage or helper
[09:40] <ppisati> vmayoral: can you complete a boot?
[09:42] <vmayoral> ppisati: i get into emergency mode https://gist.github.com/vmayoral/fc2c7ebbd679ea7d9a9b
[09:43] <ppisati> vmayoral: do you see something in dmesg?
[09:43] <ppisati> vmayoral: let me check the config
[09:44] <vmayoral> ppisati: nothing relevant that i can identify https://gist.github.com/vmayoral/193f5c1e71f5cfb9bd67
[09:44] <ppisati> FAT-fs (mmcblk0p1): IO charset iso8859-1 not found
[09:44] <ppisati> that config is missing the option
[09:44] <ppisati> two things:
[09:45] <ppisati> 1) did you check that the resulting config are the same?
[09:45] <ppisati> 2) you are using an old version #18 while we are at...
[09:45] <ppisati> 23?
[09:45] <ppisati> something like that
[09:45] <ppisati> let me take a closer look at that tree
[09:47] <ppisati> ah
[09:47] <ppisati> ok
[09:47] <ppisati> to compile that tree and get a debian package, you have to do this:
[09:48] <ppisati> export $(dpkg-architecture -aarmhf); export CROSS_COMPILE=arm-linux-gnueabihf-
[09:48] <ppisati> fdr clean; debian/rules build; fdr binary-generic
[09:48] <ppisati> the config is store in:
[09:48] <ppisati> *stored
[09:49] <ppisati> master/debian.master/config
[09:49] <ppisati> split among
[09:49] <ppisati> config.common.ubuntu
[09:49] <ppisati> and the other snippets in debian.master/config/armhf*
[09:50] <ppisati> if you did as i read in the REAME.md of that git tree:
[09:50] <ppisati> ARCH=arm ./scripts/kconfig/merge_config.sh arch/arm/configs/omap2plus_defconfig arch/arm/configs/snappy/*.config
[09:50] <ppisati> make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- uImage dtbs -j4
[09:50] <ppisati> you are missing some options
[09:50] <ppisati> so, it's up to you
[09:50] <vmayoral> i see
[09:50] <vmayoral> regarding the kernel
[09:50] <ppisati> either you add that charset in your config
[09:50] <vmayoral> i fetched it from http://kernel.ubuntu.com/git/ubuntu/ubuntu-vivid.git/ a few days ago
[09:51] <vmayoral> will rebase it now to get up to date
[09:51] <ppisati> i think your config is quite good
[09:51] <ppisati> the way you build the kernel is correct
[09:51] <ppisati> it is how it's done when you are developing
[09:51] <ppisati> it's faster
[09:51] <ppisati> it easier if you just made a change and you want to test it
[09:51] <ppisati> etc
[09:52] <ppisati> but if you want exactly our kernel packages
[09:52] <ppisati> you should follow the 'fdr *' instructions up here
[09:52] <ppisati> where fdr is an alias for
[09:52] <ppisati> 'fakeroot debian/rules'
[09:52] <ppisati> just in case
[09:53] <vmayoral> thanks a lot for explaining, is this documented somewhere?
[09:53] <ppisati> vmayoral: yep
[09:53] <ppisati> vmayoral: hold on
[09:54] <ppisati> https://wiki.ubuntu.com/KernelTeam/ARMKernelCrossCompile
[09:54] <ppisati> ok, this is a bit old
[09:54] <ppisati> since it covers the omap4 kernel
[09:54] <ppisati> and back then i was suggesting:
[09:54] <ppisati> export $(dpkg-architecture -aarmhf); export CROSS_COMPILE=arm-linux-gnueabihf-
[09:54] <ppisati> fakeroot debian/rules clean
[09:54] <ppisati> fakeroot debian/rules binary-omap4
[09:54] <ppisati> but the stuff that i pasted here is faster
[09:55] <ppisati> and works for generic
[09:55] <ppisati> (instead of binary-omap4 you use binary-generic)
[09:55] <ppisati> and it tells you how to change the config too
[09:55] <ppisati> fakeroot debian/rules editconfigs
[09:56] <ppisati> in your case it's the generic flavour that you are interested
[09:56] <vmayoral> great, I'll go ahead and reproduce it all. If it adds some value, I'll be happy to document it  and maybe this way i can contribute.
[09:57] <vmayoral> ppisati: thanks a lot for your time.
[10:00] <biezpal> ogra_, thanks for answer
[10:02] <vmayoral> ppisati: it'll be great if you guys could also consider including the PRU patches in the vivid tree. Many BBB users make use of these units on the SOC.
[10:04] <vmayoral> also, ppisati, what's you opinion on changing the kernel released on snappy images to be preemptible (pretty match activating the PREEMPT option)?. IOT devices could make good use of this kind of kernels
[10:05] <ppisati> vmayoral: ATM the BBB is the same kernel that we use across all ubuntu armhf devices
[10:05] <vmayoral> current OEM snap allows to change the kernel (i believe) so that should be manageable
[10:05] <ppisati> vmayoral: but we are having a discussion about it, where to take it, the direction we want to give it, etc
[10:05] <ppisati> right
[10:06] <ppisati> vmayoral: question - i now that you use the PRUs for the sensors on your drone, right?
[10:06] <vmayoral> ppisati: great hearing that. I don't mind compiling the kernels with PREEMPT or PREEMPT_RT options but it'll be great for many avoiding this and going straight into the official images
[10:06] <vmayoral> ppisati: yes, we use it for fast PWM generation and PPM signal processing
[10:07] <ppisati> vmayoral: ok
[10:07] <vmayoral> but there's a lot happening in the PRU world
[10:07] <ppisati> vmayoral: so, you load a binary into PRUs, right?
[10:07] <vmayoral> yes, every year, there're GSOC project that build on top
[10:07] <ppisati> vmayoral: did you try to "port" your code to the remoteproc facility?
[10:08] <vmayoral> no we have not. What i'm most concerned about is the maintainability if we were to do so.
[10:08] <vmayoral> Current Dronecode Foundation helps a lot with that
[10:08] <vmayoral> if we were to move to the PRUs...
[10:08] <ppisati> vmayoral: what you mean?
[10:11] <vmayoral> i mean that there's an existing community supporting the code based on a single (or multiple) core symmetric processors based on userspace drivers. If we were to port a  part of the code to an assymetric arch. (e.g. the PRUs) we would loose the community support
[10:11] <vmayoral> with our current size and dev. force we can't afford it
[10:12] <vmayoral> nevertheless, i'm seeing how the PRU-world grows every year and there's even people bit-banging protocols on them
[10:13] <ppisati> vmayoral: ok, sorry i'm confused now
[10:13] <ppisati> vmayoral: i asked you if you were using the PRU and you told that you were using it
[10:14] <ppisati> 12:06 < vmayoral> ppisati: yes, we use it for fast PWM generation and PPM signal processing
[10:14] <vmayoral> yes, we are
[10:14] <vmayoral> for PWM and PPM generation/processing
[10:14] <ppisati> ok
[10:14] <vmayoral> probably i misunderstood it at some point.
[10:15] <ppisati> vmayoral: so, there's are two ways to interact with the hw PRUs AFAIK
[10:15] <ppisati> vmayoral: the PRU patches that you applied
[10:16] <ppisati> vmayoral: unsupported by TI
[10:16] <ppisati> vmayoral: of their supported mechanism, the remoteproc
[10:16] <ppisati> *or
[10:16] <ppisati> since you applied thse patches to your kernel, i assume you are using the userspace driver
[10:17] <ppisati> and i was wondering if you have ever tried/considered to move it to remoteproc
[10:17] <ppisati> that's beause, part of the TI BSP kernel requires the remoteproc facility for some of its features
[10:17] <ppisati> e.g. the power management code has a requirement on it
[10:21] <vmayoral> i see, i don't have much understanding about how remoteproc works but i was assuming that any interaction with the PRUs was done through the remoteproc framework.
[10:22] <vmayoral> the patches came originally from a tree maintained by Robert Nelson who is working tightly with TI and BeagleBoard https://github.com/RobertCNelson/bb-kernel/tree/am33x-rt-v4.1/patches/pru
[10:24] <vmayoral> (AFAIK)
[10:52] <longsleep> is there a way to launch a shell in the environment of a snap?
[10:53] <longsleep> my snap fails to start, and the systemd log is not very helpful
[10:54] <ogra_> i had a nodejs based terminal once, running the shell inside teh snap env,  but the snap isnt functional currently
[10:55] <ogra_> i think there was another one in the examples or some such, but i cant remember exactly
[10:55] <longsleep> well i guess i could just set the environment variables manually
[10:56] <ogra_> do you have a start script ?
[10:56] <ogra_> you could just make it print the env to some logfile
[10:56] <longsleep> yes
[10:56] <longsleep> aha
[10:56] <longsleep> the error is cp: cannot create regular file â/server.confâ: Read-only file system
[10:57] <ogra_> looks like an unset variable
[10:57] <ogra_> SNAP_APP_PATH ?
[10:57] <ogra_> or SNAP_APP_DATA_PATH
[10:57] <longsleep> yes CONF=$SNAP_APP_DATA_PATH/server.conf
[10:58] <ogra_> weird, that should definitely be set
[10:58] <longsleep> yes it is not set when i run it manually
[10:58] <longsleep> for testing
[10:58] <longsleep> found the error now
[10:58] <ogra_> heh, indeed not
[10:58] <longsleep> sed: -e expression #1, char 88: unterminated `s' command
[10:58] <longsleep> narf
[10:58] <ogra_> heh
[10:59] <longsleep> but it would be really helpful if one would see these errors in systemd
[10:59] <ogra_> +1
[11:00] <ogra_> they used to show up when we used journald ... not sure why they dont end up in syslog now
[11:01] <longsleep> mhm let me check syslog, i was using systemctl
[11:01] <ogra_> ah
[11:04] <longsleep> ogra_: you are right, it is in syslog
[11:04] <longsleep> Jul 31 11:04:22 odroid ubuntu-core-launcher[1327]: sed: -e expression #1, char 88: unterminated `s' command
[11:05] <longsleep> thats good enough i think
[11:05] <ogra_> ah, cool
[11:05] <Chipaca> longsleep: should also be in journalctl
[11:06] <ogra_> Chipaca, do you knwo why we use both ?
[11:06] <Chipaca> but i admit to not being proficient in journalctl usage
[11:06] <ogra_> smells bloated
[11:06] <Chipaca> ogra_: because that's how we roll? :-p
[11:06] <Chipaca> ogra_: i think we want to drop syslogd, but had to bring it back for <something>
[11:06] <Chipaca> happened before i got on board
[11:06] <ogra_> ah
[11:06] <Chipaca> ogra_: but aiui it's wanted to go away
[11:06] <ogra_> ok
[11:07] <Chipaca> that is, we want to drop syslogd, but something or other depends on it still
[11:07]  * ogra_ doesnt care which one goes away, i just dont like the duplication :)
[11:07] <Chipaca> and it's bloated but not super critical
[11:07] <ogra_> yeah
[11:09] <ogra_> hmm
[11:10] <ogra_> on my BBB snappy list -v shows ubuntu-core 9 active ... webdm only shows 8
[11:14] <longsleep> mhm now i get Jul 31 11:12:02 odroid ubuntu-core-launcher[1734]: Bad system call when running sed :/
[11:14] <longsleep> time for lunch
[11:23] <Chipaca> longsleep: ooh! check syslog again, in particular for seccomp or apparmor issues
[11:23] <Chipaca> longsleep: bad system call is probably seccomp
[11:24] <Chipaca> longsleep: sc-logresolve should help you if that's the case
[11:38] <Chipaca> longsleep: super interested in what you find
[11:38] <Chipaca> but alas, lunch calls
[12:25] <longsleep> Chipaca: yeah i will investigate now - i just returned from lunch and finishing up my ice cream :P
[12:28] <ogra_> yummy
[12:28]  * Chipaca having a big cool fizzy drink instead
[12:28] <Chipaca> (no it's not beer, shaddup)
[12:29]  * ogra_ slurps a hot espresso :) 
[12:29] <ogra_> you and your frozen stuff
[12:43] <longsleep> Chipaca: so, this is all in syslog: http://paste.ubuntu.com/11973062/
[12:43] <longsleep> Chipaca: and my start script is this (for testing) http://paste.ubuntu.com/11973068/
[12:56] <jjohansen> sergiusens, ogra_: how do you build a base image with a custom kernel using ubuntu-device-flash?
[12:57] <ogra_> you need your own device tarball
[12:59] <ogra_> jjohansen, this is the script i use to create the rpi device tarball from ppisati's PPA builds http://paste.ubuntu.com/11973145/
[13:00] <jdstrand> ogra_: how much of that applies to generating something for generic-amd64?
[13:00] <ogra_> well, the format is the same in both
[13:01] <ogra_> the paths might differ though
[13:01] <ogra_> (since x86 doesnt use uboot indeed)
[13:01] <c-lob> longsleep, I saw that calling the binary directly from its folder (like /apps/appname/ver/binary) give more information than "bad system call"
[13:02] <jdstrand> I saw dtb too
[13:02] <longsleep> c-lob: all right
[13:02] <ogra_> jdstrand, jjohansen, you can ignore the System.-map, config and dtb stuff though
[13:02] <longsleep> c-lob, Chipaca : I narrowed it down to  the -i parameter in sed. sed works just fine without -i
[13:03] <Chipaca> oops, missed your syslog
[13:03] <Chipaca> sorry was pulled into a different thing
[13:03]  * Chipaca reads now
[13:03] <longsleep> Chipaca: yeah there is not much in syslog
[13:03] <longsleep> c-lob: no further details with callling /bin/sed instead just sed
[13:03] <Chipaca> longsleep: and sed -i works outside of a script?
[13:04] <longsleep> Chipaca: when i run it as root yes
[13:04] <longsleep> let me try again
[13:05] <Chipaca> jdstrand: an idea what can cause a silent "bad system call" with nothing in syslog, when running something as root under seccomp, but not when running as root wihtout it?
[13:05] <pilil> jdstrand, can you help me with question about system wrappers? After installation all apps and services should be started only through wrappers, but when one wrapper trying to launch another wrapper I receive errors like this:
[13:05] <longsleep> Chipaca: yes confirmed, sed -i works fine as root
[13:05] <pilil> $sudo /usr/bin/ubuntu-core-launcher myapp myapp_service_0.0.1 /apps/myapp/0.0.1/bin/myapp-wrapper
[13:05] <pilil> ubuntu-core-launcher: error while loading shared libraries: cannot apply additional memory protection after relocation: Permission denied
[13:05] <pilil> even with frameworks, without ubuntu-core-launcher - everything ok. Is there proper way to launch one file from another?
[13:06] <Chipaca> pilil: i don't think you should call one wrapped thing from another
[13:07] <Chipaca> inside the same app, call your things directly
[13:07] <Chipaca> outside the same app yes
[13:07] <jdstrand> pilil: Chipaca is right. if it is in your own snap, just use $SNAP_APP_PATH/path/to/your/thing
[13:07] <Chipaca> *gasp*
[13:08]  * Chipaca takes a snapshot and frames it
[13:08] <pilil> Chipaca, if I have framework LXC, how could i run commands like lxc-ls from my app?
[13:08] <jdstrand> Chipaca: it was bound to happen sometime

[13:08]  * jdstrand hugs Chipaca :)
[13:08] <jdstrand> pilil: the framework has to expose that via its framework policy
[13:09] <Chipaca> :)
[13:09] <jdstrand> pilil: so install the framework, then you can do 'snappy-security list'
[13:10] <jdstrand> pilil: then have your snap depends on the framework and use either the security-template or caps (for policy groups), or both in your yaml for the service/binary
[13:11] <jdstrand> Chipaca: yeah, that is one strange error
[13:11] <pilil> jdstrand, even if policy set up to unconfined, we have this error. Or there are some other policy?
[13:11] <jdstrand> Chipaca: (Bad system call). that isn't a seccomp denial
[13:12] <jdstrand> pilil: if the policy is unconfined, don't use a wrapper, just go into the install directory of the the you want to execute
[13:12] <jdstrand> pilil: /apps/foo/current/path/to/binary
[13:12] <Chipaca> longsleep: do you still get it if you set your policy to unconfined?
[13:13] <jdstrand> Chipaca: that feels like the launcher was compiled on one system that had that system and run on another that didn't
[13:13] <longsleep> Chipaca: how would i do that? Didnt investigate on policies yet
[13:13] <pilil> jdstrand, yes, now we deal with this way, but its look insecure, and we are researching how to improve security
[13:13] <jdstrand> s/that system/that system call/
[13:13] <longsleep> Chipaca: i have caps: networking and network-service
[13:13] <jdstrand> pilil: well, you are running unconfined so... :)
[13:13] <jdstrand> pilil: when you go confined, use the framework policy method I described
[13:14] <pilil> it was just first run, we have plans to use profiles :)
[13:14] <longsleep> Chipaca: from my syslog, does it not already run with unconfined?
[13:14] <longsleep> Chipaca: operation="profile_replace" profile="unconfined"
[13:14] <jdstrand> pilil: a framework will express what it is safe to do via its framework-policy
[13:15] <pilil> jdstrand, thanks, now I need addition research about it
[13:15] <Chipaca> longsleep: no, I don't think that's your app there
[13:16] <jdstrand> pilil: fyi, https://developer.ubuntu.com/en/snappy/guides/frameworks/ and https://developer.ubuntu.com/en/snappy/guides/security-policy/
[13:16] <longsleep> Chipaca: its not - but it has name="spreed-webrtc.sideload_spreed-webrtc_0.0.1" pid=1718 in the line?
[13:16] <pilil> jdstrand, yeah, already there, thanks
[13:17] <Chipaca> longsleep: yes; I don't know what that means (jd.strand would know), but i do know that unless you're specifying unconfined in your package.yaml, it'll be confined
[13:18] <jjohansen> ogra_: I am afraid I am missing some context to get that scipt to work for me. I assume you have mounted the image and are in its root?
[13:18] <longsleep> Chipaca: so i just add security-template: unconfined ?
[13:18] <ogra_> jjohansen, no
[13:19] <ogra_> jjohansen, i create a device tarball fs structure and tar that up after putting the files in place
[13:19] <ogra_> jjohansen, then you use it with ubuntu-device-flash to create an image (with the --device-tarball option pointing to it)
[13:19] <jdstrand> longsleep: yes
[13:19] <longsleep> Chipaca, jdstrand: When running unconfined it just works
[13:20] <jdstrand> longsleep, Chipaca: the STATUS line is just telling you that the profile was reloaded into the kernel
[13:20] <Chipaca> ta :)
[13:20] <Chipaca> i knew it was an irreal elephant, but not exactly why
[13:20] <longsleep> so, but this cannot be the solution right?
[13:20] <Chipaca> nope
[13:20] <longsleep> i mean we do want to run things confined
[13:21] <Chipaca> jdstrand: is there an easy way for longsleep to run the service confined, but via strace?
[13:21] <longsleep> well there is no strace in my snappy
[13:21] <Chipaca> longsleep: you can copy strace-the-binary from ubuntu and it'll work
[13:22] <longsleep> Right, assuming i find an armhf one somewhere - let me look
[13:22] <Chipaca> heh
[13:22] <Chipaca> you don't have an armhf notebook?
[13:22] <Chipaca> ;-p
[13:22] <Chipaca> longsleep: i can put one up for you, give me a bit
[13:22] <Chipaca> longsleep: you based on 15.04?
[13:22] <jdstrand> Chipaca: look at the line in the systemd service file for the launcher and do 'sudo strace -o /tmp/strace.out -f ubuntu-core-launcher ...'
[13:23] <jjohansen> ogra_: device-tarball is only available for touch, I am trying to do core?
[13:23] <Chipaca> jdstrand: or edit the start script similarly :)
[13:23] <Chipaca> good one
[13:24] <jjohansen> ogra_: can I just use touch instead of core?
[13:24] <Chipaca> not the start script, the systemd file
[13:24] <jdstrand> Chipaca: well, the start script will run strace under confinement. the way I described strace will be out of confinement
[13:24] <Chipaca> jdstrand: yep yep
[13:24] <jdstrand> both can be useful
[13:24] <longsleep> Chipaca: yes 15.04
[13:24] <Chipaca> jdstrand: last i tried, strace wouldn't play well seccomp
[13:24] <jdstrand> so, the service probably needs to have the cd to the WorkingDirectory and the env setup to what is in Environment
[13:25] <jjohansen> Chipaca: strace should work with seccomp, if seccomp allows ptrace
[13:25] <jdstrand> Chipaca: yeah, to run strace under confinement the security policy would have to be modifed, which took us out of 'an easy way' :)
[13:25] <jdstrand> Chipaca: I mean, I could get you there... :)
[13:26] <Chipaca> jjohansen: jdstrand: after my holidays, i'll try strace under seccomp
[13:26] <longsleep> Chipaca: ok i got strace, hold on a sec
[13:26]  * Chipaca makes a note
[13:26] <ogra_> jjohansen, sorry, --device-part= for core ... not -tarball
[13:26] <Chipaca> longsleep: ah, ok :)
[13:26]  * Chipaca had just found it
[13:28] <Chipaca> http://people.canonical.com/~john/strace_4.8.1ubuntu5_15.04_armhf fwiw
[13:28] <Chipaca> longsleep: you know what to edit to use that?
[13:29] <jjohansen> ogra_: what is the best way to mount these images, last touch images I mount were fine, these core images fail
[13:29] <jdstrand> Chipaca: for your after holidays notes> you can just drop syscalls into /var/lib/snappy/seccomp/profiles/foo. I have a feeling you'll need something for apparmor too (a simple '/path/to/strace' ixr,' and 'ptrace,' in /var/lib/apparmor/profiles/...  would probably get you very close)
[13:29] <sergiusens> jjohansen: https://github.com/longsleep/snappy-odroidc#build-snappy-image-for-odroid-c1
[13:29] <sergiusens> jjohansen: kpartx -avs img.img; mount ...; umount ...; kpartx -ds img.img
[13:30] <jjohansen> sergiusens: hrmm okay, maybe I have a corrupted image
[13:30] <longsleep> well .. Jul 31 13:30:10 odroid ubuntu-core-launcher[2762]: /apps/spreed-webrtc.sideload/0.0.1/bin/strace: test_ptrace_setoptions_for_all: unexpected signal 31
[13:30] <jjohansen> sergiusens: thanks
[13:30] <longsleep> Chipaca: did i do something wrong or strace does not work
[13:30] <Chipaca> longsleep: what did you edit?
[13:31] <sergiusens> jjohansen: parted on the img file might tell yo, but if it's x86, it should have 5 partitions
[13:31] <longsleep> my start script
[13:31] <longsleep> Chipaca: so i have a strace line now in the start script for sed
[13:31] <Chipaca> longsleep: ah. no :) edit your systemd service file
[13:31] <Chipaca> longsleep: /etc/systemd/system/somethingobvious
[13:31] <longsleep> ah ok
[13:32] <Chipaca> longsleep: and may i recommend strace -s 999 -f -o /tmp/mytrace
[13:32] <Chipaca> and then the launcher
[13:32] <Chipaca> i.e. strace -s 999 -f -o /tmp/mytrace ubuntu-core-launcher yadda yadda
[13:33]  * Chipaca notes signal 31 is USR2
[13:34] <longsleep> Chipaca: all right http://paste.ubuntu.com/11973341/
[13:34] <longsleep> oh i didnt at the parameters
[13:34] <longsleep> hold on
[13:36] <longsleep> Chipaca: and here it comes: http://paste.ubuntu.com/11973352/
[13:39] <jdstrand> longsleep: can you paste 'sudo grep audit /var/log/syslog'?
[13:39] <longsleep> sure
[13:40] <longsleep> jdstrand: http://paste.ubuntu.com/11973370/
[13:42] <Chipaca> jdstrand: nothing strange there, amirite?
[13:42] <Chipaca> i think this needs to go to a bug
[13:42] <Chipaca> i'll see if i can reproduce it, then file it myself
[13:43] <Chipaca> meanwhile, longsleep, you could do http://pastebin.ubuntu.com/11973383/
[13:43] <longsleep> Chipaca: yeah it could be related to my kernel as well
[13:43] <Chipaca> longsleep: avoid an extra exec, and an extra file move :)
[13:43] <jdstrand> Chipaca: yeah, there is no seccomp denial
[13:43] <longsleep> Chipaca: yes sure, i can go without -i
[13:44] <Chipaca> longsleep: you know -i isn't *actually* in place, yes?
[13:44] <Chipaca> it creates a tmpfile then moves it over
[13:44] <longsleep> Chipaca: yes - it creates a tmpfile
[13:44] <Chipaca> so your .new was dupe effort
[13:44] <longsleep> i see those by the way
[13:44] <Chipaca> yep, see it in strace too
[13:55] <jdstrand> ogra_: so, in thinking about it, there is no reason why in an generic-amd64 vm jjohansen can't just remount rw, put the kernel wherever grub is looking for it, remount ro and reboot, right?
[13:55] <ogra_> jdstrand, yeah
[13:55] <jdstrand> ok cool :)
[13:55] <ogra_> i thought he wanted to build an image
[13:55] <ogra_> sorry, i misunderstood that
[13:55] <jdstrand> well, that was the question posed to you, but the motivation was to test a debug kernel
[13:56] <Chipaca> :)
[13:56] <jdstrand> but now we have all this very interesting information that will just go away once kernel snaps are implemented :)
[13:56] <ogra_> yeah, for that you can just cp ... but be careful with modules ;)
[13:57] <ogra_> (initrd side specifically)
[13:57] <jdstrand> knowing jj, he isn't changing abi for what he is looking at, but note taken
[13:57] <jdstrand> jjohansen: ^
[13:58] <jdstrand> jjohansen: I suppose I should be the one to apologize for not thinking of the cp into place sooner :)
[13:59] <jjohansen> well, uh that would be nicer if it was the same kernel version
[13:59] <jjohansen> don't ask
[14:00] <jdstrand> heh, well then yes, take ogra_'s point to heart I guess :)
[14:01] <ogra_> not sure if/which modules are needed on x86
[14:01] <ogra_> perhaps it just works, else you need to repack the initrd (try if update-initramfs works)
[14:05] <jjohansen> it appears too
[14:05] <jjohansen> so I am in for some manual copying fun
[14:07] <elopio> hello!
[14:10] <longsleep> Chipaca: well i hit the next obstacle: openssl rand -hex 32 yields openssl: Operation not permitted
[14:11] <Chipaca> longsleep: sudo tail -n 100 /var/log/syslog | grep audit ?
[14:11] <jdstrand> that is certainly an apparmor denial
[14:13] <longsleep> Chipaca: http://paste.ubuntu.com/11973546/
[14:13] <Chipaca> DENIED
[14:13] <longsleep> not sure if that is related
[14:13] <Chipaca> I always read that in the quake voice
[14:14] <jdstrand> we don'twe don't allow ixr on the openssl binary. that is arguably a bug. on the one hand, it is in the platform and it is safe security wise to allow. on the other hand, it is in the platform and it adds a potential coupling to a specific ubuntu release (ie, we could update openssl and break people)
[14:14] <longsleep> mhm net_admin and block_suspend?
[14:15] <Chipaca> jdstrand: how can updating openssl break people? is the output different release on release?
[14:15] <jdstrand> Chipaca: we could drop an antiquated cipher
[14:16] <Chipaca> let's not ship antiquated ciphers in the first place! /s
[14:16] <longsleep> :D
[14:16] <longsleep> too late for that
[14:16] <jdstrand> mind you, I am speaking theoretically from the pov that has been expressed that we should only allow the minimum platform deps
[14:17] <ogra_> hey ! what about us patina fans !!
[14:17] <longsleep> in any case, i need a way to create cryptographically secure random strings, private keys and certificates
[14:17] <jdstrand> longsleep: apps aren't allowed to have net_admin - it is far too powerful (see man capabilities)
[14:18] <longsleep> jdstrand: that is helpful thanks - so are you saying openssl does require this?
[14:18] <jdstrand> longsleep: that I am not sure of
[14:19] <jdstrand> longsleep: it could be a harmless denial, but I don't see a denial for openssl itself. did you use snapcraft or deb2snap to build this snap?
[14:19] <longsleep> jdstrand: snapcraft
[14:20] <longsleep> (with my own plugin)
[14:20] <longsleep> i have caps networking and network-service
[14:20] <elopio> fgimenez: could you please write the ips to your jenkins and other machines in canonistack, in the trello card.
[14:20] <elopio> first column.
[14:21] <jdstrand> longsleep: can you try this: adjust /var/lib/apparmor/profiles/<something obvious> to have 'capability block_suspend,' before the trailing '}', then do: sudo apparmor_parser -r /var/lib/apparmor/profiles/<something obvious> then try again?
[14:24] <jdstrand> fyi, there is an open kernel bug on bad logic for checking something ipv6 related which triggers a net_admin denial (that should be harmless) that tyhicks is working on. so lets see if just block_suspend is enough
[14:25] <longsleep> jdstrand: this made no difference, audit now only shows the net_admin deny
[14:25] <longsleep> jdstrand: http://paste.ubuntu.com/11973602/
[14:26] <longsleep> (there are no denies in syslog when it runs openssl)
[14:26] <jdstrand> longsleep: ok, try to add 'net_admin' in the same way
[14:27] <longsleep> jdstrand: ned_admin DENY is gone, openssl still fails
[14:27] <fgimenez> elopio: sure
[14:28] <jdstrand> longsleep: ok, then it is something else. I haven't use snapcraft-- is the binary executable?
[14:28] <longsleep> jdstrand: why binary? openssl? i am using it from the system
[14:28] <jdstrand> longsleep: ie, the openssl binary?
[14:28] <jdstrand> I don't think you are
[14:29] <jdstrand> cause there is no apparmor policy to allow that
[14:29] <longsleep> i just run "openssl rand -hex 32"
[14:29] <jdstrand> I think snapcraft may have shipped a binary in your snap and adjusted your PATH so it seems like you are
[14:29] <longsleep> works fine as root
[14:29] <longsleep> err
[14:30] <jdstrand> find /apps/spreed-webrtc.sideload -name openssl
[14:30] <longsleep> snapcraft doesnt know about openssl
[14:30] <longsleep> its not there
[14:31] <longsleep> jdstrand: http://paste.ubuntu.com/11973632/
[14:33] <longsleep> jdstrand: so you are saying that i cannot run openssl from my snap because there is no apparmor policy to allow it?
[14:33] <longsleep> jdstrand: and i should ship openssl in my snap?
[14:35] <jdstrand> longsleep: hmm, so your snap isn't shipping openssl. this is weird. perhaps the denial is getting rate limited
[14:35] <jdstrand> longsleep: the current apparmor policy does not allow openssl, no. that is easy to for us to fix, but before doing that I want to understand what is happening
[14:36] <jdstrand> longsleep: can you add this rule to the apparmor policy in the same manner as above: /usr/bin/openssl ixr,
[14:36] <jdstrand> longsleep: then try again
[14:36] <longsleep> sure
[14:37] <longsleep> jdstrand: that helped sort of
[14:37] <longsleep> Jul 31 14:37:32 odroid ubuntu-core-launcher[3434]: unable to write 'random state'
[14:37] <longsleep> Jul 31 14:37:32 odroid ubuntu-core-launcher[3434]: message repeated 2 times: [ unable to write 'random state']
[14:39] <longsleep> jdstrand: and it did create the random strings just fine now (so that only seems to be a warning)
[14:40] <jdstrand> longsleep: ok, one last thing. can you remove the net_admin and block_suspend rules, reload the profile and try again?
[14:40] <jdstrand> longsleep: if that works, we can file a bug
[14:40] <longsleep> jdstrand: sure
[14:41] <Rlyeh> Hi
[14:41] <Rlyeh> Does ownlcoud works correctly on ubuntu-core (4)?
[14:41] <longsleep> jdstrand: still works with the unable to write 'random state' warning
[14:41] <Rlyeh> "https://192.168.1.102/owncloud/" returns Not Found!!!
[14:42] <longsleep> jdstrand: full logs: http://paste.ubuntu.com/11973690/
[14:42] <jdstrand> longsleep: oh, this is a go program?
[14:43] <longsleep> jdstrand: yes
[14:43] <jdstrand> right, so that net_admin comment I made earlier applies to you (ie, harmless denial)
[14:43] <longsleep> jdstrand: yes - it seems to work just fine
[14:44] <longsleep> jdstrand: and also the block_suspend DENIED does not seem to have any negative effect
[14:46] <Rlyeh> Solved! "https://192.168.1.102:443"
[14:48] <ogra_> :)
[14:52] <longsleep> jdstrand: the random state file can be specified with export RANDFILE="$SNAP_APP_DATA_PATH/.rnd" - then that error goes away as well
[14:55] <longsleep> jdstrand: do you want me to file a bug or will you do it yourself?
[14:55] <longsleep> Chipaca: could you reproduce the 'sed -i' issue?
[14:59] <jdstrand> longsleep: ok. it seems that kernel rate limiting was in effect and we weren't seeing all the denials. fyi: sudo sysctl -w kernel.printk_ratelimit=0
[14:59] <jdstrand> longsleep: if you could file a bug that would be great
[15:01] <longsleep> jdstrand: ok - the bug should be to allow /usr/bin/openssl ixr with the default apparmor profile right?
[15:02] <longsleep> i am adding key generation, dhparams generation, csr genration and self signing to make sure that works as well with that fix
[15:04] <jdstrand> longsleep: yes
[15:04] <jdstrand> cool
[15:06] <longsleep> should i care about the umask for private keys and stuff or does the confinement handle that?
[15:07] <Chipaca> longsleep: got half way there, got pulled of to see some FTBFS issue
[15:08] <Chipaca> related to the gcc5 move
[15:33] <longsleep> Chipaca, jdstrand If you folks are interested in my final start script: http://paste.ubuntu.com/11973970/ (it works fine when profile allows ixr for openssl.
[15:34] <Chipaca> ta
[15:34] <ogra_> hmm
[15:34] <ogra_> does package.yaml allow globbing for files to be included ?
[15:35]  * ogra_ needs to unclude the overlay/ subdir for rpi overlay dtb's, i dont want to list each dtb individually 
[15:35] <ogra_> *include
[15:35] <Chipaca> longsleep: may i suggest a "sync" after you created the conf?
[15:35] <longsleep> Chipaca: good idea thanks
[15:35] <ogra_> Chipaca, is sysnc allowed ?
[15:36] <Chipaca> we'll find out ;)
[15:36] <ogra_> heh, true
[15:36] <Chipaca> (how could it not be!)
[15:36]  * longsleep checks
[15:37] <longsleep> nope
[15:37] <longsleep> 49: /apps/spreed-webrtc.sideload/0.0.1/bin/start: sync: Operation not permitted
[15:37] <ogra_> yeah, thats what i thought :)
[15:40] <longsleep> jdstrand: so - adding the issue now, in the meantime is there any way to add a workaround to my snap?
[15:44] <longsleep> jdstrand: bug 1480366
[16:00] <longsleep> Chipaca: Maybe you can tell if i can somehow provide my own apparmor profile which allows openssl?
[16:01] <ted> mterry, I find it humorous how we basically did independent clean room implementations of get_arch() and they were basically the same :-)
[16:01] <mterry> ted, ah nice  :)
[16:01] <mterry> ted, only so many ways to do it  :)
[16:01] <Chipaca> longsleep: yes, you can. "webdm" does that, for example.
[16:01] <Chipaca> AFAIR :)
[16:01] <Chipaca> longsleep: also the docker snap
[16:01] <longsleep> Chipaca: great thanks
[16:11] <fgimenez> have a nice weekend everyone o/
[16:34] <longsleep> Chipaca: mhm  snapp.go:498: The "integration" key is deprecated, and all uses of "integration" should be rewritten
[16:35] <longsleep> Chipaca: thats how webdm does it :D
[16:36] <sergiusens> longsleep: no, webdm does it like this: http://bazaar.launchpad.net/~snappy-dev/webdm/trunk/view/head:/pkg/meta/package.yaml
[16:37] <sergiusens> with security-policy:\napprmor|seccomp entries
[16:37] <longsleep> oh i was at http://bazaar.launchpad.net/~snappy-dev/snappy-hub/webdm/view/head:/pkg/meta/package.yaml
[16:37] <longsleep> thats probably wrong then
[16:37]  * sergiusens adds note to delete snappy-hub's webdm
[16:37] <longsleep> sergiusens: thanks for the hint
[16:38] <sergiusens> np
[16:38] <sergiusens> bbs
[16:59] <longsleep> well i just read that custom apparmor profiles trigger manual review, i guess i just add a copy of openssl to my snap
[16:59]  * longsleep found that he cant just copy /usr/bin/openssl :D
[17:28] <jdstrand> longsleep: right, so stepped away for a bit. I'm going to upload this today and it will be in 15.04/edge. it won't hit stable for a few weeks
[17:29] <jdstrand> longsleep: so, since you are using snapcraft, 'just' add openssl to your list of debs and it should add it for you
[17:29] <jdstrand> longsleep: I put 'just' in quotes because I've not used snapcraft and I don't know how easy that is. but other people here do
[17:32] <longsleep> jdstrand: yes i did that - it is easy with snapcraft. Even my own plugin supports it
[17:32] <jdstrand> ok cool
[17:32] <longsleep> I have finished a working snap now, but fail to upload Service unavailable. Please try again later. ([])
[17:32] <longsleep> store seems to be borked
[17:32] <jdstrand> beuno: fyi, ^
[17:33] <beuno> looking into it
[17:35] <longsleep> jdstrand: with my debs plugin i can just add any already built deb file from url or file source into the snap. That way i can easily build armhf snaps on amd64 with snapcraft for clean room built debian packages.
[17:36] <jdstrand> neat
[17:36] <longsleep> jdstrand: http://paste.ubuntu.com/11974578/ for the snapcraft file
[17:36]  * longsleep likes snapcraft
[17:37] <jdstrand> yeah, it is really coming along aiui
[17:37] <jdstrand> mterry, ted, rsalveti, et al: ^ :)
[17:38] <beuno> longsleep, can you try and upload again, while I chase this?
[17:38] <longsleep> now if i would figure out how to push a merge request to launchpad with git i could send the patches for the debs plugin
[17:38] <longsleep> beuno: trying now
[17:39] <longsleep> beuno: nope - still Service unavailable. Please try again later. ([])
[17:39] <beuno> thanks longsleep
[17:42] <beuno> longsleep, what app is this?  everything else looks healthy
[17:42] <longsleep> heh - thats the spreed-webrtc snap i just created
[17:42] <longsleep> (with snapcraft)
[17:43] <beuno> longsleep, so, a new app instead of an update to an existing one?
[17:43] <longsleep> yes new one
[17:59] <beuno> longsleep, you seem to have hit a bug
[17:59] <beuno> some value in your metadata is too long (over 128 characters)
[17:59] <longsleep> beuno: hah - i have a talent for that
[17:59] <longsleep> probably the description
[17:59] <beuno> we'l queue up a fix, but the quickest option would be for you see which one is too long and shorten it  :)
[18:00] <longsleep> sure
[18:00] <beuno> longsleep, it's the title, I'm being told
[18:00] <longsleep> err
[18:00] <longsleep> that should not be long
[18:01] <beuno> "title": "Spreed WebRTC allows people to communicate with audio/video and transfer files over WebRTC. Open Spreed WebRTC with your browser at: https://yoursnappy:8443/ - The SSL certificate, was generated on
[18:01] <beuno>                  installation and is self signed.",
[18:01] <longsleep> thats whats in the readme.md
[18:01] <longsleep> i thought that goes into description
[18:01] <beuno> I think the format in readme.md is:
[18:01] <beuno> - title
[18:01] <beuno> - return character
[18:01] <beuno> - description
[18:01] <longsleep> Ah
[18:02] <longsleep> makes sense
[18:02] <longsleep> let me just provide the title in the web then
[18:04] <longsleep> beuno: Yes that worked. Thanks for your help!
[18:05] <beuno> longsleep, np. Sorry for the hiccup
[18:06] <longsleep> beuno: yay it even passed automatic review
[18:06] <beuno> it was probably embarrased about the bug
[18:14] <longsleep> Chipaca: I managed to put spreed-webrtc into the store (armhf only for now) sudo snappy install spreed-webrtc.longsleep if you want to give it a shot - thanks for your help!
[18:35] <Chipaca> longsleep: congrats!
[18:39] <longsleep> i am traveling the next 4 days - so it would be great if sergiusens would eventually review the updated odrodidc oem snap :P
[19:20] <mterry> ogra_, rsalveti: who has used the webcam demo successfully? I want to pick their brain
[19:25] <Chipaca> mterry: define success
[19:25] <Chipaca> i used it, and it took a pic
[19:26] <Chipaca> longsleep: sergiusens is a new dad, so all bets are off wrt his schedule :)
[19:26] <mterry> Chipaca, I'm using it and it dies with: "GD Error: gd-jpeg: JPEG library reports unrecoverable error: Not a JPEG file: starts with 0x23 0x7d" when taking a pic
[19:27] <mterry> Chipaca, I think some weirdness with my webcam I happen to have
[19:27] <mterry> Chipaca, will play with fswebcam options
[19:27] <Chipaca> mterry: AFAIK it was built for, and only tested with, logitech cameras
[19:27] <Chipaca> mterry: so that's quite likely
[19:27] <Chipaca> mterry: remind me, where was the web demo?
[19:27] <Chipaca> i'll take another look
[19:27] <mterry> Chipaca, I'm using a logitech c170...
[19:27] <Chipaca> webcam*
[19:27] <Chipaca> hey, that should work :)
[19:28] <mterry> Chipaca, webcam-webui is the snap name I believe
[19:28] <Chipaca> mterry: but the source?
[19:28] <mterry> Chipaca, oh..  https://developer.ubuntu.com/en/snappy/guides/appliance-builder-guide-webcam/ is how to build one
[19:29] <mterry> Chipaca, I don't know where the source for our package is
[19:33] <Chipaca> mterry: no worries
[19:33] <Chipaca> mterry: so, question, have you looked at the image file?
[19:33] <Chipaca> mterry: or is that error thrown by fswebcam before actually producing the image?
[19:33] <mterry> Chipaca, using "-p YUYV" fixed it!
[19:34] <mterry> Chipaca, per https://www.raspberrypi.org/forums/viewtopic.php?f=45&t=60076
[19:34] <mterry> Chipaca, fswebcam wasn't making the image at all (or rather, it was spitting out a blank black jpeg
[19:34] <Chipaca> hah! good one
[19:35] <mterry> Chipaca, thanks for looking at it anyway  :)
[19:35] <Chipaca> no worries
[19:35] <Chipaca> i'll go have another beer, this one in your honour
[19:35] <Chipaca> actually some pizza first
[19:36] <mterry> :)
[19:36] <Chipaca> don't worry, it'll be beer o'clock for you soon
[19:47] <Chipaca> mterry: --resolution is also good