=== shuduo is now known as shuduo_afk === chihchun_afk is now known as chihchun [07:16] good morning === carlo_ is now known as Guest69055 === Guest69055 is now known as clobrano === zyga-afk is now known as zyga === _morphis is now known as morphis === shuduo_afk is now known as shuduo [08:30] moin folks, first question of the wekk, can i somehow make a /var/log location available for a service in a snap? [08:34] not under confinement i think [08:35] stdout is logged to syslog though [08:35] (or to joornald if you prefer that) [08:35] *journald [08:44] ogra_: well i have a daemon which opens /var/log/blah.log and logs there until it reads its config to change the location .. kind of sucks and i was wondering if i can get around to recompile it [08:48] well, i assume somethin similar to /tmp might be possible here [08:50] ogra_: yeah - thats what i was thinking [09:39] kickinz1, whats up woth owncloud ? seems it doesnt work at all anymore since the docker update [09:40] ogra_, I started looking at it seems like it doesn't get SNAP_APP_DATA_PATH env. I need to recontruct it. [09:41] well, as long as it is known :) [09:43] ogra_, I plan to upload it soon to the store (ran out of time last week to work on it) [09:43] yeah, i didnt mean to be pushy or anything ... just noticed it on the weekend [10:12] ogra_: just checked the code for ubuntu-core-launcher, there is no way to setup additional bind mounts for snaps. So seems like i have to recompile the service :/ [10:15] longsleep, well, there is code that handles /tmp somewhere ... the same code could be used for log handling [10:15] (not by you ... ) [10:15] (transparently provided to you rather ... like /tmp is) [10:17] ogra_: yes but this would need a change in ubuntu-core-launcher - too complicated i guess [10:17] ogra_: code for tmp is here http://bazaar.launchpad.net/~snappy-dev/ubuntu-core-launcher/trunk/view/head:/src/main.c [10:17] Chipaca, ^^^ ? [10:40] longsleep: "additional bind mounts for snaps"? [10:40] i'm probably missing context here [10:45] also i probably caught you in your lunch break :) [10:45] ah [10:46] longsleep: yeah, no, no way. either recompile, or wrap it with a preloader that changes open of things in /var/log to somewhere else [10:46] there's no way to tell it to use something else? [10:48] Chipaca, well, on the phone we have per-app upstart los from ubuntu-app-launch ... i'm wondering if the same wouoldnt make sense on snappy [10:48] *logs [10:48] ogra_: the per-app upstart logs are just logging stdout/err to a predetermined file, amirite? [10:48] ogra_: it's not that it allows you to write to /var/log/blah.log [10:48] yeah [10:50] i think we might be able to tell journald to write the per-app journal to a file [10:50] but i'm not sure we want to [10:50] of anything, we might want to expose that to app devs [10:50] ie not make it the default [10:51] hmm, yeah [10:51] s/of/if/ fwiw [11:35] still no new snappy :'( === vsuojane1 is now known as vsuojanen [11:45] longsleep: I tried to pass the things you mentioned around RPi2 and other issues to Paolo, but it's not great on IRC; would you mind opening a bug describing the rpi2 smc issue? it seemed like something we could address in the kernel once and for all [11:45] longsleep: the other things... these seemed like workload specific and we just need a facility to set them === om26er_ is now known as om26er [12:17] lool, not sure what makes you think longsleep had the smsc issue :) we dont have a pckage for the RPi kernel or any way to track the bugs for it ... but paolo is aware of the issue and will work on it [12:18] ogra_: I know the smc affects everyone, it's just that longsleep brought it up; I wasn't aware of it before that and wanted to pass it cleanly to paolo [12:18] the smsc issue only affects people that have it [12:19] and it was me who brought it up [12:19] ogra_: how do you mean only affects people that have it? [12:19] ogra_: so it's for an additional ethernet adapter, not the main one? [12:19] and as i told you on friday there is no "suits all" solution [12:20] smsc is a USB NIC [12:20] only devices that use it will be affected by the turobo mode bug [12:20] ogra_: I see [12:20] ogra_: so what's the compromise here? [12:21] and you have two ways to work around it ... either by switching off turbo mode or by bumping the vm_min size up [12:21] for the RPi kernel we can easily do the latter so people dont see decreased performance ... [12:21] if you would want to do this in -generic i wouldnt just bump the min addr [12:22] since it might have other effects and the NIC is only used in sepcific boards [12:22] (not in the BBB for example, which has a real NIC, nothing USB attached) [12:24] lool: yeah rpi2 is more for ogra_ [12:25] lool: my issue was more a general one to inject kernel parameters (sysctl, proc stuff, ...) as early as possible without having to change initrd or other hacks [12:25] longsleep: ack [12:26] lool: if you want me to add a whishlist issue for that i can certainly do that [12:26] longsleep: yeah, I don't think we have anything tracking this ATM [12:26] it's all on our minds, but amongst a gazillion other issues [12:27] ogra_: I don't feel I understand why it can't be fixed properly upstream; if it can't I guess we want to document how folks would do this on top of snappy and move on [12:27] ogra_: is there an upstream pointer somewhere about this? you might have provided this on friday, but I can't find it right now [12:29] GRMBL ... [12:30] Chipaca: just saw you replies - yeah its compiled with a static path to /var/log and stuff is logged there until the config file is loaded - i guess i could hack some preloader [12:30] Chipaca: but a general approach to add more bind mounts within snaps would be nice to have [12:34] ogra_: https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=5898&hilit=ethernet%2bhot&start=50 seems to have the most complete report of the smsc issue [12:34] lool, we have like 6 bugs open for that in LP since 2009 :) [12:34] ogra_: with this linaro bug describing a workaround in linaro ubuntu images https://bugs.launchpad.net/linux-linaro/+bug/664477 [12:34] Launchpad bug 664477 in Linaro Linux "System hangs due to cascade of smsc95xx errors " [Medium,Fix released] [12:34] from 2011 [12:34] yeah, there are older ones too :) [12:35] * ogra_ researched all that last week before bringing it up here [12:35] (and discussiong it with ppisati) [12:35] for the Pi kernel we will just bump the value [12:35] but there is no generic way of fixing it [12:36] ogra_: the fix is to disable turbo mode? [12:36] you can bump the vm.min_size too [12:36] longsleep, well, that costs yoiu performance [12:37] ogra_: yeah thats why i asked [12:37] ogra_: what parameters are you actually adding now? [12:37] right, we will bump the default of vm.min_size of the RPi kernel [12:37] if I understand correctly, the speed at which frames come in overflows the memory the kernel can allocate from driver context [12:37] which is 8192 or some such currently [12:37] ah [12:38] it would seem to me the kernel config at least, or the runtime should detect this combination of options and bump the vm.min_size [12:38] but in any case, we should tune our config system to set it everywhere [12:38] well, on devices where we have that HW and have a dedicated kernel it is easy to just change the kernel default [12:38] ogra_: it's USB though, so you could plug it into anything [12:39] yeah [12:39] i wonder if it makes sense toi generally set the vm.min_size to 16k or 32 ... [12:39] i'm not sure what impact that has [12:40] (on devices that wouldnt necessarily need it) [12:40] ogra_: I believe Kconfig can set a value as a recommended default; we could probably upstream a patch to do this [12:40] i know in the past we fixed it in userspace by shipping a sysctl.d snippet by default for all arm builds [12:40] yup [12:40] grrr ... why dont we have parted on snappy :P [12:41] ogra_: cause you haven't snap-ed it :-) [12:41] * ogra_ wrangles with partition resizing for /writable ... [12:41] would be helpful if i could test on the actual install :) [12:42] lool, i havent snap'ed it because nobody did rebuild the archive for static.archive.ubuntu.com yet :P [12:42] (would be awesome if we could just do that :) ) [12:42] it's not clear whether we want static binaries of everything [12:42] consider the case where you have like 50 binaries in your snap using the same libs [12:43] no, indeed, but it makes "quickly snappin somethin" so much easier :) [12:43] ogra_: you tried snapcraft? [12:43] it's pretty good at this too [12:43] (and my kbd still needs a new g key ...) [12:43] snappin' [12:43] yeah :) [12:43] somethin' [12:44] i should add some autocorrection to xchat to replace g with ' [12:44] it works if i hit it very hard ... just not while typing it in a normal way [12:45] * ogra_ knew it was a mistake to buy a logitech kbd when he bought it [12:45] shiny technology ... krappy mechanics [13:02] ogra_, it was made for angry people [13:02] haha [13:02] ogra_, you're just too laid back === om26er_ is now known as om26er === lp|conference is now known as lazyPower [13:57] one of my services seems to get killed by systemd - main process exited, code=killed, status=31/SYS - any ideas? [13:58] well, there is surely more :) [13:58] (in some log) [13:58] no [13:58] i cannot find any more information [13:59] the service does log on stderr [13:59] it has networking and netowrk-service caps [14:00] Main PID: 1329 (code=killed, signal=SYS) [14:00] mhm === charles_ is now known as chalres === chalres is now known as charles === kyrofa_ is now known as kyrofa [14:26] mhm [14:26] mkdir("/tmp/body", 0700) = 0 [14:26] stat64("/tmp/body", {st_mode=S_IFDIR|0700, st_size=40, ...}) = 0 [14:26] +++ killed by SIGSYS +++ [14:26] Chipaca: ^^^ any idea? [14:26] where is that mkdir ? [14:27] inside your daemon ? [14:27] in my confined service [14:27] yes [14:29] longsleep: um, ouch? [14:29] longsleep: anything in syslog? [14:30] Chipaca: nope :/ [14:30] rate limiting is off [14:30] * longsleep is trying to confine nginx [14:31] well, it fails for /tmp too [14:31] mkdir("/tmp", 0700) = -1 EEXIST (File exists) [14:31] stat64("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=40, ...}) = 0 [14:31] +++ killed by SIGSYS +++ [14:31] so i think it is the stat64? [14:31] longsleep: what does it do after the stat64? [14:31] jdstrand: tyhicks: you around? [14:32] i did not look into the code - hold on === chihchun is now known as chihchun_afk [14:35] hmpf ... i thinnk i shouldnt pull parted into the initrd ... [14:36] not sure we want libreadline in there [14:37] (and ncurses through that ) [14:38] ogra_: sfdisk should be good enougth? [14:38] well, we used to use that in the past for the same purpose ... i'd prefer parted since thats what we use everywhere else though [14:38] (installer etc) [14:40] but yeah, iirc that was actually the reason why we picked sfdisk back then too === chihchun_afk is now known as chihchun [14:43] Chipaca: I am, Tyler is not [14:44] longsleep: is the SIGSYS in a service, or in commandline? [14:44] Chipaca: no idea what comes next :/ body folder is created fine and then bam [14:44] Chipaca: service [14:44] jdstrand: these lackeys always slacking off, huh [14:44] heh [14:44] he is on holiday today, back tomorrow :) [14:46] jdstrand: would a SIGSYS be seccomp doing something? [14:46] or is it unrelated to that? [14:46] seccomp should send SIGKILL [14:47] and you would see a log entry [14:47] yah [14:47] longsleep: SIGSYS usually means "bad system call" [14:47] longsleep, are you sure its not an amd64 binary ? [14:47] :) [14:48] ogra_: yes it runs fine when unconfined or run directly [14:48] ooh [14:48] longsleep: strace that :) [14:48] Chipaca: output is from strace :) [14:48] longsleep: strace -f -s 999 -o /tmp/trace yoursvchere [14:48] longsleep: i mean unconfined [14:48] ah [14:48] ok [14:48] good idea [14:48] longsleep: then we see what comes next \o/ [14:51] is there a quick way to unconfine a installed snap? [14:52] longsleep: edit the call to ubuntu-core-launcher and plug unconfined instead of the second parameter? never tried it though :) [14:52] longsleep: that'd mean editing /etc/systemd/system/something.service [14:53] longsleep: or you could just remove the call to the launcher entirely :) [14:53] ok next comes chown32("/tmp/body", 65534, -1) = 0 [14:53] guess that is the problem [14:54] so .. no change owner? [14:54] at least not to nobody i guess [14:55] longsleep: that's interesting [14:55] jdstrand: would you expect that? [14:55] jdstrand: it's clearly something in our confining apparatus doing it, but i don't understand what/how [14:56] we don't allow chown because there is no reliable uid to chown to [14:56] but, this is unconfied, no? nothing should be blocking it [14:56] i want to run it confined [14:56] (this is a web server, so it really should be) [14:56] jdstrand: we found out it was chown running it unconfined [14:56] also, we are using SCMP_ACT_KILL in the launcher so a sigsys won't be sent [14:56] jdstrand: running it confined we get sigsys, and nothing in the logs [14:57] hence a puzzled chipaca [14:57] i say we, but it's all longsleep all the time :) [14:57] yeah but what you say is how it is [14:57] seccomp can send sigsys, but only if seccomp_init is given SCMP_ACT_TRAP [14:57] which makes me think maybe i should do a test case and see if i can reproduce it [14:58] Chipaca: well, this is nginx startup [14:58] systemd can also send sigsys if a syscall filter is setup through systemd [14:58] ooooh [14:59] jdstrand: that might be it then? [14:59] are we setting that up now? it has to be something specific in the service file [14:59] jdstrand: not unless we're doing it unawares [14:59] or it's the default or something [15:00] it should not be the default [15:00] everything would break [15:00] http://www.freedesktop.org/software/systemd/man/systemd.exec.html [15:00] SystemCallFilter= [15:01] i do not see anything in the service file [15:01] http://paste.ubuntu.com/12108429/ [15:01] plus it would be redundant with the launcher syscll filtering [15:02] ok, meeting now [15:02] after meeting will reproduce with small case [15:02] then pester you again jdstrand ;) [15:02] well what i see is that chown seems to trigger sigsys [15:02] all right - thanks guys! [15:02] chown should trigger sigkill [15:03] and there should be a log entry [15:03] but I didn't think you got all the way to chown before [15:03] I thought the sigsys came before that and you only saw chown when you went unconfined [15:03] jdstrand: well strace says +++ killed by SIGSYS +++ [15:03] yes [15:05] jdstrand: here are the two strace parts http://paste.ubuntu.com/12108465/ [15:06] so i concluded the chown is the reason for the kill [15:07] longsleep: according to man 7 signal, SIGSYS is sent when a bad argument is used. are you specifying a uid to chown that doesn't exist? [15:07] jdstrand: that could be it [15:07] jdstrand: what uid should i actually use? [15:08] well, as mentioned, you shouldn't use chown at all [15:08] jdstrand: but the uid 65534 is nobody and it does exist [15:08] if you picked one that existed, then seccomp would kill you [15:08] jdstrand: well it is done by Nginx - i do not have much a say in the matter [15:09] jdstrand: so you say when i chown to an existing uid then seccomp will kill the process? [15:09] yes [15:09] thats it then [15:09] except I don't think it is [15:09] because you are seeing the wrong signal [15:10] but really, it doesn't matter, cause if you fixed whatever this problem is, seccomp would block you [15:10] ok - but it is calling chown with a uid which does exist [15:11] the reason why we don't allow it at the moment is because the launcher doesn't support syscall argument filtering and because there is no reliable uid to chown to (in general) [15:11] so, patch out the chown or otherwise tell nginx to run as root [15:12] jdstrand: nginx will not run as root, if you run it as root it will setuid to nobody [15:12] and i was trying to avoid patches :/ [15:12] it sounds like that behavior needs to be modified [15:13] fyi, syscall arg filtering is being tracked in bug #1446748 [15:13] Bug #1446748: implement seccomp filtering by argument

[15:13] bug 1446748 in ubuntu-core-security (Ubuntu) "implement seccomp filtering by argument" [Wishlist,Triaged] https://launchpad.net/bugs/1446748 [15:13] but that wouldn't be enough for your case-- you also need a non-root uid to start as [15:13] which is another unimplemented feature in snappy [15:14] from the seccomp security policy: [15:14] # snappy doesn't currently support per-app UID/GIDs so don't allow chown. To [15:14] # properly support chown, we need to have syscall arg filtering (LP: #1446748) [15:14] # and per-app UID/GIDs. [15:14] Launchpad bug 1446748 in ubuntu-core-security (Ubuntu) "implement seccomp filtering by argument" [Wishlist,Triaged] https://launchpad.net/bugs/1446748 [15:15] jdstrand: all right, so if understand this correctly, i would need both to run nginx unconfined [15:15] err confined [15:16] because nginx does fail with chown, and then will try to change the uid as it is run as root [15:17] jdstrand: so the much bigger problem is that there is no way to start a snap service with a uid other than root right? [15:18] it is a missing feature [15:20] jdstrand: ah the trick was simple after i understood this: user root root; [15:20] then it will not chown and not setuid/gid [15:20] jdstrand: thans for explaining! [15:20] +k [15:21] ok, glad that worked. still puzzled why you were sent a sigsys [15:23] jdstrand: well yeah - no i get the next sigsys .. seems like the worker processes get killed of [15:23] f === chihchun is now known as chihchun_afk [15:37] jdstrand: confirmed that i get SIGSYS not SIGKILL when running something that does chown [15:38] jdstrand: but, i do get something in the log [15:38] [Mon Aug 17 15:36:52 2015] audit: type=1326 audit(1439825813.354:13): auid=1000 uid=1000 gid=1000 ses=1 pid=994 comm="q" exe="/apps/hello-world.canonical/1.0.18/bin/q" sig=31 arch=c000003e syscall=92 compat=0 ip=0x7f65d470c4d7 code=0x0 [15:39] Chipaca: can the audit log crash, the last audit stuff i see is from ths morning :/ [15:39] longsleep: what do you get if you say "sudo sc-logresolve"? [15:39] Chipaca: nothing - no output [15:39] ah well [15:40] longsleep: i haven't known it to, but i tend to fire a kvm up, test stuff, and shut it down again [15:40] my bbb hasn't ever crashed in that way that i know of though [15:40] also i don't think it's "a thing" that can crash [15:41] Chipaca: let me try to reboot, i have the next sigsys kill - means i can only run nginx without a master process which is not suitable for production [15:41] you'd lose all logs, not just the audit ones [15:41] right [15:41] rebooting now [15:43] Chipaca: no audit logs for me but plenty of worker process 6715 exited on signal 31 [15:43] :( [15:44] it might because i'm not doing this as a service [15:44] that's next [15:45] man, i need tab completion on 'snappy install' [15:46] Chipaca: alias si [15:55] longsleep: so, as a service, SIGSYS, with log entry [15:55] e.g. Aug 17 15:51:10 localhost kernel: [ 945.918847] audit: type=1326 audit(1439826670.799:20): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=1120 comm="q" exe="/apps/xkcd-webserver.canonical/0.5/bin/q" sig=31 arch=c000003e syscall=92(chown) compat=0 ip=0x7f8da417a4d7 code=0x0 [15:55] (as output from sc-logresolve) [15:59] Chipaca: mhm ok then - so any ideas why it could not log that for me? [15:59] longsleep: well, two ideas [15:59] longsleep: my environ is not your environ -- i'm trying on a kvm with wily rolling and you're on 15.04 [15:59] Chipaca: yes and on armhf [15:59] longsleep: and, your thing is doing a whole lot more than mine is [16:00] longsleep: want to try running my thing on your thing? [16:00] longsleep: otherwise i could try running my thing in your thing :) [16:00] Chipaca: sure - do you have an armhf snap? [16:00] longsleep: no, i just compiled it and copied it over [16:01] longsleep: but i'll make one for you [16:01] give me a bit though :) [16:01] Chipaca: you can try my snap if you have an armhf environment [16:02] longsleep: i've got a bbb, but i'd have to reflash it to know where it stands (i usually break them! because, um, i just compile stuff and copy them over ;) ) [16:03] Chipaca: btw, i see the audit logs when i install or remove snaps [16:04] Chipaca: i meanm the profile status lines [16:07] longsleep: curiouser and curiouser [16:11] Chipaca: just sent you the URL as pm, that triggers signal 31 kills constantly [16:11] longsleep: http://people.canonical.com/~john/min_0_armhf.snap [16:11] Chipaca: (as the worker process is killed) [16:11] so, after looking at the kernel documentation (as opposed to the man page), it does look like the exit status of the task will be SIGSYS in we are setup for kill [16:12] jdstrand: \o/ [16:12] jdstrand: so the only question remaining is why longsleep isn't seeing it in the logs [16:12] longsleep: that snap is untested ;) but it should give you a binary and a service [16:12] perhaps kernel rate limiting is in effect? [16:12] longsleep: that run the same thing [16:12] jdstrand: longsleep mentioned he'd disabled it i think [16:14] jdstrand: sysctl -w kernel.printk_ratelimit=0 [16:14] i have that in effect [16:14] this is on armhf? I have a sneaking suspicion that there is something wrong with logging on armhf cause I have felt that sometimes messages were getting dropped when they should've been logged, but I've never been able to reproduce when I want to [16:14] yes [16:14] armhf yes [16:15] but it is just a feeling [16:15] perhaps if you reboot you'll see it [16:15] i rebooted and no change [16:15] but i see this: audit_printk_skb: 12 callbacks suppressed [16:15] hmm, yeah, I don't know then [16:15] should that happen when i have the ratelimit disabled? [16:15] right so the sysctl value will of course be dropped on reboot [16:16] i applied it again [16:16] Chipaca: your service failed to start but no audit except http://paste.ubuntu.com/12109082/ [16:17] Chipaca: and no audit when running min.q [16:17] longsleep: :( [16:18] so either its my system, my kernel or a general issue on armhf [16:18] maybe you should apply kernel.printk_ratelimit=0 to something in /etc/sysctl.d [16:18] let me try [16:18] eg, /etc/sysctl.d/99-debug-ratelimit.conf or something [16:18] then reboot [16:19] * longsleep reboots [16:21] jdstrand: well i might have an idea [16:21] systemd-journald[360]: Failed to join audit multicast group. The kernel is probably too old or multicast reading is not supported. Ignoring: Operation not permitted [16:21] interesting [16:21] i am on kernel 3.10 [16:22] otherwise no difference after reboot [16:22] is anything in dmesg? [16:22] well [16:22] sysctl -a|grep printk [16:22] kernel.printk = 4 4 1 7 [16:22] kernel.printk_delay = 0 [16:22] kernel.printk_ratelimit = 5 [16:22] kernel.printk_ratelimit_burst = 10 [16:22] it did not apply what i set [16:23] but i added the file [16:23] mhm [16:26] well the procps service does not seem to apply settings from /etc/sysctl.d [16:26] according to http://www.freedesktop.org/software/systemd/man/sysctl.d.html, systemd is supposed to do that [16:27] systemd-sysctl.service [16:27] I don't know if snappy let alone Ubuntu uses that [16:27] yes if i do systemctl restart systemd-sysctl.service it gets applied [16:27] not on boot though [16:28] interesting [16:28] /etc/init/procps.conf is the upstart job [16:28] so that won't be run [16:28] right, still my old habits [16:29] let me boot again [16:29] maybe pitti or ogra_ has insight as to how /etc/sysctl.d should be applied on boot and why systemd-sysctl.service is seemingly not being run [16:29] "systemd-sysctl.service is an early-boot service that configures sysctl(8) kernel parameters" [16:30] not sure why if you created /etc/sysctl.d/99-your-thing.conf why it wouldn't by applied [16:30] (ODROIDC)root@odroid:~# sysctl -a|grep "kernel.printk_ratelimit " [16:30] kernel.printk_ratelimit = 5 [16:30] unless there is a bug [16:30] that is after boot [16:30] and what is the name of the file you created? [16:30] systemctl restart systemd-sysctl.service [16:30] (ODROIDC)root@odroid:~# sysctl -a|grep "kernel.printk_ratelimit " [16:30] kernel.printk_ratelimit = 0 [16:31] how you suggested [16:31] cat /etc/sysctl.d/99-debug-ratelimit.conf [16:31] kernel.printk_ratelimit = 0 [16:31] seems there is a bug that those aren't being applied [16:32] let me try to add it to an existing file then [16:32] I would be surprised if that work-- I imagine the service itself isn't being started [16:32] ah i think that is a mount order issue [16:32] i cannot change the other files - those are readonly [16:33] ah not all of them [16:33] now added it to 10-zeropage.conf [16:33] that would make sense [16:34] did not work either [16:34] so there is a bug for sure [16:35] can you file it against https://bugs.launchpad.net/snappy/+filebug? [16:35] my most likely guess is that /dev/mmcblk0p4 on /etc/sysctl.d type ext4 (rw,relatime,data=ordered) is not mounted when the systemd service is run [16:35] jdstrand: sure [16:36] yeah, when you suspected mount ordering, I was thinking you were probably onto something [16:37] jdstrand, uh, i didnt even know it isnt run [16:37] thats clearly a bu [16:37] g [16:40]