[07:07] good morning [07:31] good morning === ezobn1 is now known as ezobn === Chipaca` is now known as Chipaca [08:16] * ogra_ tickles seb128 [08:16] seems someone found my G+ post about personal ... [08:16] https://lists.ubuntu.com/archives/snappy-devel/2015-September/001033.html [08:16] https://lists.ubuntu.com/archives/snappy-devel/2015-September/001036.html [08:17] seb128, i assume nobody is interested in getting bugs yet for this ? [08:17] hey ogra_ [08:17] not that I know of, no [08:31] Good morning all; happy Monday, and happy Eat a Hoagie Day! 😃 [08:39] Good morning [08:46] JamesTait: what's a hoagie? [08:46] Chipaca: I reviewed your branches, I guess next step is to get them onto the 15.04 image or is something pending from your side [08:47] Chipaca, https://en.wikipedia.org/wiki/Submarine_sandwich#Hoagie [08:48] mvo: we need to decide what to do about activation in 15.04 [08:48] * Chipaca hugs JamesTait for not sending him to lmgtfy [08:49] mvo: also maybe we don't need to backport to 15.04 if we don't want to ship these there and instead ship trunk to 15.04? [08:49] Chipaca, nah, enough people have asked that I realise not everyone watches Man vs Food. 😉 [08:49] mvo: next step for _me_ is thinking a little bit about services, and implementing the fruit of my thinkage [08:50] JamesTait: that looks like a sandwich you can't eat politely :) [08:51] Oh, definitely. [08:51] Chipaca: re trunk> https://bugs.launchpad.net/snappy/+bug/1493706 worried me a bit about that (I still think its a good idea in general, but this worried me about regressions) [08:51] Launchpad bug 1493706 in Snappy "side-loading frameworks broken" [Critical,Triaged] [08:51] Chipaca: activation? what has changed here (pardon my ignorance) [08:52] mvo: oh, maybe you didn't notice? the coreos activation doesn't build on 15.04 [08:52] mvo: it uses os.Unsetenv, which wasn't it go 1.3 [08:52] Chipaca: oh, this activation, sorry, was confused [08:52] mvo: stgraber's activation patch/fork fixes that [08:52] mvo: so we need to decide whether to ship his fork, or his patch :) [08:52] or whether we give up and vendor [08:53] yeah, let me deal with that then [08:53] mvo: and that's one we need to fix independently of backporting or not [08:53] yep [08:54] mvo, the snappy config output if you just call it vs if you change a value look very different now ... if you chnage a value it prints all of the ppp values afterwards, if you just call the config command without setting anything the ppp values are omitted [08:55] setting a value: http://paste.ubuntu.com/12395367/ vs calling snappy config http://paste.ubuntu.com/12395393/ [08:55] ogra_: hm, that looks like a bug :/ [08:55] mvo: you need to stop apologizing to me for the state of the codebase, already; i've been here long enough :) [08:57] mvo, note i also seeded libnss-myhostname on the weekend for bug 1495058 and bug 1464396 [08:57] Chipaca: sorry [08:57] bug 1495058 in Snappy "updating the hostname via snappy config does not update /etc/hosts" [Undecided,New] https://launchpad.net/bugs/1495058 [08:57] Chipaca: :P [08:57] bug 1464396 in Snappy ""sudo: unable to resolve host ..." when custom hostname is used." [Critical,Triaged] https://launchpad.net/bugs/1464396 [08:57] Chipaca: ok, I will try to stop [08:57] mvo: :D [08:57] (just as a FYI) [08:57] Chipaca: do you have a link to the build failure btw? [08:58] Chipaca: because the version of golang-systemd in the archive does not use os.Unsetenv so I'm a bit puzzled [08:58] mvo: https://code.launchpad.net/~chipaca/snappy/serve/+merge/270372 [08:58] ta [08:58] Chipaca: heh, its just our CI that is confused [09:07] Chipaca: I commented in the MP === Chipaca is now known as Chipakeitor === Chipakeitor is now known as __chip__ === __chip__ is now known as Chipaca [09:13] mvo: so with that change to dependencies it'll be happy again? [09:13] ogra_: hm, I just upgraded to ubuntu-core 147 and I do see the ppp config stuff, anything unusual aobut your image? [09:13] Chipaca: I think so, I tested locally and it looked happy [09:14] (ie. checked out the right revno and there was no os.Unsetenv() anywhere) [09:14] mvo: your diff failed here, though [09:14] :( [09:14] I will redo it with a proper branch instead of a inline diff [09:14] it needs tabs [09:14] not sure why [09:14] ah [09:14] maybe LP ate them [09:15] mvo: nah, i can patch by hand [09:16] let's see if this works now :) [09:16] mvo, nope, nothing unusual [09:17] that was a plain install of 145 and i saw/see the same on 146 and 147 [09:18] ogra_: ok, I will try harder to reproduce [09:19] hmm, the machine doesnt boot anymore after auto upgrade :/ [09:19] and indeed i have no serial board for that HW :/ [09:20] well, or it doesnt bring up the static network setup [09:43] serve landed \o/ [09:43] \o/ [09:44] sergiusens: mo'in! [09:44] Chipaca, serve me a beer [09:44] :-P [09:44] Chipaca, good morning [09:44] $ GET http://localhost:8080//1.0/packages/beer [09:44] {"metadata":null,"status":"Not Found","status_code":404,"type":"error"} [09:45] Chipaca, bummer :-) [09:45] sergiusens: easy enough to fix though [09:54] Chipaca, yeah, extend and serve myself :-P [10:06] hmm, so after autopilot upgrade it seems my static network doesnt come up anymore [10:08] mvo, does snappy config apply the /etc/network/interfaces.d file on every boot ? [10:09] Sep 14 09:18:00 aleph2 systemd[1]: ifup-wait-all-auto.service start operation timed out. Terminating. [10:09] Sep 14 09:18:00 aleph2 systemd[1]: Failed to start Wait for all "auto" /etc/network/interfaces to be up for network-online.target. [10:09] Sep 14 09:18:00 aleph2 systemd[1]: Unit ifup-wait-all-auto.service entered failed state. [10:09] * ogra_ sees that in syslog of the failed machine [10:10] oha ! [10:10] Sep 14 09:16:01 aleph2 kernel: [ 43.590490] r8169 0000:01:00.0 eth3: renamed from eth0 [10:10] thats bad [10:14] ogra_: what happened to stable network interface names? [10:14] shouldn't that be ens3 or sth [10:14] 15.04 [10:14] should it there ? [10:14] I don't know [10:14] my brain no longer supports 15.04 [10:14] * ogra_ is running 15.04 edge here [10:14] :) [10:15] it should definitely never rename it [10:15] no matter oif the name is foople0 or eth0 :) [10:17] wow and in /writable/system-data/etc/udev/rules.d i have the persistent-net.rules file with two different entries for the same MAC [10:18] thats gross [10:26] mvo, bug 1495452 [10:26] bug 1495452 in Snappy "auto upgrade unintentionally updates /etc/udev/rules.d/70-persistent-net.rules" [Undecided,New] https://launchpad.net/bugs/1495452 [10:27] pitti, is there a way to prevent this ? ^^^ [10:28] (why would it add a device at all if there is a MAC existing for it in that file already) [10:28] ogra_: that sounds like https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=765577 [10:28] Debian bug 765577 in udev-udeb "netboot install writes duplicates to 70-persistent-net.rules" [Serious,Fixed] [10:29] ogra_: is that vivid? [10:29] pitti, yeah, the 15.04 edge image that we want to release this week [10:29] (as the entire 70-persistent generator is completely gone from wily) [10:30] not sure we could drop the generator since 15.04 still seems to use the old way of naming devices [10:31] ogra_: could you clean up the file, apply http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/diff/?id=f9cb380a, and see if that helps: [10:31] ? [10:31] if so, we can SRU that [10:31] ok [10:31] Chipaca, mvo can I get an ack on https://code.launchpad.net/~sergiusens/snapcraft/collisions/+merge/270942 ? [10:31] well, i'm not really sure how to trigger it again now :( [10:32] to verify it wouldnt do that again [10:33] removing the line manually brings it up normally again ... it doesnt add it anew [10:33] ogra_: nevermind, vivid already does have that thing [10:33] well, then it doesnt help obviously :) [10:34] (teh immage build is actually from yesterday so everything in -security and -updates should be in it [10:34] ogra_: but what's weird is that the file header exists multiple times too [10:34] ) [10:34] it's actually not wrong -- there are three MACs with different names, and the rules are just duplicate, not contradicting AFAICS [10:35] oh, :f8 is eth0 and eth3 [10:35] well, except that the second entry for eth0 is named eth3 [10:35] right :) [10:36] and i also got: Sep 14 09:16:01 aleph2 kernel: [ 43.590490] r8169 0000:01:00.0 eth3: renamed from eth0 [10:36] on the first boot after the auto-upgrade [10:37] sergiusens: I have a look now [10:37] mvo, thank you very much [10:38] ……………………[ -e "$RULES_FILE" -o -e "$tmp_rules_file" ] || PRINT_HEADER=1 [10:38] ogra_: ^ hm, it shouldn't print the header again if the file already exists [10:38] ogra_: could it be that /etc/udev/rules.d/ (or the actual file) is a bind mounted dir/file, which isn't bound in initramfs already? [10:39] i. e. the generator would race with bind-mounting the file? [10:39] pitti, yeah, could be, i have to check the inird script [10:42] pitti, but would the generator run from initrd ? [10:43] the bind mount is definitely done via fstab processing [10:44] ogra_: AFAICS 70-persistent-net.rules gets copied into initramfs, but not the 75-persistent-net-generator.rules -- that shold only happen in the "real" fs [10:45] right and we cant generate the initrd on the device so the file should be either empty or not exist at all [10:45] ogra_: "lsinitramfs -l /initrd.img |grep net-gen" to double-check? [10:45] right [10:45] but if you fstab-mount it, you have a race [10:45] GEEZ ! [10:45] where does lsinitramfs come from and why dont i know about it !!! [10:47] gzip: /boot/initrd.img-3.19.0-28-generic: not in gzip forma [10:47] t [10:47] bah [10:49] (amd64)ogra@aleph2:~$ lzcat /boot/initrd.img-3.19.0-28-generic |cpio --extract --quiet --list|grep net-gen [10:49] (amd64)ogra@aleph2:~$ [10:49] nope, not there [10:50] and the 70-persistent-net.rules file would also just vanish after the bind mount anyway [10:50] it wouldnt be mangled [10:52] pitti, this is clearly triggered only by an upgrade of the readonyl image part ... so i wonder if anything in there triggers it [10:53] * ogra_ tries a snappy rollback and snappy update loop again ... [10:55] rollback comes up fine again [10:56] * ogra_ updates and reboots again [10:58] upgraded fine, didnt change the rules file [10:59] * ogra_ rolls back again and waits tile the autopilot kicks off the upgrade ... probably that causes a race [10:59] ogra_: the ppp issue you see is on arm? or in a vm? [10:59] mvo, on x86 hardware [10:59] sergiusens: the config branch as well? [10:59] ogra_: real hw? interessting, I poke some more [10:59] mvo, config branch is not done yet :-) the previous ones if you want you can review :-) [11:00] mvo, OOOH ! [11:00] mvo, I ran into collision problems when working on config (more complex and multiple parts al including ubuntu packages) [11:00] mvo, sudo vs no sudo makes the difference [11:00] ogra_: oh, of course :/ [11:02] (amd64)ogra@aleph2:~$ snappy config ubuntu-core |grep timezone [11:02] timezone: Europe/Berlin [11:02] (amd64)ogra@aleph2:~$ date [11:02] Mon Sep 14 11:01:59 UTC 2015 [11:02] ogra_, for snappy config? I thought we had 'am I root' detection there [11:02] bah, that doesnt look right either [11:02] sergiusens: yeah, I started but then got distracted [11:03] mvo ogra_, we also need input hand holding in our config for ubuntu-core [11:03] specially with ogra who typos a lot ;-) [11:03] haha [11:03] well, for configuring i used http://paste.ubuntu.com/12396598/ for this particular install [11:04] if you see any typos, please tell me :) [11:05] that the timezone doesnt get set is worrying :/ [11:08] ogra_, I don't see anything wrong, maybe check if /usr/share/zoneinfo/Europe/Berlin exists on the system? [11:08] ogra_, it works fine for me for America/Cordoba [11:08] it exists [11:09] and /etc/writable/timezone is also set correctly [11:09] sergiusens, is that on 15.04 ? [11:09] (where America/Cordoba works) [11:10] ah, now autopilot rebooted it ... /me waits if the interface gets renamed this time [11:11] bah, came up fine [11:12] ogra_, seems it doesn't work anymore http://paste.ubuntu.com/12407670/ [11:12] yeah, bad [11:13] but i wonder why ... the files are all correct [11:13] ogra_, /etc/timezone has the correct data [11:13] right, here as well [11:13] ogra_, did we drop some dependency? [11:13] ogra_, I can be blind and just blame systemd :-P [11:13] not that i'm aware of [11:14] aha [11:14] /etc/localtime points to UTC [11:15] something didnt update it along [11:15] due to the snappy config changes ? [11:15] ogra_, no, this is older [11:15] ogra_, snappy config only ever updated /etc/timezone [11:17] and that worked ? [11:17] ogra_, yeah [11:17] (just asking because the hostname update only worked half) [11:18] ogra_, hostname update has a bug logged somewhere; want to fix? shouldn't be complicated ;-) [11:18] sergiusens, fixed it on the weekend :) [11:18] * sergiusens finds ogra_ fixes things fast to scratch his itches ;-) [11:19] hey guys [11:19] so lets see what happens if i properly update /etc/localtime [11:20] (amd64)ogra@aleph2:~$ sudo cp /usr/share/zoneinfo/Europe/Berlin /etc/localtime [11:20] [sudo] password for ogra: [11:20] (amd64)ogra@aleph2:~$ date [11:20] Mon Sep 14 13:20:06 CEST 2015 [11:20] sergiusens, i cant really belive that ever worked correctly ... (i rather guess nobody wimply noticed) [11:20] *simply [11:21] these files always need to be updated alongside [11:24] mvo, do you know if meta/hooks/config can be a symlink? [11:28] sergiusens, bug 1495474 [11:28] bug 1495474 in Snappy "setting the timezone via snappy config does not actually update the time zone" [Undecided,New] https://launchpad.net/bugs/1495474 [11:31] sergiusens: I think it can and its probably a good idea too, if it does not work there might be a bug [11:32] mvo, great, because I'm working on having snapcraft just symlink ;-) [11:33] pitti, ouch ... [11:33] (amd64)ogra@aleph2:~$ mount|grep -c etc [11:33] 18 [11:33] pitti, dont you think we could simply make the generator run later ... mounting everyting of /etc in initrd sounds wrong [12:09] ppisati, hey, sorry, I lost track, when do you think I'll be able to use pwm on my rpi2? [12:11] rickspencer3: latest kernel has it working (both 3.19 and 4.2) [12:11] rickspencer3: if you are willing to do some manual hacking, you can test it right now [12:11] ppisati, can I just install rolling? [12:11] ppisati, actually, let me ask differently ... [12:12] could you send detailed instructions to the snappy mailing list on how to test it? [12:12] rickspencer3: ok, i'll do [12:13] thanks ppisati [12:13] I'm looking forward [12:21] ogra_: well, you'll just keep running into races like this [12:22] pitti, hmm, yeah :( [12:22] this is like the 10th time this issue bubbles up [12:23] ogra_: you could restart udev once again after mounting everything, but you'd need to do "start udev", "stop udev", "mount everything in /etc", "restart udev" [12:23] well, i guess that needs some discussion first ... i can add a function that parses out all the /etc dirs and mounts them indeed [12:23] do we actually restart it at all ? i thought we carry the running daemon binary over from the initrd [12:24] no, we restart it [12:24] rickspencer3, you will need a device tarball ... i'll roll one durign the week anyway for the 15.04.3 image that is planned this week [12:24] otherwise we'd keep the initramfs-tools' libc6 (and all libs) in RAM [12:24] also, the i-t udev has only a minimal set of rules [12:24] rickspencer3, we dont have automated builds for RPi yet, so there is no specific tarball for the rolling images [12:25] ogra_, ok, noted [12:25] I think if ppisati emails directions to the list, that is the best way forward [12:25] pitti, yeah, i always thought we onyl re-process the rules and keep the daemon running ... but indeed that would keep libc and friends in ram too [12:25] I have all these servos and motors and rgb leds that are hungry for some pwm [12:25] rickspencer3, not if he doesnt use the exact same setup the device tarball will use [12:26] ogra_, um, then he should do that? [12:26] I don't mind updating the image later if necessary [12:26] I would just like to be unblocked [12:26] well, why not just wait a day til i get to it and can offer the ready made tarball for testing [12:27] ogra_, if that's what ppisati's directions say to do, that is fine with me [12:27] * ogra_ is just currently busy with x86 since we have to have a specific release ready for this for 15.04.3 [12:28] (and i run into bugs over and over with nearly everything atm :/ ) [12:30] ogra_: rickspencer3 well, my directions would be how to manually install a kernel, update the dtb, etc [12:30] that's how i test new kernels [12:32] just cp'ing over the binary ? [12:32] yeah, that would work for testing indeed [12:33] ogra_: yep, literally [12:33] and the dtb too [12:33] and the modification to config.txt [12:33] i'm writing it now [12:34] cool, yeah, thats good [12:57] sergiusens: hey, there's no schema support for per-part properties [12:57] sergiusens: would you like a patch for that? [12:57] sergiusens: so that a given part type can be validated [12:57] zyga, the only reason that isn't there is because there is an alternate thing in place for that (which I want to get rid of) plugins/*.yaml [12:58] sergiusens: hmm? [12:58] sergiusens: do you want to remove all the plugin/part types? [12:58] it uses a custom rule engine to support plugins/extensions [12:58] zyga, no, I don't want to get rid of that [12:58] sergiusens: ah, sorry, I misunderstood [12:58] sergiusens: where is that thing then? [12:58] sergiusens: in other words, when adding a new plugin type, where do I patch the schema? [12:58] zyga, in the path I just gave you ;-) [12:59] zyga, it's separate [12:59] ah, that was a path, /me looks [12:59] hmm I don't see plugins/extensions [13:02] sergiusens: did you paste anything I've missed? [13:03] zyga, http://bazaar.launchpad.net/~snappy-dev/snapcraft/core/files/head:/plugins/ [13:04] sergiusens: right but that's not a json schema [13:04] sergiusens: (I know about this part) [13:04] sergiusens: if that's all then it's fine, I'm just looking around to see what to patch [13:05] zyga, how would you support a 3rd party type with the schema? [13:05] sergiusens: well, you can put all of that in the master schema [13:05] sergiusens: assuming all plugins are in trunk [13:05] sergiusens: and none are installed from a 3rd party add-on [13:05] sergiusens: with add-ons you'd have to use some more nifty features but I think it's also possible [13:05] zyga, none that you know of ;-) [13:05] sergiusens: (I wrote json-schema-validator, which we're not using) [13:06] sergiusens: my json schema is rusty but not that rusty :) [13:06] sergiusens: so with all of them in trunk you can use specific property schemas [13:06] and conditional parts [13:06] zyga, right, we'd have to hack in local references; AFAIK, jsonschema only supports http [13:06] sergiusens: it's all doable with pure json schema [13:06] sergiusens: no, I don't think you need that [13:06] sergiusens: what I was thinking of is.... [13:08] sergiusens: https://github.com/zyga/json-schema-validator/blob/master/json_schema_validator/tests/test_validator.py#L733 [13:08] sergiusens: something like this [13:08] sergiusens: you can validate per-plugin type [13:08] sergiusens: using requires you can look at the outer object [13:08] sergiusens: this is a random code example, the key idea is requires [13:09] sergiusens: https://github.com/zyga/json-schema-validator/blob/master/schema-spec/draft-zyp-json-schema-03.txt#L564 [13:09] using the 03 draft wording [13:09] zyga, I know about requires [13:09] sergiusens: anyway, this is just an optional thing [13:09] zyga, but this would only work for 'built' in plugins [13:09] sergiusens: yes [13:10] sergiusens: if the design requires 3rd party plugins then it won't work this wy [13:10] way* [13:10] (without a way to let each plugin ship any object) [13:10] zyga, exactly, it supports 3rd party plugs [13:10] sergiusens: and having a custom sub-schema for it [13:10] sergiusens: ah, I see, thanks [13:10] zyga, so we'd need schema snippets and load the references befor sending t to the validator [13:11] sergiusens: you could just validate the generic schema and then sub-validate each plugin specifically [13:11] zyga, sure, if you want to extend the schema, go ahead, I don't mind, right now all parts are part of a pattern properties [13:13] Hey zyga, can we talk about pip and checkbox? [13:13] zyga: It seems that checkbox is using multiple requirements files? [13:14] tedg: yes, we sure can [13:14] tedg: checkbox is not using requirements files the way you think [13:14] zyga: I couldn't see a way they got consolidated. [13:14] tedg: everything important is specified by setup.py meta-data [13:14] Oh, okay. [13:14] * tedg stops thinking [13:14] tedg: if you get plainbox to work then checkbox is easier :) [13:14] pitti, oh ... [13:15] zyga: So how do I know the requirements? [13:15] tedg: if I can give you a bit of advice on requirement files: follow what travis-ci allows [13:15] tedg: it's spelled in setup.py [13:15] tedg: install_requires [13:15] tedg: the standard way [13:15] pitti, i just noticed that i didnt remove "allow-hotplug eth0" from my /e/n/i setup, could that confuse the generator ? [13:15] tedg: if you do what travis-ci allows then any python project will be snappy buildable [13:15] (does it check for that in any way ?) [13:15] tedg: most projects just setup.py develop / install their deps [13:15] ogra_: no, it doesn't look at that at all [13:16] ok [13:16] zyga: I find "the standard way" doesn't really exist anywhere :-) [13:16] tedg: and using wheels is the current hotness to make that easier to do [13:16] tedg: I disagree [13:16] tedg: setuptools is the way [13:16] tedg: requirements are for separate concerns [13:16] tedg: like deployment or other weird web scenarios [13:17] tedg: if you setup.py install is broken then users complain (if they care) [13:17] tedg: but it's definitely the standard way [13:18] zyga: Travis CI just allows you to run binaries. We're not interesting in providing a bunch of scripts... [13:18] zyga: http://docs.travis-ci.com/user/languages/python/ [13:18] tedg: yes but it supports certain things for free out of the box: like using setuptools without asking [13:19] zyga: So like what we do with the python3-project plugin? [13:19] tedg: yes, except right now that implementation is broken [13:20] tedg: (for the same reasons I said earlier) [13:20] zyga: So I think there's a patch for that, we just need to work out the details. [13:20] zyga: But, to be clear, you don't want or use a requirements.txt file. [13:21] tedg, if we accept the fix from BjornT I think zyga's problem would be a non issue [13:21] tedg: it'd be nice if we support it if people want to use it (for version pinning) [13:21] tedg: but it should be optional IMHO [13:21] sergiusens: I think so too, but I haven't pinged you about your brainstorming weekend yet :-) [13:21] tedg, hasn't happened :-/ [13:21] sergiusens: Start your weekend now! ;-) [13:22] tedg, good idea [13:22] sergiusens: btw, I failed with my .run() cleanup, it's utterly hard to clean up run to be space free and run without shell [13:22] sergiusens: I will rescue a few patches from that but the whole patchset has failed (too many changes, tests change, etc) [13:23] sergiusens: So let's take BjornT's patch for now and then we'll fix it later. I think it's more important to provide features today than provide smaller snaps. [13:23] Hate taking on technical debt, but adoption is more important. [13:23] tedg, certainly [13:24] how else would the duct tape producing industry survive ? [13:25] So then elopio if you could look at my pip branch and tell me how to format the long lines in python compatible way I think it's ready too. [13:25] ogra_: I believe there is a pact between the WD-40 industry and the duct tape industry. An evil, evil pact. [13:27] elopio, sergiusens: I put pip tests in the integration suite. Is that where we want them? They take longer because they download stuff. [13:27] tedg, lol [13:29] tedg: can I review it it as well? [13:29] zyga: sure! [13:30] tedg: ok, I'll check it out now [13:30] * zyga cries for clean, logical commits :| [13:33] tedg: there's no chance to squash those commits into separate changes? [13:34] zyga: Eh, plastic surgery makes you look more beautiful, but hides who you really are :-) [13:34] tedg: well, reading a merge commit to see why it's there really is bad for review [13:34] tedg: and reviewing the whole change in one commit is not so nice [13:36] tedg: can you have a more realistic requirements please? argparse is hardly good (bundled with 3), something that has deps of its own perhaps? [13:36] zyga: You're not going to get me excited about the difficulty of reading a 200 line diff. [13:37] zyga: We're not testing PIP here, just that PIP gets called. [13:37] tedg: that's not useful [13:38] tedg: the test doesn't check that what you do works in pratice IMHO [13:38] tedg: especially that it does work if you have argparse installed on the host [13:38] tedg: or if you have something it depends on installed is also present [13:38] rickspencer3: sent [13:38] zyga: ? We're not looking at the host installation. [13:39] thanks ppisati [13:39] zyga: We're testing that it gets downloaded and installed into the part. If it's in the host the test will fail. [13:39] tedg: does it work if you use this plugin to install plainbox? [13:40] tedg: (since plainbox is installed most of the time, that's an interesting test in itself) [13:41] rickspencer3: it's a bit long and hackish, but i tested every step on a clean snappy instalation, so it should just work [13:41] (last famous words...) [13:41] haha [13:41] zyga: I looked at plainbox and it seemed to have multiple requirements files, which it doesn't support. But you said that you don't use requirements files, which is what this branch supports. [13:41] tedg: hmm [13:41] zyga: So I think the answer is no. [13:41] tedg: but what about _no_ requirements [13:41] tedg: and just "get this thing from pypi" [13:42] tedg: which is akin to pip install $stuff [13:42] zyga: No, didn't add that. I think that people mostly do that in their setup.py. No? [13:43] tedg: hmm, but then they cannot use this part, right? [13:43] tedg: or am I missing something [13:43] tedg: use case: I want to pull in some tools into my snap, those are python tools I can pip install on my host [13:43] tedg: so I use python3 parts [13:43] tedg: and the part name is the pip thing to install [13:44] tedg: I can also use requirements (e.g. to handle optional elements) [13:44] tedg: the example pip-test in your MR could look like: [13:44] zyga: This isn't about having tools on your host, this is about dependencies for your snap. You won't be able to use these tools for anything other than building/running. [13:44] parts:\n [13:44] tedg: I _know_ that was an analogy [13:44] tedg: parts: argparse: type: python2 [13:45] tedg: e.g. if I want to have a script that uses some libraries I can just use the python3/python2 parts in my snapcraft to pull them from pypi [13:45] zyga: If you were only targetting snappy, that would make some sense? But it seems like if you wanted your build system to work anywhere else you'd use setup.py and requirements.txt. [13:46] tedg: (my analogy was about moving a familiar developer/user experience) [13:46] tedg: so if I wrote a script on ubuntu [13:46] tedg: and I need some stuff from pypi (that you can normally pip install) [13:46] tedg: then I want to copy that into my snap by using a python3/python2 part type and just listing the thing I want [13:46] tedg, yes, system interactions into integration_tests is fine [13:47] So your use case is someone custom modifying a system instead of someone building something that could be deployed in production and used by other people. [13:47] tedg: the requirement file that you specify is relative to snapcraft.yaml, right? [13:47] zyga: Yes, it would be in the project directory. [13:48] tedg: ok [13:48] tedg: looking at the MR, I'd like an explanation of what the pull method is doing there [13:49] tedg: the symlink and oll the other stuff seem a bit magic [13:49] tedg: can you explain what that is doing and why? [13:49] zyga: Yeah, it's a bug. Seems that you can't change the root and specify the deb layout. [13:49] tedg: mmmm, I'm sure you can, what did you try? [13:50] tedg: the deb layout just uses dist-packages [13:50] zyga: Specifying deb layout and the root directory :-) [13:50] tedg: I did just that :) [13:50] zyga: Just the two command lines, then it put everything in /usr [13:51] tedg: hmmm, I'll look at that [13:51] tedg: I understand your branch doesn't address separation of python from the host yet [13:51] tedg: and something else that sergiusens mentioned will work on that? [13:52] tedg: specifically that pip will not install things that are on the host, so plainbox for example would be broken because its dependencies might be on the host for most developers [13:53] tedg: (and we should just force it to install everything as if system python didn't exist and didn't have any packages) [13:53] zyga: This branch allows normal setuptools to install into the parts: https://code.launchpad.net/~bjornt/snapcraft/setuptools/+merge/270733 [13:53] * zyga looks [13:53] hmmmm [13:53] this is not enough [13:53] zyga: Provide a test, we will fix it :-) [13:53] tedg: sure [13:54] IRC based TDD ;-) [13:54] tedg: one thing is that this will break if you are in a virtualenv [13:54] Hmm, that uses prefix instead of root dir. [13:54] and many python developers are ;) [13:54] my testing showed that you need to avoid importing site [13:54] and you need to explicitly run the python from stage dir, not any python [13:55] this runs any python2 and any python3 [13:55] which is wrong [13:55] to avoid importing site you need to pass -S [13:55] then it will work [13:55] The PATH is configured to only run the version from the parts install. [13:55] so that's one test case :) [13:55] tedg: I found that not to be the case, perhaps another bug was at play? [13:55] That then adjusts the directories to use the correct site directory. [13:55] tedg: ok, I'll check again now [13:56] You might make sure to run my branch in that regard, there was a small bug with environments that it fixes. I don't think it broke that for python-project though. [13:56] Only broke in python itself. [14:03] mvo, http://paste.ubuntu.com/12408693/ any idea? [14:03] sergiusens: what's the schedule for stable API for plugins/parts? [14:03] mvo maybe the profile thing doesn't like symlinks [14:03] sergiusens: as in, what's the deadline for making changes to those parts [14:04] zyga, the spec is in place, it can't change; just needs implementation [14:04] sergiusens: so all the methods need to return True? === Abhishek__ is now known as Abhishek_ [14:05] zyga, no, that's an implementation detail; but in any case once we do release, which is sometime this week it can't change as plugin writing docs would need to change as well [14:06] zyga, I say if you don't like that, get it done today ;-) [14:06] sergiusens: that's what I wanted to hear ^_^ [14:06] sergiusens: I'll talk to you in a sec, stand-up time [14:06] sergiusens: is there a public link to the spec? [14:06] sergiusens: or is it the same private doc? [14:06] zyga, I thought the gdoc I gave you was public [14:07] sergiusens: ah, I assumed otherwise [14:07] sergiusens: ok :) [14:08] sergiusens: no, sorry, this is a 15.04 image? might be a sideload issue [14:08] mvo: did you think about the sideload issue I've mentioned last week? [14:09] mvo: (to store sideload flag not in $prefix) [14:10] zyga: the origin information (.sideload, .mvo) is causing us some grief but I have not thought it through yet [14:11] zyga, mvo if it is not in the form of $name.sideload it would be in the form of /apps/$origin/$name [14:11] that was the discussion in cape town, right? [14:12] mvo, nope, this was our invention in Austin [14:12] sergiusens: I don't understand how that addresses that C/C++ stuff that hardcodes --prefix cannot be sideloaded [14:12] sergiusens: maybe I'm missing stuff, sorry, split attention [14:14] mvo, btw, I can't symlink the config hook [14:14] sergiusens: its exploding :( ? [14:15] sergiusens: the aa_exec error? [14:15] zyga, I don't know what you are talking about either; all I'm saying is that the problem you present is not solvable by hardcoding paths [14:16] mvo, aa_exec, or the runner (or snappy) how was it again that the profile for config was created? [14:16] sergiusens: ah, sorry, I didn't understand that, so what is the approach to software that does hardcode stuff today? [14:16] zyga: You shouldn't hardcode prefixes, you really need to use the environment variables there. [14:16] sergiusens: well, preach upstream, the reality is different [14:16] sergiusens: aa-clickhook is doing that still - is that on rolling or 15.04? [14:16] tedg: ^^^ [14:16] You're gonna need to chroot things for those cases. [14:16] tedg: if I want to integrate an upstream project I don't want to have to patch it) [14:16] mvo, 15.04 [14:17] tedg: that's a bit unfortunate IMHO, why is that required? [14:17] We have talked about providing some amount of automatic chroot, but haven't gotten to it. [14:17] tedg: if the prefix was deterministic from version and snap name, that would be enough for all the C software in the world [14:17] zyga: Because we don't want to commit to even /apps [14:18] tedg: since we'll break everything on 16.04 I think it would be useful to have a prefix that works in the meantime, even if we change it later [14:18] tedg: maybe the chroot thing will fix it but it sounds like an overkill to me [14:18] What you're saying is if we backed ourselves into another hardcoded filesystem layout things would be easier. But the problem is hardcoded filesystem layouts, not how they're layed out. [14:18] tedg: and (again) today, it's not working [14:18] tedg: no, hardcoded filesystem layout are a reality for the past few decades [14:19] tedg: if we require software to be relocatable then we put extra roadblock [14:19] Yes, let's fix it! [14:19] tedg: that's silly, that's not our battle to fight [14:19] Heh, frankly, we chose to fight it over three years ago. [14:19] tedg: and especially, I don't see what we gain by requiring this apart from nebulous "not hardcoded" [14:20] tedg: if that means that we cannot easily integrate major upstream software than I just see that as a disadvantage [14:22] ogra_, i'm looking into https://code.launchpad.net/~elopio/snappy/resize_test/+merge/270892, should this work in 15.04? [14:22] zyga: An opinion that you're entitled to. I think the fix for those folks is to chroot it until they realize it's 1990 already. [14:23] tedg: I think that's a bold statement from the world where _everything_ written in C hardcodes prefix and we gain exactly nothing today by not supporting that [14:23] tedg: and targetting IOT space will intersect a lot of those projects [14:23] tedg: and last time I checked ubuntu is in the 1990s you claim [14:23] tedg: and will be in 16.04 [14:24] Yes, and we're fixing that! [14:24] tedg: no, we just look the other way [14:24] tedg: are we sending patches anywhere? [14:24] tedg: we're not fixing anything, we just say we don't support de-facto standards today [14:25] fgimenez, 15.04 and rolling should be identical now wrt that code ... [14:27] zyga: Reality is that most projects provide environment variables to change almost all of those prefixes. We've even patched xcb where we needed it. For instance, all of the QML stuff is relocatable with the qml plugin. Those base libraries are all written in C. [14:27] ogra_, ok thanks, i'm getting some errors in 15.04, i'll try to fix the install-test-initramfs script to make it work in both [14:27] (well C++, but for this argument that's not important) [14:28] tedg: and what environment variables those upstream projects we've patched are respected? those of snappy? [14:29] fgimenez, oh, note that i landed the final fixes only on the weekend, so make sure you have a recent image in use [14:29] zyga: Nope, their own, we then set them up for snappy. http://bazaar.launchpad.net/~snappy-dev/snapcraft/core/view/head:/snapcraft/plugins/qml.py#L78 [14:30] ogra_, currently 147, is that ok? [14:30] yep [14:30] that should work [14:31] i did a plethora of fresh installs with 146 and 147 on actual x86 HW on the weekend here [14:31] tedg: I still think this is totally counter productive, for all the low-level plumbing stuff that we don't ship anyone that needs to ship it has now to maintain a fork just to fix this non-issue (they gain nothing from this) or hardcode /apps/$name/$version and ignore side-loaded (or reverse, non-side-loaded) snaps: [14:32] tedg: I understand what you are sying but I think it doesn't work for the long tail of things and doesn't help us in adoption today [14:39] zyga: Sure, but there are very few opportunties that you get to make things better. We're trying to take one. [14:41] sergiusens: would you oppose a snapcraft part that can install extra build dependencies? [14:41] sergiusens: dynamically from the part itself [14:41] sergiusens: and if not, what other approach should one take to handle, e.g., building bluez [14:42] zyga, iirc, there's already a build-depends you can use with parts [14:43] sergiusens: with parts yes, but what about listing those in the snapcraft.yaml itself [14:44] sergiusens: not at the part type level, at a part _instance_ level [14:44] zyga, don't use the word part so generically, it is confusing ;-) [14:45] sergiusens: sorry, so, e.g. to build bluez [14:45] sergiusens: I'd like to say that building bluez requires all the packags on the host [14:49] tedg: bluez has 34 hardcoded instances of $prefix [14:49] tedg: in executables alone [14:52] tedg: a quick source search has 100 (exactly) matches for places that need to be patched [14:52] tedg: can it be done? yes, it is useful? not for anyone consuming snappy [14:53] s/consuming/targetting/ [14:56] zyga: Then use a chroot? [14:56] tedg: how? [14:57] zyga: Have the service start a script that does a chroot to the $SNAP_APP_PATH [14:57] tedg: will it be able to access the hardware this way? [14:57] tedg: e.g. /dev, /sys, /proc [14:57] zyga: You might have to map some directories into the chroot. [14:57] tedg: I assume netlink works [14:57] tedg: how? [14:57] tedg: do I mount --bind ? something? [14:58] tedg: and still integrated with outer dbus [14:58] tedg: if you add all those up, and it works, are we still making the world better? [14:58] zyga: Getting closer ;-) [14:58] tedg: (I'm using this as an example I work on right now, it's not made up on purpose to be evil) [14:59] zyga: The reality is that no one will change unless they have a reason. If they see the environment for bluez on Snappy, they'll not use prefix as much, and eventually change. [14:59] zyga: Change is a slow process. [14:59] tedg: I won't do the bind mount, or anything else, I'll hardcode /apps/$name/$version.sideload because I don't have the time to research the other solution that is identical if it works [15:00] tedg: well, I don't want /apps to change [15:00] tedg: I just want to fix side-loading [15:00] tedg: and I'm showing my case as an argument [15:00] zyga: For instance, for frameworks it may change to /framework [15:00] tedg: I'll gladly hardcode /framework and build a new snap, that's easy [15:01] tedg: still .sideload will be a thorn [15:01] You're fighting this. Just feel the force and blend with it. Lift the speeder out of the swamp. I know you can do it. [15:02] tedg: I'll accept patches that make bluez relocatable ;) [15:02] tedg: if you want to wade thorough 100 places that need this [15:02] tedg: I'm like the model consumer of snappy [15:02] tedg: they will just totally ignore this non feature [15:02] tedg: and say you have to sideload / not sideload [15:03] tedg: this is a non feature because all the value is for snappy developers and none for the snap developers that have to now do extra work [15:06] tedg: also this: https://github.com/ninjasphere/ninjasphere-snappy/blob/master/docs/TESTING.md#known-issues [15:06] tedg: third bullet point [15:06] zyga: You're welcome to argue and dislike it, but the reality is that we're not going to hardcode the paths anytime soon. [15:07] tedg: I'll argue otherwise at the sprint :-) [15:27] Is there a way to add udev rules in a snap? [15:29] mvo, Chipaca ideas? http://paste.ubuntu.com/12409199/ [15:29] tedg, wouldn't that break confinment pretty badly? [15:30] tedg, no, only snappy hw-assign or through the gadget or oem snap [15:30] sergiusens: sudo [15:30] Chipaca, doh [15:30] sergiusens: :) [15:31] sergiusens: if you readd/write from priv'ed locations, check your EUID [15:31] sergiusens: so you're more user gefriendly [15:36] beuno: Well, we need some way to add new devices. i.e. my new cool USB device to go along with my software package. [15:36] beuno: We can parse the file. We probably don't want to just copy the file somewhere. [15:37] * beuno nods [15:37] tedg, isnt that what hw-assign actually does ? [15:38] ogra_: Oh, does it? I didn't know how it worked. I figured it was apparmor magic. [15:39] i think it is both, apparmor and udev rule creation ... jdstrand would know :) [15:41] * jdstrand reads backscroll [15:44] tedg: is there a way to add udev rules to a snap> the oem snap is capable of doing this. I believe that is 'config'urable (/me points you to mvo for that) [15:44] jdstrand, tedg here wants to do it from an app [15:44] evil evil :) [15:44] because his app will use a device he built, lets say [15:44] sergiusens: Seems that galileo doesn't use a requirements.txt, just setup.py [15:45] so, the answer to that is 'no' [15:45] a gadget could describe that [15:45] burn! [15:45] an app shouldn't [15:45] well, lets think about it a bit [15:45] so what if you ship a special USB device, lets say, a sensor [15:45] I'm stating the current implementation [15:45] over USB [15:45] jdstrand: The specific use case was galileo who does Fitbit sync, and needs to add the USB IDs for the fitbit dongle. [15:45] I know it is being reworked [15:45] or tedg's real case [15:45] ok [15:46] good [15:46] I don't think we have anybody on that atm [15:46] I'm not sure who is reworking that [15:46] actively working on it [15:46] heh [15:46] :) [15:46] but wasnt hw-assign actually meant to solve this [15:46] as least in a first shot ? [15:47] * ogra_ doesnt see why it works for webcams or serial USB controllers but not for a fitbit gadget [15:47] the real thing is that you don't want something that is untrusted (ie, an app) interacting with the system in ways that it can break out of confinement without some sort of user/admin interaction [15:47] I'm not saying it is a today thing, but I do think we need some way to add USB IDs and Bluetooth, etc, etc. [15:47] ie, an admin saying-- ok, you can access this device (fine) [15:47] right [15:47] and that is what hw-assign works for today [15:47] so this is assign-and-describe [15:47] I think that's not a way that the app controls the system as much as saying "if this exists, I'd like to know about it." [15:48] we're missing the and-describe part, common for USB [15:48] ie, snappy install allows declaring udev but not installing them without the admin (fine, not implemented, just saying a very simple idea) [15:48] Where "if this exists" could include something the system didn't know about before. [15:49] a gadget saying, this is all the stuff on the system, here are udev rules (fine, where I thought we were going, but being reworked) [15:49] an oem snap saying, these are a bunch of udev rules and snaps that can use them without me doing anything (fine, cause the oem is special) [15:50] but arbitrarily extending udev itself, that hasn't been discussed afaik [15:50] I think that works for fixed devices that are in production. But it doesn't work for makers who are building something on a Raspberry Pi and want to add functionality. [15:50] no, I understand where you are coming from [15:50] again, just stating thoughts surrounding it [15:51] it sorta sounds to me like a separate gadget snap [15:51] K, anyway. I imagine that'll have to go on the backlog today. But it was something I just ran into. [15:51] but, idk-- others are (re)defining this [15:51] jdstrand, also, gadget and oem snap are the same thing, yes? [15:52] well, oem is something that is implemented today. I thought gadget was an evolution of the oem snap [15:52] but perhaps I am wrong on that point [15:52] I... don't know [15:52] :) [15:52] I'm sure mvo does, he knows everything! [15:52] I just hear occasionally that a gadget snap is being used in ways I didn't think an oem snap would be. sergiusens could comment further I'm sure [15:53] yes, or mvo [15:54] it might be possible with sufficient yaml/input validation/templated output to have an app snap declare udev rules for snappy to add to the system [15:55] "these are the product ids" and snappy says "oh, ok, I'll add some stuff to udev so the system knows about it" [15:55] mvo: the sideload framework issue is real [15:55] Chipaca: which issue? [15:56] * zyga is curious about that [15:56] I would suggest getting p itti in on that conversation though [15:56] Chipaca: hmmm [15:56] zyga: https://bugs.launchpad.net/snappy/+bug/1493706 [15:56] Launchpad bug 1493706 in Snappy "side-loading frameworks broken" [Critical,Triaged] [15:56] Chipaca: is it because of the version number change? [15:56] mvo: let's fix it :) [15:56] Chipaca: that would be ideal indeed, I was just lazy^Wbusy [15:57] mvo: I don't know where, but i must have forgotten to do the right thing with versions somewhere :) [15:57] what is the _multi suffix? fat snaps? [15:57] or just arbitrary string [15:57] mvo: because, yes, it's trying to use a version number it should not know about [15:57] mvo: however [15:57] mvo: let me check wiht a simpler framework first [15:57] mvo: it might be docker doing things wrong :) [15:59] mvo: it's related to servcies [15:59] mvo: "start" is also not doing the right thing, somehow [15:59] mvo: here, you test it: https://public.apps.ubuntu.com/anon/download/canonical/hello-dbus-fwk.canonical/hello-dbus-fwk.canonical_1.0.1_multi.snap [16:00] uhh [16:00] ok [16:02] mvo: so [16:02] mvo: something is still taking the version from the package.yaml [16:02] :( [16:02] mvo: directly [16:02] mvo: instead of through part.Version() [16:03] mvo: found it [16:03] maybe :) [16:06] mvo: i'll have a branch up in a bit [16:13] mvo: figured it out. It's down to origin being empty for frameworks, again [16:13] the sooner we nuke that the better :( [16:15] * Chipaca runs the tests [16:16] mvo: you've got unreachable code in xgettext-go, and my vet complains :) [16:17] Chipaca: what a crybaby [16:17] Chipaca: let me fix that, vet gets cleverer each way [16:17] each day [16:17] mvo: https://code.launchpad.net/~chipaca/snappy/always-take-version-from-filesystem/+merge/270988 [16:19] mvo: if i were clever i would've noticed it the first time :) [16:20] :) [16:20] * mvo hugs Chipaca [17:17] sergiusens: https://code.launchpad.net/~zyga/snapcraft/custom-plugin/+merge/270994 [17:18] sergiusens: early feedback please [17:18] sergiusens: I've used this to build a few interesting bits [17:18] sergiusens: most notably missing are build dependency handling [17:18] sergiusens: and environment sucks (correct me if I'm wrong, this cannot be done without fixing many other things now) [17:19] sergiusens: example snapcraft.yaml for it: http://paste.ubuntu.com/12410227/ [17:23] zyga, you can build a package that depends on snapcraft I guess, but I don't thing we want commands available to do this; the original snapcraft was like this fwiw [17:25] sergiusens: can you be more explicit, I don't understand what you are saying [17:26] sergiusens: that explicit commands are bad? [17:26] sergiusens: or that something else is bad? [17:26] zyga, explicit commands [17:26] zyga, this can be driven by a Makefile in most cases, can't it? [17:31] sergiusens: yeah, perhaps [17:31] sergiusens: is there something I can do about the environment part [17:31] sergiusens: the .replaces() thing is a bit terrible but I didn't find any better ways of doing it === slangase` is now known as slangasek === ahayzen_ is now known as ahayzen [17:55] hmm, I'm seeing a weird low-level error [17:55] aa_change_onexec failed with -1 [17:55] has anyone seen this, googling suggests its related to apparmor [17:58] well, do you see any apparmor messages in syslog = [17:58] ? [17:59] ogra_: checking [18:00] anyway, ask jdstrand or tyhicks [18:00] definitely apparmor related [18:00] zyga: you need the errno value set [18:00] zyga: bah, I didn't phrase that correctly.... you need to figure out what errno is set to when aa_change_onexec() returns -1 [18:00] tyhicks: I'm not sure which part applies this, it happens when I run one of the unconfined executables [18:00] yeah [18:00] I understand [18:00] trying to hek [18:00] * zyga would love gdb now [18:01] ogra_: nothing in syslog [18:01] * jdstrand is guessing the profile isn't loaded without knowing the errno [18:01] yes, that's most likely the problem ^ [18:01] zyga: what command are you running? [18:01] hostapd_cli [18:01] hostapd works _cli doesn't [18:02] hostapd_2.4-1_amd64.snap [18:02] on chinstrap in ~zyga/ [18:02] 333K [18:02] zyga: can you paste what is in /apps/bin/hostapd_cli [18:02] yep [18:02] tyhicks: I'll take a first look at this [18:03] (I'm doing this in kvm) [18:03] ok [18:03] http://pastebin.ubuntu.com/12410753/ [18:04] jdstrand, tyhicks: without the wrapper it works okay [18:04] zyga: can you paste the output of cat /var/lib/apparmor/profiles/click_hostapd.sideload_hostapd_cli_2.4-1 [18:04] cat: /var/lib/apparmor/profiles/click_hostapd.sideload_hostapd_cli_2.4-1: No such file or directory [18:04] zyga: also, please paste the output of: sudo aa-status|grep hostapd [18:04] so that nails it [18:05] hostapd.sideload_hostapd_2.4-1 [18:05] well, maybe [18:05] let me look at the snap [18:05] http://pastebin.ubuntu.com/12410779/ [18:05] look at this, there's one hfor hostapd though [18:08] zyga: http://paste.ubuntu.com/12410801/ [18:08] zyga: rename hostapd_cli to hostapd-cli [18:09] I'm quite surprised that snappy build worked let alone snappy install [18:09] seems there is an input validation issue there [18:09] jdstrand: so the external binary must be hostapd.hostapd-cli [18:09] jdstrand: and binaries with _ are not supported? [18:09] '_' is not allowed in the pkgname, service/binary name or version [18:09] jdstrand: ah, fun [18:09] jdstrand: heh :) [18:09] jdstrand: thanks! [18:10] np [18:10] sergiusens: ^^ nothing in snacpraft assemble complained [18:10] tyhicks: fyi-- the outcome ^ [18:11] zyga, care to patch the validation thingy? [18:11] zyga, the regex [18:11] sergiusens: it might be doable in the schema [18:12] sergiusens: or is there some other validation? [18:12] zyga, it should, I have a regex there; it may be wrong [18:12] and I suck at regex [18:12] is snapcraft written in go? [18:12] cause snappy I'm quite sure has the regex in there [18:12] sergiusens: hehe [18:12] sergiusens: no worries, it should be easy [18:13] zyga: but snappy install didn't error? [18:13] that is another bug [18:13] (if it didn't) [18:14] zyga, it is missing the regex for binary name and service name http://bazaar.launchpad.net/~snappy-dev/snapcraft/core/view/head:/schema/snapcraft.yaml#L46 [18:14] a bug or MP would be great [18:15] jdstrand: there is a bug opened for that (I I think I debugged this same problem for someone once while you were on vacation) [18:15] let me look [18:17] sergiusens: fyi, snappy/click.go is wrong: const servicesBinariesStringsWhitelist = `^[A-Za-z0-9/. _#:-]*$` [18:17] sergiusens: '_' should not be in that list [18:17] jdstrand, right [18:18] sergiusens: I'll do both in a sec, some IRL activities now [18:18] neither should '#' [18:18] zyga, I'll do it, jdstrand just gave me the regex ;-) [18:18] jdstrand: nope, snappy install (remote) didn't catch it [18:18] * jdstrand looks at click-apparmor regex [18:18] sergiusens: ehe, okay :) [18:23] I think there's another bug [18:23] I'm trying to use exec: hostapd_cli and name: hostapd-cli [18:23] and it's ignored [18:24] it derives the name from exec all the time [18:24] (for the wrapper) [18:24] zyga: exec should point to the file on disk relative to the topdir of the unpacked source [18:24] ah, then yes, that is a bug [18:24] jdstrand: name with - is ignored? [18:25] eh :) [18:25] * zyga digs deeper [18:25] I'm saying if snapcraft is handling the '_' to '-' for exec vs name, that is an issue [18:25] isn't* [18:26] yeah _wrap_binaries is broken [18:26] jdstrand: it's more than that [18:26] jdstrand: it's not handling it, it's not checkign it and if you try to tell it explicitly to rename it, it ignores it [18:26] I don't doubt there is more than that [18:27] jdstrand: so what exactly cannot have _ in it? [18:27] jdstrand: the wrapper script, the target executable or both? [18:28] 'name' [18:28] jdstrand: ok [18:28] in either the service or binary yaml [18:29] jdstrand: so when I have name 'hostapd-cli' and the generated executable wrapper is hostapd_cli.wrapper is that good or bad? [18:29] it seems bad [18:30] hmm, but it works [18:30] weird [18:30] zyga: it probably depends. in and of itself, it is neither-- 'name' is specifically there to reference the binary by another name [18:31] exec is optional [18:31] sergiusens: http://paste.ubuntu.com/12410981/ [18:31] so if you nave: name: bin/foo, then you get /apps/bin/pkgname.foo [18:31] sergiusens: this is the patch that "fixes" it for me [18:31] sergiusens: but it's weird [18:31] sergiusens: and the if makes no sense anymroe [18:32] but if you use exec, you can do this: [18:32] name: foo [18:32] jdstrand: no, because then name is totally ignored [18:32] exec: bin/foo.wrapper [18:32] and then you get /apps/bin/pkgname.foo [18:32] jdstrand: I just tried that, but I didn't want to provide my bin/foo.wrapper [18:32] I'm saying how it is supposed to work [18:32] with snappy itself [18:32] mmm [18:32] yeah, I understand [18:32] I don't know what snapcraft is trying to do [18:34] eh, ignore me sergiusens [18:34] I'm tired [18:35] sergiusens: so the other bug is that snappy install didn't complain when the profile wasn't generated. your regex change will sideste that and I think any other fixes can wait until snappy absorbs click-apparmor [18:35] sidestep* [18:36] $ sudo aa-clickhook -f [18:36] ERROR: Could not parse click manifest. Skipping 'hostapd.sideload_hostapd_cli_2.4-1.json' [18:36] * jdstrand notes 'hostapd_cli' with the '_' [18:37] * Chipaca puts on his chef hat and goes shopping [18:38] sergiusens: ok, so click-apparmor itself isn't terribly picky except for: [18:38] out = n.split("_") [18:38] if len(out) != 3: [18:38] raise... [18:38] so getting rid of the '_' alone should be ok [18:38] sorry, I got sidetracked with another irc conversation [18:39] yes, that split() was the problem that I debugged, as well [18:39] this all gets back to the application id [18:39] https://wiki.ubuntu.com/AppStore/Interfaces/ApplicationId [18:40] hmm.. I found the old debugging session but it doesn't look like a bug was ever filed :/ [18:40] which is something that is pretty deeply ingrained in the system. we could support '_' in the service/binary, pkgname, version if we escaped it. but that would have to be done quite carefully [18:41] tyhicks: well, so, the system is implemented very specifically. snappy isn't handling that right [18:41] tyhicks, jdstrand there https://bugs.launchpad.net/snappy/+bug/1495662 [18:41] Launchpad bug 1495662 in Snappy trunk "Services and binaries allow _ #" [High,New] [18:41] https://bugs.launchpad.net/snappy/+bug/1470265 [18:41] Launchpad bug 1470265 in Snappy trunk "Binary with an underscore fails to produce an apparmor profile" [High,Confirmed] [18:42] sergiusens: note, https://wiki.ubuntu.com/AppStore/Interfaces/ApplicationId has regexes [18:42] there are two bugs [18:43] sergiusens: those regexes are in the context of touch of course [18:43] jdstrand, should I keep #? [18:43] sergiusens: can we resume daily builds on https://code.launchpad.net/~snappy-dev/+recipe/snapcraft-daily [18:43] sergiusens: some people are bloked by this [18:43] zyga, in a bit [18:43] sergiusens: I was surprised to see it. it won't break apparmor [18:43] at least, it shouldn't :) [18:43] sergiusens: what do you mean? [18:43] sergiusens: so, your call [18:44] jdstrand, the tools block it, right? [18:44] jdstrand, if so I will, as it should be safe (store side won't have uninstallable packages) [18:44] click-review-tools that is [18:45] that's true. I'm quite sure they will [18:45] give me a sec [18:45] btw, why didn't the tools run for zyga? [18:49] jdstrand, not sure [18:51] - lint:hooks_valid:hostapd#cli [18:51] malformed application name: 'hostapd#cli' [18:51] so, yes, the review tools will complain [18:53] $ sudo aa-clickhook [18:53] ERROR: Could not generate AppArmor profile for 'hostapd.sideload_hostapd#cli_2.4-1.json'. Skipping [18:53] sergiusens: please remove '#' [18:53] jdstrand, got it [18:53] I bet that regex is coming from easyprof [18:54] yes [18:54] if re.search(r'^[a-zA-Z0-9][a-zA-Z0-9_\+\-\.:~]+$', s): [18:54] return True [18:54] return False [18:54] jdstrand, the regex in click.go allows spaces too :-/ [18:54] sergiusens: we should remove that too [18:55] not that the regex in easyprof isn't caring about APP_ID [18:55] click-apparmor cares about APP_ID [18:55] (which is correct) [18:56] (ie, easyprof and click-apparmor are at different layers and their respective checks are valid for their respective layers) [18:56] sergiusens: alright, rather than going through all this, how about you fix it and ask me to review? [18:58] sergiusens: fyi, easyprof is as strict as it is to ease quoting and avoid shell metachars [18:59] apparmor itself can have much more in the profile name. if we want to talk about letting other stuff in later, then we can have a discussion about that [19:00] '_' will always have to be handled carefully [19:00] (cause of APP_ID) [19:00] anyhoo [19:00] * jdstrand -> writing docs [19:49] sergiusens: so to be able to use stage-packages in my plugin I need to add it to /plugin/foo.yaml. Do we want that? Or do we want to include the global stuff automatically? [19:55] tedg, you missed my collisions branch [19:55] sergiusens: Sorry, never ran into it. [19:56] tedg, https://code.launchpad.net/~sergiusens/snapcraft/collisions/+merge/270942 :-) [19:56] * tedg rimshot [19:56] tedg, more like a sad trombone ;-) [19:56] LOL, oh, ouch! :-) [19:58] tedg, I also have this which I just noticed I need to wrap into the wrap exe stuff https://code.launchpad.net/~sergiusens/snapcraft/config/+merge/270989 [20:00] sergiusens: Is there some way we can share schema with snappy? [20:00] sergiusens: Seems like we should be "all they have plus" [20:00] tedg, once we add code to support schemas inside snappy we can [20:01] Ah, okay. Just want to ensure we don't have more keys like this. [20:01] Should be able to add them in one place. [20:06] This setup.py is now using 2.5GB of RAM. Not sure when I should call it. [20:08] Oh my, it wasn't hung. [21:04] tedg, what are you building? [21:04] sergiusens: nova [21:04] But it seems to be hardcoding the path to the python interpreter. [21:05] Which surprised me. [21:06] I'll have to play with it more later though. Need to go play the trophy husband for a bit. [22:55] hi, I have built a snap package that is failing automated reviews, but some of the checks contradict the docs (eg, snappy-systemd_package_yaml_ports_lp_format) [22:56] anyone around to help me understand how to improve the snap package?