/srv/irclogs.ubuntu.com/2015/09/14/#snappy.txt

fgimenezgood morning07:07
dholbachgood morning07:31
=== ezobn1 is now known as ezobn
=== Chipaca` is now known as Chipaca
* ogra_ tickles seb12808:16
ogra_seems someone found my G+ post about personal ...08:16
ogra_https://lists.ubuntu.com/archives/snappy-devel/2015-September/001033.html08:16
ogra_https://lists.ubuntu.com/archives/snappy-devel/2015-September/001036.html08:16
ogra_seb128, i assume nobody is interested in getting bugs yet for this ?08:17
seb128hey ogra_08:17
seb128not that I know of, no08:17
JamesTaitGood morning all; happy Monday, and happy Eat a Hoagie Day! 😃08:31
zyga-phoneGood morning08:39
ChipacaJamesTait: what's a hoagie?08:46
mvoChipaca: I reviewed your branches, I guess next step is to get them onto the 15.04 image or is something pending from your side08:46
JamesTaitChipaca, https://en.wikipedia.org/wiki/Submarine_sandwich#Hoagie08:47
Chipacamvo: we need to decide what to do about activation in 15.0408:48
* Chipaca hugs JamesTait for not sending him to lmgtfy08:48
Chipacamvo: also maybe we don't need to backport to 15.04 if we don't want to ship these there and instead ship trunk to 15.04?08:49
JamesTaitChipaca, nah, enough people have asked that I realise not everyone watches Man vs Food. 😉08:49
Chipacamvo: next step for _me_ is thinking a little bit about services, and implementing the fruit of my thinkage08:49
ChipacaJamesTait: that looks like a sandwich you can't eat politely :)08:50
JamesTaitOh, definitely.08:51
mvoChipaca: re trunk> https://bugs.launchpad.net/snappy/+bug/1493706 worried me a bit about that (I still think its a good idea in general, but this worried me about regressions)08:51
ubottuLaunchpad bug 1493706 in Snappy "side-loading frameworks broken" [Critical,Triaged]08:51
mvoChipaca: activation? what has changed here (pardon my ignorance)08:51
Chipacamvo: oh, maybe you didn't notice? the coreos activation doesn't build on 15.0408:52
Chipacamvo: it uses os.Unsetenv, which wasn't it go 1.308:52
mvoChipaca: oh, this activation, sorry, was confused08:52
Chipacamvo: stgraber's activation patch/fork fixes that08:52
Chipacamvo: so we need to decide whether to ship his fork, or his patch :)08:52
Chipacaor whether we give up and vendor08:52
mvoyeah, let me deal with that then08:53
Chipacamvo: and that's one we need to fix independently of backporting or not08:53
mvoyep08:53
ogra_mvo, the snappy config output if you just call it vs if you change a value look very different now ... if you chnage a value it prints all of the ppp values afterwards, if you just call the config command without setting anything the ppp values are omitted08:54
ogra_setting a value: http://paste.ubuntu.com/12395367/ vs calling snappy config http://paste.ubuntu.com/12395393/08:55
mvoogra_: hm, that looks like a bug :/08:55
Chipacamvo: you need to stop apologizing to me for the state of the codebase, already; i've been here long enough :)08:55
ogra_mvo, note i also seeded libnss-myhostname on the weekend for bug 1495058 and bug 146439608:57
mvoChipaca: sorry08:57
ubottubug 1495058 in Snappy "updating the hostname via snappy config does not update /etc/hosts" [Undecided,New] https://launchpad.net/bugs/149505808:57
mvoChipaca: :P08:57
ubottubug 1464396 in Snappy ""sudo: unable to resolve host ..." when custom hostname is used." [Critical,Triaged] https://launchpad.net/bugs/146439608:57
mvoChipaca: ok, I will try to stop08:57
Chipacamvo: :D08:57
ogra_(just as a FYI)08:57
mvoChipaca: do you have a link to the build failure btw?08:57
mvoChipaca: because the version of golang-systemd in the archive does not use os.Unsetenv so I'm a bit puzzled08:58
Chipacamvo: https://code.launchpad.net/~chipaca/snappy/serve/+merge/27037208:58
mvota08:58
mvoChipaca: heh, its just our CI that is confused08:58
mvoChipaca: I commented in the MP09:07
=== Chipaca is now known as Chipakeitor
=== Chipakeitor is now known as __chip__
=== __chip__ is now known as Chipaca
Chipacamvo: so with that change to dependencies it'll be happy again?09:13
mvoogra_: hm, I just upgraded to ubuntu-core 147 and I do see the ppp config stuff, anything unusual aobut your image?09:13
mvoChipaca: I think so, I tested locally and it looked happy09:13
mvo(ie. checked out the right revno and there was no os.Unsetenv() anywhere)09:14
Chipacamvo: your diff failed here, though09:14
mvo:(09:14
mvoI will redo it with a proper branch instead of a inline diff09:14
mvoit needs tabs09:14
Chipacanot sure why09:14
Chipacaah09:14
mvomaybe LP ate them09:14
Chipacamvo: nah, i can patch by hand09:15
Chipacalet's see if this works now :)09:16
ogra_mvo, nope, nothing unusual09:16
ogra_that was a plain install of 145 and i saw/see the same on 146 and 14709:17
mvoogra_: ok, I will try harder to reproduce09:18
ogra_hmm, the machine doesnt boot anymore after auto upgrade :/09:19
ogra_and indeed i have no serial board for that HW :/09:19
ogra_well, or it doesnt bring up the static network setup09:20
Chipacaserve landed \o/09:43
sergiusens\o/09:43
Chipacasergiusens: mo'in!09:44
sergiusensChipaca, serve me a beer09:44
sergiusens:-P09:44
sergiusensChipaca, good morning09:44
Chipaca$ GET http://localhost:8080//1.0/packages/beer09:44
Chipaca{"metadata":null,"status":"Not Found","status_code":404,"type":"error"}09:44
sergiusensChipaca, bummer :-)09:45
Chipacasergiusens: easy enough to fix though09:45
sergiusensChipaca, yeah, extend and serve myself :-P09:54
ogra_hmm, so after autopilot upgrade it seems my static network doesnt come up anymore10:06
ogra_mvo, does snappy config apply the /etc/network/interfaces.d file on every boot ?10:08
ogra_Sep 14 09:18:00 aleph2 systemd[1]: ifup-wait-all-auto.service start operation timed out. Terminating.10:09
ogra_Sep 14 09:18:00 aleph2 systemd[1]: Failed to start Wait for all "auto" /etc/network/interfaces to be up for network-online.target.10:09
ogra_Sep 14 09:18:00 aleph2 systemd[1]: Unit ifup-wait-all-auto.service entered failed state.10:09
* ogra_ sees that in syslog of the failed machine 10:09
ogra_oha !10:10
ogra_Sep 14 09:16:01 aleph2 kernel: [   43.590490] r8169 0000:01:00.0 eth3: renamed from eth010:10
ogra_thats bad10:10
Chipacaogra_: what happened to stable network interface names?10:14
Chipacashouldn't that be ens3 or sth10:14
ogra_15.0410:14
ogra_should it there ?10:14
ChipacaI don't know10:14
Chipacamy brain no longer supports 15.0410:14
* ogra_ is running 15.04 edge here 10:14
Chipaca:)10:14
ogra_it should definitely never rename it10:15
ogra_no matter oif the name is foople0 or eth0 :)10:15
ogra_wow and in /writable/system-data/etc/udev/rules.d i have the persistent-net.rules file with two different entries for the same MAC10:17
ogra_thats gross10:18
ogra_mvo, bug 149545210:26
ubottubug 1495452 in Snappy "auto upgrade unintentionally updates /etc/udev/rules.d/70-persistent-net.rules" [Undecided,New] https://launchpad.net/bugs/149545210:26
ogra_pitti, is there a way to prevent this ? ^^^10:27
ogra_(why would it add a device at all if there is a MAC existing for it in that file already)10:28
pittiogra_: that sounds like https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=76557710:28
ubottuDebian bug 765577 in udev-udeb "netboot install writes duplicates to 70-persistent-net.rules" [Serious,Fixed]10:28
pittiogra_: is that vivid?10:29
ogra_pitti, yeah, the 15.04 edge image that we want to release this week10:29
pitti(as the entire 70-persistent generator is completely gone from wily)10:29
ogra_not sure we could drop the generator since 15.04 still seems to use the old way of naming devices10:30
pittiogra_: could you clean up the file, apply http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/diff/?id=f9cb380a, and see if that helps:10:31
pitti?10:31
pittiif so, we can SRU that10:31
ogra_ok10:31
sergiusensChipaca, mvo can I get an ack on https://code.launchpad.net/~sergiusens/snapcraft/collisions/+merge/270942 ?10:31
ogra_well, i'm not really sure how to trigger it again now :(10:31
ogra_to verify it wouldnt do that again10:32
ogra_removing the line manually brings it up normally again ... it doesnt add it anew10:33
pittiogra_: nevermind, vivid already does have that thing10:33
ogra_well, then it doesnt help obviously :)10:33
ogra_(teh immage build is actually from yesterday so everything in -security and -updates should be in it10:34
pittiogra_: but what's weird is that the file header exists multiple times too10:34
ogra_)10:34
pittiit's actually not wrong -- there are three MACs with different names, and the rules are just duplicate, not contradicting AFAICS10:34
pittioh, :f8 is eth0 and eth310:35
ogra_well, except that the second entry for eth0 is named eth310:35
ogra_right :)10:35
ogra_and i also got: Sep 14 09:16:01 aleph2 kernel: [   43.590490] r8169 0000:01:00.0 eth3: renamed from eth010:36
ogra_on the first boot after the auto-upgrade10:36
mvosergiusens: I have a look now10:37
sergiusensmvo, thank you very much10:37
pitti……………………[ -e "$RULES_FILE" -o -e "$tmp_rules_file" ] || PRINT_HEADER=110:38
pittiogra_: ^ hm, it shouldn't print the header again if the file already exists10:38
pittiogra_: could it be that /etc/udev/rules.d/ (or the actual file) is a bind mounted dir/file, which isn't bound in initramfs already?10:38
pittii. e. the generator would race with bind-mounting the file?10:39
ogra_pitti, yeah, could be, i have to check the inird script10:39
ogra_pitti, but would the generator run from initrd ?10:42
ogra_the bind mount is definitely done via fstab processing10:43
pittiogra_: AFAICS 70-persistent-net.rules gets copied into initramfs, but not the 75-persistent-net-generator.rules -- that shold only happen in the "real" fs10:44
ogra_right and we cant generate the initrd on the device so the file should be either empty or not exist at all10:45
pittiogra_: "lsinitramfs -l /initrd.img |grep net-gen" to double-check?10:45
pittiright10:45
pittibut if you fstab-mount it, you have a race10:45
ogra_GEEZ !10:45
ogra_where does lsinitramfs come from and why dont i know about it !!!10:45
ogra_gzip: /boot/initrd.img-3.19.0-28-generic: not in gzip forma10:47
ogra_t10:47
ogra_bah10:47
ogra_(amd64)ogra@aleph2:~$ lzcat /boot/initrd.img-3.19.0-28-generic |cpio --extract --quiet --list|grep net-gen10:49
ogra_(amd64)ogra@aleph2:~$10:49
ogra_nope, not there10:49
ogra_and the 70-persistent-net.rules file would also just vanish after the bind mount anyway10:50
ogra_it wouldnt be mangled10:50
ogra_pitti, this is clearly triggered only by an upgrade of the readonyl image part ... so i wonder if anything in there triggers it10:52
* ogra_ tries a snappy rollback and snappy update loop again ... 10:53
ogra_rollback comes up fine again10:55
* ogra_ updates and reboots again 10:56
ogra_upgraded fine, didnt change the rules file10:58
* ogra_ rolls back again and waits tile the autopilot kicks off the upgrade ... probably that causes a race10:59
mvoogra_: the ppp issue you see is on arm? or in a vm?10:59
ogra_mvo, on x86 hardware10:59
mvosergiusens: the config branch as well?10:59
mvoogra_: real hw? interessting, I poke some more10:59
sergiusensmvo, config branch is not done yet :-) the previous ones if you want you can review :-)10:59
ogra_mvo, OOOH !11:00
sergiusensmvo, I ran into collision problems when working on config (more complex and multiple parts al including ubuntu packages)11:00
ogra_mvo, sudo vs no sudo makes the difference11:00
mvoogra_: oh, of course :/11:00
ogra_(amd64)ogra@aleph2:~$ snappy config ubuntu-core |grep timezone11:02
ogra_    timezone: Europe/Berlin11:02
ogra_(amd64)ogra@aleph2:~$ date11:02
ogra_Mon Sep 14 11:01:59 UTC 201511:02
sergiusensogra_, for snappy config? I thought we had 'am I root' detection there11:02
ogra_bah, that doesnt look right either11:02
mvosergiusens: yeah, I started but then got distracted11:02
sergiusensmvo ogra_, we also need input hand holding in our config for ubuntu-core11:03
sergiusensspecially with ogra who typos a lot ;-)11:03
ogra_haha11:03
ogra_well, for configuring i used http://paste.ubuntu.com/12396598/ for this particular install11:03
ogra_if you see any typos, please tell me :)11:04
ogra_that the timezone doesnt get set is worrying :/11:05
sergiusensogra_, I don't see anything wrong, maybe check if /usr/share/zoneinfo/Europe/Berlin exists on the system?11:08
sergiusensogra_, it works fine for me for America/Cordoba11:08
ogra_it exists11:08
ogra_and /etc/writable/timezone is also set correctly11:09
ogra_sergiusens, is that on 15.04 ?11:09
ogra_(where America/Cordoba works)11:09
ogra_ah, now autopilot rebooted it ... /me waits if the interface gets renamed this time11:10
ogra_bah, came up fine11:11
sergiusensogra_, seems it doesn't work anymore http://paste.ubuntu.com/12407670/11:12
ogra_yeah, bad11:12
ogra_but i wonder why ... the files are all correct11:13
sergiusensogra_, /etc/timezone has the correct data11:13
ogra_right, here as well11:13
sergiusensogra_, did we drop some dependency?11:13
sergiusensogra_, I can be blind and just blame systemd :-P11:13
ogra_not that i'm aware of11:13
ogra_aha11:14
ogra_/etc/localtime points to UTC11:14
ogra_something didnt update it along11:15
ogra_due to the snappy config changes ?11:15
sergiusensogra_, no, this is older11:15
sergiusensogra_, snappy config only ever updated /etc/timezone11:15
ogra_and that worked ?11:17
sergiusensogra_, yeah11:17
ogra_(just asking because the hostname update only worked half)11:17
sergiusensogra_, hostname update has a bug logged somewhere; want to fix? shouldn't be complicated ;-)11:18
ogra_sergiusens, fixed it on the weekend :)11:18
* sergiusens finds ogra_ fixes things fast to scratch his itches ;-)11:18
zygahey guys11:19
ogra_so lets see what happens if i properly update /etc/localtime11:19
ogra_(amd64)ogra@aleph2:~$ sudo cp /usr/share/zoneinfo/Europe/Berlin /etc/localtime11:20
ogra_[sudo] password for ogra:11:20
ogra_(amd64)ogra@aleph2:~$ date11:20
ogra_Mon Sep 14 13:20:06 CEST 201511:20
ogra_sergiusens, i cant really belive that ever worked correctly ... (i rather guess nobody wimply noticed)11:20
ogra_*simply11:20
ogra_these files always need to be updated alongside11:21
sergiusensmvo, do you know if meta/hooks/config can be a symlink?11:24
ogra_sergiusens, bug 149547411:28
ubottubug 1495474 in Snappy "setting the timezone via snappy config does not actually update the time zone" [Undecided,New] https://launchpad.net/bugs/149547411:28
mvosergiusens: I think it can and its probably a good idea too, if it does not work there might be a bug11:31
sergiusensmvo, great, because I'm working on having snapcraft just symlink ;-)11:32
ogra_pitti, ouch ...11:33
ogra_(amd64)ogra@aleph2:~$ mount|grep -c etc11:33
ogra_1811:33
ogra_pitti, dont you think we could simply make the generator run later ... mounting everyting of /etc in initrd sounds wrong11:33
rickspencer3ppisati, hey, sorry, I lost track, when do you think I'll be able to use pwm on my rpi2?12:09
ppisatirickspencer3: latest kernel has it working (both 3.19 and 4.2)12:11
ppisatirickspencer3: if you are willing to do some manual hacking, you can test it right now12:11
rickspencer3ppisati, can I just install rolling?12:11
rickspencer3ppisati, actually, let me ask differently ...12:11
rickspencer3could you send detailed instructions to the snappy mailing list on how to test it?12:12
ppisatirickspencer3: ok, i'll do12:12
rickspencer3thanks ppisati12:13
rickspencer3I'm looking forward12:13
pittiogra_: well, you'll just keep running into races like this12:21
ogra_pitti, hmm, yeah :(12:22
pittithis is like the 10th time this issue bubbles up12:22
pittiogra_: you could restart udev once again after mounting everything, but you'd need to do "start udev", "stop udev", "mount everything in /etc", "restart udev"12:23
ogra_well, i guess that needs some discussion first ... i can add a function that parses out all the /etc dirs and mounts them indeed12:23
ogra_do we actually restart it at all ? i thought we carry the running daemon  binary over from the initrd12:23
pittino, we restart it12:24
ogra_rickspencer3, you will need a device tarball ... i'll roll one durign the week anyway for the 15.04.3 image that is planned this week12:24
pittiotherwise we'd keep the initramfs-tools' libc6 (and all libs) in RAM12:24
pittialso, the i-t udev has only a minimal set of rules12:24
ogra_rickspencer3, we dont have automated builds for RPi yet, so there is no specific tarball for the rolling images12:24
rickspencer3ogra_, ok, noted12:25
rickspencer3I think if ppisati emails directions to the list, that is the best way forward12:25
ogra_pitti, yeah, i always thought we onyl re-process the rules and keep the daemon running ... but indeed that would keep libc and friends in ram too12:25
rickspencer3I have all these servos and motors and rgb leds that are hungry for some pwm12:25
ogra_rickspencer3, not if he doesnt use the exact same setup the device tarball will use12:25
rickspencer3ogra_, um, then he should do that?12:26
rickspencer3I don't mind updating the image later if necessary12:26
rickspencer3I would just like to be unblocked12:26
ogra_well, why not just wait a day til i get to it and can offer the ready made tarball for testing12:26
rickspencer3ogra_, if that's what ppisati's directions say to do, that is fine with me12:27
* ogra_ is just currently busy with x86 since we have to have a specific release ready for this for 15.04.312:27
ogra_(and i run into bugs over and over with nearly everything atm :/ )12:28
ppisatiogra_: rickspencer3 well, my directions would be how to manually install a kernel, update the dtb, etc12:30
ppisatithat's how i test new kernels12:30
ogra_just cp'ing over the binary ?12:32
ogra_yeah, that would work for testing indeed12:32
ppisatiogra_: yep, literally12:33
ppisatiand the dtb too12:33
ppisatiand the modification to config.txt12:33
ppisatii'm writing it now12:33
ogra_cool, yeah, thats good12:34
zygasergiusens: hey, there's no schema support for per-part properties12:57
zygasergiusens: would you like a patch for that?12:57
zygasergiusens: so that a given part type can be validated12:57
sergiusenszyga, the only reason that isn't there is because there is an alternate thing in place for that (which I want to get rid of) plugins/*.yaml12:57
zygasergiusens: hmm?12:58
zygasergiusens: do you want to remove all the plugin/part types?12:58
sergiusensit uses a custom rule engine to support plugins/extensions12:58
sergiusenszyga, no, I don't want to get rid of that12:58
zygasergiusens: ah, sorry, I misunderstood12:58
zygasergiusens: where is that thing then?12:58
zygasergiusens: in other words, when adding a new plugin type, where do I patch the schema?12:58
sergiusenszyga, in the path I just gave you ;-)12:58
sergiusenszyga, it's separate12:59
zygaah, that was a path, /me looks12:59
zygahmm I don't see plugins/extensions12:59
zygasergiusens: did you paste anything I've missed?13:02
sergiusenszyga, http://bazaar.launchpad.net/~snappy-dev/snapcraft/core/files/head:/plugins/13:03
zygasergiusens: right but that's not a json schema13:04
zygasergiusens: (I know about this part)13:04
zygasergiusens: if that's all then it's fine, I'm just looking around to see what to patch13:04
sergiusenszyga, how would you support a 3rd party type with the schema?13:05
zygasergiusens: well, you can put all of that in the master schema13:05
zygasergiusens: assuming all plugins are in trunk13:05
zygasergiusens: and none are installed from a 3rd party add-on13:05
zygasergiusens: with add-ons you'd have to use some more nifty features but I think it's also possible13:05
sergiusenszyga, none that you know of ;-)13:05
zygasergiusens: (I wrote json-schema-validator, which we're not using)13:05
zygasergiusens: my json schema is rusty but not that rusty :)13:06
zygasergiusens: so with all of them in trunk you can use specific property schemas13:06
zygaand conditional parts13:06
sergiusenszyga, right, we'd have to hack in local references; AFAIK, jsonschema only supports http13:06
zygasergiusens: it's all doable with pure json schema13:06
zygasergiusens: no, I don't think you need that13:06
zygasergiusens: what I was thinking of is....13:06
zygasergiusens: https://github.com/zyga/json-schema-validator/blob/master/json_schema_validator/tests/test_validator.py#L73313:08
zygasergiusens: something like this13:08
zygasergiusens: you can validate per-plugin type13:08
zygasergiusens: using requires  you can look at the outer object13:08
zygasergiusens: this is a random code example, the key idea is requires13:08
zygasergiusens: https://github.com/zyga/json-schema-validator/blob/master/schema-spec/draft-zyp-json-schema-03.txt#L56413:09
zygausing the 03 draft wording13:09
sergiusenszyga, I know about requires13:09
zygasergiusens: anyway, this is just an optional thing13:09
sergiusenszyga, but this would only work for 'built' in plugins13:09
zygasergiusens: yes13:09
zygasergiusens: if the design requires 3rd party plugins then it won't work this wy13:10
zygaway*13:10
zyga(without a way to let each plugin ship any object)13:10
sergiusenszyga, exactly, it supports 3rd party plugs13:10
zygasergiusens: and having a custom sub-schema for it13:10
zygasergiusens: ah, I see, thanks13:10
sergiusenszyga, so we'd need schema snippets and load the references befor sending t to the validator13:10
zygasergiusens: you could just validate the generic schema and then sub-validate each plugin specifically13:11
sergiusenszyga, sure, if you want to extend the schema, go ahead, I don't mind, right now all parts are part of a pattern properties13:11
tedgHey zyga, can we talk about pip and checkbox?13:13
tedgzyga: It seems that checkbox is using multiple requirements files?13:13
zygatedg: yes, we sure can13:14
zygatedg: checkbox is not using requirements files the way you think13:14
tedgzyga: I couldn't see a way they got consolidated.13:14
zygatedg: everything important is specified by setup.py meta-data13:14
tedgOh, okay.13:14
* tedg stops thinking13:14
zygatedg: if you get plainbox to work then checkbox is easier :)13:14
ogra_pitti, oh ...13:14
tedgzyga: So how do I know the requirements?13:15
zygatedg: if I can give you a bit of advice on requirement files: follow what travis-ci allows13:15
zygatedg: it's spelled in setup.py13:15
zygatedg: install_requires13:15
zygatedg: the standard way13:15
ogra_pitti, i just noticed that i didnt remove "allow-hotplug eth0" from my /e/n/i setup, could that confuse the generator ?13:15
zygatedg: if you do what travis-ci allows then any python project will be snappy buildable13:15
ogra_(does it check for that in any way ?)13:15
zygatedg: most projects just setup.py develop / install their deps13:15
pittiogra_: no, it doesn't look at that at all13:15
ogra_ok13:16
tedgzyga: I find "the standard way" doesn't really exist anywhere :-)13:16
zygatedg: and using wheels is the current hotness to make that easier to do13:16
zygatedg: I disagree13:16
zygatedg: setuptools is the way13:16
zygatedg: requirements are for separate concerns13:16
zygatedg: like deployment or other weird web scenarios13:16
zygatedg: if you setup.py install is broken then users complain (if they care)13:17
zygatedg: but it's definitely the standard way13:17
tedgzyga: Travis CI just allows you to run binaries. We're not interesting in providing a bunch of scripts...13:18
tedgzyga: http://docs.travis-ci.com/user/languages/python/13:18
zygatedg: yes but it supports certain things for free out of the box: like using setuptools without asking13:18
tedgzyga: So like what we do with the python3-project plugin?13:19
zygatedg: yes, except right now that implementation is broken13:19
zygatedg: (for the same reasons I said earlier)13:20
tedgzyga: So I think there's a patch for that, we just need to work out the details.13:20
tedgzyga: But, to be clear, you don't want or use a requirements.txt file.13:20
sergiusenstedg, if we accept the fix from BjornT I think zyga's problem would be a non issue13:21
zygatedg: it'd be nice if we support it if people want to use it (for version pinning)13:21
zygatedg: but it should be optional IMHO13:21
tedgsergiusens: I think so too, but I haven't pinged you about your brainstorming weekend yet :-)13:21
sergiusenstedg, hasn't happened :-/13:21
tedgsergiusens: Start your weekend now! ;-)13:21
sergiusenstedg, good idea13:22
zygasergiusens: btw, I failed with my .run() cleanup, it's utterly hard to clean up run to be space free and run without shell13:22
zygasergiusens: I will rescue a few patches from that but the whole patchset has failed (too many changes, tests change, etc)13:22
tedgsergiusens: So let's take BjornT's patch for now and then we'll fix it later. I think it's more important to provide features today than provide smaller snaps.13:23
tedgHate taking on technical debt, but adoption is more important.13:23
sergiusenstedg, certainly13:23
ogra_how else would the duct tape producing industry survive ?13:24
tedgSo then elopio if you could look at my pip branch and tell me how to format the long lines in python compatible way I think it's ready too.13:25
tedgogra_: I believe there is a pact between the WD-40 industry and the duct tape industry. An evil, evil pact.13:25
tedgelopio, sergiusens: I put pip tests in the integration suite. Is that where we want them? They take longer because they download stuff.13:27
ogra_tedg, lol13:27
zygatedg: can I review it it as well?13:29
tedgzyga: sure!13:29
zygatedg: ok, I'll check it out now13:30
* zyga cries for clean, logical commits :|13:30
zygatedg: there's no chance to squash those commits into separate changes?13:33
tedgzyga: Eh, plastic surgery makes you look more beautiful, but hides who you really are :-)13:34
zygatedg: well, reading a merge commit to see why it's there really is bad for review13:34
zygatedg: and reviewing the whole change in one commit is not so nice13:34
zygatedg: can you have a more realistic requirements please? argparse is hardly good (bundled with 3), something that has deps of its own perhaps?13:36
tedgzyga: You're not going to get me excited about the difficulty of reading a 200 line diff.13:36
tedgzyga: We're not testing PIP here, just that PIP gets called.13:37
zygatedg: that's not useful13:37
zygatedg: the test doesn't check that what you do works in pratice IMHO13:38
zygatedg: especially that it does work if you have argparse installed on the host13:38
zygatedg: or if you have something it depends on installed is also present13:38
ppisatirickspencer3: sent13:38
tedgzyga: ? We're not looking at the host installation.13:38
rickspencer3thanks ppisati13:39
tedgzyga: We're testing that it gets downloaded and installed into the part. If it's in the host the test will fail.13:39
zygatedg: does it work if you use this plugin to install plainbox?13:39
zygatedg: (since plainbox is installed most of the time, that's an interesting test in itself)13:40
ppisatirickspencer3: it's a bit long and hackish, but i tested every step on a clean snappy instalation, so it should just work13:41
ppisati(last famous words...)13:41
rickspencer3haha13:41
tedgzyga: I looked at plainbox and it seemed to have multiple requirements files, which it doesn't support. But you said that you don't use requirements files, which is what this branch supports.13:41
zygatedg: hmm13:41
tedgzyga: So I think the answer is no.13:41
zygatedg: but what about _no_ requirements13:41
zygatedg: and just "get this thing from pypi"13:41
zygatedg: which is akin to pip install $stuff13:42
tedgzyga: No, didn't add that. I think that people mostly do that in their setup.py. No?13:42
zygatedg: hmm, but then they cannot use this part, right?13:43
zygatedg: or am I missing something13:43
zygatedg: use case: I want to pull in some tools into my snap, those are python tools I can pip install on my host13:43
zygatedg: so I use python3 parts13:43
zygatedg: and the part name is the pip thing to install13:43
zygatedg: I can also use requirements (e.g. to handle optional elements)13:44
zygatedg: the example pip-test in your MR could look like:13:44
tedgzyga: This isn't about having tools on your host, this is about dependencies for your snap. You won't be able to use these tools for anything other than building/running.13:44
zygaparts:\n13:44
zygatedg: I _know_ that was an analogy13:44
zygatedg: parts: argparse: type: python213:44
zygatedg: e.g. if I want to have a script that uses some libraries I can just use the python3/python2 parts in my snapcraft to pull them from pypi13:45
tedgzyga: If you were only targetting snappy, that would make some sense? But it seems like if you wanted your build system to work anywhere else you'd use setup.py and requirements.txt.13:45
zygatedg: (my analogy was about moving a familiar developer/user experience)13:46
zygatedg: so if I wrote a script on ubuntu13:46
zygatedg: and I need some stuff from pypi (that you can normally pip install)13:46
zygatedg: then I want to copy that into my snap by using a python3/python2 part type and just listing the thing I want13:46
sergiusenstedg, yes, system interactions into integration_tests is fine13:46
tedgSo your use case is someone custom modifying a system instead of someone building something that could be deployed in production and used by other people.13:47
zygatedg: the requirement file that you specify is relative to snapcraft.yaml, right?13:47
tedgzyga: Yes, it would be in the project directory.13:47
zygatedg: ok13:48
zygatedg: looking at the MR, I'd like an explanation of what the pull method is doing there13:48
zygatedg: the symlink and oll the other stuff seem a bit magic13:49
zygatedg: can you explain what that is doing and why?13:49
tedgzyga: Yeah, it's a bug. Seems that you can't change the root and specify the deb layout.13:49
zygatedg: mmmm, I'm sure you can, what did you try?13:49
zygatedg: the deb layout just uses dist-packages13:50
tedgzyga: Specifying deb layout and the root directory :-)13:50
zygatedg: I did just that :)13:50
tedgzyga: Just the two command lines, then it put everything in /usr13:50
zygatedg: hmmm, I'll look at that13:51
zygatedg: I understand your branch doesn't address separation of python from the host yet13:51
zygatedg: and something else that sergiusens mentioned will work on that?13:51
zygatedg: specifically that pip will not install things that are on the host, so plainbox for example would be broken because its dependencies might be on the host for most developers13:52
zygatedg: (and we should just force it to install everything as if system python didn't exist and didn't have any packages)13:53
tedgzyga: This branch allows normal setuptools to install into the parts: https://code.launchpad.net/~bjornt/snapcraft/setuptools/+merge/27073313:53
* zyga looks13:53
zygahmmmm13:53
zygathis is not enough13:53
tedgzyga: Provide a test, we will fix it :-)13:53
zygatedg: sure13:53
tedgIRC based TDD ;-)13:54
zygatedg: one thing is that this will break if you are in a virtualenv13:54
tedgHmm, that uses prefix instead of root dir.13:54
zygaand many python developers are ;)13:54
zygamy testing showed that you need to avoid importing site13:54
zygaand you need to explicitly run the python from stage dir, not any python13:54
zygathis runs any python2 and any python313:55
zygawhich is wrong13:55
zygato avoid importing site you need to pass -S13:55
zygathen it will work13:55
tedgThe PATH is configured to only run the version from the parts install.13:55
zygaso that's one test case :)13:55
zygatedg: I found that not to be the case, perhaps another bug was at play?13:55
tedgThat then adjusts the directories to use the correct site directory.13:55
zygatedg: ok, I'll check again now13:55
tedgYou might make sure to run my branch in that regard, there was a small bug with environments that it fixes. I don't think it broke that for python-project though.13:56
tedgOnly broke in python itself.13:56
sergiusensmvo, http://paste.ubuntu.com/12408693/ any idea?14:03
zygasergiusens: what's the schedule for stable API for plugins/parts?14:03
sergiusensmvo maybe the profile thing doesn't like symlinks14:03
zygasergiusens: as in, what's the deadline for making changes to those parts14:03
sergiusenszyga, the spec is in place, it can't change; just needs implementation14:04
zygasergiusens: so all the methods need to return True?14:04
=== Abhishek__ is now known as Abhishek_
sergiusenszyga, no, that's an implementation detail; but in any case once we do release, which is sometime this week it can't change as plugin writing docs would need to change as well14:05
sergiusenszyga, I say if you don't like that, get it done today ;-)14:06
zygasergiusens: that's what I wanted to hear ^_^14:06
zygasergiusens: I'll talk to you in a sec, stand-up time14:06
zygasergiusens: is there a public link to the spec?14:06
zygasergiusens: or is it the same private doc?14:06
sergiusenszyga, I thought the gdoc I gave you was public14:06
zygasergiusens: ah, I assumed otherwise14:07
zygasergiusens: ok :)14:07
mvosergiusens: no, sorry, this is a 15.04 image? might be a sideload issue14:08
zygamvo: did you think about the sideload issue I've mentioned last week?14:08
zygamvo: (to store sideload flag not in $prefix)14:09
mvozyga: the origin information (.sideload, .mvo) is causing us some grief but I have not thought it through yet14:10
sergiusenszyga, mvo  if it is not in the form of $name.sideload it would be in the form of /apps/$origin/$name14:11
mvothat was the discussion in cape town, right?14:11
sergiusensmvo, nope, this was our invention in Austin14:12
zygasergiusens: I don't understand how that addresses that C/C++ stuff that hardcodes --prefix cannot be sideloaded14:12
zygasergiusens: maybe I'm missing stuff, sorry, split attention14:12
sergiusensmvo, btw, I can't symlink the config hook14:14
mvosergiusens: its exploding :( ?14:14
mvosergiusens: the aa_exec error?14:15
sergiusenszyga, I don't know what you are talking about either; all I'm saying is that the problem you present is not solvable by hardcoding paths14:15
sergiusensmvo, aa_exec, or the runner (or snappy) how was it again that the profile for config was created?14:16
zygasergiusens: ah, sorry, I didn't understand that, so what is the approach to software that does hardcode stuff today?14:16
tedgzyga: You shouldn't hardcode prefixes, you really need to use the environment variables there.14:16
zygasergiusens: well, preach upstream, the reality is different14:16
mvosergiusens: aa-clickhook is doing that still - is that on rolling or 15.04?14:16
zygatedg: ^^^14:16
tedgYou're gonna need to chroot things for those cases.14:16
zygatedg: if I want to integrate an upstream project I don't want to have to patch it)14:16
sergiusensmvo, 15.0414:16
zygatedg: that's a bit unfortunate IMHO, why is that required?14:17
tedgWe have talked about providing some amount of automatic chroot, but haven't gotten to it.14:17
zygatedg: if the prefix was deterministic from version and snap name, that would be enough for all the C software in the world14:17
tedgzyga: Because we don't want to commit to even /apps14:17
zygatedg: since we'll break everything on 16.04 I think it would be useful to have a prefix that works in the meantime, even if we change it later14:18
zygatedg: maybe the chroot thing will fix it but it sounds like an overkill to me14:18
tedgWhat you're saying is if we backed ourselves into another hardcoded filesystem layout things would be easier. But the problem is hardcoded filesystem layouts, not how they're layed out.14:18
zygatedg: and (again) today, it's not working14:18
zygatedg: no, hardcoded filesystem layout are a reality for the past few decades14:18
zygatedg: if we require software to be relocatable then we put extra roadblock14:19
tedgYes, let's fix it!14:19
zygatedg: that's silly, that's not our battle to fight14:19
tedgHeh, frankly, we chose to fight it over three years ago.14:19
zygatedg: and especially, I don't see what we gain by requiring this apart from nebulous "not hardcoded"14:19
zygatedg: if that means that we cannot easily integrate major upstream software than I just see that as a disadvantage14:20
fgimenezogra_, i'm looking into https://code.launchpad.net/~elopio/snappy/resize_test/+merge/270892, should this work in 15.04?14:22
tedgzyga: An opinion that you're entitled to. I think the fix for those folks is to chroot it until they realize it's 1990 already.14:22
zygatedg: I think that's a bold statement from the world where _everything_ written in C hardcodes prefix and we gain exactly nothing today by not supporting that14:23
zygatedg: and targetting IOT space will intersect a lot of those projects14:23
zygatedg: and last time I checked ubuntu is in the 1990s you claim14:23
zygatedg: and will be in 16.0414:23
tedgYes, and we're fixing that!14:24
zygatedg: no, we just look the other way14:24
zygatedg: are we sending patches anywhere?14:24
zygatedg: we're not fixing anything, we just say we don't support de-facto standards today14:24
ogra_fgimenez, 15.04 and rolling should be identical now wrt that code ...14:25
tedgzyga: Reality is that most projects provide environment variables to change almost all of those prefixes. We've even patched xcb where we needed it. For instance, all of the QML stuff is relocatable with the qml plugin. Those base libraries are all written in C.14:27
fgimenezogra_, ok thanks, i'm getting some errors in 15.04, i'll try to fix the install-test-initramfs script to make it work in both14:27
tedg(well C++, but for this argument that's not important)14:27
zygatedg: and what environment variables those upstream projects we've patched are respected? those of snappy?14:28
ogra_fgimenez, oh, note that i landed the final fixes only on the weekend, so make sure you have a recent image in use14:29
tedgzyga: Nope, their own, we then set them up for snappy. http://bazaar.launchpad.net/~snappy-dev/snapcraft/core/view/head:/snapcraft/plugins/qml.py#L7814:29
fgimenezogra_, currently 147, is that ok?14:30
ogra_yep14:30
ogra_that should work14:30
ogra_i did a plethora of fresh installs with 146 and 147 on actual x86 HW on the weekend here14:31
zygatedg: I still think this is totally counter productive, for all the low-level plumbing stuff that we don't ship anyone that needs to ship it has now to maintain a fork just to fix this non-issue (they gain nothing from this) or hardcode /apps/$name/$version and ignore side-loaded (or reverse, non-side-loaded) snaps:14:31
zygatedg: I understand what you are sying but I think it doesn't work for the long tail of things and doesn't help us in adoption today14:32
tedgzyga: Sure, but there are very few opportunties that you get to make things better. We're trying to take one.14:39
zygasergiusens: would you oppose a snapcraft part that can install extra build dependencies?14:41
zygasergiusens: dynamically from the part itself14:41
zygasergiusens: and if not, what other approach should one take to handle, e.g., building bluez14:41
sergiusenszyga, iirc, there's already a build-depends you can use with parts14:42
zygasergiusens: with parts yes, but what about listing those in the snapcraft.yaml itself14:43
zygasergiusens: not at the part type level, at a part _instance_ level14:44
sergiusenszyga, don't use the word part so generically, it is confusing ;-)14:44
zygasergiusens: sorry, so, e.g. to build bluez14:45
zygasergiusens: I'd like to say that building bluez requires all the packags on the host14:45
zygatedg: bluez has 34 hardcoded instances of $prefix14:49
zygatedg: in executables alone14:49
zygatedg: a quick source search has 100 (exactly) matches for places that need to be patched14:52
zygatedg: can it be done? yes, it is useful? not for anyone consuming snappy14:52
zygas/consuming/targetting/14:53
tedgzyga: Then use a chroot?14:56
zygatedg: how?14:56
tedgzyga: Have the service start a script that does a chroot to the $SNAP_APP_PATH14:57
zygatedg: will it be able to access the hardware this way?14:57
zygatedg: e.g. /dev, /sys, /proc14:57
tedgzyga: You might have to map some directories into the chroot.14:57
zygatedg: I assume netlink works14:57
zygatedg: how?14:57
zygatedg: do I mount --bind ? something?14:57
zygatedg: and still integrated with outer dbus14:58
zygatedg: if you add all those up, and it works, are we still making the world better?14:58
tedgzyga: Getting closer ;-)14:58
zygatedg: (I'm using this as an example I work on right now, it's not made up on purpose to be evil)14:58
tedgzyga: The reality is that no one will change unless they have a reason. If they see the environment for bluez on Snappy, they'll not use prefix as much, and eventually change.14:59
tedgzyga: Change is a slow process.14:59
zygatedg: I won't do the bind mount, or anything else, I'll hardcode /apps/$name/$version.sideload because I don't have the time to research the other solution that is identical if it works14:59
zygatedg: well, I don't want /apps to change15:00
zygatedg: I just want to fix side-loading15:00
zygatedg: and I'm showing my case as an argument15:00
tedgzyga: For instance, for frameworks it may change to /framework15:00
zygatedg: I'll gladly hardcode /framework and build a new snap, that's easy15:00
zygatedg: still .sideload will be a thorn15:01
tedgYou're fighting this. Just feel the force and blend with it. Lift the speeder out of the swamp. I know you can do it.15:01
zygatedg: I'll accept patches that make bluez relocatable ;)15:02
zygatedg: if you want to wade thorough 100 places that need this15:02
zygatedg: I'm like the model consumer of snappy15:02
zygatedg: they will just totally ignore this non feature15:02
zygatedg: and say you have to sideload / not sideload15:02
zygatedg: this is a non feature because all the value is for snappy developers and none for the snap developers that have to now do extra work15:03
zygatedg: also this: https://github.com/ninjasphere/ninjasphere-snappy/blob/master/docs/TESTING.md#known-issues15:06
zygatedg: third bullet point15:06
tedgzyga: You're welcome to argue and dislike it, but the reality is that we're not going to hardcode the paths anytime soon.15:06
zygatedg: I'll argue otherwise at the sprint :-)15:07
tedgIs there a way to add udev rules in a snap?15:27
sergiusensmvo, Chipaca ideas? http://paste.ubuntu.com/12409199/15:29
beunotedg, wouldn't that break confinment pretty badly?15:29
sergiusenstedg, no, only snappy hw-assign or through the gadget or oem snap15:30
Chipacasergiusens: sudo15:30
sergiusensChipaca, doh15:30
Chipacasergiusens: :)15:30
Chipacasergiusens: if you readd/write from priv'ed locations, check your EUID15:31
Chipacasergiusens: so you're more user gefriendly15:31
tedgbeuno: Well, we need some way to add new devices. i.e. my new cool USB device to go along with my software package.15:36
tedgbeuno: We can parse the file. We probably don't want to just copy the file somewhere.15:36
* beuno nods15:37
ogra_tedg, isnt that what hw-assign actually does ?15:37
tedgogra_: Oh, does it? I didn't know how it worked. I figured it was apparmor magic.15:38
ogra_i think it is both, apparmor and udev rule creation ... jdstrand would know :)15:39
* jdstrand reads backscroll15:41
jdstrandtedg: is there a way to add udev rules to a snap> the oem snap is capable of doing this. I believe that is 'config'urable (/me points you to mvo for that)15:44
beunojdstrand, tedg here wants to do it from an app15:44
ogra_evil evil :)15:44
beunobecause his app will use a device he built, lets say15:44
tedgsergiusens: Seems that galileo doesn't use a requirements.txt, just setup.py15:44
jdstrandso, the answer to that is 'no'15:45
jdstranda gadget could describe that15:45
beunoburn!15:45
jdstrandan app shouldn't15:45
beunowell, lets think about it a bit15:45
beunoso what if you ship a special USB device, lets say, a sensor15:45
jdstrandI'm stating the current implementation15:45
beunoover USB15:45
tedgjdstrand: The specific use case was galileo who does Fitbit sync, and needs to add the USB IDs for the fitbit dongle.15:45
jdstrandI know it is being reworked15:45
beunoor tedg's real case15:45
beunook15:45
beunogood15:46
beunoI don't think we have anybody on that atm15:46
jdstrandI'm not sure who is reworking that15:46
beunoactively working on it15:46
jdstrandheh15:46
beuno:)15:46
ogra_but wasnt hw-assign actually meant to solve this15:46
ogra_as least in a first shot ?15:46
* ogra_ doesnt see why it works for webcams or serial USB controllers but not for a fitbit gadget15:47
jdstrandthe real thing is that you don't want something that is untrusted (ie, an app) interacting with the system in ways that it can break out of confinement without some sort of user/admin interaction15:47
tedgI'm not saying it is a today thing, but I do think we need some way to add USB IDs and Bluetooth, etc, etc.15:47
jdstrandie, an admin saying-- ok, you can access this device (fine)15:47
ogra_right15:47
ogra_and that is what hw-assign works for today15:47
beunoso this is assign-and-describe15:47
tedgI think that's not a way that the app controls the system as much as saying "if this exists, I'd like to know about it."15:47
beunowe're missing the and-describe part, common for USB15:48
jdstrandie, snappy install allows declaring udev but not installing them without the admin (fine, not implemented, just saying a very simple idea)15:48
tedgWhere "if this exists" could include something the system didn't know about before.15:48
jdstranda gadget saying, this is all the stuff on the system, here are udev rules (fine, where I thought we were going, but being reworked)15:49
jdstrandan oem snap saying, these are a bunch of udev rules and snaps that can use them without me doing anything (fine, cause the oem is special)15:49
jdstrandbut arbitrarily extending udev itself, that hasn't been discussed afaik15:50
tedgI think that works for fixed devices that are in production. But it doesn't work for makers who are building something on a Raspberry Pi and want to add functionality.15:50
jdstrandno, I understand where you are coming from15:50
jdstrandagain, just stating thoughts surrounding it15:50
jdstrandit sorta sounds to me like a separate gadget snap15:51
tedgK, anyway. I imagine that'll have to go on the backlog today. But it was something I just ran into.15:51
jdstrandbut, idk-- others are (re)defining this15:51
beunojdstrand, also, gadget and oem snap are the same thing, yes?15:51
jdstrandwell, oem is something that is implemented today. I thought gadget was an evolution of the oem snap15:52
jdstrandbut perhaps I am wrong on that point15:52
beunoI... don't know15:52
beuno:)15:52
beunoI'm sure mvo does, he knows everything!15:52
jdstrandI just hear occasionally that a gadget snap is being used in ways I didn't think an oem snap would be. sergiusens could comment further I'm sure15:52
jdstrandyes, or mvo15:53
jdstrandit might be possible with sufficient yaml/input validation/templated output to have an app snap declare udev rules for snappy to add to the system15:54
jdstrand"these are the product ids" and snappy says "oh, ok, I'll add some stuff to udev so the system knows about it"15:55
Chipacamvo: the sideload framework issue is real15:55
zygaChipaca: which issue?15:55
* zyga is curious about that15:56
jdstrandI would suggest getting p itti in on that conversation though15:56
mvoChipaca: hmmm15:56
Chipacazyga: https://bugs.launchpad.net/snappy/+bug/149370615:56
ubottuLaunchpad bug 1493706 in Snappy "side-loading frameworks broken" [Critical,Triaged]15:56
mvoChipaca: is it because of the version number change?15:56
Chipacamvo: let's fix it :)15:56
mvoChipaca: that would be ideal indeed, I was just lazy^Wbusy15:56
Chipacamvo: I don't know where, but i must have forgotten to do the right thing with versions somewhere :)15:57
zygawhat is the _multi suffix? fat snaps?15:57
zygaor just arbitrary string15:57
Chipacamvo: because, yes, it's trying to use a version number it should not know about15:57
Chipacamvo: however15:57
Chipacamvo: let me check wiht a simpler framework first15:57
Chipacamvo: it might be docker doing things wrong :)15:57
Chipacamvo: it's related to servcies15:59
Chipacamvo: "start" is also not doing the right thing, somehow15:59
Chipacamvo: here, you test it: https://public.apps.ubuntu.com/anon/download/canonical/hello-dbus-fwk.canonical/hello-dbus-fwk.canonical_1.0.1_multi.snap15:59
mvouhh16:00
mvook16:00
Chipacamvo: so16:02
Chipacamvo: something is still taking the version from the package.yaml16:02
mvo:(16:02
Chipacamvo: directly16:02
Chipacamvo: instead of through part.Version()16:02
Chipacamvo: found it16:03
Chipacamaybe :)16:03
Chipacamvo: i'll have a branch up in a bit16:06
Chipacamvo: figured it out. It's down to origin being empty for frameworks, again16:13
Chipacathe sooner we nuke that the better :(16:13
* Chipaca runs the tests16:15
Chipacamvo: you've got unreachable code in xgettext-go, and my vet complains :)16:16
mvoChipaca: what a crybaby16:17
mvoChipaca: let me fix that, vet gets cleverer each way16:17
mvoeach day16:17
Chipacamvo: https://code.launchpad.net/~chipaca/snappy/always-take-version-from-filesystem/+merge/27098816:17
Chipacamvo: if i were clever i would've noticed it the first time :)16:19
mvo:)16:20
* mvo hugs Chipaca16:20
zygasergiusens: https://code.launchpad.net/~zyga/snapcraft/custom-plugin/+merge/27099417:17
zygasergiusens: early feedback please17:18
zygasergiusens: I've used this to build a few interesting bits17:18
zygasergiusens: most notably missing are build dependency handling17:18
zygasergiusens: and environment sucks (correct me if I'm wrong, this cannot be done without fixing many other things now)17:18
zygasergiusens: example snapcraft.yaml for it: http://paste.ubuntu.com/12410227/17:19
sergiusenszyga, you can build a package that depends on snapcraft I guess, but I don't thing we want commands available to do this; the original snapcraft was like this fwiw17:23
zygasergiusens: can you be more explicit, I don't understand what you are saying17:25
zygasergiusens: that explicit commands are bad?17:26
zygasergiusens: or that something else is bad?17:26
sergiusenszyga, explicit commands17:26
sergiusenszyga, this can be driven by a Makefile in most cases, can't it?17:26
zygasergiusens: yeah, perhaps17:31
zygasergiusens: is there something I can do about the environment part17:31
zygasergiusens: the .replaces() thing is a bit terrible but I didn't find any better ways of doing it17:31
=== slangase` is now known as slangasek
=== ahayzen_ is now known as ahayzen
zygahmm, I'm seeing a weird low-level error17:55
zygaaa_change_onexec failed with -117:55
zygahas anyone seen this, googling suggests its related to apparmor17:55
ogra_well, do you see any apparmor messages in syslog =17:58
ogra_?17:58
zygaogra_: checking17:59
ogra_anyway, ask jdstrand or tyhicks18:00
ogra_definitely apparmor related18:00
tyhickszyga: you need the errno value set18:00
tyhickszyga: bah, I didn't phrase that correctly.... you need to figure out what errno is set to when aa_change_onexec() returns -118:00
zygatyhicks: I'm not sure which part applies this, it happens when I run one of the unconfined executables18:00
zygayeah18:00
zygaI understand18:00
zygatrying to hek18:00
* zyga would love gdb now18:00
zygaogra_: nothing in syslog18:01
* jdstrand is guessing the profile isn't loaded without knowing the errno18:01
tyhicksyes, that's most likely the problem ^18:01
jdstrandzyga: what command are you running?18:01
zygahostapd_cli18:01
zygahostapd works _cli doesn't18:01
zygahostapd_2.4-1_amd64.snap18:02
zygaon chinstrap in ~zyga/18:02
zyga333K18:02
jdstrandzyga: can you paste what is in /apps/bin/hostapd_cli18:02
zygayep18:02
jdstrandtyhicks: I'll take a first look at this18:02
zyga(I'm doing this in kvm)18:03
tyhicksok18:03
zygahttp://pastebin.ubuntu.com/12410753/18:03
zygajdstrand, tyhicks: without the wrapper it works okay18:04
jdstrandzyga: can you paste the output of cat /var/lib/apparmor/profiles/click_hostapd.sideload_hostapd_cli_2.4-118:04
zygacat: /var/lib/apparmor/profiles/click_hostapd.sideload_hostapd_cli_2.4-1: No such file or directory18:04
jdstrandzyga: also, please paste the output of: sudo aa-status|grep hostapd18:04
zygaso that nails it18:04
zyga   hostapd.sideload_hostapd_2.4-118:05
jdstrandwell, maybe18:05
jdstrandlet me look at the snap18:05
zygahttp://pastebin.ubuntu.com/12410779/18:05
zygalook at this, there's one hfor hostapd though18:05
jdstrandzyga: http://paste.ubuntu.com/12410801/18:08
jdstrandzyga: rename hostapd_cli to hostapd-cli18:08
jdstrandI'm quite surprised that snappy build worked let alone snappy install18:09
jdstrandseems there is an input validation issue there18:09
zygajdstrand: so the external binary must be hostapd.hostapd-cli18:09
zygajdstrand: and binaries with _ are not supported?18:09
jdstrand'_' is not allowed in the pkgname, service/binary name or version18:09
zygajdstrand: ah, fun18:09
zygajdstrand: heh :)18:09
zygajdstrand: thanks!18:09
jdstrandnp18:10
zygasergiusens: ^^ nothing in snacpraft assemble complained18:10
jdstrandtyhicks: fyi-- the outcome ^18:10
sergiusenszyga, care to patch the validation thingy?18:11
sergiusenszyga, the regex18:11
zygasergiusens: it might be doable in the schema18:11
zygasergiusens: or is there some other validation?18:12
sergiusenszyga, it should, I have a regex there; it may be wrong18:12
sergiusensand I suck at regex18:12
jdstrandis snapcraft written in go?18:12
jdstrandcause snappy I'm quite sure has the regex in there18:12
zygasergiusens: hehe18:12
zygasergiusens: no worries, it should be easy18:12
jdstrandzyga: but snappy install didn't error?18:13
jdstrandthat is another bug18:13
jdstrand(if it didn't)18:13
sergiusenszyga, it is missing the regex for binary name and service name http://bazaar.launchpad.net/~snappy-dev/snapcraft/core/view/head:/schema/snapcraft.yaml#L4618:14
sergiusensa bug or MP would be great18:14
tyhicksjdstrand: there is a bug opened for that (I I think I debugged this same problem for someone once while you were on vacation)18:15
tyhickslet me look18:15
jdstrandsergiusens: fyi, snappy/click.go is wrong: const servicesBinariesStringsWhitelist = `^[A-Za-z0-9/. _#:-]*$`18:17
jdstrandsergiusens: '_' should not be in that list18:17
sergiusensjdstrand, right18:17
zygasergiusens: I'll do both in a sec, some IRL activities now18:18
jdstrandneither should '#'18:18
sergiusenszyga, I'll do it, jdstrand just gave me the regex ;-)18:18
zygajdstrand: nope, snappy install (remote) didn't catch it18:18
* jdstrand looks at click-apparmor regex18:18
zygasergiusens: ehe, okay :)18:18
zygaI think there's another bug18:23
zygaI'm trying to use exec: hostapd_cli and name: hostapd-cli18:23
zygaand it's ignored18:23
zygait derives the name from exec all the time18:24
zyga(for the wrapper)18:24
jdstrandzyga: exec should point to the file on disk relative to the topdir of the unpacked source18:24
jdstrandah, then yes, that is a bug18:24
zygajdstrand: name with - is ignored?18:24
zygaeh :)18:25
* zyga digs deeper18:25
jdstrandI'm saying if snapcraft is handling the '_' to '-' for exec vs name, that is an issue18:25
jdstrandisn't*18:25
zygayeah _wrap_binaries is broken18:26
zygajdstrand: it's more than that18:26
zygajdstrand: it's not handling it, it's not checkign it and if you try to tell it explicitly to rename it, it ignores it18:26
jdstrandI don't doubt there is more than that18:26
zygajdstrand: so what exactly cannot have _ in it?18:27
zygajdstrand: the wrapper script, the target executable or both?18:27
jdstrand'name'18:28
zygajdstrand: ok18:28
jdstrandin either the service or binary yaml18:28
zygajdstrand: so when I have name 'hostapd-cli' and the generated executable wrapper is hostapd_cli.wrapper is that good or bad?18:29
zygait seems bad18:29
zygahmm, but it works18:30
zygaweird18:30
jdstrandzyga: it probably depends. in and of itself, it is neither-- 'name' is specifically there to reference the binary by another name18:30
jdstrandexec is optional18:31
zygasergiusens: http://paste.ubuntu.com/12410981/18:31
jdstrandso if you nave: name: bin/foo, then you get /apps/bin/pkgname.foo18:31
zygasergiusens: this is the patch that "fixes" it for me18:31
zygasergiusens: but it's weird18:31
zygasergiusens: and the if makes no sense anymroe18:31
jdstrandbut if you use exec, you can do this:18:32
jdstrandname: foo18:32
zygajdstrand: no, because then name is totally ignored18:32
jdstrandexec: bin/foo.wrapper18:32
jdstrandand then you get /apps/bin/pkgname.foo18:32
zygajdstrand: I just tried that, but I didn't want to provide my bin/foo.wrapper18:32
jdstrandI'm saying how it is supposed to work18:32
jdstrandwith snappy itself18:32
zygammm18:32
zygayeah, I understand18:32
jdstrandI don't know what snapcraft is trying to do18:32
zygaeh, ignore me sergiusens18:34
zygaI'm tired18:34
jdstrandsergiusens: so the other bug is that snappy install didn't complain when the profile wasn't generated. your regex change will sideste that and I think any other fixes can wait until snappy absorbs click-apparmor18:35
jdstrandsidestep*18:35
jdstrand$ sudo aa-clickhook -f18:36
jdstrandERROR: Could not parse click manifest. Skipping 'hostapd.sideload_hostapd_cli_2.4-1.json'18:36
* jdstrand notes 'hostapd_cli' with the '_'18:36
* Chipaca puts on his chef hat and goes shopping18:37
jdstrandsergiusens: ok, so click-apparmor itself isn't terribly picky except for:18:38
jdstrandout = n.split("_")18:38
jdstrandif len(out) != 3:18:38
jdstrand    raise...18:38
jdstrandso getting rid of the '_' alone should be ok18:38
tyhickssorry, I got sidetracked with another irc conversation18:38
tyhicksyes, that split() was the problem that I debugged, as well18:39
jdstrandthis all gets back to the application id18:39
jdstrandhttps://wiki.ubuntu.com/AppStore/Interfaces/ApplicationId18:39
tyhickshmm.. I found the old debugging session but it doesn't look like a bug was ever filed :/18:40
jdstrandwhich is something that is pretty deeply ingrained in the system. we could support '_' in the service/binary, pkgname, version if we escaped it. but that would have to be done quite carefully18:40
jdstrandtyhicks: well, so, the system is implemented very specifically. snappy isn't handling that right18:41
sergiusenstyhicks, jdstrand there https://bugs.launchpad.net/snappy/+bug/149566218:41
ubottuLaunchpad bug 1495662 in Snappy trunk "Services and binaries allow _ #" [High,New]18:41
tyhickshttps://bugs.launchpad.net/snappy/+bug/147026518:41
ubottuLaunchpad bug 1470265 in Snappy trunk "Binary with an underscore fails to produce an apparmor profile" [High,Confirmed]18:41
jdstrandsergiusens: note, https://wiki.ubuntu.com/AppStore/Interfaces/ApplicationId has regexes18:42
tyhicksthere are two bugs18:42
jdstrandsergiusens: those regexes are in the context of touch of course18:43
sergiusensjdstrand, should I keep #?18:43
zygasergiusens: can we resume daily builds on https://code.launchpad.net/~snappy-dev/+recipe/snapcraft-daily18:43
zygasergiusens: some people are bloked by this18:43
sergiusenszyga, in a bit18:43
jdstrandsergiusens: I was surprised to see it. it won't break apparmor18:43
jdstrandat least, it shouldn't :)18:43
zygasergiusens: what do you mean?18:43
jdstrandsergiusens: so, your call18:43
sergiusensjdstrand, the tools block it, right?18:44
sergiusensjdstrand, if so I will, as it should be safe (store side won't have uninstallable packages)18:44
sergiusensclick-review-tools that is18:44
jdstrandthat's true. I'm quite sure they will18:45
jdstrandgive me a sec18:45
jdstrandbtw, why didn't the tools run for zyga?18:45
sergiusensjdstrand, not sure18:49
jdstrand - lint:hooks_valid:hostapd#cli18:51
jdstrandmalformed application name: 'hostapd#cli'18:51
jdstrandso, yes, the review tools will complain18:51
jdstrand$ sudo aa-clickhook18:53
jdstrandERROR: Could not generate AppArmor profile for 'hostapd.sideload_hostapd#cli_2.4-1.json'. Skipping18:53
jdstrandsergiusens: please remove '#'18:53
sergiusensjdstrand, got it18:53
jdstrandI bet that regex is coming from easyprof18:53
jdstrandyes18:54
jdstrand if re.search(r'^[a-zA-Z0-9][a-zA-Z0-9_\+\-\.:~]+$', s):18:54
jdstrand    return True18:54
jdstrandreturn False18:54
sergiusensjdstrand, the regex in click.go allows spaces too :-/18:54
jdstrandsergiusens: we should remove that too18:54
jdstrandnot that the regex in easyprof isn't caring about APP_ID18:55
jdstrandclick-apparmor cares about APP_ID18:55
jdstrand(which is correct)18:55
jdstrand(ie, easyprof and click-apparmor are at different layers and their respective checks are valid for their respective layers)18:56
jdstrandsergiusens: alright, rather than going through all this, how about you fix it and ask me to review?18:56
jdstrandsergiusens: fyi, easyprof is as strict as it is to ease quoting and avoid shell metachars18:58
jdstrandapparmor itself can have much more in the profile name. if we want to talk about letting other stuff in later, then we can have a discussion about that18:59
jdstrand'_' will always have to be handled carefully19:00
jdstrand(cause of APP_ID)19:00
jdstrandanyhoo19:00
* jdstrand -> writing docs19:00
tedgsergiusens: so to be able to use stage-packages in my plugin I need to add it to /plugin/foo.yaml. Do we want that? Or do we want to include the global stuff automatically?19:49
sergiusenstedg, you missed my collisions branch19:55
tedgsergiusens: Sorry, never ran into it.19:55
sergiusenstedg, https://code.launchpad.net/~sergiusens/snapcraft/collisions/+merge/270942 :-)19:56
* tedg rimshot19:56
sergiusenstedg, more like a sad trombone ;-)19:56
tedgLOL, oh, ouch! :-)19:56
sergiusenstedg, I also have this which I just noticed I need to wrap into the wrap exe stuff https://code.launchpad.net/~sergiusens/snapcraft/config/+merge/27098919:58
tedgsergiusens: Is there some way we can share schema with snappy?20:00
tedgsergiusens: Seems like we should be "all they have plus"20:00
sergiusenstedg, once we add code to support schemas inside snappy we can20:00
tedgAh, okay. Just want to ensure we don't have more keys like this.20:01
tedgShould be able to add them in one place.20:01
tedgThis setup.py is now using 2.5GB of RAM. Not sure when I should call it.20:06
tedgOh my, it wasn't hung.20:08
sergiusenstedg, what are you building?21:04
tedgsergiusens: nova21:04
tedgBut it seems to be hardcoding the path to the python interpreter.21:04
tedgWhich surprised me.21:05
tedgI'll have to play with it more later though. Need to go play the trophy husband for a bit.21:06
pindongahi, I have built a snap package that is failing automated reviews, but some of the checks contradict the docs (eg, snappy-systemd_package_yaml_ports_lp_format)22:55
pindongaanyone around to help me understand how to improve the snap package?22:56

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!