/srv/irclogs.ubuntu.com/2016/01/27/#snappy.txt

elopiokyrofa: there is now https://github.com/ubuntu-core/snapcraft/pull/266/files00:03
=== chihchun_afk is now known as chihchun
nhainesSo here's probably an annoying question.  What's the latest snappy image for a Raspberry Pi 2?07:56
nhainesI'm going to sort of assume it's https://people.canonical.com/~mvo/all-snaps/rpi2-all-snap.img.xz.08:05
fgimenezgood morning08:07
mvonhaines: thats the very latest 16.04 image, its a bit rocky at this point because we are adding lots of new features08:09
fgimenezelopio, hey, still awake!? :) take a look at http://10.55.32.97:8080/log/all08:09
mvonhaines: if you prefer something stable to work with, the 15.04 rpi2 image is also available08:09
nhainesmvo: I just got an RPi2 with a PyGlow module, and was hoping to play around with it.  :)08:09
mvonhaines: http://releases.ubuntu.com/vivid/ubuntu-15.04-snappy-armhf-raspi2.img.xz08:09
fgimenezelopio, "Checking comment on PR #379 for job github-snappy-integration-tests-cloud"08:09
nhainesmvo: is there no 15.10-based image?08:10
elopiofgimenez: yes. Everything seems correct, except that m-o can't connect :(08:10
=== lu is now known as Guest98847
fgimenezelopio, http://stackoverflow.com/questions/8378235/javax-net-ssl-sslexception-java-security-invalidalgorithmparameterexception-th maybe we are missing an ssl store08:10
=== l is now known as Guest57199
nhainesmvo: I'll skip the "why" and ask where I can sort of figure out where to look for up-to-date information.08:11
elopiofgimenez: I installed the ca-certificates-java.08:11
elopiostill no luck.08:11
fgimenezelopio, ok i'll take over from here, thx! and get some rest08:11
mvonhaines: here is a bit of background info on the 16.04 image state https://lists.ubuntu.com/archives/snappy-devel/2016-January/001400.html08:17
mvonhaines: we only have 15.04 and 16.04 currently08:17
elopiofgimenez: ok, I give up now anyway. The old jenkins doesn't have the java certs installed anyway and it works.08:18
fgimenezelopio, maybe something is missing in trusty, i'll investigate. the current images have been built with the new keys, right?08:19
elopiofgimenez: the rolling edge all snaps, yes.08:20
elopiofgimenez: to generate the 15.04 we need snappy-cloud-image not to pass os, kernel and gadget parameters to udf08:20
fgimenezelopio, ok, i'll rebuild that one and create a new floating ip and a testing project in github08:20
elopioI guess we can make that the default is not to pass the parameters.08:20
fgimenezelopio, that's already done iirc, let me check08:21
fgimenezelopio, https://github.com/ubuntu-core/snappy-cloud-image/blob/master/pkg/image/image.go#L9408:21
elopiofgimenez: for some reason that didn't work: http://162.213.35.179:8080/job/create-cloud-image-vivid/1/console08:23
fgimenezelopio, ok i'll check that too08:24
nhainesmvo: gotcha.  Will I be able to upgrade from 15.04 to 16.04, or will that require a reflash?08:35
mvonhaines: it will require a reflash. if you want to avoid that the 16.04 image is probably better but it under heavy development :)08:40
nhainesmvo: finished flashing the 16.04 image, ran 'snappy enable-classic', then 'snappy shell classic', ran 'sudo apt install nethack-console'.  Worked but game not available.  Sad.  :)08:46
nhainesAnyway, reflashing isn't currently a problem, so I'll manage either way.  Thanks for the help.  :)08:50
JamesTaitGood morning all!  Happy Wednesday, and happy Chocolate Cake Day! 🙌 🎂09:44
mvoJamesTait: hey, good morning!09:45
mvoJamesTait: how is the snap.yaml stuff going? I would love to upload the first snaps ;)09:45
JamesTaitThings are moving now I've managed to nail down the requirements.  I expect to have something to land and test on staging today.09:48
MikaelaNow you made me want chocolate cake  :(10:05
ogra_hmm, ubuntu_geek is gone11:27
asackyrofa: sergiusens: wdyt of http://paste.ubuntu.com/14678463/ (and then make plugins that support, honor those global flags)11:33
Sweet5harkutlemming: so I tested the vagrant stuff on two machines and the "stable" image loops on both, so is pretty unusable ...11:33
Sweet5harkutlemming: also both machines are very different (one intel i7 one amd opteron 6272 etc.) ...11:34
ogra_asac, i dont like "+j_count = 0"11:34
asacwhy?11:34
Sweet5harkutlemming: the "edge" image seems to work though, so using that for now ...11:34
asac<=0 -> use auto counrt11:34
asaccount :)11:34
ogra_asac, it should check /proc/cpuinfo and default to the number of cores instead11:34
asacogra_: did you read the code?11:35
ogra_meh11:35
asac:)11:35
ogra_didnt read to the end11:35
asacsearch for it... its doing that11:35
asachehe11:35
ogra_yeah11:35
ogra_my head feels like wrapped in a 2m thick layer of cotton wool11:35
asacwithout that its pretty sleep inducing to build a kernel :)11:35
ogra_(hard to get through doors this way :P )11:35
ogra_yeah, sorry11:36
asachehe11:36
asacyou went to the doc?11:36
asac:)11:36
asacor just bad day/headaches?11:36
didrocksogra_: give it one meter more and you will know my current state :)11:36
ogra_just got up after a 12h flight :)11:36
dholbachdid you guys pick up the ubuflu?11:36
ogra_at least i only cracked my back and didnt catch a cold or ubuflu as usual11:37
dholbachI did /o\11:37
didrocksdholbach: yeah, a little bit11:37
asacpfft... 12h is easy :P11:37
didrocksseb128 as well11:37
asacogra is getting old :)11:37
asaclol11:37
asacwell, get some rest then11:37
ogra_asac, well, over all it was a 17h tour11:37
ogra_but yeah, getting old11:38
asacbetter if you are available strong for less time its better than weak for long time :)11:38
ogra_and i cant take much rest, so much to do11:38
ogra_... generic initrd ... all-snaps in livec-rootfs ... dragonboard images ... aall in the next 10 days11:39
ogra_mvo, so for the initrd, when do you want to merge the two pieces ... during snap creation or earlier ?11:40
ogra_(i assume we want to sign the file inside the snap )11:40
ogra_s/ sign the file inside/ a signed file inside/11:41
* dholbach hugs didrocks, ogra_ and seb12811:41
dholbachhope you're all better soon again11:42
* ogra_ hugs dholbach didrocks and seb128 11:42
* didrocks hugs dholbach, ogra_ and seb128 as well, (sharing germs!)11:42
didrockssame for you :)11:42
* ogra_ puts up a virus scanner to catch the worse bits 11:42
dholbachthis is the infirmary :)11:42
asacbig ubuflu it seems :)11:50
sergiusensasac, set verbose seems very specific to the system (it is different on others)12:02
sergiusensasac, if you create a bug, I'm happy to look into it12:02
asacsergiusens: not sure what you mean12:03
asaclots of build systems have verbose flags to see the real command line12:04
asaclike V=1 etc.12:04
asacsergiusens: look at this for an example with scons.py12:05
asacsergiusens: https://github.com/asac/snapcraft/tree/multithread-and-verbose-infra12:05
sergiusensdholbach, didrocks seb128 get better! And maybe ogra_ too12:05
* Sweet5hark stacks up on phone desinfectors ...12:05
sergiusensdidrocks, I have thought of a probably better way to do pkg-config without changing pkg-config, but by wrapping the call. I'll experiment and propose12:06
dholbachthanks sergiusens12:07
didrocksthanks sergiusens!12:07
ogra_thanks sergiusens :)12:08
didrockssergiusens: ah, not uninteresting, yeah, wrapping would be less instrusive12:08
sergiusensasac, I'm a bit skeptical about this, but we can explore12:08
sergiusensdidrocks, wow, get some rest, thoes double negatives are hard to follow ;-)12:08
didrockssergiusens: ahah, yeah, tireness is twisting my mind! :-)12:09
asacsergiusens: what makes you think?12:09
asacsergiusens: a) it surely needs  to be a global CLI option, no?12:09
ogra_asac, are we philosophical today ?12:09
asacb) bc, you want to be able to toggle this on and off from CLI and not hack on yaml to change behaviour12:10
sergiusensasac, lets start with `snapcraft snap`, does it work with how you did it?12:11
asacsergiusens: oi if you want to talk about how to implement the global option, then yeah lets talk12:12
sergiusensasac, verbose makes sense to be global, the -j could be a yaml thing though (just because of what I mention before); if not you need to expose these globally12:12
pindongasergiusens, hi there... I was told you had some thoughts on the vendoring approach taken here? https://github.com/ubuntu-core/snapcraft/pull/19712:12
asacsergiusens: so thats interesting question. think right now you can only change this if you run build explicitely12:12
asacwhich could be fine if you think about it, but i would prefer if the option woudl be everywhere12:12
asacsergiusens: no -j is not yaml12:12
asacits cli12:12
asacyou want to toggle on/off for debugging etc.12:12
asacat best all are global12:13
asacfelt bad to put it for top level options, but we could do that12:13
asacthought maybe you know a trick how a command would still parse the options12:13
sergiusenspindonga, I'm really not comfortable with vendoring; especially for things that the team is not upstream for12:14
asacsergiusens: anyway, thanks for checking. what i want are those common.XXX functions that have the behaviour as its now for build, just better so it works for snap etc.12:14
asacand default target12:14
pindongasergiusens, so what do you suggest we do? here's the problems I see12:14
asaci can go and make plugisn that support these do it12:15
pindonga1. ssoclient is only provided as a package on xenial, thus depending on this will mean snapcraft will only work in xenial once this change lands12:15
sergiusenspindonga, can't this land in xenial?12:15
sergiusenspindonga, that is fine12:15
sergiusenspindonga, xenial is the only target for snapcraft 2.012:15
sergiusens2.x12:15
pindonga2. the upload also requires the storeapi code, which if not vendored will also require us to package/land click-toolbelt on xenial (same caveat)12:15
pindongaalso, packaging and getting this into ubuntu seems to be taking quite some time12:16
pindongaand I fear we'll miss feature freeze12:16
sergiusenspindonga, landing in ubuntu is part of the workflow though12:16
sergiusenspindonga, maybe ask dholbach for help with packaging, I'm sure he is more than glad to help12:16
pindongasergiusens, ok12:17
pindongabeuno, fyi... looks like we'll have to go the packaging route in the end12:17
dholbachpindonga: what do you need to get into the distro?12:17
pindongadholbach, click-toolbelt12:17
* pindonga checks what the latest status of that was12:18
dholbachr57?12:18
ogra_sergiusens, oh, no LTS backports planned for snapcraft ?12:18
dholbachthere's no new changelog entry in debian/changelog in lp:click-toolbelt?12:18
pindongadholbach, yes, version 0.5.112:18
dholbachis this normally uploaded through auto-landing?12:19
pindongadholbach, this is a new package, hasn't reached the archive yet afaict12:19
dholbachpindonga: so r57 == 0.5.1?12:19
pindongaso no idea what to answer there12:19
pindongayes12:19
dholbachok ; isee12:19
dholbachlet me review it12:19
sergiusensogra_, yes, 16.04 LTS12:20
beunopindonga, sergiusens, I don't understand why it needs to be pacakged?12:20
sergiusensogra_, if it were written in go I'd be more confortable12:20
beunoalso, I wouldn't use the name click-toolbelt if we do12:20
beunoit12:20
sergiusensbeuno, we have no guarantees of maintainership; and it is bad practice to vendor in the archive12:21
pindongabeuno, in that case we'll have to split the storeapi code into a diff project and package that12:21
beunosergiusens, it12:21
pindongaeven more work12:21
NoiZeRHi, is it possible to use *.so files that are compiled on ubuntu lts 14.04 on a raspberry pi to make a snap and use them as a framework?12:21
sergiusensbeuno, if it is a fork; then it is missing tests which was said was not necessary12:21
sergiusensbeuno, this is just proper ubuntu work, nothing really out of the ordinary12:21
beunoI don't understand, upload is a core part of snapcraft12:21
beunowhy would we want to have it separate?12:22
sergiusensbeuno, yeah, the libraries used though, that's different12:22
ogra_sergiusens, hmm, that creates a gap ... most people will only upgrade once 16.04.1 comes out12:22
sergiusensbeuno, well, the conversation started with the fact that there are no tests as it is tested upstream, but if it indeed is a fork, then it is not vendored, but a fork12:22
sergiusensand should have all the required tests12:22
pindongabeuno, sergiusens what's vendored is the store api code (just so we don't have to duplicate the code in both places)12:23
pindongaI could as easily do the latter removing one thing to vendor12:23
beunosergiusens, the old code is being deprecated12:23
pindongathe other (ssoclient) would still be needed via dep (which is already in xenial)12:23
beunowe're keeping it around so the phone can still upload clicks for a little while12:23
beunowe're merging in that code into snapcraft, and killing off the other tool as soon as it's sensible12:24
sergiusensbeuno, as long as you maintain this part of the code12:24
kyrofaGood morning12:24
beunosergiusens, by you, you mean all of us?12:24
beunoI mean, this was suppose to be part of snapcraft12:24
NoiZeRkyrofa hi i have an other question about snappy can you help me out a bit?12:24
beunoand pindonga is pitching in so more things can be done on other bits12:25
kyrofaNoiZeR, sure :)12:25
beunoif there are concerns about code quality, we can surely split it up and land it in chunks12:25
sergiusensbeuno, if we are going to be upstream... https://github.com/ubuntu-core/snapcraft/pull/197/files#diff-175727a8f2ba0ea9e34a671983b5ad2dR3612:25
sergiusensbeuno, that is indeed wrong, why does it say `vendor`?12:25
NoiZeRkyrofa so i uses a Raspberry Pi and i have some libs compiled on a Raspberry Pi with ubuntu 14.04 LTS but is it possible to make a snap of it en use it as a framework?12:26
beunosergiusens, because it hasn't been clear how to pull these things together12:26
NoiZeRkyrofa it is an *.so file12:26
NoiZeRkyrofa thx in advance xD12:26
beunosergiusens, dude, you're leading snapcraft. There was the store upload feature on the roadmap, and I volunteered to get someone to work on that code since we have some domain experts12:27
kyrofaNoiZeR, potentially. Libraries still often need other libraries, which would also need to be included in the .snap12:27
beunoI don't get why you'd want to try keep that split as a separate thing12:27
kyrofaNoiZeR, why are you wanting to make a framework instead of a normal .snap, by the way?12:27
beunosergiusens, if you'd rather develop the feature yourself, pindonga has plenty of other things to work on12:27
sergiusensbeuno, I just said, remove `vendor` add tests and it is fine12:27
beunocool12:27
pindongaok12:27
sergiusensbeuno, I am worried that when snap-ids come all this code will be useless, that's why I am worried12:28
NoiZeRkyrofa im making a system where users can make some application for my application. But they need the Library that is installed on the Raspberry Pi12:28
NoiZeRkyrofa so the developers can make third party modules for the device12:29
beunosergiusens, I understand, we'll need to change quite a few tools around it12:29
beunosergiusens, nessita is currently planning that change12:29
beunosergiusens, I don't think the core of this will change (uploading)12:29
kyrofaNoiZeR, alright. Note that frameworks are more for sharing some sort of service, not sharing libraries. Not to mention frameworks won't exist in 16.04 (they're being replaced with something else)12:29
asacsergiusens: beuno: i feel that some coupling of store and snapcraft is unavoidable with the consequence that there will be changes that need invest on both sides... to ensure things move smoothly, couldnt you couple store and snapcraft through integration tests that run on landings on both sides. in this way if store wants to land changes that break snapcraft they will have to facilitate that the snapcraft change also happens?12:29
beunobut there are plenty of things to add: assertions, name reservations, etc12:30
asacand vv12:30
ogra_NoiZeR, as i understand there are no framework snaps anymore ... but you can create library snaps that your app can consume (but nobody elses but your snaps will be able to use them, they are bound to teh developer)12:30
sergiusenswe have autopackage tests enabled; but the store does not land on ubuntu so it is up to the store to implement those testing hooks12:30
sergiusensasac, ^12:30
kyrofaNoiZeR, your users might be better served by your making some sort of snapcraft plugin to integrate your libs into their .snaps12:31
beunoasac, I don't think we'd be breaking each other in this case, the server might change, and if it does the client has to regardless of where it lives12:31
kyrofaNoiZeR, or just make a build system that makes it easy12:31
asacsergiusens: if snapcraft already tests store interactions through autopackage tests on landings thats fine12:31
ogra_yeah, what kyrofa said12:31
NoiZeRkyrofa, a build system what do you mean with that?12:31
NoiZeRkyrofa can you use snaps into snaps?12:32
asacbeuno: sure, just saying that snapcraft maintainer would feel less reluctant to land more code that couples store to snapcraft if i knew there are tests for the integration running in store landing process too...12:32
kyrofaNoiZeR, is your library built by a build system? e.g. cmake, autotools, etc.?12:32
NoiZeRcmake12:32
asac(just my way to  say, i wouldnt hold back with more coupling just because something will change; we ahve to do it anyway so ratehr think how to make it efficient and not how can we wait longer)12:32
NoiZeRkyrofa cmake and an other lib by qibuild but12:32
kyrofaNoiZeR, cmake is good-- snapcraft supports cmake with a plugin12:32
NoiZeRkyrofa but cmake needed qibuild so i need to install the python library aswel i think12:33
NoiZeRkyrofa but how thats works i don't know. Just i know its needed to be installed with pip install qibuild12:34
beunoasac, sure thing12:34
dholbachpindonga: AFAICS the package is good to go, minor nits would be: the package description could be a bit longer, copyright in d/copyright could be extended to be 2013-2016, and there's no manpage for click-toolbelt - I didn't quite follow the discussion you had above about having a separate package or something.... do you know if everyone's happy with the name click-toolbelt and the placing of all bits in the code?12:35
dholbachfrom a mere packaging POV I'm happy to upload if everything else is settled12:36
pindongadholbach, thanks but it looks like we won't need it after all12:36
pindongawe're moving that code into snapcraft itself12:36
dholbachok...12:36
dholbachlet me know if I can be of any help12:36
kyrofaNoiZeR, okay I'm not familiar with that... hold on12:36
kyrofaNoiZeR, let me do some research into qibuild real quick :)12:36
NoiZeRkyrofa np12:36
NoiZeRkyrofa its with the NAO robot related12:37
kyrofaNoiZeR, but my point is this-- your job would be easier if you focused on building your snap with snapcraft rather than bundling the .snap yourself12:37
kyrofaNoiZeR, since .snaps must include their dependencies12:37
NoiZeRkyrofa ok wait a sec :p So the i need to just add the library to my snaps and then it will work correctly? And can the other snapps uses this library?12:39
kyrofaNoiZeR, each .snap will need to include your lib and its dependencies12:39
kyrofaNoiZeR, this allows your lib to update without potentially breaking the user's snaps before they're ready for it, etc.12:40
NoiZeRkyrofa but every snapp does need to get this library into there snap?12:41
NoiZeRkyrofa or is it possible to share a coulple of library12:41
ogra_NoiZeR, what exactly will your users do ?12:42
ogra_do you expect them to build a snap themselves to make use of your project ?12:42
ogra_(with an emphasis on *build* )12:43
NoiZeRogra_ yes So they need to make some snaps and the other snap need to call them12:43
ogra_cut that sentencs at "and" :)12:43
ogra_*sentence12:44
NoiZeR_ogra so you can it sea like android is my snap and they will install some snapps and that are the apps12:44
ogra_NoiZeR, they need to build something ... why not focus on that then insteasd of binary usage12:44
ogra_provide them the easiest way to do that build step and you win12:44
NoiZeRogra_ what do you mean exactly with that?12:45
ogra_make that build step bundle what they need ...12:45
ogra_into their snap12:45
asacdholbach: https://developer.ubuntu.com/en/snappy/ snappy log is broken here12:45
dholbachwhat is broken?12:46
NoiZeRogra_ so that they always include the needed libraries?12:46
ogra_NoiZeR, exactly12:47
NoiZeRogra_ ok but thats one solution but is there a way in the future to share libraries just like in a normal linux?12:49
ogra_"a normal linux" ?12:49
NoiZeRjust lik i install the C++ library on my Ubuntu 14.04 and i can include the files (the library is en *.so file)12:50
NoiZeRand some python stuff12:51
ogra_you can make shared library snaps today ... but only shared for *your* snaps ....  i.e. they are bound to your account so you cant break our security model ... random dependencies like in other package managers wont be possible12:51
dholbachasac: what is broken on thage page?12:51
kyrofaogra_, and you can't really do that today-- you'll be able to in 16.0412:51
dholbach*that page?12:51
kyrofaNoiZeR, so for 15.04, the real answer to your question is no. Snappy is a bit different from other packaging systems-- you can't share dependencies like that12:52
ogra_kyrofa, heh, right, my brain is still in a different TZ from my body :)12:52
kyrofaogra_, ha! You home now?12:52
ogra_yep12:52
kyrofaogra_, might take you a week man, sorry :P12:52
ogra_but brain is still over the atlantic somewhere ... following up at snail speed12:52
ogra_yeah12:53
NoiZeRogra_ so when i install all the apps as the same user it will work correctly12:53
NoiZeR?12:53
NoiZeRwith shared libaries?12:53
ogra_right, so that you can use the sname libs from a single snap ...12:54
ogra_but others wont be able to consume them12:54
asacdholbach: the snappy logo doesnt load here12:54
asacright on top right12:54
asacits broken image12:54
asacdholbach: do you see it?12:55
dholbachyou're right -I can see the problem too12:55
NoiZeRogra_ the account is that on the snap store that you mean or just a user on the device?12:55
ogra_NoiZeR, snap store12:55
asacdholbach: good :)12:55
dholbachasac: it wasn't clear to me that "snappy log" was supposed to be "snappy logo" :)12:55
asacyeah, typos ftw12:55
ogra_NoiZeR, if any random person would be able to put libs there that any other random person can use, you would completely break the security model of app isolation ...12:56
dholbachasac: https://assets.ubuntu.com/sites/ubuntu/latest/u/img/cloud/tools/snappy/snappy.png was dropped it seems12:56
dholbachdavidcalle: ^ better not to rely on assets12:56
ogra_NoiZeR, and are security wise back to deb/rpm12:56
sergiusenskyrofa, morning btw12:56
kyrofasergiusens, hey there :)12:56
asacdholbach: hmm. will you check it out?12:56
dholbachyes12:57
ogra_NoiZeR, meaning to give *every* person in the app store root on your device ...12:57
asacok let me know. wanted to download it for a rpesentation12:57
asac:")12:57
dholbachasac: fixed12:59
asacneat13:00
ogra_NoiZeR, i.e. in the deb/rpm world *every* package maintainer has full root access to your system ... in snappy every store account has root inside their area of the system, but not in the system itself or in other accounts ... if lib snaps would allow cross account usage that woulod mean the provider of the lib has full root access to all snaps that consume it13:01
NoiZeRok I understand that13:02
asacdholbach: there also exists an orange snappy logo somewhere13:02
asacthat has the word snappy printed13:02
asaccan you browse assets etc. to see if they are there?13:02
ogra_NoiZeR, so in the case where your users have to build something from source anyway, just focus on making that step the best experience by providing a snapcraft plugin that automatically takes care for everything13:03
NoiZeRogra_ kyrofa thx for the help when i have some questions will ask them here13:04
ogra_so the users can concentrate on their code13:04
ogra_great, that is why we are here :)13:04
kyrofaNoiZeR, sounds good :)13:05
kyrofaogra_, when will the dragonboard be here: https://developer.ubuntu.com/en/snappy/start/ ?13:09
ogra_kyrofa, within the next 10 days i hope13:10
kyrofaogra_, awesome! I just received mine yesterday. Do you happen to have a writeup I can follow?13:10
ogra_we need auto-builds first ...13:10
ogra_kyrofa, http://people.canonical.com/~ogra/snappy/dragonboard/README13:11
ogra_if you are on xenial the latest in-archive u-d-f shoudl work btw ... else you need to download the one from that dir13:12
kyrofaogra_, excellent, thank you!13:13
loologra_: hey, did you have to patch snapdragon u-boot for things to work with snappy?13:20
ogra_lool, http://bazaar.launchpad.net/~ogra/+junk/dragonboard/files13:20
ogra_only the config13:20
ogra_there is also a README about all i had to do13:20
loologra_: ok, CONFIG_SUPPORT_RAW_INITRD is what I suspected13:20
loologra_: ah I didn't see the u-boot part in your original dragonboard README hence the Q13:21
ogra_yeah, and support for our uboot.env13:21
Sweet5harkso where is the snappy command docs? I see there is a "snappy man" command than dumps a raw manpage ... but "man" doesnt seem to be available or installable on snappy?13:21
ogra_lool, yep, that work pre-dates snappy work13:21
ogra_lool, afaik even linaro now uses my boot setup ;)13:21
ogra_for normal linux13:21
loologra_: cool13:22
loolurgh CONFIG_SYS_REDUNDAND_ENVIRONMENT13:22
ogra_yeah13:22
loolredundan*d*13:22
ogra_hehe13:22
loolI wonder if it's a germanism13:23
ogra_no idea who originally created the option13:23
sergiusensasac, can you check my comment https://github.com/ubuntu-core/snapcraft/pull/257/files ?13:23
ogra_BBB uses it which is why all other boards inherited it for having a working fw_setenv/printenv (and uboot-go) we use in snappy13:24
loologra_: uboot-go?13:25
ogra_(the option just adds a 4byte header to uboot.env ... not even sure why thats "redundant" in any way)13:25
sergiusensogra_, mvo asac lool so initrd and multiple kernel snaps; mvo is thinking we should create a full initrd on kernel snap build time; that seems hard to do for snaps that we don't control though (and becomes hard to maintain)13:25
loolit's also a problem for lvm, crypto and stuff13:26
ogra_sergiusens, well, i suppose we might want to sign the initrc file inside the snap13:26
ogra_*initrd13:26
loolbesides, isn't there core snappy logic setting up writable bind mounts from initrd?13:26
ogra_so you cant really assemble at snap install time13:26
sergiusensogra_, so if we ever update the OS, we will need to rebuild all the kernel snaps (or someone)13:26
mvosergiusens: I'm not sure I'm thinking this, this is what I'm doing currently, we discussed cat'ing multiple initirds13:26
ogra_lool, we are at the step before running the initrd :)13:26
ogra_sergiusens, ?13:27
ogra_sergiusens, do you plan to ship any parts of that in the os snap ?13:27
* ogra_ wouldnt13:27
sergiusensmvo, oh, sorry, this is based on a quick SCaLE conversation with ogra_13:27
mvoogra_: btw, did you had a chance to test http://people.canonical.com/~mvo/all-snaps/dragon-all-snap.img.xz ?13:27
sergiusensogra_, so when talking to mvo half of the initrd would be in the OS snap; the other half in the kernel13:28
ogra_my vision would be that we have a generic and a modules part ... and that we merge them at kernel snap creation time into one file we can sign13:28
mvosergiusens: no worries13:28
ogra_mvo, not yet ... will do so beofre EOD13:28
mvoogra_: on a grub system we don't extract anything, we load directly from the squashfs13:28
mvoogra_: thanks!13:28
mvoogra_: so extracting and combining would be a step backward :(13:29
ogra_sergiusens, mvo, yeah, i dont think that will fly nicely ... both parts should live in the kernel snap13:29
ogra_the other option would be to merge them from the bootloader on the fly .... but that will likely result in very slow boots and probably also require quite some bootloader hackery13:30
mvomaybe I'm missing something, but why would both parts live in the kernel snap? it seems if one is in the OS snap, that is ok? as long as it has no modules or other kernel dependant stuff ?13:30
ogra_mvo, how would you boot that ?13:30
mvoit seems like grub supports loading multiple initirds?13:30
ogra_does uboot too ?13:30
ogra_i never tried that13:30
mvoogra_: grub has a concept of multiple initirds, so grub would load both, never tried it either, I'm not saying it works, just that that looks like a really good option :)13:31
mvoogra_: uboot is a bit of a problem :(13:31
ogra_that might result in horrid math stuff when you create a gadget to get all the load addresses right13:31
mvoogra_: but on the upside, on uboot we need to extract the files, so we can do merging with uboot13:31
loolmvo: so cat-ing multiple initrds would work technically; is this the solution we want to implement for 16.04? we could13:31
mvoogra_: you mean horrible math for uboot?13:31
ogra_yeah13:32
mvolool: at least on grub it sounds like the best option we have13:32
loolmvo: we tested the multiple initrds yesterday, it works13:32
mvolool: \o/13:32
mvolool: you rock13:32
ogra_where to load the bits so they dont overlaop and can be used as one file from ram etc etc13:32
loolmvo: for u-boot, it's technically possible, but I personally would prefer if we did u-boot -> grub -> multiple initrds13:32
ogra_eeek !13:32
mvoogra_: so for uboot we can extract+conact so that the bootloader does not have to do terrible math13:32
loolor we just cat the initds together on disk since u-boot systems are handled specially anyway13:32
mvolool: yeah, I think the later is the better option13:33
loolthe good thing about u-boot -> grub is that it gives us squashfs again13:33
mvolool: ohhhh13:33
ogra_lool, well, that gets us issues with signed initrds13:33
mvolool: thats very interessting13:33
ogra_lool, why would you do uboot->grup and not just grub ?13:33
loolit's a recent patchset from Linaro (end of Dec) that enables the EFI interface in u-boot for grub-efi to use13:33
mvoogra_: right, so if we have a signed kernel we can not concat, can we? because it has to be signed ? :(13:33
loologra_: it's too much work for us13:33
ogra_having two bootloaders sounds super awful13:33
loolpeople could also do efi/tianocore instead of u-boot, but this is not our battle13:34
ogra_lool, rhight, which is why i'D stay with uboot alone until we can do a full switch to grub13:34
loologra_: there are always tons of firmware before us13:34
loolthe question is whether we want to support both u-boot and grub or only grub everywhere in some way13:34
mvoif we get grubs multiple-initird + direct loading from squashfs blobs, that would be a big plus. even if it means two bootloaders13:34
loolI prefer the latter (grub only interface for snappy)13:34
dholbachkyrofa, sergiusens: are we going to add a new milestone for snappy?13:34
dholbacherr, snapcraft13:35
ogra_it will be slow and error prone to chainload13:35
ogra_not to mention the maintenance burden13:35
ogra_supporting uboot in snappy seems like the smaller amount of work in the end13:36
mvoogra_: error prone in what way? I expect that nothing would ever update uboot, it would just be like a mbr. what kinds of errors will happen? or is it the grub part that is the concern? (sorry for these silly questions, trying to understand the concern better)13:36
sergiusensdholbach, yeah, but given the semantic thing, not sure it should be 2.1 or 2.0.2 depending on what we have :-)13:36
loolI haven't actually tested the grub-efi stuff though13:36
sergiusensdholbach, I'll sort it out today13:36
mvolool: heh :)13:36
dholbachsergiusens: cool13:36
mvoogra_: nevermind, maybe we should have a quick call at some point to discuss the various pros/cons13:38
ogra_mvo, it is simply one layer more than needed ... it will at least be a lot slower to boot ... error prone just in the way of having to make sure the configs keep working together when something changes13:38
mvoogra_: *nod*13:39
mvoogra_: don't get me wrong, I'm not advocating it, just trying to explore the options and the various pluses/minuses13:39
ogra_yeah, i get that13:39
ogra_preferably we'd just have a postinst step in the os and kernel snaps that creates a merged file when either of them gets updated ...13:40
loolfinally found the thread again http://lists.denx.de/pipermail/u-boot/2015-December/239054.html13:41
ogra_but if we sign the file at build time and require signed initrsds that will not work13:41
sergiusensfrom a snapcraft PoV we can do either; a kernel plugin which downloads the generic initrd from the archive which we use, but that brings in update issues if the generic thing changes under us13:41
ogra_well, its all a matter of signed or not signed ... if we dont require the initrd we boot to be signed we can easily merge at any point13:42
mvoogra_: exactly, this is why I like the idea about loading multiple-initirds via the bootloader. but I do get it that its horrid with uboot13:42
ogra_mvo, well, its just a lot more work to do the math ... and it is board specific so porting gets a lot harder13:43
ogra_once you have all the numbers it wont be much harder13:44
ogra_but often porting to a new arm board even finding the right address and RAM sizes for booting with one initrd can be really hard13:44
* mvo nods13:44
ogra_so we double up the hardest bit :)13:45
noizer_kyrofa an other question ps im NoiZeR from before todauy13:46
noizer_*today13:46
kyrofanoizer_, sure13:47
noizer_kyrofa can i mount the snappy sd card in an ubuntu system and then place the *.so files on it. Can the snaps then use the lib?13:48
asacsergiusens: pushed updaetd rev with comment13:51
asacsergiusens: not sure what mvos comment means13:51
asacmvo: you say we should produce the complete initrd in kernel snap?13:51
* asac reads more backlog13:54
asacmvo: so current plan is to make mmodules/firmware only initrd in kernel snap and general one in OS snap and concat them preferable at bootloader13:54
asacthis would be exactly waht we would i fiigure13:54
asacyou are hesitant to that approach?13:55
asacmaybe having a call would help13:55
asac:)13:55
ogra_asac, that approach means that porting is a lot harder13:55
ogra_but in the light of signed initrds that might be the only workable one13:56
ogra_if we dont sign or dont require them signed there are more options (and less harder ones)13:56
asacbootloader requirement yet13:56
asaci think we coudl still merge them on disk13:56
asacfor bootloaders that cant concat load13:57
ogra_not if they need to be signed13:57
asacsure not13:57
asacbut not all platforms will need signing for hardware assertions13:57
ogra_so if we ever want to enforce them signed, merging at load time is the only option13:57
asacsurely i dont want to ship core logic like our general boot code etc. in the kernel snap13:57
asacthats even worse imo :)13:57
asacright. think we want merging at load time13:58
ogra_thats what mvo and lool propose13:58
asacwhat do they propose?13:58
asacmerge at load time? yes +1 on that13:58
ogra_asac, it will just make porting to new arm HW a horror trip for most porters13:58
asacnew arm HW probably gets a recent uboot which should be ok... i am more worried about old hardware13:58
ogra_usually upstreams dont even give you a load address for a single initrd13:59
ogra_it is hard enough to find the right bits and pieces for one13:59
asacsure13:59
asaci am mostly interested if and when we would get the load concat feature for pi2, beagle and dragon13:59
ogra_now you need to figure out two of them and your uboot version needs to be recent enough too13:59
asacif thats soonish, then i am bullish going down that oath13:59
ogra_that would rule aout about half of the boards we have community suported today14:00
asacyes, as i said i would love to also see a on-disk on update merge option14:00
ogra_not every board is mainline or even uses a uboot version from the last 12 months14:00
asacbut its tricky due to how kernel and OS can move independently14:00
ogra_no, thats not the prob14:00
asacsure it is... you need to regen the combination of both14:01
asacso you can fall back14:01
ogra_both can have postinst magic to merge if either changes14:01
asacat least its not trivial14:01
asacyes, but you have to keep the combination of before the update14:01
ogra_but you break the file signature for signed initrd14:01
asacso you cannot just use the kernel or OS path14:01
asacbut have to mix them14:01
asacsure14:01
asacthats just a feature problem, but i think mvo is resistant for the reason i mention14:02
ogra_you have one initrd per installed snap14:02
ogra_(that is ... two kernel snaps, two OS snaps... i.e. 4 initrds)14:02
ogra_there is still the option i wanted to do initially :)14:03
ogra_and simply not merge them at all14:03
ogra_but loop mount and link the contents14:03
ogra_all from inside the generic initrd14:03
dholbachkyrofa: thanks for the reviews14:08
dholbachas far as I could I fixed everything up14:09
kyrofadholbach, awesome! And no problem :) . I'm going back through it again real quick14:10
kyrofadholbach, almost there on the variables14:18
kyrofadholbach, I know it's tough to hit a moving target-- thanks for your efforts!14:18
dholbachkyrofa: no worries - it's great to work with you guys as a team :)14:23
kyrofadholbach, agreed :)14:23
asachmm14:23
asacso is lib/firmware in a versioned subdir or not?14:23
asaci see both things on my desktop14:23
asacbut apparently only old stuff in /lib/firmware/VERSION14:23
ogra_lib/firmware isnt14:23
ogra_it comes from linux-image-extra14:24
ogra_but there is a versioned subdir with the kernel specific FW14:24
asacbut why do i have stuff in unverseioned /lib/firmware?14:24
asachmm14:24
ogra_thats plain FW14:24
asacok so what the kernel installs goes into a versioned dir?14:24
ogra_not bound to the kernel version14:24
ogra_right14:24
ogra_lib/firmware/$kvers14:25
asacnot really14:25
ogra_that has device trees and kernel specific FW files14:25
asacso upstream firmeare_install creawtes stuff14:25
asacwithout version14:25
asacit does for modules14:25
asacso i am confused what goes were14:25
ogra_hmm, thats beyond me then ... afaik generic firmware comes from linux-generic-extra14:25
asaci have something in my kernel firmware_install that is in /lib/firmware14:26
ogra_and specific FW should end up in the versioned dir14:26
ogra_ask the kernel team then14:26
asacapw: where do we put firmware? what goes in unversioned dir and what goes in versioned subdir of /lib/modules?14:26
asacwhen using install_firmware kernel target it installs it withyout any version dir14:26
asacand part of the stuff it puts there is in our unversioned dir e.g. atmsar11.fw14:26
ogra_in lib/modules the only bits you should have in the toplevel are the dependency files from depmod14:27
apwasac, firmware which is in a linux-firmware is in an unversioned directory, stuff that comes out of the kernel source goes in a version specific directory14:27
asacok, do we carry a patch or something against upstream?14:27
apwasac, we supply the path we want it installed to which we put the version number in14:27
asacor is it user space tool outside kernel that knows that location and loads stuff?14:27
apwasac, the debian package version number14:27
* ogra_ wonders what asac is after here 14:28
asacok. well, think for our smnap its fine to have it not versioned as its really independent anyway on FS14:28
asaclets see what happens :)14:28
ogra_asac, well, it really depends on the search paths in the different tools14:29
asacjust stick to whatever firmware_install spits out feels safe14:29
asacright. i was wondering waht does the firmware searching14:29
asacif its part of the kernel then its fine14:29
asacotherwise have to check out14:29
ogra_why would you change what we have ?14:29
asaci am not changing anything14:29
* ogra_ tries to understand your target14:29
asaci am building upstream kernel from source14:29
asacinto a snap14:29
ogra_well, then just use make install with adjusted target dir and be done14:30
asacso what loads the firmware :)?14:30
asaci can do that14:30
asacbut only if it works :) ...14:30
asacneed to conf that whoever knows about search paths14:30
ogra_heh, why wouldnt it14:30
asacis in the OS14:30
asacwell, uif the kernel itself looks for it14:30
asacthen it would probably prefer to have it where it expects it \14:31
asace.g. maybe we patched the kernel or use special config when building the in archive kernels to adjust the search path14:31
ogra_but make install should cover this14:31
ogra_and afaik you can still run any ubuntu with a home built mainline kernel (if your config is fine indeed)14:32
ogra_so userspace shouldnt matter much14:32
asacok so the kernel itself looks for the firmware in the path?14:33
asacthen its all fine without doing anything14:33
* asac keeps it :)14:33
ogra_well, not the kernel but the respective drivers i think14:33
sergiusenselopio, standup?14:34
ogra_just boot it, i'm sure your drivers will complain if they cant find firmware :)14:34
asacbooting is for the weak :P14:34
ogra_lol14:34
asacknowing is better14:34
asaci cant boot it as we still have to figure the initd story above14:34
asaci am just getting ready to whatever there14:34
asacwell i can boot, but not simply by using udf etc.14:35
ogra_you can always use the system-image way :)14:35
kyrofaelopio, you up yet?14:35
asacmy initrd doesnt even have an init right now :)14:36
asacguess showing how to include atmsar11.fw in the initrd is as good as any :)14:36
ogra_for what is that ?14:36
asacno idea :)14:36
ogra_doesnt sound disk specific14:37
asacjust to show how you can use your snapcraft.yaml to select firmeware that goes into your initrd14:37
asacof course not14:37
ogra_and thats all you want from the initrd14:37
asacif you have a better example out of what pops out of x86_64_defconfig let me know14:37
asaclet me give you the list to pick one from14:37
* asac waits for build to finish14:37
* asac thinks he would prefer to have 32 cores14:38
asacogra_: snapcraft-master/examples/kernel$ find parts/kernel/install/lib/firmware/ | pastebinit14:38
asachttp://paste.ubuntu.com/14679351/14:38
asactell me which fw is better to pick as an example :)14:38
asacmaybe the e100 stuff? :)14:39
asacanyway, good for now14:42
asaclool: i got firmware feature now too. whats next?14:42
asacogra_: if you want to play, check out: https://github.com/asac/snapcraft/blob/kbuild-kernel-wip/examples/kernel/snapcraft.yaml14:43
asacbut no init in initrd so dont hope too much :)14:43
loolasac: I'm working on a snapcraft snippet for a partner right now; I want to test the grub-efi patchset on dragonboard next to confirm feasability of multi initrd14:44
asacgotcha14:44
loolbut worst case, we just cat the initrds together on arm14:44
asacright14:44
asacbut mvo hates that idea14:44
loolcat-ing?14:44
asacbut guess thats what he has to get over with14:44
loolit's not different to the copy we're doing today anyway14:44
mvohate is such a strong word :)14:44
loolsince we dont have squashfs loading on arm anyway14:44
mvoif thats the way, thats the way14:44
asaclool: its because of rollback and keeping the rightg initrd combos14:45
asacwith independent kernel and OS snap14:45
mvowe need to keep all the previous cats around though so that we can rollback14:45
mvobut that should be ok14:45
asacits a bit tricky, but guess nothing mvo cant solve14:45
mvoits just more book keeping14:45
loolgenerally we need to agree on what happens to A/B with kernel and OS updates separate14:45
loolI guess OS was never considered part of it14:45
asacguess cat /path/to/os/version1/initrd /path/to/kernel/version2/initrd > boot/initrd-version1-version214:45
asacdang :)14:45
loolbut now that it contributes an initrd snippet, it should trigger an A/B switch, shouldn't it?14:45
mvoasac: exactly14:46
asacyeah we need to track latest OS and latest kernel and previous combo14:46
asacfor the bootloader now14:46
asaclong term i think snappy should have a tx log so you can stepwise go back and see exactly what combo of packages was installed14:46
asac:P14:46
mvoasac: indeed14:46
asacbut thats nerdy i figure without having more advanced usecases14:46
asacbut still would be great to know how snappy system evolved to its current state14:46
mvoasac: I guess we agree that we explore grub multiple initrd and for uboot concat even though that means no signed initrds on uboot(?)14:47
asacmvo: i would like to start with concat on arm yes14:48
asacthe option lool explores would be still nice14:48
asacbut as an option14:48
asacand as a requirement if you want secure boot of course14:48
elopiofgimenez: did you have luck with m-o?14:48
ogra_asac, any reason to not just glob lib/* ?14:48
asacfor those that cant do it we can validate signatures in our concat code of both pieces and put the merge sig next to it14:48
asacogra_: sure, the idea is that you can explicitely pick what you want for the initrd ... into the main snap all that gets installed with firmware_install gets in14:49
asace.g. after initrd the whole list you have in paste is avail14:49
ogra_lool, mvo, asac, why not loop mount an img file with modules and fw and create the right links instead of merging two files14:49
asacwell, we want to leverage the ability to use drivers from initrd14:49
asacto mount the FS14:49
asaci think14:49
ogra_yes14:49
asacif we dont want that we can avoid all of this alltogether14:49
asacif you loop mount the toher initrd you need to have the driver for that place14:50
asacwhich you might not have14:50
asacat that stage14:50
* ogra_ would create a loop mountable file for lib/modules|firmware ... and simply have *all* modules available from the start ... then move mount that during boot to the rootfs lib/14:50
asacwhere do you put that mountable file though?14:51
asacpoint of initrd is taht it gest shovelled into mem magically14:51
asacwithout drivers14:51
ogra_and simply make sure the kernel has vfat and loop mount support (plus whatever fs is used inside the loop file)14:51
ogra_asac, next to the initrd14:51
asacright. if we want to put that requirement up (e.g.t hat you can mount boot without modules)14:51
ogra_into /boot14:51
ogra_which is guaranteed to be vfat14:51
asacthen we can just not do an initrd in kernel snap14:51
ogra_which we always support in our kernel14:51
asacbut we said for now we want to explore if we can do it right14:51
pindongasergiusens, elopio, PR updated --> https://github.com/ubuntu-core/snapcraft/pull/19714:52
asac*shrug*14:52
ogra_i find that more "right" than merging two initrd halves :)14:52
asactwo initrd halves sounds right :P14:52
asachehe14:52
asacyou coudl get both from nfs14:52
ogra_not to me :) ... but i'm not tied to the loop mount idea14:52
asacnicely in uboot14:52
asacor from the cloud of course14:53
ogra_sure14:53
asaci was hoping we woudl do somethign like that14:53
ogra_but you have to do that on bootloader level anyway14:53
mvoor from pxe14:53
asacbut i couldnt get clear enough buy in so i think for now its best to just do it14:53
asacwith modules in initrd14:53
asacand concat14:53
asacand optimize later14:54
ogra_then you can as well just provide a tool that repackst the initrd on the fly14:54
ogra_that would be way less complex and effectively the same as merging the files14:54
asacnot so sure... i cant figure what that would mean right now, but we have understood the merge in bootloader approach as well as merge on disk14:55
asacso lets unblock and do it :)14:56
asacand its only one mvo patch away14:56
asacand of course we have to produce an indep initrd on builders somewhat14:56
asacbut i think lool/mvo know what to do there14:56
fgimenezelopio, yes, it seem to be working now, after update-ca-certificates -f14:56
fgimenezelopio, i'm finishing some tests and then i'll put this into the ubuntucore/jenkins-ubuntu container, so that the next deploy will include it14:57
asacelopio: fgimenez: did you guys think about having good tests for our kernel auto rolblack feature?14:59
asacthink would be key to have a strong stoory there14:59
asaclike having a bunch of instrucmented kernel modules taht make stuff crash or loop infinite in certain ways at certain times of boot process and ensure that our auto rollback never breaks14:59
mvoasac: we have tests for this in the integration suite that break the kernel snap14:59
mvoasac: but more cases of course is always better15:00
asacmvo: with reboots?15:00
mvoasac: yes15:00
asaci mean does that run on real hardware etc.?15:00
ogra_asac, i can do the generic initrd within the next 2h ... thats just copy paste from code we use elsewhere15:00
asacVM?15:00
mvoasac: on kvm, but elopio was working on tests on the BBB too for the kernel/os (or was it rpi2?)15:00
=== opmodoro is now known as lcapra
ogra_but i think having to define "modules and firmware that go into the kernel" vs "just having all of it available at any time" is the wrong approach15:00
asacmvo: thats cool if we have... how do we break the kernel? i think we need 3 cases: kernel does not exist at all, kernel panics early but does not reboot, kernel panics but ddoes reboot, kernel just gets stuck forever never reaching health check15:00
ogra_*that go into the initrd15:01
asacok 4:)15:01
ogra_... sorry15:01
asacogra_: we dont define what goes into kernel15:01
asacwe define what goes into kernel at initrd stage15:01
asacafter that all is available of course15:01
asacjust in case thats not understood15:01
ogra_asac, yes, i corrected myself above15:01
ogra_i think thats wrong15:01
asackk15:01
mvoasac: yeah, I'm not sure which of those, I have a look now15:02
ogra_all modules should always be available ... my loop mount way would provide that15:02
asacyou would need the kernel snap on th vfat15:02
asacand just mount it15:02
asaci dont think mvo wants that15:02
ogra_the modules img15:02
asacwell, its close to the same15:03
ogra_not the kernel snap :)15:03
mvowell, I see a gap here when it comes to e.g. PXE booting15:03
asaccan just be the kernel snap i figure15:03
asacright15:03
ogra_mvo, why ?15:03
mvoto support that our kernel would have to have all network drivers build-in15:03
ogra_its just amnnother file you have to add to the PXE config15:03
ogra_PXE already has to load initrd and kernel binaries15:04
ogra_so you add a third file it loads to give you the modules img15:04
ogra_the rest happens from the initrd15:04
mvoand where will that file be stored, i.e.. how does the kernel/initrd access it?15:04
ogra_in ram ?15:05
mvoon the computer15:05
mvoI get that15:05
ogra_8sure15:05
mvoI mean, how does the kernel/initird access it? how is it availalbe to it?15:05
ogra_how is the initrd available to the kernel in that setup ?15:05
ogra_(you load it to ram and uncompress from there ... not much different to loop mount it from there if your kernel has the necessary bits included (which is has))15:06
mvoogra_: I know how inird works, but I do not know how you can load an image file into ram and tell the kernel to loop mount it from ram, that part is what I'm missing15:07
ogra_me too, but i shouldnt be hard to figure out15:08
ogra_PXE surely has the mechanics to get us that15:08
ogra_(the load into ram part)15:08
mvoogra_: ok, it sounds a lot like a second initird to me :)15:08
ogra_mvo, except that it is not ... because it is the same modules.img your booted system uses15:08
mvoogra_: the load to ram part I'm not worried about, sure, pxe can do that. the "how does the kernel get it from ram and mount it part" is what I'm not sure about. I'm sure its possible, I wonder it its straightforward15:09
ogra_a second initrd is essentially what we do today (selection of modules included) just chopped into two parts15:09
mvoogra_: right, the model you propose is different, but conceptually it sounds a lot like a initird to me15:09
ogra_what i'm proposing is to drop the duplication of modules altogether15:09
ogra_and re-use the modules.img on the booted system15:10
mvoogra_: I think we agree on that15:10
mvoogra_: oh, sorry, now I see what you mean15:10
ogra_so you dont have to define firmware or modules in the snapcraft.yaml like asac currently does15:10
ogra_this model is easy to implement in the vfat case ... but yeah, it will need some research and coding for PXE ... in the end though, PXE should give us the img content *somewhere*15:12
asacmvo: can you tell me what to do in the end?15:12
mvoogra_: its a very interessting idea. for pxe there are some more things to consider, the modules dir is ~200mb big, so a netboot would have to download that and put it into ram.15:12
asaci think what ogra is proposing needs more thinking15:13
asacbut thats just me15:13
asacif you feel comfortable to do that just tell me what the kernel snap should spit out/include\15:13
asacand i can replace initrd with that15:13
asacinitrd-modules-only that is15:13
ogra_mvo, yeah, one would hope that someone had thought about it by today, given i promote that setup since the last orlando UDS :P15:14
mvoasac: well, the most simple way forward is that we require the kernel snap to be able to boot without loading modules. this way we can have the os snap provide the initird and we are unblocked and can research more options15:14
asacif we do it we shouldnt spit out a separate modules.img15:14
asacbut just loop mount the kernel snap though15:14
asacmvo: yes, but note that this is not true for the ubuntu packages15:14
* ogra_ wanted that setup for normal ubuntu installs since ages :P15:14
asacso we deviate from ubuntu15:14
asacfor the snapcraft built ones15:14
mvoogra_: you mean the pxe case? or the how-to-get-from-ram-inito-a-mounted-dir thing?15:14
asacthat was another reason i didint like to do that in this step15:14
ogra_mvo, the modules.img15:15
ogra_in general15:15
ogra_i had a long BOF session at the last orlando UDS where iheavily promoted it ... sadly it never happpened15:15
mvoasac: right, I think this only works if we do the same to our kernel, i.e. move squashfs in instead of having it as a module15:15
mvoogra_: was anyone there against it?15:15
mvoogra_: or did it just not happen15:15
mvo?15:15
* ogra_ has a hard time to type ... the cats are so happy i'm back that i barely am allowed to use the kbd today 15:16
ogra_mvo, the latter ...15:16
mvook15:16
ogra_even the kernel team was interested back then15:16
asacmvo: i dont think our kernel te4am will put that requirement up15:17
ogra_after a year or so i stopped pushing for it though ... and never got to it myself ... the idea is ages old though and could have been researched by today ... including PXE15:17
asacthey want nfs boot and friends15:17
asachence my hesitation15:17
asacnfs using crazy wifi modules15:17
asacetc.15:17
asacit would mean to include almost all network and disk drivers in the kernel statically15:17
asace.g. no-fly15:17
ogra_asac, well15:18
mvoasac: right, multiple initird is my best answer then15:18
asacbasically all that is in current initrd15:18
ogra_asac, today we include almost all network modules in the initrd15:18
asacyes15:18
asacall those would then go into the kernel statically15:18
asacso its huge15:18
ogra_you only move it around15:18
asacno15:18
asacif its module they dont get loade15:18
asacd15:18
ogra_sure15:18
asacif its static its in the kernel all time15:18
ogra_it gets loaded into ram with the initrd.img ... it wont be used later indeed15:19
asace.g. only what is needed vs all in ram15:19
asacyep15:19
ogra_but if you load a 19M kernel.img or a 19M initrd.img doesnt make much difference15:19
ogra_both is bad15:19
kyrofasergiusens, this was our previous standup: "So do you know anything about that?" "No, we'll have to ask sergio." "Okay, have a good day!"15:19
sergiusenspindonga, I'll look now15:19
sergiusenskyrofa, lol15:20
* sergiusens reads the full backlog first15:20
mvoogra_: the initird is freeed again once the boot is complete, no?15:20
asacyes15:20
ogra_mvo, yes, indeed ...15:20
kyrofasergiusens, so it's a good thing ;)15:20
ogra_but the loading will be the same, no matter where your modules live15:20
ogra_s/modules/drivers/15:21
asacsure, but the mem waste will be something i am sure some folks in kernel/foundations/server team will not take time to even consider to consider15:21
elopioasac: on our suite we have a more or less generic way to break every snap. It would be great if you help us extend it with common ways to break a kernel update.15:21
ogra_asac, there is no mem waste ... thats what i'm saying15:21
asacelopio: right, see the four classes of issues i mentioned avbove15:21
ogra_at most there is a move of the existing waste15:22
elopioasac: and it currently works on rpi and kvm. And it will work on bbb as soon as its kernel works again.15:22
asacif we have one instrumentation for each of that i would feel pretty good to start with15:22
* elopio reads back.15:22
asacelopio: think it was somewhere close to where i highlighted you :)15:22
* asac realizes lots of lines were said here after15:23
mvoasac: I think we have break systemd and break kernel, probably more15:23
mvoasac: fgimenez and elopio did a really good job here, we even found kernel bugs this way :)15:23
asacright, kernel has subclasses that i mentioned15:23
mvoasac: yes, sorry for just glossing over this15:23
ogra_there is a prob when the kernl doesnt actually panic though15:26
ogra_and just hangs15:26
mvoyep15:26
* ogra_ had that during dragonboard porting15:27
mvoat least in this scenaro we get back to a good state once you power cycle15:27
elopioasac: https://github.com/ubuntu-core/snappy/tree/master/integration-tests/tests Look at the ones starting with "failover". We have loops, crashes and empty files.15:27
elopioSadly all the ones that touch the kernel have open bugs :)15:27
ogra_yeah15:27
asacelopio: ok so we have all the cases i metioned? great!15:28
asacelopio: now i would like to use those to make a demo out oof it (e.g. so folks can experience this themselves))15:28
asacis that easy?15:28
asacok calls now15:28
asacttyl15:28
elopioasac: it is. go run ./integration-tests/main.go -filter failoverSuite15:29
fgimenezasac, if you have a running vm, "go run ./integration-tests/main.go -ip <vm_ip> -port <vm_port> -filter failoverSuite" makes clearer the failover working15:31
=== l__ is now known as lcapra1
pindongasergiusens, thanks for the review... sorry I missed those vendor bits there, just pushed the fixes16:10
sergiusenspindonga, no worries; I'm doing live testing now for the final bits16:16
sergiusenselopio, help with figuring this out would be appreciated btw https://travis-ci.org/ubuntu-core/snapcraft/jobs/105199856 :-)16:30
elopiosergiusens: merge with this one: https://github.com/ubuntu-core/snapcraft/pull/26616:32
Sweet5harksooo, how do I strace in snappy?16:43
ogra_by using the snappy-debug snap i guess16:43
Sweet5harkogra_: how?16:44
ogra_it should ship an strace binary16:45
qenghoAre there any snappy images that take the new package format?16:45
* ogra_ doesnt know "how" though ... in the past i used th hack up the wrapper script for such stuff ... but that isnt possible in the new world 16:45
ogra_qengho, yeah http://people.canonical.com/~mvo/all-snaps/16:46
Sweet5harkogra_: sudo snappy install snappy-debug && strace yields comman not found16:46
Sweet5harkand 'find /apps/ -name strace' doesnt enlight me either16:48
ogra_well, it would be snappy-debug.strace ... but yeah, thats not there16:49
ogra_so ignore me then (i dont use snappy-debug much anyway, someone who does might be better to answer this )16:50
kyrofaogra_, yeah, snappy-debug only includes the security tool right now16:53
ogra_yeah16:54
sergiusensogra_, I would argue that one would enter the classic dimension and attach to a pid16:56
sergiusensif more than attaching is required, I am not sure how to do this then :-)16:56
sergiusensexcept by using the binary directly16:56
sergiusenssince snappy-debug does not provide strace itself16:57
kyrofaSweet5hark, any chance you're running an all-snaps image?16:58
kyrofaqengho, can you define "new"? i.e. squashfs-based?16:58
NoiZeR_Hi how can i deploy my snap that i made on Ubuntu 14.04 LTS deploy on my snappy OS16:59
ogra_sergiusens, thats a pretty good idea .... we should have documentation for such stuff :)16:59
qenghokyrofa: indeed.16:59
kyrofaqengho, yes, but it's under heavy development right now. If you're okay with that, check out http://people.canonical.com/~mvo/all-snaps/16:59
ogra_NoiZeR_, i usually scp it over and then use sudo snappy install --allow-unauthenticated /path/to/snap16:59
ogra_scp it to /tmp or /home/ubuntu17:00
kyrofaNoiZeR_, or use snappy-remote17:00
ogra_yeah, if that still works17:00
kyrofaogra_, I don't recommend scping to /tmp as it's tmpfs right now.  You need that space to verify signatures :P17:00
kyrofaogra_, unless you're on rolling/all-snaps I guess17:00
NoiZeR_i try to scp it17:01
Sweet5harkkyrofa: please elaborate? I run the edge image in vagrant on amd64 ....17:01
Sweet5harkkyrofa: (because the "stable" image was looping on two machines for me)17:02
kyrofaSweet5hark, if you're willing to be even more cutting-edge, use the amd64 image out of http://people.canonical.com/~mvo/all-snaps/ and you can use the "Classic Dimension." Are you familiar with that at all?17:02
kyrofaSweet5hark, then you can do as sergiusens recommended17:02
kyrofawith strace17:02
Sweet5harkkyrofa: nope, im not familiar with any of this ;)17:02
kyrofaSweet5hark, good, no problem!17:02
ogra_kyrofa, ah, indeed, depends how big your snap is :) (i usually use ~/ anyway)17:03
kyrofaSweet5hark, if you use one of those images (which will become rolling eventually), you can use the classic dimension with snappy enable-classic, which allows you to use .debs and stuff. Good for prototyping17:03
sergiusensogra_, well https://github.com/ubuntu-core/snapcraft/blob/master/docs/debug.md17:03
sergiusensthank dholbach17:03
kyrofaogra_, yeah, true enough17:04
NoiZeR_Wtf where can i drop files on snappy17:04
kyrofaNoiZeR_, you mean with scp?17:04
NoiZeR_i tried in /temp but he sais read only file system17:04
NoiZeR_yes17:04
kyrofaNoiZeR_, try the ubuntu user's home directory17:05
NoiZeR_ok i pushed it on /home/ubuntu17:06
NoiZeR_thx17:06
Sweet5harkkyrofa: how do i best deploy that all-snaps image in vagrant/virtualbox?17:06
kyrofaSweet5hark, for vbox I suggest just downloading that amd64 image and converting it into a vdi17:07
* dholbach hugs sergiusens and kyrofa... and jdstrand! :)17:07
* ogra_ thanks dholbach 17:07
dholbachI just extracted and updated the information from the the appdev manual :)17:07
dholbachthanks for the reviews!17:07
kyrofadholbach, which is a magical mythical document17:08
kyrofaNice to get that stuff out there17:08
dholbachI hope it will very soon be just part of the realm of magic, myths and fantasies :-)17:08
kyrofaSweet5hark, however, I'm not familiar enough with vagrant to help you much there, sorry :(17:08
kyrofadholbach, ha!17:09
dholbachall right my friends - I call it a day - see you all tomorrow! :)17:09
kyrofaSeeya dholbach :)17:09
kyrofaSweet5hark, does what I suggest for vbox make sense though?17:10
sergiusensoh vagrant17:12
sergiusensmvo, we might want to update utlemming on how to get the latest rolling stuff17:12
Sweet5harkkyrofa: I hope so? I created a libreoffice snap with snapcraft, set up an snappy env with vagrant/edge, installed the snap and with some intermediate LD_LIBRARY_PATH hacking foo got it to at least spit out an answer to "libreoffice --help" ...17:14
kyrofaSweet5hark, ooo!17:14
sergiusenskyrofa, ooo feels like open office, this is libre office being discussed :-P17:15
* kyrofa rolls his eyes17:15
Sweet5harkkyrofa: everything beyond that ("libreoffice --cat" "libreoffice --convert-to pdf" for now) though returns RC 77 suggesting that libreoffice thinks it needs to restart and setup a profile in ~/.config/libreoffice or somesuch ...17:16
kyrofaSweet5hark, are you getting apparmor denials?17:17
Sweet5harkkyrofa: and now I want to know about the nature of "somesuch" -- so strace and tools like that to see what libreoffice is doing would be helpful ...17:17
Sweet5harkkyrofa: oh, ahem *cough*17:18
kyrofaSweet5hark, approximately a million of them, perhaps?17:18
Sweet5harka minor detail: Im sudoing/running as root for now because .... #YOLO17:18
Sweet5harkkyrofa: were should I check for apparmor denials17:19
Sweet5hark?17:19
ogra_Sweet5hark, you might want to tell libO to search for the profile under SNAP_APP_USER_DATA_PATH instead17:19
kyrofaSweet5hark, those are logged to syslog, but try running snappy-debug. First install it with `snappy install snappy-debug`17:19
kyrofaThen run it with `snappy-debug.security scanlog` (use sudo if you're not su'd)17:20
kyrofaSweet5hark, it should spit all sorts of recommendations if you're getting denials. Look through there for the config files17:20
ogra_Sweet5hark, well, you really dont want to sudo an app ... it will try to write stuff to /root most likely17:21
kyrofaogra_, not right now-- sudo still gets $HOME->/home/ubuntu17:22
ogra_instead you want the right app wrappers to adjust paths and such for a normal user17:22
ogra_your app will actually run as root anyway17:22
ogra_kyrofa, weven after the command wrapper fired up as root and dumped all of env ?17:22
sergiusenskyrofa, fyi I did this instead of the manual m https://github.com/ubuntu-core/snapcraft/pull/26817:22
kyrofaogra_, indeed, $HOME is real uid, not effective17:23
ogra_ah, cool17:23
kyrofaogra_, so if you su, then what you say applies17:23
ogra_right17:23
sergiusensogra_, don't spread old var names :-)17:23
Sweet5harkogra_:  Im happy when libreoffice runs as root for a start. once it does that, Ill care about the rest.17:23
ogra_sergiusens, shhh17:23
ogra_sergiusens, i simply dont have the new ones in my head yet :P17:24
NoiZeR_kyrofa so i made a snap with this yaml file name: firstsnap17:24
NoiZeR_version: 0.117:24
NoiZeR_# The vendor for the snap (replace 'Vendor <email@example.com>')17:24
NoiZeR_vendor: Vendor <email@example.com>17:24
NoiZeR_summary: test17:24
NoiZeR_description: test17:24
NoiZeR_icon:  icon.jpg17:24
NoiZeR_parts:17:24
NoiZeR_    cam:17:24
NoiZeR_        plugin: python217:24
NoiZeR_        source: run.py17:24
ogra_eek !17:24
NoiZeR_but how can i run it?17:24
sergiusensNoiZeR_, use paste.ubuntu.com please17:24
kyrofaNoiZeR_, ugh, please pastbin17:24
* ogra_ points to paste.ubuntu.com17:24
NoiZeR_kyrofa ok xD17:24
ogra_NoiZeR_, not at all ...17:24
ogra_you would have to define an "app" entry for that17:24
ogra_to either have it run as a service or as executable app17:24
sergiusensthis is not the first time I see someone trying to put a file under source, not sure how to make it clear it is not the intended way17:25
kyrofaogra_, I think it's still safe to assume 15.04 here, no?17:25
sergiusenswe need a better error message there17:25
ogra_kyrofa, well, it is probably safer to ask nowadays :)17:25
kyrofaogra_, touche17:25
NoiZeR_ogra_ ok but what does i need to change then?17:25
ogra_rolling is actually starting to get semi usable now :)17:25
NoiZeR_http://pastebin.com/ycB5RGGP17:25
Sweet5harkkyrofa: sooo, I ran "sudo snappy-debug.security scanlog" in one shell, which provides no output. then I ran my command in the other shell -- still no output.17:26
NoiZeR_ogra_ kyrofa here is the pastbin17:26
ogra_NoiZeR_, look at the snapcraft-examples ... there should also be apps among them17:26
kyrofaSweet5hark, huh, no denials eh? Check syslog just for sanity's sake real quick?17:26
Sweet5harkkyrofa: nothing screaming at me from syslog -- last line is a cronjob note17:27
sergiusenskyrofa, if Sweet5hark is running unconfined, that may be the issue17:27
kyrofasergiusens, oh, good point. Sweet5hark are you unconfined?17:27
* sergiusens reboots and will lose all backlogs; yay irc with no backend to support the backend17:28
Sweet5harkkyrofa: elaborate?17:28
kyrofasergiusens, you need a bouncer17:28
sergiusenskyrofa, that is the backend to support the backend ;-)17:28
kyrofasergiusens, :P17:28
kyrofaSweet5hark, have you done anything special regarding security policies?17:29
Sweet5harkkyrofa: nope17:29
kyrofaSweet5hark, okay, then you're just not getting denials. That's okay, but I do agree then-- you need to resort to strace17:29
kyrofaSweet5hark, in which case I suggest you download that amd64 all snaps image17:30
kyrofaSweet5hark, do you know how to get that working in vbox?17:30
ogra_why not kvm ?17:33
kyrofaogra_, he asked for vbox17:33
ogra_(saves you from converting the img)17:33
ogra_NoiZeR_, look at the snapcraft.yaml of the godd example for an executable app ... i think thats the most simple thing you can do17:36
ogra_for a service look at the snapcraft.yaml of the shout example17:37
ogra_you want the "apps:" entries17:37
* kyrofa thinks cmake printing 100% more than once is an oxymoronic feature designed specifically to tease17:39
pindongasergiusens, pushed fixes... didn't do anything about upload progress bar yet... I think that could be added later in a separate PR (as an improvement on top of this)18:01
sergiusenspindonga, sounds good18:01
ogra_ooooh !18:01
ogra_http://www.cnx-software.com/2016/01/27/orange-pi-one-quad-core-arm-linux-development-board-launched-for-9-99/18:01
ogra_someone needs to do a snappy image for that !18:02
kyrofasergiusens, does snapcraft 2.0 no longer have a force?18:05
sergiusenskyrofa, we removed it to do it properly, remember?18:06
ogra_kyrofa, i think sergiusens just forgot to build it with the --luke option :P18:06
sergiusenskyrofa, as in, we still need to get together and design it18:06
kyrofaogra_, groan18:06
sergiusenshah18:06
kyrofasergiusens, indeed, just double-checking. I was just taking a look at bug #147790418:07
ubottubug 1477904 in Snapcraft "snapcraft.yaml needs to make all stages dirty (was: the can't add file without recreating entire package)" [High,Triaged] https://launchpad.net/bugs/147790418:07
sergiusenskyrofa, ah, you want to tackle that one? It is almost like auto-force :-P18:09
kyrofasergiusens, just marking steps dirty via modification times is easy... but is that the best way? If they just add one little unrelated bit to this copy plugin part down there, now I have to re-pull and build the linux kernel part I have at the top?18:10
kyrofasergiusens, can we be smarter about this?18:11
sergiusenskyrofa, do you want to look at this one today? :-)18:12
sergiusensif so, hangout time18:12
kyrofasergiusens, there are others I can look at as well-- I imagine you're a bit swamped18:13
sergiusenskyrofa, I feel swamped, not sure that I am swamped :-P Yeah, better tomorrow as it is quite complex; I've been thinking about splitting into three parts; but better do it tomorrow, yeah18:15
kyrofasergiusens, yeah, sounds good :)18:15
kyrofasergiusens, we should start having planning meetings every once in a while18:16
kyrofasergiusens, this issue, --force, plugin api improvements, etc18:17
sergiusenskyrofa, yeah, together with our bug triaging meetings ;-)18:17
sergiusensheh18:17
sergiusenslet me set that up18:17
kyrofasergiusens, yeah, though I feel like we've gotten better about that18:18
sergiusenskyrofa, so friday it is18:20
kyrofasergiusens, excellent, thank you :)18:21
pindongasergiusens, thanks so much for approving that PR!19:05
sergiusensnp, looked good, did what it said it would do; so thanks!19:05
mvosergiusens: yes, I was hoping to have ud-f- in the archive19:14
mvosergiusens: codereview *cough* ;)19:14
sergiusensmvo, didn't I review already? In any case, I think simplestreams is on trusty19:32
sergiusenselopio, ideas http://paste.ubuntu.com/14682191/ ?20:19
elopiosergiusens: is that local?20:28
elopioxenial?20:29
=== chihchun is now known as chihchun_afk
elopiosergiusens: I had a similar problem, it didn't find fixtures. I worked it around with PYTHONPATH=/usr/lib/python3/dist-packages ./runtests.sh unit20:34
elopioto me it seemed like python3.5 crazyness.20:34
sergiusenselopio, k, yeah local on xenial20:42
sergiusenselopio, kyrofa aside from requesting an integration tests, comments appreciated https://github.com/ubuntu-core/snapcraft/pull/270/files20:55
kyrofasergiusens, awesome!20:56
sergiusenskyrofa, elopio and now we have an integration test21:51

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!