/srv/irclogs.ubuntu.com/2015/10/22/#snappy.txt

=== chihchun_afk is now known as chihchun
=== chihchun is now known as chihchun_afk
fgimenezgood morning07:05
dholbachgood morning07:09
mvoogra_: good morning, it looks like we have dnsmasq and tss both with uid 109 in the image right now, could you have a look please? in wily, not sure about stable, just noticed on wily07:10
mvohey dholbach07:10
mvofgimenez: good morning07:10
dholbachhey mvo07:10
fgimenezhey mvo, dholbach :) mvo, do we have a rc image?07:11
mvofgimenez: not yet, sorry, I'm working on it now, Chipaca almost fixed the network bug (which is great)07:11
fgimenezmvo, awesome! ok np07:12
dholbachhola fgimenez :)07:12
davidcalleMorning o/07:15
clobranobuongiorno :)07:51
=== chihchun_afk is now known as chihchun
JamesTaitGood morning all; happy Thursday, and happy Wily Werewolf welease day! 😃08:35
dasjoeJamesTait always seems like the most enthusiastic morning person08:42
JamesTaitIt's all a charade.  I'm a grumpy old curmudgeon really. 😉08:47
Chipacamvo: do you have a branch of almost-next-15.04?08:59
mvoChipaca: 15.04 is up-to-date for the release, I think the one outstanding issue is the ordering09:00
Chipacamvo: and is the latest edge built on that?09:00
Chipacajust to minimise the delta i'm working with09:00
mvoChipaca: yes, well, I can build a new image now09:00
Chipacamvo: i mean, will u-d-f build it with that09:01
Chipacaor is that what you offered to build?09:01
mvoChipaca: biulding a new amd64 15.04/edge image with todays lp:snappy/15.04 branch so that you have something super close to the stable release09:01
mvoChipaca: does that help you?09:01
Chipacamvo: indeedly it does :)09:02
Chipacaand i'll branch that and propose a branch on that09:02
mvoChipaca: thanks, you can also work on trunk right now, we have no delta and I inspected the changes (of course) before merging trunk to 15.04 and it looks very sane (famous last words!)09:04
Chipacaah! perfect then09:04
mvoChipaca: I triggered the image build, new 15.04/edge image should be ready in ~30min or so09:04
mvoChipaca: image build09:13
mvoChipaca: now it just needs to get imported09:13
Chipacamvo: it seems there was a 15.04/edge built overnight09:14
Chipacamvo: working with that for now09:14
mvoyeah09:14
mvosure, that will be fine as well, the delta is small09:14
ogra_mvo, whats bad about that ? (tss)09:18
ogra_and yes, i seeded nss3 for a test because it showed up on some list from olli09:18
mvoogra_: two usernames with the same uids09:18
ogra_uh ?09:18
ogra_thats not possible09:18
mvoogra_: yeah, olli asked me to revert that seeding and he said he will follow up09:19
mvoogra_: let me double check about the uid, maybe its something local09:19
ogra_right, as i said, it was for testing (and obviously worth it :) )09:19
ogra_/etc/passwd is readonly09:19
ogra_and checked during build09:19
ogra_argh09:20
ogra_so no, tss is a requirement for the rootfs encryption stuff09:21
ogra_not related to my nss3 seeding09:21
ogra_and yes, seems thats my bug :(09:21
ogra_(didnt check enough that the tss uid is not already there)09:21
* ogra_ bumps the UID to 11009:22
mvoogra_: thanks, no worries. is this 15.04 too?09:25
ogra_yes09:25
ogra_well, let me check :P09:25
mvook, please let me know once the fix is upload and I trigger a new amd64 image09:26
mvothere is also a fix from Chipaca pending, so no worries we are not golden anyway yet :)09:26
ogra_mvo, not in stable .... phew09:27
mvoyay09:27
ogra_now i wonder why we dont have dnsmasq there09:28
ogra_oh, because ubuntu-fan only landed in rolling back then09:28
ogra_(on purpose)09:28
ogra_that was before all the crazy reqs. came in, we shoudl ask if we need fan on stable too now09:28
ogra_uploaded to the PPA09:33
Chipacahttps://code.launchpad.net/~chipaca/snappy/firstboot-ordering/+merge/275244 is ready for review gents09:33
Chipacafgimenez: if you could try this ^ wrt firstboot bringing up eth0, that'd be cool09:34
fgimenezChipaca, ok on it09:38
Chipacafgimenez: much appreciated!09:39
Chipacafgimenez: i've fake-firstbooted ~20 times and it worked every time, fwiw09:40
ogra_would be nice to know if this by chance also fixes the BBB09:44
ogra_seems racy enough to be possible :)09:44
ogra_/join #ubuntu-release-party09:46
ogra_oops :P09:46
mvoChipaca: \o/ once you fix lands we should have a golden image09:56
Chipacaogra_: this should be less racy than before10:04
Chipacaogra_: so maybe :)10:04
fgimenezChipaca, works great on kvm \o/10:09
mvoChipaca: (I said it before, but I say it again) \o/10:10
Chipacafgimenez: and ... bbb? :)10:11
fgimenezChipaca, ogra i'll try bbb now10:11
Chipacawhee :)10:11
fgimenezChipaca, yep :)10:11
Chipacafgimenez: could you put a +1 on the branch based on the success of kvm tests?10:14
fgimenezChipaca, sure10:14
Chipacafgimenez: even if it doesn't fix bbb, it fixes amd64, so we can build/release that at least10:15
fgimenezChipaca, there's one error in the test suite on kvm, but not related to eth0 on first boot "Error: open /etc/default/watchdog.wvUpOFylUogg: read-only file system"10:21
Chipacahm!10:21
Chipacaeeeep10:22
Chipacaogra_: you around?10:22
Chipacamvo: this might be bad10:22
ogra_what ?!?10:22
Chipacaogra_: we have just /etc/default/watchdog as rw, not the directory itself?10:22
ogra_wh would anything write to it on boot ?10:22
Chipacanot boot, this is the integration test suite10:23
Chipacatesting setting watchdog10:23
ogra_yeah, makin the directory writable would kind of defeat the purpose10:23
mvouh, atomicWrite ?10:23
ogra_yeah10:23
ogra_needs special hacks10:23
ogra_(like hostname)10:23
Chipacamvo: yes10:23
mvobäää10:23
ogra_yup10:23
Chipacamea culpa, i broke that10:24
mvowell, its the right thing to do :/10:24
ogra_oh is that a function you control (doing the atomic write ?=10:24
Chipacayes10:24
Chipacawell10:24
Chipacaogra_: it used to not do an atomic write at all10:25
Chipacaogra_: not even a sync10:25
Chipacamaybe the right thing to do is to edit it in writable?10:25
ogra_well, if we want to keep it we need to handle the file like the others in /etc/writable10:25
Chipacawould that work?10:25
ogra_i.e. copy it over at image build time and create a link10:25
Chipacai meant edit /writable/system-data/etc/default/watchdog directly10:27
Chipacamvo: ogra_: would editing that directly, from snappy config, be reasonable?10:27
Chipacaand would the rename work with the mount10:28
Chipacaaugh10:28
Chipacamvo: or i just revert the atomicwrite10:28
Chipacaand we think about what we've done10:28
mvoChipaca: directly editing /w/s-d/e/d/watchdog would work indeed10:29
mvoChipaca: lets do that for now, we need to re-investigate the writable paths system anyway before 16.0410:30
Chipacamvo: i'm not sure it'll work10:32
ogra_mvo, Chipaca http://paste.ubuntu.com/12893223/10:32
ogra_that should work10:32
* Chipaca hugs ogra_ 10:32
ogra_i assume we want that in stable too ?10:33
Chipacamvo: editing it in /writable doesn't work because the file is the thing mounted, by inode. New file -> new inode10:33
Chipacamvo: afacit at least10:33
mvoogra_: thanks10:33
mvoChipaca: uh, you are right of course10:33
Chipacaogra_: let me check the others i switched over to atomic, just in case10:34
ogra_well, it would work if you made the whole dir writable ... but that means all /etc/default files become writable which isnt so great10:34
Chipacaogra_: the timezone file, the network files (interfaces and pp), the modprobe file, the modules-load file, the watchdog file, and the hostname file10:36
ogra_intefaces has the whole dir writable10:36
ogra_modprobe stuff too10:36
Chipacamvo: we still need a branch for this10:37
ogra_hostname and timezone are already handled10:37
Chipacamvo: (on it)10:37
ogra_so we shoudl eb fie10:37
ogra_*shouold be10:37
Chipacaogra_: but the timezone file is not written to /etc/writable10:38
ogra_(RaspberryPi2)ubuntu@localhost:~$ ls /etc/writable/10:38
ogra_hostname  localtime  timezone10:38
ogra_my install disagrees :)10:38
ogra_(RaspberryPi2)ubuntu@localhost:~$ ls -lh /etc/timezone10:39
ogra_lrwxrwxrwx 1 root root 17 Jul 24 16:36 /etc/timezone -> writable/timezone10:39
Chipacaogra_: yes, but when you set it, it's written to /etc/timezone directly10:39
ogra_which is a link10:39
Chipacaogra_: which means the atomic code will try to create /etc/timezone.somecrap10:39
ogra_no, it should do the atomic write in /etc/writable10:40
ogra_(this is why we have that dir at all :) )10:40
Chipacaogra_: what about localtime?10:40
ogra_we should have called it /etc/atomic insztead, to make the purpose more clear ;)10:40
ogra_same thing10:40
ogra_as well as hostname10:40
Chipacaogra_: and watchdog.conf10:41
ogra_the filesystem takes care here10:41
ogra_ah, not only the /etc/default/watchdog file ?10:41
ogra_ok, adding10:41
Chipacaogra_: i think that covers it10:42
ogra_good10:42
ogra_uploaded to the PPA for wily ... now doing stable10:43
Chipacamvo: https://code.launchpad.net/~chipaca/snappy/use-etc-writable-for-atomics/+merge/27530210:45
mvoChipaca: thanks10:46
ogra_stable uploaded too10:47
mvoogra_: \o/10:47
Chipacainteresting that the bbb tests caught this and not the kvm ones10:48
ogra_http://themcphails.uk/snappy.jpg10:48
ogra_:)10:49
ogra_Chipaca, errr10:49
ogra_no10:49
Chipacaogra_: no?10:50
ogra_you want to edit the original places10:50
ogra_the filesystem should care for the rest10:50
ogra_dont edit in /etc/writable10:50
Chipacaogra_: buh.. buh... ?10:50
ogra_your atomic write should follow the link before doing the backup copy10:51
ogra_and the filesystem should manage that for you, so there should eb no need to touch /etc/writable directly10:51
Chipacasiiigh10:52
mvoChipaca, ogra_: I'm not sure if our code is doing that, I think it would replace the link (but thats from memory only)10:52
ogra_(at least all other writin tools do it like rthat ... this setup comes from the phone where it exists since years )10:52
Chipacamvo: it isn't, but it should perhaps10:52
ogra_it definitely should10:52
ogra_everythjing else does10:52
Chipacamvo: so, abort, abort, etc10:52
mvodone10:52
Chipacamvo: tarmac didn't care and merged it anyway. NEver mind, follow up branch will revert.11:09
Chipacamvo: https://code.launchpad.net/~chipaca/snappy/atomic-follow-symlinks/+merge/27531011:26
ogra_looks good11:28
fgimenezChipaca, first boot works fine in bbb too, now running the suite11:28
ogra_fgimenez, no mopre missing address ?11:29
fgimenezogra_ iirc that was happening on rolling only, let me check11:29
ogra_it could be the same cause11:29
fgimenezogra_, yes, rolling/edge https://bugs.launchpad.net/snappy/+bug/1503329, i'll try the fix there too11:30
ubottuLaunchpad bug 1503329 in Snappy "not getting an ipv4 address on the first boot" [Critical,Confirmed]11:30
ogra_fgimenez, though it could also be the double UID (for dnsmasq vs tss) which would be fixed in the next image11:31
ogra_especially if it is rolling only ... only rolling has this UID bug11:31
=== olli_ is now known as olli
fgimenezogra_, ack thanks11:36
=== chihchun is now known as chihchun_afk
* ogra_ sighs about the RPi devicetree issues :/12:01
ogra_mvo, please remind me that we talk about that once the release is out ... ^^^12:02
mvoogra_: ok12:02
mvoChipaca: thanks!12:02
Chipacaaugh12:14
mvoChipaca: more crisis?12:15
mvohm, apparently12:15
Chipacamvo: :)12:15
Chipacaor :-/12:15
Chipacadepending on the flavour of your cynicism12:15
* mvo hugs Chipaca12:15
mvoall good12:15
Chipacamvo: there's an explicit test for us not following symlinks there12:17
mvoChipaca: geh12:18
mvoChipaca: let me check that, I wonder why12:18
mvoChipaca: I have some vague memory why this might be useful to not follow symlinks12:18
Chipacamvo: security, usually12:18
mvoChipaca: yeah, I think so, I think it came up as part of the recent click issue12:19
Chipacamvo: for example :)12:20
mvoChipaca: hmmmmmmmmmmmm, so indeed we added this recently as a result of click issue12:21
Chipacayep12:21
mvoChipaca: so lets not merge it and use /etc/writable and look into this after the reelease again12:21
mvoChipaca: its a mess anyway12:21
mvoChipaca: or do you have an alternative suggestion :) ?12:22
Chipacamvo: adding a flags parameter to AtomicWriteFile, which defaults to nofollow12:22
mvoChipaca: even better12:23
=== matiasb_ is now known as matiasb
Chipacamvo: updated lp:~chipaca/snappy/atomic-follow-symlinks if you want to take a look12:45
mvoChipaca: done, thanks again12:46
mvofgimenez: r223 for amd64 is ready, if you could give this a early testrun14:00
mvoelopio: -^14:00
mvoI will build arm/i386 too now14:00
fgimenezmvo, thanks on it14:00
mvothank you!14:00
fgimenezmvo, the watchdog config test is still failing with a different error now "Error: open /etc/default/writable/default: no such file or directory"14:16
mvofgimenez: aha, thanks. let me check14:16
mvoChipaca: -^14:16
mvoor maybe ogra_ ---^ ?14:18
ogra_hrn14:18
ogra_yeah, that might be  wrong link ...14:19
mvohttp://paste.ubuntu.com/12894264/14:19
ogra_hmm, no, that looks right14:19
ogra_how does /etc/writable look like ?14:20
mvoogra_: http://paste.ubuntu.com/12894275/14:20
ogra_looks fine too14:21
mvoogra_: that looks right? isn't there a "/" missing in the first one?14:21
ogra_oooh !!14:21
ogra_no14:21
ogra_there would be a ../ missing14:21
ogra_damned14:22
ogra_that needs more logic in livecd-rootfs :/14:22
ogra_will fix after meeting14:23
mvota14:23
elopiofgimenez: why don't you do a show and tell on the next ubuntu summit?14:29
fgimenezelopio, well, i doubt i have anything interesting to show or tell ;)14:30
elopioI think it would be cool to show the integration suite on kvm. But maybe the stuff you did on snappy-tests-job, or some other cool thing you liked.14:31
elopiofgimenez: it doesn't have to be a full hour. I'd be interested also in things like the api test suite, or the cli one, or the failover tests.14:33
elopioballoons: help me out here :)14:33
fgimenezelopio, ok i'll have a look, are you going to do something for the summit?14:34
balloonsfgimenez, definitely! You can demo something for 5-10 mins and take a couple questions. It can be really quick if you wish14:34
elopiofgimenez: no. I prefer to help you with yours, if you need a hand.14:34
ogra_mvo, http://paste.ubuntu.com/12894355/ does that look ok to you ?14:36
mvoogra_: yes, thank you14:37
ogra_(assuming we might get more files in /etc/default to be writable one day)14:37
ogra_uploading then14:37
elopiofgimenez: any chance you have an old 15.04 edge around? like #142?14:37
fgimenezballoons, elopio ok thanks, let's see if there's something interesting enough (or at all :) in all this stuff..14:38
fgimenezelopio, nope, i've seen the bug, maybe from old stables?14:38
elopiofgimenez: I'll try re-updates from different stables. If the bug doesn't happen there, then it's not important.14:39
fgimenezelopio, sgtm14:40
elopiofgimenez: have you confirmed if Chipaca's fix solves the problem?14:42
fgimenezelopio, yes, eth0 is up on first boot for both amd64 and bbb, it's already included in r22314:43
elopiocool. Then I'll go and take a walk with the dog while the automated suite runs here.14:44
fgimenezelopio, haven't tried it yet on rolling/edge bbb, maybe it solves the ipv4 issue too14:44
elopiobbs14:44
elopiofgimenez: ah, I'll leave that one flashing and check.14:44
elopiofgimenez: hey, btw, I started getting tls handshake timeouts when doing u-d-f.14:46
elopioI still think that's a problem in the store servers. Just not sure what to do about it to prove it.14:47
fgimenezelopio, from udf too? i think i saw that some time ago, yes, it seems to be a problem of the store14:47
Chipacaogra_: mvo: status update wrt symlinks plz (but no rush)14:51
ogra_Chipaca, fix is in the PPA ... waiting for promotion and new image14:51
Chipacaneat-o14:52
Chipacai'll go get myself a cuppa tea then14:52
kgunntedg: hey so, made some good headway last night on the mir yaml, but wondering...at least one of the paths wanted arch specific input, is there a way to late bind that in the yaml ? so it'd insert whatever arch your building on?14:52
tedgkgunn: No, I don't think we have anything for that.14:53
tedgkgunn: One design floating around had variables, but I don't think they've made it to a final design.14:53
tedgkgunn: We need to do some more thinking around cross-building and architectures, we're weak there right now.14:53
kgunntedg: ok..understood14:54
kgunnjust felt naughty14:54
tedgkgunn: In a good way? ;-)14:55
kgunnlol14:55
kgunnno, most definitely not14:56
=== ubott2 is now known as ubottu
=== Odd_Blok1 is now known as Odd_Bloke
ogra_new stable image building15:02
ogra_fgimenez, elopio ^^^ with a fix for /etc/default/watchdog15:02
=== moul_ is now known as moul
=== mariogrip_ is now known as mariogrip
=== Ursinha_ is now known as Ursinha
ogra_Xenial-changes Subscription results15:14
ogra_Your subscription request has been received, and will soon be acted upon....15:14
ogra_:D15:14
=== Chipaca` is now known as Chipaca
=== Tristit1a is now known as Tristitia
elopiofgimenez: I'm getting an error on the config test: open /etc/default/writable/default: no such file or directory15:42
ogra_elopio, yes, thats is what we are doing the current re-spin for15:42
ogra_225 should be fixed15:43
elopioogra_: oh, I thought that was already on 225.15:43
elopiosorry, on 22415:43
ogra_aaand ... 225 hit the system-image server this second :)15:43
ogra_so happy testing :)15:43
elopiook good, the tests work :)15:43
* elopio u-d-fs15:43
ogra_let me know how it goes ... i *belive* it is fixed ... but religion might not help here ;)15:44
elopioI bet religion will just make it worse ;)15:45
ogra_heh, yeah15:48
fgimenezelopio, ogra_ it still fails for me "Error: rename /etc/writable/watchdog.conf.HGxhokjLnsKB /etc/writable/watchdog.conf: device or resource busy"15:51
ogra_huh ?15:51
ogra_thats a different file though15:51
ogra_Chipaca, ^^^ did your atomic write fix not land yet ?15:52
ogra_"device or resource busy" is also a very interesting error message here15:53
elopioError: rename /etc/writable/watchdog.conf.elXUoCIALnmQ /etc/writable/watchdog.conf: device or resource busy15:53
elopiosame here.15:53
ogra_yeah15:53
ogra_so the fix for /etc/default/watchdog works ...15:53
ogra_but we are back at the initial error from this morning15:54
ogra_weird15:54
ogra_i thought Chipaca's fix had landed already15:54
Chipacaogra_: that looks like the one with my fix16:05
ogra_so why do we get that weird error ?16:05
Chipaca"device or resource busy" is not the error from this morning16:05
Chipacabecause16:05
Chipacait's still mounted16:05
Chipacai'd wager16:05
ogra_why would it not be mounted16:05
Chipacaogra_: i mean, we used to mount /etc/watchdog.conf16:06
ogra_ooooh !16:06
ogra_crap16:06
Chipacaoooh.16:06
Chipaca:)16:06
Chipacaogra_: only guessing, but that's what i'd look at, myself16:06
ogra_yeah16:07
ogra_nobody updated ubuntu-core-config16:07
Chipacasilly nobody16:07
=== dpm is now known as dpm-afk
ogra_uploaded (for vivid yet)16:10
fgimenezleaving, good evening o/16:10
ogra_bah, and i cant land it for wily16:12
ogra_oh, i can16:13
ogra_hah16:13
ogra_ok, wily fix uploaded too16:15
seb128wah, nobody fixed 1.6 so wily released with 1.5?16:16
ogra_seb128, seemingly slipped through, we'Re all super busy doing a release :)16:19
Chipacaoff to a parents' evening, will bbl16:34
ogra_elopio, 226 building16:34
elopiotesting 226 now17:18
ogra_yay, thanks17:19
* ogra_ was about to ping that it is done :) 17:19
=== charles__ is now known as charles
elopioogra_: Chipaca: the test now passes. Trying the rest.17:35
ogra_YAY !17:35
* ogra_ relaxes 17:35
elopioogra_: oh, don't relax. I have food for thought for you: how can we put a regression test that doesn't require us to regenerate an image to check it?17:37
ogra_elopio, well, you could replace the changed parts if yu know exactly which are the ones ... but that only works if neither initrd nor firstboot is involved17:41
ogra_(and indeed if it isnt bootloader or kernel stuff)17:41
ogra_in the watchdog case this wouldnt have helped17:41
elopioogra_: where can I see what you changed to fix it?17:42
ogra_usually in the MPs but for certain packages like livecd-rootfs or ubuntu-core-config you can only see it in the package diff of the PPA17:42
ogra_https://launchpad.net/~snappy-dev/+archive/ubuntu/image/+packages ... if you expand either of the "ubuntu-core-config" packages on that page there is a diff below17:43
elopioI see. This is pretty hard, because it's not like an api that snappy consumes.17:47
ogra_yeah, thats plumbing ...17:47
elopioogra_: will this core-config be a snap eventually too?17:48
ogra_what you can definitely look at is http://people.canonical.com/~ogra/core-image-stats/17:48
ogra_no, its one of the debs we create the rootfs from17:48
ogra_the rootfs will become a snap but not the parts used to build it17:48
ogra_for 226 the changelog is http://people.canonical.com/~ogra/core-image-stats/vivid/20151022.5.changes17:49
elopioogra_: so this goes back to being able to create a rootfs on demand. Once we build it, we can plug it into an existing image and check.18:16
elopionot perfect, but will reduce the feedback time.18:16
ogra_elopio, well, why not generate a complete image then ?18:36
elopioogra_: I guess you are right. But we need to isolate the changes as much as possible.18:38
elopiomaybe we can make a contract between snappy and what ubuntu-core-config produces. And then test just that.18:38
elopioanyway, I think that the next step would be to generate a complete image, as you say.18:39
elopioogra_: what's wrong with this? http://pastebin.ubuntu.com/12896455/19:13
elopioI don't understand the error.19:13
mvoelopio: hi, so whats the status? how is the imgae looking?19:29
mvoogra_: or you maybe, how is the image looking?19:32
elopiomvo: all good with kvm, already a couple of full successful runs + exploring.19:35
elopiotests are currently running on bbb and rpi.19:35
elopiomvo: I think we are good to publish to alpha to test the upgrade and rollback19:36
mvoelopio: excellent, let me do that then19:36
elopiomvo: so, will you trigger staging tomorrow?19:38
elopioand who should we notify to get the cert team testing alpha?19:38
mvoelopio: I can stay up some more to release today, depends a bit how long the test runs will take19:39
mvoelopio: and when its stable I will tell the cert guys to use it (thats how we usually do it)19:40
mvoelopio: amd64/armhf/i386 ready in alpha19:41
elopiomvo: great. I'll download and flash19:42
mvoelopio: thanks, just keep me updated, I will be around for ~1h at least, otherwise I will continue (early) in my morning19:43
ogra_mvo, its all fine (like elopio said) ... had to do a re-spin because we forgot to drop the watchdog stuff from ubuntu-core (so it was still bind mounted)19:57
mvoogra_: aha, thanks19:58
elopiomvo: tested alpha 14 -> 15, rollback, update again, and then the full suite on kvm20:24
elopiothe boards in progress. But that's a lot slower. Things look good.20:25
mvoelopio: yay, thanks20:26
elopioI think last time ricmm told us to notify the chinese team to test the alpha before the release, but I'm not sure about that part of the process.20:26
mvoelopio: let me ask20:28

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!