=== chihchun_afk is now known as chihchun === chihchun is now known as chihchun_afk [07:05] good morning [07:09] good morning [07:10] ogra_: good morning, it looks like we have dnsmasq and tss both with uid 109 in the image right now, could you have a look please? in wily, not sure about stable, just noticed on wily [07:10] hey dholbach [07:10] fgimenez: good morning [07:10] hey mvo [07:11] hey mvo, dholbach :) mvo, do we have a rc image? [07:11] fgimenez: not yet, sorry, I'm working on it now, Chipaca almost fixed the network bug (which is great) [07:12] mvo, awesome! ok np [07:12] hola fgimenez :) [07:15] Morning o/ [07:51] buongiorno :) === chihchun_afk is now known as chihchun [08:35] Good morning all; happy Thursday, and happy Wily Werewolf welease day! 😃 [08:42] JamesTait always seems like the most enthusiastic morning person [08:47] It's all a charade. I'm a grumpy old curmudgeon really. 😉 [08:59] mvo: do you have a branch of almost-next-15.04? [09:00] Chipaca: 15.04 is up-to-date for the release, I think the one outstanding issue is the ordering [09:00] mvo: and is the latest edge built on that? [09:00] just to minimise the delta i'm working with [09:00] Chipaca: yes, well, I can build a new image now [09:01] mvo: i mean, will u-d-f build it with that [09:01] or is that what you offered to build? [09:01] Chipaca: biulding a new amd64 15.04/edge image with todays lp:snappy/15.04 branch so that you have something super close to the stable release [09:01] Chipaca: does that help you? [09:02] mvo: indeedly it does :) [09:02] and i'll branch that and propose a branch on that [09:04] Chipaca: thanks, you can also work on trunk right now, we have no delta and I inspected the changes (of course) before merging trunk to 15.04 and it looks very sane (famous last words!) [09:04] ah! perfect then [09:04] Chipaca: I triggered the image build, new 15.04/edge image should be ready in ~30min or so [09:13] Chipaca: image build [09:13] Chipaca: now it just needs to get imported [09:14] mvo: it seems there was a 15.04/edge built overnight [09:14] mvo: working with that for now [09:14] yeah [09:14] sure, that will be fine as well, the delta is small [09:18] mvo, whats bad about that ? (tss) [09:18] and yes, i seeded nss3 for a test because it showed up on some list from olli [09:18] ogra_: two usernames with the same uids [09:18] uh ? [09:18] thats not possible [09:19] ogra_: yeah, olli asked me to revert that seeding and he said he will follow up [09:19] ogra_: let me double check about the uid, maybe its something local [09:19] right, as i said, it was for testing (and obviously worth it :) ) [09:19] /etc/passwd is readonly [09:19] and checked during build [09:20] argh [09:21] so no, tss is a requirement for the rootfs encryption stuff [09:21] not related to my nss3 seeding [09:21] and yes, seems thats my bug :( [09:21] (didnt check enough that the tss uid is not already there) [09:22] * ogra_ bumps the UID to 110 [09:25] ogra_: thanks, no worries. is this 15.04 too? [09:25] yes [09:25] well, let me check :P [09:26] ok, please let me know once the fix is upload and I trigger a new amd64 image [09:26] there is also a fix from Chipaca pending, so no worries we are not golden anyway yet :) [09:27] mvo, not in stable .... phew [09:27] yay [09:28] now i wonder why we dont have dnsmasq there [09:28] oh, because ubuntu-fan only landed in rolling back then [09:28] (on purpose) [09:28] that was before all the crazy reqs. came in, we shoudl ask if we need fan on stable too now [09:33] uploaded to the PPA [09:33] https://code.launchpad.net/~chipaca/snappy/firstboot-ordering/+merge/275244 is ready for review gents [09:34] fgimenez: if you could try this ^ wrt firstboot bringing up eth0, that'd be cool [09:38] Chipaca, ok on it [09:39] fgimenez: much appreciated! [09:40] fgimenez: i've fake-firstbooted ~20 times and it worked every time, fwiw [09:44] would be nice to know if this by chance also fixes the BBB [09:44] seems racy enough to be possible :) [09:46] /join #ubuntu-release-party [09:46] oops :P [09:56] Chipaca: \o/ once you fix lands we should have a golden image [10:04] ogra_: this should be less racy than before [10:04] ogra_: so maybe :) [10:09] Chipaca, works great on kvm \o/ [10:10] Chipaca: (I said it before, but I say it again) \o/ [10:11] fgimenez: and ... bbb? :) [10:11] Chipaca, ogra i'll try bbb now [10:11] whee :) [10:11] Chipaca, yep :) [10:14] fgimenez: could you put a +1 on the branch based on the success of kvm tests? [10:14] Chipaca, sure [10:15] fgimenez: even if it doesn't fix bbb, it fixes amd64, so we can build/release that at least [10:21] Chipaca, there's one error in the test suite on kvm, but not related to eth0 on first boot "Error: open /etc/default/watchdog.wvUpOFylUogg: read-only file system" [10:21] hm! [10:22] eeeep [10:22] ogra_: you around? [10:22] mvo: this might be bad [10:22] what ?!? [10:22] ogra_: we have just /etc/default/watchdog as rw, not the directory itself? [10:22] wh would anything write to it on boot ? [10:23] not boot, this is the integration test suite [10:23] testing setting watchdog [10:23] yeah, makin the directory writable would kind of defeat the purpose [10:23] uh, atomicWrite ? [10:23] yeah [10:23] needs special hacks [10:23] (like hostname) [10:23] mvo: yes [10:23] bäää [10:23] yup [10:24] mea culpa, i broke that [10:24] well, its the right thing to do :/ [10:24] oh is that a function you control (doing the atomic write ?= [10:24] yes [10:24] well [10:25] ogra_: it used to not do an atomic write at all [10:25] ogra_: not even a sync [10:25] maybe the right thing to do is to edit it in writable? [10:25] well, if we want to keep it we need to handle the file like the others in /etc/writable [10:25] would that work? [10:25] i.e. copy it over at image build time and create a link [10:27] i meant edit /writable/system-data/etc/default/watchdog directly [10:27] mvo: ogra_: would editing that directly, from snappy config, be reasonable? [10:28] and would the rename work with the mount [10:28] augh [10:28] mvo: or i just revert the atomicwrite [10:28] and we think about what we've done [10:29] Chipaca: directly editing /w/s-d/e/d/watchdog would work indeed [10:30] Chipaca: lets do that for now, we need to re-investigate the writable paths system anyway before 16.04 [10:32] mvo: i'm not sure it'll work [10:32] mvo, Chipaca http://paste.ubuntu.com/12893223/ [10:32] that should work [10:32] * Chipaca hugs ogra_ [10:33] i assume we want that in stable too ? [10:33] mvo: editing it in /writable doesn't work because the file is the thing mounted, by inode. New file -> new inode [10:33] mvo: afacit at least [10:33] ogra_: thanks [10:33] Chipaca: uh, you are right of course [10:34] ogra_: let me check the others i switched over to atomic, just in case [10:34] well, it would work if you made the whole dir writable ... but that means all /etc/default files become writable which isnt so great [10:36] ogra_: the timezone file, the network files (interfaces and pp), the modprobe file, the modules-load file, the watchdog file, and the hostname file [10:36] intefaces has the whole dir writable [10:36] modprobe stuff too [10:37] mvo: we still need a branch for this [10:37] hostname and timezone are already handled [10:37] mvo: (on it) [10:37] so we shoudl eb fie [10:37] *shouold be [10:38] ogra_: but the timezone file is not written to /etc/writable [10:38] (RaspberryPi2)ubuntu@localhost:~$ ls /etc/writable/ [10:38] hostname localtime timezone [10:38] my install disagrees :) [10:39] (RaspberryPi2)ubuntu@localhost:~$ ls -lh /etc/timezone [10:39] lrwxrwxrwx 1 root root 17 Jul 24 16:36 /etc/timezone -> writable/timezone [10:39] ogra_: yes, but when you set it, it's written to /etc/timezone directly [10:39] which is a link [10:39] ogra_: which means the atomic code will try to create /etc/timezone.somecrap [10:40] no, it should do the atomic write in /etc/writable [10:40] (this is why we have that dir at all :) ) [10:40] ogra_: what about localtime? [10:40] we should have called it /etc/atomic insztead, to make the purpose more clear ;) [10:40] same thing [10:40] as well as hostname [10:41] ogra_: and watchdog.conf [10:41] the filesystem takes care here [10:41] ah, not only the /etc/default/watchdog file ? [10:41] ok, adding [10:42] ogra_: i think that covers it [10:42] good [10:43] uploaded to the PPA for wily ... now doing stable [10:45] mvo: https://code.launchpad.net/~chipaca/snappy/use-etc-writable-for-atomics/+merge/275302 [10:46] Chipaca: thanks [10:47] stable uploaded too [10:47] ogra_: \o/ [10:48] interesting that the bbb tests caught this and not the kvm ones [10:48] http://themcphails.uk/snappy.jpg [10:49] :) [10:49] Chipaca, errr [10:49] no [10:50] ogra_: no? [10:50] you want to edit the original places [10:50] the filesystem should care for the rest [10:50] dont edit in /etc/writable [10:50] ogra_: buh.. buh... ? [10:51] your atomic write should follow the link before doing the backup copy [10:51] and the filesystem should manage that for you, so there should eb no need to touch /etc/writable directly [10:52] siiigh [10:52] Chipaca, ogra_: I'm not sure if our code is doing that, I think it would replace the link (but thats from memory only) [10:52] (at least all other writin tools do it like rthat ... this setup comes from the phone where it exists since years ) [10:52] mvo: it isn't, but it should perhaps [10:52] it definitely should [10:52] everythjing else does [10:52] mvo: so, abort, abort, etc [10:52] done [11:09] mvo: tarmac didn't care and merged it anyway. NEver mind, follow up branch will revert. [11:26] mvo: https://code.launchpad.net/~chipaca/snappy/atomic-follow-symlinks/+merge/275310 [11:28] looks good [11:28] Chipaca, first boot works fine in bbb too, now running the suite [11:29] fgimenez, no mopre missing address ? [11:29] ogra_ iirc that was happening on rolling only, let me check [11:29] it could be the same cause [11:30] ogra_, yes, rolling/edge https://bugs.launchpad.net/snappy/+bug/1503329, i'll try the fix there too [11:30] Launchpad bug 1503329 in Snappy "not getting an ipv4 address on the first boot" [Critical,Confirmed] [11:31] fgimenez, though it could also be the double UID (for dnsmasq vs tss) which would be fixed in the next image [11:31] especially if it is rolling only ... only rolling has this UID bug === olli_ is now known as olli [11:36] ogra_, ack thanks === chihchun is now known as chihchun_afk [12:01] * ogra_ sighs about the RPi devicetree issues :/ [12:02] mvo, please remind me that we talk about that once the release is out ... ^^^ [12:02] ogra_: ok [12:02] Chipaca: thanks! [12:14] augh [12:15] Chipaca: more crisis? [12:15] hm, apparently [12:15] mvo: :) [12:15] or :-/ [12:15] depending on the flavour of your cynicism [12:15] * mvo hugs Chipaca [12:15] all good [12:17] mvo: there's an explicit test for us not following symlinks there [12:18] Chipaca: geh [12:18] Chipaca: let me check that, I wonder why [12:18] Chipaca: I have some vague memory why this might be useful to not follow symlinks [12:18] mvo: security, usually [12:19] Chipaca: yeah, I think so, I think it came up as part of the recent click issue [12:20] mvo: for example :) [12:21] Chipaca: hmmmmmmmmmmmm, so indeed we added this recently as a result of click issue [12:21] yep [12:21] Chipaca: so lets not merge it and use /etc/writable and look into this after the reelease again [12:21] Chipaca: its a mess anyway [12:22] Chipaca: or do you have an alternative suggestion :) ? [12:22] mvo: adding a flags parameter to AtomicWriteFile, which defaults to nofollow [12:23] Chipaca: even better === matiasb_ is now known as matiasb [12:45] mvo: updated lp:~chipaca/snappy/atomic-follow-symlinks if you want to take a look [12:46] Chipaca: done, thanks again [14:00] fgimenez: r223 for amd64 is ready, if you could give this a early testrun [14:00] elopio: -^ [14:00] I will build arm/i386 too now [14:00] mvo, thanks on it [14:00] thank you! [14:16] mvo, the watchdog config test is still failing with a different error now "Error: open /etc/default/writable/default: no such file or directory" [14:16] fgimenez: aha, thanks. let me check [14:16] Chipaca: -^ [14:18] or maybe ogra_ ---^ ? [14:18] hrn [14:19] yeah, that might be wrong link ... [14:19] http://paste.ubuntu.com/12894264/ [14:19] hmm, no, that looks right [14:20] how does /etc/writable look like ? [14:20] ogra_: http://paste.ubuntu.com/12894275/ [14:21] looks fine too [14:21] ogra_: that looks right? isn't there a "/" missing in the first one? [14:21] oooh !! [14:21] no [14:21] there would be a ../ missing [14:22] damned [14:22] that needs more logic in livecd-rootfs :/ [14:23] will fix after meeting [14:23] ta [14:29] fgimenez: why don't you do a show and tell on the next ubuntu summit? [14:30] elopio, well, i doubt i have anything interesting to show or tell ;) [14:31] I think it would be cool to show the integration suite on kvm. But maybe the stuff you did on snappy-tests-job, or some other cool thing you liked. [14:33] fgimenez: it doesn't have to be a full hour. I'd be interested also in things like the api test suite, or the cli one, or the failover tests. [14:33] balloons: help me out here :) [14:34] elopio, ok i'll have a look, are you going to do something for the summit? [14:34] fgimenez, definitely! You can demo something for 5-10 mins and take a couple questions. It can be really quick if you wish [14:34] fgimenez: no. I prefer to help you with yours, if you need a hand. [14:36] mvo, http://paste.ubuntu.com/12894355/ does that look ok to you ? [14:37] ogra_: yes, thank you [14:37] (assuming we might get more files in /etc/default to be writable one day) [14:37] uploading then [14:37] fgimenez: any chance you have an old 15.04 edge around? like #142? [14:38] balloons, elopio ok thanks, let's see if there's something interesting enough (or at all :) in all this stuff.. [14:38] elopio, nope, i've seen the bug, maybe from old stables? [14:39] fgimenez: I'll try re-updates from different stables. If the bug doesn't happen there, then it's not important. [14:40] elopio, sgtm [14:42] fgimenez: have you confirmed if Chipaca's fix solves the problem? [14:43] elopio, yes, eth0 is up on first boot for both amd64 and bbb, it's already included in r223 [14:44] cool. Then I'll go and take a walk with the dog while the automated suite runs here. [14:44] elopio, haven't tried it yet on rolling/edge bbb, maybe it solves the ipv4 issue too [14:44] bbs [14:44] fgimenez: ah, I'll leave that one flashing and check. [14:46] fgimenez: hey, btw, I started getting tls handshake timeouts when doing u-d-f. [14:47] I still think that's a problem in the store servers. Just not sure what to do about it to prove it. [14:47] elopio, from udf too? i think i saw that some time ago, yes, it seems to be a problem of the store [14:51] ogra_: mvo: status update wrt symlinks plz (but no rush) [14:51] Chipaca, fix is in the PPA ... waiting for promotion and new image [14:52] neat-o [14:52] i'll go get myself a cuppa tea then [14:52] tedg: hey so, made some good headway last night on the mir yaml, but wondering...at least one of the paths wanted arch specific input, is there a way to late bind that in the yaml ? so it'd insert whatever arch your building on? [14:53] kgunn: No, I don't think we have anything for that. [14:53] kgunn: One design floating around had variables, but I don't think they've made it to a final design. [14:53] kgunn: We need to do some more thinking around cross-building and architectures, we're weak there right now. [14:54] tedg: ok..understood [14:54] just felt naughty [14:55] kgunn: In a good way? ;-) [14:55] lol [14:56] no, most definitely not === ubott2 is now known as ubottu === Odd_Blok1 is now known as Odd_Bloke [15:02] new stable image building [15:02] fgimenez, elopio ^^^ with a fix for /etc/default/watchdog === moul_ is now known as moul === mariogrip_ is now known as mariogrip === Ursinha_ is now known as Ursinha [15:14] Xenial-changes Subscription results [15:14] Your subscription request has been received, and will soon be acted upon.... [15:14] :D === Chipaca` is now known as Chipaca === Tristit1a is now known as Tristitia [15:42] fgimenez: I'm getting an error on the config test: open /etc/default/writable/default: no such file or directory [15:42] elopio, yes, thats is what we are doing the current re-spin for [15:43] 225 should be fixed [15:43] ogra_: oh, I thought that was already on 225. [15:43] sorry, on 224 [15:43] aaand ... 225 hit the system-image server this second :) [15:43] so happy testing :) [15:43] ok good, the tests work :) [15:43] * elopio u-d-fs [15:44] let me know how it goes ... i *belive* it is fixed ... but religion might not help here ;) [15:45] I bet religion will just make it worse ;) [15:48] heh, yeah [15:51] elopio, ogra_ it still fails for me "Error: rename /etc/writable/watchdog.conf.HGxhokjLnsKB /etc/writable/watchdog.conf: device or resource busy" [15:51] huh ? [15:51] thats a different file though [15:52] Chipaca, ^^^ did your atomic write fix not land yet ? [15:53] "device or resource busy" is also a very interesting error message here [15:53] Error: rename /etc/writable/watchdog.conf.elXUoCIALnmQ /etc/writable/watchdog.conf: device or resource busy [15:53] same here. [15:53] yeah [15:53] so the fix for /etc/default/watchdog works ... [15:54] but we are back at the initial error from this morning [15:54] weird [15:54] i thought Chipaca's fix had landed already [16:05] ogra_: that looks like the one with my fix [16:05] so why do we get that weird error ? [16:05] "device or resource busy" is not the error from this morning [16:05] because [16:05] it's still mounted [16:05] i'd wager [16:05] why would it not be mounted [16:06] ogra_: i mean, we used to mount /etc/watchdog.conf [16:06] ooooh ! [16:06] crap [16:06] oooh. [16:06] :) [16:06] ogra_: only guessing, but that's what i'd look at, myself [16:07] yeah [16:07] nobody updated ubuntu-core-config [16:07] silly nobody === dpm is now known as dpm-afk [16:10] uploaded (for vivid yet) [16:10] leaving, good evening o/ [16:12] bah, and i cant land it for wily [16:13] oh, i can [16:13] hah [16:15] ok, wily fix uploaded too [16:16] wah, nobody fixed 1.6 so wily released with 1.5? [16:19] seb128, seemingly slipped through, we'Re all super busy doing a release :) [16:34] off to a parents' evening, will bbl [16:34] elopio, 226 building [17:18] testing 226 now [17:19] yay, thanks [17:19] * ogra_ was about to ping that it is done :) === charles__ is now known as charles [17:35] ogra_: Chipaca: the test now passes. Trying the rest. [17:35] YAY ! [17:35] * ogra_ relaxes [17:37] ogra_: oh, don't relax. I have food for thought for you: how can we put a regression test that doesn't require us to regenerate an image to check it? [17:41] elopio, well, you could replace the changed parts if yu know exactly which are the ones ... but that only works if neither initrd nor firstboot is involved [17:41] (and indeed if it isnt bootloader or kernel stuff) [17:41] in the watchdog case this wouldnt have helped [17:42] ogra_: where can I see what you changed to fix it? [17:42] usually in the MPs but for certain packages like livecd-rootfs or ubuntu-core-config you can only see it in the package diff of the PPA [17:43] https://launchpad.net/~snappy-dev/+archive/ubuntu/image/+packages ... if you expand either of the "ubuntu-core-config" packages on that page there is a diff below [17:47] I see. This is pretty hard, because it's not like an api that snappy consumes. [17:47] yeah, thats plumbing ... [17:48] ogra_: will this core-config be a snap eventually too? [17:48] what you can definitely look at is http://people.canonical.com/~ogra/core-image-stats/ [17:48] no, its one of the debs we create the rootfs from [17:48] the rootfs will become a snap but not the parts used to build it [17:49] for 226 the changelog is http://people.canonical.com/~ogra/core-image-stats/vivid/20151022.5.changes [18:16] ogra_: so this goes back to being able to create a rootfs on demand. Once we build it, we can plug it into an existing image and check. [18:16] not perfect, but will reduce the feedback time. [18:36] elopio, well, why not generate a complete image then ? [18:38] ogra_: I guess you are right. But we need to isolate the changes as much as possible. [18:38] maybe we can make a contract between snappy and what ubuntu-core-config produces. And then test just that. [18:39] anyway, I think that the next step would be to generate a complete image, as you say. [19:13] ogra_: what's wrong with this? http://pastebin.ubuntu.com/12896455/ [19:13] I don't understand the error. [19:29] elopio: hi, so whats the status? how is the imgae looking? [19:32] ogra_: or you maybe, how is the image looking? [19:35] mvo: all good with kvm, already a couple of full successful runs + exploring. [19:35] tests are currently running on bbb and rpi. [19:36] mvo: I think we are good to publish to alpha to test the upgrade and rollback [19:36] elopio: excellent, let me do that then [19:38] mvo: so, will you trigger staging tomorrow? [19:38] and who should we notify to get the cert team testing alpha? [19:39] elopio: I can stay up some more to release today, depends a bit how long the test runs will take [19:40] elopio: and when its stable I will tell the cert guys to use it (thats how we usually do it) [19:41] elopio: amd64/armhf/i386 ready in alpha [19:42] mvo: great. I'll download and flash [19:43] elopio: thanks, just keep me updated, I will be around for ~1h at least, otherwise I will continue (early) in my morning [19:57] mvo, its all fine (like elopio said) ... had to do a re-spin because we forgot to drop the watchdog stuff from ubuntu-core (so it was still bind mounted) [19:58] ogra_: aha, thanks [20:24] mvo: tested alpha 14 -> 15, rollback, update again, and then the full suite on kvm [20:25] the boards in progress. But that's a lot slower. Things look good. [20:26] elopio: yay, thanks [20:26] I think last time ricmm told us to notify the chinese team to test the alpha before the release, but I'm not sure about that part of the process. [20:28] elopio: let me ask