[00:42] mwhudson: Heya [00:42] niemeyer: hi [00:42] mwhudson: We have very good coverage on assertions on the system design doc [00:42] ah yes, i've seen that i think [00:42] mwhudson: Do you have a moment to chat about the upcoming image? [00:43] niemeyer: it's been kept up to date as the implementation has happened? [00:43] niemeyer: sure [00:43] mwhudson: Yeah, I think it's quite precise stlil [00:43] cool, i shall re-read it then [00:43] mwhudson: There might be a couple of minor assertions missing there, but otherwise it's good [00:43] niemeyer: chat here or some kind of voip? [00:44] mwhudson: We'd like to finish our pending activities for RTM by EOD tomorrow, so that we can cook a candidate image by Monday [00:44] mwhudson: Here is fine to start, we can jump on a call if we need more bandwidth [00:45] mwhudson: So I'd like to sync with you regarding open points on your end [00:45] niemeyer: that would be great, i've been feeling a bit disconnected [00:45] well not disconnected, just a bit unsure of some things [00:46] mwhudson: Yeah, the usual problem of living on different sides of the world [00:46] that and coming in late to a fast moving project :) [00:46] mwhudson: We've had a few network-related branches landing recently [00:46] yes [00:46] mwhudson: INdeed [00:47] mwhudson: I've been very far from that work, so I'm not sure what your plans for it are [00:47] mwhudson: is it working already? [00:47] niemeyer: more or less, it works fine in kvm [00:48] niemeyer: the ui does not let you configure wlan interfaces, but if you inject an ifupdown config to make that work, it shouldn't get in the way [00:48] mwhudson: Cool, that's a start [00:49] mwhudson: How do you mean? wlan feels like the basic stuff [00:50] niemeyer: netplan only just gained support for wlan, and we haven't written the ui yet [00:50] niemeyer: nothing fundamental, just lack of time [00:52] mwhudson: Ok.. hmm [00:52] mwhudson: So is there anything pending that we need to do today to have a proper image on Monday? [00:52] i can hack like crazy and see what i can get done in the 3 hours of work i have left in the week ... [00:52] niemeyer: not that i am aware of [00:53] niemeyer: on the networking side [00:53] niemeyer: on the user side, there was some confusion about whether cloud-init should accept the traditional cloud-init syntax for creating users [00:53] mwhudson: Something you'd like to be tested during the next month as we put those images to run [00:54] mwhudson: Yeah, I've seen the bug # [00:54] niemeyer: sorry, is that a question? [00:54] mwhudson: Yeah, sorry, it was part of the previous one [00:55] the biggest open bits are around user management but that's not happening for RTM aiui [00:55] mwhudson: re. the cloud-init feature, I'm not sure.. one alternative would be to teach it to talk to snapd and call create-user [00:56] mwhudson: Yeah, there's an open point I noted down today for GA.. we still need to associated the system user created with a snapd user [00:56] niemeyer: well it should likely be able to do that, sure, but that's not what i meant by "accept the traditional cloud-init syntax for creating users" [00:57] mwhudson: Ah? [00:57] which is username/password/ssh key, not email for use with sso [01:01] niemeyer: yes, snapd user tied to system user and some way for console-conf to tell that this has happened [01:01] mwhudson: Ideally we'd also put that under the same system [01:02] mwhudson: So we have a single path to create ubuntu core users, and the outcome is always the same [01:02] (owner semantics, credentials for snapd, etc) [01:04] niemeyer: that certainly makes sense [01:07] mwhudson: Cool.. so for RTM, if you have any critical changes you'd like to see tested in the candidate image, today or your Monday should be fine, and we'll try to get the changes in on time [01:08] mwhudson: Post RTM, let's nail down networking and those credentials/ownership details [01:08] niemeyer: ack. how should i provide any such fixes? upload to ppa:canonical-foundations/ubuntu-image and email canonical-snapcraft@ ? [01:17] Yeah, or push a PR to snapd, if that's the case [01:17] mwhudson: ^ [01:18] niemeyer: talking of which! https://github.com/snapcore/snapd/pull/1830 [01:18] PR snapd#1830: firstboot: only configure en* interfaces by default [01:22] mwhudson: LGTM [01:22] mwhudson: As a minor, would be nice to fix indentation on this file [01:22] mwhudson: I'd suggest 4 spaces consistently [01:23] mwhudson: one space makes a bit too hard on the eye [01:23] mwhudson: and the first level is using 2 [01:24] niemeyer: fair enough, fixing === King_InuYasha is now known as Son_Goku [02:06] mwhudson: Much nicer, thank you [02:51] ah haha can i customize the kernel command line of a snappy image? [02:51] /etc/default/grub is read only... [03:09] PR # closed: snapcraft#652, snapcraft#652, snapcraft#652, snapcraft#652, snapcraft#652, snapcraft#652, snapcraft#652, snapcraft#652, snapcraft#652, snapcraft#652, snapcraft#652, snapcraft#652, snapcraft#652, snapcraft#652, snapcraft#652 === chihchun_afk is now known as chihchun [04:01] where can I read about assertions? [04:10] hmm.. I think the qmake snapcraft plugin should include g++ build package [05:23] oh man, cleanbuild can be painful [05:36] pitti: good morning! latest yakkety adt in snapd is failing because it runs out of diskspace (the container). is that a known issue? http://autopkgtest.ubuntu.com/browse.cgi/packages/s/snapd/yakkety/amd64 [05:38] pitti: and one more question - the new autopkgtest tests we have are based on spread. this is a little bit unusual as it usually drives the testsuite over ssh on a different host, for adt I point the test simply to localhost, but it still needs to connect via ssh (because that is how spread works). in my adt local vm this is all working great, however in the dc I get failures to connect to localhost [05:42] mvo, jdstrand: networkd is supposed to send the hostname on a DHCP request, so if the server keeps a hostname -> IP mapping that shold work; but if it relies on the client to save the previous lease, we don't do that across reboots right now [05:43] mvo, jdstrand: if that's needed, please file a bug (but why bother..) [05:43] PR snapd#1830 closed: firstboot: only configure en* and eth* interfaces by default [05:44] mvo: disk space failure is known, bug 1619285 [05:44] Bug #1619285: cc_growpart fails on yakkety

[05:44] mvo: we'll do a mass-retry after this is fixed, s_moser is working on it [05:45] mvo: hm, sshd should be running in the instances, that's how we connect to it after all [05:48] pitti: "Discarding adhoc:ubuntu-16.04-64, cannot connect: cannot connect to adhoc:ubuntu-16.04-64: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain", would love to play with it some more, works in my local adt qemu. and its really cool, we run 85 integration spread tests now in there which much more than with the old system, I'm super keen to land this but I need some test uploads to a real [05:48] adt to figure out why its not wokring just now. ideas appreciated (or a wiki page), I had hoped I could use yakkety but... [05:49] PR snapd#1821 closed: ensure snapd.refresh.service waits for snapd.firstboot.services [05:51] mvo: Changing password for ubuntu. [05:51] chpasswd: (user ubuntu) pam_chauthtok() failed, error: [05:51] mvo: and that's not just it? [05:52] pitti: where do you see that? that may well be it [05:52] mvo: anyway, if this happens with the current -proposed package, I can start a test manually with --shell and give you ssh access to the instance so that you can poke aroud [05:52] https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-yakkety/yakkety/i386/s/snapd/20160901_214402@/log.gz [05:52] mvo: the i386 snapd failure [05:52] mvo: amd64 runs into the growpart failure, the i386 one got further [05:52] mvo: well, where did *you* see it? [05:53] pitti: yay, I think this gives me enough, I will try some more yakkety, I had no idea that i386 would work, as long as I have one working, its all good [05:53] pitti: thank you [05:54] mvo: so, I'm just saying don't do five uploads to debug this, the latency will kill all of however much patience you still have left :) [05:55] pitti: heh, ok! *maybe* its just a single one, this log gives me some clues [05:56] pitti: should I add "breaks-testbed" because of the modifications I make? [05:56] mvo: irrelevant for the infra, but a nice warning for people trying to run it locally, yes [05:56] ta [05:56] mvo: well, actually not irrelevant if you have more than one test (this decides whether the testbed gets reset in between tests) [05:57] pitti: its just a single (large test) currently [05:58] * mvo adds it [05:59] pitti: thanks a bunch for your help, much appreciated [06:00] mvo: no worries; please let me know if you need ssh to a testbed instance [06:12] ogra_: can you please check if you have the system-boot partition on your pi2, i.e. if it shows up with ls -l /dev/disk/by-label ? its not there on amd64 so firstboot breaks in different ways [06:34] how to install lxd on all-snap image ? [06:40] mvo, nope [06:40] ogra_: its not there? [06:40] no [06:40] ogra_: then firstboot fails for this reason [06:40] ogra_: easy enough to fix, I can push a new gadget, we had the same issue on amd64 [06:40] ogra_: a world of pain [06:41] filesystem-label: system-boot [06:41] it is in gadget.yaml [06:41] ogra_: yeah but that is not the partition label [06:42] well, it uses an msdos table ... not sure we even planned for partition labels there [06:42] ogra_: no idea either, I wonder if http://paste.ubuntu.com/23123142/ helps [06:42] (that might need more code in u-image) [06:42] ogra_: I can look at the old u-d-f code [06:42] ogra_: I checked, there is code in u-i for setting the fs-label, I just suspect its the wrong one [06:42] yeash, that smells like it should help [06:42] I can see lxd here: https://uappexplorer.com/app/lxd.stgraber but can't install, am I missing something ? [06:43] ogra_: I push it, what could possibly go wrong [06:43] heh, indeed === ogra_ is now known as ogra [06:45] meh, store and bzr out of sync! [06:46] stgraber, around ? Can you tell if I can currently install lxd on all-snap image ? [06:46] (on a RPi) [07:02] hey hey [07:18] PR snapd#1831 opened: interfaces: modem-manager: ignore camera [07:26] PR snapd#1831 closed: interfaces: modem-manager: ignore camera [07:41] o/ === ant_ is now known as Guest84579 [08:06] ogra: hey, do you remmember the /etc/os-release thing [08:06] zyga, see my bug comments :P [08:08] (i linked all related bugs) [08:08] ogra: ah, I see now; too bad we cannot use variant [08:08] ogra: but whatever works :) [08:08] well, i guess we can just use variant [08:09] i personally find it cleaner ... [08:09] it sounds like the natural field to use for this [08:10] what worries me more is how we can mangle it in a clean way ... [08:10] technically the change needs to be in base-files [08:10] instead of being a hack in the build system [08:10] i'm not sure how we should do that [08:11] yep, I don't know either [08:11] update-alternatives perhaps? [08:11] uh [08:11] or dpkg-divert something [08:11] pitti, as somone who has touched base-files often ... any idea how we could cleanly mangle the content of os-release without evil hacks ? [08:12] PR snapd#1768 closed: many: automatically restart all-snap devices after os/kernel updates [08:12] zyga, well, then a sed or mv/cp is as good ... given we dont use dpkg at all on the aime [08:12] s/aime/img/ [08:13] ogra: you are right [08:13] ogra: talked to steve, my changes to the gadget were incorrect, I fixed it now in the u-i code [08:13] dpkg-divert really only makes sense regarding debs ... [08:16] ogra: new ubuntu-image snap is uplaoded [08:16] oh, its in the store already ? [08:16] hah [08:16] * ogra grabs [08:16] yep [08:17] only tested on kvm so far and need to go and work on snapd code next [08:17] error: cannot fetch and check prerequisites for the model assertion: account-key [Jv8_JiHiIzJVcO9M55pPdqSDWUvuhfDIBJUS-3VW7F_idjix7Ffn5qMxB21ZQuij] not found [08:18] hmpf [08:19] ogra: yeah, one sec [08:20] ogra: export UBUNTU_IMAGE_SKIP_COPY_UNVERIFIED_MODEL=1 [08:20] moves on [08:21] mvo, did you already publish teh new gadget ? http://paste.ubuntu.com/23123319/ ... [08:23] ogra: sorry, now [08:24] ogra: I don't actually know much about base-files, but WDYM with "mangle"? [08:24] hmm, does u-image cache something ? [08:24] still the same [08:25] pitti, Bug #1619420 [08:25] Bug #1619420: snappy removal of dpkg-query breaks lsb_release --all [08:25] and bug 1481086 [08:25] Bug #1481086: Need API/cli method to know if "is_snappy" [08:26] mvo, didnt change :/ [08:28] mvo, also, any reason why you reverted the README change in the gadget ? [08:28] ogra: sounds like a mistake [08:28] ah, yeah, it was in the "merge from steve" merge :) [08:28] ogra: yeah, that probably confuse dme, sorry, shall I fix or will you? [08:29] ogra: I uploaded a new rev24 of the pi2 snap, jsut to be sure [08:29] i can fix it, but i still get the same u-image error [08:29] * ogra retries once again [08:30] ogra: seems to be ok here, maybe a store delay? [08:30] seems to work now [08:30] at least it sits silent there [08:30] aaaand ... i got an image ... [08:30] * ogra dd's [08:30] ogra: it does? my latest snap should have proper progress [08:31] well, for the downloads [08:31] everything after is still quiet [08:31] (the image creation itself) [08:31] aha, ok [08:31] thats fine [08:31] was worried for a second [08:32] ogra@anubis:~/datengrab/images/snappy$ ls -l u-image-pi2.img [08:32] -rw-r--r-- 1 root root 4000000000 Sep 2 10:29 u-image-pi2.img [08:32] the size is really funny [08:32] we probably need something slightly smaller [08:32] yup, i have a bug open for that [08:33] iirc we reduced it in u-d-f because some usb sticks are not up to it [08:33] aha, yeah, we can make it a lot smaller indeed :) [08:33] Bug #1619362 [08:33] Bug #1619362: images should not be largely bigger than the actual content [08:33] 300MB should be enough [08:34] (effectively i'd like u-i to just compute the content, add a few MB wiggle room and roll that .. [08:34] ) [08:34] as long as we resize on first boto we can be as small as the content [08:34] *boot [08:35] * ogra tries conv=sparse for dd ... according to steve that should not write the zeros [08:36] woah [08:36] and he seems to be right ... that was a fast dd [08:36] ogra: yeah, I meant to add this to godd automatically but ENOTIME :/ [08:37] * ogra boots [08:37] reading /kernel.img [08:37] ** Unable to read file /kernel.img ** [08:37] reading /initrd.img [08:37] nope ... :( [08:37] ogra: uh, that is worse than before [08:37] yeah... waiting til i get a uboot prompt [08:37] ogra: maybe I did a incorrect uboot.env from a wrong uboot.in? [08:38] ogra: I think I regenerated it, maybe using the wrong readme? could you double check that maybe? [08:39] http://paste.ubuntu.com/23123371/ filesystem looks fine [08:40] ogra: yeah, I'm sure the env is wrong for some reaon [08:40] no snap_kernel and snap_core in the env [08:41] sounds more like u-i than the default env [08:41] all other defaults are correct [08:49] ogra: hm, hm ok [08:58] PR snapd#1802 closed: image,overlord/boot,snap: metadata from asserts for image snaps [08:59] WOAH [09:00] i just wanted tzo check the vars on yesterdays dragonboard image [09:00] snap_kernel=dragonboard-kerne.snap [09:00] how the heck did that boot at all ?!?! [09:01] * ogra reboots the board [09:01] hmm, despite the mangled value it seems to use the correct one on boot [09:01] how weird is that [09:02] snap_kernel=dragonboard-kernel_7.snap [09:02] hmm ... now it is correct [09:03] probably a serial tty glitch then [09:08] U-Boot> setenv snap_kernel pi2-kernel_11.snap [09:08] U-Boot> setenv snap_core ubuntu-core_424.snap [09:08] mvo, boots fine with that ^^^ [09:10] bah, still the cloud-init failures [09:11] * ogra goes to make coffee ... that'll take 10-15min til cloud-init gives up [09:20] ogra@localhost:~$ snap list [09:20] No snaps are installed yet. Try 'snap install hello-world'. [09:20] ogra@localhost:~$ [09:20] mvo, in that regard nothing changed ^^^ [09:24] ogra: what is the systemctl status snapd.firstboot output? [09:24] give it a bit, i just rebooted [09:25] oh come on cloud-init [09:25] sigh [09:26] this is such a pain :/ [09:29] ogra: yeah, I think I will merge the branch from cyphermox into my snap [09:29] ogra: its just terrible otherwise [09:29] mvo, http://paste.ubuntu.com/23123469/ [09:29] nothing exciting [09:30] ogra: hm, snap changes? [09:30] ogra: and snap change 1 ? [09:30] ogra@localhost:~$ snap changes [09:30] error: no changes found [09:30] ogra@localhost:~$ snap change 1 [09:30] error: cannot find change with id "1" [09:31] and i guess if i greop for the model assertion error ... i will find the same thing [09:31] ogra: oh, yeah, I suspect that is the case [09:31] ogra@localhost:~$ sudo grep model /var/log/syslog [09:31] Sep 2 08:31:11 localhost kernel: [ 0.000000] Machine model: Raspberry Pi 2 Model B Rev 1.1 [09:31] Sep 2 08:31:12 localhost snap[2612]: error: need a model assertion [09:31] yeps [09:31] ogra: we are close to having a real model assertion [09:31] but first things first ... whatever sho7uld set snap_kernel and snap_core failed to do so [09:32] this only booted because i manually tweaked them [09:32] sigh ... and the user isnt in adm ... thats also painful [09:40] mvo, ARGH ! [09:41] http://paste.ubuntu.com/23123498/ [09:41] kernel is gone ... [09:41] i assume the vfat we create with u-i is different from what u-d-f did create [09:45] ogra: its gone now? [09:45] ogra: was it there before? [09:45] ogra: and vanished on boot? [09:45] i booted it a few times, yes [09:45] looks like something did an fsck [09:45] and moved the files to fsck000X.rec [09:46] i bet it doesnt get along with "pi2-kernel_11.snap/" not being 8+3 [09:47] for whatever reason [09:47] so i assume u-i creates the wrong type of vfat [09:47] uh, the fsck ones are indeed terrible [09:47] ogra: thats alarming [09:48] well, good that we hit it now :) [09:48] i ran the former u-d-f image for about a month ... with at least one reboot per day (for the new ubuntu-core) ... so i know the u-d-f setup was 100% robust [09:49] there must be something different in the vfat creation [09:51] ogra: let me fix that [09:51] +1 [09:52] ogra: I think (because the filesystem is small) it was auto-selecting fat16 [09:52] ogra: if you can boot it, there should be a way to verify [09:52] well, its an SD :) [09:53] ogra: heh, ok [09:53] Gerät Boot Start Ende Sektoren Größe Id Typ [09:53] bah [09:53] Gerät Boot Start Ende Sektoren Größe Id Typ [09:53] /dev/sdc1 * 2048 264191 262144 128M c W95 FAT32 (LBA) [09:53] /dev/sdc2 264192 61315071 61050880 29,1G 83 Linux [09:53] hmm [09:54] hm indeed [09:56] the start byte is different when i compare to an SD from u-d-f [09:56] 2048 vs 8192 [09:56] ogra, do you know if anybody has snapped x11 for rpi? [09:56] abeato, wont happen i guess [09:57] mvo, didnt you add some magic math to u-d-f back then for exactly that ? [09:57] i remember something like that [09:57] abeato, Mir and XMir is the answer ... [09:57] we wont allow plain X11 afaik [09:58] ogra, sure, but we do not have those yet, isn't it? [09:58] (snapped) [09:58] we do have mir [09:58] but we dont have images with graphics drivers [09:59] ogra, hmm, will that happen any time soon for rpi? [09:59] between RTM and GA [10:00] ogra, git it, thanks [10:00] *got it :p [10:05] mvo, i actually think it is the mkfs.vfat call that might differ in u-i [10:05] cmd := []string{"mkfs.vfat", "-F", "32", "-n", string(part.label)} [10:05] thats what u-d-f has [10:05] oh, and [10:05] if size != "512" { [10:05] cmd = append(cmd, "-s", "1") [10:05] } [10:05] cmd = append(cmd, "-S", size, dev) [10:06] it computes the exact sector size [10:07] (and hands it over to mkfs) [10:07] ogra: oh, interessting [10:08] i have a suspicion that we use something less reliable in u-i [10:08] like mtools instead of mkfs [10:08] (because the former doesnt need root) [10:10] ogra: u-i also uses mkfs.vfat [10:10] but with different options i guess [10:10] ogra: but different parameters, let me try settings -s1 [10:11] func sectorSize(dev string) (string, error) { [10:11] out, err := exec.Command("blockdev", "--getss", dev).CombinedOutput() [10:11] if err != nil { [10:11] that is what it uses to determine the size [10:11] has there already been landings on master of snapd to prevent or change method for installing local unasserted snaps? [10:12] ogra: I think its ok to always force -s1 aiui if size is 512 -s1 is also used (implicitely) [10:12] blockdev --getss .... [10:12] sergiusens: do you remember why vfat -s1 was used? [10:12] sergiusens: in u-d-f? [10:13] mvo, note -s != -S [10:13] ah, i found --force-dangerous, dont worry :) [10:13] -s SECTORS-PER-CLUSTER [10:13] Specify the number of disk sectors per cluster. Must be a power of 2, i.e. 1, 2, 4, 8, ... 128. [10:13] -S LOGICAL-SECTOR-SIZE [10:13] Specify the number of bytes per logical sector. Must be a power of 2 and greater than or equal to 512, i.e. 512, 1024, 2048, 4096, 8192, [10:13] 16384, or 32768. [10:14] we always append -S [10:14] PR snapd#1832 opened: Added time-date-control interface for org.freedesktop.timedate1 [10:14] and if != 512 we *also* append "-s 1" [10:14] ogra: oh, indeed [10:15] hm, leaving that out for now then [10:16] ogra: not even sure that will work on a file [10:16] ogra@anubis:~/datengrab/images/snappy$ sudo blockdev --getss /dev/sdc1 [10:16] 512 [10:16] ogra@anubis:~/datengrab/images/snappy$ sudo blockdev --getss /dev/sdc1 [10:16] 512 [10:16] they are both identical [10:16] (thats u-i vs u-d-f image) [10:16] hmm [10:17] though i should probably check *before* and fsck ran [10:17] * ogra re-flashes the u-i image [10:18] s/and/any/ [10:19] yeah, stays 512 [10:31] ogra: do we have that candidate snap for ubuntu-core? or should I try to set it? [10:32] morning! [10:32] just try to set it from this mornings build i guess [10:33] mvo, 421-426 [10:33]