[02:47] sabdfl: so you're on this side of the pond now? === ikey is now known as ikey|zzz === JoshStrobl is now known as JoshStrobl|zzz [06:14] good morning [06:15] * zyga-ubuntu does PR pass [06:21] mvo: o/ [06:29] mvo: hey [06:29] mvo: I'm willing to do a deep dive on https://github.com/snapcore/snapd/pull/4030 [06:29] PR #4030: many: add internal squashfuse copy as "snapfuse" [06:29] mvo: unless you are already doing that [06:31] zyga-ubuntu [06:31] that seems like a super-bad idea [06:35] Son_Goku: hey [06:35] Son_Goku: I know [06:35] Son_Goku: but if it's sufficient let's do it the best we can: [06:35] Son_Goku: I'd discard the generated files, discard uneeded code, integrate it into our build system, ete [06:35] Son_Goku: I haven't looked yet, maybe it is huge, maybe it will end up being a handful of files that we can just maintian [06:36] *maintain [06:36] Son_Goku: and good morning :) [06:36] Son_Goku: you're up early, it's barely sunrise here [06:36] I'm about to pass out [06:36] my son is in a new school today [06:36] so I'm working from a coffee shop nearby [06:36] zyga-ubuntu: I think it's a really bad idea to do this at all [06:36] Son_Goku: sure but what is the alternative? [06:37] Son_Goku: if we have to do it, we have to do it [06:37] Son_Goku: we canont just change apt defaults overnight (even though I agree they are bad) [06:37] Son_Goku: those should have been changed years ago [06:37] Son_Goku: but I think that now it is too late for that [06:37] well, if people are updating graphically or using `apt upgrade`, they'd get it [06:38] just fix apt-get upgrade's behavior [06:38] Son_Goku: yes but there is always a set that doesn't [06:38] Son_Goku: I think we've seen that already with snapd before (and we reverted a change that triggered that) [06:38] Son_Goku: it will also be a (I suspect) endless discussion in debian [06:38] Son_Goku: I think it's risky business to go and change that for snapd's sake [06:39] Son_Goku: (I'd still change it but on separate schedules) [06:39] Son_Goku: and once snapd can reliably depend on things, let's yank this [06:39] then don't integrate it into the source tree [06:39] just carry it as a separate tarball bundle [06:39] mmm [06:39] that way it's harder to be lazy about it [06:39] that's interesting, perhaps that's a better idea indeed [06:40] Son_Goku: i'd require some tweaks to CI but yes, it's nicer this way [06:44] zyga-ubuntu: I wish you'd do this with a vendor overlay tarball too [06:44] basically, a tarball that only contains snapd-/vendor/ [06:45] Son_Goku: yes, I think that's a natural place to do this for releases [06:45] Son_Goku: I'll talk to mvo [06:46] this would allow me to ship the snapd-.tar.gz (or in debian parlance, snapd-.orig.tar.gz) and snapd-.vendor.tar.gz in distgit as sources while not using the vendor tarball on Fedora [06:47] which would open the door to adding an EPEL build [06:47] (aka CentOS) [06:47] zyga-ubuntu: a vendor tarball? we can do this easily [06:48] I asked for this a year ago too, btw [06:48] :( [06:48] sorry [06:48] must have forgoten [06:48] the experimental CentOS builds are super-brittle :( [06:49] zyga-ubuntu: incidentally, for your amusement, with literally no expectation of anything working: https://copr.fedorainfracloud.org/coprs/ngompa/snapcore-el7/ [06:49] Son_Goku: so snapd_2.28.5.{pristine,vendor}.tar.xz? a [06:49] mvo: yep [06:49] sounds fine with me :D [06:50] ok [06:50] * mvo will update his release script [06:50] mvo: basically, I need the base tarball to be closest to what git repo looks like, otherwise patches don't apply [06:51] mvo: though you don't have to do weird underscores and whatnot :) [06:51] unlike debian, I don't have such restrictions :) [06:51] Son_Goku: heh, ok [06:52] o/ [06:52] it can just be snapd-2.28.5.tar.xz and snapd-2.28.5.vendor.tar.xz [06:53] Son_Goku: ok, I may need to keep the current convention for people using our GH release page, so it might be snapd.vendor-only.tar.xz or I will just annouce the new format and assume that not a lot of people need to adjust their scripts [06:54] sure [06:54] that's fine [06:54] I don't particularly care if you only just upload a vendor-only tarball [06:54] as I can just get the regular one from the GitHub link as I do now [06:54] mvo: I think it's just a handful of people, we'll be fine [06:54] I just need the vendor code in a separate tarball that I can extract [06:56] PR snapd#4047 opened: snap-confine: init all arrays with `= {0,}` [07:08] mvo: .5 looks good so far [07:09] * zyga-ubuntu tweaks details on https://github.com/snapcore/snapd/pull/4008 [07:09] PR #4008: cmd/snap-update-ns: create missing mount points automatically [07:12] zyga-ubuntu: test and give karma for Fedora snapd updates [07:12] they're on 2.28.5 now [07:12] zyga-ubuntu: could you or pawel try to reproduce the error on the pi3 that sergio got? I'm still nervous about it. otoh I have no report other than from sergio about it and would love to get that confirmed [07:19] mvo: I cannot do that today, sorry, I won't be home until after 18 [07:19] mvo: it seems the error cachio got was repeated on >1 SD card he had [07:20] maybe it's just bad luck but it does seem to indicate more than random issue [07:20] Son_Goku: only after I'm home, sorry :/ [07:21] mvo: though my pi3 is still up [07:21] mvo: I could try to test something remotely [07:23] pstolowski: hey [07:23] pstolowski: is your pi3 operational? [07:24] mvo: my pi3-1 is operational, though I cannot reflash it remotely [07:31] zyga-ubuntu, morning! yes, it is [07:31] pstolowski: I think mvo needs a little bit of help and I'm not at my office today [07:31] 09:12 < mvo> zyga-ubuntu: could you or pawel try to reproduce the error on the pi3 that sergio got? I'm still nervous about it. otoh I have no report other [07:31] than from sergio about it and would love to get that confirmed [07:31] pstolowski: I can only run spread tests on pre-installed remotely [07:32] pstolowski: last week cachio ran into a bug where smoehing got corrupted in the boot process [07:32] pstolowski: it might be related to the SD card he was using (he tried two) [07:32] pstolowski: we need hands on testing to see if we can reproduce it [07:35] zyga-ubuntu, sure, happy to help. i need to reflash with latest edge image correct? [07:35] pstolowski: I think so, ideally build the image as cachio did [07:35] * zyga-ubuntu looks [07:36] pstolowski: look at this thread please: https://forum.snapcraft.io/t/snap-mode-try-on-raspberry-pi/2454 [07:38] zyga-ubuntu, k === kira is now known as Guest92109 [07:41] hello got stuck in network configuration can any one please help [07:42] Guest92109: hey, can you please ask a more precise question? [07:42] zyga-ubuntu: Sure [07:42] pstolowski: I commented on the thread with details of what I know [07:42] zyga-ubuntu, thanks [07:42] I got a machine with ubuntu core15.10 [07:43] I need to configure eth0 [07:43] I teied to edit /etc/network/interfaces file but it gives system is read-only message please help [07:46] Guest92109: 15.10 you say [07:46] hmmm [07:46] Guest92109: may I ask if you really want to use 15.10 or can you consider updating to ubuntu core 16? [07:47] zyga-ubuntu, what are the steps to build an image? [07:47] Guest92109: I don't think we support 15.10 anymore (apart from commercial contracts) as it was an early version and ubuntu core 16 has received immense fixes, features and improvements since [07:48] pstolowski: I think you need ubuntu-image and the pi3 model, one sec [07:48] pstolowski: $ sudo ubuntu-image -c beta -O pi3-beta.img pi3.model [07:48] - I am using dell edge gateway 5000 series which comes with preinstalled snappy core 15.10 and need to configure same [07:49] Guest92109: it used to ship with 15.10 with the desire to update to 16-series, new releases are only using 16; in any case let me try to find someone who can help you [07:50] pstolowski: you need the pi3.model assertion [07:50] pstolowski: cachio sent a link somewhere to a repo with test scripts but I cannot find it [07:50] mvo: do you happen to remember how to fetch the pi3 model file [07:51] Guest92109: I've asked around but I don't know how soon I'll get an answer [07:51] Guest92109: may I suggest that you go to forum.snapcraft.io and ask your question there [07:51] zyga-ubuntu: thanks in advance would be a great help i am not getting any help onlien [07:52] Guest92109: please double check that you cannot upgrade your device to 16 series, then we can all help you easily [07:52] Guest92109: I think 15.10 is not supported anymore (apart from contractual obligations) and everyone is strongly encouraged to go to 16 [07:53] zyga-ubuntu: Sure thanks but can u just answer is the read-only file system also persist in 16 series [07:53] While configuring /etc/network/interfaces file [07:53] Guest92109: yes although network configuration in 16 series is different and much much easier [07:54] Guest92109: apart from a menu-driven wizard we also use declarative YAML based nplan (netplan) configuration [07:54] Guest92109: while the system as a whole is read only the set of writable files is different [07:54] Guest92109: (between 16-series and 15.10) [07:55] kissiel: ^ [07:55] kissiel: I think you are experienced with the dell gateways more than I am [07:56] zyga-ubuntu: Thanks can i get the steps of 16 series, i will get the idea regarding the configuration [07:57] Guest92109: it's a simple process you follow using the on-screen menu (also availale via serial port) [07:57] Guest92109: you can run "console-conf" after initial setup if you wish to change the configuration [07:57] Guest92109: I'd poke around network-manager snap [07:57] kissiel: note, Guest92109 is using 15.10, I'm not sure we still ship that snap for 15.10 [07:57] zyga-ubuntu, pstolowski sorry, was distracted, do you still need the pi3 model assertion ? i [07:58] pstolowski: "snap known --remote model series=16 brand-id=canonical model=pi3" [07:58] zyga-ubuntu: ah, right. maybe joc will know more, I'll ping him when he gets online [07:59] mvo: thank you, I had a copy but I forgot how I got it [07:59] Guest92109: joc is in the UK timezone, he should be around soon [08:01] zyga-ubuntu: Thanks a lot for ur support one last thing how would i know if he comes online [08:02] mvo, zyga-ubuntu thanks! so, I'm going to rebuild the pi3 image, flash it, configure pi3, then add the assertion on it, then run ubuntu-core-upgrade spread test on it in a loop; did i miss anything? [08:03] pstolowski: the assertion is needed to build the image [08:03] ah! [08:03] pstolowski: you need to set up the pi for external backend [08:03] ok [08:03] pstolowski: this is described in tests/external-backend.md [08:04] pstolowski: have a look, you just flash & first-boot as usual [08:04] pstolowski: then run a few scripts [08:04] pstolowski: (from the tree) [08:04] pstolowski: and then you are ready for testing [08:04] pstolowski: ask if you need help, I did this on Friday [08:05] zyga-ubuntu, oh yeah, i know that, just unclear on building the image + assertion part [08:06] pstolowski: yes, what zyga-ubuntu said and good luck [08:06] <__chip__> morning peeps [08:07] __chip__: hello, good morning [08:07] <__chip__> i'm on the road today -- doing admin stuff wrt the boys' schooling [08:07] <__chip__> got my tablet with me (thus the nick) [08:07] __chip__: I'm on the road too, and my son changed school (well, maybe, trial week) [08:07] __chip__: fingers crossed for both of us : [08:07] :) [08:08] <__chip__> zyga-ubuntu: hah. this is about changing schools also (looking into a different school) [08:08] <__chip__> so, yeah, good luck to us [08:08] mvo, zyga-ubuntu ok, all clear, i see where the assertion is needed now, thanks [08:09] mvo: if you want I can keep running that test on my pi3 but I think it depends on SD card model [08:09] mvo: and some models corrput something [08:09] mvo: otherwise why would it never fail vs always fail? [08:10] ok, got the image built, falshing [08:12] zyga-ubuntu: did you build the image yourself when you did the test? or download a pre-made one? maybe its a specific combination of the ubuntu-image cmdline/kernel/gadget or something [08:13] mvo: no, I ran the pre-made one [08:14] mvo: ah, indeed [08:14] mvo: could be! [08:17] <__chip__> pedronis: are you working today? [08:22] morphis: Guest92109 here has a dell gateway with ubuntu 15.10 and need help, can you please try to render assistance? [08:23] * zyga-ubuntu is on the move again, I'll be online in ~~ hour [08:23] ttyl [08:25] Bug #1723881 opened: [Feature Request] Support pre-invoke and post-invoke commands as DPkg::Pre-Invoke and DPkg::Post-Invoke in APT [08:48] Guest92109: hi, i see you had some questions about the 5000 series gateways [08:53] mvo, it failed for me, but with slightly different output http://pastebin.ubuntu.com/25751741/ [08:54] Error: "snap_mode" not defined ? [09:05] mvo, it seems that re-running the test again after the above failure gives another unexpected failure, i assume the box is now in a bad shape and needs re-flashing before attempting tests again? [09:06] pstolowski: looking [09:07] pstolowski: could you please paste the other failure too? [09:07] pstolowski: this is very different from what cachio got, it went much further [09:08] mvo, the other failure after I re-run the tests: http://pastebin.ubuntu.com/25751799/ [09:08] pstolowski: and a different error, I would love to compare the revisions of core/gadget/kernel you have and the revision of ubuntu-image to see where things are different [09:08] mvo, external:ubuntu-core-16-arm-32 .../tests/main/ubuntu-core-upgrade# snap list [09:08] Name Version Rev Developer Notes [09:08] core 16-2.28.5 3214 canonical core [09:08] pi2-kernel 4.4.0.1075.75 43 canonical kernel [09:08] pi3 16.04-0.5 6 canonical gadget [09:09] pstolowski: maybe we can put it into sergios forum post? this way we can compare notes more easily [09:09] sure [09:10] pstolowski: the followup looks indeed just like a test issue with a strange state, might be enough to just "rm curChg" in the debug shell and run again [09:11] pstolowski: thank you, this looks promising, it does not fail in the omg-cant-explain-what-is-going on way at least (which is the case for sergios error, I can't make any sense of it) [09:12] mvo, I hope I didn't miss anything in the setup. will re-try from scratch in a while [09:13] pstolowski: if you could rm curChg and run again, that would be nice [09:14] pstolowski: miss something> well, if its really hard to reproduce it means its not wide-spread at least. i.e. we can't reproduce it with the pre-build images, we can't reproduce the same error with ubuntu-image, so it must be sometihng special [09:14] pstolowski: well, hopefully [09:18] Guest92109: joc is online, you could consult him [09:25] anyone else hitting https://bugs.launchpad.net/snapcraft/+bug/1723898 ? [09:25] Bug #1723898: Unable to build snaps using python plugin - pypi.debian.net no longer accessible [09:25] for the life of me I can't actually figure out where snapcraft is configured to use that [09:26] hrm, hrm, interfaces-cups-control test is busted [09:26] (in debian-unstable) [09:26] re [09:26] (and right on time) [09:28] zyga-ubuntu: heh :) debian-unstable cups control test is busted, that rings a bell, iirc we had artful broken as well recently. anyways, I'm debugging [09:28] kissiel: Thanks [09:29] joc: Hello I have a dell edge geteway with ubuntu 15.10 snappy core [09:30] I am having difficulty in configuring the network using the traditional ubuntu method in the interfaces file [09:30] as the system is read-only [09:33] Guest92109: if i could first make a recommendation: if you are just beginning a project then zyga & kissiel were correct in urging you to update to Series 16 Ubuntu Core, we have been working with Dell to transition customers for serveral months to Series 16 [09:34] Guest92109: in addition I believe Dell announced that your version of Ubuntu Core would go End Of Life on September 30th so you will not receive further security updates [09:35] mvo: ack [09:35] mvo: let me know if I can help in some way [09:35] mvo: I'm updating my branches [09:36] Guest92109: in the meantime the supported method for configuring the network is to use network-manager which should come preinstalled on your system [09:36] zyga-ubuntu: yeah, I keep you updated, waiting for a debug shell right now. a cheap way would be to simply disable the test [09:36] zyga-ubuntu: on debian [09:36] mvo: is there a thread? [09:36] zyga-ubuntu: otoh, it will probably come back to use for artful+1 [09:36] zyga-ubuntu: not yet, I just noticed with my most recent PR that it fails [09:36] * zyga-ubuntu was reading something while stuck in transit and is a bit freaked out about this: [09:36] ok [09:37] http://papers.mathyvanhoef.com/ccs2017.pdf [09:37] mvo: note, our spread image differs for linode and qemu [09:37] mvo: qemu uses, I believe, real sid, while linode is using stretch [09:37] mvo: I wanted to fix this a while ago but I wans't sure how [09:37] zyga-ubuntu: yeah, I'm running against linode just to be sure [09:38] zyga-ubuntu: however there was a new upstream release of cups to unstable yesterday so most likely fallout from that [09:38] joc: thanks for your valuable information [09:38] Will contact them and do the same [09:39] Can i get a doc where complete step is mentioned how to open network manager and configure [09:39] mvo: aha [09:40] mvo: that paper I linked to looks like a massive security issue that affects everything using wifi, expect surge of updates everywhere :/ [09:40] I am new to core and tried various commands using snappy but could not get the setting . Thanks in advance [09:40] morphis: I think you need to update hostapd [09:43] Guest92109: configuration should be via the "nmcli" command [09:44] pstolowski: can you try the test a few more times? [09:44] joc: sure will check and let u know :) [09:44] Guest92109: there should be plenty of documentation for that in mangpages and on the internet, I would have to check with our Dell liasons whether there is some official docs for customers [09:46] joc: Thanks a lot would be a great help [10:11] PR snapd#4048 opened: tests: fix interfaces-cups-control test for cups-2.2.5 [10:12] mvo: +1 :) [10:12] zyga-ubuntu: ta [10:12] zyga-ubuntu: silly idea - could we simply govendor the squashfuse binary :) ? [10:13] zyga-ubuntu: it looks like govendor does not care if the git repo is go [10:13] mvo: mmm, dunno, how do you envision that? a go binary linking to something that is squashfuse itself or just a "govendor sync" repo that keeps all of squashfuse code but is unrelated to go [10:14] zyga-ubuntu: the later, just to have the git checkout and we do the building ourselfs [10:14] I like that much more than the PR [10:14] * kalikiana coffee [10:15] zyga-ubuntu: its slightly crazy but maybe a nicer option than the massive embedding [10:16] mvo: yes, certainly nicer [10:16] *certainly [10:33] mvo, urgh ... i think i found your reason for the upgrade test issues ... [10:33] mvo, https://dashboard.snapcraft.io/dev/snaps/5596/configurations/ [10:34] candidate and beta use the stable gadget (ancient firmware and uboot) [10:34] if you dont start from edge this wil most likely always fail with newer kernels [10:35] pstolowski's last comment on the thread helped a great deal ... [10:36] mvo, if you dont mind, i'll switch beta and candidate to the latest edge gadget [10:37] ogra_, oh, the versions? [10:37] pstolowski, yeah, your pi3 is the stable one [10:37] (pi3 snap) [10:38] ogra_: lets not change it just yet, lets see if cachio can refresh to edge gadget to make the issue go away [10:38] ogra_: but it sounds great [10:38] ogra_: I mean, great in the sense that it sounds like a plausible theory [10:38] ogra_: and also would explain all the oddness around the issue [10:39] mvo, well, he'Äd have to build an edge image and then downgrade kerne/coe to the espective target channels he wants to test [10:39] mvo, what doesnt go well with that theory is that it only started recently [10:39] ogra_: yeah, he says it started recently, hm, hm [10:40] there must be another aspect to it since the gadget channel setup has never changed after the first stable one [10:40] but it is defintely one of the issues [10:41] ogra_, hmm, ok all the rpi3 stuff is new to me, so please bear with me... I didn't miss any setup test, did I? [10:42] pstolowski, well, if i wanted to test beta i'*d also just builld a beta image, the issue is on ou side, not on yours [10:42] *our [10:42] ok [10:43] ogra_, interestingly, i re-flashed and testing again, it a 3rd run of the test and so far so good [10:43] *it's a 3rd [10:43] pstolowski: ha! [10:43] and pi3 is still at rev 6 ? [10:43] pstolowski: so you and cachio need to compare notes what is different [10:44] ogra_, yes, rev 6 still [10:44] ++ fw_printenv snap_mode [10:44] ## Error: "snap_mode" not defined [10:44] ogra_: lets try to find what this other factor is, then we need to fix the gadget to match the beta kernel. but lets try to get to the bottom of this first [10:44] thats the thing that made me curious ... [10:44] the stable gadget has still all the old variablle names [10:45] the test should fail exactly there ... [10:45] (from http://pastebin.ubuntu.com/25751741/) [10:45] github feature idea: mark a PR as "unbreaks master" and stop testing and landing all other PRs [10:45] could help us with cases like the one today [10:46] do we have any friends there? [10:47] ogra_: hm, things should be ok if snap_mode is not definied at all (unless I miss something). when it needs to get set, snapd will set it. in what line in http://pastebin.ubuntu.com/25751741/ should it fail you mean? [10:47] mvo, sure, the point is that snap__mode doesnt even exist ... which is true for the ancient gadget but not true for all newer ones [10:48] mvo, in the old gadget only "snappy_mode" is set [10:48] the price of renaming [10:49] mvo, indeed just the missing variable isnt an issue but an indicator that yoou run old and incompatible binary bllobs and uboot [10:49] is eternal backwards compatibilty support code [10:49] ogra_: aha, yes [10:49] (said with the voice of malcom mcdowell) [10:49] *malcolm* [10:51] mvo, ogra_ ah, no, it eventually failed again http://pastebin.ubuntu.com/25752176 [10:51] PR snapd#4030 closed: many: add internal squashfuse copy as "snapfuse" [10:51] Read error on /boot/uboot/uboot.env: Success [10:51] something is not setting errno [10:52] pstolowski: if you hexdump -C /boot/uboot/uboot.env - what happens? [10:53] mvo, note that this can actuallly be a eal corruption issue due to the ancient uboot vfat version in stable (vs very new kernel in beta and candidate) [10:53] mvo, I exited the spread test shell, is it still useful if I do that after spread restored the stuff? [10:54] pstolowski, just call fw_printenv alone [10:54] pstolowski: I think so [10:54] just ssh into it [10:54] pstolowski: I think so too [10:54] the test proobably swallows output [10:54] ogra_: yeah, corruption of the vfat would make a lot of sense. what version of uboot do we have there and in edge? might be worthwhile to do a quick scan of the uboot git log to see if anything related to vfat is there [10:55] ogra_, mvo, zyga-ubuntu fw_printenv output: http://pastebin.ubuntu.com/25752195/ [10:55] mvo, some 2015 version iirc, we neve properlly documented it and it isnt built fom source [10:55] that onlly changed in edge when we switched to actual uboot source builds [10:56] pstolowski, well, that doesnt look corrupt at all [10:56] pstolowski: the first call looks very short [10:56] mvo, seeing that i take back my vfat theory :P [10:56] mvo, yeah, ignore it, probably I made a mistake copying [10:56] there must be some issue how the test uses fw_printenv instead [10:57] ogra_: well: "+ fw_printenv [10:57] Read error on /boot/uboot/uboot.env: Success" is the error from the test [10:57] pstolowski: anything in dmesg? [10:57] pstolowski: related to read error or so? [10:58] mvo, snappy_mode in there? [10:58] mvo, yes ... but nmanualy it woks [10:58] pstolowski, yeah, eftover [10:58] *left [10:58] ogra_: yeah, still odd, how could the test use it incorrectly? also no errno set :/ [10:58] pstolowski, there should also be a snap_mode *if* you ae in any of try/trying ... it gets unset afterwards [10:59] PR snapd#4048 closed: tests: fix interfaces-cups-control test for cups-2.2.5 [10:59] mvo, well, i dont know, but manually running it obviously works ... perhaps missing env ? or does the test unmont snappy-boot at some point ? [11:00] ogra_: I don't think it does [11:00] mvo, nothing stands out in dmesg.. except for [ 6.989717] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities [11:00] [ 7.004959] EXT4-fs (mmcblk0p2): couldn't mount as ext2 due to feature incompatibilities [11:00] pstolowski: what ogra said, snappy_mode= can be ignored [11:01] pstolowski, thats /writable ;) and normal noise from the ext4 driver looping over fs capabilities [11:01] mvo: could it be (in any way) related to the fact that more recent ubuntu versions use incompatible feature set for ext4 [11:01] (i whish we could just quieten that) [11:01] csum_metadata [11:01] chsum_metadata* [11:01] zyga-ubuntu, wrong path :P we talk about a vfat ;) [11:02] ogra_: p2 is vfat? [11:02] ogra_: I thought p1 is [11:02] zyga-ubuntu, system-boot is [11:02] everywhere [11:05] mvo, the original test ooutput has "mesg: ttyname failed: Inappropriate ioctl for device" https://paste.ubuntu.com/25728858/ [11:05] mvo, i wonder if that could inflluence fw_printenv functionality somehow [11:06] ogra_: I think thats something from spread actually, but it might be, I am looking at fw_printenv from the uboot git now [11:06] mvo, better look at the one fgrom the deb source [11:06] thats what we use [11:06] (and it might be far behind git) [11:06] ogra_: the xenial one? [11:06] yep [11:08] 2016.01+dfsg1-2ubuntu3 ... nearly 2y old [11:08] how can 2016 be nearly two years old? [11:08] well, [11:08] nearly ok [11:08] january 2016 ... [11:09] (i could have said 22 months if you'd prefer that :P ) [11:10] my tests also had a couple of 'mesg: ttyname failed: Inappropriate ioctl for device' errors [11:10] yeah [11:10] must not be the issue but it might indicate that you tty is somewhat different to the one you have when running fw_printenv manually [11:11] something is surelly with the env [11:16] pstolowski, ogra_ http://git.denx.de/?p=u-boot.git;a=blob;f=tools/env/fw_env.c;h=ab06415898c2f718e8a926879379b1f8d74ca55d;hb=HEAD#l760 is the only place where this error is produced (looks the same in the 2016 version) [11:16] too bad it doesn't display rc [11:17] pstolowski: I would argue the code in there is incorrect [11:17] it may be just a partial read [11:17] zyga-ubuntu: yeah, exactly [11:17] (truncated environment?) [11:17] zyga-ubuntu: you mean something else truncated the uboot.env? [11:17] mvo: last week we learned how error messages suck [11:17] mvo: well, not sure if really truncated [11:17] mvo: I didn't read the rest of the code [11:17] mvo: I meant that if you have a fixed buffer [11:17] mvo: and then read until EOF [11:17] well, uboot.env isnt truncated if you manually run fw_printenv [11:18] mvo: then eventually you will get a partially complete buffer [11:18] zyga-ubuntu: there might be a signal [11:18] mvo: unless there's some assumption about the alignment of the file [11:18] mvo: "alignment" in blocks [11:18] zyga-ubuntu: but yeah, I think the code in there is slightly naive [11:18] mvo: might be but we don't know [11:18] zyga-ubuntu: yeah, sadly it does not print rc which is the crucial bit [11:18] s/is/would be/ :/ [11:18] mvo: maybe we just need to patch snapd to always write the environment in multiples of some block size [11:18] mvo, zyga-ubuntu, but why does it work manually then ? [11:18] mvo: and fill the rest with zeros [11:19] ogra_: I don't suppose it is corrected by that process somewhere [11:19] ogra_: maybe it fails, then gets fixed, and then is correct [11:19] ogra_: we don't have snapshots of the file during the process to say [11:19] well, you could stop the test before it calls fw_pintenv the first time i suppose [11:20] and then manually run the command to see [11:21] pstolowski: could you please rerun the test just to see if it fails in a similar way again? plus please update the forum post so that all the "evidence" is in a single place [11:21] ogra_: it doesn't fail for me, I cannot reflash beta sadly (AFK) [11:22] zyga-ubuntu: I still suspect a signal that might cause the short read, but what signal could it be? usually it would be something like SIGWINCH but that is probably not it on a spread run and no other signal springs to my mind [11:23] phone [11:23] * mvo lunch [11:25] re [11:26] mvo: so I was thinking that it could be a signal _but_ I'm somewhat skeptical; let me check something [11:27] SIGWINCH 28,28,20 Ign Window resize signal (4.3BSD, Sun) [11:27] SIGWINCH is ignored by default [11:27] i guess we dont call out to fw_setenv when writing thhe file but use our own write finction ? [11:27] *function [11:28] (from snapd that is) [11:28] Interruption of system calls and library functions by signal handlers [11:28] If a signal handler is invoked while a system call or library [11:28] function call is blocked, then either: [11:28] * the call is automatically restarted after the signal handler [11:28] returns; or [11:28] * the call fails with the error EINTR. [11:28] ogra_: I think so [11:29] If a blocked call to one of the following interfaces is interrupted [11:29] by a signal handler, then the call will be automatically restarted [11:29] after the signal handler returns if the SA_RESTART flag was used; [11:29] otherwise the call will fail with the error EINTR: [11:29] * read(2), readv(2), write(2), writev(2), and ioctl(2) calls on [11:29] "slow" devices. A "slow" device is one where the I/O call may [11:29] block for an indefinite time, for example, a terminal, pipe, or [11:29] socket. If an I/O call on a slow device has already transferred [11:29] some data by the time it is interrupted by a signal handler, then [11:29] the call will return a success status (normally, the number of [11:29] bytes transferred). Note that a (local) disk is not a slow device [11:29] according to this definition; I/O operations on disk devices are [11:29] not interrupted by signals. [11:32] so I think the C code for uboot env handling may not be setting SA_RESTART in any place [11:32] * zyga-ubuntu looks [11:34] mvo, will do [11:39] mvo: still no idea which signal it might be :/ [11:39] ideally we'd rebuild that old uboot [11:39] and set a breakpoint and see [11:39] but ... [11:39] well [11:39] maybe we can? [11:39] pstolowski: can you reproduce this? [11:39] pstolowski: or does it happen only once? [11:40] pstolowski: ideally we'd have a copy of the binary that has this problem and the file that causes it [11:40] pstolowski: and then see what gdb tells us [11:40] it prints the build date as very first line duing boot on serial [11:41] ogra_: any chance we can grab debug symbols for that build somewhere/ [11:41] nly via re-building ... i thinbk it gets stripped by default (it is a bootloade optimized for size after all) [11:42] ogra_: any -dbg package equivalent? [11:42] noot reallly [11:43] with the new setup it would be easy since we now build from upsteam tags ... stable was random local builds originally ... we need the string it prints, with lluck that had the commit in the version [11:44] *has [11:44] aha [11:44] well, that depends on if pstolowski can reproduce this :) [11:44] ogra_: could you provide a build for pawel to try [11:44] ogra_: just that command with debug symbols with ~~ similar version (as close as you can get it) [11:45] https://www.denx.de/wiki/DULG/DebuggingUBoot [11:46] seems it is actually not stripped (nothing theer says you need a debug build) [11:47] pstolowski: can you run gdb and see if that build has symbols? [11:47] https://www.denx.de/wiki/view/DULG/Debugging is a bit more informative and llinks to tips and tricks [11:51] * zyga-ubuntu breaks, I'll probably miss standup but I may show up if lucky [11:58] just re-flashed to retest [12:16] re [12:21] Pharaoh_Atem: hey, can you remind me that repo sync issue in fedora/linode [12:21] Pharaoh_Atem: linode operators are in #linode but on OFTC, not freenode [12:22] Pharaoh_Atem: if you can give me those details again I'll collect everything and open a ticket [12:26] zyga-ubuntu, pstolowski just finished reading backlog. lets see if it is reproducible and then we attack it [12:27] mvo, ... and i'm having issues now on 1st boot after flashing. it's taking forever on 'random: nonblocking pool is initialized' [12:28] not sure if this is entropy issue or what. console-conf doesn't show up [12:28] no, thats a red herring [12:28] it is simply the last thing the kernel prints before switching consoles [12:28] pstolowski: hm, is the network cable attached? [12:28] mvo, no [12:29] that might then take 2min for the timeout [12:29] ogra_, i've been waiting much much longer already [12:29] hmm [12:30] mvo, i was actually wobndering if we should rip out the hardcoded DHCP default for eth0 that we add during build [12:30] at least for physical images (probably cloud still needs it) [12:31] mvo: I'll be away from home for at least three hours, after that I can spend any time debugging and exploring this with hands on [12:31] mvo: I'll try to focus on coding but I can help pstolowski with gdb [12:32] zyga-ubuntu: thanks, I think we are good for now, if pstolowski can reproduce that is good enough for now [12:48] ok, i reflashed and this time it didn't hang on boot. weird. [12:54] pstolowski: does the uboot env CLI tool fail the same way? [12:54] * zyga-ubuntu logs into ubuntu session, brb [12:56] zyga-ubuntu, not yet, the test is running [12:56] aha [12:56] * zyga-ubuntu joins the standup [13:14] PR snapcraft#1615 closed: schema: improve invalid app, hook, and part errors [13:16] sergiusens: hi - raised https://bugs.launchpad.net/snapcraft/+bug/1723945 based on the conversation we had last week [13:16] Bug #1723945: provide support for use of DEB_HOST_MULTIARCH in environment variables [13:16] does that sounds about right? [13:29] Issue snapcraft#1445 closed: ROS user oriented story [13:29] Issue snapcraft#1446 closed: MOOS user oriented story [13:29] PR snapcraft#1577 closed: lxd: don't inject local snaps on a different arch [13:32] PR snapcraft#1610 closed: tests: reenable the cleanbuild integration test [13:35] PR snapcraft#1609 closed: tests: remove the duplicate nodejs integration tests [13:44] jamespage yeah, that should cover it [13:45] sergiusens: also what would you think of https://bugs.launchpad.net/snapcraft/+bug/1723957 [13:45] Bug #1723957: cleanbuild: add PPA sources [13:45] I'm using cleanbuild to produce test binaries; currently hacking in my ppa sources using cloud-init in lxd user.user-data config [13:51] I don't understand this behaviour at all: https://forum.snapcraft.io/t/cleanbuild-remote-on-pi-grabs-wrong-arch-lxc-image/691/17 [13:52] cleanbuild should not be forcing the version of snapd to anything EXCEPT "the latest" stable, unless the user specifically tells snapcraft otherwise [13:53] and doing the forcing by injecting host-side snaps is just plain wrong [13:53] IMO [13:53] and no, that opinion is not humble in any way :-p [13:55] pstolowski: how is that test going? the second run of the core-refresh testing? [13:56] diddledan: snapd? do you mean the core snap? snapcraft attempts to install the same version in the container that's being used on the host [13:57] mvo, well.. [13:57] yeah, core [13:57] mvo, http://pastebin.ubuntu.com/25753195/ [13:57] diddledan: so if eg. you're using beta on the host to test something, the container will be using that, same goes for snapcraft itself [13:58] mvo, last failure unrelated, probably wifi dropped or something [13:58] I think it's wrong to be injecting at all, foreign arch or not. Core should be updated using snap processes not by injecting random snaps. We should not be designing weird behaviour for the edge case of "I want to test something" by default [13:59] an "I want to test something" should be an opt-in not a default [13:59] pstolowski: do you have a monitor attached, i.e. can you actually see what is on screen when its in this "2017-10-16 15:41:20 Error debugging external:ubuntu-core-16-arm-32:tests/main/ubuntu-core-upgrade : EOF" state? [14:01] the whole point of cleanbuild is that the build is in a clean environment. The moment you start injecting stuff to match the host it is no-longer "clean" [14:01] diddledan: I wouldn't agree it's "random". If you're using snapcraft natively without a container, it uses whatever core you are running. The experience should as much as possible be the same [14:03] You should of course be testing that your snap works with stable before you release it, in both cases [14:03] mvo, no, no monitor (but i've usb-serial). but hmm.. i cannot access this pi3 via wifi anymore [14:03] maybe if you think we should be doing this, the name "cleanbuild" is wrong. So let's rename it to "meh, frankenstein, YMMV" [14:03] mvo, if serial console doesn't lie, then the last thing that happened was systemd[1]: Started Journal Service [14:04] pstolowski: hm, that looks harmless [14:04] pstolowski: I wonder why it stops responding [14:04] pstolowski: fwiw, cachio saw some of these errors too (EOF) but we never got to the bottom of it, might actually be multiple causes just showing the same error [14:04] mvo, wifi seems dead, no led blinking [14:05] and let's change the documentation to say "this is not a clean environment. any attempt to get a system as close to build service is never going to work. We made it this way on purpose because we hate consistency" [14:07] diddledan: Can we take a step back and you tell me what your expectation of cleanbuild is? If this is your impression that's certainly not what anyone wants :-) [14:10] mvo, I just powered it again; on normal boot that Systemd Journal Service message is followed by brcmf initialization, which apparently didn't happen [14:10] my expectation is that cleanbuild builds in a clean environment. Is it wrong to assume that when the documentation says "cleanbuild Create a snap using a clean environment..." that it will build using a clean environment? [14:11] pstolowski: hm, do you think you could put it on a cable connection? to make tests more reliable? [14:12] mvo, yeah, that might be a good idea [14:13] also, when the documentation says "Create a snap using a pristine environment managed by lxd" I expect the environment to be unaltered [14:14] I certainly do NOT expect for implementation in my host to "leak" through into the LXD container [14:16] diddledan: So even if you're doing, say, `snap install core --edge`, you wouldn't want the container to use the same channel? [14:16] no [14:16] diddledan: Would you apply the same reasoning to persistent containers? [14:17] if I did that then I would want a switch to tell snapcraft to change to using a different channel [14:17] yes, I would [14:18] diddledan: One thing we could look into is a prompt like `You're using the "core" snap from the "edge" channel. Do you want to use the same in the container?` [14:18] most people building snaps are NOT going to want to change the core snap to test things because they're not developing the core snap nor snapd nor snapcraft [14:19] if they really need to test their snap builds against a different core then they will know that they want a specific core and should then be tasked with telling snapcraft that specifically [14:21] diddledan: Right. We have slightly different perspectives here. To my mind, the same core snap increases predictability. But I see the point you're making [14:24] diddledan: Would you mind opening a new forum topic for it? [14:24] the problem that you wrote the PR in the link above with core being injected to foreign arch snaps is a symptom, not the cause. the cause is that we're assuming the user wants to match their host with their build. Now your PR says "match the host EXCEPT on remote builds." where are these exceptions going to end? why not just allow the user to specify the core they want to use rather than guessing [14:25] diddledan: The PR applies to both the core and snapcraft. We have to do this anyway to be able to get the correct snapcraft version. [14:25] If we decide to not push the core snap out of the box, that's a separate decision [14:26] Or maybe add a way to choose the channel explicitly etc. [14:28] diddledan: To be clear, we have to push snapcraft because it's calling itself. === ikey|zzz is now known as ikey [14:41] kalikiana: https://forum.snapcraft.io/t/dont-match-the-core-snap-to-the-host-in-cleanbuild-environments/2486 [14:42] diddledan: Thanks! [14:42] Will reply in a bit [14:42] np [14:43] kalikiana just ensure it always comes from the stable channel [14:47] sergiusens: Yeah, that we can do. To be clear, though, we still want snapcraft itself to be injected from the host or same channel, right? [14:48] Bug #1723974 opened: snap client doesn't work with Croatian language/characters [14:50] kalikiana: I must apologise if I may have seemed aggressive above, I was just frustrated at my inability to articulate why I felt cleanbuild was behaving wonky [14:50] group hug! [14:51] :-p [14:51] * kalikiana hugs diddledan [14:51] yey [14:51] for anyone who doesn't like hugs, as punishment I shall not hug you! :-D [14:52] kalikiana for persistent containers, most likely yes, for cleanbuild, only if it is in stable === JoshStrobl|zzz is now known as JoshStrobl [14:56] Good point. Will think about how we can do that [15:20] mvo, 5 runs of pi3 ubuntu-core-upgrade test passed when on ethernet [15:21] heh [15:21] yay ? [15:23] pstolowski: interessting. so lets compare notes with cachio when he is back tomorrow and see what the differences are [15:23] pstolowski: thanks a lot for your test! [15:23] this is starting to get curious ... [15:23] pstolowski: that gives me some peace of mind. this was an image produced with all bits from beta, right? [15:24] ogra_: to me it already was last week :) but yeah, it is very inconclusive [15:24] did you bump the gadget in beta ? [15:24] we should really do that [15:24] mvo, sudo ubuntu-image -c beta -O pi3-beta.img pi3.model, that's how I created it [15:25] pstolowski: great [15:27] pstolowski: lets see what sergio did - maybe it makes a difference if it was created without sudo or something else, also the revision of ubuntu-image. I'm sure we can now track it down better [15:27] ogra_, ethernet may have nothing to do here, it was just a re-test for yet another failure I pasted above where on the last run I lost access to my pi3 over wifi [15:27] ogra_: I did not touch the gadget [15:27] ogra_: I would like to wait until we solved this mystery, lets cange as little as possible [15:27] mvo, well, then it is interesting that this kernel boots at all, since the dtb will be incomplete [15:28] (i assume half the devices you have with the edge gadget wont simply be there or only half working) [15:28] ogra_: yeah, I think that is an interessting datapoint as well, the old dtb seems to be fine with most new kernels [15:28] well [15:29] the unstable wifi behavoiur might already be an indicator here [15:29] (we only added the fully fixed wifi stuff after stable got released) [15:29] it is like a lottery :) [15:29] ogra_: yeah, not saying its great, but interessting how uch apparently worked [15:30] yeah [15:30] ogra_: yeah, maybe that is the answer, sergio just draw the short straw [15:30] yep [15:32] PR snapd#4049 opened: debian,vendor: import github.com/snapcore/squashfs and use [15:34] sergiusens, kalikiana, elopio: someone _banned_ me. Who hates me today? [15:34] I can't rejoin :P [15:34] have you been rude ? [15:34] :) [15:34] kyrofa: sorry [15:34] ogra_, sometimes it slips [15:34] haha [15:35] elopio, hahaha, was it you? Fess up [15:48] nice, we have a bug in the croatian translation that causes snap to panic [15:48] that's #1723974 [15:48] Bug #1723974: snap client doesn't work with Croatian language/characters [15:55] also: “Pokušajte "snap install hello-world".” [15:56] i guess re-exec should better use C.UTF-8 ;) [15:59] re === ShalokShalom_ is now known as ShalokShalom [16:20] mvo: do you know who I need to pester to fix a buggy translation? [16:26] elopio can we make that timing thing the default on tests or is that something you did manually? [16:27] this was a looong meeting :-D I need food now to re-ful [16:31] sergiusens: I added a timer in setup and teardown. It makes things verbose, and only the outliers are useful. As a second step after fixing the outliers, I was thinking to add a max time, and fail the tests that cross that line. [16:31] but if you want, I can get it printed in travis. [16:36] kyrofa: the ros unit tests should not connect to the server, right? https://launchpadlibrarian.net/341066726/buildlog_ubuntu-artful-amd64.snapcraft_2.34-201710161334-d0be68b~ubuntu17.10.1_BUILDING.txt.gz [16:38] elopio, huh, looks like it is [16:38] * kyrofa looks [16:45] elopio, indeed, it seems that it is [16:45] But it shouldn't be... [16:46] Oh oh, wrong test [16:47] elopio, fixing now [16:47] PR snapd#4050 opened: cmd/snap: tell translators about arg names and descs req's [16:49] Hey guys, could you help me out making a snap? https://askubuntu.com/questions/965455/how-do-i-create-an-opengles2-and-glfw3-snap [16:50] PR snapcraft#1614 closed: apt repo: allow insecure repos [16:55] mark__: hello [16:55] hi [16:56] mark__: I think the most effective way forward for you will be to ask in the forum [16:56] I did [16:57] mark__: ah! ok then you're set :-) [16:57] Thanks. [17:03] elopio, fixed [17:05] thank you! [17:06] mark__: about your other issue though [17:07] mark__: what distro are you on? [17:07] lubuntu [17:07] elopio I am not so confident about killing tests that take too long. It is an extra burden and in practice doesn't work really well (considering that we already get our runners killed for long running test executions anyways) [17:08] mark__: what's the output of `which stopwatch`? [17:08] mark__: which -a stopwatch, even [17:09] of both commands [17:09] the output is "/snap/bin/stopwatch" [17:10] mark__: and if you run stopwatch, it complains about /usr/local/bin/stopwatch? [17:10] Actually it's fixed now. I don't know, maybe because after the question I make a new snap and installed/removed it multiple times. [17:11] mark__: did you used to have it in /usr/local ? [17:11] So now" snap run stopwatch" and "stopwatch" produce the same results [17:11] * Chipaca mangles the english language for fun & profit [17:11] mark__: that error looks like at some point you had a /usr/local/bin/stopwatch, and bash cached it [17:12] yeah, I had it installed manually before [17:12] using custom install.sh and uninstall.sh scripts [17:12] mark__: for what it's worth, when that happens, bash has a builtini "hash" that lets you tweak that cache [17:12] hash -r, if memory serves, updated the entries in it [17:12] or maybe it reset it [17:12] one of those :-) [17:12] oh, so that's solved then... [17:12] thanks [17:12] no problem [17:12] glad it was that :-) [17:13] I still can't load the graphics drivers though :( [17:14] mark__: yep, that's something else (and you'll need a snapcrafter for it i reckon) [17:14] kyrofa: wait, that test hasn't landed in master? [17:15] mark__: have you looked at one of the sample snaps that use opengl? [17:15] mark__: (no idea, just trying to help) [17:15] elopio, ah, yes indeed it has [17:15] elopio, I can propose a quick PR to fix it rather than putting it in the ament PR if you like [17:15] It's two lines [17:16] I suppose autopkgtest is barfing on it huh [17:17] kyrofa: is the PPA recipe. I'd appreciate if you make the PR, so I can move on with the PPA. [17:17] elopio, easy peasy. Two seconds [17:20] PR snapcraft#1619 opened: tests: don't hit internet in ros2 units [17:20] elopio, there you are ^^ [17:28] * sergiusens commutes to a location with something to eat lunch [17:28] * zyga-ubuntu rests after a long day [17:53] PR snapcraft#1620 opened: plugins: build-attributes is already in the state [18:07] popey: can you review https://github.com/snapcrafters/corebird/pull/13 for me please? :-) [18:07] PR snapcrafters/corebird#13: add wayland support for artful (17.10) [18:08] PR snapcraft#1621 opened: integration tests: skip catkin test on non-xenial [18:09] elopio, take a look at that ^^ one when you get a sec as well [18:28] Hey elopio, I have another thought for our tests. What if we had a Docker container pre-configured with all our dependencies. That would save a lot of setup time, wouldn't it? [18:28] I mean, the static tests that take a few seconds on my machine take over 5 minutes in travis [19:41] hi [19:41] I was wondering why we have to connect manually some interfaces with snap connect xxxx:home for example ? [19:42] why it is not done automatically ? [19:52] gouchi: some interfaces don't autoconnect for security reasons. Have a look at this wiki https://github.com/snapcore/snapd/wiki/Interfaces [19:53] kalikiana: thank you === Pharaoh_Atem is now known as Conan_Kudo === Conan_Kudo is now known as Pharaoh_Atem [20:27] PR snapcraft#1620 closed: plugins: build-attributes is already in the state [21:21] hi, noob question: how does Snapcraft interact with Github - if I give it access to my Github account, will it edit anything regarding the repo or Github presentation? For example create some sort of link to the Snap on the Snapcraft website [21:38] Arthur_D it shouldn't, the scope is a bit broad, but it is I think the no brainer one that allows webhooks [21:39] instructions on manually hooking it all up are being worked on to avoid this scenario, it would however require more manual interaction and setup [21:42] the reason I ask is because I contribute a bit to a project but don't want to create some sort of official or official looking snap, mainly wanted to test [21:57] PR snapcraft#1619 closed: tests: don't hit internet in ros2 units [21:57] PR snapcraft#1621 closed: integration tests: skip catkin test on non-xenial [23:12] PR snapcraft#1622 opened: schema: sync patterns with snapd