[06:53] <zyga-ubuntu> o/
[06:55] <mvo> hey zyga-ubuntu
[06:59] <zyga-ubuntu> hey, sorry for being late, just waking up
[07:00] <zyga-ubuntu> mvo: did you see my comment here? https://github.com/snapcore/snapd/pull/4034#discussion_r144391820
[07:00] <mup> PR #4034: dbus: ensure io.snapcraft.Launcher.service is created on re-exec <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4034>
[07:01] <mup> PR snapd#4034 closed: dbus: ensure io.snapcraft.Launcher.service is created on re-exec <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4034>
[07:01] <mvo> zyga-ubuntu: I did not, sorry
[07:02] <mvo> zyga-ubuntu: fixing that now in an extra branch :/
[07:02] <zyga-ubuntu> mvo: no worries, thank you
[07:06] <mup> PR snapd#4036 opened: interfaces/dbus: drop unneeded check for release.ReleaseInfo.ForceDevMode <Created by mvo5> <https://github.com/snapcore/snapd/pull/4036>
[07:06] <mvo> zyga-ubuntu: -^
[07:06] <zyga-ubuntu> ay
[07:07] <zyga-ubuntu> thank you!
[07:15] <mvo> zyga-ubuntu: 4035 needs a second review, that is also 2.28 material (mostly to make cachios life easier)
[07:16] <zyga-ubuntu> k
[07:17] <zyga-ubuntu> done
[07:17] <mup> PR snapd#4035 closed: Fix econnreset test when the download needs a retry <Created by sergiocazzolato> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4035>
[07:39]  * zyga-ubuntu needs a 2nd coffee
[07:39] <zyga-ubuntu> sorry guys I just feels so sleepy today
[07:40] <zyga-ubuntu> mvo: question about 2.28, I noticed that some PRs were going into master, are you tracking the cherry-pick list/
[07:45] <zyga-ubuntu> hey john
[07:48] <Chipaca> zyga-ubuntu: 'sup
[07:49] <Chipaca> at this end, my 'just add conditions' branch is turning into a bit of a monster
[07:49] <Chipaca> the kind of thing i then need to split into three
[07:50] <Chipaca> i think i'll split it now, even if it's a bit of an ouroboros
[07:52] <mvo> zyga-ubuntu: yes, release/2.28 should be up-to-date
[07:52] <mvo> zyga-ubuntu: i.e. all the things got cherry-picked
[07:52] <mvo> zyga-ubuntu: I'm trying to make sense of the pi3 issue that sergio reported right now but I don't have a pi3 so its a bit guesswork. could you run the specific test on a pi3?
[07:52] <mvo> (you have one, right?)
[07:53] <mvo> zyga-ubuntu: its external:ubuntu-core-16-arm-32:tests/main/ubuntu-core-upgrade - ideally with debug to get an idea what is going on inside it
[07:54] <zyga-solus> mvo: yes
[07:54] <zyga-solus> mvo: setting this up now
[07:55] <mup> PR snapd#4036 closed: interfaces/dbus: drop unneeded check for release.ReleaseInfo.ForceDevMode <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4036>
[07:55] <mvo> zyga-solus: thank you. one of the logs says "snap "core" has changes in progress
[07:56] <mvo> zyga-solus: so it might be auto-refreshing to something while the test is running
[08:11] <zyga-solus> mvo: sorry for the lag, the card didn't boot and I had to arrange keyboard/monitor; flashing edge image now
[08:12] <zyga-solus> mvo: offtopic, if you are using the wayland version of gnome on your x250 I found the UI scaling option to be invaluable and working very well (on artfuL)
[08:13] <mvo> zyga-solus: my x250 is still on zesty, I think I upgrade today
[08:13] <mvo> zyga-solus: I updated the forum topic from sergio, its a very strange behaviour
[08:14] <mvo> zyga-solus: it looks like the reboot does not go through uboot or at least not through the normal uboot snap_boot handler
[08:14] <mvo> zyga-solus: i.e. snap_mode=try should never happen after a reboot because the bootloader checks/changes that
[08:14] <mvo> zyga-solus: so I'm curious what you see if you watch the terminal
[08:14] <mup> Bug #1718942 changed: running favorited snap shows two icons in Ubuntu dock <artful> <Snappy:Fix Released> <gnome-shell-extension-ubuntu-dock (Ubuntu):Fix Released> <https://launchpad.net/bugs/1718942>
[08:15] <mvo> zyga-solus: eh, watch the attached monitor
[08:15] <zyga-solus> mvo: I'll be ready to fire this in ~10 minutes
[08:15] <zyga-solus> right
[08:15] <zyga-solus> I can also plug in serial tty if you think that is useful
[08:15] <mvo> zyga-solus: the more logging the better. its one of those "can't happen" scenarios
[08:16] <zyga-solus> ok
[08:16] <mvo> zyga-solus: I mean, the only explaination I have right now is craziness like kexec or similar
[08:16] <mvo> i.e. bypass uboot entirely
[08:21] <zyga-solus> ok, serial also attached
[08:21] <zyga-solus> just waiting for the copy to finish
[08:21]  * zyga-solus worries that the card may be busted or just super slow
[08:23] <zyga-ubuntu> writes are ~1MB/s
[08:24] <mvo> zyga-ubuntu: I can  recommend scandisk extreme, that is what I'm using and they are pretty swift. but I guess that is not helpful right now :)
[08:24]  * zyga-ubuntu tweets something
[08:26] <zyga-ubuntu> https://twitter.com/zygoon/status/918754849063358464
[08:28] <zyga-ubuntu> mvo: just as a sanity check, I'll be testing this revision 4eafe60b201706ebd317891ab226832b45d23f12
[08:28] <mvo> zyga-ubuntu: fwiw, I ordered a pi3 now too, shoudl have done that a long time agi
[08:28] <zyga-ubuntu> mvo: copy complete, let's see if it boots
[08:28] <mvo> zyga-ubuntu: autsch :( what happend?
[08:29] <mvo> zyga-ubuntu: and thanks for the extra test for 4eafe60b201706ebd317891ab226832b45d23f12
[08:29] <mvo> zyga-ubuntu: there is a spread test for it, but but but :)
[08:29] <mvo> zyga-ubuntu: better safe than sorry
[08:30] <zyga-ubuntu> mvo: nothing, they cards just stopped working one day
[08:30] <zyga-ubuntu> oddly there's nothing on serial port
[08:30] <mvo> zyga-ubuntu: on the montior?
[08:30] <zyga-ubuntu> the monitor is showing normal boot stuff
[08:30] <zyga-ubuntu> maybe I got the wires wrong?
[08:31] <mvo> zyga-ubuntu: I vaguely remember the serial port was not enabled for some reason on the pi3 but I might be wrong. ogra_ will know
[08:31] <zyga-ubuntu> I'll make sure after initial setup
[08:34] <mvo> ta
[08:35] <zyga-ubuntu> ok, board up and running, I'll check serial
[08:36] <ogra_> mvo, zyga-ubuntu serial is enabled in non-stable
[08:36] <zyga-ubuntu> ogra_: I'm on the edge image
[08:36] <ogra_> it was nitialllly not enablled on the pi
[08:36] <zyga-ubuntu> ogra_: how can I check it's enabled? I didn't see any gettys in the process list
[08:37] <zyga-ubuntu> mvo: funny, I saw "snap repair run"
[08:37] <zyga-ubuntu> root      1144  0.0  0.2   4340  2192 ttyS0    Ss+  08:37   0:00 /bin/bash /usr/share/subiquity/console-conf-wrapper --serial
[08:37] <zyga-ubuntu> I see this, could that be it?
[08:38] <ogra_> check /boot/uboot/cmdline.txt on a running system or cmdline.txt on the system-boot partition when checking the SD from a PC
[08:39] <zyga-ubuntu> dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty0 elevator=deadline rng_core.default_quality=700
[08:39] <ogra_> there you go
[08:39] <ogra_> both are enabled
[08:39]  * zyga-ubuntu swaps RX/TX
[08:40] <zyga-ubuntu> yep
[08:40] <ogra_> :)
[08:40] <zyga-ubuntu> ok, ready for testing
[08:42] <zyga-ubuntu> mvo: ok, shall I run one specific test or all of main?
[08:43] <zyga-ubuntu> external:ubuntu-core-16-arm-32:tests/main/ubuntu-core-upgrade  ?
[08:43] <zyga-ubuntu> running that
[08:46] <mvo> zyga-ubuntu: yes, that one
[08:47] <zyga-ubuntu> I'll keep you posted
[08:47] <zyga-ubuntu> I think this will take a while
[08:47] <mvo> zyga-ubuntu: yes :/ its the last piece of the puzzle
[08:47] <mvo> zyga-ubuntu: once that is under control we can do 2.28.5
[08:49] <zyga-ubuntu> sorry for not having core prepared last night
[08:53] <zyga-ubuntu> mvo: 1st restart
[08:54] <zyga-ubuntu> I really wished I ran this with --verbose-but-without-protocol-dump
[08:55] <zyga-ubuntu> mvo: 2nd reboot
[08:58] <zyga-ubuntu> mvo: 3rd reboot
[09:00] <zyga-ubuntu> mvo: 4th reboot
[09:00] <zyga-ubuntu> (man, that's a lot)
[09:00] <mvo> zyga-ubuntu: yes
[09:00]  * zyga-ubuntu wasn't aware this test did so much rebooting
[09:02] <zyga-ubuntu> restoring now
[09:02] <zyga-ubuntu> mvo: what do we do if it passes/
[09:04]  * zyga-ubuntu looks at his dead SD cards and wonders if there will be a uSD-sized HDD one day
[09:04] <mvo> zyga-ubuntu: we ask cachio to use a second sd card and retry
[09:05] <zyga-ubuntu> mvo: http://pastebin.ubuntu.com/25731170/
[09:05] <zyga-ubuntu> mvo: you can retry the test from office.zygoon.pl
[09:05] <zyga-ubuntu> mvo: I should just change the network config to static IP
[09:09] <mvo> zyga-ubuntu: thats fine
[09:10] <mvo> zyga-ubuntu: I assume the card from sergio is in a strange state. however I really want to get to the bottom of it, need to wait for him but I will prepare 2.28.5
[09:11] <zyga-ubuntu> mvo: ok
[09:13] <zyga-ubuntu> mvo: you should be able to ssh into office.zygoon.pl and ssh into pi3-1 now
[09:13] <zyga-ubuntu> mvo: and run the test any number of times
[09:14] <zyga-ubuntu> mvo: ttyUSB1 is now attached to the serial console
[09:15] <mvo> zyga-ubuntu: I will run it again a couple of times but I suspect it will just work. but who knows
[09:16] <zyga-ubuntu> mvo: I ran prepare-remote as my own user, not sure if you have to repeat
[09:16] <mvo> ta
[09:22]  * kalikiana coffee break
[09:23] <zyga-ubuntu> I'll try to organize my attic over weeekend
[09:23] <zyga-ubuntu> and move the lab devices there
[09:23] <zyga-ubuntu> this should give me a more convenient x86 box available as lab controller
[09:31] <ogra_> mvo, hmm, any idea why we did get these upload error mails for core ?
[09:31] <ogra_> (the snaps seems to be where they should be)
[09:32] <ogra_> " __all__: Can not create a new snap for core, the name is already taken and owned by someone else." is also a weird error i havent seen before
[09:34] <mvo> ogra_: should be fixed by noe
[09:34] <mvo> ogra_: by now
[09:34] <mvo> ogra_: did you get "Store authorization failed for core" mails earlierß
[09:34] <ogra_> yeah, i see they are where they should be, i was just wondering what it was
[09:34] <mvo> ?
[09:34] <ogra_> nope
[09:35] <ogra_> only one per arch for the last build
[09:36] <mvo> ogra_: it was using the wrong authorization for the automatic store uploads, it is now using the canonical store account again (as it should) so things are fine again. the token did expire (valid for 1y)
[09:36] <ogra_> ah
[09:36] <ogra_> thanks
[09:46] <popey> mvo: zyga-ubuntu does core in edge (rev 3201) have the nvidia fixes?
[09:47] <zyga-solus> popey: no idea
[09:47]  * zyga-solus looks
[09:48] <mvo> popey: yes
[09:48] <popey> oooh
[09:48]  * popey refreshes
[09:48] <mvo> popey: it should have them, including the fixes for nvidia-375+
[09:48] <mvo> popey: please let us know :)
[09:48] <popey> Try and stop me :)
[09:48] <zyga-ubuntu> mvo: question
[09:48] <zyga-ubuntu>   edge:      16-2.28.4+git434.8c9d7e9 (3201) 87MB -
[09:48] <zyga-ubuntu> what is the git revision?
[09:48] <zyga-ubuntu> and what is the dot there?
[09:49] <ogra_> the first one is just a counter
[09:49] <ogra_> iirc
[09:49] <mvo> zyga-ubuntu: the git revision is 8c9d7e9 but for lp:snapd-vendor
[09:51] <zyga-ubuntu> ahhh
[09:51] <zyga-ubuntu> thanks
[09:51] <popey> not fixed
[09:51] <zyga-ubuntu> what /o\
[09:51] <popey> http://paste.ubuntu.com/25731340/
[09:51] <zyga-ubuntu> hmmm
[09:52] <zyga-ubuntu> mvo: we don't refresh udev profiles
[09:52]  * zyga-ubuntu looks to confirm
[09:52] <zyga-ubuntu>             shouldRefresh := (backend.Name() == interfaces.SecuritySecComp || backend.Name() == interfaces.SecurityAppArmor)
[09:53] <zyga-ubuntu> yes /o\
[09:53] <zyga-ubuntu> mvo: I'll make a branch
[09:53] <zyga-ubuntu> darn
[09:53] <mvo> zyga-ubuntu: yes, darn indeed
[09:54] <zyga-ubuntu> should we just do all?
[09:54] <zyga-ubuntu> or too risky?
[09:54] <zyga-ubuntu> I wanted to do all since day one but we were conservative then
[09:54]  * ogra_ wonders if there could be any bad impact on core 
[09:55] <mvo> zyga-ubuntu: lets add udev for 2.28 and all for master ?
[09:55] <ogra_> i.e. with custom gadgets where people have custom slots
[09:55] <zyga-ubuntu> k
[09:55] <mvo> zyga-ubuntu: will we udev reload a lot?
[09:56] <mvo> i.e. will this be slow or cause storms of udev events?
[09:56] <zyga-ubuntu> mvo: done https://github.com/snapcore/snapd/pull/4037
[09:56] <mup> PR #4037: overlord/ifacestate: refresh udev backend on startup <Created by zyga> <https://github.com/snapcore/snapd/pull/4037>
[09:56] <zyga-ubuntu> mvo: it's only done if something is really different
[09:56] <zyga-ubuntu> mvo: so should be ok-ish
[09:56] <mvo> \o/
[09:56] <zyga-ubuntu> mvo: though
[09:56] <zyga-ubuntu> mvo: we re-run trigger anyway
[09:56] <ogra_> mvo, i wouldnt be that much concerned about storms of events but about device name changes or some such ...
[09:56] <zyga-ubuntu> mvo: as we don't know if the rules on disk were loaded
[09:56] <zyga-ubuntu> mvo: let's try this and see what we get
[09:56] <zyga-ubuntu> popey: thank you for catching this
[09:56] <mup> PR snapd#4037 opened: overlord/ifacestate: refresh udev backend on startup <Created by zyga> <https://github.com/snapcore/snapd/pull/4037>
[09:57] <popey> zyga-ubuntu: no problemo
[09:57] <popey> I am keen to play zzt ;)
[09:57] <popey> Uh, I mean, open my irc client
[09:57] <mvo> zyga-ubuntu: could we do something very targeted for 2.28?
[10:01] <mvo> zyga-ubuntu: I targed it to master now, this way we can land in edge, trigger build and ask popey to refresh, then we cherry pick to 2.28. sounds good?
[10:04] <zyga-ubuntu> mvo: done
[10:04] <mvo> zyga-ubuntu: ta!
[10:04] <zyga-ubuntu> mvo: I have two branches though
[10:04] <mvo> thats fine
[10:04] <zyga-ubuntu> mvo: one for 2.28 with just udev
[10:04] <zyga-ubuntu> mvo: and one for master with all
[10:05] <mup> PR snapd#4039 opened: overlord/ifacestate: refresh all security backends on startup <Created by zyga> <https://github.com/snapcore/snapd/pull/4039>
[10:05] <mvo> ok
[10:06] <mvo> I'm setting up a 16.04/nvidia machine in the meantime
[10:06]  * mvo feels uneasy not being able to reproduce
[10:07] <mup> PR snapd#4040 opened: interfaces: add USB interface number attribute in udev rule for serial-port and hidraw interfaces <Created by musicguitar> <https://github.com/snapcore/snapd/pull/4040>
[10:07] <zyga-ubuntu> mvo: install 16.04 and stable
[10:07] <zyga-ubuntu> mvo: then refresh to beta
[10:07] <zyga-ubuntu> mvo: no wait, install a snap before you do the beta refresh
[10:07] <zyga-ubuntu> mvo: that should trigger this
[10:07] <popey> my zyga desktop is still powered on and available if you need it
[10:08] <zyga-ubuntu> popey: no, it's fine, I think this patch will change that correctly
[10:08] <popey> kk
[10:08] <zyga-ubuntu> mvo: meanwhile 3 runs on pi3-1 are green, still going
[10:09] <mvo> great
[10:14] <ackk> why can't I see prints() in snapcraft unittests?
[10:14] <ackk> *print()s
[10:16] <zyga-ubuntu> ackk: probably intercepted by the testing framework
[10:17] <ackk> might be something in testscenarios I guess
[10:18] <ackk> oh wait, FakeTerminal
[10:34]  * zyga-solus -> lunch
[10:40] <mup> PR snapcraft#1617 opened: Add options to configure applications socket activation <Created by albertodonato> <https://github.com/snapcore/snapcraft/pull/1617>
[10:42] <zyga-ubuntu> mvo: welcome back
[10:42] <zyga-ubuntu> what did you change?
[10:43] <mvo> zyga-ubuntu: switch to xorg from wayland
[10:44] <kalikiana> ppisati: Hey! I just saw your snapcraft PR
[10:48] <kalikiana> ppisati: Thanks for looking into it! Do you need any help with it? I'm thinking we could have a very simple unit test for it
[10:48] <zyga-ubuntu> re
[10:48] <zyga-ubuntu> ok, let's look at the tests
[10:49] <ppisati> kalikiana: if you feel like writing a unit test for it, go ahead - i'm really bad with that stuff
[10:50] <kalikiana> Yeah, I can do that.
[10:51] <kalikiana> ppisati: Just to check first, should the return be ''? In the new PR there's no value
[10:53] <mup> PR snapd#4039 closed: overlord/ifacestate: refresh all security backends on startup <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4039>
[10:53] <mvo> zyga-ubuntu: I baby-sit a core build now, then we can test the fix
[10:55] <ppisati> kalikiana: empty string or nothing have the same effect
[10:55] <zyga-solus> mvo: ta
[10:57] <zyga-ubuntu> mvo: https://github.com/snapcore/snapd/pull/4037 is green now
[10:57] <mup> PR #4037: overlord/ifacestate: refresh udev backend on startup <Created by zyga> <https://github.com/snapcore/snapd/pull/4037>
[10:57] <kalikiana> ppisati: Hmmm this `'CROSS_COMPILE={}'.format(None)` gives me `'CROSS_COMPILE=None'`, though, which seems wrong?
[10:58] <kalikiana> I guess it might be ignored by kbuild?
[10:58] <zyga-solus> kalikiana: why wrong?
[10:58] <zyga-solus> I mean, it's valid python :)
[10:58] <zyga-solus> just looks wrong as a string
[10:58] <kalikiana> zyga-solus: Well, by wrong I mean the result looks wrong
[10:59] <kalikiana> Literally in terms of human eyes
[10:59] <mvo> zyga-ubuntu: yeah, waiting for confirmation before the merge mainly
[10:59] <kalikiana> zyga-solus: This gets passed to kbuild
[11:00]  * kalikiana knows very little about what's normal in kbuild terms
[11:03] <zyga-ubuntu> mvo: tests passed on pi3
[11:04] <zyga-ubuntu> mvo: though the counter is off :)
[11:04] <zyga-ubuntu> zyga@fyke:~/snapd$ SPREAD_EXTERNAL_ADDRESS=192.168.3.3:22 spread -v -debug -reuse -repeat 5  external:ubuntu-core-16-arm-32:tests/main/ubuntu-core-upgrade
[11:04] <zyga-ubuntu> 2017-10-13 12:47:47 Successful tasks: 6
[11:04] <zyga-ubuntu> wat?
[11:04] <zyga-ubuntu> -repeat is -repeat-in-addition-to-the-one-you-selected
[11:06] <mvo> zyga-ubuntu: heh, ok
[11:07] <mvo> zyga-ubuntu: I get lunch while waiting for the builds I think
[11:07] <kalikiana> ppisati: aside from that detail, are you going to add an error message for the non-x86 case? or do you want me to do that also?
[11:08] <kalikiana> I think what you suggested in the bug sounds pretty good
[11:08] <kalikiana> the use case of the op is exactly that
[11:16] <ppisati> kalikiana: if arch != x86_64, it raises the exception there so you get a 'Cross compiling XXX on YYY is not supported' error
[11:29] <kalikiana> ppisati: Yeah, was just wondering if it should be more specific in this case, like "Cross compiling for i686 is only supported from x86-64"
[11:30] <kalikiana> Although maybe the existing one is good enough
[11:32] <kalikiana> ppisati: Can you add me to your branch? Then I can push the tests
[11:50] <mup> PR snapd#4041 opened: cmd,packaging: enable apparmor on openSUSE <Created by zyga> <https://github.com/snapcore/snapd/pull/4041>
[11:56] <mup> PR snapd#4037 closed: overlord/ifacestate: refresh udev backend on startup <Created by zyga> <Merged by stolowski> <https://github.com/snapcore/snapd/pull/4037>
[11:57] <kalikiana> ppisati: Nevermind, seems I was able to push already. Again, thanks for working on this
[12:03] <jdstrand> mvo: is a 2.28.5 expected in beta/candidate today?
[12:03] <jdstrand> mvo: hi btw :)
[12:04] <mvo> jdstrand: yes
[12:04] <mvo> popey: could you try edge please with your example snap?
[12:05] <mvo> jdstrand: we are just doing on confirmation about https://github.com/snapcore/snapd/pull/4039
[12:05] <mup> PR #4039: overlord/ifacestate: refresh all security backends on startup <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4039>
[12:05] <mvo> jdstrand: well, a variant for 2.28 that only refreshes udev in addition to seccomp and apparmor
[12:05] <popey> mvo: will do
[12:05] <mvo> jdstrand: this is needed to ensure the broken profiles get flushed
[12:05] <mvo> popey: thank you!
[12:05]  * popey notes a new install bar thing
[12:06] <mvo> popey: yes, new feature, if you like it send \o/ to Chipaca
[12:06] <kalikiana> sergiusens: not sure if you're on yet, but FYI I'll be on a train in ~30 and have a longer lunch, killing some overtime from the late nights this week, but will check in later/ be available for q's
[12:06] <popey> I thought my terminal was broken initially
[12:06] <popey> :)
[12:06] <popey> still broken I'm afraid
[12:07] <mvo> zyga-ubuntu: -^
[12:07] <zyga-ubuntu> ack
[12:07] <popey> http://paste.ubuntu.com/25731820/
[12:07] <zyga-ubuntu> popey: can you look at /etc/rules.d/
[12:08] <zyga-ubuntu> popey: look at the file for the snap
[12:08] <zyga-ubuntu> popey: can you paste it?
[12:08] <popey> i assume you mean /etc/udev/rules.d
[12:08] <mvo> it is the right version, that git revision contains the fix
[12:08] <popey> alan@hal:/etc/udev/rules.d$ pastebinit 70-snap.zzt.rules
[12:08] <popey> http://paste.ubuntu.com/25731823/
[12:08] <zyga-ubuntu> popey: yes, sorry
[12:08] <zyga-ubuntu> popey: okay
[12:08] <zyga-ubuntu> popey: please go to /run/udev/tags
[12:09] <zyga-ubuntu> popey: then to app in question
[12:09] <zyga-ubuntu> and list files there and paste here
[12:09] <popey> http://paste.ubuntu.com/25731827/
[12:09] <zyga-ubuntu> this looks like a bug in udev
[12:09] <zyga-ubuntu> ok
[12:09] <zyga-ubuntu> can you please
[12:09] <mvo> jdstrand: are people asking on the sprint already?
[12:10] <zyga-ubuntu> sudo udevadm control --reload
[12:10] <zyga-ubuntu> sudo udevadm trigger
[12:10] <zyga-ubuntu> oh man
[12:10] <popey> done
[12:11] <popey> (still busted)
[12:11] <zyga-ubuntu> mvo: we do "sudo udevadm control --reload-rules"
[12:11] <zyga-ubuntu> mvo: but help has only --reload
[12:11]  * zyga-ubuntu checks
[12:11] <zyga-ubuntu> popey: ok, thank you, no new theories yet
[12:11] <popey> ok
[12:11] <popey> lunch
[12:11] <popey> biab
[12:11] <zyga-ubuntu> popey: the files in /run... didn't change?
[12:11]  * popey looks
[12:11] <popey> same time/date on them
[12:12] <zyga-ubuntu> k
[12:12] <zyga-ubuntu> popey: if you remove the nvidia ones
[12:12] <zyga-ubuntu> and then run the app?
[12:12] <jdstrand> mvo: no, but last night I noticed in the forum someone with multiple network devices having trouble with network-control
[12:12] <popey> so rm /run/udev/tags/snap_zzt_zzt/*nvidia*  ?
[12:12] <zyga-ubuntu> yp
[12:12] <zyga-ubuntu> yep
[12:12] <mvo> jdstrand: ta
[12:12] <jdstrand> mvo: mwinter and the frr snap
[12:12] <popey> success
[12:13] <zyga-ubuntu> mvo: ok, I just checked that "--reload" and "--reload-rules" are the same (it's an alias)
[12:13] <zyga-ubuntu> popey: ok
[12:13] <zyga-ubuntu> can you reboot
[12:13] <zyga-ubuntu> I think it's a bug in udev
[12:13] <popey> ugh
[12:13] <zyga-ubuntu> mvo: we can do a cleanup thing that would go around all the udev snap tags and rm those files
[12:13] <zyga-ubuntu> mvo: (via the udev backend)_
[12:13] <popey> ok, will reboot
[12:13] <zyga-ubuntu> mvo: +1/-1?
[12:14] <kalikiana> elopio: We really need to split up the tests, two of my PRs now "failed" because of timeouts, and they were fine before... maybe we can discuss it a bit later if you have some time
[12:14] <zyga-ubuntu> mvo: I think this could go to udev's Initialize (in master) and to udev init() in 2.28
[12:14] <jdstrand> mvo: thanks for working on this
[12:15] <Son_Goku> zyga-ubuntu: what happens if the snap-confine is linked to libapparmor and apparmor isn't available?
[12:15] <zyga-ubuntu> Son_Goku: at runtime as in compiled but disabled on boot? nothing bad, it's been supported for months this way
[12:16] <Son_Goku> well, that is, userspace support is there, but not kernel space
[12:16] <zyga-ubuntu> jdstrand: not sure if you have the time for that but I did some updates on https://github.com/snapcore/snapd/pull/4008 -- I don't need a security review but another "still on the right/wrong" track review
[12:16] <mup> PR #4008: cmd/snap-update-ns: create missing mount points automatically <Created by zyga> <https://github.com/snapcore/snapd/pull/4008>
[12:16] <mvo> zyga-ubuntu: hm, but people will have to reboot anyway?
[12:16] <zyga-ubuntu> Son_Goku: same
[12:16] <zyga-ubuntu> mvo: no
[12:16] <zyga-ubuntu> mvo: if we remove those files then it's ok
[12:16] <Son_Goku> so then why didn't you do this with openSUSE before?
[12:16] <zyga-ubuntu> mvo: (popey just tested that)
[12:16] <zyga-ubuntu> Son_Goku: because before the backend was not enabled at all and snap-confine didn't run without any app profiles
[12:16] <mvo> zyga-ubuntu: oh, he did? I thought he said it did not work, if it does, excellent, lets do it
[12:17] <zyga-solus> mvo: please check backlog to ensure I'm not confusing things
[12:17] <mvo> zyga-ubuntu: didn't he say "<zyga-ubuntu> I think it's a bug in udev"
[12:17] <mvo> zyga-ubuntu: eh, I mean, didn't he say he will reboot?
[12:17] <mvo> popey: hey, did removing the files help?
[12:17] <zyga-solus> mvo: he didn't reboot yet (I assumed, as that was his IRC system)
[12:17] <zyga-solus> mvo: better to ask though, sure :)
[12:18] <mvo> zyga-solus: aha, I see "success"
[12:18] <mvo> zyga-solus: I'm still slightly confused, lets see what he says :)
[12:18]  * zyga-solus moved from kitchen back to office desk
[12:18] <zyga-solus> yes
[12:18] <mvo> zyga-solus: but yeah, lets add workaround/cleanup
[12:18] <zyga-solus> I worry we don't have a full picture of udev behavior
[12:18] <zyga-solus> mvo: ok, I'm on it
[12:18] <mvo> ta
[12:20] <zyga-ubuntu> mvo: question, shoud snap-confine do this (so that we know it runs before 1st snap runs)
[12:20] <popey> zyga-ubuntu: after rebooting, apps work (however, that's what ogra did without any updates)
[12:20] <zyga-ubuntu> mvo: or should snapd do this (but it may start after the snap)_
[12:20] <zyga-ubuntu> popey: if you go to stable
[12:20] <zyga-ubuntu> popey: they should stop to work again
[12:20] <zyga-ubuntu> popey: then you can go to edge
[12:20] <popey> ok, lets test
[12:20] <zyga-ubuntu> popey: and they will stay broken
[12:21] <zyga-ubuntu> popey: and removing those files from /run will fix it again (assuming you are on edge)
[12:21] <zyga-ubuntu> popey: that's my theory
[12:21] <popey> ok
[12:21] <zyga-ubuntu> popey: udev doesn't remove tags for nvidia because of $unknown reason (and probably sysfs)
[12:22] <popey> reverted to stable, and things are not as clear cut.
[12:24] <zyga-ubuntu> meaning?
[12:24] <popey> zzt and ohmygiraffe fail with the x issue
[12:24] <popey> can't create a gl context
[12:24] <popey> irccloud loads fine
[12:24] <zyga-ubuntu> can you look at udev rules
[12:24] <zyga-ubuntu> ah
[12:24] <zyga-ubuntu> wait
[12:24] <zyga-ubuntu> right
[12:24] <zyga-ubuntu> popey: because stable doesn't reload udev rules :)
[12:24] <zyga-ubuntu> popey: so you have to disable/enable
[12:25] <popey> remove/install the snap good enough?
[12:25] <zyga-ubuntu> yes thouh disable/enable is faster
[12:25] <popey> nope
[12:25] <popey> removed / installed zzt, still broken
[12:25] <popey> (on stable core)
[12:25] <zyga-ubuntu> good
[12:25] <popey> ok
[12:25] <zyga-ubuntu> now go to edge
[12:26] <zyga-ubuntu> and it should stay broken
[12:26] <popey> ok
[12:26] <zyga-ubuntu> then remove the files and it should work
[12:26] <zyga-ubuntu> and then reboot after saying on edge and it should work out of the box
[12:26] <zyga-ubuntu> (that's my theory)
[12:26] <zyga-ubuntu> sorry for tha hands-on exercise :/
[12:26] <mvo> zyga-ubuntu: hm, hm, ideally snap-confine, I'm slightly worried that the code will be more complex there
[12:26] <zyga-ubuntu> mvo: right
[12:26] <zyga-ubuntu> mvo: maybe snapd is "good enough"
[12:27] <zyga-ubuntu> Chipaca: ^ what do you think?
[12:27] <popey> went to edge and zzt works now
[12:27] <zyga-ubuntu> popey: !!
[12:27] <zyga-ubuntu> popey: without removing?
[12:27] <Chipaca> zyga-ubuntu: sorry, what's the context?
[12:27] <mvo> popey: please try beta that should fix the gl context bug
[12:27] <popey> ohmygiraffe works too
[12:27] <zyga-ubuntu> Chipaca: we're trying to figure out where to put some clenaup code: in snap-confine/C or in snapd/go
[12:27] <popey> yes, without removing
[12:27] <Chipaca> zyga-ubuntu: what does the cleanup code do?
[12:28] <Chipaca> zyga-ubuntu: and is it distro-specific, or will everybody need it?
[12:28] <zyga-ubuntu> popey: then something doesn't hold, maybe that's just my confusion about stable not having refresh for udev
[12:28] <zyga-ubuntu> Chipaca: rm /run/udev/tags/snap_*/*nvidia
[12:28] <zyga-ubuntu> -f
[12:28] <zyga-ubuntu> (literally that)
[12:29] <zyga-ubuntu> I think it's a generic thing
[12:29] <mvo> zyga-ubuntu: we could add a systemd unit?
[12:29] <zyga-ubuntu> mvo: I'm puzzled,
[12:29] <zyga-ubuntu> mvo: not in reexec easily
[12:29] <Chipaca> zyga-ubuntu: who wrote those files?
[12:29] <zyga-ubuntu> Chipaca: udev
[12:29] <popey> mvo: on beta ohmygiraffe fails, segfault
[12:29] <mvo> zyga-ubuntu: hm, good point
[12:29] <zyga-ubuntu> Chipaca: it seems (though I may be wrong) that udev doens't remove them
[12:29] <Chipaca> zyga-ubuntu: how much do we love udev? (no ignore this question)
[12:29] <popey> same for zzt
[12:29] <zyga-ubuntu> Chipaca: with a retractable baton
[12:29] <zyga-ubuntu> no, I'm kidding
[12:29] <zyga-ubuntu> udev is fine, it's just an edge case
[12:29] <mvo> popey: what version of the nvidia driver do you have installed? apt list --installed nvidia-*
[12:30] <zyga-ubuntu> popey: note that unless you use edge you will not see reliable behavior
[12:30] <Chipaca> zyga-ubuntu: how do these things persist over reboot?
[12:30] <zyga-ubuntu> popey: as non-edge doesn't reload udev
[12:30] <Chipaca> zyga-ubuntu: or is it a "upgrade to new snapd, now wrong udev rules" thing?
[12:30] <zyga-ubuntu> popey: and your piror revision of snapd is relevant to "works/doesnt"
[12:30] <zyga-ubuntu> Chipaca: those don't persist, it's just a classic reexec fix
[12:30] <zyga-ubuntu> Chipaca: it seems, though we need to test this, that udev doesn't remove those tags
[12:31] <zyga-ubuntu> mvo: can you try testing this
[12:31] <zyga-ubuntu> mvo: on your machine
[12:31] <mvo> zyga-ubuntu: I need to do a bit of fiddling, but probably
[12:31] <zyga-ubuntu> mvo: just look if /run/udev/tags/snap_*/*nvidia go away
[12:31] <zyga-ubuntu> mvo: after going from stable to edge, with reexec without a reboot
[12:32] <zyga-ubuntu> mvo: note that after going to stable from edge you will have fake state with udev from edge and rest from stable so it's not a reliable measurement
[12:33] <zyga-ubuntu> s/fake/deceiving/
[12:33] <zyga-ubuntu> mvo: and download caching branch, perfect for that :) thank you for making it
[12:35] <zyga-ubuntu> mvo: I'll hold and do the fixup in go if we need to
[12:35] <zyga-ubuntu> I need to break, my back is killing me :/
[12:39] <mvo> zyga-ubuntu: ok, I can reproduce it with "hiri"
[12:39] <mvo> zyga-ubuntu: I get "udev_enumerate_scan failed"
[12:40] <mvo> zyga-ubuntu: I installed it before, then I refresh core to edge and the nvidia-files are still in run/udev/tags and hiri still fails to start
[12:41] <mvo> zyga-ubuntu: if I move those files away hiri starts
[12:41] <Chipaca> hmmm
[12:41] <ogra_> did you add code to e-generate all rulles ?
[12:41] <ogra_> *re-generate
[12:41] <ogra_> if so, just rm them ?
[12:41] <ogra_> (before re-generation)
[12:41] <Chipaca> if the problem goes away with a reboot, and only happens on nvidia, would it be fair to publish an errata and fix it for 2.29?
[12:42] <mvo> zyga-ubuntu: if I move the files back, things fail again
[12:42] <Chipaca> or are there headless, unattended devices out there that use nvidia?
[12:44] <popey> zyga-ubuntu: mvo you don't need me for a bit do you?
[12:45] <mvo> popey: I can reproduce it now too, thank you. we may ask you in ~1-2h to double check our fix
[12:46] <popey> kk
[12:46] <mvo> Chipaca, zyga-ubuntu just double checked (it was kind of expected) - a reboot makes the problem go away. but I'm not happy telling our users that bth
[12:51] <Chipaca> mvo: is this breakage only in .4?
[12:51] <Chipaca> or is it in .1 also?
[12:53] <Son_Goku> mvo: that snapfuse thing looks gross
[12:54] <Son_Goku> is it really true that `apt upgrade` won't upgrade components that pull in new dependencies?
[12:54] <Son_Goku> that seems like broken behavior
[12:54] <popey> you want apt dist-upgrade
[12:54] <pstolowski> zyga-solus, Chipaca, mvo i'm not coming to the standup today, i've a guy coming to mount blinders etc in my apartment
[12:54] <popey> its not broken, it's documented, desired
[12:55] <Chipaca> popey: not sure how desired, given 'apt update' does the dist upgrade :-)
[12:55] <Chipaca> upgrade*
[12:55] <mvo> Son_Goku: firefighting right now, but happy to tell the full (sad) backstory, in a nutshell popey is right, its the behaviour of apt-get upgrade nice 1997 (roughtly) and some people use it, probably more out of ignorance than anything else
[12:55] <Chipaca> pstolowski: you're so lazy man
[12:55] <popey> eh?
[12:56] <Chipaca> popey: apt upgrade == apt-get dist-upgrade
[12:56] <popey> update doesn't dist-upgrade
[12:56] <mvo> Chipaca: almost
[12:56] <Chipaca> mvo: la la la la
[12:56] <Chipaca> :-)
[12:56] <mvo> Chipaca, popey apt upgrade will add new package but never remove packages
[12:56] <popey> Chipaca: Ah okay, you said "apt update" does dist-upgrade
[12:56]  * mvo fixed that back in the day
[12:57] <Chipaca> popey: yes, because my brain is a not particularly firm blancmange
[12:57] <popey> :D
[12:57] <zyga-ubuntu> mvo: re,
[12:57] <Son_Goku> now I'm glad that apt never won out in the rpm distros
[12:57] <Son_Goku> that's really stupid behavior
[12:58] <zyga-ubuntu> mvo: ok
[12:58] <popey> It was often desired on systems where you were happy to upgrade existing software you've audited, but didn't want new stuff sneaking in
[12:58] <zyga-ubuntu> mvo: so now that you can reproduce
[12:58] <popey> *shrug*
[12:58] <zyga-ubuntu> mvo: can you please confirm that after going to a build with udev reload patch we still keep the nvidia tag files in /run/udev/tags?
[12:58] <zyga-ubuntu> mvo: (i.e. it's a bug in udev)
[12:59] <Chipaca> popey: AIUI it's merely flipping a default config, where apt-get uses one default and apt uses another, but i might've misunderstood
[12:59] <Son_Goku> that's an incredibly stupid default
[12:59] <mvo> zyga-ubuntu: I did that, I refreshed to edge and the files are still there
[12:59] <popey> It's not a default
[12:59] <popey> you typed the wrong command :)
[12:59] <mvo> zyga-ubuntu: fwiw, I am testing a small cleanup
[12:59] <popey> holding it wrong etc
[12:59] <Son_Goku> :/
[12:59]  * Chipaca wonders if Son_Goku accidentally ate trollflakes instead of cornflaes for breakfast
[13:00] <zyga-ubuntu> mvo: thank you
[13:00] <mvo> Son_Goku: its a feature with its time and its place, however the sad thing is that it got cargo culted mostly because "apt-get dist-upgrade" sounds scary (and would remove stuff) and people use apt-get upgrade for the wrong reasons. fortunately apt upgrade fixes it
[13:01] <Son_Goku> yeah, I've seen people say "never use dist-upgrade" at work
[13:01] <Son_Goku> afaik, all dist-upgrade does is process Breaks/Replaces (aka, do proper upgrade)
[13:02] <Son_Goku> mvo: I just used smart most of the time before the new apt command became available ;)
[13:02] <popey> no
[13:02] <popey> People are silly :)
[13:02] <Chipaca> *shocked*
[13:02]  * Son_Goku gasps
[13:02] <Chipaca> mvo: i'll be in the standup in a minute (getting tea)
[13:02] <zyga-ubuntu> Chipaca: https://www.youtube.com/watch?v=0du72G8CeOY
[13:04] <ogra_> popey, but *you* know the weirdest people
[13:04] <popey> hello ogra_
[13:04]  * ogra_ grins
[13:06] <Chipaca> popey: just to double-check, your nick rhymes with "ropey", not with "popeye", right?
[13:06] <popey> yes
[13:06] <Chipaca> phew
[13:14] <diddledan> popey, the dopey
[13:15] <diddledan> popey: you're dopey, right? :-p
[13:15] <popey> people call me poppy or popeye too
[13:15] <popey> i used to get a surprising number of hits to my blog for people searching for "popey and olive oil"
[13:15] <diddledan> haha
[13:15] <diddledan> you ought to make a page for that
[13:16] <diddledan> give them something to actually read
[13:16]  * ikey wonders how many hits come from "in London a knight a popey interred"
[13:17] <ikey> *london lies
[13:17]  * ikey wakes up
[13:17] <diddledan> wakey wakey ikey
[13:17] <ikey> ty lol
[13:18] <diddledan> popey needs to invent a boxing match special move, similar to the rope-a-dope
[13:19] <ogra_> after alll we're the only disto who has A Pope ;)
[13:19]  * diddledan pontificates
[13:19] <ikey> lol
[13:19] <diddledan> I still say we should get popey canonised
[13:20] <ogra_> to shoot him at what exactly though ?
[13:20] <ikey> ogra_, the hopes and dreams of the innocent
[13:20] <ogra_> heh
[13:20] <diddledan> nono, turn him into a saint
[13:21] <ikey> now when you say saint
[13:21] <ogra_> before or after we shot him ?
[13:21] <diddledan> lol
[13:21] <ikey> you dont mean like howard saint right?
[13:21] <diddledan> well usually they need to be dead
[13:21] <diddledan> popey: we're killing you off
[13:21] <ikey> thats quite the job requirements
[13:21] <ogra_> https://www.mysteryscenemag.com/images/film_tv/the_saint_w_halo.jpg
[13:22] <diddledan> haha, great series
[13:22] <ikey> "Required: 10 years experience with being generally nice, 8 years experience with being generally not alive"
[13:22] <diddledan> they did a straight-to-netflix movie with some random fella and eliza dushku (of Buffy the Vampire and Dollhouse)
[13:23] <ikey> ooo its friday the 13th
[13:23] <diddledan> :-o
[13:23]  * ikey avoids hockey all day
[13:23] <ogra_> it so is
[13:23]  * diddledan goes back to bed
[13:23] <ikey> if i meet anyone called jason im noping myself back all the way home
[13:24] <ikey> or even more terrifying than that
[13:24] <ikey> if i bump into lindsay lohan
[13:24]  * ikey shudders
[13:26] <ogra_> http://dlisted.com/wp-content/uploads/2014/05/lilomolly2014.jpg
[13:26] <Son_Goku> the work buildsystem is down today :(
[13:26] <Son_Goku> this... is... not good
[13:26] <ikey> the picture or the build system?
[13:27] <ikey> i mean both are pretty out there
[13:27] <ogra_> so it is a vacation buildsystem then
[13:27] <ikey> lol
[13:27] <Son_Goku> ikey: both
[13:27] <Son_Goku> :D
[13:37] <mup> PR snapd#4042 opened: snap-confine: cleanup incorrectly created nvidia udev tags <Created by mvo5> <https://github.com/snapcore/snapd/pull/4042>
[13:55] <mup> PR snapd#4041 closed: cmd,packaging: enable apparmor on openSUSE <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4041>
[13:57] <zyga-ubuntu> jdstrand: hey, can you please review https://github.com/snapcore/snapd/pull/4042
[13:57] <mup> PR #4042: snap-confine: cleanup incorrectly created nvidia udev tags <Created by mvo5> <https://github.com/snapcore/snapd/pull/4042>
[14:00] <zyga-ubuntu> jdstrand: it's urgent for us to get this before EOD
[14:24] <cachio> mvo, zyga-ubuntu running the test with logging enabled
[14:24] <cachio> I had to restart the screen to enable it
[14:28] <Son_Goku> zyga-ubuntu: can you give positive karma to the snapd updates in Fedora, since popey can't?
[14:32] <mvo> cachio: thank you
[14:32] <cachio> mvo, https://paste.ubuntu.com/25732470/
[14:32] <mvo> cachio: oh my
[14:32] <cachio> mvo, does it help?
[14:32] <mvo> cachio: that explains why we don't have updated defaults
[14:33] <mvo> cachio: can you mail me your uboot.env file from the card? the one that it cannot read?
[14:33] <mvo> cachio: oh, nevermind - uEnv.txt? we don't use this since ages. sorry. ogra_ is the error/warning from https://paste.ubuntu.com/25732470/ expected on the pi3?
[14:33] <ogra_> yeah, thats normal
[14:33] <mvo> cachio: -^
[14:34] <ogra_> it normally just skips
[14:34] <ogra_> (we should remove the old code that looks for it and prints the message but it isnt any blocker in any way)
[14:34] <cachio> mvo, I sent you the screen log
[14:36] <mvo> cachio: hmmmm, "^MFAT: Misaligned buffer address (3ab0b570)
[14:36] <mvo> "
[14:36] <mvo> ogra_: -^ have you seen this before?
[14:37] <carroarmato0> Hello, I'm trying to build a snap package containing a Go application. I managed to snap it correctly, however I encountered the following issue security tag snap.12to8.12to8 not allowed   due to the fact that the app's name starts with a number.... any way to fix this without having to rename the upstream project ?
[14:38] <mvo> cachio, ogra_ the relevant part is http://paste.ubuntu.com/25732499/ I think - we try to store snap_mode=trying but that fails with the given error and so the system boots in try mode and it goes into the installed core. so mystery solved, now we need to understand why the buffer is misaligned and what that means. the error comes from uboot
[14:38] <ogra_> mvo, no, thats also normal
[14:38] <mvo> ogra_: it is?
[14:39] <ogra_> yes
[14:39] <zyga-solus> Son_Goku: no, because we're going to do .5
[14:39] <zyga-solus> :/
[14:39] <zyga-solus> Son_Goku: we found a number of issues
[14:39] <Son_Goku> god damn it
[14:39] <zyga-solus> Son_Goku: I was meaning to tell you but I forgot, sorry
[14:39] <Chipaca> carroarmato0: that looks like a bug to me
[14:39] <zyga-solus> Son_Goku: that's all the week really :/
[14:39] <ogra_> mvo, and you will find that nothing gets corrupted, thats about the ram buffer only
[14:39] <ogra_> cachio, mvo, rather check the content by stopping the boot and lloking at the printenv output
[14:40] <ogra_> *looking
[14:40] <mvo> ogra_: well, what cachio is seeing is a device that boots in snap_mode=try and never gets to "snap_mode=trying" (or snap_mode="")
[14:40] <zyga-solus> mvo: hmm
[14:40] <mvo> ogra_: and *if* the error is not a red-herring that would fit the picture
[14:40] <ogra_> right, but none of the messages are anything out of the ordinary
[14:40] <zyga-solus> mvo: I saw some misaligned fat errors in dos tools
[14:40] <zyga-solus> mvo: it was fixed >> xenial AFAIK
[14:40]  * zyga-solus looks
[14:41] <Chipaca> carroarmato0: is your snap public?
[14:41] <ogra_> cachio, mvo, there are corruption issues where uboot doesnt init the SD coreectly and prints timeout messages *before* even loading uboot.env
[14:41] <ogra_> that happens with some new sandisk cards
[14:41] <carroarmato0> Chipaca:  no, but there is an old existing issue for it on launchpad
[14:41] <ogra_> (which are also not working i.e. in my desktop SD reader)
[14:42] <cachio> ogra_, it was working until the last build
[14:42] <ogra_> (i.e. a general issue with these cards and linux)
[14:42] <carroarmato0> Chipaca:  kinda figured maybe someone knows a workaround for it or has been fixed in the meantime :)
[14:42] <carroarmato0> Chipaca:  hang on, I'll look it up
[14:42] <zyga-solus> mvo: this error comes from the following patch https://lists.denx.de/pipermail/u-boot/2015-September/228813.html
[14:42] <cachio> I tried with 2 cards and I see the same error
[14:42] <carroarmato0> Chipaca:  https://bugs.launchpad.net/ubuntu/+source/snapcraft/+bug/1636016
[14:42] <mup> Bug #1636016: snapcraft should warn of invalid name <snapcraft (Ubuntu):Confirmed> <https://launchpad.net/bugs/1636016>
[14:42] <zyga-solus> ogra_: ah, you're talking about that bug where some optimization would clobber the cards and there's a growing blacklist
[14:43] <ogra_> cachio, well, do you see any timeout messages something like "sdhc ... something, something ... increating timeout by ... millisecs"
[14:43] <zyga-solus> ogra_: right/
[14:43] <ogra_> zyga-solus, mainly the sandisk extreme cards
[14:43] <ogra_> the very latest batch seems to have some issues
[14:43] <Chipaca> carroarmato0: I can confirm it's a bug, yes
[14:43] <Chipaca> carroarmato0: and i've got a minimal reproducer
[14:44] <ogra_> i can get them to mount on my laptop but my PC doesnt see them at all ... both on xenial both using the same USB reader
[14:44] <zyga-solus> is that the card that cachio is using?
[14:44] <ogra_> i think he used an ultra crad
[14:44] <ogra_> *card
[14:44] <ogra_> (if thats the same he showed me in YNC)
[14:44] <zyga-solus> ogra_: very appropriate tweet from this morning https://twitter.com/zygoon/status/918754849063358464
[14:44] <ogra_> *NYC
[14:44] <Chipaca> carroarmato0: it's a bug in snapd, not snapcraft, though
[14:44] <ogra_> heh
[14:45] <Chipaca> carroarmato0: or … hm
[14:45] <zyga-solus> Chipaca: let's roll the fix into 2.28
[14:45] <Chipaca> carroarmato0: reading #1589613 it might not be
[14:45] <mup> Bug #1589613: Snap app names are too permissive <verification-done> <Canonical Click Reviewers tools:Invalid> <Snapcraft:Fix Released by kyrofa> <snapcraft (Ubuntu):Fix Released> <snapd (Ubuntu):Fix Released by kyrofa> <snapcraft (Ubuntu Xenial):Fix Released> <snapd (Ubuntu Xenial):Fix Released>
[14:45] <mup> <snapcraft (Ubuntu Yakkety):Fix Released> <snapd (Ubuntu Yakkety):Fix Released by kyrofa> <https://launchpad.net/bugs/1589613>
[14:45] <mvo> ogra_: any ideas what might cause a pi3 from staying at "snap_mode=try" would be appreciated. if you could guide him to any further debug ddata, that would be great. I don't really see (except for a write failure) how the uboot does not go from "snap_mode=try" to "snap_mode=trying"
[14:45] <Chipaca> so
[14:45] <Chipaca> snap and app names
[14:45]  * mvo takes a short break
[14:45] <carroarmato0> Chipaca:   I search for other snap packages which might start with a number in their name. Stumbled upon the 0ad game. https://uappexplorer.com/snap/ubuntu/play0ad   I'm not sure if they worked around the problem by naming it  play0ad rather than 0ad... or they just chose to name it like that right off the bat
[14:45] <Chipaca> we explicitly allow them to start with a number
[14:45] <ogra_> mvo, as i said ... the printout from "printenv" when intercepting the boot
[14:46] <ogra_> cachio, ^^^
[14:46] <carroarmato0> Chipaca:   snapcraft has no issue at all generating snaps beginning with a number
[14:46] <Chipaca> but then snap-confine aborts
[14:46] <mvo> ogra_: will that happen before or after "snappy_boot" ran? because snappy_boot is when uboot toggles from snap_mode=try to trying and saves the env
[14:46] <Chipaca> zyga-solus: so, why does snap-confine reject valid app names?
[14:47] <carroarmato0> Chipaca:  but the issue occurs after you installed the snap and try executing the command
[14:47] <carroarmato0> Chipaca:  I can provide you the  snapcraft.yaml if you want
[14:47] <ogra_> mvo, i think the countdown runs before anything gets changed ... :/
[14:47] <Chipaca> carroarmato0: no, thank you, i have a minimal reproducer as i said
[14:47] <zyga-solus> Chipaca: let me look
[14:47] <zyga-solus> Chipaca: probably because we don't reuse the regexp
[14:48] <Chipaca> zyga-solus: that's the how, not the why :-)
[14:48] <zyga-solus> looking
[14:48] <Chipaca> zyga-solus: there's even a comment in snap.ValidateName which says "keep in sync wiht these two others"
[14:48] <Chipaca> zyga-solus: we obviously haven't :-)
[14:48] <zyga-solus> I know, I wrote the comment
[14:49] <Chipaca> zyga-solus: the regexp in snapd is basically [a-z0-9]* except it must have at least one [a-z]
[14:50] <zyga-solus> what's the snap name?
[14:50] <carroarmato0> Chipaca: zyga-solus:  it's  12to8
[14:50] <zyga-solus> 12to8, gotcha
[14:50] <Chipaca> carroarmato0: anything starting with a number
[14:50] <carroarmato0> Chipaca: zyga-solus:  indeed
[14:50] <zyga-solus> testing
[14:50] <Chipaca> zyga-solus: http://pastebin.ubuntu.com/25732561/
[14:50] <cachio> ogra_, I don't see that
[14:51] <cachio> ogra_, I use an extreme sdcard
[14:52] <zyga-solus> hmm
[14:52] <zyga-solus> +       // Regression test: 12to8
[14:52] <zyga-solus> +       sc_snap_name_validate("12to8", &err);
[14:52] <zyga-solus> +       g_assert_null(err);
[14:52] <zyga-solus> Chipaca: this passes
[14:52] <zyga-solus> maybe we ... have another routine?
[14:52] <Chipaca> zyga-solus: it's the app name that's the problem, not the snap name
[14:52] <zyga-solus> maybe not snap name but security token
[14:52] <zyga-solus> what's the error message?
[14:52] <Chipaca> security tag snap.123test.123test not allowed
[14:52] <zyga-solus> right
[14:52] <zyga-solus> we use
[14:52] <zyga-solus> +       // Regression test: 12to8
[14:52] <zyga-solus> +       sc_snap_name_validate("12to8", &err);
[14:52] <zyga-solus> +       g_assert_null(err);
[14:52] <zyga-solus> or I cannot copy and paste
[14:53] <Chipaca> :-)
[14:53] <mvo> cachio: so the log is nice, it shows are doing a single reboot only, so somehow the snap_mode is not changed *or* it is not written to disk, one of those
[14:53]  * Chipaca hugs zyga-solus 
[14:53] <zyga-solus> wow, copy is greyed out
[14:53] <zyga-solus> WTF
[14:54] <zyga-solus>             "^snap\\.([a-z](-?[a-z0-9])*)\\.([a-zA-Z0-9](-?[a-zA-Z0-9])*|hook\\.[a-z](-?[a-z])*)$";
[14:54] <zyga-solus> must be vim with some fancy pants terminal mouse support in solus
[14:54] <zyga-solus> so that's the regexp
[14:54] <Chipaca> zyga-solus: yeah, that's notably different
[14:54]  * zyga-solus compares to snap.go
[14:55] <zyga-solus> I see
[14:55] <zyga-solus> Chipaca: changing
[14:55] <Chipaca> zyga-solus: ^[a-zA-Z0-9](?:-?[a-zA-Z0-9])*$
[14:55] <carroarmato0> Chipaca:  zyga-solus:  0:)  http://www.osnews.com/images/comics/wtfm.jpg
[14:55] <Chipaca> zyga-solus: also the hook one will fail
[14:56] <Chipaca> carroarmato0: i think zyga-solus's wtf was about copy being greyed out, but it doesn't mean you're not right
[14:56] <cachio> mvo, ogra_ this is the printenv https://paste.ubuntu.com/25732587/
[14:56] <zyga-solus> Chipaca: I think we have one more copy
[14:56] <cachio> do you see anything weird on there?
[14:56] <Chipaca> carroarmato0: also I don't know if 0:) is a smiling angel or a shouting person with a unibrow
[14:56] <Chipaca> zyga-solus: we do
[14:56] <Chipaca> probably
[14:57] <cachio> mvo, ogra_ this is stopping the boot
[14:57] <zyga-solus> Chipaca: I think in snap-update-ns
[14:57] <Chipaca> we always do things in threes
[14:57] <zyga-solus> in the C code there
[14:57] <carroarmato0> Chipaca:  smiling angel it is
[14:57] <ogra_> cachio, thats fine, gimme a sec
[14:58] <ogra_> cachio, try: run snappy_boot
[14:58] <ogra_> that should move on
[14:58] <zyga-solus> Chipaca: what is (?:...)
[14:58] <ogra_> see if that changes anything, prints errrors etc
[14:59] <kyrofa> Chipaca, carroarmato0 disallowing numbers at the beginning is on purpose, I remember the discussion
[15:00] <zyga-solus> kyrofa: we changed that when 2048 came out
[15:00] <zyga-solus> or something like that
[15:00] <carroarmato0> zyga-solus:  hahahaha
[15:00] <carroarmato0> zyga-solus:  priorities :D
[15:00] <zyga-solus> kyrofa: it's just internal misalignment in snapd
[15:00] <cachio> ogra_, in progress
[15:01]  * ogra_ is in a meetzing now
[15:01] <zyga-solus> woah
[15:01] <zyga-solus> Chipaca: why do we allow UPPER CASE snap names?
[15:01] <zyga-solus> Chipaca: in snap-confine
[15:01] <zyga-solus> man this is hairy
[15:01] <kyrofa> zyga-solus, so... snapcraft and the store need to be updated now? Do we have a cross-project bug or anything?
[15:01] <ogra_> for developers with tourette ?
[15:01] <Chipaca> zyga-solus: not snap names, but yes snap apps
[15:02] <zyga-solus> kyrofa: not sure, I think it's only snap-confine accepting the same set as snapd (for now)
[15:02] <zyga-solus> Chipaca: aah
[15:02] <zyga-solus> that makes sense
[15:02] <Chipaca> kyrofa: do they though?
[15:03] <Chipaca> carroarmato0: did you publish your snap?
[15:03] <carroarmato0> Chipaca:   no I haven't. Have been playing around with it to learn how snaps work
[15:03] <Chipaca> kyrofa: i ask because carroarmato0 here built a snap with an app both of which started with a number, so at least snapcraft seems to be a'ight
[15:03] <kyrofa> Chipaca, ah, no indeed
[15:03] <kyrofa> Chipaca, I was mistaken. We must have made that change along with you
[15:04] <Chipaca> kyrofa: it's as if we sometimes talked to each other
[15:04] <carroarmato0> Chipaca:  the application is opensource, so if you want I could look into publishing it
[15:04] <Chipaca> kyrofa: sounds horrible
[15:04] <kyrofa> Chipaca, blech, you think I'd remember
[15:04] <Chipaca> carroarmato0: there's no rush
[15:04] <Chipaca> kyrofa: it probably involved beer
[15:04] <kyrofa> Probably
[15:04] <carroarmato0> Chipaca:  but not sure if I'll keep it updated though
[15:05] <zyga-solus> libsnap-confine-private/classic-test.c:55:13: warning: ‘test_is_on_classic_with_long_line’ defined but not used [-Wunused-function]
[15:05] <Chipaca> carroarmato0: probably best to not do it until you're sure, then
[15:05]  * zyga-solus prepares more fixes
[15:05] <carroarmato0> Chipaca: indeed, don't want to deploy abandonware :)
[15:05] <Chipaca> carroarmato0: you could install xbill-xaw and use it to reflect about what you nearly did
[15:06]  * zyga-solus looks at other parts of the code
[15:06] <zyga-solus> carroarmato0: the good news is that this will land today and will be in the release
[15:06] <carroarmato0> zyga-solus:  oh cool.
[15:07] <carroarmato0> zyga-solus:  as soon as it lands will keep the ticket on launchpad updated
[15:07] <zyga-solus> Chipaca: can you review please
[15:07] <mup> PR snapd#4043 opened: cmd/snap-confine: update valid security tag regexp <Created by zyga> <https://github.com/snapcore/snapd/pull/4043>
[15:07] <zyga-solus> I'll do one more PR
[15:09] <cachio> ogra_, it did not started any more
[15:09] <cachio> https://paste.ubuntu.com/25732634/
[15:09] <carroarmato0> zyga-solus: Chipaca:  I will be going now.  Thanks for the assistance. Will keep an eye out for the update and update the issue in launchpad. :) cheers
[15:09] <cachio> ogra_, it hangs
[15:09] <zyga-solus>  o/
[15:12] <cachio> mvo, ogra_ I'll be off 30 minutes
[15:12] <ogra_> cachio, hmm, are you sure it hangs and didnt just switch to tty console ?
[15:12] <Chipaca> zyga-solus: won't that fail if a snap called 123 has the temerity of shipping a hook?
[15:12] <zyga-solus> ikey: hey, where is the indent git tree for me to clone/build
[15:12] <Chipaca> 123test*
[15:13] <zyga-solus> Chipaca: looking
[15:13] <cachio> ogra_, I can't connect through ssh anymore
[15:13] <ogra_> cachio, and no screen attched ?
[15:13] <ogra_> (could be that the direct running of snappy_boot simplly omits the serial tty option from the cmdline)
[15:13] <ikey> zyga-solus, oh didnt know you wanted me to push it
[15:13] <ikey> gizza sec
[15:14] <cachio> ogra_, just 1 screen and it is attached
[15:14] <zyga-solus> ikey: yeah, it could save me a hop to an ubuntu machine :)
[15:14] <ogra_> cachio, nothing on that ?
[15:14] <ikey> ya giz 2 secs
[15:15] <cachio> ogra_, the output of that screen is what I pasted
[15:15] <Chipaca> oSoMoN: did you call play0ad play0ad because 0ad didn't work, or because of some other reason?
[15:15] <ogra_> cachio, i meant a HDMI screen ... not the screen app :)
[15:16] <ikey> zyga-solus, https://dev.solus-project.com/source/indent.git
[15:16] <zyga-solus> Chipaca: +       g_assert_true(verify_security_tag("snap.123test.hook.configure", "123test"));
[15:16] <ikey> and i stuffed the build in unstable repo for you
[15:16] <zyga-solus>  
[15:16] <zyga-solus> this passes
[15:16] <zyga-solus> ikey: thank you!
[15:17] <zyga-solus> Chipaca: did I understand your question correctly? this is correct, right?
[15:17] <oSoMoN> Chipaca, "0ad" is not allowed because it starts with a digit… there's a bug filed for that, let me try and find it
[15:17] <zyga-solus> oSoMoN: it should be allowed (by snapd) nowadays
[15:17] <Chipaca> zyga-solus: did you change the regexp, or did I misread it the first time?
[15:18] <zyga-solus> Chipaca: I didn't change it
[15:18] <zyga-solus> Chipaca: I force-pushed to fix one tests that now passes
[15:18] <Chipaca> zyga-solus: I shall put on the dunce hat and sit in the corner and thing about cookies
[15:19] <cachio> ogra_, seending a pic by email
[15:19] <ogra_> ok
[15:19] <cachio> I'll need to go now
[15:20] <Chipaca> zyga-solus: +1
[15:20] <zyga-solus> Chipaca: I'll make fmt (after installing indent) and push
[15:20] <cachio> ogra_, which is your email?
[15:21] <ogra_> cachio, oga@ubuntu.com (or @canonical.com as you like)
[15:21] <oSoMoN> Chipaca, I can't find that bug again, I'd swear I had filed one
[15:21] <cachio> oga or ogra?
[15:21] <oSoMoN> zyga-solus, oh that's good news, I'll try again when I get some spare time
[15:21] <ogra_> cachio, damn ... my R key is shaky ... ogra indeed
[15:22] <cachio> ogra_, sent
[15:23] <zyga-solus> ikey|afk: woot, I also learned about using different profiles :)
[15:23] <ikey|afk> :D
[15:23] <ikey|afk> -p main-x86_64 ^^
[15:23] <ikey|afk> ok bbiab
[15:24] <mup> PR snapd#4044 opened: cmd/libsnap: enable two stranded tests <Created by zyga> <https://github.com/snapcore/snapd/pull/4044>
[15:24] <zyga-solus> right :)
[15:24] <zyga-solus> mvo: https://github.com/snapcore/snapd/milestone/13 we have three branches now
[15:27] <zyga-solus> jdstrand: kind ping about https://github.com/snapcore/snapd/pull/4042
[15:27] <mup> PR #4042: snap-confine: cleanup incorrectly created nvidia udev tags <Created by mvo5> <https://github.com/snapcore/snapd/pull/4042>
[15:28] <zyga-solus> ogra_: hey
[15:28] <zyga-solus> ogra_: so, I'm lost with this FAT/pi3 issue
[15:28] <zyga-solus> ogra_: can you please explain it to me how you understand it?
[15:30] <ogra_> zyga-solus, i dont yet ...
[15:30] <ogra_> zyga-solus, point is that none of the messages mentioned yet were anything unusual
[15:31] <zyga-solus> did cachio upload his image anywhere?
[15:31] <zyga-solus> (of his card)
[15:31] <ogra_> he mailed it to me ... and it shows that i was wrong assuming "run snappy_boot" would be enough ...
[15:31] <zyga-solus> no, I mean his whole SD card image
[15:31] <ogra_> it is useless (misses bits from the cmdline)
[15:32] <ogra_> nope
[15:32] <ogra_> only a pic from the screen
[15:33] <ogra_> zyga-solus, i doubt that would help you though
[15:34] <ogra_> ah
[15:34] <ogra_> [    0.000000] Kernel command line: 8250.nr_uarts=1 dma.dmachans=0x7f35 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2709.boardrev=0xa22082 bcm2709.serial=0x1dffa1eb smsc95xx.macaddr=B8:27:EB:FF:A1:EB bcm2708_fb.fbswap=1 bcm2709.uart_clock=48000000 vc_mem.mem_base=0x3dc00000 vc_mem.mem_size=0x3f000000  dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty0 elevator=deadline root=/dev/disk/by-label/writable net.ifnames=0
[15:34] <ogra_> init=/lib/systemd/systemd ro panic=-1 fixrtc snap_core=core_x1.snap snap_kernel=pi2-kernel_43.snap
[15:34] <ogra_> so manually rrunning snappy_boot switched to the core_x1 properly ... at least we know that bit
[15:36] <ogra_> hmm
[15:36] <ogra_> and the cmdline actually looks correct ...
[15:36] <ogra_> so why does he end up in an initrd prompt then ...
[15:39] <ogra_> https://imgur.com/a/YA3p9 is the image he mailed ... and that looks actually morre like a generral SD breakage
[15:40] <ogra_> sadly the actual error would be several lines above ... what you see thee is onlly fallout
[15:41] <ogra_> hmm
[15:41] <ogra_> unless core_x1.snap is actually corrupt ...
[15:43] <ogra_> mvo, do we know how exactly core_x1.snap was assembled ? i'm wondering if the bootloader doesnt actually DTRT here and simply switch back as it should
[15:43] <ogra_> that photo seems to agree that there is something wrong with the rootfs assembling
[15:44] <ogra_> anyway ... i need to go afk too for at least 2h
[15:47] <zyga-solus> ogra_: yeah, it looks like the SD card reads lorem ipsum on all blocks
[15:47] <zyga-solus> ogra_: I think it's a test-made core
[15:47] <zyga-solus> ogra_: looking at the test ...
[15:47] <ogra_> zyga-solus, well, i wouldnt go that far ... but probably on the core squashfs
[15:48] <zyga-solus> http://pastebin.ubuntu.com/25732826/
[15:48] <zyga-solus> this is the test that was executed
[15:48]  * zyga-solus reads
[15:48] <zyga-solus> actually
[15:49] <zyga-solus> since this runs on ubuntu-core-16 target, I suspect it may contain a core that was built from the git tree
[15:49] <zyga-solus> (with some hackery obviously)
[15:49] <zyga-solus> but not sure, I've never used external targets
[15:49]  * zyga-solus looks
[15:50] <zyga-solus>     0) cmd="snap install --dangerous /var/lib/snapd/snaps/core_$(cat prevBoot).snap" ;;
[15:50] <zyga-solus> it looks like just previous core snap is sideloaded
[15:51] <ogra_> i wonder why he doesnt just use --revision and the store
[15:51] <ogra_> (saves you from using --dangerous ... )
[15:52] <zyga-solus> probably simplicity
[15:52] <ogra_> anyway, really out now (need to mow the lawn before it gets to dark)
[15:52] <zyga-solus> ok
[15:52] <zyga-solus> lol
[15:52] <zyga-solus> it's dark already here :)
[15:52] <ogra_> the simplicity of building a complete core vs using one from the store ?
[15:52]  * zyga-solus imagines ogra with a flashlight and a mower 
[15:52] <zyga-solus> ogra_: it's not building one, it's just installing one on disk
[15:53] <zyga-solus> simplicity of passing --dangerous :)
[15:53] <ogra_> haha, not that dark yet ... i want to manager to finish at least half the garden before 19:00
[15:53] <ogra_> ah
[15:53] <zyga-solus> go
[15:53] <zyga-solus> :)
[16:05] <cachio> zyga-solus, the upload failed
[16:05] <cachio> grrrrr
[16:06] <cachio> zyga-solus, I'll build it on jenkins
[16:06] <cachio> zyga-solus, so you can download from there
[16:07] <mvo> zyga-solus: hey, thanks, looking at the PRs now
[16:08] <zyga-suse> mvo: thank you
[16:08] <mvo> cachio_: hey, anything new, new theories? I was also wondering if the same problem is reproducable in the edge core
[16:09] <cachio_> mvo, no, did you see the image that I sent?
[16:09] <zyga-suse> mvo: no new theories
[16:10] <mvo> cachio_: no, could you paste me the link again please?
[16:10] <mvo> zyga-suse: did you manage to reproduce it as well?
[16:10] <cachio_> mvo, which link?
[16:10] <mvo> cachio_: the image
[16:11] <mvo> cachio_: aha, i have it now
[16:11] <zyga-solus> mvo: no, no luck, cachio_ didn't upload the image he was using, I suspect it may be his card more than the image though
[16:11] <ogra_> mvo, https://imgur.com/a/YA3p9
[16:12] <mvo> cachio_: autsch - when/how did that happen? i.e. after what action?
[16:12] <ogra_> mvo, i dont think it is the bootloader at all but the self assembled core_x1.snap
[16:12] <ogra_> mvo, after i asked him to stop the boot and "run snappy_boot" manually
[16:12] <ogra_> mvo, the dmesg output shows the cmdline is fine so i suspect there is something with the core snap itself
[16:13] <ogra_> mvo, which woulld mean the bootloader does exactly the right thing and rolls back
[16:13] <zyga-solus_> re, sorry
[16:14] <mvo> ogra_: that is just the core from /var/lib/snapd/snapd/core_$(currentBoot), so its essentially the snap downloaded from the store
[16:15] <mvo> cachio_: do you have the busybox prompt in front of you?
[16:15] <mvo> cachio_: if so, does ls /var/lib/snapd/snaps show a core_x1.snap ?
[16:15] <ogra_> mvo, why is it called core_x1.snap then ?
[16:15] <ogra_> oh
[16:15] <ogra_> because it was sideloaded before
[16:15] <ogra_> indeed
[16:16] <ogra_> silly me
[16:17] <cachio_> mvo, let me connect a keyboard
[16:17] <mvo> cachio_: thanks
[16:20] <cachio_> doesn't work the keyboard with the pi3 at this momento
[16:21] <cachio_> mvo, I tried with 2 and not working
[16:22] <ogra_> likely no usbhid module in the initrd ... :/
[16:22] <ogra_> you coudl try to boot again with console=tty1 dropped from the cmdline.txt
[16:22] <ogra_> that should give you the initrd shell on serial
[16:27] <cachio_> ogra_, tty1 or tty0?
[16:27] <ogra_> well, the "not serial" one :)
[16:28] <cachio_> ogra_, I have this time dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty0 elevator=deadline
[16:28] <ogra_> remove "console=tty0"
[16:29] <ogra_> then all output should only go to serial
[16:29] <ogra_> (including the initrd shell)
[16:29] <cachio_> ok
[16:30] <ogra_> the first console= arg on a kernel cmdline always means ""here prints the kernel to" the seond means "print all userspace output to this" (i.e. after starting initrd) ... if you have only one console= it will be used for all output
[16:30] <ogra_> (and input indeed)
[16:30] <cachio_> excelent
[16:32] <cachio_> ogra
[16:32] <cachio_> (initramfs) ls /var/lib
[16:32] <cachio_> ls: /var/lib: No such file or directory
[16:32] <ogra_> ls /root/var/lib
[16:32] <ogra_> the writable partition should be mounted under /root
[16:32] <cachio_> ok
[16:32] <mvo> ogra_, cachio_ - so there is a new kernel in edge,beta,candidate for pi2, uploaded 2days, 7h ago
[16:33] <mvo> cachio_, ogra_ I wonder if using the stable kernel/gadget makes a difference?
[16:34] <mvo> cachio_ ogra_ or if the previous version of the kernel (r42) would help.
[16:34] <cachio_> mvo, we can try
[16:35] <mvo> cachio_: does this match the time when the problems started? 2d ago?
[16:35] <cachio_> mvo, yes
[16:35] <ogra_> ogra@pi3:~$ snap list pi2-kernel
[16:35] <ogra_> Name        Version        Rev  Developer  Notes
[16:35] <ogra_> pi2-kernel  4.4.0.1075.75  43   canonical  kernel
[16:36] <ogra_> mvo, that kernel silently upgraded on my pi3 here
[16:36] <ogra_> (and works fine)
[16:36] <ogra_> mvo, you cant boot the stable kernel ... it will fail
[16:36] <mvo> ogra_: right, I'm sure its fine most of the time, but I'm a bit desperate right now
[16:37] <mvo> ogra_ cachio_ so no stable kernel, I can make r42 available again I think
[16:37] <ogra_> (well, it might boot but many bits will be missing or not work)
[16:37] <ogra_> yeah, use edge
[16:37] <ogra_> so cachio_ can refresh to edge
[16:37] <mvo> ogra_: the thing is, this test started to fail ~2d ago and all changes in snapd/core we did are unrelated to booting
[16:37] <ogra_> yeah
[16:38] <mvo> ppisati: I switched the pi2-kernel r42 to edge for a short time
[16:38] <mvo> cachio_: snap refresh --edge pi2-kernel
[16:38] <mvo> cachio_: should give you r42 now
[16:38] <mvo> cachio_: this version is ~3 weeks old
[16:38]  * ogra_ goes to pack up the lawnmower and stuff 
[16:38] <ogra_> brb
[16:39] <cachio_> mvo,  I need 10 minutes
[16:39] <cachio_> I am flashing again
[16:40] <mvo> cachio_: sure, thanks a lot for your diligence in this issue. I'm excited about this potential clue, this is the best theory in the last few hours
[16:45] <jdstrand> mvo, zyga-solus: re 4042, done
[16:46]  * jdstrand steps away
[16:46] <kyrofa> Any snapcraft peeps around today?
[16:46] <kyrofa> GH is super quiet
[16:47] <ogra_> kyrofa, talk to that kyrofa guy, i think he works on snapcraft at times ;)
[16:47] <cachio_> mvo, refreshing ...
[16:49] <jdstrand> mvo, zyga-solus: after thinking about it, please answer https://github.com/snapcore/snapd/pull/4042#issuecomment-336506560
[16:49] <mup> PR #4042: snap-confine: cleanup incorrectly created nvidia udev tags <Created by mvo5> <https://github.com/snapcore/snapd/pull/4042>
[16:54] <kyrofa> ogra_, I HAVE been! And I'm STILL lonely
[16:54]  * ogra_ hugs kyrofa 
[16:55] <kyrofa> Haha
[16:58] <zyga-solus> re
[16:58] <zyga-solus> jdstrand: looking
[16:58] <zyga-solus> jdstrand: thank you for your review
[16:59] <zyga-solus> mvo: I'm baby-sitting family computers (sigh), I'll look at the patch and spread test in ~30
[17:00] <zyga-solus> mvo: shall we call it a (release) day and regroup early tomorrow to build a core snap and push it for testing
[17:02] <zyga-solus> mvo: please reply to jdstrand's question as well, I'm checking the tree
[17:07] <cachio_> mvo, running test
[17:07] <cachio_> using kernel 42
[17:14] <mvo> zyga-solus: maybe, I can push the release to beta, thats fine. what patch/spread are you refering to? for 4042?
[17:14] <mvo> cachio_: ohhhh, I'm curious how it goes
[17:15] <zyga-solus> mvo: patch requested by jdstrand and spread test to ensure this works
[17:15] <zyga-solus> mvo: please see his feedback
[17:16] <mvo> zyga-solus: excellent, thank you
[17:17] <kalikiana> kyrofa: would you feel like having a look at this PR? https://github.com/snapcore/snapcraft/pull/1613 not mine for a change, tho I added the unit test for it
[17:17] <mup> PR snapcraft#1613: cross compilation: enable cross compilation of i386 kernel on x86-64 … <Created by piso77> <https://github.com/snapcore/snapcraft/pull/1613>
[17:18] <cachio_> mvo, same issue
[17:18] <kyrofa> kalikiana, sure. I've got some up for review as well
[17:18] <zyga-solus> mvo: but disclaimer, still hacking on family crap PCs
[17:19] <kalikiana> kyrofa: I'm just looking at the queue now, so will get right to it
[17:19] <mvo> cachio_: /o\
[17:21] <ogra_> cachio_, grep . /sys/class/mmc_host/mmc0/mmc0:*/* 2>/dev/null
[17:21] <ogra_> can you paste the output ?
[17:21] <ogra_> (on the pi)
[17:22] <kalikiana> kyrofa: quick q on https://github.com/snapcore/snapcraft/pull/1614 - do you have any reference/ bug for the change? I think it makes complete sense, just wondering if there's some background to follow up on later
[17:22] <mup> PR snapcraft#1614: apt repo: allow insecure repos <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/1614>
[17:22] <kyrofa> kalikiana, no, it's the ROS error we're getting in the autopkgtests for anything beyond xenial
[17:23] <zyga-solus> ogra_: good idea
[17:23] <zyga-solus> ogra_: we can see the manufacturer ID
[17:23] <mvo> zyga-solus: I will deal with the feedback from jamie, no worries, fix the PC :)
[17:23] <ogra_> zyga-solus, and the vendor ... and compare it to http://paste.ubuntu.com/25566047/ and http://pastebin.ubuntu.com/25566134/ which are two cards i know are failing regulary
[17:23] <cachio_> ogra_, https://paste.ubuntu.com/25733320/
[17:23] <zyga-solus> mvo: I'd rather not but "bla bla bla, internet is broken (shouting)"
[17:24] <zyga-solus> ogra_: yeah, good idea
[17:24] <ogra_> yeah, but no match
[17:24] <ogra_> its a sandisk ... but not a mmodel i know as failing
[17:25] <kalikiana> kyrofa: Ah. Alright. Maybe file a bug for it as a reminder.. approved in any case
[17:25] <kyrofa> kalikiana, we need a discussion around it, I need to write a forum post
[17:25] <kyrofa> At least... I think that's what I'm supposed to do these days :P
[17:26] <mvo> zyga-solus: do you remember what PR introduced adding the nvidia udev tags?
[17:26] <zyga-solus> mvo: I think the one from garry, let me find it
[17:26] <kalikiana> kyrofa: True, bugs don't work as well for proper discussions - which reminds me, wasn't there something else you were going to post in the forum? I'm trying to recall, it was something I was going to chime in on...
[17:27] <mvo> zyga-solus: I want to improve my comment to make sure i refer to the right PR
[17:27] <kyrofa> kalikiana, yeah, our ideal for slow tests
[17:27] <mvo> zyga-solus: I think I can use 4022 as the reference, nevermind
[17:27] <ogra_> cachio_, any chance you can try with another SD to make sure its not that ?
[17:28] <kalikiana> kyrofa: Yup, that's it. Feel free to poke me once you post it, in case I don't see it right away
[17:29] <zyga-solus> mvo: one sec
[17:29] <cachio_> ogra_, I tried with 2
[17:29] <cachio_> ogra_, I can try another one
[17:29] <ogra_> both sandisk ?
[17:29] <zyga-solus> mvo: https://github.com/snapcore/snapd/pull/3617/files
[17:29] <mup> PR #3617: interfaces/builtin: use udev tagging more broadly <Created by adglkh> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/3617>
[17:29] <zyga-solus> this one
[17:33] <cachio_> ogra_, yes
[17:33] <ogra_> do you have a non-sandisk ?
[17:33] <cachio_> ogra_, yes
[17:33] <cachio_> a toshiba one
[17:33] <ogra_> can you try with that one ?
[17:34] <cachio_> yes
[17:34] <ogra_> just to make sure ...
[17:40] <cachio_> ogra_, running
[17:46]  * Chipaca quietly walks out the back and pops a beer open
[17:48] <mvo> cachio_: how is it looking so far? with the different card?
[17:49] <zyga-solus_> mvo: I see the patch now
[17:50] <Son_Goku> openSUSE infrastructure is now down
[17:50] <zyga-solus_> mvo: woah, nice spread test
[17:50] <zyga-solus_> mvo: ^^
[17:50] <zyga-solus_> mvo: you will have failing opensuse tests
[17:50] <Son_Goku> mvo: all openSUSE tests need to be disabled for the next two days
[17:51] <zyga-solus_> two days? are they re-painting the data center?
[17:51] <Son_Goku> they're doing maintenance on the power systems
[17:51] <mvo> Son_Goku: woah, thank you
[17:51] <mvo> zyga-solus_: thanks
[17:51] <Son_Goku> it broke us here at work too ;)
[17:52] <mvo> zyga-solus_: we can simply disable suse then (set to manual)
[17:52] <zyga-solus_> mvo: yes
[17:52] <zyga-solus_> mvo: let's do it in this PR
[17:52] <zyga-solus_> mvo: so that we can land one PR instead of two
[17:52] <cachio_> mvo, ogra_ with the toshiba card same error
[17:52] <zyga-solus_> Son_Goku: thank you for the tip
[17:52] <zyga-solus_> cachio_: stamp and send the card to mvo for debugging
[17:52] <mvo> zyga-solus_: looks like opensuse is already on manual
[17:52] <zyga-solus_> cachio_: (physical card)
[17:52] <ogra_> cachio_, thanks, then it is definitely not a card issue
[17:52] <zyga-solus_> mvo: ooops, I didn't know that
[17:53] <zyga-solus_> (not great but I'll deal with that)
[17:53] <Son_Goku> mvo: note that the openSUSE repositories will be up
[17:53]  * zyga-solus_ writes this down
[17:53] <Son_Goku> but osc build will not work
[17:56] <zyga-solus_> mvo: what is this for?
[17:56] <zyga-solus_> +    mv $SNAP_MOUNT_DIR/core/current.renamed $SNAP_MOUNT_DIR/core/current
[17:56] <zyga-solus_> it seems unrelated
[17:56] <mvo> zyga-solus_: the previous test is causing havoc
[17:56] <zyga-solus_> aah
[17:56] <zyga-solus_> thanks
[17:56] <mvo> zyga-solus_: its breaking things on purpose
[17:56] <zyga-solus_> right
[17:56] <zyga-solus_> and not doing restore, gotcha
[17:57] <zyga-solus_> cachio_: I'm 90% serious about that SD card
[17:57] <cachio_> zyga-solus_, I live in cordoba, it could take 1 week :)
[17:58] <zyga-solus_> cachio_: that's fine
[17:58] <zyga-solus_> cachio_: I think it's something we should understand for 2.29.x
[17:59] <ogra_> cachio_, send a pidgeon ;)
[17:59] <ogra_> cachio_, so you see this prob since 2.28.4 landed ? could you do the same test with a candidate image (which still has 2.28.1 ) ?
[18:00] <mvo> cachio_: or use the same image and "snap refresh --candidate core"
[18:00] <mvo> cachio_: and then do the test?
[18:01] <cachio_> mvo, refreshing
[18:01] <mvo> ta
[18:01] <Son_Goku> zyga-solus_: for building the packages for openSUSE, are we using `osc build`?
[18:03] <mup> PR snapd#4044 closed: cmd/libsnap: enable two stranded tests <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4044>
[18:03] <mvo> zyga-solus_: 4044 does not cherry-pick without conflicts, I will skip that one (as its tests only)
[18:04] <mup> PR snapd#4043 closed: cmd/snap-confine: update valid security tag regexp <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4043>
[18:04]  * mvo waits for tests for 4042
[18:24] <kalikiana> elopio: I think you might want to re-trigger these tests? I see some network outage there causing "failures"
[18:24] <kalikiana> https://github.com/snapcore/snapcraft/pull/1602
[18:24] <mup> PR snapcraft#1602: tests: add the slow tag for ros snapd integration test <Created by elopio> <https://github.com/snapcore/snapcraft/pull/1602>
[18:25]  * elopio checks
[18:25] <elopio> kalikiana: we need Kyle's branch first to get everything to green.
[18:26] <kalikiana> kyrofa: I see some conflicts here https://github.com/snapcore/snapcraft/pull/1602
[18:26] <mup> PR snapcraft#1602: tests: add the slow tag for ros snapd integration test <Created by elopio> <https://github.com/snapcore/snapcraft/pull/1602>
[18:26] <kalikiana> elopio: Hm? I thought it's the other way around?
[18:26] <kalikiana> Maybe I'm getting "hen and egg" confusion here
[18:26] <kyrofa> elopio, +1 it then, let's get it in there
[18:26] <kalikiana> The failures at least look like false negatives to me
[18:26] <kyrofa> elopio, well, perhaps we should run an autopkgtest on it?
[18:27] <kyrofa> kalikiana, not the zesty one
[18:27] <elopio> well, yes, we need everything to be green :D
[18:27] <kyrofa> kalikiana, it dies with a NO_PUBKEY
[18:27] <elopio> 1603 and 1614 are chicken and egg.
[18:28] <kalikiana> Oha. Indeed.
[18:28] <kalikiana> kyrofa: Right, *that* PR. Now I see where I was confused :-D
[18:29]  * kalikiana morns launchpad prerequisite branches for a moment
[18:29] <kyrofa> kalikiana, yeah I feel ya
[18:30] <kyrofa> elopio, not chicken and egg, 1614 is required for 1603. It's more like chicken and human
[18:30] <kyrofa> Human requires the chicken in order to eat it
[18:31] <kalikiana> lol
[18:31] <kalikiana> kyrofa: Now I see your old avatar as I'm reading this, where you look very ominous and slightly philosophical
[18:32] <kyrofa> My old avatar? I looked like a child in my old one, not philosophical
[18:32] <elopio> well everything is approved. We just need Sergio to push the merge button.
[18:33] <kyrofa> elopio, I finally have that power again, do we need to wait?
[18:33] <cachio_> mvo, https://paste.ubuntu.com/25733615/
[18:33] <cachio_> I'll need to reflash againg
[18:35] <elopio> kyrofa: no need to wait. Maybe just run autopkgtest in your 1614 to check that the error is indeed gone.
[18:35] <kyrofa> elopio, zesty good enough? Or artful as well?
[18:35] <mvo> cachio_: oh, so the same error ?
[18:35] <jdstrand> mvo: hey, are you using the system-image seed to build the core snap or something else?
[18:35] <mvo> cachio_: snap change 6 ? does that show the same
[18:36] <cachio_> mvo, I could not go to 2.28.1
[18:36] <mvo> jdstrand: we are using live-build, why?
[18:36] <mvo> cachio_: aha, so what is this pastebin about, what version was used in the test?
[18:36] <kyrofa> snappy-m-o, autopkgtest something something
[18:36] <snappy-m-o> Computer says nooo. See logs for details:
[18:36] <snappy-m-o>  Command '['/tmp/tmpc66uz1ql/retry_autopkgtest.sh', 'something', 'something']' returned non-zero exit status 1
[18:36] <kyrofa> Oh good, I was close
[18:36] <jdstrand> mvo: and does live-build use seeds?
[18:37] <mvo> jdstrand: I think so, why?
[18:37] <jdstrand> mvo: we have a sizing request and want to be looking at the right thing to understand what is exactly being pulled in and how
[18:37] <elopio> kyrofa: just one, anyone.
[18:37] <kyrofa> snappy-m-o, autopkgtest 1614 zesty:amd64
[18:37] <snappy-m-o> kyrofa: I've just triggered your test.
[18:37] <mvo> jdstrand: aha, I think we have some room here, plus base snaps will make this a lot easier in the near future
[18:37] <cachio_> mvo, I was in 28.4 after run the test
[18:37] <mvo> jdstrand: /usr/share/snappy/dpkg.list
[18:37] <mvo> cachio_: ok
[18:37] <cachio_> mvo, I tried to refresh to candidate but it failed
[18:38] <mvo> cachio_: oh, that failed too
[18:38] <jdstrand> mvo: yep, aware of that file. ok thanks
[18:38] <cachio_> I am refreshing again from a new one
[18:38] <mvo> cachio_: so it seems refreshes are broken for all cores
[18:38] <mvo> cachio_: what kind of error do you get? i.e. snap change 8
[18:40] <cachio_> mvo, 1 minute, I am checking in my other pu3
[18:41] <cachio_> I need a bigger usb hub
[18:41] <zyga-solus_> re
[18:42] <zyga-solus_> mvo: ack about skipping that PR
[18:42] <zyga-solus_> how are things?
[18:50] <Pharaoh_Atem> mvo: the dates are wrong in the fedora changelog
[18:50] <Pharaoh_Atem> you put Tue Oct 11, but Oct 11 was a Wednesday
[18:58] <cachio_> mvo, same error with 28.1
[18:58] <cachio_> mvo, I am gonna run with another pi3
[18:59] <cachio_> to see if the problem is in the board
[19:20] <mup> PR snapcraft#1618 opened: states: add scriptlets to build state <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/1618>
[19:37] <mvo> cachio_: thanks a lot
[19:38] <mvo> zyga-solus_: thinks are so so
[19:38] <mvo> zyga-solus_: if we are happy about 4042 I will merge it into 2.28
[19:41] <cachio_> mvo, I don't know what else to try
[19:41] <cachio_> mvo, I'll try to save in disk the images that I use for beta validation
[19:42] <cachio_> so if we need to reproduce something we have the images used for the tests
[19:44] <mvo> cachio_: yeah, I ordered a pi3, should be here early next week and I can try to make sense of this. testing in a second pi3 is certainly an idea
[19:51] <cachio_> mvo, I already tested in 2 pi3
[19:52] <mvo> cachio_: oh and always the same
[19:53] <cachio_> mvo, yes
[19:53] <cachio_> this is the output of the change
[19:53] <cachio_> https://paste.ubuntu.com/25734001/
[19:53] <cachio_> from beta to candidate after the test fail
[19:53] <mvo> cachio_: same error
[19:53] <cachio_> mvo, ye
[19:53] <cachio_> s
[19:58] <mup> PR snapd#4042 closed: snap-confine: cleanup incorrectly created nvidia udev tags <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4042>
[19:59] <zyga-solus_> re
[20:01] <mvo> zyga-solus_: any last concerns about 4042 or release/2.28? if not I will push to beta
[20:01] <mvo> zyga-solus_: cachio_ had the refresh error now with 2.28.1 too, all a bit of a myserty, maybe you can try to reproduce with your pi3 using the same steps on monday. I will do so as well once my pi3 is here
[20:02]  * mvo spread tests 4042
[20:04] <zyga-solus_> mvo: let's see
[20:04] <zyga-solus_> nick zyga-solus
[20:05] <zyga-solus> mvo: no, that's fine
[20:05] <zyga-solus> I read your improved patch and I was very happy about the test
[20:05] <zyga-solus> mvo: let's see if this is releasable
[20:05] <zyga-solus> man you should skip monday with a day like this
[20:13] <mvo> zyga-solus: lets get it to beta and annouce in the forum, then we can keep debugging the pi3 issue, I would like to understand that before we go to stable anyway
[20:17] <zyga-solus> mvo: agreed
[20:17] <zyga-solus> mvo: thank you for staying so late tonight
[20:17] <zyga-solus> mvo: you made all of this possible!
[20:23] <kyrofa> elopio, other, unrelated errors in https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-zesty-snappy-dev-snapcraft-daily/zesty/amd64/s/snapcraft/20171013_184638_e91b2@/log.gz
[20:24] <kyrofa> Are those real errors, or should we re-run?
[20:26] <cachio_> mvo, also pawel has a pi3
[20:27] <cachio_> he has 2 in fact
[20:29] <mup> PR snapd#4045 opened: snap-confine: cleanup broken nvidia (2.28) <Created by mvo5> <https://github.com/snapcore/snapd/pull/4045>
[20:29] <mvo> zyga-solus: -^ because the cherry-pick does not apply cleanly
[20:34] <ryebot> is it possible to `snap download` for a different arch, somehow? Alternatively, what url is the snap client hitting to perform the download? Does it need auth?
[20:34] <zyga-ubuntu> mvo: ack
[20:35] <zyga-ubuntu> the spread test conflicted?
[20:35] <zyga-ubuntu> mvo: reviwed
[20:36] <mvo> zyga-ubuntu: yeah, its a new test, I forgot about that
[20:48] <zyga-ubuntu> mvo: hmm
[20:48] <zyga-ubuntu> core                     16-2.28.4+git434.8c9d7e9  3201  canonical     core,disabled
[20:49] <zyga-ubuntu> how did this happen?
[20:49] <zyga-ubuntu> aha
[20:50] <zyga-ubuntu> I think my fault http://pastebin.ubuntu.com/25734299/
[20:50] <zyga-ubuntu> hmm
[20:50] <zyga-ubuntu> then again, not sure
[20:50] <elopio> kyrofa: :( I had not seen those container builds errors before...
[20:55] <mvo> zyga-ubuntu: hm, is it hanging there?
[20:55] <mvo> zyga-ubuntu: it might be the new security re-generation?
[20:56] <zyga-ubuntu> mvo: root     21944  0.0  0.0   8864  1092 ?        D    12:31   0:00 /snap/core/current/usr/lib/snapd/snap-confine snap.core.hook.configure /usr/lib/snapd/snap-ex
[20:56] <zyga-ubuntu> no, it's in D state
[20:56] <zyga-ubuntu> unkillable
[20:56] <zyga-ubuntu> I tried to gdb attach to it, gdb is also borked
[20:57] <zyga-ubuntu> nothing in syslog
[20:59] <zyga-ubuntu> I'll reboot :/
[21:00] <gouchi> Hi
[21:01] <gouchi> we want to be able to browse / and even if we have home, removable-media and udisks2 interfaces it seems we can't browse it
[21:01] <gouchi> https://github.com/libretro/retroarch-snap/blob/master/snapcraft.yaml#L12
[21:02] <gouchi> is there other interface we need to enable ?
[21:04] <kalikiana> elopio: would you mind giving https://github.com/snapcore/snapcraft/pull/1587#issuecomment-336521348 another look? I got a +1 from Kyle
[21:04] <mup> PR snapcraft#1587: lifecycle: clean after deleting container <Created by kalikiana> <https://github.com/snapcore/snapcraft/pull/1587>
[21:04] <elopio> Sure
[21:05] <zyga-ubuntu> mvo: I had to hard-reboot, that process prevented systemd to reboot
[21:06] <zyga-ubuntu> mvo: ok, I'm tracking stable now
[21:06] <zyga-ubuntu> I'll refresh as soon as you push the new update
[21:06] <gouchi> we will check log first
[21:08] <kalikiana> elopio: Thanks!
[21:08]  * kalikiana wrapping up the late night shift now - so much for my plan of killing overtime :-D
[21:24] <mup> PR snapd#4045 closed: snap-confine: cleanup broken nvidia (2.28) <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4045>
[21:40] <mup> PR snapd#4046 opened: release: snapd 2.28.5 <Created by mvo5> <https://github.com/snapcore/snapd/pull/4046>
[21:45]  * zyga-solus noticed that "snap refresh" perpetually refreshes atom
[21:45] <zyga-solus> which is following stable
[21:45] <zyga-solus> and always at revision 36
[21:45] <zyga-solus> but still refreshes each time anyway
[21:45] <zyga-solus> but that's for tomorrow
[21:45] <zyga-solus> o/