[03:31] <sergiusens> elopio: rollback fails for you, right?
[03:33] <rsalveti> sergiusens: fails for me as well
[03:33] <rsalveti> sergiusens: check https://bugs.launchpad.net/snappy/+bug/1474126
[03:33] <sergiusens> rsalveti: I have this fix I've been trying to test for a while now :-/
[03:34] <sergiusens> rsalveti: first a bad sdcard with fully corrupted content, now this...
[03:34] <rsalveti> right, this corruption issue seems to be quite normal
[03:34] <sergiusens> rsalveti: oh, and not sure I ever noticed, but it doesn't look like vfat
[03:34] <rsalveti> seems at the moment we call fsck for the vfat partition things break
[03:34] <sergiusens> rsalveti: HARDWA~1.YAM
[03:35] <rsalveti> sergiusens: that is vfat
[03:35] <sergiusens> yeah
[03:35] <sergiusens> rsalveti: don't we get long file names?
[03:36] <rsalveti> that's an extension
[03:36] <rsalveti> it seems fsck changed it
[03:36] <rsalveti> it is supported by the original flash
[03:36] <rsalveti> all good and so on, but at the time it runs fsck, boom
[03:37] <rsalveti> the time it worked fine I had a FSCK... file in the partition
[03:37] <rsalveti> FSCK0000.REC
[03:37] <rsalveti> when it failed I didn't get this file, but the content changed
[03:38] <rsalveti> not sure yet when we call fsck
[03:38] <sergiusens> rsalveti: fsck is part of our new init?
[03:38] <rsalveti> doesn't look like it
[03:38] <rsalveti> maybe part of the boot process, done by systemd, not sure
[03:40] <rsalveti> going to bed now, we can check tomorrow
[03:40] <sergiusens> rsalveti: ok, then you don't get to test my fix
[03:40] <sergiusens> :-P
[03:40] <sergiusens> until tomorrow
[03:40] <rsalveti> right :-)
[03:40] <sergiusens> rsalveti: smart in using the dup files btw ;-)
[03:40] <rsalveti> propose it anyway
[03:41] <rsalveti> great
[03:41] <sergiusens> rsalveti: I will, as soon as I can do some street testing
[03:41] <rsalveti> alright, later man, have a good night
[03:42] <sergiusens> good night
[07:02] <fgimenez> good morning
[07:12] <dholbach> good morning
[08:55] <ogra_> yippie !!!
[08:55] <ogra_> https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/wily/ubuntu-core-system-image/+build/32370
[08:55] <ogra_> we has arm64 images :)
[09:05] <tbr> ogra_: oh. does that mean I'll be able to try something on the dragonborad?
[09:06] <ogra_> tbr, well, no idea if it boots or how our bootloader setup on arm64 is at all
[09:06] <tbr> (yes, I'm able to bring my own hw-adaptation. Just rebuilt the kernel to get usb-networking)
[09:07] <tbr> the state of bootloaders on aargh64 is 'special'.
[09:07] <ogra_> at least all packages build and enable rolling a rootfs, not sure i can say more about these images :)
[09:07] <JamesTait> Good morning all; happy Pandemonium Day! ð
[09:08] <ogra_> uuuuuh
[09:08] <tbr> that's fine. I might give it a go, but will probably have some questions
[09:08]  * tbr wishes JamesTait a merry Setting Orange
[09:09]  * JamesTait sets his orange on the desk, to eat later.
[10:53] <Chipaca>             sourceFiles |= set([os.path.join(root, d) for d in dirs])
[10:53] <Chipaca> mterry:
[10:53] <Chipaca> oh, no mterry
[11:54] <rsalveti> morning
[11:54] <Chipaca> mo'in
[12:20] <rsalveti> ogra_: mvo: did you guys had time to check the email sergiusens sent yesterday?
[12:20] <rsalveti> about bug 1474126
[12:37] <mvo> rsalveti: not yet, sorry, let me do that now. I was absorbed in snapcraft stuff, mterry has too many good ideas :)
[12:37]  * mvo looks now
[12:37] <rsalveti> mvo: haha, no worries :-)
[12:38] <ogra_> hmm
[12:38]  * ogra_ didnt notice a mail 
[12:38] <ogra_> checking again
[12:38] <rsalveti> ogra_: it seems sergiusens forgot to include you
[12:38] <rsalveti> but was basically about this bug ^
[12:39] <rsalveti> he created https://code.launchpad.net/~sergiusens/snappy/noUselessUpdate/+merge/264665 to fix 1474125
[12:39] <rsalveti> but we can't test it easily because of bug 1474126
[12:39] <ogra_> right, i saw that
[12:40] <ogra_> so as i commented on the bug, it could well be that fatwrite messes up the filesystem
[12:40]  * ogra_ doesnt really trust it 
[12:40] <ogra_> still so new and rarely used anywhere
[12:44] <rsalveti> ogra_: yeah, looks like it
[12:44] <rsalveti> ogra_: I did the update, then decided to run fsck.vfat and it was all good
[12:44] <rsalveti> but once I rebooted, boom
[12:44] <ogra_> i wonder if it is something simple ... like fatwrite not being aboe to truncate files ...
[12:45] <ogra_> so you would have to delete them first
[12:45] <ogra_> i didnt really find many bugs though ... i guess it is really rarely used
[12:45] <rsalveti> ogra_: mvo: one other question I had, why do we keep /boot/uboot mounted all the time?
[12:45] <ogra_> we shouldnt (i dont know why, ask jodh ? )
[12:46] <mvo> rsalveti: I don't think there is a good reason
[12:46] <ogra_> also, why dont we sync mount it ?
[12:46] <mvo> I think we do that, no?
[12:46]  * ogra_ doesnt see a sync option 
[12:46] <mvo> the sync mount I mean
[12:46] <rsalveti> /dev/mmcblk0p1 on /boot/uboot type vfat (rw,relatime,sync,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
[12:46] <rsalveti> we do use the sync option
[12:46] <ogra_> hmm
[12:46] <mvo> ogra_: outdated initrd ;) ?
[12:46] <rsalveti> let me remove the fatwrite piece and check again
[12:47] <ogra_> (RaspberryPi2)ubuntu@localhost:~$ mount|grep boot
[12:47] <ogra_> /dev/mmcblk0p1 on /boot/uboot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
[12:47] <ogra_> weird
[12:47] <rsalveti> it seems that makes fsck.vfat unhappy and then it changes the fs
[12:47] <ogra_> shouldnt be to outdated
[12:47] <rsalveti> I was on image 4 (bbb)
[12:48] <ogra_> (15.04 stable image 3 though)
[12:48] <ogra_> but i thought that should have sync set
[12:50] <mvo> ogra_: yeah, we set to sync a good while ago, in the 15.04 timeframe
[12:50] <mvo> odd
[12:50] <ogra_> right
[12:50] <ogra_> i use the initrd from cdimage with removed modules
[12:50] <ogra_> nothing else changed
[12:51] <ogra_> rsalveti, are our tools able to remount /boot rw before operating on it ?
[12:52] <ogra_> just switching it to ro might be a bit dangerous if they dont :)
[12:53] <rsalveti> /boot/uboot is always rw
[12:54] <ogra_> yes, it shouldnt
[12:54] <ogra_> but if we make it ro, can the tools deal with it ?
[12:55] <ogra_> hmm
[12:55] <ogra_>                 else
[12:55] <ogra_>                         bootdir="$ubootdir"
[12:55] <ogra_>                         echo "$boot_partition $bootdir auto sync 0 2" >> "$fstab"
[12:55] <ogra_>                 f
[12:55] <rsalveti> I'd expect it to only be mounted when needed, since it's a vfat partition
[12:55] <ogra_> so that is what it is supposed to write to fstab
[12:55] <rsalveti> yup
[12:55] <ogra_> (RaspberryPi2)ubuntu@localhost:~$ grep uboot /etc/fstab
[12:55] <ogra_> /dev/mmcblk0p1 /boot/uboot auto defaults 0 0
[12:55] <ogra_> and that is what my fstab has
[12:56] <rsalveti> /dev/mmcblk0p1 /boot/uboot auto sync 0 2
[12:56] <rsalveti> bbb, image 3
[12:56] <ogra_> rpi image 3
[12:56] <ogra_> how weird
[12:59] <ogra_> i wonder if systemd re-writes the fstab entry somehow
[12:59] <ogra_> i dont see any explanation in the code why it would be "auto default"
[13:21] <rsalveti> mvo: oh, and https://code.launchpad.net/~sergiusens/snappy/noUselessUpdate/+merge/264665 reminds me that we need to prioritize the work to update the upgrader before doing the upgrade
[13:21] <tedg> Morning folks
[13:21] <rsalveti> not super urgent, but would be good for us to put some hands on it later this month
[13:21] <tedg> Chipaca, Were you able to get a libxcbcommon patch working?
[13:21] <Chipaca> tedg: i'm not sure, right now
[13:22] <Chipaca> um
[13:22] <Chipaca> read: i have a working patch
[13:22] <Chipaca> but i'm not sure the patch works
[13:22] <Chipaca> is that clearer? i'm not sure :)
[13:22] <tedg> Hmm, how do you know it's a working patch then? :-)
[13:23] <mvo> rsalveti: indeed
[13:24] <mvo> tedg: it builds, so we shipped it.
[13:24] <tedg> \o/
[13:26] <Chipaca> tedg: and the not sure is because http://pastebin.ubuntu.com/11877692/
[13:27] <sergiusens> morning
[13:28] <tedg> Chipaca, There seems to be reads from /home/ubuntu/thing/lib and /apps/qmlapp.sideload/0/usr/lib, seems there should only be one of those?
[13:28] <rsalveti> sergiusens: morning :-)
[13:29] <Chipaca> tedg: it is one, there's a symlink but i'll unify so it melts minds less
[13:30] <tedg> If you're running confined, symlinks don't help.
[13:30] <Chipaca> not running confined
[13:31] <rsalveti> sergiusens: ogra_: mvo: yeah, if I remove the fatwrite command it all works fine
[13:32] <mvo> hey sergiusens, good morning
[13:32] <Chipaca> tedg: http://pastebin.ubuntu.com/11877734/
[13:35] <mvo> Chipaca: hm, that looks like progress compared to yesterday, no?
[13:35] <Chipaca> tedg: oh, wait, that one is using the wrong socket
[13:35] <tedg> Chipaca, Ah, so to the next step :-)  Looks like it can't find the Mir socket. Probably not running as user 1000
[13:35] <Chipaca> tedg: with the right socket: http://pastebin.ubuntu.com/11877746/
[13:35] <tedg> Yeah, need to set MIR_SOCKET probably.
[13:35] <ogra_> rsalveti, well, how do you switch ? by manually editing the file ?
[13:35] <Chipaca> yeh, i'd commented it out, because with that it finds it but still can't connect (?)
[13:36] <Chipaca> but it gets further ahead with it, so
[13:36] <Chipaca> permission denied
[13:36] <Chipaca> dammit
[13:36] <Chipaca> http://pastebin.ubuntu.com/11877751/
[13:36] <Chipaca> ^ with the right perms, and the right socket
[13:37] <tedg> What Mir server are we using? Demo Server? System compositor?
[13:37] <Chipaca> demo server
[13:37] <Chipaca> which is what is in the mir package
[13:38] <tedg> K, I don't think that it requires notification that we're going to connect. I think it stubs out the auth parts.
[13:38] <Chipaca> tedg: the permission denied was on the socket itself
[13:38] <Chipaca> tedg: but i don't know if you're talking about that :)
[13:39] <tedg> Kinda both :-)
[13:39] <tedg> Mir also does it's own auth depending on the server.
[13:39] <Chipaca> ah, ok
[13:39] <tedg> There are some MIR debugging environment variables, let me try to find them.
[13:40] <Chipaca> the mir wrapper does take care of the socket and its permissions, but i'm running it directly myself to have the console output
[13:40] <tedg> Well, you can get the console output from systemd
[13:42] <Chipaca> i know i should be able to, but couldn't find it in journalctl
[13:42] <tedg> Oh, my Mir branch was only 200K objects out of date...
[13:43] <ogra_> rsalveti, so i'd really like to get rid of using fatwrite, i wonder if we could ship a uboot.env file wih a binary environment instead ... only containing the a/b variable, that way we can just run saveenv to switch it (if anything reads the snappy_stamp.txt file from userspace we need to handle that separately though)
[13:44] <tedg> Chipaca, I think this is the useful variable: MIR_CLIENT_RPC_REPORT=log
[13:44] <Chipaca> tedg: in the environ of the server, client, both?
[13:45] <tedg> Chipaca, Client, though sever can also be used if we're confused still :-)
[13:45] <tedg> Chipaca, Here is what I wrote down: MIR_SERVER_{DISPLAY_REPORT,COMPOSITOR_REPORT,CONNECTOR_REPORT,MSG_PROCESSOR_REPORT,SESSION_MEDIATOR_REPORT}=log
[13:45] <mvo> sergiusens: thanks for your branch, I'm trying to reproduce this now
[13:45] <Chipaca> oh yeah baby
[13:46] <Chipaca> tedg: http://pastebin.ubuntu.com/11877784/
[13:46] <mvo> rsalveti: is there anything unusual about your sd card? particularly big maybe?
[13:47] <tedg> Chipaca, Huh, perhaps look at the server?
[13:47] <Chipaca> tedg: that was on the server, fwiw
[13:47] <tedg> Otherwise we're gonna need #ubuntu-mir's help
[13:47] <tedg> Oh, try on the client.
[13:48] <Chipaca> tedg: i get no extra input from the client with those, nor with MIR_CLIENT_ of same
[13:53] <Chipaca> tedg: i've got to run, will read when i return
[13:53] <tedg> Chipaca, K, I'll ping some central time folks when they log in as well.
[13:53] <Chipaca> ta
[13:54] <Chipaca> and i'll get the different bits and bobs up when i return, also, so others can reproduce
[13:55] <sergiusens> mvo: np
[13:59] <ogra_> pitti, could anything in systemd touch/modify fstab during boot ?
[14:00]  * ogra_ really doesnt get how the /boot/uboot entry can end up differently in fstab than what was echoed into it 
[14:00] <pitti> ogra_: read for sure; changing, I'm not aware of anything
[14:01] <ogra_> i have this code ... echo "$boot_partition $bootdir auto sync 0 2" >> "$fstab"
[14:01] <ogra_> and end up with:
[14:01] <ogra_> (RaspberryPi2)ubuntu@localhost:~$ grep boot /etc/fstab
[14:01] <ogra_> /dev/mmcblk0p1 /boot/uboot auto defaults 0 0
[14:02] <ogra_> i dont get how that can happen
[14:02] <pitti> so that's not system-image writing fstab from a template and writable-paths or so?
[14:02] <pitti> and is /etc/fstab writable in the first place?
[14:02] <ogra_> no, it is the initrd script checking for uboot and then echoing this line into fstab
[14:03] <pitti> so /etc/fstab is a symlink to /etc/writable/fstab or so?
[14:03] <ogra_> to my knowledge thats the only place we handle /boot/uboot
[14:03] <ogra_> fstab is a bindmount
[14:03] <ogra_> into /run/image.fstab
[14:04] <pitti> so perhaps your echo happens before the bind mount?
[14:04] <ogra_> http://paste.ubuntu.com/11877845/
[14:04] <ogra_> thats the code snippet
[14:04] <ogra_> (it happens afterwards)
[14:06] <ogra_> i dont see anything wrong with that code
[14:06] <pitti> what does handle_writable_paths() do?
[14:06] <pitti> it doesn't happen to assign fstab= to something else by chance?
[14:06] <ogra_> it loops over the writable paths file and echos the lines as proper fstab lines
[14:06] <ogra_> oh
[14:06]  * ogra_ checks for uboot in writable-paths
[14:07] <ogra_> (RaspberryPi2)ubuntu@localhost:~$ grep boot /etc/system-image/writable-paths
[14:07] <ogra_> (RaspberryPi2)ubuntu@localhost:~$
[14:07] <ogra_> nope
[14:09] <rsalveti> mvo: nothing unusual, no
[14:09] <rsalveti> ogra_: we got logic from our userspace to check for that stamp file
[14:09] <rsalveti> to then change the boot mode because it was successful
[14:09] <rsalveti> since it boots with "try" first
[14:10] <ogra_> rsalveti, right, so the uboot.env would have to add it to cmdline and the initrd would have to echo that into the stamp file
[14:10] <ogra_> so that uboot.env still stays the master and writin only happens from the kernel vfat driver
[14:10] <ogra_> and not from uboot
[14:11] <rsalveti> right, I guess that's a way
[14:11] <rsalveti> env + kernel cmdline + logic in initrd
[14:11] <ogra_> yeah
[14:11] <ogra_> just requires that all uboots we support can handle such an env
[14:12] <ogra_> at least for upgradeable devices
[14:12] <rsalveti> ogra_: are we shipping uboot with bbb?
[14:12] <rsalveti> or are we booting the uboot that is already flashed in the board?
[14:12] <ogra_> we are shipping a uboot.img
[14:12] <ogra_> (and SPL too i think)
[14:12] <rsalveti> that uboot is super old
[14:13] <ogra_> hmm
[14:13] <ogra_> i wonder if you guys actually boot the EMMc one
[14:13] <rsalveti> easy to check
[14:14] <rsalveti> ogra_: yeah, we're loading the spl and uboot that is already flashed in the emmc
[14:14] <mvo> ogra_: I like your idea
[14:14] <ogra_> aha !
[14:15] <rsalveti> because to boot from the sd card completely you have to hold the button while turning on the board
[14:15] <ogra_> rsalveti, so likely a different fatwrite then
[14:15] <rsalveti> U-Boot SPL 2014.10-dirty (Dec 18 2014 - 22:07:26)
[14:15] <ogra_> or wipe the MMC
[14:15] <rsalveti> U-Boot 2014.10-dirty (Dec 18 2014 - 22:07:26)
[14:15] <rsalveti> the one we ship
[14:15] <rsalveti> right
[14:15] <rsalveti> U-Boot SPL 2014.04-00015-gb4422bd (Apr 22 2014 - 13:24:29)
[14:15] <rsalveti> U-Boot 2014.04-00015-gb4422bd (Apr 22 2014 - 13:24:29)
[14:15] <rsalveti> the one we boot
[14:16] <ogra_> well, less dirty at least :P
[14:16] <ogra_> damn ...
[14:16] <ogra_> didnt zyga's patch land that was supposed to enforce that ?
[14:16] <rsalveti> ogra_: nops
[14:17] <rsalveti> since the only way to force our bootloader is by not having the original one or by pressing the button
[14:17] <ogra_> https://code.launchpad.net/~zyga/snappy-hub/fix-1464275/+merge/261833
[14:17] <mvo> hm, I did a fresh flash of my sd card some minutes ago and I have: "U-Boot 2014.10-dirty (Dec 18 2014 - 22:07:26)" in my boot message - but I did press the button in the past, will it remember that?
[14:17] <rsalveti> maybe you wiped out your emmc
[14:17] <ogra_> well, it is supposed to remember it
[14:18] <ogra_> but i have no clue how that works
[14:18] <mvo> both is likely
[14:18] <rsalveti> don't think it remembers it
[14:18] <mvo> ok
[14:18] <rsalveti> at the moment you unplug power and plug it again, it should try emmc first again
[14:18] <ogra_> so looking at zyga'S patch there ... it would still use the MMC uboot
[14:18] <ogra_> but enforce loading the SD
[14:19] <ogra_> so that wouldnt actualyl help
[14:19] <rsalveti> no, when you run that you're already in uboot
[14:19] <ogra_> right :/
[14:19]  * ogra_ sees no way around this except wiping the MMC on first boto or some such
[14:20] <ogra_> *boot
[14:20] <ogra_> ... not great :/
[14:20] <rsalveti> ogra_: or do your env dance suggestion you said
[14:21] <rsalveti> I'm checking now if our fatwrite is better
[14:21] <ogra_> well, will that help ? are we sure the bootx command behaves the same ... etc etc ?
[14:21] <ogra_> its not only fatwrite
[14:22] <ogra_> i ean, we can indeed use my suggestion to take fatwrite out of the equation ... but there are 6 months between the two uboots ... i would be surprised if we wouldnt hit other issues
[14:22] <ogra_> *mean
[14:24] <ogra_> (8 monts actually ... temporary math weakness :P )
[14:25] <mvo> sergiusens, rsalveti: hrm, so I tried to test 15.04a3 -> 15.04a4 -> 15.04a3 but when I rollback I get the following error http://paste.ubuntu.com/11877942/ on boot. this is new, right?
[14:25] <mvo> ogra_: -^
[14:26] <ogra_> mvo, no, thats exactly the issue we are diuscussing
[14:26] <ogra_> *discussing
[14:26] <mvo> ogra_: thats the corruption?
[14:26] <ogra_> fatwrite updates the file and trashes the vfat
[14:26] <mvo> ok, cool
[14:26] <ogra_> if the fatwrite from the older uboot is used
[14:26] <mvo> so  â¦ I get this with "U-Boot 2014.10-dirty (Dec 18 2014 - 22:07:26)
[14:26] <mvo> "
[14:27] <rsalveti> yeah, so that shows that our uboot is busted as well
[14:27] <mvo> I never see a different U-Boot boot string
[14:27] <ogra_> yeah
[14:27] <mvo> *weeeh*
[14:27] <rsalveti> and another interesting thing, my ethernet doesn't work with our uboot
[14:27] <rsalveti> it only works with the uboot provided by the board
[14:28] <rsalveti> got bbb rev c
[14:28] <ogra_> wow. thats weird
[14:29] <rsalveti> mvo: that's the issue that elopio had for quite a while
[14:29] <rsalveti> but that we were never able to find out why
[14:29] <sergiusens> mvo: that's the second bug
[14:29] <sergiusens> mvo: to test the fix for the first bug, resync the kernel and initrd from /boot ;-)
[14:30] <elopio> good morning.
[14:31] <mvo> sergiusens: :) just commented, I will look into adding a test now, but a cup of tea first :)
[14:31] <elopio> fgimenez: looking at your failures, for some reason systemctl status docker_docker-daemon_*.service prints nothing for you on the stdout.
[14:31] <elopio> I'll check what's going on.
[14:31] <elopio> fgimenez: lets skip our meeting today, unless you want to discuss about something.
[14:31] <mvo> sergiusens: honestly this fatwrite thing is really alarming to me shows that we really need some bbb auto testing
[14:32] <fgimenez> elopio, ok np, i'll check the output of systemctl
[14:34] <rsalveti> mvo: yeah
[14:34] <rsalveti> on the good side plars is getting that going soon :-)
[14:34] <rsalveti> but yeah, it's super bad
[14:36] <sergiusens> mvo: to be fair, this is an untested code path
[14:36] <ogra_> sergiusens, rolling back and upgradin is untested ?
[14:36] <rsalveti> guess the main problem is when we're changing kernel versions
[14:37] <elopio> fgimenez: I see, the *.service seems to work on 15.04 but not on rolling.
[14:37] <rsalveti> when updating/rolling back with the same kernel version I think that's mostly fine
[14:37] <rsalveti> but might still get broken I believe, and why elopio got it from time to time
[14:37] <sergiusens> ogra_: for new kernels, yes
[14:37] <ogra_> ah, it is our first new kernel ?
[14:37] <ogra_> k
[14:38] <fgimenez> elopio, yes, the output is from rolling/edge 107
[14:39] <sergiusens> ogra_: I don't think so, but channel changing for testing this are all flawed IMO (unless it is indeed a channel change test)
[14:39] <fgimenez> elopio, btw i have a branch for selecting the release, branch and revision for the tests almost done, will ping you when pushed
[14:40] <elopio> fgimenez: nice.
[14:40] <mvo> rsalveti: yeah, thats the good part, we have the tests and a plan and all that :)
[14:41] <mvo> and thats probably what the erle guys also experienced from time to time
[14:41] <ogra_> yeah
[14:42] <elopio> pitti: do you know why this works on systemd 219 but not on 222?
[14:42] <elopio> systemctl status docker_docker-daemon_*.service
[14:48] <pitti> elopio: not off the top of my head; what does "not work" mean?
[14:48] <elopio> pitti: on 219 it returs the status. On 222 it returns nothing.
[14:52] <pitti> elopio: indeed; mind filing a bug about this, so that we can track this? (busy with something else right now)
[14:52] <elopio> pitti: sure thing.
[14:52] <pitti> elopio: thanks; I'll bisect it
[14:53] <sergiusens> rsalveti: ogra_ if we revert the fix in uenv.txt, will we get something workable again (wrt fatwrite)?
[14:53] <ogra_> sergiusens, no
[14:53] <ogra_> that fix wont change anything
[14:54] <sergiusens> ogra_: why did this only start now?
[14:54] <ogra_> sergiusens, dunno, did we do this kind of test before ? (multiple updates and rollbacks)
[14:55] <sergiusens> ogra_: yes we have
[14:56] <sergiusens> ogra_: just the previous release was a 3 to 4 jump between releases
[14:56] <ogra_> probably it only happenms if the fat content changes between two fatwrites
[14:56] <ogra_> which now happens due to the kernel upgrade
[14:58] <ogra_> (i'm really just guessing here ... fact is that it doesnt break if fatwrite isnt used at all)
[14:59] <rsalveti> right, I believe it's because of the newer kernel
[15:00] <ogra_> do we know if the uboot fat driver can handle long filenames  ?
[15:00] <elopio> pitti: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1474426
[15:00] <ogra_> we also started using the longer filenames for the kernel
[15:00] <elopio> fgimenez: for now, I'll modify GetCurrentVersion to get a packageName.
[15:01] <ogra_> (thanks to the version)
[15:03] <pitti> elopio: thanks
[15:16] <plars> ogra_: rsalveti: sorry, just now catching up after I got off a call - yes this is what I was complaining about a long time ago when trying to get BBB to boot properly - that we just accidentally boot snappy on it because we have a working uboot on emmc, and that uboot could easily break on the sd card if you don't force it to boot from there
[15:17] <ogra_> plars, yeah, but there is no way for us to enforce SD ...
[15:17] <ogra_> unless we wipe it
[15:17] <ogra_> well, wipe the MMC
[15:17] <plars> ogra_: there is one other way
[15:18] <mvo> ohhhh?
[15:18] <ogra_> how ?
[15:18] <plars> ogra_: holding the s2 button down, or jumpering lcd_data02 I think, to gnd will do it
[15:18] <ogra_> right
[15:18] <plars> it forces booting from the sd (including bootloader)
[15:18] <ogra_> or solder a wire ...
[15:18] <plars> ogra_: well, a jumper wire is so much easier :)
[15:18] <ogra_> not realy a softwareish solution :)
[15:19] <ogra_> and we cant really destroy peoples MMC installs
[15:19] <plars> ogra_: so my current plans for instrumenting bbb include this, and use s2 along with a stable image on emmc to flash the sd, then force booting onto the sd
[15:19] <plars> ogra_: but if/when we get the recovery stuff, then we won't want to do that of course - and it becomes really critical that we don't break uboot
[15:20] <ogra_> yeah :(
[15:20] <plars> ogra_: but that makes me wonder, do we really need to change the bootloader that often? except maybe at the beginning?
[15:20] <plars> ogra_: once it stabilizes shouldn't we be able to mostly leave it alone?
[15:21] <ogra_> well, its not our bootloader that has issues ... it is the case where we boot the onboard one and use fatwrite from it
[15:21] <plars> if you are updating it only ever 12-16 months, it's much easier to be super careful and make sure we don't break it before landing the change
[15:21] <plars> ah
[15:22] <plars> ogra_: why is it calling fatwrite from the onboard one?
[15:22] <ogra_> the uboot scripts execute it to set the a or b partition in a file on the boot partition
[15:22] <plars> oh, right
[15:23] <plars> so this is just the nature of bbb, by default it tries to load the bootloader from emmc, unless you press the s2 button after poweron
[15:24] <beuno> FWIW, in both my bbb
[15:24] <beuno> I didn't need to press anything
[15:24] <beuno> it boots from the SD if it's there
[15:24] <plars> beuno: yeah, that's accidental and the source of the problem :)
[15:25] <beuno> plars, this is why I don't do hardware  :p
[15:25] <plars> beuno: I've been complaining about this for a while now, what's happening is that it's loading the bootloader off of your emmc
[15:26] <plars> but the bbb default image is set up to check if there's something it can load from the sd
[15:26] <elopio> pitti: other question, is there a way to wait for a service to be loaded and running?
[15:27] <pitti> elopio: another unit should just have After=foo.service
[15:27] <elopio> pitti: but not inside a unit. Like I install a package, and I want a command to wait until it's up before using it.
[15:28] <pitti> elopio: other than polling "systemctl is-active foo.service" I don't know; usually invoke-rc.d waits until it's up already
[15:28] <pitti> elopio: asking on #systemd (freenode) might be useful, perhaps I'm missing some kind of "wait-active"
[15:28] <elopio> pitti: ack, thanks.
[15:29] <elopio> is-active is already an improvement to what I have :)
[15:29] <pitti> elopio: other than that a while/is-active/sleep loop would work for now
[15:30] <pitti> elopio: that smells a bit weird -- you  have a package whose postinst auto-starts a service, but it's not up when apt-get finishes?
[15:30] <plars> ogra_: if there's a known fix for this uboot bug, I'm sure rcn-ee would be open to adding it in his images, then we could point people who are running the default image on emmc to update to that?
[15:30] <pitti> elopio: packages are expected to do that; i. e. that sounds like a bug in that .service, or in the package
[15:30] <elopio> pitti: well, on snaps we don't have post-inst.
[15:30] <ogra_> plars, well, the issue is to use fatwrite at all :P
[15:31] <plars> oh
[15:31] <ogra_> plars, i'm looking at a way to not use it at all
[15:31] <elopio> I'm not sure if we can make something to wait on install.
[15:31] <plars> ok
[15:31] <pitti> elopio: ah, ok
[15:31] <pitti> elopio: but what would start the shipped units then?
[15:31] <pitti> elopio: whatever that ^ is, if it does "systemctl start", that's also synchronous, i. e. waits until it's started
[15:32] <pitti> elopio: unless you call it with --no-block
[15:34] <elopio> pitti: hum, so maybe that works. I'm just following this comment that says:
[15:34] <elopio> "FIXME: sucks, needed because "systemctl start" does not wait until the service is really started."
[15:34] <elopio> maybe that's not true anymore.
[15:34] <pitti> that has never been true
[15:34] <pitti> it might be true that the .sevice's Exec* commands do not wait
[15:35] <pitti> i. e. do whatever you need to do to wait until the started service is up
[15:35] <pitti> but then the polling loop doesn't help you either
[15:35] <pitti> or it specifies a wrong Type= (simple instead of forking, etc.)
[15:36] <elopio> pitti: I see. Oh, but here's another case where we do need to wait:
[15:36] <elopio> if ssh starts before the service starts, then the test might run before the service is running.
[15:37] <pitti> elopio: ah, this might be something to fix in autopkgtest
[15:37] <pitti> elopio: i. e. not start before systemctl list-jobs becomes empty
[15:38] <elopio> pitti: oh, I like that.
[15:38] <plars> Is there a good description somewhere of pxe booting snappy?
[15:47] <elopio> I see that the service has no type, which defaults to simple, which should wait until the service is up. So if it doesn't, is a package bug.
[15:48] <elopio> fgimenez: are you ok if I remove all this waits, and see how it goes? If it fails, we report bugs to fix the packages.
[15:50] <fgimenez> elopio, yes, much better. If we need to wait for something we could use goroutines and channels too
[15:50] <elopio> fgimenez: I was trying to use a ticker to poll every 1 second, and made a mess instead :)
[15:51] <elopio> I need to get better at parallelization.
[15:52] <fgimenez> elopio, me too :) we could encapsulate all this logic somehow so that we can reuse, we'll need it again for sure
[15:53] <elopio> fgimenez: btw there's this tiny branch that needs review: https://code.launchpad.net/~elopio/snappy/test_the_tests_tests/+merge/264170
[15:54] <fgimenez> elopio, sure
[15:56] <rsalveti> lool: why are we not using the u-boot from the archive?
[15:56] <rsalveti> it seems we're using the one from http://people.canonical.com/~lool/snappy-bbb/
[15:56] <ogra_> lool, did your patches land upstream (so that they would be in the wily archive uboot) ?
[15:57] <rsalveti> sergiusens: how can I retrieve the content of the beaglebone oem snap from the store?
[16:03] <elopio> ok, xkcd fails without the sleep. So I suppose there's a bug on the networking-service template.
[16:07] <sergiusens> rsalveti: get it from te link
[16:07] <sergiusens> rsalveti: https://search.apps.ubuntu.com/api/v1/package/beagleblack the anondownload one
[16:08] <sergiusens> rsalveti: or just lp:snappy-hub and look for the snappy-systems branch (I would create a series, but I think asac is still owning the snappy-hub project).
[16:08] <tedg> Uhg, so I installed the Mir demo server before setting up SSH.
[16:09] <tedg> Is there a way to figure out the IP address KVM gave something?
[16:09] <rsalveti> sergiusens: cool, thanks
[16:09] <sergiusens> tedg: kvm only does lo by default, so you would need to redir anyways and not care about the ip
[16:09] <tedg> I used virt-manager so it sets up the interface.
[16:09] <sergiusens> tedg: I do alias kvm_snappy='kvm -m 1500 -redir :8022::22 -redir :8080::8080 -redir :4200::4200'
[16:10] <sergiusens> tedg: oh, then I don't know :-)
[16:30] <jobot> Hello, I have snappy on a beaglebone black using the image from here https://developer.ubuntu.com/en/snappy/start/#try-beaglebone. I have installed owncloud and would like it to use storage from a usb drive, but I cannot seem to mount the usb to have it writeable. Any guidance appreciated. Thanks
[16:56] <elopio> jobot: I could mount my usb drive on bbb.
[16:57] <elopio> and then I can use chown to give the ubuntu user write permissions on the mount.
[16:57] <elopio> what's the problem you are getting
[16:57] <elopio> ?
[17:09] <jobot> Thanks. It's when I set owncloud storage to /mountlocation , it says that it's not writable
[17:10] <jobot> I guess I didn't chown it. Are there any other mount settings required?
[17:14] <elopio> jobot: I haven't used owncloud yet, so I don't know. But I can write to my usb.
[17:19] <jobot> ok I will give it a go. Thanks!
[19:25]  * rsalveti is still trying to clone git://git.denx.de/u-boot.git
[19:28] <ogra_> did it grow so much the recent years ?
[19:28] <rsalveti> ogra_: no, just super slow to clone
[19:28] <rsalveti> 30kb/s
[19:28] <ogra_> nice, feels like home :P
[19:28] <rsalveti> hahah
[19:29] <plars> someone picked up the other phone and messed up the modem? :)
[19:54] <plars> elopio: is https://code.launchpad.net/~snappy-dev/snappy/selftest still the right place to look for snappy selftests?
[19:55] <plars> elopio: or is there newer test code somewhere else?
[20:01] <sergiusens> plars: it's all in trunk now
[20:06] <plars> elopio: sergiusens: awesome, thanks, looking through the readme now... it looks like it may prompt me for a ssh password? does it try a default or can you give it one in advance to try?
[20:18] <sergiusens> plars: don't think so, but I can't attest to that
[20:18] <sergiusens> :-)
[20:18] <plars> sergiusens: hmm, I can't even get it to run at the moment, but it's probably some go issue
[20:18] <plars> it doesn't seem to work like the previous adt stuff did
[20:19] <sergiusens> plars: build on x86 and run on arm? maybe missing some cross compile flags there ;)
[20:19] <sergiusens> elopio should be able to help there
[20:19] <plars> sergiusens: yeah, I'll talk to him when he's back around
[20:32] <rsalveti> sergiusens: what removes snappy-stamp.txt?
[20:46] <sergiusens> rsalveti: nothing
[20:47] <sergiusens> rsalveti: is it being removed?
[20:47] <rsalveti> sergiusens: that's the file created by u-boot, just wanted to know about the logic that handles it
[20:47] <rsalveti> in the userspace side
[20:48] <rsalveti> doc says it's undone by /lib/systemd/system/ubuntu-core-snappy.service
[20:48] <rsalveti> which doesn't exist
[20:49] <rsalveti> so someone removes /boot/uboot/snappy-stamp.txt
[20:49] <sergiusens> rsalveti: it's not called ubuntu-core-snappy.service, that's one problem I guess
[20:50] <rsalveti> right :-)
[20:51] <sergiusens> rsalveti: there seems to be a job missing ...
[20:51] <rsalveti> the file gets removed
[20:51] <sergiusens> rsalveti: it's ubuntu-snappy.boot-ok.service
[20:51] <sergiusens> rsalveti: oh, nvm, it's a task most likely and systemctl doesn't list those by defualt
[20:52] <sergiusens> rsalveti: sudo journalctl -u ubuntu-snappy.boot-ok.service
[21:03] <rsalveti> sergiusens: right, so it's snappy itself that handles it
[21:03] <rsalveti> ExecStart=/usr/bin/snappy booted
[21:04] <sergiusens> rsalveti: oh, yeah, sorry, forgot that was only obvious to 3 people :-P
[21:04] <sergiusens> rsalveti: I fixed a bug there not long ago to avoid needless writes if you may recall
[21:04] <rsalveti> sergiusens: yeah, I wonder if the problem is because the file size is 0 for the stamp file
[21:05] <rsalveti> trying to write some data on it now
[21:05] <sergiusens> rsalveti: it should never be zero
[21:05] <rsalveti> not a lot of changes from upstream at the fat code
[21:05] <sergiusens> rsalveti: oh, the stamp file, not snappy-system.txt
[21:05] <sergiusens> got it
[21:05] <rsalveti> fatwrite mmc ${mmcdev}:${mmcpart} 0x0 ${snappy_stamp} 0
[21:05] <rsalveti> yeah, stamp
[21:06] <sergiusens> rsalveti: snappy booted does not erase that iirc, it's the bootloader; this way we can rollback if the kernel is corrupted
[21:06] <sergiusens> as an example
[21:07] <sergiusens> rsalveti: nvm, http://bazaar.launchpad.net/~snappy-dev/snappy/snappy/view/head:/partition/bootloader_uboot.go#L203
[21:07] <sergiusens> rsalveti: I forgot how this all works! lool you around?
[21:07] <rsalveti> yeah, saw that
[21:13] <rsalveti> sergiusens: didn't break this time with a file != 0
[21:13] <rsalveti> let me build a new image with your fix so I can also test the rollback
[21:22] <sergiusens> rsalveti: interesting, so if uEnv.txt fatwrote this with something > 0 it would be fine?
[21:23] <rsalveti> sergiusens: that is what I'm trying to confirm
[21:24] <rsalveti> sergiusens: elopio: give this a try, change /boot/uboot/snappy-system.txt
[21:24] <rsalveti> with image 3
[21:25] <rsalveti> change "fatwrite mmc ${mmcdev}:${mmcpart} 0x0 ${snappy_stamp} 0" to "fatwrite mmc ${mmcdev}:${mmcpart} ${loadaddr} ${snappy_stamp} 10"
[21:25] <rsalveti> then do the update
[21:27] <sergiusens> rsalveti: k
[21:28] <sergiusens> rsalveti: will take a shortcut and format system-b
[21:32]  * sergiusens waits
[21:38] <rsalveti> yeah, tried twice and wasn't able to get the same corruption
[21:46] <rsalveti> hm, ubuntu-device-flash is the one creating snappy-system.txt
[22:23] <sergiusens> rsalveti: yes it is
[22:24] <sergiusens> rsalveti: it's not updatable
[22:30] <rsalveti> sergiusens: yeah, guess we can fix this all when we split the updater
[22:31] <rsalveti> migrating 124 to alpha
[22:31] <rsalveti> argh, there is one importer running
[22:47] <sergiusens> rsalveti:  I can add a fix for u-d-f to at least have new images work correctly
[22:48] <rsalveti> sergiusens: I'm waiting the next alpha to show up and will do a bit more testing
[22:48] <rsalveti> but got a local change already for that
[22:48] <rsalveti> importer takes forever lately
[22:56] <rsalveti> wget https://public.apps.ubuntu.com/anon/download/canonical/beagleblack.canonical/beagleblack.canonical_1.8_all.snap
[22:56] <rsalveti> HTTP request sent, awaiting response... 500 INTERNAL SERVER ERROR
[22:56] <rsalveti> beuno: any issue with the store?
[22:58] <rsalveti> yeah, unable to create a beagle image
[22:58] <rsalveti> sigh
[23:18] <sergiusens> rsalveti: it downloads fine here
[23:18] <rsalveti> sergiusens: yeah, it's working again
[23:22] <beuno> rsalveti, looking at outages, for future reference, noodles is usually up and running at this hour
[23:22] <beuno> or just ask in #u1-internal
[23:22] <rsalveti> alright, thanks
[23:22] <rsalveti> was giving this error for about 10 minutes for me
[23:24] <beuno> rsalveti, looks like a storm in the internal cloud
[23:25] <beuno> being managed by IS
[23:25] <rsalveti> alright :-)
[23:30] <rsalveti> sergiusens: elopio: alpha 5 is now available
[23:33] <beuno> (winds are picking up, might get a bit bumpier before it gets better)