[03:31] elopio: rollback fails for you, right? [03:33] sergiusens: fails for me as well [03:33] sergiusens: check https://bugs.launchpad.net/snappy/+bug/1474126 [03:33] Launchpad bug 1474126 in Snappy "failed to rollback from 15.04 alpha #4 to #3" [Critical,Confirmed] [03:33] rsalveti: I have this fix I've been trying to test for a while now :-/ [03:34] rsalveti: first a bad sdcard with fully corrupted content, now this... [03:34] right, this corruption issue seems to be quite normal [03:34] rsalveti: oh, and not sure I ever noticed, but it doesn't look like vfat [03:34] seems at the moment we call fsck for the vfat partition things break [03:34] rsalveti: HARDWA~1.YAM [03:35] sergiusens: that is vfat [03:35] yeah [03:35] rsalveti: don't we get long file names? [03:36] that's an extension [03:36] it seems fsck changed it [03:36] it is supported by the original flash [03:36] all good and so on, but at the time it runs fsck, boom [03:37] the time it worked fine I had a FSCK... file in the partition [03:37] FSCK0000.REC [03:37] when it failed I didn't get this file, but the content changed [03:38] not sure yet when we call fsck [03:38] rsalveti: fsck is part of our new init? [03:38] doesn't look like it [03:38] maybe part of the boot process, done by systemd, not sure [03:40] going to bed now, we can check tomorrow [03:40] rsalveti: ok, then you don't get to test my fix [03:40] :-P [03:40] until tomorrow [03:40] right :-) [03:40] rsalveti: smart in using the dup files btw ;-) [03:40] propose it anyway [03:41] great [03:41] rsalveti: I will, as soon as I can do some street testing [03:41] alright, later man, have a good night [03:42] good night === chihchun_afk is now known as chihchun [07:02] good morning [07:12] good morning === erkules_ is now known as erkules === chihchun is now known as chihchun_afk [08:55] yippie !!! [08:55] https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/wily/ubuntu-core-system-image/+build/32370 [08:55] we has arm64 images :) [09:05] ogra_: oh. does that mean I'll be able to try something on the dragonborad? [09:06] tbr, well, no idea if it boots or how our bootloader setup on arm64 is at all [09:06] (yes, I'm able to bring my own hw-adaptation. Just rebuilt the kernel to get usb-networking) [09:07] the state of bootloaders on aargh64 is 'special'. [09:07] at least all packages build and enable rolling a rootfs, not sure i can say more about these images :) [09:07] Good morning all; happy Pandemonium Day! 😃 [09:08] uuuuuh [09:08] that's fine. I might give it a go, but will probably have some questions [09:08] * tbr wishes JamesTait a merry Setting Orange [09:09] * JamesTait sets his orange on the desk, to eat later. [10:53] sourceFiles |= set([os.path.join(root, d) for d in dirs]) [10:53] mterry: [10:53] oh, no mterry [11:54] morning [11:54] mo'in [12:20] ogra_: mvo: did you guys had time to check the email sergiusens sent yesterday? [12:20] about bug 1474126 [12:20] bug 1474126 in Snappy "failed to rollback from 15.04 alpha #4 to #3" [Critical,Confirmed] https://launchpad.net/bugs/1474126 [12:37] rsalveti: not yet, sorry, let me do that now. I was absorbed in snapcraft stuff, mterry has too many good ideas :) [12:37] * mvo looks now [12:37] mvo: haha, no worries :-) [12:38] hmm [12:38] * ogra_ didnt notice a mail [12:38] checking again [12:38] ogra_: it seems sergiusens forgot to include you [12:38] but was basically about this bug ^ [12:39] he created https://code.launchpad.net/~sergiusens/snappy/noUselessUpdate/+merge/264665 to fix 1474125 [12:39] but we can't test it easily because of bug 1474126 [12:39] bug 1474126 in Snappy "failed to rollback from 15.04 alpha #4 to #3" [Critical,Confirmed] https://launchpad.net/bugs/1474126 [12:39] right, i saw that [12:40] so as i commented on the bug, it could well be that fatwrite messes up the filesystem [12:40] * ogra_ doesnt really trust it [12:40] still so new and rarely used anywhere [12:44] ogra_: yeah, looks like it [12:44] ogra_: I did the update, then decided to run fsck.vfat and it was all good [12:44] but once I rebooted, boom [12:44] i wonder if it is something simple ... like fatwrite not being aboe to truncate files ... [12:45] so you would have to delete them first [12:45] i didnt really find many bugs though ... i guess it is really rarely used [12:45] ogra_: mvo: one other question I had, why do we keep /boot/uboot mounted all the time? [12:45] we shouldnt (i dont know why, ask jodh ? ) [12:46] rsalveti: I don't think there is a good reason [12:46] also, why dont we sync mount it ? [12:46] I think we do that, no? [12:46] * ogra_ doesnt see a sync option [12:46] the sync mount I mean [12:46] /dev/mmcblk0p1 on /boot/uboot type vfat (rw,relatime,sync,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro) [12:46] we do use the sync option [12:46] hmm [12:46] ogra_: outdated initrd ;) ? [12:46] let me remove the fatwrite piece and check again [12:47] (RaspberryPi2)ubuntu@localhost:~$ mount|grep boot [12:47] /dev/mmcblk0p1 on /boot/uboot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro) [12:47] weird [12:47] it seems that makes fsck.vfat unhappy and then it changes the fs [12:47] shouldnt be to outdated [12:47] I was on image 4 (bbb) [12:48] (15.04 stable image 3 though) [12:48] but i thought that should have sync set [12:50] ogra_: yeah, we set to sync a good while ago, in the 15.04 timeframe [12:50] odd [12:50] right [12:50] i use the initrd from cdimage with removed modules [12:50] nothing else changed [12:51] rsalveti, are our tools able to remount /boot rw before operating on it ? [12:52] just switching it to ro might be a bit dangerous if they dont :) [12:53] /boot/uboot is always rw [12:54] yes, it shouldnt [12:54] but if we make it ro, can the tools deal with it ? [12:55] hmm [12:55] else [12:55] bootdir="$ubootdir" [12:55] echo "$boot_partition $bootdir auto sync 0 2" >> "$fstab" [12:55] f [12:55] I'd expect it to only be mounted when needed, since it's a vfat partition [12:55] so that is what it is supposed to write to fstab [12:55] yup [12:55] (RaspberryPi2)ubuntu@localhost:~$ grep uboot /etc/fstab [12:55] /dev/mmcblk0p1 /boot/uboot auto defaults 0 0 [12:55] and that is what my fstab has [12:56] /dev/mmcblk0p1 /boot/uboot auto sync 0 2 [12:56] bbb, image 3 [12:56] rpi image 3 [12:56] how weird [12:59] i wonder if systemd re-writes the fstab entry somehow [12:59] i dont see any explanation in the code why it would be "auto default" === chihchun_afk is now known as chihchun [13:21] mvo: oh, and https://code.launchpad.net/~sergiusens/snappy/noUselessUpdate/+merge/264665 reminds me that we need to prioritize the work to update the upgrader before doing the upgrade [13:21] Morning folks [13:21] not super urgent, but would be good for us to put some hands on it later this month [13:21] Chipaca, Were you able to get a libxcbcommon patch working? [13:21] tedg: i'm not sure, right now [13:22] um [13:22] read: i have a working patch [13:22] but i'm not sure the patch works [13:22] is that clearer? i'm not sure :) [13:22] Hmm, how do you know it's a working patch then? :-) [13:23] rsalveti: indeed [13:24] tedg: it builds, so we shipped it. [13:24] \o/ [13:26] tedg: and the not sure is because http://pastebin.ubuntu.com/11877692/ [13:27] morning [13:28] Chipaca, There seems to be reads from /home/ubuntu/thing/lib and /apps/qmlapp.sideload/0/usr/lib, seems there should only be one of those? [13:28] sergiusens: morning :-) [13:29] tedg: it is one, there's a symlink but i'll unify so it melts minds less [13:30] If you're running confined, symlinks don't help. [13:30] not running confined [13:31] sergiusens: ogra_: mvo: yeah, if I remove the fatwrite command it all works fine [13:32] hey sergiusens, good morning [13:32] tedg: http://pastebin.ubuntu.com/11877734/ [13:35] Chipaca: hm, that looks like progress compared to yesterday, no? [13:35] tedg: oh, wait, that one is using the wrong socket [13:35] Chipaca, Ah, so to the next step :-) Looks like it can't find the Mir socket. Probably not running as user 1000 [13:35] tedg: with the right socket: http://pastebin.ubuntu.com/11877746/ [13:35] Yeah, need to set MIR_SOCKET probably. [13:35] rsalveti, well, how do you switch ? by manually editing the file ? [13:35] yeh, i'd commented it out, because with that it finds it but still can't connect (?) [13:36] but it gets further ahead with it, so [13:36] permission denied [13:36] dammit [13:36] http://pastebin.ubuntu.com/11877751/ [13:36] ^ with the right perms, and the right socket [13:37] What Mir server are we using? Demo Server? System compositor? [13:37] demo server [13:37] which is what is in the mir package [13:38] K, I don't think that it requires notification that we're going to connect. I think it stubs out the auth parts. [13:38] tedg: the permission denied was on the socket itself [13:38] tedg: but i don't know if you're talking about that :) [13:39] Kinda both :-) [13:39] Mir also does it's own auth depending on the server. [13:39] ah, ok [13:39] There are some MIR debugging environment variables, let me try to find them. [13:40] the mir wrapper does take care of the socket and its permissions, but i'm running it directly myself to have the console output [13:40] Well, you can get the console output from systemd [13:42] i know i should be able to, but couldn't find it in journalctl [13:42] Oh, my Mir branch was only 200K objects out of date... [13:43] rsalveti, so i'd really like to get rid of using fatwrite, i wonder if we could ship a uboot.env file wih a binary environment instead ... only containing the a/b variable, that way we can just run saveenv to switch it (if anything reads the snappy_stamp.txt file from userspace we need to handle that separately though) [13:44] Chipaca, I think this is the useful variable: MIR_CLIENT_RPC_REPORT=log [13:44] tedg: in the environ of the server, client, both? [13:45] Chipaca, Client, though sever can also be used if we're confused still :-) [13:45] Chipaca, Here is what I wrote down: MIR_SERVER_{DISPLAY_REPORT,COMPOSITOR_REPORT,CONNECTOR_REPORT,MSG_PROCESSOR_REPORT,SESSION_MEDIATOR_REPORT}=log [13:45] sergiusens: thanks for your branch, I'm trying to reproduce this now [13:45] oh yeah baby [13:46] tedg: http://pastebin.ubuntu.com/11877784/ [13:46] rsalveti: is there anything unusual about your sd card? particularly big maybe? [13:47] Chipaca, Huh, perhaps look at the server? [13:47] tedg: that was on the server, fwiw [13:47] Otherwise we're gonna need #ubuntu-mir's help [13:47] Oh, try on the client. [13:48] tedg: i get no extra input from the client with those, nor with MIR_CLIENT_ of same [13:53] tedg: i've got to run, will read when i return [13:53] Chipaca, K, I'll ping some central time folks when they log in as well. [13:53] ta [13:54] and i'll get the different bits and bobs up when i return, also, so others can reproduce [13:55] mvo: np [13:59] pitti, could anything in systemd touch/modify fstab during boot ? [14:00] * ogra_ really doesnt get how the /boot/uboot entry can end up differently in fstab than what was echoed into it [14:00] ogra_: read for sure; changing, I'm not aware of anything [14:01] i have this code ... echo "$boot_partition $bootdir auto sync 0 2" >> "$fstab" [14:01] and end up with: [14:01] (RaspberryPi2)ubuntu@localhost:~$ grep boot /etc/fstab [14:01] /dev/mmcblk0p1 /boot/uboot auto defaults 0 0 [14:02] i dont get how that can happen [14:02] so that's not system-image writing fstab from a template and writable-paths or so? [14:02] and is /etc/fstab writable in the first place? [14:02] no, it is the initrd script checking for uboot and then echoing this line into fstab [14:03] so /etc/fstab is a symlink to /etc/writable/fstab or so? [14:03] to my knowledge thats the only place we handle /boot/uboot [14:03] fstab is a bindmount [14:03] into /run/image.fstab [14:04] so perhaps your echo happens before the bind mount? [14:04] http://paste.ubuntu.com/11877845/ [14:04] thats the code snippet [14:04] (it happens afterwards) [14:06] i dont see anything wrong with that code [14:06] what does handle_writable_paths() do? [14:06] it doesn't happen to assign fstab= to something else by chance? [14:06] it loops over the writable paths file and echos the lines as proper fstab lines [14:06] oh [14:06] * ogra_ checks for uboot in writable-paths [14:07] (RaspberryPi2)ubuntu@localhost:~$ grep boot /etc/system-image/writable-paths [14:07] (RaspberryPi2)ubuntu@localhost:~$ [14:07] nope [14:09] mvo: nothing unusual, no [14:09] ogra_: we got logic from our userspace to check for that stamp file [14:09] to then change the boot mode because it was successful [14:09] since it boots with "try" first [14:10] rsalveti, right, so the uboot.env would have to add it to cmdline and the initrd would have to echo that into the stamp file [14:10] so that uboot.env still stays the master and writin only happens from the kernel vfat driver [14:10] and not from uboot [14:11] right, I guess that's a way [14:11] env + kernel cmdline + logic in initrd [14:11] yeah [14:11] just requires that all uboots we support can handle such an env [14:12] at least for upgradeable devices [14:12] ogra_: are we shipping uboot with bbb? [14:12] or are we booting the uboot that is already flashed in the board? [14:12] we are shipping a uboot.img [14:12] (and SPL too i think) [14:12] that uboot is super old [14:13] hmm [14:13] i wonder if you guys actually boot the EMMc one [14:13] easy to check [14:14] ogra_: yeah, we're loading the spl and uboot that is already flashed in the emmc [14:14] ogra_: I like your idea [14:14] aha ! [14:15] because to boot from the sd card completely you have to hold the button while turning on the board [14:15] rsalveti, so likely a different fatwrite then [14:15] U-Boot SPL 2014.10-dirty (Dec 18 2014 - 22:07:26) [14:15] or wipe the MMC [14:15] U-Boot 2014.10-dirty (Dec 18 2014 - 22:07:26) [14:15] the one we ship [14:15] right [14:15] U-Boot SPL 2014.04-00015-gb4422bd (Apr 22 2014 - 13:24:29) [14:15] U-Boot 2014.04-00015-gb4422bd (Apr 22 2014 - 13:24:29) [14:15] the one we boot [14:16] well, less dirty at least :P [14:16] damn ... [14:16] didnt zyga's patch land that was supposed to enforce that ? [14:16] ogra_: nops [14:17] since the only way to force our bootloader is by not having the original one or by pressing the button [14:17] https://code.launchpad.net/~zyga/snappy-hub/fix-1464275/+merge/261833 [14:17] hm, I did a fresh flash of my sd card some minutes ago and I have: "U-Boot 2014.10-dirty (Dec 18 2014 - 22:07:26)" in my boot message - but I did press the button in the past, will it remember that? [14:17] maybe you wiped out your emmc [14:17] well, it is supposed to remember it [14:18] but i have no clue how that works [14:18] both is likely [14:18] don't think it remembers it [14:18] ok [14:18] at the moment you unplug power and plug it again, it should try emmc first again [14:18] so looking at zyga'S patch there ... it would still use the MMC uboot [14:18] but enforce loading the SD [14:19] so that wouldnt actualyl help [14:19] no, when you run that you're already in uboot [14:19] right :/ [14:19] * ogra_ sees no way around this except wiping the MMC on first boto or some such [14:20] *boot [14:20] ... not great :/ [14:20] ogra_: or do your env dance suggestion you said [14:21] I'm checking now if our fatwrite is better [14:21] well, will that help ? are we sure the bootx command behaves the same ... etc etc ? [14:21] its not only fatwrite [14:22] i ean, we can indeed use my suggestion to take fatwrite out of the equation ... but there are 6 months between the two uboots ... i would be surprised if we wouldnt hit other issues [14:22] *mean [14:24] (8 monts actually ... temporary math weakness :P ) [14:25] sergiusens, rsalveti: hrm, so I tried to test 15.04a3 -> 15.04a4 -> 15.04a3 but when I rollback I get the following error http://paste.ubuntu.com/11877942/ on boot. this is new, right? [14:25] ogra_: -^ [14:26] mvo, no, thats exactly the issue we are diuscussing [14:26] *discussing [14:26] ogra_: thats the corruption? [14:26] fatwrite updates the file and trashes the vfat [14:26] ok, cool [14:26] if the fatwrite from the older uboot is used [14:26] so … I get this with "U-Boot 2014.10-dirty (Dec 18 2014 - 22:07:26) [14:26] " [14:27] yeah, so that shows that our uboot is busted as well [14:27] I never see a different U-Boot boot string [14:27] yeah [14:27] *weeeh* [14:27] and another interesting thing, my ethernet doesn't work with our uboot [14:27] it only works with the uboot provided by the board [14:28] got bbb rev c [14:28] wow. thats weird [14:29] mvo: that's the issue that elopio had for quite a while [14:29] but that we were never able to find out why [14:29] mvo: that's the second bug === bjf is now known as __bjf [14:29] mvo: to test the fix for the first bug, resync the kernel and initrd from /boot ;-) [14:30] good morning. [14:31] sergiusens: :) just commented, I will look into adding a test now, but a cup of tea first :) [14:31] fgimenez: looking at your failures, for some reason systemctl status docker_docker-daemon_*.service prints nothing for you on the stdout. [14:31] I'll check what's going on. [14:31] fgimenez: lets skip our meeting today, unless you want to discuss about something. [14:31] sergiusens: honestly this fatwrite thing is really alarming to me shows that we really need some bbb auto testing [14:32] elopio, ok np, i'll check the output of systemctl [14:34] mvo: yeah [14:34] on the good side plars is getting that going soon :-) [14:34] but yeah, it's super bad [14:36] mvo: to be fair, this is an untested code path [14:36] sergiusens, rolling back and upgradin is untested ? [14:36] guess the main problem is when we're changing kernel versions [14:37] fgimenez: I see, the *.service seems to work on 15.04 but not on rolling. [14:37] when updating/rolling back with the same kernel version I think that's mostly fine [14:37] but might still get broken I believe, and why elopio got it from time to time [14:37] ogra_: for new kernels, yes [14:37] ah, it is our first new kernel ? [14:37] k [14:38] elopio, yes, the output is from rolling/edge 107 [14:39] ogra_: I don't think so, but channel changing for testing this are all flawed IMO (unless it is indeed a channel change test) [14:39] elopio, btw i have a branch for selecting the release, branch and revision for the tests almost done, will ping you when pushed [14:40] fgimenez: nice. [14:40] rsalveti: yeah, thats the good part, we have the tests and a plan and all that :) [14:41] and thats probably what the erle guys also experienced from time to time [14:41] yeah [14:42] pitti: do you know why this works on systemd 219 but not on 222? [14:42] systemctl status docker_docker-daemon_*.service [14:48] elopio: not off the top of my head; what does "not work" mean? [14:48] pitti: on 219 it returs the status. On 222 it returns nothing. [14:52] elopio: indeed; mind filing a bug about this, so that we can track this? (busy with something else right now) [14:52] pitti: sure thing. [14:52] elopio: thanks; I'll bisect it [14:53] rsalveti: ogra_ if we revert the fix in uenv.txt, will we get something workable again (wrt fatwrite)? [14:53] sergiusens, no [14:53] that fix wont change anything [14:54] ogra_: why did this only start now? [14:54] sergiusens, dunno, did we do this kind of test before ? (multiple updates and rollbacks) [14:55] ogra_: yes we have [14:56] ogra_: just the previous release was a 3 to 4 jump between releases [14:56] probably it only happenms if the fat content changes between two fatwrites [14:56] which now happens due to the kernel upgrade [14:58] (i'm really just guessing here ... fact is that it doesnt break if fatwrite isnt used at all) [14:59] right, I believe it's because of the newer kernel [15:00] do we know if the uboot fat driver can handle long filenames ? [15:00] pitti: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1474426 [15:00] Launchpad bug 1474426 in systemd (Ubuntu) "systemctl status globbing does not work on version 222" [Undecided,New] [15:00] we also started using the longer filenames for the kernel [15:00] fgimenez: for now, I'll modify GetCurrentVersion to get a packageName. [15:01] (thanks to the version) === chihchun is now known as chihchun_afk [15:03] elopio: thanks [15:16] ogra_: rsalveti: sorry, just now catching up after I got off a call - yes this is what I was complaining about a long time ago when trying to get BBB to boot properly - that we just accidentally boot snappy on it because we have a working uboot on emmc, and that uboot could easily break on the sd card if you don't force it to boot from there [15:17] plars, yeah, but there is no way for us to enforce SD ... [15:17] unless we wipe it [15:17] well, wipe the MMC [15:17] ogra_: there is one other way [15:18] ohhhh? [15:18] how ? [15:18] ogra_: holding the s2 button down, or jumpering lcd_data02 I think, to gnd will do it [15:18] right [15:18] it forces booting from the sd (including bootloader) [15:18] or solder a wire ... [15:18] ogra_: well, a jumper wire is so much easier :) [15:18] not realy a softwareish solution :) [15:19] and we cant really destroy peoples MMC installs [15:19] ogra_: so my current plans for instrumenting bbb include this, and use s2 along with a stable image on emmc to flash the sd, then force booting onto the sd [15:19] ogra_: but if/when we get the recovery stuff, then we won't want to do that of course - and it becomes really critical that we don't break uboot [15:20] yeah :( [15:20] ogra_: but that makes me wonder, do we really need to change the bootloader that often? except maybe at the beginning? [15:20] ogra_: once it stabilizes shouldn't we be able to mostly leave it alone? [15:21] well, its not our bootloader that has issues ... it is the case where we boot the onboard one and use fatwrite from it [15:21] if you are updating it only ever 12-16 months, it's much easier to be super careful and make sure we don't break it before landing the change [15:21] ah [15:22] ogra_: why is it calling fatwrite from the onboard one? [15:22] the uboot scripts execute it to set the a or b partition in a file on the boot partition [15:22] oh, right [15:23] so this is just the nature of bbb, by default it tries to load the bootloader from emmc, unless you press the s2 button after poweron [15:24] FWIW, in both my bbb [15:24] I didn't need to press anything [15:24] it boots from the SD if it's there [15:24] beuno: yeah, that's accidental and the source of the problem :) [15:25] plars, this is why I don't do hardware :p [15:25] beuno: I've been complaining about this for a while now, what's happening is that it's loading the bootloader off of your emmc [15:26] but the bbb default image is set up to check if there's something it can load from the sd [15:26] pitti: other question, is there a way to wait for a service to be loaded and running? [15:27] elopio: another unit should just have After=foo.service [15:27] pitti: but not inside a unit. Like I install a package, and I want a command to wait until it's up before using it. [15:28] elopio: other than polling "systemctl is-active foo.service" I don't know; usually invoke-rc.d waits until it's up already [15:28] elopio: asking on #systemd (freenode) might be useful, perhaps I'm missing some kind of "wait-active" [15:28] pitti: ack, thanks. [15:29] is-active is already an improvement to what I have :) [15:29] elopio: other than that a while/is-active/sleep loop would work for now [15:30] elopio: that smells a bit weird -- you have a package whose postinst auto-starts a service, but it's not up when apt-get finishes? [15:30] ogra_: if there's a known fix for this uboot bug, I'm sure rcn-ee would be open to adding it in his images, then we could point people who are running the default image on emmc to update to that? [15:30] elopio: packages are expected to do that; i. e. that sounds like a bug in that .service, or in the package [15:30] pitti: well, on snaps we don't have post-inst. [15:30] plars, well, the issue is to use fatwrite at all :P [15:31] oh [15:31] plars, i'm looking at a way to not use it at all [15:31] I'm not sure if we can make something to wait on install. [15:31] ok [15:31] elopio: ah, ok [15:31] elopio: but what would start the shipped units then? [15:31]