| sergiusens | elopio: rollback fails for you, right? | 03:31 |
|---|---|---|
| rsalveti | sergiusens: fails for me as well | 03:33 |
| rsalveti | sergiusens: check https://bugs.launchpad.net/snappy/+bug/1474126 | 03:33 |
| ubottu | Launchpad bug 1474126 in Snappy "failed to rollback from 15.04 alpha #4 to #3" [Critical,Confirmed] | 03:33 |
| sergiusens | rsalveti: I have this fix I've been trying to test for a while now :-/ | 03:33 |
| sergiusens | rsalveti: first a bad sdcard with fully corrupted content, now this... | 03:34 |
| rsalveti | right, this corruption issue seems to be quite normal | 03:34 |
| sergiusens | rsalveti: oh, and not sure I ever noticed, but it doesn't look like vfat | 03:34 |
| rsalveti | seems at the moment we call fsck for the vfat partition things break | 03:34 |
| sergiusens | rsalveti: HARDWA~1.YAM | 03:34 |
| rsalveti | sergiusens: that is vfat | 03:35 |
| sergiusens | yeah | 03:35 |
| sergiusens | rsalveti: don't we get long file names? | 03:35 |
| rsalveti | that's an extension | 03:36 |
| rsalveti | it seems fsck changed it | 03:36 |
| rsalveti | it is supported by the original flash | 03:36 |
| rsalveti | all good and so on, but at the time it runs fsck, boom | 03:36 |
| rsalveti | the time it worked fine I had a FSCK... file in the partition | 03:37 |
| rsalveti | FSCK0000.REC | 03:37 |
| rsalveti | when it failed I didn't get this file, but the content changed | 03:37 |
| rsalveti | not sure yet when we call fsck | 03:38 |
| sergiusens | rsalveti: fsck is part of our new init? | 03:38 |
| rsalveti | doesn't look like it | 03:38 |
| rsalveti | maybe part of the boot process, done by systemd, not sure | 03:38 |
| rsalveti | going to bed now, we can check tomorrow | 03:40 |
| sergiusens | rsalveti: ok, then you don't get to test my fix | 03:40 |
| sergiusens | :-P | 03:40 |
| sergiusens | until tomorrow | 03:40 |
| rsalveti | right :-) | 03:40 |
| sergiusens | rsalveti: smart in using the dup files btw ;-) | 03:40 |
| rsalveti | propose it anyway | 03:40 |
| rsalveti | great | 03:41 |
| sergiusens | rsalveti: I will, as soon as I can do some street testing | 03:41 |
| rsalveti | alright, later man, have a good night | 03:41 |
| sergiusens | good night | 03:42 |
| === chihchun_afk is now known as chihchun | ||
| fgimenez | good morning | 07:02 |
| dholbach | good morning | 07:12 |
| === erkules_ is now known as erkules | ||
| === chihchun is now known as chihchun_afk | ||
| ogra_ | yippie !!! | 08:55 |
| ogra_ | https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/wily/ubuntu-core-system-image/+build/32370 | 08:55 |
| ogra_ | we has arm64 images :) | 08:55 |
| tbr | ogra_: oh. does that mean I'll be able to try something on the dragonborad? | 09:05 |
| ogra_ | tbr, well, no idea if it boots or how our bootloader setup on arm64 is at all | 09:06 |
| tbr | (yes, I'm able to bring my own hw-adaptation. Just rebuilt the kernel to get usb-networking) | 09:06 |
| tbr | the state of bootloaders on aargh64 is 'special'. | 09:07 |
| ogra_ | at least all packages build and enable rolling a rootfs, not sure i can say more about these images :) | 09:07 |
| JamesTait | Good morning all; happy Pandemonium Day! 😃 | 09:07 |
| ogra_ | uuuuuh | 09:08 |
| tbr | that's fine. I might give it a go, but will probably have some questions | 09:08 |
| * tbr wishes JamesTait a merry Setting Orange | 09:08 | |
| * JamesTait sets his orange on the desk, to eat later. | 09:09 | |
| Chipaca | sourceFiles |= set([os.path.join(root, d) for d in dirs]) | 10:53 |
| Chipaca | mterry: | 10:53 |
| Chipaca | oh, no mterry | 10:53 |
| rsalveti | morning | 11:54 |
| Chipaca | mo'in | 11:54 |
| rsalveti | ogra_: mvo: did you guys had time to check the email sergiusens sent yesterday? | 12:20 |
| rsalveti | about bug 1474126 | 12:20 |
| ubottu | bug 1474126 in Snappy "failed to rollback from 15.04 alpha #4 to #3" [Critical,Confirmed] https://launchpad.net/bugs/1474126 | 12:20 |
| mvo | rsalveti: not yet, sorry, let me do that now. I was absorbed in snapcraft stuff, mterry has too many good ideas :) | 12:37 |
| * mvo looks now | 12:37 | |
| rsalveti | mvo: haha, no worries :-) | 12:37 |
| ogra_ | hmm | 12:38 |
| * ogra_ didnt notice a mail | 12:38 | |
| ogra_ | checking again | 12:38 |
| rsalveti | ogra_: it seems sergiusens forgot to include you | 12:38 |
| rsalveti | but was basically about this bug ^ | 12:38 |
| rsalveti | he created https://code.launchpad.net/~sergiusens/snappy/noUselessUpdate/+merge/264665 to fix 1474125 | 12:39 |
| rsalveti | but we can't test it easily because of bug 1474126 | 12:39 |
| ubottu | bug 1474126 in Snappy "failed to rollback from 15.04 alpha #4 to #3" [Critical,Confirmed] https://launchpad.net/bugs/1474126 | 12:39 |
| ogra_ | right, i saw that | 12:39 |
| ogra_ | so as i commented on the bug, it could well be that fatwrite messes up the filesystem | 12:40 |
| * ogra_ doesnt really trust it | 12:40 | |
| ogra_ | still so new and rarely used anywhere | 12:40 |
| rsalveti | ogra_: yeah, looks like it | 12:44 |
| rsalveti | ogra_: I did the update, then decided to run fsck.vfat and it was all good | 12:44 |
| rsalveti | but once I rebooted, boom | 12:44 |
| ogra_ | i wonder if it is something simple ... like fatwrite not being aboe to truncate files ... | 12:44 |
| ogra_ | so you would have to delete them first | 12:45 |
| ogra_ | i didnt really find many bugs though ... i guess it is really rarely used | 12:45 |
| rsalveti | ogra_: mvo: one other question I had, why do we keep /boot/uboot mounted all the time? | 12:45 |
| ogra_ | we shouldnt (i dont know why, ask jodh ? ) | 12:45 |
| mvo | rsalveti: I don't think there is a good reason | 12:46 |
| ogra_ | also, why dont we sync mount it ? | 12:46 |
| mvo | I think we do that, no? | 12:46 |
| * ogra_ doesnt see a sync option | 12:46 | |
| mvo | the sync mount I mean | 12:46 |
| rsalveti | /dev/mmcblk0p1 on /boot/uboot type vfat (rw,relatime,sync,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro) | 12:46 |
| rsalveti | we do use the sync option | 12:46 |
| ogra_ | hmm | 12:46 |
| mvo | ogra_: outdated initrd ;) ? | 12:46 |
| rsalveti | let me remove the fatwrite piece and check again | 12:46 |
| ogra_ | (RaspberryPi2)ubuntu@localhost:~$ mount|grep boot | 12:47 |
| ogra_ | /dev/mmcblk0p1 on /boot/uboot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro) | 12:47 |
| ogra_ | weird | 12:47 |
| rsalveti | it seems that makes fsck.vfat unhappy and then it changes the fs | 12:47 |
| ogra_ | shouldnt be to outdated | 12:47 |
| rsalveti | I was on image 4 (bbb) | 12:47 |
| ogra_ | (15.04 stable image 3 though) | 12:48 |
| ogra_ | but i thought that should have sync set | 12:48 |
| mvo | ogra_: yeah, we set to sync a good while ago, in the 15.04 timeframe | 12:50 |
| mvo | odd | 12:50 |
| ogra_ | right | 12:50 |
| ogra_ | i use the initrd from cdimage with removed modules | 12:50 |
| ogra_ | nothing else changed | 12:50 |
| ogra_ | rsalveti, are our tools able to remount /boot rw before operating on it ? | 12:51 |
| ogra_ | just switching it to ro might be a bit dangerous if they dont :) | 12:52 |
| rsalveti | /boot/uboot is always rw | 12:53 |
| ogra_ | yes, it shouldnt | 12:54 |
| ogra_ | but if we make it ro, can the tools deal with it ? | 12:54 |
| ogra_ | hmm | 12:55 |
| ogra_ | else | 12:55 |
| ogra_ | bootdir="$ubootdir" | 12:55 |
| ogra_ | echo "$boot_partition $bootdir auto sync 0 2" >> "$fstab" | 12:55 |
| ogra_ | f | 12:55 |
| rsalveti | I'd expect it to only be mounted when needed, since it's a vfat partition | 12:55 |
| ogra_ | so that is what it is supposed to write to fstab | 12:55 |
| rsalveti | yup | 12:55 |
| ogra_ | (RaspberryPi2)ubuntu@localhost:~$ grep uboot /etc/fstab | 12:55 |
| ogra_ | /dev/mmcblk0p1 /boot/uboot auto defaults 0 0 | 12:55 |
| ogra_ | and that is what my fstab has | 12:55 |
| rsalveti | /dev/mmcblk0p1 /boot/uboot auto sync 0 2 | 12:56 |
| rsalveti | bbb, image 3 | 12:56 |
| ogra_ | rpi image 3 | 12:56 |
| ogra_ | how weird | 12:56 |
| ogra_ | i wonder if systemd re-writes the fstab entry somehow | 12:59 |
| ogra_ | i dont see any explanation in the code why it would be "auto default" | 12:59 |
| === chihchun_afk is now known as chihchun | ||
| rsalveti | mvo: oh, and https://code.launchpad.net/~sergiusens/snappy/noUselessUpdate/+merge/264665 reminds me that we need to prioritize the work to update the upgrader before doing the upgrade | 13:21 |
| tedg | Morning folks | 13:21 |
| rsalveti | not super urgent, but would be good for us to put some hands on it later this month | 13:21 |
| tedg | Chipaca, Were you able to get a libxcbcommon patch working? | 13:21 |
| Chipaca | tedg: i'm not sure, right now | 13:21 |
| Chipaca | um | 13:22 |
| Chipaca | read: i have a working patch | 13:22 |
| Chipaca | but i'm not sure the patch works | 13:22 |
| Chipaca | is that clearer? i'm not sure :) | 13:22 |
| tedg | Hmm, how do you know it's a working patch then? :-) | 13:22 |
| mvo | rsalveti: indeed | 13:23 |
| mvo | tedg: it builds, so we shipped it. | 13:24 |
| tedg | \o/ | 13:24 |
| Chipaca | tedg: and the not sure is because http://pastebin.ubuntu.com/11877692/ | 13:26 |
| sergiusens | morning | 13:27 |
| tedg | Chipaca, There seems to be reads from /home/ubuntu/thing/lib and /apps/qmlapp.sideload/0/usr/lib, seems there should only be one of those? | 13:28 |
| rsalveti | sergiusens: morning :-) | 13:28 |
| Chipaca | tedg: it is one, there's a symlink but i'll unify so it melts minds less | 13:29 |
| tedg | If you're running confined, symlinks don't help. | 13:30 |
| Chipaca | not running confined | 13:30 |
| rsalveti | sergiusens: ogra_: mvo: yeah, if I remove the fatwrite command it all works fine | 13:31 |
| mvo | hey sergiusens, good morning | 13:32 |
| Chipaca | tedg: http://pastebin.ubuntu.com/11877734/ | 13:32 |
| mvo | Chipaca: hm, that looks like progress compared to yesterday, no? | 13:35 |
| Chipaca | tedg: oh, wait, that one is using the wrong socket | 13:35 |
| tedg | Chipaca, Ah, so to the next step :-) Looks like it can't find the Mir socket. Probably not running as user 1000 | 13:35 |
| Chipaca | tedg: with the right socket: http://pastebin.ubuntu.com/11877746/ | 13:35 |
| tedg | Yeah, need to set MIR_SOCKET probably. | 13:35 |
| ogra_ | rsalveti, well, how do you switch ? by manually editing the file ? | 13:35 |
| Chipaca | yeh, i'd commented it out, because with that it finds it but still can't connect (?) | 13:35 |
| Chipaca | but it gets further ahead with it, so | 13:36 |
| Chipaca | permission denied | 13:36 |
| Chipaca | dammit | 13:36 |
| Chipaca | http://pastebin.ubuntu.com/11877751/ | 13:36 |
| Chipaca | ^ with the right perms, and the right socket | 13:36 |
| tedg | What Mir server are we using? Demo Server? System compositor? | 13:37 |
| Chipaca | demo server | 13:37 |
| Chipaca | which is what is in the mir package | 13:37 |
| tedg | K, I don't think that it requires notification that we're going to connect. I think it stubs out the auth parts. | 13:38 |
| Chipaca | tedg: the permission denied was on the socket itself | 13:38 |
| Chipaca | tedg: but i don't know if you're talking about that :) | 13:38 |
| tedg | Kinda both :-) | 13:39 |
| tedg | Mir also does it's own auth depending on the server. | 13:39 |
| Chipaca | ah, ok | 13:39 |
| tedg | There are some MIR debugging environment variables, let me try to find them. | 13:39 |
| Chipaca | the mir wrapper does take care of the socket and its permissions, but i'm running it directly myself to have the console output | 13:40 |
| tedg | Well, you can get the console output from systemd | 13:40 |
| Chipaca | i know i should be able to, but couldn't find it in journalctl | 13:42 |
| tedg | Oh, my Mir branch was only 200K objects out of date... | 13:42 |
| ogra_ | rsalveti, so i'd really like to get rid of using fatwrite, i wonder if we could ship a uboot.env file wih a binary environment instead ... only containing the a/b variable, that way we can just run saveenv to switch it (if anything reads the snappy_stamp.txt file from userspace we need to handle that separately though) | 13:43 |
| tedg | Chipaca, I think this is the useful variable: MIR_CLIENT_RPC_REPORT=log | 13:44 |
| Chipaca | tedg: in the environ of the server, client, both? | 13:44 |
| tedg | Chipaca, Client, though sever can also be used if we're confused still :-) | 13:45 |
| tedg | Chipaca, Here is what I wrote down: MIR_SERVER_{DISPLAY_REPORT,COMPOSITOR_REPORT,CONNECTOR_REPORT,MSG_PROCESSOR_REPORT,SESSION_MEDIATOR_REPORT}=log | 13:45 |
| mvo | sergiusens: thanks for your branch, I'm trying to reproduce this now | 13:45 |
| Chipaca | oh yeah baby | 13:45 |
| Chipaca | tedg: http://pastebin.ubuntu.com/11877784/ | 13:46 |
| mvo | rsalveti: is there anything unusual about your sd card? particularly big maybe? | 13:46 |
| tedg | Chipaca, Huh, perhaps look at the server? | 13:47 |
| Chipaca | tedg: that was on the server, fwiw | 13:47 |
| tedg | Otherwise we're gonna need #ubuntu-mir's help | 13:47 |
| tedg | Oh, try on the client. | 13:47 |
| Chipaca | tedg: i get no extra input from the client with those, nor with MIR_CLIENT_ of same | 13:48 |
| Chipaca | tedg: i've got to run, will read when i return | 13:53 |
| tedg | Chipaca, K, I'll ping some central time folks when they log in as well. | 13:53 |
| Chipaca | ta | 13:53 |
| Chipaca | and i'll get the different bits and bobs up when i return, also, so others can reproduce | 13:54 |
| sergiusens | mvo: np | 13:55 |
| ogra_ | pitti, could anything in systemd touch/modify fstab during boot ? | 13:59 |
| * ogra_ really doesnt get how the /boot/uboot entry can end up differently in fstab than what was echoed into it | 14:00 | |
| pitti | ogra_: read for sure; changing, I'm not aware of anything | 14:00 |
| ogra_ | i have this code ... echo "$boot_partition $bootdir auto sync 0 2" >> "$fstab" | 14:01 |
| ogra_ | and end up with: | 14:01 |
| ogra_ | (RaspberryPi2)ubuntu@localhost:~$ grep boot /etc/fstab | 14:01 |
| ogra_ | /dev/mmcblk0p1 /boot/uboot auto defaults 0 0 | 14:01 |
| ogra_ | i dont get how that can happen | 14:02 |
| pitti | so that's not system-image writing fstab from a template and writable-paths or so? | 14:02 |
| pitti | and is /etc/fstab writable in the first place? | 14:02 |
| ogra_ | no, it is the initrd script checking for uboot and then echoing this line into fstab | 14:02 |
| pitti | so /etc/fstab is a symlink to /etc/writable/fstab or so? | 14:03 |
| ogra_ | to my knowledge thats the only place we handle /boot/uboot | 14:03 |
| ogra_ | fstab is a bindmount | 14:03 |
| ogra_ | into /run/image.fstab | 14:03 |
| pitti | so perhaps your echo happens before the bind mount? | 14:04 |
| ogra_ | http://paste.ubuntu.com/11877845/ | 14:04 |
| ogra_ | thats the code snippet | 14:04 |
| ogra_ | (it happens afterwards) | 14:04 |
| ogra_ | i dont see anything wrong with that code | 14:06 |
| pitti | what does handle_writable_paths() do? | 14:06 |
| pitti | it doesn't happen to assign fstab= to something else by chance? | 14:06 |
| ogra_ | it loops over the writable paths file and echos the lines as proper fstab lines | 14:06 |
| ogra_ | oh | 14:06 |
| * ogra_ checks for uboot in writable-paths | 14:06 | |
| ogra_ | (RaspberryPi2)ubuntu@localhost:~$ grep boot /etc/system-image/writable-paths | 14:07 |
| ogra_ | (RaspberryPi2)ubuntu@localhost:~$ | 14:07 |
| ogra_ | nope | 14:07 |
| rsalveti | mvo: nothing unusual, no | 14:09 |
| rsalveti | ogra_: we got logic from our userspace to check for that stamp file | 14:09 |
| rsalveti | to then change the boot mode because it was successful | 14:09 |
| rsalveti | since it boots with "try" first | 14:09 |
| ogra_ | rsalveti, right, so the uboot.env would have to add it to cmdline and the initrd would have to echo that into the stamp file | 14:10 |
| ogra_ | so that uboot.env still stays the master and writin only happens from the kernel vfat driver | 14:10 |
| ogra_ | and not from uboot | 14:10 |
| rsalveti | right, I guess that's a way | 14:11 |
| rsalveti | env + kernel cmdline + logic in initrd | 14:11 |
| ogra_ | yeah | 14:11 |
| ogra_ | just requires that all uboots we support can handle such an env | 14:11 |
| ogra_ | at least for upgradeable devices | 14:12 |
| rsalveti | ogra_: are we shipping uboot with bbb? | 14:12 |
| rsalveti | or are we booting the uboot that is already flashed in the board? | 14:12 |
| ogra_ | we are shipping a uboot.img | 14:12 |
| ogra_ | (and SPL too i think) | 14:12 |
| rsalveti | that uboot is super old | 14:12 |
| ogra_ | hmm | 14:13 |
| ogra_ | i wonder if you guys actually boot the EMMc one | 14:13 |
| rsalveti | easy to check | 14:13 |
| rsalveti | ogra_: yeah, we're loading the spl and uboot that is already flashed in the emmc | 14:14 |
| mvo | ogra_: I like your idea | 14:14 |
| ogra_ | aha ! | 14:14 |
| rsalveti | because to boot from the sd card completely you have to hold the button while turning on the board | 14:15 |
| ogra_ | rsalveti, so likely a different fatwrite then | 14:15 |
| rsalveti | U-Boot SPL 2014.10-dirty (Dec 18 2014 - 22:07:26) | 14:15 |
| ogra_ | or wipe the MMC | 14:15 |
| rsalveti | U-Boot 2014.10-dirty (Dec 18 2014 - 22:07:26) | 14:15 |
| rsalveti | the one we ship | 14:15 |
| rsalveti | right | 14:15 |
| rsalveti | U-Boot SPL 2014.04-00015-gb4422bd (Apr 22 2014 - 13:24:29) | 14:15 |
| rsalveti | U-Boot 2014.04-00015-gb4422bd (Apr 22 2014 - 13:24:29) | 14:15 |
| rsalveti | the one we boot | 14:15 |
| ogra_ | well, less dirty at least :P | 14:16 |
| ogra_ | damn ... | 14:16 |
| ogra_ | didnt zyga's patch land that was supposed to enforce that ? | 14:16 |
| rsalveti | ogra_: nops | 14:16 |
| rsalveti | since the only way to force our bootloader is by not having the original one or by pressing the button | 14:17 |
| ogra_ | https://code.launchpad.net/~zyga/snappy-hub/fix-1464275/+merge/261833 | 14:17 |
| mvo | hm, I did a fresh flash of my sd card some minutes ago and I have: "U-Boot 2014.10-dirty (Dec 18 2014 - 22:07:26)" in my boot message - but I did press the button in the past, will it remember that? | 14:17 |
| rsalveti | maybe you wiped out your emmc | 14:17 |
| ogra_ | well, it is supposed to remember it | 14:17 |
| ogra_ | but i have no clue how that works | 14:18 |
| mvo | both is likely | 14:18 |
| rsalveti | don't think it remembers it | 14:18 |
| mvo | ok | 14:18 |
| rsalveti | at the moment you unplug power and plug it again, it should try emmc first again | 14:18 |
| ogra_ | so looking at zyga'S patch there ... it would still use the MMC uboot | 14:18 |
| ogra_ | but enforce loading the SD | 14:18 |
| ogra_ | so that wouldnt actualyl help | 14:19 |
| rsalveti | no, when you run that you're already in uboot | 14:19 |
| ogra_ | right :/ | 14:19 |
| * ogra_ sees no way around this except wiping the MMC on first boto or some such | 14:19 | |
| ogra_ | *boot | 14:20 |
| ogra_ | ... not great :/ | 14:20 |
| rsalveti | ogra_: or do your env dance suggestion you said | 14:20 |
| rsalveti | I'm checking now if our fatwrite is better | 14:21 |
| ogra_ | well, will that help ? are we sure the bootx command behaves the same ... etc etc ? | 14:21 |
| ogra_ | its not only fatwrite | 14:21 |
| ogra_ | i ean, we can indeed use my suggestion to take fatwrite out of the equation ... but there are 6 months between the two uboots ... i would be surprised if we wouldnt hit other issues | 14:22 |
| ogra_ | *mean | 14:22 |
| ogra_ | (8 monts actually ... temporary math weakness :P ) | 14:24 |
| mvo | sergiusens, rsalveti: hrm, so I tried to test 15.04a3 -> 15.04a4 -> 15.04a3 but when I rollback I get the following error http://paste.ubuntu.com/11877942/ on boot. this is new, right? | 14:25 |
| mvo | ogra_: -^ | 14:25 |
| ogra_ | mvo, no, thats exactly the issue we are diuscussing | 14:26 |
| ogra_ | *discussing | 14:26 |
| mvo | ogra_: thats the corruption? | 14:26 |
| ogra_ | fatwrite updates the file and trashes the vfat | 14:26 |
| mvo | ok, cool | 14:26 |
| ogra_ | if the fatwrite from the older uboot is used | 14:26 |
| mvo | so … I get this with "U-Boot 2014.10-dirty (Dec 18 2014 - 22:07:26) | 14:26 |
| mvo | " | 14:26 |
| rsalveti | yeah, so that shows that our uboot is busted as well | 14:27 |
| mvo | I never see a different U-Boot boot string | 14:27 |
| ogra_ | yeah | 14:27 |
| mvo | *weeeh* | 14:27 |
| rsalveti | and another interesting thing, my ethernet doesn't work with our uboot | 14:27 |
| rsalveti | it only works with the uboot provided by the board | 14:27 |
| rsalveti | got bbb rev c | 14:28 |
| ogra_ | wow. thats weird | 14:28 |
| rsalveti | mvo: that's the issue that elopio had for quite a while | 14:29 |
| rsalveti | but that we were never able to find out why | 14:29 |
| sergiusens | mvo: that's the second bug | 14:29 |
| === bjf is now known as __bjf | ||
| sergiusens | mvo: to test the fix for the first bug, resync the kernel and initrd from /boot ;-) | 14:29 |
| elopio | good morning. | 14:30 |
| mvo | sergiusens: :) just commented, I will look into adding a test now, but a cup of tea first :) | 14:31 |
| elopio | fgimenez: looking at your failures, for some reason systemctl status docker_docker-daemon_*.service prints nothing for you on the stdout. | 14:31 |
| elopio | I'll check what's going on. | 14:31 |
| elopio | fgimenez: lets skip our meeting today, unless you want to discuss about something. | 14:31 |
| mvo | sergiusens: honestly this fatwrite thing is really alarming to me shows that we really need some bbb auto testing | 14:31 |
| fgimenez | elopio, ok np, i'll check the output of systemctl | 14:32 |
| rsalveti | mvo: yeah | 14:34 |
| rsalveti | on the good side plars is getting that going soon :-) | 14:34 |
| rsalveti | but yeah, it's super bad | 14:34 |
| sergiusens | mvo: to be fair, this is an untested code path | 14:36 |
| ogra_ | sergiusens, rolling back and upgradin is untested ? | 14:36 |
| rsalveti | guess the main problem is when we're changing kernel versions | 14:36 |
| elopio | fgimenez: I see, the *.service seems to work on 15.04 but not on rolling. | 14:37 |
| rsalveti | when updating/rolling back with the same kernel version I think that's mostly fine | 14:37 |
| rsalveti | but might still get broken I believe, and why elopio got it from time to time | 14:37 |
| sergiusens | ogra_: for new kernels, yes | 14:37 |
| ogra_ | ah, it is our first new kernel ? | 14:37 |
| ogra_ | k | 14:37 |
| fgimenez | elopio, yes, the output is from rolling/edge 107 | 14:38 |
| sergiusens | ogra_: I don't think so, but channel changing for testing this are all flawed IMO (unless it is indeed a channel change test) | 14:39 |
| fgimenez | elopio, btw i have a branch for selecting the release, branch and revision for the tests almost done, will ping you when pushed | 14:39 |
| elopio | fgimenez: nice. | 14:40 |
| mvo | rsalveti: yeah, thats the good part, we have the tests and a plan and all that :) | 14:40 |
| mvo | and thats probably what the erle guys also experienced from time to time | 14:41 |
| ogra_ | yeah | 14:41 |
| elopio | pitti: do you know why this works on systemd 219 but not on 222? | 14:42 |
| elopio | systemctl status docker_docker-daemon_*.service | 14:42 |
| pitti | elopio: not off the top of my head; what does "not work" mean? | 14:48 |
| elopio | pitti: on 219 it returs the status. On 222 it returns nothing. | 14:48 |
| pitti | elopio: indeed; mind filing a bug about this, so that we can track this? (busy with something else right now) | 14:52 |
| elopio | pitti: sure thing. | 14:52 |
| pitti | elopio: thanks; I'll bisect it | 14:52 |
| sergiusens | rsalveti: ogra_ if we revert the fix in uenv.txt, will we get something workable again (wrt fatwrite)? | 14:53 |
| ogra_ | sergiusens, no | 14:53 |
| ogra_ | that fix wont change anything | 14:53 |
| sergiusens | ogra_: why did this only start now? | 14:54 |
| ogra_ | sergiusens, dunno, did we do this kind of test before ? (multiple updates and rollbacks) | 14:54 |
| sergiusens | ogra_: yes we have | 14:55 |
| sergiusens | ogra_: just the previous release was a 3 to 4 jump between releases | 14:56 |
| ogra_ | probably it only happenms if the fat content changes between two fatwrites | 14:56 |
| ogra_ | which now happens due to the kernel upgrade | 14:56 |
| ogra_ | (i'm really just guessing here ... fact is that it doesnt break if fatwrite isnt used at all) | 14:58 |
| rsalveti | right, I believe it's because of the newer kernel | 14:59 |
| ogra_ | do we know if the uboot fat driver can handle long filenames ? | 15:00 |
| elopio | pitti: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1474426 | 15:00 |
| ubottu | Launchpad bug 1474426 in systemd (Ubuntu) "systemctl status globbing does not work on version 222" [Undecided,New] | 15:00 |
| ogra_ | we also started using the longer filenames for the kernel | 15:00 |
| elopio | fgimenez: for now, I'll modify GetCurrentVersion to get a packageName. | 15:00 |
| ogra_ | (thanks to the version) | 15:01 |
| === chihchun is now known as chihchun_afk | ||
| pitti | elopio: thanks | 15:03 |
| plars | ogra_: rsalveti: sorry, just now catching up after I got off a call - yes this is what I was complaining about a long time ago when trying to get BBB to boot properly - that we just accidentally boot snappy on it because we have a working uboot on emmc, and that uboot could easily break on the sd card if you don't force it to boot from there | 15:16 |
| ogra_ | plars, yeah, but there is no way for us to enforce SD ... | 15:17 |
| ogra_ | unless we wipe it | 15:17 |
| ogra_ | well, wipe the MMC | 15:17 |
| plars | ogra_: there is one other way | 15:17 |
| mvo | ohhhh? | 15:18 |
| ogra_ | how ? | 15:18 |
| plars | ogra_: holding the s2 button down, or jumpering lcd_data02 I think, to gnd will do it | 15:18 |
| ogra_ | right | 15:18 |
| plars | it forces booting from the sd (including bootloader) | 15:18 |
| ogra_ | or solder a wire ... | 15:18 |
| plars | ogra_: well, a jumper wire is so much easier :) | 15:18 |
| ogra_ | not realy a softwareish solution :) | 15:18 |
| ogra_ | and we cant really destroy peoples MMC installs | 15:19 |
| plars | ogra_: so my current plans for instrumenting bbb include this, and use s2 along with a stable image on emmc to flash the sd, then force booting onto the sd | 15:19 |
| plars | ogra_: but if/when we get the recovery stuff, then we won't want to do that of course - and it becomes really critical that we don't break uboot | 15:19 |
| ogra_ | yeah :( | 15:20 |
| plars | ogra_: but that makes me wonder, do we really need to change the bootloader that often? except maybe at the beginning? | 15:20 |
| plars | ogra_: once it stabilizes shouldn't we be able to mostly leave it alone? | 15:20 |
| ogra_ | well, its not our bootloader that has issues ... it is the case where we boot the onboard one and use fatwrite from it | 15:21 |
| plars | if you are updating it only ever 12-16 months, it's much easier to be super careful and make sure we don't break it before landing the change | 15:21 |
| plars | ah | 15:21 |
| plars | ogra_: why is it calling fatwrite from the onboard one? | 15:22 |
| ogra_ | the uboot scripts execute it to set the a or b partition in a file on the boot partition | 15:22 |
| plars | oh, right | 15:22 |
| plars | so this is just the nature of bbb, by default it tries to load the bootloader from emmc, unless you press the s2 button after poweron | 15:23 |
| beuno | FWIW, in both my bbb | 15:24 |
| beuno | I didn't need to press anything | 15:24 |
| beuno | it boots from the SD if it's there | 15:24 |
| plars | beuno: yeah, that's accidental and the source of the problem :) | 15:24 |
| beuno | plars, this is why I don't do hardware :p | 15:25 |
| plars | beuno: I've been complaining about this for a while now, what's happening is that it's loading the bootloader off of your emmc | 15:25 |
| plars | but the bbb default image is set up to check if there's something it can load from the sd | 15:26 |
| elopio | pitti: other question, is there a way to wait for a service to be loaded and running? | 15:26 |
| pitti | elopio: another unit should just have After=foo.service | 15:27 |
| elopio | pitti: but not inside a unit. Like I install a package, and I want a command to wait until it's up before using it. | 15:27 |
| pitti | elopio: other than polling "systemctl is-active foo.service" I don't know; usually invoke-rc.d waits until it's up already | 15:28 |
| pitti | elopio: asking on #systemd (freenode) might be useful, perhaps I'm missing some kind of "wait-active" | 15:28 |
| elopio | pitti: ack, thanks. | 15:28 |
| elopio | is-active is already an improvement to what I have :) | 15:29 |
| pitti | elopio: other than that a while/is-active/sleep loop would work for now | 15:29 |
| pitti | elopio: that smells a bit weird -- you have a package whose postinst auto-starts a service, but it's not up when apt-get finishes? | 15:30 |
| plars | ogra_: if there's a known fix for this uboot bug, I'm sure rcn-ee would be open to adding it in his images, then we could point people who are running the default image on emmc to update to that? | 15:30 |
| pitti | elopio: packages are expected to do that; i. e. that sounds like a bug in that .service, or in the package | 15:30 |
| elopio | pitti: well, on snaps we don't have post-inst. | 15:30 |
| ogra_ | plars, well, the issue is to use fatwrite at all :P | 15:30 |
| plars | oh | 15:31 |
| ogra_ | plars, i'm looking at a way to not use it at all | 15:31 |
| elopio | I'm not sure if we can make something to wait on install. | 15:31 |
| plars | ok | 15:31 |
| pitti | elopio: ah, ok | 15:31 |
| pitti | elopio: but what would start the shipped units then? | 15:31 |
| pitti | elopio: whatever that ^ is, if it does "systemctl start", that's also synchronous, i. e. waits until it's started | 15:31 |
| pitti | elopio: unless you call it with --no-block | 15:32 |
| elopio | pitti: hum, so maybe that works. I'm just following this comment that says: | 15:34 |
| elopio | "FIXME: sucks, needed because "systemctl start" does not wait until the service is really started." | 15:34 |
| elopio | maybe that's not true anymore. | 15:34 |
| pitti | that has never been true | 15:34 |
| pitti | it might be true that the .sevice's Exec* commands do not wait | 15:34 |
| pitti | i. e. do whatever you need to do to wait until the started service is up | 15:35 |
| pitti | but then the polling loop doesn't help you either | 15:35 |
| pitti | or it specifies a wrong Type= (simple instead of forking, etc.) | 15:35 |
| elopio | pitti: I see. Oh, but here's another case where we do need to wait: | 15:36 |
| elopio | if ssh starts before the service starts, then the test might run before the service is running. | 15:36 |
| pitti | elopio: ah, this might be something to fix in autopkgtest | 15:37 |
| pitti | elopio: i. e. not start before systemctl list-jobs becomes empty | 15:37 |
| elopio | pitti: oh, I like that. | 15:38 |
| plars | Is there a good description somewhere of pxe booting snappy? | 15:38 |
| elopio | I see that the service has no type, which defaults to simple, which should wait until the service is up. So if it doesn't, is a package bug. | 15:47 |
| elopio | fgimenez: are you ok if I remove all this waits, and see how it goes? If it fails, we report bugs to fix the packages. | 15:48 |
| fgimenez | elopio, yes, much better. If we need to wait for something we could use goroutines and channels too | 15:50 |
| elopio | fgimenez: I was trying to use a ticker to poll every 1 second, and made a mess instead :) | 15:50 |
| elopio | I need to get better at parallelization. | 15:51 |
| fgimenez | elopio, me too :) we could encapsulate all this logic somehow so that we can reuse, we'll need it again for sure | 15:52 |
| elopio | fgimenez: btw there's this tiny branch that needs review: https://code.launchpad.net/~elopio/snappy/test_the_tests_tests/+merge/264170 | 15:53 |
| fgimenez | elopio, sure | 15:54 |
| rsalveti | lool: why are we not using the u-boot from the archive? | 15:56 |
| rsalveti | it seems we're using the one from http://people.canonical.com/~lool/snappy-bbb/ | 15:56 |
| ogra_ | lool, did your patches land upstream (so that they would be in the wily archive uboot) ? | 15:56 |
| rsalveti | sergiusens: how can I retrieve the content of the beaglebone oem snap from the store? | 15:57 |
| elopio | ok, xkcd fails without the sleep. So I suppose there's a bug on the networking-service template. | 16:03 |
| sergiusens | rsalveti: get it from te link | 16:07 |
| sergiusens | rsalveti: https://search.apps.ubuntu.com/api/v1/package/beagleblack the anondownload one | 16:07 |
| sergiusens | rsalveti: or just lp:snappy-hub and look for the snappy-systems branch (I would create a series, but I think asac is still owning the snappy-hub project). | 16:08 |
| tedg | Uhg, so I installed the Mir demo server before setting up SSH. | 16:08 |
| tedg | Is there a way to figure out the IP address KVM gave something? | 16:09 |
| rsalveti | sergiusens: cool, thanks | 16:09 |
| sergiusens | tedg: kvm only does lo by default, so you would need to redir anyways and not care about the ip | 16:09 |
| tedg | I used virt-manager so it sets up the interface. | 16:09 |
| sergiusens | tedg: I do alias kvm_snappy='kvm -m 1500 -redir :8022::22 -redir :8080::8080 -redir :4200::4200' | 16:09 |
| sergiusens | tedg: oh, then I don't know :-) | 16:10 |
| jobot | Hello, I have snappy on a beaglebone black using the image from here https://developer.ubuntu.com/en/snappy/start/#try-beaglebone. I have installed owncloud and would like it to use storage from a usb drive, but I cannot seem to mount the usb to have it writeable. Any guidance appreciated. Thanks | 16:30 |
| elopio | jobot: I could mount my usb drive on bbb. | 16:56 |
| elopio | and then I can use chown to give the ubuntu user write permissions on the mount. | 16:57 |
| elopio | what's the problem you are getting | 16:57 |
| elopio | ? | 16:57 |
| jobot | Thanks. It's when I set owncloud storage to /mountlocation , it says that it's not writable | 17:09 |
| jobot | I guess I didn't chown it. Are there any other mount settings required? | 17:10 |
| elopio | jobot: I haven't used owncloud yet, so I don't know. But I can write to my usb. | 17:14 |
| jobot | ok I will give it a go. Thanks! | 17:19 |
| === He4dShOt_ is now known as He4dShOt | ||
| * rsalveti is still trying to clone git://git.denx.de/u-boot.git | 19:25 | |
| ogra_ | did it grow so much the recent years ? | 19:28 |
| rsalveti | ogra_: no, just super slow to clone | 19:28 |
| rsalveti | 30kb/s | 19:28 |
| ogra_ | nice, feels like home :P | 19:28 |
| rsalveti | hahah | 19:28 |
| plars | someone picked up the other phone and messed up the modem? :) | 19:29 |
| plars | elopio: is https://code.launchpad.net/~snappy-dev/snappy/selftest still the right place to look for snappy selftests? | 19:54 |
| plars | elopio: or is there newer test code somewhere else? | 19:55 |
| sergiusens | plars: it's all in trunk now | 20:01 |
| plars | elopio: sergiusens: awesome, thanks, looking through the readme now... it looks like it may prompt me for a ssh password? does it try a default or can you give it one in advance to try? | 20:06 |
| sergiusens | plars: don't think so, but I can't attest to that | 20:18 |
| sergiusens | :-) | 20:18 |
| plars | sergiusens: hmm, I can't even get it to run at the moment, but it's probably some go issue | 20:18 |
| plars | it doesn't seem to work like the previous adt stuff did | 20:18 |
| sergiusens | plars: build on x86 and run on arm? maybe missing some cross compile flags there ;) | 20:19 |
| sergiusens | elopio should be able to help there | 20:19 |
| plars | sergiusens: yeah, I'll talk to him when he's back around | 20:19 |
| rsalveti | sergiusens: what removes snappy-stamp.txt? | 20:32 |
| sergiusens | rsalveti: nothing | 20:46 |
| sergiusens | rsalveti: is it being removed? | 20:47 |
| rsalveti | sergiusens: that's the file created by u-boot, just wanted to know about the logic that handles it | 20:47 |
| rsalveti | in the userspace side | 20:47 |
| rsalveti | doc says it's undone by /lib/systemd/system/ubuntu-core-snappy.service | 20:48 |
| rsalveti | which doesn't exist | 20:48 |
| rsalveti | so someone removes /boot/uboot/snappy-stamp.txt | 20:49 |
| sergiusens | rsalveti: it's not called ubuntu-core-snappy.service, that's one problem I guess | 20:49 |
| rsalveti | right :-) | 20:50 |
| sergiusens | rsalveti: there seems to be a job missing ... | 20:51 |
| rsalveti | the file gets removed | 20:51 |
| sergiusens | rsalveti: it's ubuntu-snappy.boot-ok.service | 20:51 |
| sergiusens | rsalveti: oh, nvm, it's a task most likely and systemctl doesn't list those by defualt | 20:51 |
| sergiusens | rsalveti: sudo journalctl -u ubuntu-snappy.boot-ok.service | 20:52 |
| rsalveti | sergiusens: right, so it's snappy itself that handles it | 21:03 |
| rsalveti | ExecStart=/usr/bin/snappy booted | 21:03 |
| sergiusens | rsalveti: oh, yeah, sorry, forgot that was only obvious to 3 people :-P | 21:04 |
| sergiusens | rsalveti: I fixed a bug there not long ago to avoid needless writes if you may recall | 21:04 |
| rsalveti | sergiusens: yeah, I wonder if the problem is because the file size is 0 for the stamp file | 21:04 |
| rsalveti | trying to write some data on it now | 21:05 |
| sergiusens | rsalveti: it should never be zero | 21:05 |
| rsalveti | not a lot of changes from upstream at the fat code | 21:05 |
| sergiusens | rsalveti: oh, the stamp file, not snappy-system.txt | 21:05 |
| sergiusens | got it | 21:05 |
| rsalveti | fatwrite mmc ${mmcdev}:${mmcpart} 0x0 ${snappy_stamp} 0 | 21:05 |
| rsalveti | yeah, stamp | 21:05 |
| sergiusens | rsalveti: snappy booted does not erase that iirc, it's the bootloader; this way we can rollback if the kernel is corrupted | 21:06 |
| sergiusens | as an example | 21:06 |
| sergiusens | rsalveti: nvm, http://bazaar.launchpad.net/~snappy-dev/snappy/snappy/view/head:/partition/bootloader_uboot.go#L203 | 21:07 |
| sergiusens | rsalveti: I forgot how this all works! lool you around? | 21:07 |
| rsalveti | yeah, saw that | 21:07 |
| rsalveti | sergiusens: didn't break this time with a file != 0 | 21:13 |
| rsalveti | let me build a new image with your fix so I can also test the rollback | 21:13 |
| sergiusens | rsalveti: interesting, so if uEnv.txt fatwrote this with something > 0 it would be fine? | 21:22 |
| rsalveti | sergiusens: that is what I'm trying to confirm | 21:23 |
| rsalveti | sergiusens: elopio: give this a try, change /boot/uboot/snappy-system.txt | 21:24 |
| rsalveti | with image 3 | 21:24 |
| rsalveti | change "fatwrite mmc ${mmcdev}:${mmcpart} 0x0 ${snappy_stamp} 0" to "fatwrite mmc ${mmcdev}:${mmcpart} ${loadaddr} ${snappy_stamp} 10" | 21:25 |
| rsalveti | then do the update | 21:25 |
| sergiusens | rsalveti: k | 21:27 |
| sergiusens | rsalveti: will take a shortcut and format system-b | 21:28 |
| === howefield_afk is now known as howefield | ||
| * sergiusens waits | 21:32 | |
| rsalveti | yeah, tried twice and wasn't able to get the same corruption | 21:38 |
| rsalveti | hm, ubuntu-device-flash is the one creating snappy-system.txt | 21:46 |
| sergiusens | rsalveti: yes it is | 22:23 |
| sergiusens | rsalveti: it's not updatable | 22:24 |
| rsalveti | sergiusens: yeah, guess we can fix this all when we split the updater | 22:30 |
| rsalveti | migrating 124 to alpha | 22:31 |
| rsalveti | argh, there is one importer running | 22:31 |
| sergiusens | rsalveti: I can add a fix for u-d-f to at least have new images work correctly | 22:47 |
| rsalveti | sergiusens: I'm waiting the next alpha to show up and will do a bit more testing | 22:48 |
| rsalveti | but got a local change already for that | 22:48 |
| rsalveti | importer takes forever lately | 22:48 |
| rsalveti | wget https://public.apps.ubuntu.com/anon/download/canonical/beagleblack.canonical/beagleblack.canonical_1.8_all.snap | 22:56 |
| rsalveti | HTTP request sent, awaiting response... 500 INTERNAL SERVER ERROR | 22:56 |
| rsalveti | beuno: any issue with the store? | 22:56 |
| rsalveti | yeah, unable to create a beagle image | 22:58 |
| rsalveti | sigh | 22:58 |
| sergiusens | rsalveti: it downloads fine here | 23:18 |
| rsalveti | sergiusens: yeah, it's working again | 23:18 |
| beuno | rsalveti, looking at outages, for future reference, noodles is usually up and running at this hour | 23:22 |
| beuno | or just ask in #u1-internal | 23:22 |
| rsalveti | alright, thanks | 23:22 |
| rsalveti | was giving this error for about 10 minutes for me | 23:22 |
| beuno | rsalveti, looks like a storm in the internal cloud | 23:24 |
| beuno | being managed by IS | 23:25 |
| rsalveti | alright :-) | 23:25 |
| rsalveti | sergiusens: elopio: alpha 5 is now available | 23:30 |
| beuno | (winds are picking up, might get a bit bumpier before it gets better) | 23:33 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!