/srv/irclogs.ubuntu.com/2016/05/17/#ubuntu-devel.txt

=== salem_ is now known as _salem
=== yuning-afk is now known as yuning
pittiGood morning05:00
pittihallyn: "systemctl status" says "bad", looks like this unit wasn't actually installed, or has some error? Alias= are translated into symlinks at install time05:02
pittistgraber: the libnl3 package caused many network-manager failures, so it got pulled after a quick discussion in #u-r; Adam later replaced trusty-updates with a reversion05:03
pittistgraber: right, that wouldn't have helped users who already upgraded, only those who didn't yet05:03
infinitypitti: Should never delete without a revert.  The revert allows those who upgraded but didn't reboot to have a higher chance of being fixed before things go south.05:09
infinitypitti: Anyhow, thanks to stgraber's script freaking out, I went on the warpath to fix it all.05:09
pittiinfinity: thanks! understood for next time (I ran out of time on Fri); i. e. I'll just leave it to some US folks right away next time05:10
infinitypitti: Yeah, acting quickly was good, but if you couldn't revert, nick highlighting (or given the severity, phoning) someone who can DTRT would help.05:11
infinitypitti: I suspect a whole lot of grandparents with broken networks phoned their grandkids over the weekend.05:11
infinitypitti: But hopefully it's mostly settled out by now. :/05:11
sarnoldpitti: sorry to bother you directly, could you take a loot at my systemd timers for my archive sync? http://paste.ubuntu.com/16471760/05:37
sarnoldpitti: the goal is to run the rsync, and once that's finished, run the zfs snapshots, then wait four hours before repeating05:38
infinitysarnold: Is it coincidence or intentional that 4h is the delay we use to trigger non-Canonical mirrors?05:38
infinitysarnold: (The followup question being "why don't you ask IS to trigger you?")05:39
sarnoldinfinity: I had the impression from IS that they only wanted to push-sync Big Mirrors and wanted everyone else to periodically pull05:39
infinityPerhaps so, yes.05:39
sarnoldinfinity: the 4hours was slightly informed by the "six times a day" mention I saw somewhere.. mostly I wanted lots of snapshots to work with in a hurry, and considered switching to five or six hours between syncs later on..05:40
infinitysarnold: Well, 4h is a good cadence if you're pulling from a tier2 mirror, since that's when we trigger them. :)05:41
infinityIf you're pulling from archive.ubuntu.com, all bets are off, they update constantly.05:41
sarnoldpulling from archive.ubuntu.com, it's remarkably speedy over rsync :)05:42
infinityAnyhow, I have no opinions on your units.  I still live in a mindset where replacing cron was silly.05:42
sarnoldI even found the 'fast' IP address and did terrible terrible things with /etc/hosts that I'm not proud of.05:42
infinityWhy mangle hosts?05:42
sarnoldbecause one IP was five to ten times fsater than the others for me05:43
infinityrsync rsync://ip.add.re.ss::foo?05:43
infinityI mean, why bother pinning down the hostname instead of just using the IP.05:43
infinityrsync doesn't vhost.05:43
sarnoldhmm. good point. :)05:43
sarnoldthanks05:43
cpaelzergood morning05:44
sarnoldre: replacing cron, believe me, I'm half tempted to return this to cron. but getting the one to run after the other finished, under two different user accounts, seemed like a worthwhile enough feature to try out05:44
sarnoldmorning cpaelzer :)05:44
infinitysarnold: Alternately, it might be that what you're looking for is "us.archive.ubuntu.com" :P05:44
infinityFor reasons I've never fully understood, archive.u.c is a round-robin of gb and us.05:45
infinitySo its usefulness fluctuates based on time of day and phase of moon.05:45
sarnoldinfinity: even within the us hosts one was way faster than the other05:46
infinitysarnold: Curious.05:46
infinitysarnold: IS might want to know about that.  Pretty sure they sit on the same VLAN.  Probably the same switch on the same rack even.  If one sucks, that's odd.05:46
sarnoldespecially since I saw roughly no difference between mtr results between the two. the fast one was 85 ms away, the slow one 80 ms away, and stddev was <5 ms on both05:46
sarnoldinfinity: they are actually on different routers and different uplinks05:47
pittisarnold: the main issue is that you don't want to use a .target for those05:47
pittisarnold: as the .target is never stopped (there is no such thing as an "oneshot target", this doesn't make sense)05:48
pittisarnold: so once it's running, the timer won't do anything any more05:48
sarnoldpitti: ah :)05:48
infinitysarnold: Hrm.  Not from me, they aren't.05:48
sarnoldinfinity: hmm.05:48
pittisarnold: so, drop the .target, have the .timer trigger the rsync unit, and have that Requires/Before= the snapshot unit05:48
pittisarnold: and as both are oneshot, they will be restarted on the next timer05:49
pittisarnold: also, you don't need to add Before= to one and After= to the other, once is enough05:49
sarnoldinfinity: 91.189.91.23 was the fastest -- 91.189.91.26 was the slowest.05:49
sarnoldpitti: excellent, good to know, I thought it seemed inelegant for Before/After to be anything except duals of each other, but that forum post was so sure.. (I should have known. :)05:50
infinitysarnold: Very weird indeed.  I have identical routes to both.  Under severe load, economy would probably be faster due to being a slightly larger machine, but I'm not sure they have enough outbound bandwidth to actually cripple them.05:51
* infinity shrugs.05:51
sarnoldinfinity: .23 would routinely give me 4-8 MBps, .26 would routinely give me 120KBps.05:52
infinitysarnold: Ouch.05:52
infinitysarnold: If that's reproducible, I suspect IS would definitely like to know.05:52
sarnold..especially since I wanted to see ~50MBps from these things :) heh05:53
* pitti needs to disappear for a few hours for a long appointment05:53
sarnoldpitti: thanks for the help05:53
infinitypitti: Have fun storming the castle.05:54
ginggsmorning! freecad doesn't want to migrate because freecad-doc (arch:all) was dropped. would someone decruft please?06:25
infinityginggs: Done.07:03
ginggsinfinity: thanks!07:04
sil2100cjwatson: hey! We had the 503 again during two image builds - is there a way to re-try those builds somehow?08:53
sil2100cjwatson: since essentially the livefs builds succeeded, would be a waste of time to rebuild those from scratch08:53
sil2100cjwatson: also, it's a bit worrying that it happens so frequently recently, in the past I don't remember it happening even once08:54
cjwatsonsil2100: as cdimage@nusakan, run the usual build command (you can get it from cdimage/etc/crontab) but without --live09:13
sil2100Oh, ok09:13
sil2100Good to know, thanks!09:13
=== Guest11953 is now known as alvesadrian
=== alvesadrian is now known as adrian
brendandwhy would it be the case that lxdbr0 isn't visible in ifconfig, after i've run lxd init?10:17
brendandi have of course specified to set up networking10:17
=== hikiko is now known as hikiko|ln
juliankinfinity: Coverage report for APT https://apt.alioth.debian.org/coverage/index.html :D10:27
pittibrendand: what does "systemctl status lxd-bridge.service" say?10:31
pittibrendand: note that both lxd.service itself and also the bridge are only started on demand (via socket activation)10:32
pittibrendand: i. e. you won't actually see the bridge up until you try to start your first container10:32
=== yuning is now known as yuning-afk
brendandpitti, hi btw!10:36
pittihey brendand, how are you?10:36
brendandpitti, i'm good. you know i'm back on the inside, right?10:37
pittibrendand: yeah, I read it on the ML; welcome back!10:37
brendandok, didn't realise it only starts with the containers10:37
pittibrendand: you didn't hold up on the outside for very long :)10:37
brendandpitti, yep joining the same club as such luminaries as xnox and mvo :)10:38
* pitti waves to lamont to :)10:38
brendandpitti, no, unfortunately it didn't last long at all - or fortunately maybe!10:38
brendandpitti, oh lamont too, heh - he's on my team10:38
Laneythe outside world is that bad eh?10:39
brendandLaney, yeah it's a dark and scary place10:40
brendandpitti, so it looks like: http://paste.ubuntu.com/16473414/10:42
brendandgreen is good10:43
brendandbut 'failed to setup lxd-bridge' seems bad10:43
pittiright, so lxd-bridge.start should not succeed in this case10:44
brendandi feel like i did something very bad to my lxd. even launch doesn't work10:48
farbluehi all :) Maybe not the right channel to ask but I'm working with fan networking and wondered if anyone knew why the docs suggest docker should be configured with iptables=false when using fan networking11:12
xnoxbrendand, so you are the new Yo-Yo =) as elmo called me the other day.11:17
=== hikiko|ln is now known as hikiko
farblueanyone here know about fan networking?11:35
sladenfarblue: sorry know.  A possible reason for disabling iptables is that iptables is already being used to perform the static-nat that FAN-network requries11:37
sladenfarblue: however, I would stress that I don't know, and it's not something I've ever used11:38
farblueok, that makes some sense. I’m struggling to work out what the problem is but while my containers can talk to each other and the host can talk to them, they don’t seem able to talk to the outside world :(11:39
sladenfarblue: isn't that the point :)11:39
farbluewell, no, not really. I don’t want anything other than the docker hosts to be able to talk to the containers but it is quite useful for containers to be able to talk out to other services11:41
sladenkirkland: you are quoted on the PR for the fan-networking/OpenStack stuff.  If you're around, would you be able to follow-up with farblue11:41
* sladen reads with amusement "There are several class A network addresses reserved for Class E". No, there are several /8s reserved for Class E.11:43
farblueI think half the problem is that docker does all this ‘magic’ stuff under the hood and I strongly suspect turning off iptables in the docker config does more than expected11:43
sladenfarblue: nb. kirkland is normally on US time, so likely to be around later rather than now11:45
farblueok, I’ll keep working on the problem and ask again later :)11:46
sladenfarblue: yes, but please stay on IRC!11:48
farbluesure :)11:48
farblueif iptables management is disabled in docker then it doesn’t create the iptables FORWARD rules or create the DOCKER chain or the DOCKER-ISOLATION chain and it doesn’t add the PREROUTING rules - which I suspect is rather more than I need disabled11:49
=== Guest48104 is now known as adrian
=== _salem is now known as salem_
=== alexabreu is now known as alex-abreu
=== fginther` is now known as fginther
barryLaney: hi.  have you had a chance to look at the libpeas PPA?13:35
Laneyhi barry, not yet13:36
LaneyI'll let you know13:36
Laneyinside CSS currently13:36
barryLaney: cool, no worries13:36
barryand thanks!13:36
=== tinoco is now known as tinocoff
rharperinfinity: pitti: so for the nm/libnl cluster;  where did I mess up in the process?  in the libnl bug, once we found out the nm break happened in -proposed, I built an update (attached v3 of the debdiff to the bug) and requested a sponser;  I can't upload myself.13:49
rbasakrharper: I think it was adding verification-done. This implies that it's OK to copy the current version in -proposed to -updates.13:51
rharperrbasak: hrm, yes;  I was pretty sure that we tested proposed; that we had unbroken proposed13:52
rharperwhich I thought meant that v3 was in proposed already13:52
rharperthat is, I had to create the v3 with Breaks to *unbreak* anyone testing proposed;13:53
rharperand ISTR that someone helped unbreak proposed by uploading the  newer version;  but maybe that didn't happen in which case changing the tag wasn't correct;13:54
rbasakI understand how it went wrong - lots of moving pieces, especially when complicated by the extra coordination needed by sponsorship.13:54
rbasakI'd say that marking verification-done needs double checking of everything, including what version is where and that everything known is accounted for (in this case the Breaks, together with everything else).13:55
rbasakI wonder if it would be better to mark verification-failed and start again in this sort of case, to reduce the chance of error.13:56
rharperrbasak: so, where would I check to ensure the right versions are in place?  pulling the Packages version from the the archive ?13:57
rharperrbasak: as in start a new bug?  wading back into those it's quite a lot to take in after one has paged it out of memory context13:57
rbasakrharper: rmadison, or https://launchpad.net/ubuntu/+source/libnl3 etc. That should tell you what version of what is where, and sometimes Launchpad has useful diffs to show you to save you downloading and checking.13:58
rharperrbasak: cool13:58
rbasakrharper: not necessarily starting a new bug, but marking verification-failed, having the SRU team delete from -proposed, and re-uploading new fixes or something. That's what I was thinking. I don't know if this is a good idea or not in general, but I feel that it might have reduced the probability of error in this particular case.13:58
rharperah13:59
pittirharper: I think that should have come up during testing, and indeed I considered teh v-done as "good enough" (when mass-processing SRUs I don't/can't check every detail again)14:02
=== alvesadrian is now known as adrian
pittirharper: not easy to put a finger on a particular point in the process, it was sort of an emergent corner case between sponsoring, verification, and not prodding about removing a known-bad proposed version14:03
pittirharper: i. e. the half-done version should have been removed from the queue, or marked as v-failed after accepting into -proposed, I think14:03
rharperpitti: yeah; I think not being able to make those changes myself makes things harder since a sponsor has to grok all of what happened as well;14:04
rbasakrharper: thank you for asking the question BTW. respect++.14:05
pittirharper: everyone can mark as v-failed, and everyone needs to prod the archive admins about dropping an upload from the queue14:05
pittirbasak: indeed, echoing rbasak; thanks for thinking about the post-mortem14:06
rharperrbasak: only way to learn =)  also; I felt really bad since it happened while I was out; everyone here was putting out a fire I started.  =(14:06
hallynpitti: could the 'systemctl status' -> 'bad' error be related to bug 1579922 ?14:06
ubottubug 1579922 in init-system-helpers (Ubuntu) "dh_systemd_enable fails due to 'preset' when service file is renamed" [Undecided,New] https://launchpad.net/bugs/157992214:06
pittihallyn: yes, very plausibly14:07
pittihallyn: I did see teh report and your call for feedback; didn't get to that yet, sorry14:07
hallynnp.  i assume it has to do with the manual work i do in libvirt-bin.preinst, i suspect i should move another file or something.14:09
rbasakpitti: it still feels wrong that there's only one person standing in the way of failure here though. Do we need more staffing on the SRU team so closer reviews are possible before accepting v-done?14:09
hallyni didn't get around to reproducing with a set of empty dummy packages14:09
pittirbasak: not really realistic, I think14:09
pittisecond-guessing v-done would mean a wholly new second v-really-done, and that's pointless as it doesn't add anything new14:10
pittirbasak: in this case, more testing results from "real users" would have helped14:10
rbasakI'm thinking something like "<random> v-done; I checked X and Y"; "<SRU> <reviews what should be checked> what about Z?"14:11
pittirbasak: and we run the SRU process under the assumption that some folks run -proposed and report regressions14:11
pittiwhich apparently didn't happen despite this being in -proposed for like half a year14:11
rbasakSo trusting results from v-done, but confirming that the right things were checked.14:11
rbasakIn this case we all knew what was needed, so I don't think additional testing would have helped.14:11
rbasakWhat I think we needed was clarity over what we already knew.14:12
rbasak(between us)14:12
pittirbasak: to me as an SRU process guy it wasn't at all obvious from this bug that this broke NM14:12
pittiit was supposed to fix NM :)14:12
rbasakWell, it fixed libnl3 but exposed a bug in NM.14:12
rbasakpitti: how about a tag to say "this is complex, needs two full reviewers please?"14:13
rbasakMost SRUs are trivial compared to this one.14:13
pittiyeah, SRUs depending on each other are rare14:14
pittiwhich is probably why it's easy to forget to add Depends:/Breaks:14:14
rbasakDoesn't even need SRU team. Just another uploader.14:14
pittiand given enough luck (installing NM first), even ten testers would have confirmed it works14:15
pittithe root cause was to introduce a change that depended on a change in NM without saying so14:15
rharperpitti: we didn't know it broke NM until we uploaded to -proposed; mostly because the testing done was on server (with no NM);14:15
pittiand that most probably wasn't even known initially14:15
pittiexactly14:15
rbasakAnother issue is that after the initial problem was revealed, we told testers that we knew about the issue.14:15
pittiat that point we should have marked this as v-failed14:16
pittior ask an AA to pull the package14:16
rbasakAfter that test results become muddled because we don't know if the tester accounted for the original regression proposed or not. They might fail the new version but not report because they think we already know because of the failure in the initial version.14:16
rbasakYeah. I guess one simple thing to do is to delete from -proposed immediately if we know that it isn't suitable to land in -updates.14:17
rbasakAnd if that isn't immediate (eg. SRU team member online to immediately help with a replacement), then v-failed.14:17
pittithat or tagging v-failed, which can be done by everyone14:18
juliankinfinity: pitti (?): bdmurray (?): Looking for endorsements on https://wiki.ubuntu.com/JulianAndresKlode/DeveloperApplication-PPU (or could I rather apply for core-dev?).14:29
juliankmvo is not here, which is somewhat unfortunate14:31
* juliank contributes in somewhat unusual ways, so there is not that much evidence of what he does (mostly just told mvo to syncpackage something)14:33
seb128could somebody from the SRU team look at unity-control-center / unity-settings-daemon in the xenial queue? they are small patches and things the oem team is wanting14:35
juliankAnyone thought about AppArmor profiles for APT's http and https methods? I'm currently implementing seccomp for the former, but I thought it might make sense to also get some AppArmor safety (not an expert on that, though)14:48
juliankI should probably also chroot() the method into the partial dir14:51
rbasakjuliank: that sounds like a good idea. AIUI those run in separate processes so should be fairly easy to do?14:51
juliankrbasak: I think the problem is that it's not clear where they are fetching to. APT has CWD, /var/lib/apt/lists/partial and /var/cache/apt/archives/partial14:52
juliankBut some other programs also use them to fetch stuff into other dirs14:52
rbasakI see.14:53
juliankAlso we have proxy scripts we need to run :/14:53
juliankBut we could at least use AppArmor to protect against writes to /usr and friends14:53
rbasakThat would certainly be a useful start.14:55
juliankAlthough: They currently cannot do that anyway (under normal circumstances) since they run as a separate user14:57
naccrbasak: still around?15:33
coreycbarges, hi, regarding the neutron-* stable release for xenial bug 1580674.15:57
ubottubug 1580674 in neutron-openvswitch (Juju Charms Collection) "[SRU] mitaka point releases" [Undecided,New] https://launchpad.net/bugs/158067415:57
coreycbxenial and yakkety are currently at 8.0.0-0ubuntu115:58
coreycbupstream will release the first beta of 9.0.0 for newton in two weeks which we'll then upload to yakkety15:59
coreycbara, can we get an exception to upload the stable release to xenial before that?15:59
coreycbarges, ^  (sorry ara)15:59
aracoreycb, no worries :)15:59
infinityrharper: Absolutely what went wrong is that it should have been marked v-failed, not v-done.16:19
infinityrharper: It would have flipped back to v-needed when the new version was uploaded (which never happened).16:19
bdmurraypitti: wrt bug 1446537, would it be possible to tag bugs apport-hook-error that have an attachment named "HookError.*" so it is easier to find issues like this?16:34
ubottubug 1446537 in apport (Ubuntu) "apport hook fails in add_info with TypeError: 'str' does not support the buffer interface" [High,Fix committed] https://launchpad.net/bugs/144653716:34
=== tinocoff is now known as tinoco
farblueI’m still working with the fan networking and it seems to me that a docker container on the fan network doesn’t have a route out of the fan unless you enable iptables management in docker16:48
farbluecould it be because the fan network is overlay’d on top of a physical ethernet interface that doesn’t actually have a route out anywhere?16:51
farblueanyhow, home time for me now.16:51
argescoreycb: i'm not sure what exception you mean. the version in xenial can't be >> than the version in yakkety17:04
coreycbarges, ok I wasn't sure if there were any exceptions to that rule.  we can probably upload the xenial version to yakkety for the next 2 weeks.17:10
kirklandfarblue: hi -- you're interested in fan networking?17:36
coreycbarges, can you reject the mitaka neutron* uploads?  there are 4 packages for 8.1.0-0ubuntu1.18:07
coreycbarges, these are for xenial18:07
argescoreycb: ok18:09
argesdone18:10
coreycbarges, thx18:10
bdmurraypitti: I ended up pushing a change to the ubuntu hook to the yakkety branch of apport re apport-hook-error18:39
=== jgrimm is now known as jgrimm-afk
sarnoldpitti: thanks for your help yesterday, my timer ran twice without trouble since then :)18:42
rharperinfinity: ok; thanks19:05
=== jgrimm-afk is now known as jgrimm
shadeslayerchrisccoulson: hi, I was wondering, https://launchpadlibrarian.net/259046127/firefox_46.0.1+build1-0ubuntu0.16.04.1_46.0.1+build1-0ubuntu0.16.04.2.diff.gz seems to have extra things that are not present in https://code.launchpad.net/~mozillateam/firefox/firefox.xenial20:03
shadeslayercould you please update the branch? :)20:04
shadeslayerchrisccoulson: specifically : https://bazaar.launchpad.net/~mozillateam/firefox/firefox.precise/revision/107020:05
karstensrageare there any backporters that can help with  #1562434 and #156183721:25
karstensrageive tested them up the wazoo with ppa's and vms21:26
karstensragethey are a library and a pam module so im not sure how to propagate that information21:26
karstensrageand its already in xenial, tested and working21:27
karstensragei will help anyone with configuring PAM in some way and credentials are required that can be provided as well21:28
karstensrageor is there someone that would consider testing so that the backporter doesnt have to just take my word for it?21:54
karstensragedebfx, you possibly, youre the newest approved backporter?22:01
karstensrageLaney? broder?22:03
=== salem_ is now known as _salem
sarnoldbueller?22:03
karstensrageit sure feels like that22:04
Unit193Tempting to try getting a backport instead of SRU, because the former seems easier and more likely. >_>22:05
karstensrageSRU?22:07
sarnoldstable release update22:07
Unit193karstensrage: Different matter at hand, but st..dang.22:07
sarnoldthe process for updating existing packages in published releases22:07
karstensragegetting it in xenial proposed was easy22:07
karstensragegetting a backport is proving frustratingly impossible22:08
Unit193karstensrage: FWIW, part of that is limited number of backporters too, iirc.22:08
karstensrageyeah ive heard it all22:08
Unit193LP 1582713, LP 1562358.22:09
ubottuLaunchpad bug 1582713 in python-googleapi (Ubuntu) "1.5.0 xenial update needed due to oauth2client changes" [Undecided,New] https://launchpad.net/bugs/158271322:09
ubottuLaunchpad bug 1562358 in python-googleapi (Ubuntu Xenial) "python-googleapi is incompatible with oauth2client >= 2.x" [Undecided,New] https://launchpad.net/bugs/156235822:09
mapreriI wonder, is a SRU acceptable just to fix an appstream issue (including bigger icons)?  my pet package is not showing up in that gnome-software thinghy allegedly for this...22:16
sarnoldmapreri: it looks sort of like the infrastructure should be there, there's icons-* appstream packages in the -updates, -security, and -proposed pockets22:20
sarnolds/packages/blobs/22:20
mapreriwell, that's what I'd expect from an archive that supports DEP-11, I'd be surprised if those files weren't there.22:21
=== JanC is now known as Guest42191
=== JanC_ is now known as JanC

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!