/srv/irclogs.ubuntu.com/2014/03/27/#ubuntu-devel.txt

ScottKdoko: Are you going to upload the rebuilds for extensions to drop 3.3 support?00:30
=== freeflying__ is now known as freeflying
=== BenC- is now known as BenC
=== _salem is now known as salem_
=== salem_ is now known as _salem
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
msxhello everyone :) I've enjoying 14.04-desktop_amd64 since almost a month now and so far it's shaping as an amazing release - rock solid, fast... really beautiful. However I've found this strange issue I'm unable to debug, may be you can shed some light here guys: whenever I return from susped, there will be a sustained load average of +1. On normal conditions like a fresh boot, the usual load when idle03:00
msxcould be as low as 0.01 (kudos!!!) but after waking from suspend you won't be seeing the load average drop 1.10~1.2003:00
msxAny idea? How could I debug this issue?03:00
msxs/i've/i'v been/g :)03:02
TheMusomsx: Have you looked at what is eating your resources in system monitor, or using top in a terminal?03:41
TheMusomsx: No actual idea what it could be, but knowing what process is using resources is somewhere to start looking at least.03:41
msxTheMuso: hey!03:43
msxTheMuso: well, actually, top doesn't shows anything unusual; in fact the consume is as about as normal as always BUT the loadaverage is always from 1.10 upwards03:45
msxTheMuso: if the load average would start with a 0.x everthing would be perfectly normal03:45
msxi'm very intrigued about what could be happening ^_^03:46
=== vibhav is now known as Guest48466
pittiGood morning06:11
pittislangasek: wooooow, https://code.launchpad.net/~xnox/phablet-tools/py2-3/+merge/205608 landed at last \o/06:20
pittibdmurray, ev: errors now running on prodstack> niice!06:34
=== achernya_ is now known as achernya
=== tvoss|afk is now known as tvoss
pittitvoss: guten Morgen07:27
=== jamesh_ is now known as jamesh
dholbachgood morning07:29
* pitti hugs dholbach07:29
* dholbach hugs pitti back07:29
=== qengho is now known as CardinalFang
=== CardinalFang is now known as qengho
zygagood morning08:50
pittiev, zyga, fginther: I've been looking for an ack timeout in rabbitmq -- i. e. that a message gets re-queued if the accepting worker doesn't ack it within a given time; I only found some references that this isn't possible, but they are from 200909:01
pittizyga: sorry, I meant vila09:02
pittiev, vila, fginther: do you know if that's possible?09:02
vilapitti: so far, we've managed to always ack (outcome reported whether it succeeded or failed)09:03
vilapitti: as fginther hinted, we want to dig that far more09:03
pittivila: so this wouldn't cover the case if the test process gets stuck, but stays alive09:03
pittivila: if the worker node dies properly, the AMQP connection will be terminated and the message requeued, so that's fine09:04
pittibut if a test runs indefinitely long, it won't help09:04
evI'm certain it's possible (otherwise why have ack'ed messages at all?). I'll dig in a little bit.09:04
vilapitti: but adt-run will timeout in that case no ?09:04
evOh I misread.09:04
pittivila: usually yes; just playing the "what could possibly go wrong" game :)09:05
vilapitti: the testbed and the worker are distinct instances09:05
vilapitti: yeah, right ;)09:05
vilapitti: planned to be digged to the bottom ;) Feel free to vanguard, I won't be far ;)09:05
pittivila: I've seen an LXC host kernel-panic and stop doing anything for heavily loaded container tests09:06
pittiand I'm not sure whether that already was sufficiently dead to drop network connections09:06
pittithere's a per-queue and per-message TTL, but that will just kill messages after no worker grabs it for that time, so that's the opposite of what we want09:07
vilapitti: LXC host meaning the host handling the container running adt-run right ?09:08
pittiright09:08
pittivila: I guess we could implement this manually; the controller would traverse the list of pending (unacked) requests regularly (assuming that this is accessible), and re-queue the ones which haven't been acked after 6 hours09:09
vilapitti: so, this design has indeed a fatal failure mode, we avoid that by using (today) a nova instance for the testbed, (future) a MAAS instance for bare metal09:09
vilapitti: but in any case, the worker should not be subject to such a failure mode in production (it's ok for local use as it's faster)09:09
pittivila: well, "should" -- it happened; kernel bugs are everywhere :)09:10
pitti(or hw)09:10
vilapitti: yes, that's why the worker (controlling the testbed) should not be on the same host09:10
pittivila: hm, I think I disagree09:11
vilapitti: so it can monitor the testbed in all scenarios09:11
pittiit introduces more things that can go wrong, and doesn't really solve the problem09:11
pittias now you need the same kind of sync between controller and worker, AND worker and testbed09:11
pittitwice the number of hosts which can fail, and twice the amount of synchronization/acks to do09:12
vilapitti: err, it removes one thing that can go wrong: a user test crashing the infra09:12
pittivila: we'll run tests in QEMU, that's not the problem09:12
pittivila: the problem was that the host kernel, or the host's hard disk freaked out on high loads (from LXC, but I don't think that matters that much)09:13
vilapitti: right, QEMU is not lxc, that's what a nova instance provides09:13
vilapitti: so we separate root causes between worker and testbed and can handle them differently09:14
pittivila: I'll see whether it's possible for the controller to get/check/kill unacked requests09:14
vilapitti: so the worker can fail, the message is not acked, another worker takes the ticket09:14
pittivila: you mean it does get acked, but with a failure result?09:14
vilapitti: yes09:15
pittivila: because "not acked → another worker takes ticket" is exactly what I'm looking for :)09:15
vilapitti: I haven't written tests for that yet, but that's a strong assumption in our design, if it's not guaranteed, we'll have to re-design ;)09:16
pittivila: which assumption do you mean? relying on workers to always eventually acking requests?09:16
vilapitti: I've been assured it was a safe assumption though ;)09:16
pitti(or dying properly with closing net connections)09:16
pittivila: well, maybe it is09:17
vilapitti: either the worker ack the message or the message becomes available in the queue after some time09:17
pittivila: I'm not sure how much active "pinging" rabbitmq does to the accepting worker to determine whether its connection is still alive09:17
vilapitti: yup, that's exactly the point that needs to be tested09:17
pittivila: ok; it seems you didn't run into this kind of trouble so far, so perhaps this can be put on the backburner for now?09:18
* pitti tries what happens when SIGSTOPping the worker09:18
vilapitti: that's what we've done, I'm not super comfortable with that but it's on my radar09:18
vilapitti: but yes, we didn't run into that case (yet)09:19
pittivila: ok; thanks for your help! (and sorry for keeping bothering you)09:19
vilapitti: not bothering at all, happy to share the knowledge there as we acquire it ;)09:19
pittiit seems I have a habit that the very first problems I'm thinking of in these new systems are the ones which are impossible or underdocumented :)09:19
vilapitti: same here, which led to a reputation of being over concerned ;)09:20
vilauntil things break ;)09:20
pittivila: so if I sigstop a worker (so that it definitively can't serve its network connection), the rabbit server still thinks it's active09:29
vilapitti: and to make things clear: the actual design is under construction, so more eyeballs help09:29
vilapitti: right, so there should be some way[s] to configure that, hopefully on a per-queue basis09:30
pittionly if I control-C it it puts the message back into the queue09:30
vilapitti: meeting starting right now09:31
pittivila: ack09:31
vilapitti: oh.. interesting, does that mean SIGSTOP must be handled or we miss some safety net already provided in other cases ?09:31
pittivila: no, I was just simulating what happens if the worker suddenly stops responding to net requeusts without closing its network connection properly09:32
pittivila: i. e. the equivalent of what could happen on a kernel panic and processes end up in deep kernel sleep, etc.09:32
pittivila, ev: ah, http://www.rabbitmq.com/tutorials/tutorial-two-python.html officially documents the absence of timeouts09:35
cjwatsonseb128: thanks - I need to test it but I've stuck a patch derived from that one in my working tree so I don't forget about it09:36
pittiso if that turns out to be an actual problem, we can regularly iterate over these queues and kill the workers; but that can (and has to) be bolted on top of the system anyway, so IMHO no need to do this in phase 109:36
seb128cjwatson, thanks09:36
=== oSoMoN_ is now known as oSoMoN
pittivila, ev: but hard-switching off (lxc-stop -k) the worker DTRT, so I think that covers the most common problems09:40
jamespagedoko_, figured out that eclipse-* failures - gnumail osgi metadata was not quite right - need a drop of javax.activiation09:42
jamespagefix uploaded09:42
vilapitti: from your url above: 'RabbitMQ will redeliver the message only when the worker connection dies.' so it should be a matter of waiting for the connection to die right ? That can delay nacks, but they will occur no ?09:43
vilapitti: at worst, a worker can lose the connection and fail to ack and the same message will be handled twice, but I think handling duplicates will be easier than losing messages09:44
=== funkyHat_ is now known as funkyHat
zygahey, quick question, what is the syntax for debian/control, that says package depends on the same version of another binary package from the same source package10:10
zygalike libfoo and foo that need to be updated together10:10
zygaI recall something like ${Source:vesrsion}10:10
zygabut I cannot find any docs about that10:10
zyga${binary:Version}10:11
zyga?10:11
cjwatsonzyga: package-name (= ${binary:Version})10:11
cjwatsonzyga: deb-substvars(5)10:11
=== funkyHat_ is now known as funkyHat
zygacjwatson: thanks10:11
cjwatsonThere are subtleties in the event that one of the binaries is arch-dependent and the other is arch-independent10:12
pittivila: yes, duplicates isn't a problem in the adt case10:22
vilapitti: thanks, I had troubles here, I may have missed some msgs10:25
pittivila: no you didn't miss anything; I was AFK for a bit10:25
vilapitti: did you reply to: 'from your url above: 'RabbitMQ will redeliver the message only when the worker connection dies.' so it should be a matter of waiting for the connection to die right ?' ?10:26
pittivila: right10:26
vilapitti: i.e. you mentioned a test you did, could it be that the connection wasn't yet dead ?10:26
pittivila: it logically wasn't10:26
pittivila: i. e. it never got a RST, but it wasn't really alive either as the worker was stopped10:27
pittivila: which might approximate what happens if a system freezes due to a hardware failure or kernel panic or similar10:27
vilapitti: but a TCP connection should ultimately die (what's the timeout there, 20 mins or something ?), right ?10:27
pittivila: but as I said, we can handle that using nagios and regular ping queues, and just killing hanging workers10:28
pittivila: I don't know, I'm afraid10:28
pittivila: I just waited a few minutes, certainly not 2010:28
vilapitti: ack10:28
vilapitti: right, one other assumption we did early on, was that rabbit itself is considered reliable so if a msg is received, it won't be lost10:29
pittivila: so, perhaps this is a non-issue as the connection will eventually die; it seems rabbit is being used by quite a number of people, if that was a real problem I'd think that there was a common solution by now10:29
* vila nods10:29
pittivila: so either way, I think we have the tools to deal with this problem, I was just curious whether I coudl do something like [...], ack_timeout='4 hours'10:30
vilapitti: a consequence of the above is that we focus on making sure msgs are properly queued/dequeued and don't have to handle inter components failure modes otherwise10:30
vilapitti: yeah :-/10:30
vilapitti: but you mentioned TTL at one point ?10:31
pittivila: *nod*, I assume that part is okay; I was playing around quite heavily with randomly killing multiple workers etc., and I never lost one10:31
pittivila: yes, that's the other way around: if a message isn't being picked up by TTL, it gets deleted10:31
vilaha, hmm, the opposite of what I'd like ;)10:31
pittivila: so we certainly want to know when that happens (-> again, housekeeping cron job), but not kill messages, but fix the AWOL workers instead10:32
pittiso I don't think we want to use that feature10:32
vilapitti: agreed10:32
vilapitti: oh, you mean it's opt-in ?10:32
pittivila: yes: http://www.rabbitmq.com/ttl.html10:32
vilaexcellent10:33
pittivila: which is certainly handy for some use cases (just not our's here)10:33
vilapitti: yup10:33
* vila go reads those pages and stop annoying pitti with silly questions10:33
pittivila: no no, that's fine :) this is all new to me, too10:34
vilapitti: ha, right, one more idea comes back: we can have a different queue where workers send heartbeats so they can be killed if they fail to do so, not sure it's worth it at that point but that's one way to monitor them10:35
pittivila: right, I proposed that yesterday (to fginther, I believe)10:36
pittiah no, to ev10:36
vilapitti: hehe GMTA ;)10:36
pittivila: anyway, we can do that, or regularly check pending and unack'ed queue entries in rabbit, or similar10:36
vilaor rather, good ideas spread, spontaneously ;)10:36
pittivila: we can have a fanout queue, and every worker has to ack in 30 seconds; the set of expected workers comes from the results in swift10:37
pittivila: that was roughly my proposal10:37
vilapitti: right, I tend to prefer separate means for separate goals, but... real life decides10:37
evpitti, vila: we must not be the first people to face this problem11:13
evI wonder if there's decent prior work11:13
evthough the current approach seems sound11:14
evpitti, vila: perhaps we already covered this, but could the workers not just open('/srv/tr_worker/var/heartbeat.stamp', 'w').close() ? Then the watchdog could just check timestamps to know if it needs to kill the daemon?11:19
evwhatever we do, we should definitely have a test for os.kill (worker_pid, signal.SIGSTOP)11:20
pittiev: the watchdog sshs in? If it can ssh, then we shouldnt' have the problem of "hung kernel" in the first place, but yes11:20
pittiev: if that's any easier than checking unack'ed messages in the queue11:21
vilaev: what pitti said, we need to cope with unreachable workers11:21
evpitti: no need. Just put the watchdog on every system11:21
pittiev: the watchdog will be affected by crashing hw/kernel just as the adt worker, though11:21
vilaev: that's the same as sending a heartbeat so someone still have to monitor the heartbeats from a *different* instance or we're back to square one11:22
evwe can have a separate check for "do you respond to ssh? No. Okay, `nova delete` to you."11:22
pittibut of course, if the watchdog/ssh don't respond, that's already a "died" condition11:22
pitti*nod*11:22
evthen juju automatically respawns the worker. In the case of the test runner. Not sure how to do this with lxc containers11:22
evbut I'm assuming there's something comparable11:22
evor you could use juju to manage them11:22
pittiyes11:22
evsince it does that now11:22
pittijuju-local can do that11:23
evwhoop11:23
pittior just lxc-stop / lxc-start again11:23
evcan I just say that I'm so excited for this? :)11:23
vilayeah, <vm tech> start/stop11:23
pittiexcited> +111:23
=== oSoMoN_ is now known as oSoMoN
pittiev, vila: FYI, I'm working on a spec here: https://wiki.debian.org/MartinPitt/DistributedDebCI11:40
pittiev, vila: once it's ready, I'll send mail for review/commenting11:40
vilapitti: ack11:40
=== marcoceppi is now known as marco_traveling
=== _salem is now known as salem_
=== MacSlow is now known as MacSlow|lunch
rbasakcjwatson: any opinion on openssh 6.6? It's "primarily a bugfix release" but it seems quite late now. I just triaged bug 1298280.12:45
ubottubug 1298280 in openssh (Ubuntu) "Update OpenSSH to 6.6" [Wishlist,Triaged] https://launchpad.net/bugs/129828012:45
cjwatsonrbasak: I already have it staged and plan to land it12:45
rbasakAh - thanks!12:45
cjwatsonwas talking with Marc about it a few days back12:46
=== roadmr_afk is now known as roadmr
=== MacSlow|lunch is now known as MacSlow
smoserslangasek, https://bugs.launchpad.net/ubuntu/+source/plymouth/+bug/1160079 ? that going to get in ?13:48
ubottuLaunchpad bug 1160079 in plymouth (Ubuntu) "plymouth aborts in cloud images" [Medium,Confirmed]13:48
=== roadmr_afk is now known as roadmr
=== roadmr is now known as roadmr_afk
=== roadmr_afk is now known as roadmr
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
=== doko_ is now known as doko
=== mbiebl_ is now known as mbiebl
rbasakmterry: please could you review https://bugs.launchpad.net/ubuntu/+source/juju-quickstart/+bug/1273865/comments/19, which answers your question on this MIR bug?15:59
ubottuLaunchpad bug 1273865 in websocket-client (Ubuntu Trusty) "[MIR] juju-quickstart, python-jujuclient, urwid, websocket-client" [High,New]15:59
rbasakmterry: we have a FFe bug waiting too - if we can get this acked, is this sufficient for main?15:59
cjwatsonrbasak: openssh> btw you can always feel free to test stuff in git://anonscm.debian.org/pkg-ssh/openssh.git :-)15:59
rbasakcjwatson: don't look at me. I just triaged the bug! :)16:00
cjwatsonif you feel the urge16:00
cjwatsonok :)16:00
cjwatsonit's always worth a try16:00
rbasak:)16:00
mterryrbasak, yeah both MIR and FFe would be sufficient for main.  I'm looking at that bug16:07
knoctedoes William Hua hang out here?16:16
rbasakmterry: fyi, there was some concurrent chat about this in #juju just now. jamespage will update the bug.16:17
jamespagemterry, rbasak: commented on bug16:20
Chipacajcastro: hi there. were you aware the askubuntu login thing (via our sso) isn't working?16:21
jcastrono16:22
Laneyknocte: he is attente on this network, but email might be better to reach him16:22
Chipacajcastro: (as an aside, can't you change it to use sso and not lp?)16:22
knocteLaney: I'm trying here because email didn't work..16:22
jcastroChipaca, we've asked them to but never put a priority on it16:23
bdmurraypitti: could we get the dbg syms updated for lightdm?16:23
Laneywell, you have the nick16:23
Chipacajcastro: fair enough; maybe if it's broken they'll fix it and change it at the same time :)16:23
Laneytry a PM16:23
seb128knocte, what do you want from him? maybe others can help you?16:24
bdmurraypitti: for version 1.9.13-0ubuntu116:24
knocteseb128: a hint on where to start looking to try to fix https://bugs.launchpad.net/bugs/128660516:24
ubottuLaunchpad bug 1286605 in unity-gtk-module (Ubuntu) "Unity global menu causes handlers of the "activate" signal of Gtk.Action to be emptied" [Medium,Triaged]16:24
jcastroChipaca, http://meta.askubuntu.com/questions/2837/moving-to-ubuntu-single-sign-on16:24
jcastrocan you bump that and then I can ping someone16:24
=== shadeslayer is now known as shadeslayer_
=== shadeslayer_ is now known as shadeslayer
Chipacajcastro: maybe what's happening is that it's under way right now? (and instead of removing the button they disabled it?)16:25
jcastrono clue16:26
Chipacajcastro: I can't bump that, no16:26
Chipacajcastro: because I can't log in16:26
seb128knocte, k, yeah you want to talk to attente ... he's in utc+10 tz atm though, so he's probably sleeping atm16:26
jcastrooh, ok, I can try to hunt someone down16:26
knocteok thanks seb16:26
jcastroChipaca, lp login works for me16:27
Chipacajcastro: when i try to add a comment there is no lp button, so it does look like it's ongoing16:27
jcastroI just tried it16:27
infinitypitti: How do you feel about the state of psql SRUs?  Are they ready to go today, despite being a tiny bit early?16:27
Chipacajcastro: i'm trying chrome and firefox, no luck in either16:27
jcastroChipaca, when you click the LP button it takes you to the text field, where you have to type in your name and then click log in16:27
Chipacajcastro: the button does nothing16:27
Chipacajcastro: there is no text field16:27
argesstgraber: hi. I'm going through your fixes for ifupdown and seeing if they apply to precise. Do you think backporting bug 1072518's fix to precise seems reasonable?16:38
ubottubug 1072518 in ifupdown (Ubuntu) "Restart networking crashes dbus and the desktop manager" [Critical,Fix released] https://launchpad.net/bugs/107251816:38
argesjust the preventing ability to stop/restart networking iteractively16:39
stgraberarges: hmm, as much as I'd like to see this gone, it'd be a behavior change that some people may (as wrong as it is), rely on16:41
argesstgraber: that was my concern as well...16:41
argesstgraber: i'll leave that one out then. and just ensure that documentation gets updated16:42
infinityIt's pretty shocking the number of people who think "restart networking" is the way to reconfigure interfaces.  I wish I knew where this came from originally.16:42
stgraberinfinity: at least that will simply fail in 14.04, I'm sure we'll get some more bug reports because of that, but at least it won't trash their system anymore...16:43
argeshttps://help.ubuntu.com/12.04/serverguide/network-configuration.html  suggests 'sudo /etc/init.d/networking restart' hmm16:46
argesthat's fixed in the 14.04 guide16:47
infinitystgraber: I already have one bug report because of it, yes.16:47
=== bfiller is now known as bfiller_afk
infinitystgraber: Not sure if the error lacks appropriate verbosity, or if the user in question was just silly (I'm assuming the latter).16:48
infinityarges: Argh.16:48
stgraberinfinity: we can't print something on screen when they hit it, so upstart will report a failed to stop and the actual reason is in upstart's log file at /var/log/upstart/networking.log16:48
stgraberstgraber@castiana:~/Desktop/rcu$ sudo restart networking16:49
stgraberrestart: Job failed to restart16:49
stgraberstgraber@castiana:~/Desktop/rcu$ sudo tail /var/log/upstart/networking.log16:49
stgraberStopping or restarting the networking job is not supported.16:49
stgraberUse ifdown & ifup to reconfigure desired interface.16:49
infinitystgraber: Yeah, that's pretty opaque.16:49
infinitystgraber: Is there really not a sane way to echo to the controlling tty?  Having *everything* in logs isn't always the best default.16:50
* arges files a bug against serverguide...16:50
slangasekwell, a) don't call restart directly, use 'sudo service networking restart'16:51
infinityslangasek: That latter is still wrong...16:51
infinity(In the cases when it works, it's by accident, not design)16:52
slangasekinfinity: right, which is b) someone could patch service to tail the log16:52
infinitycjwatson: Pretty sure I've praised it at least once before in the last few months, but live-installer on a fast storage device, whee.  2-second base installs.16:54
infinitycjwatson: Don't know if it's ever been pondered, but why not swap base-installer for live-installer in netboot as well, and just publish base.squashfs?16:56
infinity(Oh, I guess that would mean publishing it to the package mirror, not the CD mirror, that would be a problem)16:56
infinitySo, nevermind that.16:56
alowinfinity: how did the node stuff go on top of libv8?16:57
infinityalow: Fine, I just needed to add one patch to your set.16:57
infinityalow: http://paste.ubuntu.com/7163741/ <-- without that, my ppc64el systems tried to configure as ppc (ie: 32-bit), which didn't work out so well.16:58
seb128bdmurray, hey, do you have access  to extra datas for e.u.c reports than the ones in the UI?16:59
seb128bdmurray, e.g https://errors.ubuntu.com/problem/85c403c4a05cd32a48a73b226340850faa45e78516:59
bdmurrayinfinity: could you have a look at the patch in bug 1296386?16:59
ubottubug 1296386 in casper (Ubuntu) "[PATCH] Remove 23etc_modules script" [Undecided,New] https://launchpad.net/bugs/129638616:59
seb128bdmurray, is there any way to know why retracing fails, or see if those users are running ppas or get access to a dump of a report?16:59
* dholbach hugs hggdh16:59
cjwatsoninfinity: yeah, mirroring would be the problem16:59
* hggdh hugs dholbach :-)17:00
tarpmandholbach: hggdh++17:00
bdmurrayseb128: I'll have a look at the retracer logs for any of those examples.17:00
infinitybdmurray: Yes, but not on beta day.  Want to keep it on your list of things to nag about? :P17:00
seb128bdmurray, thanks17:00
alowinfinity: Thanks - I'll add that to our code17:00
bdmurrayinfinity: what makes you think I have such a list?17:01
infinitybdmurray: A hunch.17:01
slangasekbdmurray: I think he said you should assign it to him17:01
bdmurrayslangasek: and keep it on the nag list. ;-)17:01
bdmurrayseb128: outdated debug symbol package for liblttng-ust0: package version 2.4.0~rc4-1ubuntu2 dbgsym version 2.1.1-217:04
seb128bdmurray, that's not likely the problem though17:04
infinityalow: It looks like Debian's moved on from .25 to .26.. Do you know if that brings any dangers with it (ABI transition, API breaks, etc)?17:04
bdmurrayseb128: well that's why it failed to retrace17:04
infinityalow: Considering it for us, but only if it's really low risk, since we're close to release.17:05
seb128bdmurray, can you see from the log what qtubuntu-android version is installed?17:05
bdmurrayseb128: its not listed in Dependencies.txt so probably not17:06
seb128bdmurray, I though that apport had smartness to resolve the packages containing the files listed in the stacktrace?17:06
seb128or in the procmap rather17:07
alowinfinity: I don't see a big risk moving to 0.10.26 - let me go look at the deltas17:07
bdmurrayseb128: yes it does, but it just picks the latest package version in the pocket (-updates, etc) when trying to retrace17:07
seb128bdmurray, don't we have also some magic to see when a ppa was in use to warn about it?17:07
bdmurrayseb128: yeah, that'd show up in Tags as 'third-party-packages'17:08
bdmurrayseb128: well but maybe only for dependencies and not stuff in ProcMaps17:08
seb128bdmurray, ok, I guess we are just without a solution for that one then17:09
seb128bdmurray, thanks for checking/helping!17:09
infinityDoes someone upstarty want to tell me why minutes after I've booted, sysvinit jobs seem to run again?17:10
infinityslangasek: ^?17:10
infinityslangasek: http://paste.ubuntu.com/7163794/ <-- Note the last few lines.17:11
infinityslangasek: I see this on my buildds too, and it does genuinely seem to be starting again, not for the first time, as on the buildds, I get apache erroring because it's already running, etc.17:11
slangasekinfinity: the ondemand init script is special17:12
slangasekinfinity: and doesn't point to anything running again17:12
infinityslangasek: The ondemand thing should be a red herring here, it execs itself in the background.  Unless that's what's causing the rest of this.17:12
slangasekinfinity: it shouldn't be causing any of the rest of it; but you said "note the last few lines", maybe you want to be more specific? :)17:12
slangasekyou mean the apparmor profiles stuff?17:12
infinityslangasek: When I said "last few lines", I mean "hey look, I'm booted and logged in and, oh neat, a couple of minutes later, I'm starting apparmor and restoring resolver state".17:13
infinityslangasek: And, like I said, on my buildds, I see this with apache and launchpad-buildd too.17:13
infinityFor a long time, I thought it was an oddity with booting without an initrd, so didn't care, but this example is a standard install with an initrd.17:13
slangasekinfinity: well here's a question, why the heck is something like "dns-clean" running in runlevel 2 instead of in rcS?17:17
infinityslangasek: Dunno, but apparmor *is* in rcS, so that's still weird.17:17
alowinfinity: RE: 0.10.25 -> 0.10.26  http://blog.nodejs.org/2014/02/18/node-v0-10-26-stable/ Mostly it's a bunch of bug fixes. Possibly more changes than you'd like to see, but no API level changes that I'm aware of.17:17
slangasekinfinity: but I don't know what to make of those messages, anyway; I'm not aware of ever having seen this, and it's even weirder that you're getting messages from /etc/rcS.d/S??apparmor + /etc/rc2.d/S??dns-clean17:17
mterryjamespage, regarding juju-quickstart -- there would be value in having it in main as an installer for the universe packages, eh?  I'm not sure what the MIR policy on such installer packages is -- we let ubuntuone do that.  But I'm asking the other MIR team members17:18
alowinfinity: IMHO - being 'current' has quite a bit of value.17:18
infinityslangasek: I imagine it's (re-)running all of rcS and rc2, those are just the only two with output.17:18
infinityslangasek: But haven't ever had the time to try to dissect it.17:18
slangasekinfinity: I could imagine blaming it on a late network interface start, but dns-clean appears to only be wired up for ppp interfaces17:19
infinityslangasek: And, again, wouldn't explain the apparmor start.17:19
slangasekinfinity: the output from those init scripts is using the standard lsb interfaces; it makes no sense at all that they would be the only two with output17:19
infinityslangasek: Meh.  Well, I'll have to look into it again when it's not today.17:20
slangasekinfinity: fwiw the fact that you also have an 'ntpdate' process in your ps output also points me this way17:21
infinityslangasek: But there's something weird going on.  Like I said, on my buildds, where it seems clear it's double-starting some things (because the late sysvinit messages on the console also whine about failing due to ports in use), something's up...17:21
slangasekinfinity: so if you want further debugging, please try to reproduce when booted with --verbose, then capture the upstart-induced dmesgry17:22
infinityslangasek: Right.  Will do $later.17:23
slangasekbut I'm 70% sure that what you're seeing is deferred activation triggered by the network interface coming up17:23
slangasekI just don't see the path it's taking17:23
infinityslangasek: Does that explanation explain the late apparmor?17:23
infinityslangasek: Do we defer all of sysvinity until post-network?17:23
infinitysysvinit, too.17:23
slangasekinfinity: in the sense that there are per-network-interface apparmor hooks (/etc/init/network-interface-security.conf), maybe?  But I don't see where that triggers the init script17:24
slangasekinfinity: and yes, rc2 waits for static network configuration17:24
slangasekso that explains late running of dns-clean (and apache), but not double-running17:25
infinityslangasek: I'll dig deeper later and see if I can come up with something more coherent than months of confused memories.17:31
jdstrandinfinity: curious-- is apparmor starting twice or just very late? jjohansen has seen it start late and, for example, not load profiles (eg firefox) until much later17:34
jdstrandinfinity: which of course means that his firefox is running unconfined17:34
jdstrandwe have a medium term solution for pregenerating apparmor cache in kernel postinst, and know how to do it, which would then allow us to not only improve boot performance after install/first boot with new kernel, but allows us to do an upstart job very early since it won't slow boot17:36
infinityjdstrand: Without more debugging, hard to say for sure if it's twice or late.17:36
jdstrand(or systemd unit, doesn't matter)17:37
infinityjdstrand: But if pregeneration and an upstart job is something you think you can safely land at this point in the cycle, that sounds like a better method anyway.17:37
jdstrandinfinity: is this a server or desktop?17:37
infinityjdstrand: These are all servers.17:37
jdstrandinfinity: we cannot17:37
jdstrandthis is 14.10 at this point unfortunately17:37
=== DrKranz is now known as DktrKranz
infinityAhh, well.  A man can dream.17:37
jdstrandinfinity: so, you could quickly login and do a tcpdump that continues to run17:38
mdeslaurinfinity: please delay the LTS until 14.10. thanks.17:38
jdstrandinfinity: then login to another console and do 'sudo aa-status'17:38
infinityjdstrand: Right, I might ask for debugging tips some day that isn't today. :)17:38
jdstrandand look for 'X processes are unconfined but have a profile defined'17:38
jdstrandinfinity: ack17:38
jdstrandwe've always known there was a race17:38
infinityCertainly, if the first start is that delayed, the idea that a bunch of stuff might be unconfined (like, half my system?) seems a bit less than ideal.17:39
jdstrandbut in practice, people never really hit it17:39
jdstrandwe still don't have widespread bug reports17:39
jdstrandwell, any actually17:39
jdstrandit has just been observed on occasion17:39
jdstrandinfinity: on a server it is less of a concern cause the profile load happens in the upstart job17:40
jdstrandbefore the exec17:40
jdstrandthe whole generate profile cache in postinst is possibly SRUable btw17:41
infinityjdstrand: So, if it's happening in an upstart job, does that mean the console spew from the sysv job is basically redundant and meaningless?17:41
jdstrandI think we'll have most of the kernel bits-- beyond that it is mostly kernel packaging and an apparmor SRU17:42
jdstrandinfinity: no-- the policy load happens in multiple stages. the network-security job makes sure dhclient is handled. upstart jobs are modified to load policy before the exec, like with mysql. then there is everything else that doesn't have an upstart job, like tcpdump17:43
jdstrandit is the last stage that is the most problematic17:43
jdstrandcause that happens via sysvinit which is racy17:44
jdstrandwe could move that to an upstart job now, but people would probably get fired17:44
jdstrand:)17:44
jdstrand"you increased boot time by how long!?!"17:45
jdstrandthe good news is that we have a plan, we know what to do, and it is scheduled work for later17:45
jdstrandthat is quite a bit better than even 3 months ago17:46
jdstrand(we discussed it at our team sprint recently)17:46
infinityjdstrand: I like the precompiling notion.  People notice postinsts sucking a bit less than they notice boot time.17:47
sbeattieinfinity, jdstrand: it really looks like from infinity's paste that it's starting late, not running twice; note the profile loads im dmesg are from /etc/init/network-interface-security.conf, not from the sysvinit apparmor script17:47
infinityjdstrand: I suspect we all remember the good ol' days when we used to always run "depmod" on boot too...17:48
jdstrandinfinity: (also, the fact that an upstart job loads the mysql policy only to have the sysv script do it later is redundant-- but in that case the cache is present for at least on of them, so it is as fast as the file can be read from the disk)17:48
jdstrandinfinity: re sucking> absolutely17:48
jdstrandsbeattie: oh, you are right :\17:49
infinity(Of course, from my weird POV, I notice postinst sucking more than boot time, because I upgrade constantly and never reboot, but I'm preeeeetty sure that's not normal)17:49
jdstrandsomething in trusty is delaying the apparmor sysv run...17:50
jdstrand(sometimes)17:50
jdstrandlike I said, we might need to do an SRU17:50
infinityjdstrand: Could it be that slangasek's assertion about network interfaces only delaying rc2 is wrong, and it in fact delays all of sysvinit?  That would pretty much explain that paste.17:50
infinity(Since I'm hung up on ntpdate)17:51
jdstrand(that said, if someone could figure outwhy it is late and fix it, that would be excellent. /me can't reproduce)17:51
slangasekno, it could not17:51
jdstrandI unfortunately am not an expert with upstart job interactions17:51
infinityslangasek: Kay.  Then back to the weird conclusion and the "not debugging today" thing.17:51
infinityjdstrand: Anyhow, I see this on tons of machines and can pretty much reproduce at will, so I'll get together with you guys to see if we can unwind it a bit later.17:52
slangasekoh, wait17:52
* slangasek scratches his head17:52
jdstrandinfinity: that is good to know17:52
infinity(That was a fresh beta2 install from only moments earlier, so nothing weird there at all)17:52
slangasekok, so /etc/init.d/rcS is run from the bottom of /etc/init/rc-sysinit.conf, which does key on static network up17:53
slangasekI thought we had better interleaving of /etc/rcS.d with the rest of the system17:53
slangasekjdstrand: so yeah, any late apparmoring that you're seeing is actually related to us having made /etc/rc2.d non-racy, without noticing that this also holds up /etc/rcS.d :/17:53
slangasekproper fix would be to split out /etc/rcS.d handling, and make it wait for the local filesystem but not the network17:54
infinityslangasek: So, curious followup.  Why do I have a console?17:54
jdstrandslangasek: oh! it is great to know the cause17:54
infinityslangasek: My hvc0 job is "start on stopped rc RUNLEVEL=[2345]", is that basically a complete lie, given how we're running rcS/rc2?17:55
=== shadeslayer_ is now known as shadeslayer
jdstrandslangasek: since I use network-manager, the fact that I don't see it locally is consistent with your understanding of the issue, correct?17:56
slangasekjdstrand: correct17:56
jdstrandI never saw it in a vm cause I almost always do desktop installs in them17:57
slangasekinfinity: ummmm ok, yeah, I don't know why you would have a console with output from rc2.d showing up then17:57
slangasekinfinity: because 'stopped rc RUNLEVEL=[2345]' should mean the console starts only after all of rc2.d is done running17:58
infinityslangasek: That's what I would think, yes.17:58
jdstrandslangasek: so, is there something that can be done for 14.04 for this? (like I said, we can't to the cache pregeneration/upstart job change before 14.10/SRU)17:58
infinityslangasek: And really, while delayed boots are annoying (and they are), the larger complaint is actually that these late starts scribble all over the getty, making it confusingly non-obvious that a login prompt has happened. :P17:59
=== marco_traveling is now known as marcoceppi
slangasekjdstrand: file a bug on upstart, cut'n'paste my comment above about the fix, and we'll see if we can pull it off sanely before 14.0417:59
infinityslangasek: I saw the same with the ppc64el cloud images too, but assumed cloud-init was just doing something silly.17:59
infinity(Which it might be, but..)17:59
jdstrandslangasek: ack, thanks! shall I tag it special somehow or assign it to someone or is the bug enough?18:00
tester56_Is there I way I can rebuild a package without installing build-deps on my system? means I trigger something like  dpkg-buildpackage, and it looks for the build-deps, installs them to a temporary directory, compiles everything, moves the final deb package to a destination folder and removes the temporary directory (if specified)?18:02
tester56_*a way18:03
tester56_damn ... how is that ... I am still here :-)18:04
jdstrandslangasek: fyi, bug #129853918:05
ubottubug 1298539 in upstart (Ubuntu) "apparmor rcS.d sysv initscript is running too late" [Undecided,New] https://launchpad.net/bugs/129853918:05
tarpmantester56_: you're looking for pbuilder or sbuild18:05
tarpmantester56_: alternatively, upload it to a ppa and let launchpad do the work18:06
slangasekjdstrand: ta18:06
tester56_tarpman: thanks, what would you recommend pbuild?18:06
tarpmantester56_: pardon?18:07
tester56_tarpman: sorry, I wanted to ask: which of the two options would you recommend, pbuilder or sbuild18:07
tester56_tarpman: judging from the ubuntuwiki it seems pbuilder is better documented18:08
tarpmantester56_: no strong opinion. personally I use sbuild; so do the buildds. others think pbuilder is better for the personal case18:08
tester56_tarpman: both use chroot?18:08
tarpmanyes18:09
tester56_tarpman: do they install something like ubuntu-minimal?18:09
tarpmantester56_: sbuild does debootstrap --variant=buildd, which is even slightly smaller than that. not sure about pbuilder18:09
tester56_tarpman: thanks I will look into both18:13
tarpmantester56_: cheers18:13
=== bfiller_afk is now known as bfiller
jamespagemterry, ack - I've expressed what I think the position is - but I'm happy to leave the decision to the MIR team18:25
mterryjamespage, I think MIR team is happy to consider the two juju MIRs as coupled18:25
mterryi.e. like not for 14.0418:25
dokoLogan_, https://jenkins.qa.ubuntu.com/view/Trusty/view/AutoPkgTest/job/trusty-adt-pycxx/19:22
jtaylorfwiw I can't reproduce neither ci.debian19:29
jtaylordoes gccs [atu]san stuff make sense for shared libraries?19:30
jtaylorbecause libtool strips it ...19:30
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
argeshallyn: hey looking at your upload for libvirt in saucy. mdeslaur mentioned that there may be a security update to libvirt soon. can this fix wait to be re-based on top of the security update?20:08
mdeslaurarges: it's going to take me a while before I can push out the security update, might as well do the -proposed cycle first20:08
argesmdeslaur: ok that's good to know20:09
argesbdmurray: ^^^20:09
bdmurraymdeslaur: okay, thanks20:10
Logan_doko: wassup?20:38
=== e^0 is now known as ffio
dokoLogan_, you did sync the package, now the autopkg test is failing.. are you aware of this?21:01
NoskcajWhere can i see the jenkins test failures? I Asked for a sync of pycxx which is aparently not testing properly21:04
jtaylorNoskcaj: https://jenkins.qa.ubuntu.com/view/Trusty/view/AutoPkgTest/job/trusty-adt-pycxx/21:12
Noskcajthanks21:15
LaneyNoskcaj: In general linked from http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html21:15
NoskcajDoes the failure mean anything to you guys?21:17
NoskcajI have to go to school now, will be back later21:18
jtaylorNoskcaj: I can't reproduce it, neither can ci.debian.net21:27
LaneyI can21:27
jtaylorI guess we shuld sync cython to trusty21:30
jtayloroh it is nevermind21:30
jtaylorpts out of date21:30
Laneypython3-cxx-dev has broken symlinks in /usr/share/doc/python3-cxx-dev/examples/21:30
jtaylorhm upgrade issue?21:31
jtaylorlooks fine for me21:31
Laneyinstalled in a chroot21:31
darkxstgjs i386 tests failed all of a sudden, and I dont have enough internet atm to look into it...21:33
jtayloroO21:35
jtaylorwere are those symlinks coming from21:35
jtaylormy chroot built package does not have them21:36
infinityjtaylor: cython is synced, but FTBFS on ppc64el.  And doko keeps ignoring the tests.  Grr. :P21:36
jtaylormaybe the numpy ppcel64 patch does not work21:37
Laneyjtaylor: huh, they're coming from pkgstripfiles21:37
jtaylorLaney: why is that creating broken symlinks?21:37
infinityIt really shouldn't.21:37
infinityIt's pretty careful about what it links and how.21:38
jtaylorwhy is pycxx in main anyhow?21:39
jtaylorits not exactly a much used package21:39
jtaylorthe few users it has just embed it21:39
jtaylorwhat does pkgstripfiles do? no manpage :(21:40
LaneyMany things, one of which is making symlinks of identical /usr/share/doc/ files where a dependency exists and they're built from the same package in order to save space21:40
infinityLaney: Those links should never dangle, though.21:43
LaneyNo, if they end up doing so it's a bug21:43
infinityLaney: It's very careful to only link to packages that the package depends on.21:43
infinityOh, I wonder if one of these has a replaces on the other or something and it's causing havoc.21:46
infinity    install|upgrade)21:47
infinity        test ! -L /usr/share/doc/python3-cxx-dev || rm /usr/share/doc/python3-cxx-dev21:47
infinity    ;;21:47
infinityWhee.21:48
jtaylorhm whats wrong there?21:50
LaneyNah, it's this:21:51
Laney+ ln -s ../python-cxx-dev/examples/setup.py /home/laney/temp/pycxx-6.2.5/debian/python3-cxx-dev/usr/share/doc/python3-cxx-dev/examples/setup.py21:51
infinityLaney: There's nothing wrong with that.21:51
infinityLaney: If python-cxx-dev ships that file.21:51
infinityWhich it claims to: /usr/share/doc/python-cxx-dev/examples/setup.py21:51
infinityIt's in the dpkg file list, but it ain't on disk. :P21:52
Laney(desktop-trusty-amd64)root@iota:/home/laney/temp/pycxx-6.2.5# readlink -m  /usr/share/doc/python3-cxx-dev/examples/test_example.py21:52
Laney/usr/share/doc/python3-cxx-dev/python-cxx-dev/examples/test_example.py21:52
infinityOh!21:52
infinityI was missing the lack of extra ../21:52
LaneyYeah21:52
infinityOkay, yeah, that's a straight up bug, but... How?21:53
LaneyThis is quite a surprising bug!21:53
infinityMaybe that was never written to grok multiple levels of doc dirs.21:53
infinityCause, other than massive HTML and docs and examples, there usually aren't multple levels, and who would ship those twice? :)21:53
infinity-ln -s "../$dep/$f" "$thisfile";21:59
infinity+ln -s "$depfile" "$thisfile";21:59
infinityLaney: ^-- Betting that would fix it.21:59
infinityOh.21:59
infinityNo.21:59
infinityThat would link to the build tree.21:59
infinityWhee.21:59
infinityBut how can this work at all if $f is wrong?21:59
=== thegatta_ is now known as thegattaca
LaneyI guess you need to count the number of components in $f and go up that many levels22:01
infinityOh, right, $f is right.  I didn't read your output well enough.22:01
infinitySo it's just that, yes.22:01
infinityJust add another ../ per count of / in $f22:02
* infinity tries this bash-only without forking..22:03
Laneygood luck with that ...22:10
Laneylooping over the string doesn't count22:10
infinityLaney: Something like this (untested): http://paste.ubuntu.com/7165305/22:14
infinityErr.22:15
infinitys/prefix/f/ on that first line.22:15
infinity                slashes="${f//[^\/]}"22:16
infinityLaney: If you have a build tree available, a test would be cool. :)22:16
* infinity spins up a chroot.22:17
Laneylrwxrwxrwx root/root         0 2014-03-27 22:17 ./usr/share/doc/python3-cxx-dev/examples/setup.py -> ../../python-cxx-dev/examples/setup.py22:18
infinityThat looks more correct.22:18
infinityLaney: The rest of the doc dir still look sane?22:20
infinityLaney: If so, I shall upload that ugly looking mess.22:20
LaneyMy way was to remove the slashes and then subtract the lengths22:20
LaneyLemme run the test and see if it passes now22:20
LaneyNo more broken symlinks in there22:21
infinityLaney: Oh, was your paste with mine or yours?22:21
LaneyYours22:21
infinityKay, cool.22:21
LaneyI was playing in a separate script22:21
infinityUploading, then.  Once I give it a quick test.22:21
infinityThis probably should have a testsuite entry.22:22
=== salem_ is now known as _salem
infinityAnd, somehow, I don't care about the test right now...22:23
* infinity testbuilds and uploads.22:24
LaneyThe pycxx test needs to cp -L22:25
LaneyOther than that it works22:25
Laneyjtaylor: ^-- is that something you'd want to have in Debian?22:26
infinityDon't see why it would hurt to have it in Debian.22:26
infinityIt's a no-op on real files anyway.22:26
LaneyIt was a polite way of asking him to kindly include it there. :)22:27
infinityHeh.22:27
jtaylorsure I can add that22:27
infinityI'm a bit shocked no one's ever noticed this problem before.22:27
jtaylorisn't -L the default of cp?22:28
hallynbdmurray: i messed up the saucy libvirt update, failed to build - pushed a new one just now which fixed that.  sorry.22:28
infinityAnd wonder if it's worth a scan of a lintian lab for broken doc links.22:28
Laneyjtaylor: not with -r22:31
jtaylorright22:31
infinityLaney: Fix is in the queue.22:32
LaneyLies22:33
LaneyOkay, now it is22:33
LaneyDone22:35
LaneyUploaded pycxx, but I'd wait until after pkgbinarymangler is published to accept it22:39
infinityIndeed.22:40
* Laney goes away to play games22:41
jtaylordoko: there is a new numpy to merge, as you did the last its probably faster if you do it directly instead of me doing the sponsoring dance22:45
infinityjtaylor: Your pycxx was autoaccepted...22:48
infinityOr someone else accepted it.22:48
infinityErr.22:48
infinityLaney: ^22:48
infinityjtaylor: Not you. :P22:48
infinityjtaylor: But I guess if you upload the -L thing to Debian, we can just sync over Laney's upload.22:48
jtayloryes I'll do so tomorrow22:48
Laneyinfinity: ho hum, should have mentioned it in #-release23:19
infinityLaney: Yeah, no big deal.23:20
infinityLaney: I was going to just reupload it to get it off the list.23:20
LaneyKay23:20
LaneyNoskcaj: ^^^ your problem turned out to be quite fun (would have appreciated some initial investigation from your side though)23:24
=== infinity changed the topic of #ubuntu-devel to: Trusty Final Beta released! | Archive: Gated Review | Devel of Ubuntu (not support or app devel) | build failures -> http://qa.ubuntuwire.com/ftbfs/ | #ubuntu for support and discussion of lucid -> saucy | #ubuntu-app-devel for app development on Ubuntu http://wiki.ubuntu.com/UbuntuDevelopment | See #ubuntu-bugs for http://bit.ly/lv8soi | Patch Pilots:
=== mhall119 is now known as mhall119|offline

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!