/srv/irclogs.ubuntu.com/2016/09/12/#juju-dev.txt

wallyworldveebers: do you know if bug 1621576 is in progress?00:03
mupBug #1621576: get-set-unset config will be renamed <juju-ci-tools:Triaged> <https://launchpad.net/bugs/1621576>00:03
veeberswallyworld: it looks like curtis landed the fix in the tests for that. Seems he didn't update the bug. I'll confirm with him that it's finished (but it looks like it is)00:09
wallyworldveebers: awesome. the reason for asking is that we will be looking to land the code changes to juju to use the new command syntax00:09
wallyworldand without the ci script changes, ci will break00:09
veeberswallyworld: ack, I'll have confirmation for you tomorrow :-) But I'm pretty sure that fix is complete00:15
wallyworldty00:15
wallyworldthe juju pr needs a little fixing, so that won't be ready till when the US comes back online anyway00:16
mupBug #1560487 changed: local provider fails to create lxc container from template <canonical-is> <local-provider> <juju-core:Won't Fix> <juju-core 1.25:Triaged by alexis-bruemmer> <OPNFV:New> <https://launchpad.net/bugs/1560487>00:45
menn0axw: easy one: http://reviews.vapour.ws/r/5646/01:56
axwmenn0: LGTM01:57
menn0axw: thanks01:59
perrito666Nites, anyone happens to know dimitern's mobile?02:50
menn0wallyworld: another migration cleanup: http://reviews.vapour.ws/r/5648/03:06
wallyworldok03:07
anastasiamacwallyworld: re-instatement (?) of vsphere supported architecture - http://reviews.vapour.ws/r/5649/03:11
wallyworldok, on my list03:12
anastasiamac:)03:12
rick_h_perrito666: not here, you make it in?03:15
rick_h_wallyworld: do you have the config changes spec handy? I thought application config was just config now?04:02
wallyworldrick_h_: it is04:02
wallyworldbut not in beta1804:02
rick_h_wallyworld: did it not make b18? oh crap04:02
wallyworldPR up today, will land tonght04:02
rick_h_ah, thought b18 got all but one commit04:02
rick_h_ah, gotcha04:02
wallyworldrick_h_: sorry :-(04:02
wallyworldwe just ran out of time04:03
rick_h_wallyworld: all good, just working on my slides for tomorrow and checking my thoughts vs reality in the beta04:03
wallyworldneeded to coordinaye withCI etc04:03
rick_h_wallyworld: understand04:03
wallyworldput an asterisk :=)04:03
rick_h_yep, will work it out04:03
wallyworldrick_h_: also, i will have a fix today for machines not reporting as Down when they get killed04:04
wallyworldjust a cosmetic thing, but very annoying04:04
rick_h_wallyworld: <304:04
wallyworldespecially if you are an admin trying to script whether to enable ha or not04:05
anastasiamac... if CI gets a run without a failure... all landings I've seen today report similar failures :)04:13
perrito666Rick_h_ just getting out of the airport after 1h or more in migrations queue I wanted to message him to get dinner but I guess I'll be arriving too late04:17
rick_h_perrito666: gotcha, sucky on the queue fun04:19
perrito666Happens :) seems I picked a specially busy day04:21
perrito666Juju is in town so all these people are coming for the charmer summit, evidently :p04:23
menn0wallyworld, axw: do you know if anyone is looking into all the test timeouts in apiserver/applications04:27
menn0it's happened to me and lots of other merge attempts it seems04:27
menn0apiserver/application04:27
axwdon't know04:27
wallyworldmenn0: i'm not, i haven't been monitoring landing bot today04:27
veebersmenn0: you're seeing this in the merge job? (anastasiamac ^^)04:27
menn0wallyworld: ok... i'll start looking04:27
wallyworlddamn, something broke04:27
wallyworldmenn0: i am fixing the annoying go cookies issue04:28
menn0veebers: yep, most merge attempts today have failed because of this04:28
menn0so someone managed to land something which is failing most of the time04:28
* menn0 hopes it wasn't him :)04:28
veebersmenn0: right, I was checking to see if it was CI/infra related. I've changed which machine the next run will happen on in hopes it might help.04:28
menn0veebers: ok thanks.04:29
menn0veebers: I can't repro the problem locally of course04:29
veebersmenn0: heh :-\ always the way. FYI the last mereg that passed on that job was: "fwereade charm-life" (http://juju-ci.vapour.ws:8080/job/github-merge-juju/9167/)04:30
veebersmenn0: I'll track the next queued up job that will run on the older machine and let you know how it gets on04:30
menn0wallyworld, axw, anastasiamac: the stuck test appears to be TestAddCharmConcurrently if that rings any bells?04:30
anastasiamacmenn0: no bells but veebers pointed out the commit ^^ that seems to b the culprit :D04:31
* anastasiamac have to get a kid from school, b back l8r04:32
menn0veebers: cool, I'll start looking at that merge04:33
anastasiamacwallyworld: m considering to remove arch caching from vsphere on current pr as well.. any idea how heavily supported architectures retrieval is used?04:36
anastasiamacwallyworld: it'll b calling simplestream image retrival evry time constraints validator is constructed...04:37
wallyworldin a couple of places04:37
wallyworldtwice in one api call04:37
wallyworldwhen adding a machine i tihnk04:37
anastasiamacwallyworld: k.. i'll leave it out cached for now.. let's tackle it later for 2.1 maybe...04:38
wallyworldtha's from memory thugh04:38
wallyworldwould need to check code again04:38
anastasiamacwallyworld: k.. i've created a separate bug for it and we'll address separately then04:39
anastasiamacmayb we'll even have some help with perfomance benchmarking (veebers :D) to determine how much better/worse we'd do without caching supported architectures :)04:39
veebersheh :-)04:40
wallyworldmenn0: you're busy, if you get a chance later, here's that status fix http://reviews.vapour.ws/r/5651/. If no time, I can ask elsewhere04:45
menn0wallyworld: looking now04:47
wallyworldta04:48
menn0wallyworld: good stuff.04:55
menn0wallyworld: i'm creating a card now as the migrations prechecks will need to use this too04:55
wallyworldmenn0: thanks menno. btw did you book your flights yet? when you arriving/leaving?04:55
menn0wallyworld: I've sent the email to the agent but haven't heard back yet (unsurprisingly since they're not at work yet)04:57
wallyworldyou looking to arrive the first sat and leave the following sat?04:57
menn0wallyworld: I'm likely to be leaving on Saturday night, which gets me in on Sunday evening04:58
menn0wallyworld: leaving the sprint on Saturday morning04:58
wallyworldyou'll miss drinks :-)04:58
menn0wallyworld: possibly04:58
menn0wallyworld: sometimes they're a bit later04:58
wallyworlddepending on flights, i'm going to try and arrive sat evening04:59
menn0wallyworld: so it looks like will hit the timeout in apiserver/application twice while trying to merge. he assumed it was bug 159696005:15
mupBug #1596960: Intermittent test timeout in application tests <tech-debt> <unit-tests> <juju:Triaged> <https://launchpad.net/bugs/1596960>05:15
menn0but that one says it's only windows05:15
menn0I'm guessing his changes have made it more likely to happen05:15
wallyworlddamn, sounds pluasible05:15
wallyworldlooks messy as well05:17
axwwallyworld: will you have a chance to look at that ca-cert issue? I'm trying to stay focused on azure05:54
wallyworldaxw: yeah, i can look05:55
wallyworldaxw: just read emails, so the cert issue is just disabling the validation check IIANM05:56
axwwallyworld: see uros's latest email, there's also an issue with credentials looking up provider based on controller cloud05:57
axwwhich seems wrong...05:57
wallyworldyeah05:59
blahdeblahQuick Q: in order for a unit to log to rsyslog on node 0, should there be a rule in the secgroup that allows access to tcp port 6514?  And should juju add this automatically?07:26
=== frankban|afk is now known as frankban
wallyworldurulama: http://reviews.vapour.ws/r/5652/ FYI07:52
urulamathanks07:54
wallyworldblahdeblah: units can ask for ports to be opened on a bespoke basis07:54
wallyworldit's not something we'd do unilaterally07:55
blahdeblahwallyworld: so it wouldn't be done as part of add-unit when a machine is added via the manual provider?07:55
urulamawallyworld: been running it with that fix since axw pointed it out :)07:55
wallyworldblahdeblah: not that i am aware of. manual provider assumes pretty much that everything is in place. juju tends to try not to mess with manual machines07:57
blahdeblahwallyworld: OK - thanks07:57
wallyworldurulama: i was hinting for a review from your folks :-)07:58
wallyworldaxw: fyi urulama thinks that add-model issue may be with the controller proxy, so we're off the hook for now07:59
axwwallyworld urulama: yeah, I think it's most likely due to something around Cloud.DefaultCloud and/or Cloud.Cloud08:00
wallyworldaxw: yep, i traced to to the cli making an api call to client.Cloud() and it's all goo in core08:00
wallyworldgood08:00
wallyworldbut something missing in proxy most likely08:00
voidspacebabbageclunk: https://github.com/juju/juju/compare/master...voidspace:1534103-run-action09:55
* frobware needs to run an errand; back in an hour.09:57
fwereadevoidspace, may I have a 5-second review of http://reviews.vapour.ws/r/5653/ please?10:04
fwereadevoidspace, apparently it has been failing a bunch10:04
voidspacefwereade: ok10:05
voidspacefwereade: LGTM10:12
fwereadevoidspace, ta10:13
mupBug #1594977 changed: Better generate-image help <helpdocs> <oil-2.0> <v-pil> <juju:Triaged> <https://launchpad.net/bugs/1594977>11:55
mupBug #1622581 opened: Cryptic error message when using bad GCE credentials <juju-core:New> <https://launchpad.net/bugs/1622581>11:55
mupBug #1622581 changed: Cryptic error message when using bad GCE credentials <juju-core:New> <https://launchpad.net/bugs/1622581>12:19
fwereadeis anyone free for a ramble about cleanups with a detour into refcounting? axw, babbageclunk?13:05
babbageclunkyup yup13:12
fwereadebabbageclunk, so, the refcount stuff I extracted13:14
fwereadebabbageclunk, short version: it's safe in parallel but not in serial13:14
babbageclunkbabbageclunk: ?13:14
natefinchfwereade: that is impressive13:15
voidspacethat's impressive13:15
babbageclunkI didn't think that was a thing we needed to worry about.13:15
voidspacehard to do13:15
natefinchvoidspace: hi513:15
voidspaceo/13:15
fwereadebabbageclunk, i.e. refcount is 2; 2 separate transactions decref; one will fail, reread with refcount 1, successfuly hit 0 and detect13:15
voidspacenatefinch: :-)13:15
fwereadevoidspace, natefinch: I'm rather proud of it, indeed13:15
natefinchlol13:15
babbageclunkbut isn't serial just slow parallel?13:16
fwereadebabbageclunk, refcount is 2, one transaction gets composed of separate ops that hit same refcount: will decref to 2, but won't ever "realise" it did so, so there's no guaranteed this-will-hit-0 detection13:16
babbageclunkugh13:17
fwereadebabbageclunk, we're always composing transactions from ops based on a read state from before the txn started13:17
babbageclunkAll the asserts happen before all of the ops?13:17
fwereadebabbageclunk, yeah13:17
fwereadebabbageclunk, that's how it works13:17
babbageclunkof course. ouch. so each assert passes, but they leave it at 0 with no cleanup13:18
fwereadebabbageclunk, yeah, exactly13:18
voidspacefwereade: you have two days to fix this, right13:19
fwereadevoidspace, perhaps :)13:19
voidspace:-)13:19
fwereadevoidspace, stateful refcount thingy is one, wanton spamming of possibly-needed cleanups is another13:20
fwereadevoidspace, I'm slightly hopeful that you have a third?13:20
voidspaceI hope so too13:20
fwereadevoidspace, oh13:21
voidspaceoh, no13:21
voidspacesorry13:21
fwereadevoidspace, days, I thought you said ways13:21
voidspacefwereade: I hope I have at least a third day13:21
fwereadevoidspace, I would imagine so ;)13:21
* babbageclunk lols sadly.13:21
voidspacefwereade: unless they purge juju of everyone you know...13:21
voidspacefresh start and all that13:21
fwereadevoidspace, we have always been at war with...13:22
voidspace:-)13:22
fwereadevoidspace, babbageclunk: so on the one hand there is this problem with txns13:23
fwereadevoidspace, babbageclunk: and it's one that bites us increasingly hard as we try to isolate and decouple individual changes to the db13:24
fwereadevoidspace, babbageclunk: and I don't really have an answer to either the problem or the increased risk we take on as we further isolate ops-generation13:25
babbageclunkWhy do we compose these operations into one transaction? Shouldn't they be multiple transactions?13:28
mupBug #1622136 changed: Interfaces file source an outside file for IP assignment to management interface <juju:Triaged by rharding> <https://launchpad.net/bugs/1622136>13:28
fwereadebabbageclunk, that is basically where I'm going13:28
babbageclunkNot sure how we could prevent it though13:28
fwereadebabbageclunk, I cannot, indeed, think of a reason that the app remove ops have to be bundled into the final-unit-remove ops13:29
fwereadebabbageclunk, and in fact, that approach is itself vulnerable to that style of bug -- if we wrap up the final *2* unit-removes, we'd miss the app-remove code13:30
babbageclunkfwereade: But it would be nice if the transaction system could prevent you from combining these transactions together somehow since they're not valid.13:30
fwereadebabbageclunk, that would probably be sensible, but I can't see any non-blunt ways of doing it -- only one op per doc, I guess? but that works *hard* against any prospect of decomposition13:32
fwereadebabbageclunk, the usual escape valve is cleanup ops, ofc -- you can apply a partial change and leave a note to pick it up later, and that's great13:33
babbageclunkfwereade: can it be more fine-grained than that - one op touching any attribute of a doc in one transaction?13:33
fwereadebabbageclunk, perhaps so, but it sorta sucks not to be able to incref unitcount by 5, for example13:34
babbageclunk(Not sure how easy that would be to do in the mongo expression lang)13:34
babbageclunktrue13:34
fwereadebabbageclunk, and anything at the txn layer has sort of lost the real context of the ops, so it's likely hard/impossible to DTRT re compressing ops into one13:35
fwereadebabbageclunk, (I heartily support this style of thinking, I just don't think I can do much about it in 2 days, hence cleanups)13:36
babbageclunkfwereade: yeah, it seems like it would be hard to do that in a generic way - I can see it working for refcounts, but I'm sure the same problem can come from other things harder to reason about.13:37
babbageclunkso, cleanups!13:37
fwereadebabbageclunk, so, if we simplify unit removal (and relation removal, same latent bug) such that it doesn't even care about app refcounts, and just read life and drop in a maybe-clean-the-app-up op13:38
fwereadebabbageclunk, the cleanups will run and everyone is happy13:38
fwereadebabbageclunk, except that the time taken to remove a service once its last unit goes away has gone from ~0s to 5-10s13:39
fwereadebabbageclunk, because the cleanups run on the watcher schedule13:39
babbageclunkfwereade: Oh, 'cause that's when a cleanup will run.13:39
voidspaceso that's the "spam extra cleanup checks" approach13:39
voidspacebut removing the service once the units have gone is *mostly* admin right13:40
fwereadebabbageclunk, yeah -- and the more we do this, the better our decoupling but the more we'll see cleanups spawning cleanups and require ever more generations to actually get where we're going13:40
voidspaceor is there resource removal that only happens at cleanup time too?13:40
fwereadevoidspace, yeah, but you can't deploy another app with the same name, for example13:40
voidspaceright13:40
voidspaceis that a common need?13:41
voidspacemaybe I guess13:41
babbageclunkdo watcher polling more frequently!13:41
babbageclunk;)13:41
fwereadebabbageclunk, that is certainly an option, and it does speed things up, but it's also the sort of tuning parameter that I am loath to fiddle with without paying close attention to the Nth-order effects at various scales and so on13:42
babbageclunkWhat about rather than dropping a cleanup you drop another txn that does the removal, with an assert that the refcount's 0?13:43
fwereadebabbageclunk, can't guarantee they both apply -- that is the purpose of a cleanup, to queue another txn, really13:43
babbageclunkAh, no - the cleanup gets created in the txn, right?13:43
fwereadebabbageclunk, and you can't really write ops for future execution in the general case -- if they fail, there's no attached logic to recreate or forget about them, we can only forget13:44
fwereadebabbageclunk, voidspace: anyway, one watcher-tick delay is not so terrible13:45
babbageclunkno13:46
fwereadebabbageclunk, voidspace: so I was thinking I could just tweak the cleanup worker: expose NeedsCleanup, and check it in a loop that cleans up until nothing's left13:47
fwereadebabbageclunk, voidspace: which at least gives us freedom to explore more-staggered cleanup ops without macro-visible impact13:48
fwereadebabbageclunk, voidspace: and which I can probably get done fairly quickly13:48
voidspacesounds reasonable13:48
babbageclunk+113:49
fwereadebabbageclunk, voidspace: barring unexpected surprises in trying to separate service-remove from unit-remove13:49
fwereadebabbageclunk, voidspace: excellent, thank you13:49
voidspace# github.com/juju/juju/cmd/jujud14:11
voidspace/usr/lib/go-1.6/pkg/tool/linux_amd64/link: running gcc failed: fork/exec /usr/bin/gcc: cannot allocate memory14:11
mgzyeah, I suffer a fair bit from that14:12
mgzlinking with 1.6 takes a lot of memory14:12
voidspacetime to switch to 1.7 then I guess14:12
voidspaceI haven't seen it before and now I'm seeing it consistently with master14:16
voidspace_ok, so a reboot fixed the memory issues14:55
dimiternfrobware: hey, not sure if you've seen my PM15:34
dimiternfrobware: here's the PR I'm talking about: https://github.com/juju/juju/pull/621915:34
=== hml_ is now known as hml
fwereadebabbageclunk, voidspace_: I think I have a happier medium, in case I don't land anything else: http://reviews.vapour.ws/r/5644/15:51
fwereadebabbageclunk, voidspace_: would either of you be free to take a look before EOD?15:51
babbageclunkfwereade: Sure, looking now15:52
fwereadebabbageclunk, tyvm15:52
redirmorning juju-dev16:01
babbageclunkfwereade: Sorry, I got distracted - still looking!16:48
voidspacefwereade: you still here?16:51
fwereadebabbageclunk, voidspace: heyhey17:00
voidspacefwereade: so this implementation of a failaction operation seems to work and "do the right thing" https://github.com/juju/juju/compare/master...voidspace:1534103-run-action#diff-ae955475ac58e0d2683d2cfd6101b3f7R117:01
=== frankban is now known as frankban|afk
voidspacefwereade: which is mostly copied from runaction.go17:03
fwereadevoidspace, that certainly looks sane to me17:07
voidspacefwereade: cool, it seems to fix the bug and behave sanely - so I'll add tests and propose17:07
fwereadevoidspace, cool, tyvm17:16
perrito666hey, juju restore survives suspending the machine for 10 mins, sweet17:31
perrito666does annyone know if there is a way to list all models?17:55
perrito666fwereade: ?17:55
fwereadeperrito666, I thought there was literally a list-models?17:56
perrito666fwereade: sorry I meant in state :p17:56
fwereadeperrito666, not sure offhand, how does the list-models apiserver do it?17:56
* perrito666 accidectally mixed chia and earl grey and is not happy about the result17:56
perrito666fwereade: an ungly thing that gets models for a user17:57
perrito666I was trying to avoid constructing another one of those17:57
perrito666:p17:57
perrito666hey, there is an AllModels here17:59
perrito666nice17:59
fwereadeperrito666, well, the raw collection is pretty terrible18:01
fwereadeperrito666, but, resolved anyway ;p18:02
mbruzekhmo: http://ppa.launchpad.net/juju/devel/ubuntu/pool/main/j/juju-core/juju-core_2.0-beta15-0ubuntu1~16.04.1~juju1.debian.tar.xz18:52
perrito666is the message "Contacting juju controller <private ip>" correct here? http://pastebin.ubuntu.com/23170667/20:01
natefinchperrito666: buh... that can't be right, unless somehow you can connect to the private address of AWS from where you're running the client20:08
perrito666I cant20:09
natefinchperrito666: weird then.  probably just posting the first address in whatever list20:12
perrito666yep, after a restore, juju status will also show that address20:12
=== rmcall_ is now known as rmcall
mupBug #1622738 opened: Multi-series charms failing in 1.25.6 <juju-core:New> <https://launchpad.net/bugs/1622738>20:39
wallyworldredir i am free for a bit but need coffee so give me 5 if you still had a question22:33
redircool22:33
redirI'm here22:33
redirbut going to make tea while you make coffee22:36
perrito666wallyworld: hey, this https://bugs.launchpad.net/juju/+bug/1595720 is still happening but now its a big issue since admin users are hitting this  :(22:55
mupBug #1595720: Problems using `juju ssh` with shared models <ssh> <usability> <juju:Triaged> <https://launchpad.net/bugs/1595720>22:55
wallyworldperrito666: damn, i'll add to the list of todo items for rc1, yay23:12
wallyworldthumper: standup?23:16
thumperreview up: http://reviews.vapour.ws/r/5657/23:36
marcoceppi rick_h_ https://bugs.launchpad.net/juju/+bug/162278723:40
mupBug #1622787: If you name a credential with an @ Juju barfs <juju:New> <https://launchpad.net/bugs/1622787>23:40
rick_h_marcoceppi: lol ty for keeping the barf part23:41
thumpero/ marcoceppi23:43

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!