/srv/irclogs.ubuntu.com/2014/08/22/#juju-dev.txt

xavierbickHello everyone, I saw this bug on launchpad (https://bugs.launchpad.net/juju-core/+bug/1336473) and wanted to submit a fix to github, but I'm not quite sure what the normal procedure is for contributing a bug fix to juju.  Can I just assign myself to the bug and mark it as in progress?00:02
mupBug #1336473: Support new t2 instance types on AWS <ec2-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1336473>00:02
davecheneyxavierbick: sure00:03
xavierbickgreat, thanks davecheney00:04
perrito666davecheney: I need to make the api server able to restart jujud with a given result from a FacadeCall, much like what happens for workers but in this case the call would answer to the client forwarding the result and then restart, yet I am not very sure where to add this00:05
=== menn0 is now known as menn0-afk
davecheneyperrito666: i think someone else asked this question recently00:07
davecheneymenn0-afk: did you ask this recently00:07
perrito666davecheney: I send a similar question to the mailing list some time ago00:12
perrito666before understanding the actual problem fully00:12
davecheneyahh00:17
perrito666davecheney: that means you dont have an answerfor me right?00:50
perrito666:p00:50
davecheneyperrito666: i'm not even sure I understand the question00:53
perrito666I most likely suck at explaining it00:53
davecheneyperrito666: you want an api method to restart the apiserver, but not just exit the process ?00:54
perrito666Ill try to explain better, bear with me Its late :)00:54
perrito666I created a Client API method for Restore, which basically... restores :)00:57
perrito666now restore its a rather destructive thing which shuffles things around so, after finishing it I need to restart jujud00:58
perrito666so It was suggested by william that I should handle the return value of said call and restart jujud on a given result00:59
perrito666now, even if restarting jujud, I need to answer to the client first so it acknowledges that the command finished00:59
davecheneyperrito666: i dunno01:04
davecheneyi'd just do exit(0)01:04
perrito666davecheney: what I am not sure is where to :) there is a blurry area between state/api/client and state/apiserver/client :p which is not all that clear to me, even though is properly documented01:08
perrito666anyway, 31C at 11PM is too much for my brain I am going to sleep under the AC until the head clears off01:08
perrito666jam: if you read the abovewhen you arrive and have an answer please drop it here and Ill read it in the morning01:09
davecheney31c, fuk that noise01:09
perrito666noise?01:09
perrito66631°C <-- temp, fun fact, we are in the middle of winter01:10
davecheneybzzt01:10
wallyworlddavecheney: that status bug has been fixed in trunk, i just backported to 1.2001:40
wallyworldyou did the fix for trunk a little while back01:40
davecheneyit's sort of a fix01:41
davecheneyit's still shit01:41
davecheneywe should just stop trying to carry on if api.GetStatus screws up01:41
wallyworldsure, but at least the nil pointer is fixed and 1.20 is consistent with master01:58
wallyworldaxw: just otp, there soon02:00
axwwallyworld: nps02:00
wallyworldaxw: ready now02:05
katcowallyworld: i need to take care of some things, but i think i addressed all the issues your brought up in https://github.com/juju/juju/pull/58402:36
wallyworldkatco: awesome, thank you, will look02:37
katcowallyworld: no, thank you :)02:37
* katco away for a bit, maybe for the night.02:38
axwwallyworld: just putting my desk back together, will bbs to review02:42
wallyworldsure02:42
=== menn0-afk is now known as menn0
jamperrito666: state/api/client is the actual user/client/not-the-server side of the code, and state/apiserver/client is the stuff running inside the API server. "client" in this case is that it is the set of "client facing" APIs (Status, Deploy, Upgrade, etc), not Agent facing APIs.03:26
jamperrito666: essentially everything in state/apiserver should be mirrored as state/api, the former being the one producing the API, and the latter being the one consuming it.03:27
jamperrito666: as for restarting gracefully, the worker/Upgrader code does this by raising UpgradeReadyError03:28
jamwhich signals up the worker/runner stack that we want to restart our process.03:28
jamHowever, that is being run in a *worker* not during an API call.03:28
jamI don't have an immediate "this is clearly how to do it" but we should probably take this to an on list discussion, so that fwereade et al can get in on the conversation03:29
wallyworldaxw: manual provisioning question - in manual/provisioner.go, there's a ProvisionMachine() method; know the one i mean?03:52
axwwallyworld: yup?03:55
wallyworldin that method, we call args.Client.ProvisioningScript03:55
* axw nods03:55
wallyworldand DisablePackageCommands will be false03:55
katcowallyworld: i have to go to bed, but i just pushed up more changes. thanks for the review.03:55
wallyworldkatco: np, still going through it :-)03:55
axwwallyworld: orly? /me looks03:55
wallyworldkatco: thanks for the effort :-)03:56
* katco hands wallyworld a fine-toothed comb03:56
wallyworldlol03:56
wallyworldi don't have nits :-P03:56
katcowallyworld: yeah well, this has drug out long enough. i wanted to take advantage of you being online ;)03:56
axwwallyworld: DisablePackageCommands isn't set, so yes, it'll be false03:56
wallyworldaxw: so that means we will run upgrade/update on manual machines, no?03:57
katcook, sorry, off to bed. thanks again wallyworld, axw.03:57
axwwallyworld: ProvisioningSCript calls cloudcfg.SetAptUpgrade(false)03:57
wallyworldkatco: night03:57
axwkatco: night night03:57
axwwallyworld: the idea was that manual should only install what it needs, and not muck about with the machine otherwise03:58
wallyworldaxw: oh, so it does, so it ignores the DisablePackageCommands03:58
wallyworldwhereas now, we want to default to the current behaviour, but allow it to be overridden03:59
axwwallyworld: IIRC, DisablePackageCommands is set on the MachineConfig in the API server code03:59
axwwallyworld: probably do that in the API server code then, I think.04:00
axwapiserver/client04:00
wallyworldaxw: it's not set all for the ProvisioningScript call04:00
wallyworldit's only set on machineconfig by the lxc container manager04:00
axwwallyworld: it is. look at state/apiserver/client/client.go04:01
axwthere's a Client.ProvisioningScript method that sets up the MachienConfig, then calls manual.ProvisioningSCript04:01
wallyworldyes, i meant that nothing sets its value to true04:03
axwwallyworld: kapil's external code does :)04:03
wallyworldright04:03
wallyworldnothing in juju04:03
wallyworldso instead in juju we hard code disableaptupgrade to false04:04
wallyworldso now in the new branch, that is configurable04:04
axwwallyworld: upgrade is *always* false for manual, regardless of what you do. you can *also* disable update&install by setting DisablePackageCommands04:04
axwok04:04
axwmmm04:04
axwyeah, that sounds fine04:05
wallyworldie it defaults to disabled like now04:05
wallyworldbut the user can choose to have their box upgraded if they want04:05
axwas long as it doesn't override DisablePackageCommands04:05
wallyworldthat's the hard bit i'm pondering about - how to stay backwards compatible04:06
wallyworldsince we don't know if DisablePackageCommands was explicitly set or not04:06
wallyworldif think if it is false, we use the new config settings04:07
wallyworldif true, we disable everything regardless of the settings04:07
axwwallyworld: +104:08
axwwallyworld: the client side doesn't need to change at all04:08
wallyworldit's not "perfect", bu i think the best we can do04:08
axwjust state/apiserver/client/client.go04:08
wallyworldyep04:08
axwmanual.ProvisioningScript shouldn't force upgrade=false anymore, that needs to move into apiserver04:09
axwwhich has access to config04:09
wallyworldyep04:11
menn0wallyworld, thanks for dealing with bug 135943504:21
mupBug #1359435: Next version selection for upgrades is no longer correct <juju-core:Triaged> <juju-core 1.20:In Progress by wallyworld> <https://launchpad.net/bugs/1359435>04:21
wallyworldmenn0: no problem, thanks for noticing the issue04:22
wallyworldaxw: a small one https://github.com/juju/juju/pull/59105:17
axwlooking05:18
wallyworldaxw: thanks05:20
davecheneywallyworld: could you please kill http://juju-ci.vapour.ws:8080/job/github-merge-juju/390/console05:59
davecheneyit's not going to pass and there is a large backlog behind it05:59
wallyworlddavecheney: it will pass, why won't it?05:59
wallyworldthat uniter failure happens all the time06:00
wallyworldpasses 2nd time through06:00
davecheneywallyworld: ok,06:01
wallyworlddavecheney: did i mention i hate our tests?06:01
davecheneywallyworld: to tell06:14
wallyworldof late, most runs require the fallback -p 2 to pass06:15
wallyworldi will be cataloging the failures and trying to get people to fix them06:15
davecheneywallyworld: +1 to turning off the retry06:18
davecheneytests pass or GTFO06:19
wallyworldi think so, we'll try and get an itemised list of failures first06:19
davecheneyyou'll have no shortage of canidates then06:26
davecheneywallyworld: i'll close with this graph, http://juju-ci.vapour.ws:8080/job/github-merge-juju/buildTimeTrend06:47
davecheneythings clearly went wrong, around this commit waigani endpoint-on-bootstrap-take306:47
davecheneynot saying it was this commit06:47
wallyworlddavecheney: yeah, early after the cutover, we were building in 18 or so minutes06:48
wallyworldwe had hoped to get it down to 10 or 11 with the use of lxc or nail up instances06:49
=== wallyworld_ is now known as wallyworld
mattywmorning folks08:02
TheMuemorning08:11
voidspaceTheMue: morning08:14
voidspaceTheMue: I updated this PR to use errors.Annotate everywhere08:15
voidspaceTheMue: https://github.com/juju/juju/pull/56108:15
TheMuevoidspace: great, will take a look08:15
voidspaceTheMue: I would appreciate a quick look for the sake of sanity when you have a moment08:15
voidspacethanks!08:15
voidspaceTheMue: as I was touching the file I did all of the ones that were appropriate to change08:16
voidspace(in some cases there isn't an existing error to annotate of course, so those remain Errorf)08:16
TheMuevoidspace: I’m still fighting with changing an existing API part into a new version. it’s a rattail :/08:16
voidspacefun08:16
TheMuevoidspace: the errors package IMHO has useful funcs there too08:17
voidspaceTheMue: ok08:17
TheMuevoidspace: you’ve got a LGTM08:18
voidspaceTheMue: awesome, thanks08:19
voidspaceTheMue: and are you happy(ish) for me to merge the other PR without an additional test?08:19
voidspaceTheMue: given that we concluded that we *do* test (in provider specific ways) that only the requested ports are open08:19
TheMuevoidspace: here’s the errors package http://godoc.org/github.com/juju/errors08:20
TheMuevoidspace: have to take a deeper look myself08:20
voidspaceTheMue: thanks08:20
voidspaceI will peruse08:20
voidspaceI wonder how good the documentation is08:20
TheMuevoidspace: and regarding the other PR I can live w/o another test, yes. it only have been more general thoughts about how to ensure that only the wanted ports are open. but that would be a huge change :)08:22
voidspaceTheMue: and we have to open some ports in a provider specific way - as we have to specify some ports when we create the machine for some providers I believe08:22
voidspaceas we need some ports immediately08:22
voidspaceso a *general change* is maybe not even possible08:23
voidspacealthough we don't need the state port immediately just to bootstrap08:23
TheMuevoidspace: yep, you’re right08:23
Beretwallyworld, nice update, thanks08:57
Beretwallyworld, we'll do all we can to get you more information when we hit those LXC issues08:58
wallyworldBeret: no problem and thank you08:58
=== wallyworld_ is now known as wallyworld
perrito666morning all10:30
voidspaceperrito666: morning10:50
voidspacedimitern: TheMue: standup?10:50
TheMuedimitern, voidspace: standup?10:51
TheMuevoidspace: oh, just typed the same10:51
voidspacehah10:51
TheMue;)10:51
dimiternvoidspace, TheMue, sorry, brt10:51
dimiternvoidspace, TheMue, others? networker refactoring branch - https://github.com/juju/juju/pull/593 , I'd appreciate a review when you have some time13:00
wwitzel3dimitern: I will take a look, I'm one of the on-call reviewers today :)13:05
TheMuedimitern: will look13:06
=== uru_ is now known as urulama
dimiternwwitzel3, TheMue, cheers guys!13:26
perrito666natefinch: 1:1?13:27
perrito666lol just saw the email13:28
wwitzel3man, trying to use our existing providers as a baseline for reviewing a new provider is turning out to be confusing.13:35
wwitzel3I feel like all our existing providers implement the same behavior as each other, just in a slightly different way.13:36
wwitzel3which provider should be considered the "gold standard"?13:36
katcowwitzel3: +113:36
katcowwitzel3: i just did some work on providers, and i can't answer that.13:36
mgzwwitzel3: well, ah13:36
mgzthey're all bad in different ways :)13:37
katcowwitzel3: if i had to guess, i'd say ec2 since it's probably been around the longest?13:37
TheMuewwitzel3: always the next one, which will make it all better13:37
katcoTheMue: lol!13:37
wwitzel3I can distill these answers to "Yes, good luck"13:38
wwitzel3:D13:38
TheMuewwitzel3: +113:38
katcowwitzel3: haha... well what are you trying to review? there are probably some concepts that in their pure form remain the same13:39
wwitzel3katco: the CloudSigma provider set of PRs13:39
katcowwitzel3: link to one?13:39
wwitzel3katco: https://github.com/juju/juju/pull/17313:40
wwitzel3katco: it is also the fact that it is just 2,000+ new lines. So there is no reference built in to the diff.13:41
katcowwitzel3: yeah, i see that! tough review...13:41
wwitzel3katco: that's why I'm trying to look to our existing providers as a reference / guide13:41
katcowwitzel3: yeah, i think i'd be doing what you are. and probably bouncing between ec2 which is probably the oldest, and maybe local which is one of the newer ones. or azure.13:42
* katco dusts hands off. glad i could provide absolutely no value for you. ;)13:43
wwitzel3sanity checks are always helpful13:44
perrito666katco: well if you manage to make a career out of that you can become an argentinian real stateagent13:45
wwitzel3hah13:45
katcoperrito666: i don't have it in me, unfortunately. it would drive me crazy.13:45
perrito666katco: there is good money in it and you can code while ignoring your customers13:46
perrito666fwereade: are you around?13:47
fwereadeperrito666, yeah13:47
katcoperrito666: lol13:47
fwereadeperrito666, meant to ping yu this morning13:47
fwereadeperrito666, you were asking last night about restarting the state server from within an api call?13:48
perrito666fwereade: heh yeah, our discussion the other day end up including more people and opinions and I am not entirely sure how to implement that :p13:48
fwereadeperrito666, so, I have been 100% focused on leader-election things for the last day or two, I am probably missing context13:48
perrito666fwereade: Well I triedto implement that according to what we spoke, and suddenly I noticed that there was a gray area on state where I am not that familiar13:49
fwereadeperrito666, are there concerns about the special-error approach?13:49
fwereadeperrito666, ah, go on13:49
fwereadeperrito666, I'd expect state to be out of the loop on that sort of question13:49
perrito666fwereade: the deal is, I am not entirely sure where is that I should be catching the error to ensure the client gets the response and trigger the restart (I might not be as good explaining myself as I think I am on this particular matter)13:50
fwereadeperrito666, well -- somewhere after the function returns, and before it gets packaged up and sent back over the API13:53
fwereadeperrito666, how many choices do you have there?13:53
fwereadeperrito666, presumably it needs to be a layer that has, or can easily and sanely be given, access to the apiserver worker's tomb13:54
fwereadeperrito666, so I would guess that the apiserver worker itself would be the right one to actually handle sending on the error to its runner13:54
fwereadeperrito666, but I can well believe that we need another layer in there somewhere13:54
fwereadeperrito666, I don't immediately recall what's actually in play there13:55
perrito666fwereade: as I get it th call propagates like this: apiclient->rpc layer->(X)->apiserver13:56
perrito666and I believe I should be standing in (X)13:56
fwereadeperrito666, hmm, that sounds wrong13:56
fwereadeperrito666, the apiserver needs to know that it should stop13:57
fwereadeperrito666, and I think it has to finish the call, passing a "everything's good, restarting now!" response down via rpc, while arranging its own tidy suicide separately13:58
perrito666fwereade: I see, makes sense13:59
fwereadeperrito666, possible approach: return result *and* error in that situation13:59
fwereadeperrito666, in apiserver, define the error (type) that indicates "return these results and then terminate the api server with ErrStateRestored(?)"14:00
fwereadeperrito666, except14:01
fwereadeperrito666, can you really actually *do* the restore while everything's running?14:01
fwereadeperrito666, what's the actual sequence of events?14:02
perrito666fwereade: yes, writing14:02
natefinchperrito666: standup?14:03
perrito666natefinch: getting there, I was looking for a webcam14:04
perrito666:p14:04
TheMuedimitern: great work, but I would like a second to take a look too14:07
perrito666fwereade: so this is how it works: [Bootstrap in safe mode][Upload File]<stop mongo><restore mongo><restore conf files for juju> <start mongo><restart jujud?>14:08
fwereadeperrito666, then I don't quite understand where the API server comes in14:09
fwereadeperrito666, ah: so immediately after you've bootstrapped, you tell the API server to do all that stuff?14:10
=== ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs: 1354027 1360286
perrito666yup14:10
perrito666fwereade: so once I have a working api server I ask for it to perform the restore14:10
fwereadeperrito666, and is the API server still responding to requests while this is going on?14:10
perrito666fwereade: mm I see where you are going, I could make API server deny requests, much like what upgrade does14:12
perrito666update14:12
sinzuinatefinch, jam: We have a new regression. Can you find an engineer to fix or revert the problem? https://bugs.launchpad.net/juju-core/+bug/136028614:12
mupBug #1360286: ppc64el and i386 unittests cannot find tools <ci> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1360286>14:12
fwereadeperrito666, I'm really rather nervous about having the api server running at all while this is going on14:12
fwereadeperrito666, in particular I am nervous about what firewaller, networker, etc might do while running against incorrect state14:14
fwereadeperrito666, the provisioner might take down machines so that's the obvious risk14:14
perrito666fwereade: actually the state is down (and API will not connect to it until restarted)14:14
fwereadeperrito666, how do you ensure it's down? starting jujud will start all sorts of workers, surely, and you won't know to shut them down until you get this api request14:15
perrito666fwereade: hold, I think we could be not on the same page here, let me finish this standup and then I can give you my full attention (I would really love tho have a white board between us but that is too much asking)14:21
fwereadeperrito666, a whiteboard in one's webcam's FOV is a useful thing, I never got round to setting it up though14:22
fwereadeperrito666, could be14:22
fwereadeperrito666, natefinch: brb, but will join yu in your standup in 514:22
perrito666fwereade: we are no longer there, just ping me when you are back14:22
* perrito666 suddenly hits his head on an open window... brb14:29
fwereadeperrito666, natefinch: will be in the hangout when you're ready14:29
perrito666back14:29
ericsnowfwereade, perrito666: mind if I join?14:30
fwereadeericsnow, ofc not14:30
perrito666ericsnow: you are welcome to join14:30
ericsnowfwereade, perrito666: you guys are the best! :)14:30
perrito666aghh my ears are aching15:43
natefinchanyone have thoughts on why a machine after upgrading from 1.18 to 1.20 would says "missing API Hosts"?15:44
wwitzel3it's lonely?15:45
wwitzel3</nothelpful>15:45
perrito666natefinch: agent.conf corrupted?15:45
natefinchperrito666: interesting you should say that, he did say he had to update the agent.conf by hand to add apiaddresses15:46
natefinchor at least, he thought that was the problem, but adding it didn't help15:46
natefinchcan someone else give me a hand? I may have to leave soonish.  On #juju in canonical irc, guy's nick is ivoks, he's onsite with a customer right now.15:50
* fwereade bbs15:55
perrito666natefinch: sorry was buying foood15:56
=== urulama is now known as uru-afk
katcohow do i got about upgrading an environment to my dev version? "juju upgrade-juju --dry-run --version 1.21-alpha1" gives me "ERROR no matching tools available"16:20
natefinch--upload-tools should work I think16:23
natefinchcan anyone else help ivoks in canonical irc #juju?  I have to go16:24
natefinchsinzui, alexisb, mgz, wwitzel3 ^16:25
natefinchperrito666 was looking at iot but had to go too16:25
katconatefinch: ah i see... it doesn't like --upload-tools with --dry-run16:25
katconatefinch: that's what was tripping me up16:25
sinzuikatco, interesting. tip is definitely broken for ppc64el and i386, what arch you testing?16:26
sinzuisee bug 136028616:26
mupBug #1360286: ppc64el and i386 unittests cannot find tools <ci> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1360286>16:26
katcosinzui: amd64, i think it was just the combination of --dry-run16:26
katcosinzui: if i do --upload-tools alone, it gives me a list of available tools, and best version16:27
sinzuikatco, oh, we have one or two related bugs where --upload-tools doesn't in other combinations16:27
* sinzui looks16:27
sinzuikatco, These are the upload-tools bugs https://bugs.launchpad.net/juju-core/+bugs/?field.tag=upload-tools and bug 1307643 looks like the issue16:30
mupBug #1307643: juju upgrade-juju --upload-tools does not honor the arch <upgrade-juju> <upload-tools> <juju-core:Triaged> <https://launchpad.net/bugs/1307643>16:30
katcosinzui: interesting, thank you so much. it looks like it might? have worked?16:32
katcosinzui: i'm noticing it incrementing the agent-version16:32
sinzuikatco, status shows the new version?16:33
katcosinzui: yeah: 1.21-alpha1.x16:33
katcosinzui: where x is incrementing...16:33
sinzuikatco, upload-tool will always do that16:34
sinzuikatco, that is to distinguish a version taken from the client instead of the net16:35
katcosinzui: so how do i know that it's actually updated? state is started16:35
sinzuikatco, I trust the log 99% of the time. When I doubt, I ssh to the machine and run "jujud version"16:41
sinzuikatco, I have never seen it differ from status. Once, did this and found th reason for a failed upgrade was bad symlink. There were to jujuds on the machine. I deleted the symlink and juju correctly upgraded itself a minute later16:42
katcosinzui: lol ok, so rule of thumb: trust agent-status16:43
perrito666ericsnow: you are a cruel man, such lengthy email on a friday afternoon18:17
perrito666ericsnow: why would backup allow upload?18:19
perrito666ericsnow: I have not yet looked at the metadata but I do have a request18:20
perrito666ericsnow: lemme know when we can have a call18:21
perrito666ericsnow: ?18:56
ericsnowperrito666: hey just got back19:09
perrito666hey, read above :)19:10
ericsnowwe should hang out19:10
ericsnowmoonstone?19:10
perrito666ericsnow: sure19:10
katcorick_h__: you still around?21:57

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!