#juju-dev 2013-01-02
<fwereade> good mornings
<fwereade> lunch, bbiab
<mramm> Good morning fellow Juju-ers!
<fwereade> mramm, heyhey
<mramm> http://www.ubuntu.com/  -- 3:00 min to go
#juju-dev 2013-01-03
<fwereade> mgz, morning
<mgz> hey fwereade
<fwereade> mgz, nice holiday?
<mgz> yup! and yours?
<fwereade> mgz, yeah, very peaceful
<fwereade> jam, heyhey, hope you had a nice holiday
<jam> fwereade: hi, yeah pretty good time. I hope your time off went well also
<fwereade> jam, yes thanks, very relaxing :)
<fwereade> jam, mgz: fwiw, I have a great big pile of juju-core reviews, and I'd be very happy if you had the time to take a look at any of them today -- the good news is that a lot of them are really trivial :)
<mgz> fwereade: will have a look over them and see if there are any I have useful comments on
<fwereade> mgz, cheers
#juju-dev 2013-01-04
<fwereade> mgz, fwiw, I have 4 CLs with diffs of <100 lines that have 1 LGTM already -- if you're free at any point this afternoon I would be most grateful if you would look at them
<fwereade> mgz, they are: https://codereview.appspot.com/7001050/ https://codereview.appspot.com/7001053/ https://codereview.appspot.com/7005054/ https://codereview.appspot.com/7013048/
<fwereade> mgz, (and I have others too, if you happen to be bored... ;p)
<mgz> will do.
<fwereade> mgz, cheers
<mgz> have still not submitted to love of rietveld...
<mgz> the forced file-by-file review workflow just doesn't match what I want to do
<fwereade> mgz, I find it works pretty well for me, but mileage clearly varies -- I find that once I've done an initial skim for overall structure I'm fine
<fwereade> mgz, but then I'm probably misunderstanding how you like to review code
<fwereade> arosales, ping
<arosales> fwereade: hello
<fwereade> arosales, heyhey, hope you had a good xmas
<arosales> I did, spent some good time with the family.
<fwereade> arosales, I have a change I think we need to make to a few charms
<arosales> I hope yours was well too
<fwereade> arosales, cool, likewise :)
<arosales> good to hear
<fwereade> arosales, it's simple but may be annoying -- it's that several of them use relations with names starting with "juju-" and those are disallowed in juju-core
<fwereade> arosales, I think/hope you're the right person to talk to about how to move forward on changing them all
<arosales> I am
<fwereade> arosales, cool
<arosales> jimbaker: has also been working on reconciling differences in charms going to Go
<arosales> that sounds a bit weird
<arosales> but stated differently jimbaker has been working on the format2 work which is aimed at reconciling the differences in charms between py and Go
<fwereade> arosales, the issue is very simply that every charm implicitly provides juju-info (over the juju-info interface) -- and a charm that provides and requires relations with the same name are obviously insane
<fwereade> arosales, rather than just disallowing juju-info (which I believe is utterly necessary) it seemed smarter to reserve the juju-* namespace
<arosales> makes sense
<jimbaker> catching up
<arosales> fwereade: how do you feel about filing a bug to identify these charm porting issues
<fwereade> arosales, np at all -- the question is how you'd best like it
<arosales> I can then make a task against appropriate charms that need to be changed to support Go
<jimbaker> fwereade, i believe it was our intent to reserve the juju- namespace
<fwereade> arosales, ie what the bug should be filed against
<jimbaker> back when juju-info was first proposed after orlando 2011
<fwereade> jimbaker, excellent, I *thought* something like that had been agreed among, er, at least some of us all
<arosales> jimbaker: it sounds like we still have charm that are using a juju-* relation, correct fwereade?
<jimbaker> fwereade, i will try to dig up, although my thunderbird client has been extremely flaky recently
<fwereade> jimbaker, correct
<fwereade> jimbaker, IIRC the precise restrictions in go are as follows:
<fwereade> 1) no "juju" or "juju-*" relations are allowed
<fwereade> 2) no "juju" or "juju-*" filenames are allowed in the hooks dir
<jimbaker> well it's in our docs, https://juju.ubuntu.com/docs/implicit-relations.html
<jimbaker> "reserved namespace"
<fwereade> 3) no relations can provide the "juju" or "juju-*" interface
<fwereade> jimbaker, ok, that's fantastic -- at least it means that the charms are flat-out wrong and there's little justification for complaints about the change :)
<fwereade> jimbaker, (note that ofc it's fine to *require* over a reserved interface)
<jimbaker> fwereade, it's also restricted by the python code
<jimbaker> correct
<jimbaker> in that require is feasible (as it must!)
<fwereade> jimbaker, awesome -- we are all on the same page then, looks like it's just the charms :)
<jimbaker> i think the only additional restriction here is the juju interface name
<fwereade> arosales, against what would you like the bug to be filed?
<jimbaker> vs juju-*
<fwereade> jimbaker, I forget the reasoning behind that -- probably just that it's more confusing to allow it than to disallow it
<jimbaker> fwereade, we do have one example of the juju charm
<fwereade> jimbaker, (btw: merry xmas and happy new year :))
<jimbaker> where the name is used, although not for relation name/interface name
<fwereade> jimbaker, ah, yes, interesting -- but it doesn't quite collide -- yeah :)
<jimbaker> fwereade, thanks! and i hope it was a wonderful holiday and start of the new year too!
<fwereade> jimbaker, cheers, it was :)
<jimbaker> fwereade, in the case of the juju charm, it is a nice metahack, and makes a lot of sense. so at the least, grandfather that in
<fwereade> jimbaker, yeah, I'm fine with that
<fwereade> jimbaker, it's not disallowed by anything as it stands
<jimbaker> fwereade, cool
<jimbaker> fwereade, one thing that would be useful to do is to update charm proof for this sort of stuff
<jimbaker> such as requiring format: 2 to be declared
<jimbaker> it be warn for now, but err if a special flag is used
 * fwereade looks a bit guilty re format 2 -- he seems to recall someone else taking responsibility for that a few months ago but can't remember who, and sees that there are still open bugs related to it
<fwereade> (in juju-core, that is)
<fwereade> jimbaker, actually, *is* anything actually using format 2 yet?
<jimbaker> fwereade, i'm afraid widespread adoption is lacking
<jimbaker> the reality is that ascii usually works
<fwereade> jimbaker, I'm not certain that's actually such a bad thing, because I *think* most of the charms are working fine under juju-core anyway
<jimbaker> fwereade, where format: 2 presumably is implied
<fwereade> jimbaker, theoretically, yes -- it just seems like almost nothing actually depends on repr-style output
<jimbaker> fwereade, the reality is, you know it if you need it, or just working around via b65
<jimbaker> b64
<fwereade> jimbaker, ha, yes, I'd forgotten that side of it tbh -- the primary source of my own panic was entirely repr-style output
<jimbaker> fwereade, no doubt because it's difficult to parse in a shell script
<fwereade> jimbaker, yeah, lucky for us :)
<jimbaker> fwereade, indeed
<fwereade> jimbaker, just a thought - do the subordinates which use juju-* relations actually work at all under pyjuju?
<fwereade> jimbaker, I can't quite believe either that they do or that they don't ;p
<jimbaker> fwereade, i think the one thing we did get out of the exercise is making it quite well defined. there was no nearly testing of formatting. now there is, for both legacy and format 2
<fwereade> jimbaker, yeah, that's a definite win
<jimbaker> fwereade, they work. the real problem is lack of serialization between primary and subs
<fwereade> jimbaker, ha, that
<jimbaker> fwereade, now perhaps that's reasonable. but there's the special case of apt-get to consider
<fwereade> jimbaker, well, my view is that it's not reasonable at all, but I'm up against niemeyer and rogpeppe on that, so I don't hold out much hope
<jimbaker> so we have been going back & forth between recommending flock to guard, or use aptdcon (and make that a dependency of the juju package)
<jimbaker> fwereade, so the current status in goju is not to serialize then?
<fwereade> jimbaker, (I feel that it makes sense to see juju as ceding administratory-style control over the machine when running hooks, and that having multiple admins logged into the same machine is not a sane thing in general)
<fwereade> jimbaker, the current status in goju is to ignore the problem I'm afraid -- has pyju addressed it already?
<jimbaker> no, we ignore too
<jimbaker> hence the problem
<fwereade> jimbaker, the ?consensus? is that we should provide special apt tools
<fwereade> jimbaker, I think that's crack, so I haven't done it
<jimbaker> fwereade, you mean like a wrapper of apt-get?
<fwereade> jimbaker, (not that anyone has asked me to, but still)
<fwereade> jimbaker, yeah
<jimbaker> or a juju-apt-get?
<fwereade> jimbaker, I'm not quite sure -- there was talk of some sort of declarative magic, which also sounded kinda crackful to me, because you don't know what needs to be declared until you're running the charm, and config can change at any time
<jimbaker> fwereade, maybe that's the right solution - apt-get and friends always are guarded by flock /usr/bin/apt-get
<fwereade> jimbaker, honestly, I don't like that much, I feel that apt-get is a specific (and, ok, overwhelmingly common) case of a general problem
<fwereade> jimbaker, it may be that guarding apt-get fixes close enough to 100% of problems that it's actually the best way to go
<jimbaker> fwereade, well i do think it makes sense to support more declarative stuff in charms, eg charmhelpers
<fwereade> jimbaker, and at least it means we *don't* have to serialize hook execution across processes
<jimbaker> fwereade, maybe the right solution is:
<fwereade> jimbaker, yeah, that which can be done declaratively generally should be -- I just don;t think anyone's actually thought it through in enough detail yet
<jimbaker> we pick a standard lock file for flock
<jimbaker> apt-get and friends are wrapped by guards: flock /standard/file; this is on the path for any hooks
<jimbaker> then if a hook wants to participate in the locking, it also uses flock /standard/file/or/whatever
<fwereade> jimbaker, cautious +1, it is pretty clean
<jimbaker> fwereade, how does that proposal sound? then maybe the declarative stuff in charmhelpers could say: serialize: true, and it would automagically ensure all hooks get this treatment
<jimbaker> so opt-in for hooks, but it solves the biggest problem we have now
<fwereade> jimbaker, yeah, indeed -- would you take that proposal to the lists please? just in case someone else spots anything problematic?
<fwereade> jimbaker, but I think I like it
<jimbaker> fwereade, ok, sounds good to me - it's just an extension of what we've been discussing on #juju, including yesterday - so list discussion is definitely in order
<fwereade> jimbaker, awesome
<hazmat> fwereade, juju-* for interfaces is reserved.. but for rel names.. does it matter?
<hazmat> fwereade, so instead your suggesting they define hooks for the relation but not declare their requirement?
<hazmat> fwereade, its important for those charms to be able to define hooks for those relations.. i'm failing to see why this an issue in go.
<hazmat> effectively every generic subordinate uses them
<hazmat> http://jujucharms.com/interfaces/juju-info
<hazmat> jimbaker, arosales was there something i'm missing here
<hazmat> or are we confusing relation name with interface name
<jimbaker> hazmat, pyju restricts both relation and interface names to not be prefixed with juju-
<jimbaker> hazmat, i don't think there's much confusion here on that
<jimbaker> (if any ;)
<jimbaker> more of an issue that we were verifying goju vs pyju
<hazmat> jimbaker, not true
<arosales> hazmat: it sounded like there was a name space conflict, but perhaps that is not the issue
<hazmat> the not provide juju-info makes sense
<arosales> hazmat: fwereade may have more insights, but EOD for him
<hazmat> the change around no relation name with juju in it will cause some breakage
<jimbaker> hazmat, please clarify what you mean by not true, i suppose my short hand is for charms to provide such relations :)
<hazmat> jimbaker, <jimbaker> hazmat, pyju restricts both relation and interface names to not be prefixed with juju-
<hazmat> jimbaker, that statement is false given the number of charms i linked to which do it
<jimbaker> juju.charm.metadata.MetaData.parse restricts the namespace here
<jimbaker> hazmat, just looking at the code to see what is it says
<hazmat> jimbaker, that only applies to provides
<hazmat> jimbaker, the change he's talking about is to requires as well
<hazmat> jimbaker, the charms i linked require on juju-info and many have 'juju' in the relation name
<hazmat> and thus also hooks with that name
<jimbaker> hazmat, i didn't look at juju in the relation name, but we discuss this in charm proof. i suppose goju could be relaxed in this case
<jimbaker> needs to be discussed here i suppose
<jimbaker> i believe we described this as being restricted to provides, but can revisit w/ fwereade when he's back
<hazmat> there's close to 40 charms there that juju in the rel name.. maybe 15 unique
<jimbaker> hazmat, ack
<hazmat> er.. 24 unique
<jimbaker> so that's good evidence to change goju
#juju-dev 2013-01-05
<fwereade> hazmat, is it reasonable for a charm to have two relations with the same name?
<fwereade> hazmat, IMO it is clearly crack -- you can't specify endpoints sanely, and you can end up running the wrong hooks
<fwereade> hazmat, the fact  that charms currently happen to do it, and we've been lucky enough that the bugs have not yet been hit, is neither here nor there
<fwereade> hazmat, (and the juju-* relation names are (almost?) all juju-info, which is *definitely* crack)
<fwereade> hazmat, certainly the inconvenience of fixing any other juju-* names is minimal compared to the inconvenience of fixing the juju-info ones
<fwereade> hazmat, is there some clever way around the problem that I'm missing?
<hazmat> fwereade, the alternative is recognizing that the implicit provide is special and that hooks are not allowed for it, ie. don't attempt to execute them since those hooks are for the require side, and thus not breaking a bunch of charms.
<fwereade> hazmat, I don't think that holds water -- sure, we could paper over the hooks bug, but I don't see any way in which it's actually sane to allow relation name collisions
<fwereade> hazmat, and AFAICT that's what you're advocating
#juju-dev 2014-01-02
<rogpeppe> mornin' all, and Happy New Year!
<axw> hey rogpeppe, happy new year :)
<rogpeppe> axw: hiya
<rogpeppe> axw: hope you've had a good break
<axw> rogpeppe: yeah it was pretty relaxing, and you?
<rogpeppe> axw: yeah, good thanks.
<natefinch> fwereade, rogpeppe: I guess no meeting this morning?
<rogpeppe> natefinch: i dunno, i'd assumed there would be one, just to get us off to a start
<natefinch> rogpeppe: maybe most people are taking these two days off too?
<rogpeppe> natefinch: that's entirely possible
<fwereade> rogpeppe, would you kjoin us quickly please?
<rogpeppe> fwereade: where?
<fwereade> rogpeppe, https://plus.google.com/hangouts/_/calendar/bWFyay5yYW1tLWNocmlzdGVuc2VuQGNhbm9uaWNhbC5jb20.8sj9smn017584lljvp63djdnn8
<rogpeppe> feeling stupid trying to upgrade my system - i click the "Upgrade" button and nothing at all happens. any clues?
<rogpeppe> hmm, now running do-release-upgrade directly, with crossed fingers
<sinzui> rogpeppe, make sure ot have go installed from upstream or a PPA. trusty has golang 1.2. I haven't had a successful test run using it.
<rogpeppe> lunch
<rogpeppe> sinzui: interesting - i've been using 1.2 for a while and it seems to work ok
<rogpeppe> sinzui: we'll see how it goes - am currently downloading saucy; hopefully will succeed in upgrading to trusty later on today
<rogpeppe> hmm, do-release-upgrade has now hung up
<rogpeppe> last thing it printed was "Setting up adduser (3.113+nmu3ubuntu2) ..."
<rogpeppe> it seems to be running dpkg
<rogpeppe> but dpkg is stripped so difficult to find out what it's *actually* trying to do
<rogpeppe> hmm, looks like it *might* be blocked trying to read from stdin
<rogpeppe> oh, darn, i broke it
 * rogpeppe wishes that do-release-upgrade could continue where it left off
<rogpeppe> ha ha
<rogpeppe> diff: \/etc\/cups\/cups\-files\.conf: No such file or directory
<rogpeppe> diff: \/etc\/cups\/cups\-files\.conf\.dpkg\-new: No such file or directory
<rogpeppe> rebooting
<rogpeppe> well, i now seem to be running Trusty Tahr. that was actually quite painless.
<sinzui> rogpeppe, The only problem I have had with trusty are reports that X/Mir crashed when I didn't see it crash
<rogpeppe> sinzui: i'm hoping that by running it now, my bug reports might actually be useful to someone before the release...
<rogpeppe> sinzui: only problem i've seen so far is:
<rogpeppe> "bzr 2.6.0 is too new for pipeline 1.4"
<sinzui> rogpeppe, we see spurious failures in CI regarding "unable to connect to environment" most often with azure and canonistack. The revision that just landed has never passed on canonistack, and I cannot tell if cloud or juju is at fault. Do you see anything insightful in this log: http://162.213.35.54:8080/job/canonistack-upgrade/452/consoleFull#-2115410258bc9712e0-ba8d-4476-87e1-593fdc4da857
<sinzui> abentley, ^ can rogpeppe use/test the patch to bug 1209131 to get pipelines working again
<_mup_> Bug #1209131: Pipeline reports it is too old for newly released bzr 2.6 <bzr-pipeline:New> <https://launchpad.net/bugs/1209131>
<rogpeppe> sinzui: nothing jumps out at me
<rogpeppe> sinzui: what is wait_for_agent_update actually doing?
<rogpeppe> sinzui: and this line is weird - do you know where it's coming from?
<rogpeppe> <test-release-canonistack> unknown: 1, 0, 2
<sinzui> rogpeppe, wait is sleeping, call status, parse for agent state
<rogpeppe> sinzui: so it succeeds n times, then fails?
<sinzui> rogpeppe, I am not sure about wht the numbers mean. Each is the status of unit + the state-server
<rogpeppe> sinzui: i'm trying to see what kind of failure we're observing here
<rogpeppe> sinzui: is the problem that we're making some change, then can't connect, or is it that we're suddenly unable to connect for some unknown reason?
<sinzui> ah, well maybe I can do better than read the test is doing. I think I can get the log from the upgrade
<sinzui> rogpeppe, This is the all machines log of the failed upgrade test: http://162.213.35.54:8080/job/canonistack-upgrade/452/artifact/artifacts/all-machines-test-release-canonistack.log.gz
<rogpeppe> sinzui: so is this error happening just after an environment has been upgraded?
<sinzui> rogpeppe, yes
<rogpeppe> sinzui: i'm not that surprised then
<rogpeppe> sinzui: when you upgrade an environment, the API server will restart, dropping any existing connections
<sinzui> rogpeppe, CI just passed r2179. The rev is blessed.
 * rogpeppe needs a translation of that :-)
<rogpeppe> sinzui: what does "blessed" mean in this context?
<sinzui> rogpeppe, CI passed r2179 after 5 attempts. We could release it as 1.17.1. The log I am showing you is from the fourth failed attempt
<sinzui> bless == release candidate
<rogpeppe> sinzui: i see, so 1/5 tests passed...
<sinzui> yep.
<rogpeppe> sinzui: i suspect it might pass more often if you waited 20 seconds after upgrading before trying to connect again
<sinzui> rogpeppe, okay. thank you. I will try that.
<rogpeppe> sinzui: another possibility is to treat "connection is shut down" as a soft error, and retry again without failing the CI
<sinzui> rogpeppe, I agree. If you expect the API to drop all connections and let natural error recovery to take over, then the test is too strict
<rogpeppe> sinzui: there is perhaps an argument that the command line should be more resilient about what happens when an environment is upgraded, but it's not an easy problem to solve well.
<sinzui> I am always will to retry. Since the test is attempting to behave as a devop, it should retry
 * rogpeppe is done for the day
#juju-dev 2014-01-03
<rogpeppe> mornin' all
<rogpeppe> hmm, i wish i knew why this computer spontaneously reboots sometimes
<fwereade> rogpeppe, heh, that's quite a significant quirk
<rogpeppe> nate_finch: there seem to be only two tests currently enabled in the replicaset package. what's the reason for that?
<rogpeppe> nate_finch: (for the record, they all just passed for me when enabled, although they did take 32s to run)
<nate_finch> rogpeppe: damn, I had commented out some to do some tests, I guess I forgot to uncomment them :/
<rogpeppe> nate_finch: in general i find it's better to comment out blocks of code with // rather than /* - that way it's much more obvious that they are commented out...
<rogpeppe> nate_finch: wow, this line takes 11s to run
<rogpeppe> 	err := session.DB("local").C("system.replset").Find(nil).One(cfg)
<nate_finch> rogpeppe: weird, I wonder why that is.  In theory you're already connected and the server is up.
<rogpeppe> nate_finch: yeah, it's odd
<rogpeppe> nate_finch: also, this line takes 16s to run: 		session = instances[0].MustDialDirect()
<rogpeppe> nate_finch: and again, the instance should be already up and ready, i think
<rogpeppe> nate_finch: oh no, i tell a lie
<rogpeppe> nate_finch: it's the ping that takes 16s
<rogpeppe> nate_finch: ah, it fails with "no reachable servers"
<rogpeppe> nate_finch: weird, it's succeeding immediately now
<nate_finch> rogpeppe: inconsistency kills me.  so much rather have things fail 100% of the time than succeed 50% of the time :/
<rogpeppe> nate_finch: yeah
<nate_finch> holy crap, I just got a $2000 AWS bill
<natefinch> fwereade, rogpeppe: I don't suppose we have any connections at Amazon that could help me out with the aforementioned astronomical bill?  ^^   I evidently left a single big instance running in us-west, not my normal us-east, so I didn't see it when watching my account.
<rogpeppe> natefinch: oops
<rogpeppe> natefinch: not AFAIK
<rogpeppe> natefinch: but hopefully you'll be able to expense it
 * rogpeppe hopes he hasn't done the same ting
<rogpeppe> thing
<sinzui> natefinch, you might already know this, but in case not, you may need to file the bill under "misc" because there is a cap on the personal "cloud computing" category
<natefinch> sinzui: cool, thanks
<natefinch> sinzui: didn't know that, though it makes sense
<natefinch> well, time to go snowblow the foot or so of snow we got here the last day or two.  I'll be back in an hour and a half or so.
#juju-dev 2014-01-05
<thumper> good morning juju world
<davecheney> bonjour juju-dev
<wallyworld_> g'day
<thumper> o/
<wallyworld_> hi thumper
<thumper> hey wallyworld_
<wallyworld_> good break?
<thumper> too short
<thumper> I think I could have done with another week or two
<thumper> wallyworld_: how about you?
<wallyworld_> thumper: i could have done with another month or too - by the time you catch up with everyone etc, you haven't had a break really
<wallyworld_> two
 * thumper nods
<thumper> I'm trying to remember where I was with all my work
<wallyworld_> me too
<wallyworld_> i have a branch in progress
<wallyworld_> just need to write tests
<thumper> me too
<wallyworld_> but need to do some infrastructure for that
<thumper> and one that I need to land
<thumper> but for some reason the bot isn't updating the dependencies
<wallyworld_> \o/
<wallyworld_> the bot doesn't
<wallyworld_> you need to do it manually if i recall correctly
<wallyworld_> dependencies.sv not used by bot yet
<thumper> :(
<wallyworld_> did you need me to log in and do it?
<thumper> perhaps I should learn :)
<thumper> do we have docs on the bot?
<wallyworld_> um
<wallyworld_> maybe
<wallyworld_> not sure
<wallyworld_> the ip address is 10.55.32.52
<thumper> where is that written down?
<wallyworld_> you need to source some canonistack credentials
<wallyworld_> i just remember it
<wallyworld_> there was an email i think
<wallyworld_> let me search for it
<thumper> hmm...
#juju-dev 2015-01-04
<waigani> menn0: I picked up the nexus 6 in Melbourne over xmas
<waigani> it's huge and I think I'd feel silly talking on it in public, but other than that it's pretty nice
<menn0> waigani: cool. i haven't paid much attention to them. bigger than a nexus 5?
<waigani> menn0: yeah, a lot bigger - it's a phablet
<menn0> phabulous :)
<waigani> lol
<menn0> thumper: regarding the CI blocker (bug 1396099)
<mup> Bug #1396099: AWS/Joyent/HP/manual/maas: juju deploy error "connection is shut down" <api> <ci> <deploy> <ec2-provider> <joyent-provider> <manual-provider> <regression> <juju-core:Fix Committed by wallyworld> <https://launchpad.net/bugs/1396099>
<thumper> ya?
<menn0> thumper: that was fixed by Ian before the break
<menn0> thumper: he changed the certupdater worker to not restart the API server
<menn0> thumper: the new cert is swapped in to the existing server
<thumper> is it working?
<menn0> thumper: the AWS health tests hve been clean for all recent history
<menn0> thumper: so I say we mark it as fix committed
<thumper> menn0: sounds like we are good then
 * thumper agrees
 * menn0 updates ticket
<menn0> thumper: quick look at this please: http://reviews.vapour.ws/r/660/
 * thumper looks
<thumper> shipit
<menn0> thumper: cheers
<lazyPower> o/ Happy 2015 chaps. cheers
<thumper> hi lazyPower
<menn0> waigani: the upgrade step to fix up the minUnits collection is in
<waigani> menn0: nice
#juju-dev 2016-01-04
<wallyworld> axw: thanks for reviews, issues hopefully addressed
<axw> wallyworld: looking now
<blahdeblah> \o all
<blahdeblah> Happy New Year
<axw> blahdeblah: hiya, and to you
<wallyworld> +1
<blahdeblah> Can anyone tell me the name of the API that charms.reactive uses to send data between layers?  Is it charms.reactive.bus?
<wallyworld> blahdeblah: not done any reactive charm development, sorry
 * blahdeblah asks around elsewhere; apologies for the redundancy for those in multiple channels
<wallyworld> axw: thanks for reviews, landing now
<axw> wallyworld: cool
<axw> wallyworld: are metabox going to come through for you this week?
<wallyworld> axw: i sure hope so
<wallyworld> they said NYE that order was underway
<axw> wallyworld: I'll be interested to see it at the next sprint. probably won't get one though, I don't want a discrete card in my next laptop
<axw> dead weight
<wallyworld> axw: yeah, +1, but bublebee turns it off. trouble is, to get everything else (alloy case, 4 DDR4 slots, 2 M.2 slots, 2 HHD slots etc etc) that was the only choice without spending double
<dimitern> morning!
<dimitern> and happy new year :)
<anastasiamac> happy new year, dimitern \o/
<TheMue> happy new year to my ex-colleagues ;)
<dimitern> TheMue, o/
<dimitern> dooferlad, frobware, are you guys around today?
<dooferlad> dimitern: hi, just got here, new computer running :-D
<dimitern> dooferlad, nice! happy new year :)
<dooferlad> dimitern: happy new year!
<dimitern> dooferlad, I got a 4K monitor and now it works way better with the scaling and the past issues I was having are gone
<dimitern> dooferlad, I'd appreciate a look at https://github.com/juju/juju/pull/4029 when you have some time ;)
<dooferlad> dimitern: *click* (email can wait)
<dimitern> dooferlad, cheers!
<dimitern> voidspace, o/ happy new year :)
<voidspace> dimitern: same to you :-)
<dooferlad> dimitern, voidspace,
<dooferlad> dimitern, voidspace: hangout
<voidspace> dooferlad: morning
<voidspace> dimitern: dooferlad: sorry about that
<dooferlad> voidspace: no problem
<voidspace> dimitern: dooferlad: so last year I landed the worker that imports spaces, plus fixed some bugs in it, but it has a few TODOs and needs tests
<voidspace> dimitern: dooferlad: the last couple of days, when you were off, I spent working with Andy on a MAAS problem
<voidspace> dimitern: dooferlad: we probably need to talk to dimitern about that
<voidspace> dimitern: dooferlad: but it can wait for frobware
<dimitern> voidspace, ok, let's chat when andy's back then
<voidspace> dimitern: yep
<voidspace> dimitern: this was the DNS settings not being set on container templates, due to the change to manual network interfaces in 1.25
<voidspace> dimitern: there is a simple fix, but that seems like a huge change that landed on 1.25
<dimitern> voidspace, right! yeah, I remember that
<dimitern> voidspace, we're getting far behind master as well :/
<voidspace> dimitern: we should merge master
<voidspace> dimitern: we'll do it as a merge rather than a rebase though, right?
<voidspace> dimitern: I can propose that
<dimitern> voidspace, yeah, that's safer
<dimitern> voidspace, cheers
<voidspace> dimitern: oooh, conflicts
<voidspace> fun
<voidspace> only 3 files, one of which is dependencies.tsv
<dimitern> voidspace, only 3 files with conflicts? That's not too bad :)
<voidspace> hah
<voidspace> I guess so
<dooferlad> voidspace: master --> feature branch should be done as a "merge commit", which is what "git merge master feature-branch-name" does. https://www.atlassian.com/git/tutorials/merging-vs-rebasing/conceptual-overview seems to be a good doc on the subject that I need to read.
<voidspace> dooferlad: well, all I *can* do is propose a branch for the bot to land
<voidspace> dooferlad: anything else will require frobware
<voidspace> hmmm
<voidspace> dooferlad: if I'm *on* the feature branch - isn't that command the equivalent of "git merge upstream/master"
<dooferlad> voidspace: yes
<voidspace> good, thanks
<voidspace> that's what I've done
<frobware> voidspace, I'm back. 1:1 now?
<voidspace> frobware: hey, hi
<voidspace> frobware: happy new year
<voidspace> frobware: gimme 5 to get coffee
<frobware> voidspace, sure. HNY
<frobware> jam: ping & Happy New Year. Want to catch up later today?
<voidspace> frobware: you still there?
<voidspace> frobware: you left the hangout
<perrito666> morning all
<voidspace> dimitern: frobware suggests that as we have the demo coming up it may be better *not* to merge master unless there's anything we specifically need
<voidspace> dimitern: frobware thinks it's just adding potential breakage
<dimitern> voidspace, frobware, I agree - we can do it after next week
<dimitern> frobware, perrito666, morning btw :)
<frobware> dimitern, morning
<frobware> dimitern, let's use the standup HO
<dimitern> frobware, sure, omw
<frobware> dimitern, don't think the HO issue is at my end as I was just talking with voidspace
<dimitern> frobware, let's try again?
<frobware> dimitern, OK
<mup> Bug #1530840 opened: juju status-history too noisy with update-status <kanban-cross-team> <landscape> <juju-core:New> <https://launchpad.net/bugs/1530840>
<dimitern> frobware, one last try? :)
<mup> Bug #1530840 changed: juju status-history too noisy with update-status <landscape> <juju-core:New> <https://launchpad.net/bugs/1530840>
<mup> Bug #1530840 opened: juju status-history too noisy with update-status <landscape> <juju-core:New> <https://launchpad.net/bugs/1530840>
<dimitern> dooferlad, thanks for the review!
 * fwereade is in london; is rapidly becoming overwhelmed with passport things and bank things and other tedium; is likely to be gone for much of today and weds and fri; marking self out on calendar; but ping me if you need me and you might be lucky
<dimitern> fwereade, safe travels :)
<fwereade> dimitern, cheers :)
<frobware> dimitern, ping. my ISP issues have gone away
<frobware> dimitern, can we sync?
<dimitern> frobware, sure, let's
<dimitern> frobware, I'm in the standup HO
<cherylj> frobware: you around today?
<frobware> cherylj, Happy New Year!
<cherylj> frobware: thanks, you too!
<cherylj> frobware, jam, did a unit test get written for http://reviews.vapour.ws/r/3436/ ?
<frobware> cherylj, I haven't caught up with jam today but I was going to add a unit test today, then we can review and consider for merging. ok?
<cherylj> frobware: sounds good.  Thanks!
<frobware> cherylj, I need to catch up with dooferlad first though...
<cherylj> k
<frobware> dooferlad, HNY && can we sync for 15 mins?
<dooferlad> frobware: give me 5 mins - doing some network configuration
<frobware> dooferlad, ack
<frobware> dooferlad, sounds like my (ISPs) morning. :)
<dooferlad> frobware: ok, ready.
<dooferlad> frobware: I am lurking in https://plus.google.com/hangouts/_/canonical.com/juju-sapphire
<perrito666> mgz:  ping
<ericsnow> katco: standup?
<frobware> dooferlad, omw
<mgz> perrito666: yo
<perrito666> have moment for a priv chat?
<mgz> perrito666: for you...
<natefinch> cherylj: what bugs are we supposed to be working on for bug squad?
<cherylj> natefinch: I'm working on that list now
<rick_h_> cherylj: also please check with alexisb/mramm on some issues the qa folks have getting tests going on the controller branch as potentials please
<cherylj> rick_h_: will do
<rick_h_> cherylj: as we should treat those as 2.0 alpha blockers atm based on some small irc conversations
<cherylj> rick_h_: sounds good.  jog, sinzui,  abentley, do you have bug #s for stuff blocking controller rename branch testing?
<sinzui> cherylj: I was just asking the same question. I expect to know shortly
<cherylj> k, thanks!
<abentley> cherylj: Here are the bugs marked as affecting controller-rename: https://bugs.launchpad.net/juju-core/controller-rename/
<cherylj> thanks, abentley
<frobware> voidspace, please could you take a look over dimitern's PR as we want to land this soon-ish for the demo. thx. http://reviews.vapour.ws/r/3448/
<voidspace> frobware: ok
<frobware> voidspace, thx
<voidspace> it's kind of huge...
<voidspace> dimitern: I'm worried about params.Address having a SpaceName added, does this mean that all APIs returning or taking an address should be version bumped?
<natefinch> cherylj: bug fixes for 2.0 should still be on master, I assume?
<voidspace> dimitern: frobware: reviewed, with several questions/comments
<cherylj> natefinch: yes
<natefinch> cherylj: cool thanks
<frobware> dimitern, do you have some time after the network call to talk about the unit test case for the DNS issue?
<dimitern> frobware, ok, sure
<dimitern> voidspace, thanks!
<jam> frobware: hey, I did intend to meet with you today, it looks like my IRC client didn't actually connect properly today. I got your message from hours ago just recently
<frobware> jam: I futzed around with what turned out to be an ISP throttling issue so it may be related.
<cherylj> frobware, dimitern can you ping me when you're free.  We need to talk about another potential blocker for 1.25.2
<frobware> cherylj, I'm around
<cherylj> frobware, sinzui, mgz, jog:  https://plus.google.com/hangouts/_/canonical.com/cheryl-jennings?authuser=0
<mgz> you named the hangout after yourself?
<mgz> are you the topic? :P
<cherylj> mgz: you should wait until I leave, then you can talk about me
<mup> Bug #1530957 opened: New EC2 Korea region <juju-core:New> <https://launchpad.net/bugs/1530957>
<natefinch> perrito666: can you review me? http://reviews.vapour.ws/r/3452/
<perrito666> well not you, but certainly your code
<natefinch> heh
<natefinch> thx
<perrito666> ship it
<mup> Bug #1530992 opened: Can't remove machine from Juju <juju-core:New> <https://launchpad.net/bugs/1530992>
<mup> Bug #1530992 changed: Can't remove machine from Juju <juju-core:New> <https://launchpad.net/bugs/1530992>
<mup> Bug #1530992 opened: Can't remove machine from Juju <juju-core:New> <https://launchpad.net/bugs/1530992>
<perrito666> wallyworld: awake so early?
<wallyworld> been up for a while :-)
<perrito666> good to see you are trying that not overwork thing....
<perrito666> is standup happening today?
<wallyworld> yep
<perrito666> good, I had to nurse my google calendar back into health after all that shaking near december
<cherylj> Hey axw, did you see bug 1527681?
<mup> Bug #1527681: azure provider does not appear to be opening ports <juju-core:Triaged> <https://launchpad.net/bugs/1527681>
<axw> cherylj: no, I did not. will take a look later on
<cherylj> thanks, axw
<perrito666> axw: you are up too? what is it with you people and getting up so early?
<axw> perrito666: I wouldn't normally, had to administer antibiotics
<axw> perrito666: house full of chest- and ear- infected people
<perrito666> axw: well that is what you get for getting vacation advice from wallyworld
<perrito666> axw: wallyworld says the calendar that anastasia is off, would you guys like to call now? (also it would help me finish my day :p)
<wallyworld> perrito666: sorry, in another meeting
<cherylj> jog: did you guys get a chance to test the destroy-environment --force with 1.8.2?
<cherylj> and checking for leaked container IPs?
<cherylj> jog, mgz, I also attempted to deploy containers with frobware's DNS patch on 1.8.2, and saw that it didn't fix the issue
<cherylj> but it worked on 1.9
<mgz> cherylj: aha
<cherylj> yeah.  Awesome :(
<jog> mgz, did you start a new run of something?
<mgz> jog: yup, see ^, I rebuilt the feature branch with the fix so we had a current run on that
<mgz> given current info it should just be exatly the same
<mgz> wait, < then ^, different channel
<cherylj> ah, I'm not on 1.8.2.  I'm on 1.8.3+bzr4053-0ubuntu1.
<cherylj> not that it'll make too much difference, maybe
<jog> mgz, I had re-run 3430. Containers get past pending with that older build but this new run started before I was able to snapshot the leases file.
<mgz> hm, the problem with the --keep-env thing on short timeout
<cherylj> jog, mgz, I'm wondering if this is a bug with MAAS?    I see that a device gets created for the container, and that the device is removed (or at least doesn't show up on the devices list in the dashboard)
<cherylj> but I don't see a release of the IP
<cherylj> let me try it with 1.9 and see if there's a difference.
<mgz> well, the point is we were not previously exercsing this maas 1.8 networking code at all previously
<mgz> so, even if it's maas not working as api promises, it's our problem because the change is juju using those apis where we didn't before
<wallyworld> perrito666: standup?
<perrito666> going
<jog> mgz, I'll going to try a maas-1_8-deployer run with Juju 1.25.0... do you have anything you plan to kick off?
<mgz> jog, nope, but be aware the current queue isn't empty
<jog> yeah, I see that now
#juju-dev 2016-01-05
<natefinch> waigani: I keep seeing issues about "lingo ls" and misread it as "lingo is" ... and expect it to have the same output as "juju is" :)
<waigani> natefinch: hehe
<mup> Bug #1531064 opened: lxd: cannot create multiple environments <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1531064>
<mup> Bug #1531064 changed: lxd: cannot create multiple environments <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1531064>
<mup> Bug #1531064 opened: lxd: cannot create multiple environments <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1531064>
<frobware> cherylj, I tried adding a container with my patch on 1.8.2 and it worked - worked in that /etc/resolv.conf was populated. Do you not see this on on 1.8.3? (Will install 1.8.3 in some spare time)
<voidspace> dimitern: thanks!
<dooferlad> dimitern: which branches are you wanting reviews on?
<dooferlad> dimitern: the ones that are over two weeks old?
<dimitern> dooferlad, nope, just the last one
<dooferlad> dimitern: on it
<dimitern> dooferlad, the other two will stay like that while I tease bits of them and propose them separately
<dimitern> dooferlad, ta!
<dooferlad> dimitern: you have a review
<dimitern> dooferlad, awesome! thanks
<frobware> dimitern, take a look at https://bugs.launchpad.net/juju-core/+bug/1528217/comments/12
<mup> Bug #1528217: 1.25.2 doesn't set up DNS information with MAAS <blocker> <ci> <maas> <network> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1528217>
<frobware> cherylj, mgz: ^^
<mgz>  frobware: looking
<mgz> frobware: yeah, I get the same on 1.8.3, double space where ip should be and all
<frobware> mgz, so might just be a regression in 1.8.3
<frobware> mgz, debugging some more.
<mgz> I hae a maas checkout somewhere...
<mgz> there's not much in 1.8.3
<frobware> mgz, as in not much different?
<mgz> indeed
<mgz> http://paste.ubuntu.com/14410247
<dimitern> frobware, looking
<dimitern> frobware, hmm.. that means the ConfigType: "dhcp" wasn't taking effect?
<dimitern> frobware, is that the result of changing static to dhcp in the InterfaceInfo ?
<frobware> dimitern, no. That's my same change tested against .2 and .3.
<frobware> dimitern, so 1.8.3, as reported, is behaving differently.
<dimitern> frobware, hmm - yeah, but the lack of address in /e/n/i is troubling - got the logs to compare what happened?
<frobware> dimitern, host and container is still up. Which would help?
<dimitern> frobware, the machine log of the host - looking for things like "created device ..."
<dimitern> frobware, both from 1.8.2 and 1.8.3
<dimitern> frobware, logged the result of claim-sticky-ip-address should follow shortly after creating the device
<dimitern> s/logged the/M-t/
<frobware> dimitern, attached to https://bugs.launchpad.net/juju-core/+bug/1528217
<mup> Bug #1528217: 1.25.2 doesn't set up DNS information with MAAS <blocker> <ci> <maas> <network> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1528217>
<dimitern> frobware, looking
<frobware> dimitern, not sure the 1.8.2 log has sticky in it. hmm.
<dimitern> frobware, ah, sorry - those logs are not enough - machine-0 logs for both are actually better
<dimitern> frobware, as the device registration etc. is handled by the environ provisioner (running on the api server(s) only)
<frobware> dimitern, let's HO
<frobware> dimitern, we can do stuff live
<dimitern> frobware, ok, omw to standup HO
<mattyw> fwereade, ping?
<fwereade> mattyw, pong
<frobware> mgz, cherylj: can you take a look on your 1.8.3 installations to see if the discovered interfaces (maas-eth0) has a default gateway. I'm guessing not. Explanation in: https://bugs.launchpad.net/juju-core/+bug/1528217/comments/15
<mup> Bug #1528217: 1.25.2 doesn't set up DNS information with MAAS <blocker> <ci> <maas> <network> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1528217>
<mgz> frobware: was reading the maas provide code yesterday, there's a branch that checks for default gateway and, ah I see doesn't actually do anything if it's empty
<mgz> difference between our 18 and 19 is somewhat interesting:
<mgz> http://paste.ubuntu.com/14411019
<mgz> http://paste.ubuntu.com/14411028
<frobware> mgz, and the last pastebin was 1.9?
<mgz> frobware: yup
<frobware> mgz, as you say, interesting!
<frobware> dooferlad, voidspace, dimitern: http://reviews.vapour.ws/r/3451/ - PTAL now that we know why it fails on 1.8.3.
<voidspace> frobware: will look shortly, just picking up girl from school
<voidspace> brb
<cherylj> frobware: yes, I see that there is no default gateway for maas-eth0
<frobware> dimitern, can we really just s/network.ConfigStatic/network.ConfigDHCP/ in the provisioner? Apart from fixing up the failing tests is there anything else to be wary of with this change?
<dimitern> frobware, looking at that PR
<dimitern> frobware, I believe it's safe to do that, but only when AddressAllocationEnabled() is false
<frobware> dimitern, right
<dimitern> frobware, well.. and pending a subsequent live tests :)
<frobware> dimitern, makes the tests interesting then
<frobware> dimitern, My sample size of 1 indicates that 1 container comes with as expected.
<frobware> s/with/up
<dimitern> frobware, you've got a review btw
<dimitern> frobware, that's good :) - was that on 1.8.2 without any manual setting of gateways ?
<frobware> dimitern, it had a default gateway. I can try without now. Thanks for the review.
<dimitern> frobware, np - well, without GW we've seen it breaks, right?
<frobware> dimitern, fatally.
<BrunoR> marcoceppi: is there something new regarding http://review.juju.solutions/review/2398 ?
<dimitern> frobware, yeah.. I think at this point we only need to confirm if 1.8.3 vs 1.8.2 is broken "by default" when installed from scratch, and if it is, that settles it (and we have a simple workaround when it breaks - add GW)
<dooferlad> dimitern/voidspace: having checked out github.com/juju/charm and switched to the unsable-v6 branch I tried to run the tests and get a bunch of errors. I have installed the dependencies. Thoughts?
<dimitern> dooferlad, tests in juju/charm or in juju/juju ?
<dooferlad> in juju/charm
<dimitern> dooferlad, paste the errors?
<dooferlad> dimitern: http://pastebin.ubuntu.com/
<mgz> dooferlad: slight problem...
<dooferlad> mgz: ?
<dimitern> no ID :)
<dooferlad> http://pastebin.ubuntu.com/14411294/
<dooferlad> doh!
<dooferlad> I think it may be that charm_test.go helpfully imports gopkg.in/juju/charm.v6-unstable
<dooferlad> instead of, well, "charm"
<dimitern> dooferlad, double-check the branch juju/charm is on is v6-unstable (commit ID 0229ee1
<dooferlad> dimitern: I think it is all about the imports. I checked it out into github.com/juju/charm, not gopkg.in/juju/charm.v6-unstable
<dimitern> dooferlad, hmm.. well gopkg.in/juju/charm.v6-unstable is backed by github.com/juju/charm @v6-unstable
<dooferlad> dimitern: yep, I just find it strange to hack in a gopkg.in directory.
<dimitern> dooferlad, yeah, quirks of gopath
<dooferlad> dimitern: also a quirk of using package thing_test instead of package thing -- you have to import the thing you are testing instead of just being in it.
<cherylj> frobware: could this lack of GW somehow also lead to the leaking of IPs?
<cherylj> or maybe other things are weird with 1.8.3
<dimitern> dooferlad, right; I've confirmed locally that even though I have e.g. github.com/juju/charm and gopkg.in/juju/charm.v6-unstable, the former's tip is an year older than the latter's tip
<dimitern> I guess that's when we switched to gopkg.in import paths
<voidspace> frobware: I can't look at the diff in reviewboard as it seems to conflict
<voidspace> will try github
<cherylj> frobware: can you ping me when you get a minute?
<frobware> cherylj, ping
<frobware> cherylj, have 20 mins before the OS call
<natefinch> cherylj: you can ignore my request to edit that bug squad list page, I just had wanted to mark my bug as fixed, but I should have known you'd be on top of it. :)
<cherylj> frobware: https://plus.google.com/hangouts/_/canonical.com/frobware-cherylj?authuser=0
<dimitern> frobware, dooferlad, voidspace, OS call?
<voidspace> dimitern: omw
<voidspace> dimitern: frobware: dooferlad: if you get a chance please http://reviews.vapour.ws/r/3457/
<katco> natefinch: ericsnow: forgot to mention, i have a doc. appt tomorrow morning overlapping with the standup. bumped it up a bit
<ericsnow> katco: oh yeah, and I have a dentist appt. today :)
<katco> ericsnow: calendar please :)
<ericsnow> katco: added it yesterday :)
<katco> ericsnow: cool ty
<natefinch> katco: I got nothing all week :)
<katco> natefinch: yay :D
<marcoceppi> BrunoR: not at the moment, there are some oddities I'm looking into
<marcoceppi> BrunoR: you may want to join #juju to continue discussion
<dimitern> voidspace, looking
<voidspace> dimitern: thanks
<voidspace> dimitern: they're intended to be complete tests, so it might be worth reviewing apiserver/common/networkingcommon/spaces.go along with it
<voidspace> it wasn't really reviewed the first time round
<voidspace> that's *mostly* moved code though rather than new code
<dimitern> voidspace, reviewed
<voidspace> dimitern: thanks
<voidspace> dimitern: cool
<voidspace> good idea on the line wrapping, they're annoying
<dimitern> voidspace, yeah, cheers
<cherylj> hey ericsnow, I updated bug 1521777 with the behavior needed, and a link to the changes I had done so far
<mup> Bug #1521777: Allow for upgrades to 2.0 <juju-core:In Progress by cherylj> <juju-core 1.25:Fix Committed by cherylj> <https://launchpad.net/bugs/1521777>
<ericsnow> cherylj: thanks
<cherylj> couldn't update the card, I got a permission error
<katco> natefinch: reviewed your pr
<frobware> cherylj, dimitern: I did a clean install of MAAS 1.8.2 and I get not default gateway for the discovered maas-eth0 interface, even when updating the Cluster controllers managed network.
<frobware> s/and I get/and I do NOT get/
<dimitern> frobware, hmm this means both 1.8.2 and 1.8.3 are consistently broken then
<frobware> dimitern, I would have to try 1.9 to see if it is the same.
<dimitern> .. unless the default GW is manually set
<dimitern> frobware, +1
<frobware> dimitern, we have not noticed because containers were using dhcp.
<mgz> prettty sure 1.9 is set up with the bits as before
<frobware> mgz, question is was the default gateway set manually?
<mgz> perrito666: pr #3408 does not fill me with joy
<mgz> both requiring provider api calls in an upgrade step and putting uuids in a string field seem horrible
<perrito666> that is the number of github or rb?
<mgz> perrito666: my bad, review ^, pr #3986
<perrito666> oh, good because 3408 is quite nice and you where hurting my feelings
<perrito666> mgz: well 2 things
<perrito666> 1) google does not provide any other way, because of reasones
<perrito666> 2) api calls in upgrade steps are about to become a thing, since we suddenly need to change stuff upstream to support big changes like multienv same name
<mgz> perrito666: we don't really expect upgrade steps to fail
<mgz> this absolutely can
<perrito666> that said, I accept any form of criticism and will try to deal with it :)
<perrito666> mm, we already have steps like these for storage, definitely something we will have to deal with
<perrito666> ftr I think that not expecting anything to fail is wishful thinking on our part
<mgz> meh, and I haven't got good answers on the naming that aren't major infra changes
<mgz> perrito666: but line 65 in provider/openstack/upgrade.go is certainly a problem
<mgz> you're not even logging/propogating the nova error
<perrito666> mgz: what pr is this now?
<mgz> the openstack sec group change
<perrito666> mgz: line 65 is return errors.Annotatef(err, "upgrading security group name from %q to %q", group.Name, newName)
<mgz> right
<perrito666> aaand that err is the one that comes from calling nova?
<mgz> provider returns an error, the step fails part way through upgrade and we lose the provider error
<perrito666> that error bubles up all the way to the upgrade step, I feel like I am missing something here
<mgz> sorry, we don't lose the error, but we don't complete the step
<perrito666> well, I assumed that if the step failed and given that the identity of the upgradestep function allowed for an error that was the way to proceed: stop and err out
<mgz> probably, but we don't retry steps till a new version so life seems complicated
<perrito666> mmm, perhaps I should add some redundancy to my own step in that case
<mgz> well, I'm not sure, I don't think individual steps should
<mgz> but the framework above and below probably should
<perrito666> you seem to be luring me into adding redundancy to upgrade steps
<mgz> no, but I'm pointing out an issue with it given how things work at present
<perrito666> sad, you could easily play the part of a british elf luring inocent devs into complex features :p
<frobware> cherylj, switch the provisioner to use DHCP again on 1.25 works based on some very limited testing. and it works without having to worry about having a default gateway. We need to decide how we should proceed from here.
<frobware> cherylj, (and some of that was almost a real sentence.) LOL
<frobware> cherylj, I can continue with the move back to DHCP and fix the unit test failures but as we've seen most of the failures have come from live usage.
<frobware> cherylj, I'm trying to avoid fixing the fix ... which is fixing another fix, and so on.
<cherylj> frobware: I'm chatting with the maas guys now about the IPs leaking for devices... I can ask them about the default behavior with 1.8.2 and the default gateway
<cherylj> frobware: If it's the case that 1.8.2 has the same behavior that by default, there's no default gateway, then I think we need to use the DHCP again.
<cherylj> We can't make users specify a gateway to be able to use juju after an upgrade.
<frobware> cherylj, playing devils ... OK
<cherylj> does that sound reasonable?
<frobware> cherylj, as an end-user, sure. :)
<cherylj> frobware: haha, yeah :)
<frobware> cherylj, I think we should consider landing my change anyway.
<frobware> cherylj, and on top of that we switch back to DHCP. Thoughts?
<cherylj> frobware: yeah, that was my assumption
<perrito666> anyone very savvy on hooks willing to spare a few moments with me?
<frobware> cherylj, I do get a default gw with 1.9
<frobware> cherylj, that was noted on the back of a clean fresh 1.9 install
<cherylj> frobware: ok, thanks for checking
<cherylj> frobware: have you had the chance to try 1.8.2?
<frobware> cherylj, yes. 1.8.2 no default GW. ditto for 1.8.3.
<frobware> cherylj, my 1.9 was for completeness.
<cherylj> ah, ok
<frobware> cherylj, I vote for 1.9. :)
<cherylj> has 1.9 even been released yet?
<frobware> cherylj, I *think* so. I now have: 1.9.0+bzr4533-0ubuntu1
<cherylj> ah, ok
<frobware> cherylj, I previously had MAAS Version 1.9.0 (rc4+bzr4533)
<frobware> cherylj, I need to EOD. Tomorrow I'll switch to DHCP, address the unit tests and perhaps we create another feature branch for testing.
<cherylj> frobware: ok.  And just fyi - regarding the IP addresses leaking:
<cherylj> mpontillo> cherylj: ah, in 1.8.3 it might be different - I was thinking 1.9. in 1.9 we changed the internals of how we associate IPs with devices, so it may work better
<frobware> cherylj, do we know if it leaks with 1.8.2 as well?
<cherylj> not yet
<cherylj> the qa team hasn't had the chance to downgrade the MAAS
<cherylj> frobware: if you switch to DHCP, would that eliminate using devices to register container IPs?
<frobware> cherylj, need to double-check with dimiter
<cherylj> frobware: ok, because if the device IPs are not released when the devices are deleted, we may need to back out that change.
<frobware> cherylj, so the change works for 1.9, doesn't this then become a 1.8 issue?
<frobware> cherylj, I'm now confused. Back out which change?  Because we want IPs to get cleaned up.
<cherylj> frobware: technically, yes,  but we can't ship something that breaks people running on 1.8
<cherylj> frobware: the change that registers containers as devices.
<cherylj> frobware: if you need to EOD, we can continue this conversation tomorrow
 * frobware maÃ±ana 
<cherylj> jog, mgz, do you know if the 1.8.3 maas you guys are using has a static IP range defined?
<jog> cherylj, see #maas... yes
<katco> natefinch: did you see my review?
<natefinch> katco: yes, was just about to ask you about it
<natefinch> katco: are side comments a problem?  I actually find them to be easier to read because they don't get in the way of the code
<katco> natefinch: not a problem, i just don't think they conform to go's standard practices
<katco> natefinch: i don't think the tooling picks them up, only top comments
<katco> natefinch: also the comments don't conform to the standard anyway (lead with variable name)
<natefinch> katco: when you have multiple comments in a block like that, they get printed out enmass: https://godoc.org/github.com/natefinch/godocgo/sub#pkg-constants
<katco> natefinch: if this weren't go, i'd say side-comments were the way to go here
<natefinch> katco: also, pretty sure I've seen side comments on std lib constants in similar situations.... couldn't point you at it currently.
<natefinch> katco: finally, the point of starting with the name of the thing is so grep will pick up the comment with what it's commenting, which will still work here, since it's on the same line
<katco> natefinch: the two official sources i can find say to do doc comments: https://golang.org/doc/effective_go.html#commentary http://blog.golang.org/godoc-documenting-go-code
<natefinch> katco: hmm, can't find an example of constants with side comments in the stdlib, only struct fields.    meh.  I can do top comments, I'm not married to it
<katco> natefinch: fwiw i agree with what you're saying; but consistency > preference
<natefinch> katco: yep, that's fair.
<katco> natefinch: haven't looked; have you reviewed all of ericsnow's patches? i'm moving onto his now
<ericsnow> katco: sorry (forgot), I've landed the first one and the second is merging
<ericsnow> katco: working on the third now
<katco> ericsnow: oh, did they already have ship-its?
<ericsnow> katco: yep
<katco> ericsnow: oh cool, so no review needed?
<ericsnow> katco: perhaps on the remaining one of mine?
<katco> ericsnow: sure thing
<katco> ericsnow: listresources endpoint?
<natefinch> katco: yes, I reviewed all his and gave them shipits pending addressing of review comments
<katco> natefinch: cool, gj, ty
<ericsnow> katco: yep
<ericsnow> katco, natefinch: I also found an intermittent bug in the feature branch and now have a patch up:  http://reviews.vapour.ws/r/3458/
<katco> ericsnow: awesome! good find :)
<ericsnow> katco: it was annoying me :)
<katco> ericsnow: natefinch: do you have time to hop into moonstone? i have a resources question
<ericsnow> katco: sure
<natefinch> katco: yep, one sec
<natefinch> ericsnow, katco:  Dang, should have looked a little further down.  I still don't like using stub's CheckCallNames because it tests the order of the calls, which is an implementation detail that could change, causing the test to fail erroneously.  But that's more of a comment about how the stub itself works, not this change.
<ericsnow> natefinch: checking the order of calls is exactly the point though
<ericsnow> natefinch: we want to ensure that the internal boundary is used in the right order
<natefinch> ericsnow: but it's not. it's testing like a dozen things that aren't actually required for the function to work correctly.
<natefinch> ericsnow: I've had to change tests that shouldn't have failed because the order of function calls changed without the actual logic changing.
<ericsnow> natefinch: I'm not clear on how the order of calls could change without the logic changing
<natefinch> ericsnow: I forget the exact example, but it was basically two function calls that could have been done in any order, and switching them caused a test to fail.
<natefinch> ericsnow: but in this case, we don't have a requirement that the files be uploaded in the same order as they are given on the command line.... except now there's a requirement enshrined in the test that says it must be so... so if we decide we want to upload the smallest one first, or upload them alphabetically or concurrently.. this test has to change.
<ericsnow> natefinch: correct
<katco> ericsnow: natefinch: the function calls would have to be associative (e.g. multiplication or something)
<ericsnow> natefinch: it will reflect the interaction with the dependencies
<natefinch> ericsnow: but it's not a requirement, and you've made it one. The only requirement is that tthe results of openresource get passed to upload
<ericsnow> natefinch: the point of the patch is that the order *does* matter
<natefinch> ericsnow: also, it's now a requirement that we create the client before opening the first resource... why?
<natefinch> ericsnow: it makes the test care too much about very specific details of the order of calls in the implementation that it shouldn't care about.  It makes the tests fragile.
<ericsnow> natefinch: testing the behavior of a function relative to its internal interface (dependencies) is just as important as relative to its external interface
<ericsnow> natefinch: is your concern strictly with testing the order of interaction with dependencies?
<ericsnow_afk> dentist
<natefinch> ericsnow_afk: well, yes.  We should be checking that what is returned from OpenResource is sent to upload for the respective resource... and if somehow the code can go back in time and do the upload first, well, good for it ;)
<natefinch> I wish I could go back and make my reviews into a "fix it, then ship it!"
<natefinch> katco: btw, thanks for the review on my stuff
<wallyworld> menn0: hi, welcome back. quick one - this bug has been fixed in recent jujus i think, right? ie we clean up orphaned txns bug 1528261
<mup> Bug #1528261: jujud won't start, log shows a lot of "cannot set addresses of machine XXX: cannot find transaction ObjectIdHex("xxxxxx"); <juju-core:Incomplete> <juju-core 1.20:Triaged> <https://launchpad.net/bugs/1528261>
<menn0> wallyworld: hi! looking
<menn0> wallyworld: there's a few things with that
<menn0> wallyworld: all the unnecessary addressupdater txns have been fixed
<menn0> wallyworld: and we do clean up the txn collection
<wallyworld> menn0: the issue arrose when mongo got its replicaset all screwed up
<menn0> wallyworld: but the pancis in the ticket aren't fixed automatically in newer juju verisons
<wallyworld> ok, at least we're part way there
<menn0> wallyworld: someone will need to shut down the state servers, get the replicaset working again and then run mgopurge to fix the txns
<wallyworld> ok, ty
<wallyworld> niedbalski: ^^^^^^^
<wallyworld> i'm not sure of mario's nic to tell him also
<niedbalski> Mmike, ^^
<Mmike> thxn!
<wallyworld> Mmike: niedbalski: so an upgrade to 1.24 or 1.25 is required plus the mgopurge
<Mmike> how do I upgrade when I can't get jujud started?
<wallyworld> i think running mgopurge first will help, well hopefully?
<wallyworld> so shut down state servers, fix replicaset, run mgopurge
<menn0> Mmike, niedbalski, wallyworld: the upgrade to 1.24 isn't a strict requirement to get this env working again. It'll just help reduce the chance of it happening again.
<wallyworld> yes
<Mmike> ack
<Mmike> where do I find mgopurge?
<menn0> Mmike: I'll compile it for you... it's not easily available
<waigani> axw or menn0: this needs a review when you have a moment please: http://reviews.vapour.ws/r/3422
<wallyworld> menn0: we should consider packaging that binary with juju, like we do for plugins
<Mmike> menn0, thnx!
<Mmike> menn0, no rush, it's almost 11PM here and I'll doze off shortly
<menn0> Mmike: ok. how shall I get it to you?
<menn0> wallyworld: that could work.
<Mmike> menn0, how large is it? Email, if it's not largish, or put it to mombin somewhere and allow me to access it? (mario is the username)
<wallyworld> menn0: yeah, would be handy as i suspect we'll need it again
<menn0> Mmike: it's not too big
<Mmike> then email could do it
<Mmike> wallyworld, menn0, thnx, lads!
<Mmike> I'm dozing off now.
<wallyworld> Mmike: np, menn0 did all the work :-)
<Mmike> wallyworld, you did the connecting :) If you were in sales... :)
<wallyworld> lol
<perrito666> wallyworld: I might be a couple of mins late for the standup, grocery shopping
<wallyworld> sure, np
<perrito666> wallyworld: axw alexisb ready
<alexisb> wallyworld, I am running late
<wallyworld> np
<wallyworld> perrito666: we are here already
#juju-dev 2016-01-06
<wallyworld> anastasiamac: pretty please http://reviews.vapour.ws/r/3459/
<anastasiamac> wallyworld: looking :D
<wallyworld> ty
<wallyworld> anastasiamac: commeted
<wallyworld> hopefully it makes sense
<anastasiamac> wallyworld: looks awesome \o/ thank u - shipited it
<anastasiamac> :D
<wallyworld> yay :-) ty
<menn0> axw: ship it
<axw> menn0: cheers
<menn0> axw: of course, now I see that wallyworld beat me to it
<menn0> oh well... double ship it
<axw> :)
<wallyworld> anastasiamac: CI should run structured metadata tests shortly, let's keep fingers crossed
<wallyworld> axw: anastasiamac: new laptop arrived :-D will be offline for a bit to swap hard drive and stuff
<axw> wallyworld: excellent. enjoy
<wallyworld> i will :-D
<natefinch> axw: are you familiar with this test? https://github.com/juju/juju/blob/controller-rename/featuretests/cmd_juju_controller_test.go#L118
<axw> natefinch: nope, maybe waigani or menn0
<axw> both of whom are probably EOD now ...
<natefinch> weak
<waigani> axw hey, sorry I was having dinner
<axw> waigani: no problem, I don't think anybody expects you to work through dinner ;)
<waigani> heh
<waigani> axw is there something up with that test?
<axw> waigani: no idea, sorry. all I know is that natefinch wants to know about it :)
<waigani> oh right, hehe, I mis read comment - thought you were looking for me
<waigani> and he's gone - oh well. tomorrow is another day. He can catch me then.
<axw> anastasiamac: FYI, https://github.com/juju/juju/pull/4042
<anastasiamac> axw: looking :D
<mattyw> good morning juju
<TheMue> good morning mattyw
<mattyw> TheMue, hey hey! You don't belong here ;)
<mattyw> TheMue, happy new year mate, how's it going?
<TheMue> mattyw: it's a public channel, I'm still interested, so I never left :)
<TheMue> mattyw: happy new year to you too.
<mattyw> TheMue, very happy to have you around, how's erlang?
<TheMue> mattyw: technology at new job is fine, but the team is totally missing any kind of process or common habbits like iterations, retrospectives, reviews, etc.
<TheMue> mattyw: it's a fight to push it into the right direction
<mattyw> TheMue, that's awesome - gives means you can implement it the way it should be done :)
<TheMue> mattyw: got aware how good we're doing the job here at juju
<voidspace> dimitern: you really need to run the git pre-push hooks
<voidspace> dimitern: there's a Warningf call with no formatting directive on maas-spaces tip...
<dimitern> voidspace, oh is there..
<voidspace> dimitern: corrected it in the branch I'm about to merge (the one you reviewed yesterday
<voidspace> dimitern: :-p
<dimitern> yeah I was meaning to set up the hook
<voidspace> :-)
<dimitern> at least now with the new laptop I have no excuse like before - everything's much faster
<voidspace> cool
<voidspace> dimitern: your first comment on this review: http://reviews.vapour.ws/r/3457/
<voidspace> dimitern: "please fix the imports grouping" - it looks to me like the grouping is correct
<voidspace> dimitern: non juju/juju in one group and juju/juju separately
<voidspace> ah
<voidspace> dimitern: no - juju/testing is wrong
<voidspace> dimitern: never mind
<mup> Bug #1531444 opened: azure: add public mapping of series->Publisher:Offering:SKU <juju-core:Triaged> <https://launchpad.net/bugs/1531444>
<mup> Bug #1531444 changed: azure: add public mapping of series->Publisher:Offering:SKU <juju-core:Triaged> <https://launchpad.net/bugs/1531444>
<dimitern> ah, so I did have pre-push hook, but the symlink was broken in .git/hooks/
<voidspace> heh
<voidspace> that doesn't help
<dimitern> now I have it on and it did catch that warning
<mup> Bug #1531444 opened: azure: add public mapping of series->Publisher:Offering:SKU <juju-core:Triaged> <https://launchpad.net/bugs/1531444>
<dimitern> voidspace, dooferlad, frobware, when you have some time, I'd appreciate a review on http://reviews.vapour.ws/r/3462/
<dimitern> it might be easier to also look at the PR's individual commits
<dooferlad> dimitern: https://github.com/juju/charm/pull/186 -- I hope this is what you wanted!
<dimitern> dooferlad, cheers, looking
<dimitern> uh oh "No description provided."
 * dimitern is surprised Storage .. `yaml: ",omitempty"` works for "storage: ..."
<dimitern> I thought only bson lowercases the field names
<dooferlad> I think json and YAML parses both convert to lowercase on output and interpret lowercase as Title Case on input. Haven't tested it though. Not sure if all-caps becomes all lowercase.
<dimitern> dooferlad, reviewed
<dimitern> and please, add a PR description
<dimitern> e.g. describing the added section with an example YAML excerpt?
<dimitern> dooferlad, also, adding cards + PR link for that PR and the following 2 would be great
<dooferlad> dimitern: reviewed your pull request. Basically, I am on a TODO needs a bug or card quest at the moment.
<jam> axw: are you still around?
<jam> I'm running into problems with maas 1.9 and KVM nodes not detecting storage during commissioning
<dimitern> dooferlad, fair enough
<dooferlad> dimitern: Will catch up with cards later and get a decent description in. Lunch now.
<dimitern> dooferlad, sure, np
<jam> dimitern: have you seen ^^
<voidspace> frobware: dimitern: dooferlad: http://reviews.vapour.ws/r/3463/
<dimitern> jam, I'm using KVMs on vmaas 1.9 all the time, and haven't seen that issue (even recently had to recommission all the nodes)
<dimitern> voidspace, looking
<voidspace> tnx
<dimitern> jam, haven't yet tried 1.9.0 though - still on 1.9rc4
<jam> k
<jam> it might be fine on regular MAAS and it is a problem with storage detection on KVM instances.
<dimitern> voidspace, reviewed
<voidspace> dimitern: ta
<voidspace> dimitern: heh :-)
<voidspace> should have done that already
<dimitern> :)
<dooferlad> dimitern: could you help me out with what I am supposed to be doing next? I am not clear on what the next steps should be. Perhaps a hangout where we spec out some cards?
<dimitern> dooferlad, sure, let's HO in like ~20m if you can?
<dimitern> I just got some food
<dooferlad> dimitern: perfect.
<dimitern> dooferlad, hey, I added 2 cards and tried to describe the next steps - can you see if that's enough for you to go on with it?
<dooferlad>  dimitern: yep
<dimitern> if not, let's HO I guess
<dooferlad> dimitern: will have to actually dig into code before I can tell if there is enough info in those cards, but they look good
<dimitern> dooferlad, sure
<dooferlad> dimitern: if you could mrege my changes (https://github.com/juju/charm/pull/186) that would be great. I don't have write access.
<dimitern> dooferlad, I don't either, but mgz confirmed the charm repo is under the merge bot, so $$merge$$ will take care of it ... however - I've added one last comment
<dimitern> dooferlad, sorry to be a pest, but "service wordpress wants to bind to endpoint foo to space bar, which isn't defined by the charm" needs rephrasing
<dooferlad> dimitern: ok
<dimitern> dooferlad, e.g. to `service "wordpress" wants to bind endpoint "foo" to space "bar", which isn't defined by the charm`
<dimitern> dooferlad, do you mind doing a follow-up for that?
<dimitern> or something like that
<dooferlad> dimitern: sure
<dimitern> the important bit is to quote names so they stand up in the message
<dimitern> dooferlad, maybe even ", but the endpoint is not defined by the charm" ?
<dooferlad> dimitern: yea, sounds good]
<dimitern> dooferlad, tyvm
<dooferlad> dimitern: https://github.com/juju/charm/pull/187
<dimitern> dooferlad, reviewed
<dooferlad> you and your %q
<natefinch> %q is one of my favorite features of Go :)
<dimitern> :) it's from experience - looking at MBs of logs and trying to find an issue quickly
<perrito666> natefinch: you are happy with little
<dimitern> dooferlad, thanks
<natefinch> perrito666: sometimes :)
<dooferlad> natefinch: try "%# v" some time -- it will blow your mind.
<natefinch> dooferlad: oh, I know :)
<natefinch> dooferlad: but sometimes it's the little things that are nicest.  Like auto-quoting the parameters in a print string :)
<natefinch> cherylj: got a fix for #1529126, I see your name on git blame, care to do a super quick and easy review? http://reviews.vapour.ws/r/3464/diff/#
<mup> Bug #1529126: cmdControllerSuite.TestControllerDestroy unittest <ci> <intermittent-failure> <test-failure> <unit-tests> <juju-core:Incomplete> <juju-core controller-rename:Triaged by natefinch> <https://launchpad.net/bugs/1529126>
<dooferlad> dimitern: https://github.com/juju/bundlechanges/pull/15
<dimitern> dooferlad, looking
<dimitern> dooferlad, LGTM
<cherylj> natefinch: sure, looking now
<cherylj> ah, this is the stuff waigani added for the undertaker work he did
<voidspace> dimitern: heh, if you rename your maas default space you get warnings from the juju client
<voidspace> dimitern: probably on every operation...
<voidspace> my default space is not called "default"
<natefinch> voidspace: such a noncomformist
<voidspace> natefinch: always! :-D
<dimitern> :)
<dooferlad> mgz: https://github.com/juju/bundlechanges/pull/15 seems to be being ignored by the merge bot. Is it still under bot control?
<mgz> still?
<mgz> it's the jujugui gating
<mgz> not ours
<dooferlad> ah, my mistake.
<mgz> see the previous prs for how that works
<mgz> :shipit: I believe
<dooferlad> thought I had seen our bot, but maybe I just have too many github tabs open :-)
<dooferlad> mgz: thanks, that was it.
<marcoceppi> I need help, upcoming demo to MS about new azure provider today, I can't bootstrap: http://paste.ubuntu.com/14422068/
<marcoceppi> 1.26-alpha3-trusty-amd64
<marcoceppi> katco: do you know who I could bother about this^?
<katco> marcoceppi: hey, let me take a look at the pastebin. unfortunately this is axw's realm i believe
<marcoceppi> katco: oh, axw is eod isn't he?
<katco> marcoceppi: he's asleep right now (aus)
<marcoceppi> yup :(
<katco> marcoceppi: as an aside, best thing to do in these situations is ping vanguard in #juju on private network
<marcoceppi> I can't get the old provider to give me the right instances and I seem to be having issues creating a networkcontroller
<marcoceppi> katco: I think I need to finish reading the instructions
<katco> marcoceppi: ok. most interesting thing right now is the final error message. curious if you could manually do a put to this address with correct info: https://management.azure.com/subscriptions/776757db-9bc9-43bc-994e-761d0ce6c309/resourceGroups/juju-azure-west-environment-26ac52cb-0709-424a-891a-0e41adffbb8d/providers/Microsoft.Network/virtualnetworks/juju-internal?api-version=2015-05-01-preview
<marcoceppi> katco: I need authorization
<katco> marcoceppi: (sad trombone)
<marcoceppi> katco: but later in the release notes I notice it mention "registering components"
<marcoceppi> and network is one of them
<marcoceppi> trying that now
 * marcoceppi forces hand onto face
<marcoceppi> katco: it's always when I ask for help, that I find the answer, I didn't have networking enabled on my account
<katco> marcoceppi: :) not a problem at all
<katco> marcoceppi: just glad it's working
<frobware> cherylj, can we do a quick HO to verify whether I see IP addrs being released in my 1.8.2 setup
<cherylj> frobware: sure:  https://plus.google.com/hangouts/_/canonical.com/maas1-8-2?authuser=0
<frobware> cherylj, do we want a separate bug for the lack of gateway?
<frobware> cherylj, as you say, bug 1528217 is really about lack of DNS
<mup> Bug #1528217: 1.25.2 doesn't set up DNS information with MAAS <blocker> <ci> <maas> <network> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1528217>
<cherylj> frobware: good point.  I'll open one up for you, give me a few
<natefinch> ericsnow: in your review tyou say there can only ever be one of a resource, but that seems impossible.  Any time you upload a new revision of a resource, you'll by definition have at least two: the old one, which some of the units will still be using, and the new one, which the units will gradually switch over to.
<ericsnow> natefinch: when you upload it replaces the one that was there
<natefinch> ericsnow: but units are still using that one
<ericsnow> natefinch: ah, right, but that has not been addressed yet in the spec
<ericsnow> natefinch: let's keep it simpler for now
<natefinch> ericsnow: I guess.  I'd bet money that we'll want the list sorted... and in fact, even in the spec right now, it appears that the resources from the store are sorted to the top, though of course that may just be coincidence.
<ericsnow> natefinch: yeah, it may be that the store resources should be shown first
<ericsnow> natefinch: for now let's just go unsorted and ask for clarification since it's ambiguous
<ericsnow> natefinch: (mostly)
<natefinch> ericsnow: that's fine, it's vcs, I can always get the code back.
<ericsnow> natefinch: exactly! :)
<ericsnow> natefinch: I've done that a time or two :)
<natefinch> man I hate that the format framework stuff uses interface{} and then immediately requires you to typeconvert to the since type it must be.
<natefinch> s/since/single
<natefinch> ericsnow: so about the FormattedCharmResource and FormattedSvcResource - I disagree that they're not structured data.  There's quite a bit of logic in the FormatSvcTabular function that needs to transform the structured data into unstructured text.  It's less true of FormatCharmTabular, but I'd have to be switching over a raw string in FormatSvcTabular for Origin at the very least.  Origin and DataType are enums, even in this data, it's only when t
<natefinch> hey're output that we make them into strings.
<natefinch> I do 100% agree that the lower() function should be in the formatter, not on the data types, though.
<ericsnow> natefinch: I don't think the logic belongs in the outputter
<natefinch> ericsnow: isn't that the definition of what it's for?  Take this data and spew it out in a specifc way.
<ericsnow> natefinch: its only job it to convert the formatted data into the output we desire
<natefinch> yes. we need logic there to know how to merge logical columns into visible columns.
<ericsnow> natefinch: it isn't to format the data items, but to present (combine) them in a particular way
<natefinch> and we should be basing the logic of how to combine the columns on strongly typed data, not strings that happen to match some other string
<ericsnow> natefinch: if the type matters then we should handle it in the formatter
<natefinch> The format struct defines the logic columns of data. The outputter uses those to decide how to output the actual columns.  Putting the logic of how to combine the columns in the format struct is mixing concerns.  The format struct shouldn't have logic. That's why I think you were right that the lower() functions belong in the outputter code, not on the struct's data types.
<perrito666> marcoceppi: did you ever try to deploy openstack using lxd/lxc only? I recall you where doing something like that
<marcoceppi> perrito666: wayne and paige were working on it, I never really got much into it
<perrito666> mm, I might try to finish that, I am in need of an openstack
<natefinch> I guess the question is, do you want the service resource's wacky revision/timestamp column printed out in json and yaml?  If yes, then it should be in the struct... I don't think it belongs there.
<cherylj> alexisb, dimitern, frobware, it is worth noting that master has this problem as well, since the fix for bug #1483879 landed there as well
<mup> Bug #1483879: MAAS provider: terminate-machine --force or destroy-environment don't DHCP release container IPs <bug-squad> <destroy-machine> <landscape> <maas-provider>
<mup> <sts> <juju-core:Fix Released by dimitern> <juju-core 1.24:Won't Fix> <juju-core 1.25:Fix Released by dimitern> <https://launchpad.net/bugs/1483879>
<frobware> cherylj, ack
<cherylj> since master is still in devel, we can probably get away with just noting the problem in the release notes.
<cherylj> and not back out the fix.
<natefinch> cherylj: ping on that review?
<cherylj> natefinch: yeah, I'm still looking.  For some reason, I've got a lot of meetings today :)
<natefinch> cherylj: no problem, thought maybe you'd just forgotten to hit publish or something. No rush.
<cherylj> natefinch: just to sanity check - did you re-run the race check with your changes?
<natefinch> cherylj: I did :)  But a very good question to ask.
<cherylj> yay!
<cherylj> natefinch: shipit!
<natefinch> cherylj: yay, thanks! :)
<mup> Bug #1531589 opened: debug-log does not work with local provider on xenial + 1.25.0 <juju-core:New> <https://launchpad.net/bugs/1531589>
<mup> Bug #1531589 changed: debug-log does not work with local provider on xenial + 1.25.0 <juju-core:New> <https://launchpad.net/bugs/1531589>
<mup> Bug #1531589 opened: debug-log does not work with local provider on xenial + 1.25.0 <juju-core:New> <https://launchpad.net/bugs/1531589>
<natefinch> ericsnow: made a compromise on the enums - they're string enums now, so I can remove all the BS with making them a stringer and the marshal functions etc.  Really cleans up the code.  I wanted to keep them with specific named values however, so the logic in FormatSvcTabular can switch over real values.  Also, the conversion functions from resource to output is actually fairly valuable buffer to protect us from those values ever changing.
<ericsnow> natefinch: how is the matter handled in "juju status"?
<natefinch> ericsnow: let me go find the right code under the status component.... OH WAIT.
<ericsnow> natefinch: haha
<natefinch> ericsnow: which is to say: it'll take me a big to find the right bit of code
<natefinch> s/big/bit
<ericsnow> natefinch: cmd/juju/status/tabular_output.go
<perrito666> natefinch: if it makes you feel better, even if status where a component, it would still suck
<natefinch> me: who the heck wrote this?  <git blame>
<natefinch> 2a763456 (Eric Snow         2015-07-28 09:41:01 -0600  32)      p := func(values ...interface{}) {
<perrito666> * blame is destroying friendships since forever
<natefinch> haha
<katco> natefinch: =|
<natefinch> anyway.... :)
<perrito666> wait until it degenerates into blame hunts
<natefinch> honestly, I think git blame has like a 30% chance of coming up eric just based on line count alone
<perrito666> git blame lies though
<natefinch> yeah, true true
<perrito666> if eric moved the line it will blame him
<natefinch> merges and stuff
<natefinch> ericsnow: the answer is... I don't know.  I don't really see any enums in the status structs... but maybe some of them are and I just can't tell because someone made them into raw strings. :D
<ericsnow> natefinch: I'm (just) guessing that the formatter did all the work :)
<perrito666> I might help you there, what is the problem?
<natefinch> perrito666: I have a status struct that has enums, because one of the formatters needs to perform some logic to determine what to output... and I didn't want to have a switch that compared raw strings to some magic values.
<natefinch> perrito666: http://reviews.vapour.ws/r/3445/#comment21101
<natefinch> perrito666: I've simplified the code somewhat since that, to make Origin and DataType into typed strings, so we can drop the marshal crap.  But eric thinks they should just be raw strings
<ericsnow> natefinch: I still think the formatter should be in charge of the logic stuff that outputters need
<perrito666> natefinch: I am going to side with eric in this one
<perrito666> status is already a nightmare to follow
<perrito666> if you add that generate magic, even though it is nice, it will be impossible
<natefinch> perrito666: no no... I've simplified it... let me push my changes
<ericsnow> natefinch: e.g. the formatter sets a "HasRevisionNumber" field to true
<natefinch> ericsnow: I guess if you then mark those as not getting serialized to yaml/json...  just seems like, if you do all that, then the outputter is just determining the column names and what columns to output.... I guess I just expected to separate the data and the view in different spots.  This seems like it's lumping all the views together in with the data.
<ericsnow> natefinch: my understanding is that the outputter is just in charge of spitting out the data it's been given (or a subset) in some particular output format
<ericsnow> natefinch: the issue here is the "subset" part
<ericsnow> natefinch: you're saying that the formatter should render all the info the outputter needs to do its job
<ericsnow> natefinch: I agree with that
<ericsnow> natefinch: I'm arguing for explicitly providing the info and you are arguing for piggybacking on some of the formatted data (which is reasonable)
<ericsnow> natefinch: this is all relative to what we are already doing: check for magic strings in certain fields in the outputter to make the decision
<ericsnow> natefinch: I agree that sort of implicit knowledge is problematic
<ericsnow> natefinch: so let's make it explicit with extra fields in the formatted data
<ericsnow> natefinch: if you are worried about JSON/YAML, then don't export the extra fields
<ericsnow> natefinch: if they aren't exported then the marshalers ignore them
<natefinch> ericsnow: right...
<natefinch> I still think it's merging the views in the wrong spot.... but it sounds like this is just the way they work, so I'll move those tabular columns into nonexported fields in the struct.
<ericsnow> natefinch: thanks!
<perrito666> the lxd provider works?
 * perrito666 does not feel like deploying a full openstack by hand
<natefinch> perrito666: yes it works
<natefinch> perrito666: it requires a few manual steps right now, which are detailed in the default environments.yaml output
<natefinch> perrito666: (a few one-time manual steps)
<perrito666> I guess it is behind a flag right?
<natefinch> perrito666: you need to compile with go 1.3 or higher
<perrito666> 1.5 should be enough
<natefinch> perrito666: indeed
<natefinch> perrito666: there's no feature flag
<perrito666> well, EOD
<axw> marcoceppi: are you awake still?
<axw> marcoceppi: you need to add features to your account... I think it's in the release notes, lemme check
<katco> axw: i think he figured it out
<axw> ah
<axw> thanks katco
<katco> axw: he did indeed have to add a feature to his account
<axw> yup, I also needed to keep reading :)
<katco> axw: lol!
<marcoceppi> axw: yeah, I stopped reading the instructions
<marcoceppi> axw: when I finished reading them, I found what I needed
<axw> marcoceppi: goodo :)  did it all work ok after that?
<marcoceppi> axw: so far so good. I was able to deploy to Standard_D12 instances which the old provider didn't allow
<axw> marcoceppi: cool
<marcoceppi> axw: I've got a problem
<marcoceppi> axw: juju expose with the new ARM doesnt' seem to work
<axw> marcoceppi: https://bugs.launchpad.net/juju-core/+bug/1527681
<mup> Bug #1527681: azure provider does not appear to be opening ports <juju-core:Fix Committed by axwalk> <https://launchpad.net/bugs/1527681>
<marcoceppi> well that puts a damper onthis demo
<axw> marcoceppi: can you build off master?
<marcoceppi> axw: it takes 20 mins to deploy and I have a demo in 30 :)
<marcoceppi> axw: I'm going to poke a few holes manually
<axw> marcoceppi: :/
<marcoceppi> if I can figure out how
<axw> marcoceppi: you'll need to create rules in the network security group
<marcoceppi> axw: found them, thanks
<marcoceppi> axw: I'll build from master next time
#juju-dev 2016-01-07
<marcoceppi> axw: demo went great, thanks!
<axw> marcoceppi: sweet :)
<axw> marcoceppi: did you show windows/centos workloads?
<marcoceppi> axw: no, just ubuntu stuff, this was mostly a juju benchmarking talk but they asked about it
<axw> ah ok
<marcoceppi> axw: I plan to have an ubuntu -> centos -> windows + benchmarking example for the follow up conversation
<axw> marcoceppi: awesome, I'll be keen to see that in action
<axw> wallyworld: could you please take a look over http://reviews.vapour.ws/r/3461/ some time today?
<wallyworld> sure
<natefinch> ericsnow: are you around?
<wallyworld> axw: lgtm, can we hold off landing until current CI run finishes on sm branch. i may be able to merge that to master first
<axw> wallyworld: sounds good
<axw> thanks
<anastasiamac> \o/
<mup> Bug #1531718 opened: syslog full of "rsyslogd-2089: netstream session ... will be closed due to error" <juju-core:New> <https://launchpad.net/bugs/1531718>
<mup> Bug #1531719 opened: Runaway memory allocation in jujud unit agent <juju-core:New> <https://launchpad.net/bugs/1531719>
<wallyworld> axw: may as well land that bootstrap branch. there's one remaining failure in sm branch, which is upgrading from 1.20
<axw> wallyworld: okey dokey
<wallyworld> axw: seen a409 before? http://pastebin.ubuntu.com/14427956/   get errors adding both centos7 and win2012 machines
<axw> wallyworld: urgh. no, take a look in the portal. navigate to the resource group, there will be a bar graph with some red - click on the red and it'll take you to the errors
<wallyworld> ok
<axw> wallyworld: what location are you using?
<wallyworld> West US it seems
<axw> wallyworld: ok, should be fine
<wallyworld> lol
<wallyworld> "Operation results in exceeding quota limits of Core. Maximum allowed: 4, Current in use: 1,
<wallyworld> sigh
<wallyworld> axw: i'm using a free trial, but 1 < 4 last time i looked
<wallyworld> oh wait, there's more message
<wallyworld> it says Additonal Requested 4
<wallyworld> but i'm only starting 3 machines
<wallyworld> bootstrap, plus one each of centos and win
<wallyworld> maybe it's cores, not machines
<axw> wallyworld: yeah those A3 machines are multi-core I think
<wallyworld> yeah had same thought
<axw> wallyworld: ask antonio to add you to the enterprise account
<wallyworld> will try smaller machines
<wallyworld> will have to
<axw> wallyworld: my trial account was just converted. I still have my own subscription, it's just not paid by me
<wallyworld> ah cool
<wallyworld> axw: so far so good with smaller machine, but fark it is soooooooooooooo slooooooooooow
<wallyworld> how do eople use it
<wallyworld> s/how/why
<axw> wallyworld: tis a bit
<wallyworld> s/a bit/a lot
<axw> wallyworld: I do like the interface and API though
<wallyworld> yeah, that is nice
<wallyworld> like everything else MS
<wallyworld> looks flashy, shit underneath
<axw> wallyworld: I've added a separate section to the release notes about the bootstrap changes
<wallyworld> awesome, ty. i was going to ask the same thing
<wallyworld> people have been wanting that for a while
<axw> it's kinda separate from strucutred metadata
<wallyworld> yep
<axw> wallyworld: you want me to do that cloud yaml bootstrap one next?
<axw> wallyworld: needs credentials branch first
<wallyworld> axw: yes please, i was hoping to get credentials unmarshalling done before hand, but i might run out of time
<wallyworld> credentials marshalling landing now
<axw> wallyworld: unmarshallign would be a prereq for bootstrap
<wallyworld> yeah
<wallyworld> i'll try and get to it tonight
<wallyworld> but if not i might have to leave it with you sorry
<axw> np
<wallyworld> i was optimistic about how much i'd be able to get done ahead of you
<wallyworld> axw: i'm aiming to have a bootstrap demo for end of next week
<wallyworld> i think that's doable
<dimitern> morning o/
<TheMue> morning dimitern
<dimitern> voidspace, dooferlad, frobware, please have a look at http://reviews.vapour.ws/r/3465/ when you have some time
<dimitern> TheMue, hey :)
<TheMue> dimitern: always with one eye in juju-land ;)
<dimitern> TheMue, cheers!
<TheMue> dimitern: that's so nice with open-source
<voidspace> dimitern: looking
<voidspace> dimitern: do you need to "continue" for *all* those possible errors?
<fwereade> late xmas present: http://thecodelesscode.com/case/218
<voidspace> dimitern: hmmm... it's only after you've successfully got the interface object and extracted a name that errors cause a skip
<voidspace> dimitern: so I guess it's alright
<voidspace> dimitern: LGTM
<anastasiamac> fwereade: tyvm :D cheers to u too \o/
<anastasiamac> fwereade: and right in time for some xmases (russian one is today..)
<fwereade> anastasiamac, ah yes, I forgot that -- happy xmas :)
<mattyw> does anyone know of a function that will take a unit string and return back the service name? I know it's trivial to do, but I wondered if it had been turned into a function
<dimitern> voidspace, tyvm
<wallyworld> Mmike: hey, did that mgopurge work?
<mattyw> ^^ found it https://godoc.org/github.com/juju/names#UnitService
<anastasiamac> tyvm
<Mmike> wallyworld: hola! I never received it, I figured menn0 would email it to me, but haven't received anything yet.
<wallyworld> Mmike: damn ok, i'll follow up
<wallyworld> Mmike: i was going to ask you to update that bug 1528261 with you results
<Mmike> wallyworld: np! :) thnx!
<mup> Bug #1528261: jujud won't start, log shows a lot of "cannot set addresses of machine XXX: cannot find transaction ObjectIdHex("xxxxxx"); <juju-core:Incomplete> <juju-core 1.20:Triaged> <https://launchpad.net/bugs/1528261>
<Mmike> wallyworld: I am re-doing what I did before holidays right now, I'll have an update within hour and a half, or os
<wallyworld> Mmike: great ok, if you could comment on the bug that would be great
<Mmike> yup, willdo
<dooferlad> dimitern, frobware: hangout time
<frobware> omw
<dimitern> omw
 * fwereade out for a bit
<mattyw> fwereade, are you around?
<jam> frobware: can you check something in your MAAS install?
<frobware> jam: sure
<jam> If you go to one of the VM instances in virt-manager, can you tell me what bus the disk is connected to?
<jam> Under Virtual Disk, I have an "Advanced Options" which lists "Disk bus:"
<frobware> jam: virtio
<jam> yeah, mine as well
<jam> can you run "sudo lsblk there" ?
<jam> sorry
<jam> "sudo lsblk" there
<frobware> jam: on the source path? (/var/lib/libvirt/images/maas19-node5.img)?
<jam> frobware: from inside the VM
<frobware> jam: presumably ... yep
<frobware> jam: http://pastebin.ubuntu.com/14429283/
<jam> frobware: do you know how you created that disk? I'm just noticing the size on mine is tiny, presumably because qcow can grow, and doesn't have a min size.
<jam> sorry, max size
<jam> either that or MAAS is just refusing to think about a storage that is only 200KB in size.
<frobware> jam: I created the disk through virt-manager's UI
<jam> ok
<jam> kmaas uses "qemu-img create -f qcow2 $PATH 32G"
<jam> which sounds like it can grow to 32GB
<jam> but 'lsblk' is reporting something much smaller
<jam> frobware: yeah, I think by default virt-manager checks the "preallocate" box
<frobware> jam: what does this report?  sudo virt-filesystems -l -a /var/lib/libvirt/images/maas19-node5.img
<frobware> jam: on the qemu-img create version?
<frobware> jam: I'm pretty sure they are preallocated as my /var looks like: /dev/mapper/crucial--vg-var--volume  493G  326G  142G  70% /var
<jam> frobware: virt-manager preallocates by default, qemu create does not
<frobware> jam: and I have 30 nodes(??)...
<frobware> jam: I tried `qemu-img create -f qcow2 -o preallocation=metadata vm-100-disk-3.qcow2  1G' which seems to generate 1G.
<jam> frobware: yeah, I tried that as well, but the disk thing seems to be sparse
<frobware> jam: without the preallocate I get 193K
<jam> frobware: because the "df -h" doesn't change
<jam> frobware: confirmed that this makes MAAS happy
<jam> I'm guessing they added code that cared about the size that lsblk returned
<jam> and without preallocation it gets a vblk that is is only 200K or so.
<jam> and then MAAS says "that's not useable"
<frobware> jam: I just created a node based on my previous qemu-imag ... preallocate ... which seems to commission OK && deploy.
<jam> frobware: right, I just deleted the disk image (without touching virt-manager) and reran "qemu-img create" with preallocation set
<jam> and without changing anything else, maas accepted and found the storage on the machine
<jam> so I'm guessing the 1.9rc4 used to notice the disk, but didn't care about the size
<jam> and 1.9.0 is saying "that disk is too small, ignore it"
<frobware> jam: actually.... my deploy failed. hmm.
<jam> frobware: bootstrap is making progress
<jam> not sure if everything will be happy, but it didn't stop at the same point
<jam> frobware: ugh. timeout (I think). "is started but not deployed"
<jam> but I see it downloading stuff in the console
<frobware> jam: that's where my deploy failed (cloud-init) looked like it couldn't create stuff on the disk.
<jam> frobware: it seems that cloud-init took 4m to download 40MB from the maas host
<jam> that seems like a performance problem
<jam> my load on this machine is super high
<jam> currently 15
<jam> (I also had the problem that I had left it running during commisioning, so I had to notice and ask it to reboot, so my timeouts are probably a bit off, but the perf issue is weird)
<jam> frobware: dimitern: So it looks like I can almost bootstrap now, but I'm running into the "no IP address for eth1" bug
<jam> It doesn't appear that dimitern's patch has made it into the maas-spaces feature branch has it?
<dimitern> jam, is that on maas-spaces?
<jam> dimitern: I'm getting the failure in maas-spaces
<jam> I tried "juju bootstrap --upload-tools" and the last line was: "failed to bootstrap environment: getting addresses: getting node interfaces: no ip_address for link #0 on interface #0 eth1
<dimitern> jam, yeah - so that's the same issue thedac was having, and my PR (currently being tested by the merge bot and should land very soon) fixes that
<jam> I'm also noticing that apt-http-proxy doesn't seem to affect the initial cloud-init update
<jam> at least, it doesn't seem to hit the proxy I just set up
<jam> yeah, I see it running now
<jam> I hope it lands first try
<jam> looks like it landed
<perrito666> voidspace: this was just taken a few blocks from here, are you by any chance in argentina? :p https://mmi488.whatsapp.net/d/VPt3jR5VJKCkgy9_4aHti1aOabI/ArCjonYv8CfDvL8KhO5kfWhBAof_QrxONa9F59MKBn2b.jpg
<TheMue> perrito666: *rofl*
<dimitern> frobware, here's the revert PR: http://reviews.vapour.ws/r/3472/
<dimitern> still testing on 1.9
<frobware> dimitern, OK. Was going through the process
<frobware> dimitern, trying to identify which merge commits we want to revert
<dimitern> and I'd appreciate if you also test it on 1.8 as my setup is funny with the IPv6 an all, so the test can't complete because of it
<frobware> dimitern, I think I only have 4 - does that sound about right?
<dimitern> frobware, yeah, perfect
<frobware> dimitern, OK will pull your branch in a bit.
<dimitern> frobware, ta!
<mup> Bug #1531878 opened: TestBootstrapImage fails on i386 and ppc64el <i386> <ppc64el> <test-failure> <unit-tests> <juju-core:Incomplete> <juju-core structured-metadata:Triaged> <https://launchpad.net/bugs/1531878>
<frobware> dimitern, I got two merge conflicts. Did you see more than that?
<dimitern> frobware, I can't remember the exact number
<frobware> dimitern, bridgescript.go and environs.go
<dimitern> frobware, the bridgescript one is due to adding head -n1 in one of the follow-up PRs
<dimitern> frobware, and the environs.go (dummy provider) is changed in at least 2 of the 3 PRs IIRC
<frobware> dimitern, I have a slightly different tree to you: https://github.com/frobware/juju/tree/1.25-redux
<dimitern> frobware, I'm comparing now mine and yours against upstream 1.25
<mup> Bug #1531886 opened: Status fails during upgrade <upgrade-juju> <juju-core:Incomplete> <juju-core structured-metadata:Triaged> <https://launchpad.net/bugs/1531886>
<frobware> dimitern, did you squash your commits for the revert? I was looking at `git log' in your tree and only see the 1 commit.
<frobware> dimitern, asking because if we were to revert the revert for 1.25.3 it would be nice to see them applied as individual merge commits again.
<dimitern> frobware, both branches mine and yours have the same diff
<frobware> dimitern, they're not identical
<dimitern> frobware, well, I didn't do it by reverting individual commits, but manually c/p chunks of the PRs
<frobware> dimitern, I think it is better to record them as individual reverts
<dimitern> frobware, oh, yeah - in 2 places they differ in environ.go (dns[i] -> dns[l] and a comment + in the tests a name)
<dimitern> but those are irrelevant
<frobware> dimitern, I find the log history makes it clear what happened. http://pastebin.ubuntu.com/14430590/
<dimitern> frobware, I have a habit of not trusting git to do the right thing.. so that's why I did it manually
<frobware> dimitern, I trust git more than Emacs. :)
<dimitern> frobware, hehe
<dimitern> frobware, well, I'm fine if you want to do it your way - esp. after I verified both diffs
<frobware> dimitern, yes - it's good that we have, more or less, the same result.
<frobware> dimitern, the 'more or less' is just cosmetic AFAICS
<frobware> dimitern, as a preferences I prefer the discrete steps.
<frobware> dimitern, I will test anyway. Trying on 1.8.2 right now.
<dimitern> frobware, sounds good, I finished my tests - all fine (well, apart from 2 issues)
<frobware> dimitern, new?
<dimitern> frobware, the issues are - IPv6 causing issues when used in 1.8-
<dimitern> frobware, and it doesn't seem to work well with vlans on top of a single nic
<frobware> dimitern, and both those issues went away with devices?
<dimitern> frobware, the IPv6 issue I fixed as part of these PRs
<dimitern> frobware, like skipping IPv6 subnets when discovering which one to use for address allocation
<dimitern> frobware, otherwise it barfs on the first one, which ultimately causes allocation to fail (with A-C on)
<dimitern> frobware, now retrying with 1 NIC only, no VLANs on 1.9
<frobware> cherylj, mgz: could we take mine or dimiter's change as a feature branch and test the backed out changes like we did for the DNS issue? I want to land/merge this with confidence that we don't endlessly break things.
<frobware> dimitern, ^^
<dimitern> frobware, the VLAN issues are weird - the bridge script can't seem to handle the "drop eth0.100 and replace it by juju-br0.100, which has vlan-raw-device eth0"
<frobware> dimitern, I think that's better on the maas-spaces branch
<dimitern> as it complains eth0.100 already exists
<dimitern> yeah it is indeed
<frobware> dimitern, cut+paste the /e/n/i and try running the bridge-script from the maas-spaces branch
<dimitern> frobware, good idea - will try on another node
<frobware> dimitern, shamefully I don't think the script reads from stdin.
 * frobware blushes.
<bogdanteleaga> anyone used the new arm provider?
<dimitern> bogdanteleaga, there's an arm provider now?
<dimitern> news to me
<bogdanteleaga> :)
<bogdanteleaga> azure arm provider
<bogdanteleaga> to be more exact
<katco> jcastro: so for bug 1495978
<mup> Bug #1495978: Juju does not deploy CentOS images in LXC <adoption> <centos> <containers> <juju> <lxc> <system> <juju-core:Triaged> <https://launchpad.net/bugs/1495978>
<katco> jcastro: for the local case, you can probably use the new lxd provider
<dimitern> ah :)
<katco> jcastro: it looks for a default image name, and that image could be a centos image
<frobware> bogdanteleaga, I'm curious as to why the "arm" part makes a difference. Azure as a provider I can understand but why does the architecture make a diff?
<dimitern> no, not yet
<jcastro> katco: oh if the new provider makes this bug go away then absolutely!
<dimitern> frobware, that's not an arch, it's the MS's acronym for the new az api
<bogdanteleaga> frobware: it's just a fancy name that probably everybody will misunderstand at first
<bogdanteleaga> it just comes from azure resource manager
<jcastro> katco: what's the tldr on when we can start playing with the lxd provider?
<katco> jcastro: well it's not a 1:1 match; what i'm describing would only work for the local case
<frobware> bogdanteleaga, ah - I see.
<katco> jcastro: i.e. matty's comment about lxc/kvm
<jcastro> mbruzek: ^^^
<frobware> dimitern, you should be able to run the script on your laptop - no need for a deployed node
<katco> jcastro: https://blueprints.launchpad.net/juju-core/+spec/charmer-experience-lxd-provider
<katco> jcastro: it was going to be in 1.26, it's currently landed in master, but will be released in 2.0 looks like
<jcastro> ok so should I tell eco to start banging on it for the alpha then?
<dimitern> frobware, yeah, but first need to get the /e/n/i
<dimitern> frobware, btw I added both of us to the revert card and linked your redux branch in there
<frobware> dimitern, ok - thought you were going to deploy a node from maas-spaces branch to run the script.
<katco> jcastro: sure; it's very stable. was slated to be released
<frobware> dimitern, ack
<katco> jcastro: just got pushed back because of our change in release schedule
<jcastro> katco: ok so if we were to do the charmer summit at end of this month with the LXD provider ... would you consider that a good decision?
<katco> jcastro: absolutely.
<cmars> jcastro, i've been using the lxd provider quite a bit, and find it to be very stable
<katco> jcastro: very confident in the stability
<frobware> dimitern, heh. when you click on the leankit link the card flashes at you...
<jcastro> wow that totally changes my life
<katco> jcastro: and it's soooo much better for local stuff
<jcastro> omg, I am doing this today then
<katco> jcastro: doooo it! lemme know if you have issues
<jcastro> what do I need? dev ppa?
<dimitern> frobware, yeah - some "improved UX" attempt I guess
<katco> jcastro: you need lxd installed and then >= 1.26-alpha2 of juju
<jcastro> katco: ok so we should be banging on this already, I'll get the word out
<jcastro> I had no idea it was this far along, awesome
<katco> jcastro: sorry, how could we have gotten the word out better?
<rick_h_> jcastro: yea, the alpha release ppa
<frobware> dimitern, do you have objections if we merge my branch? It's easier to look at the individual commits to see what was reverted.
<dimitern> frobware, not at all - go for it
<frobware> dimitern, I wanted to do some more testing first anyway. then hear from cherylj and mgz as to whether we want to test the change on a feature branch first.
<frobware> dimitern, it's great given the two approaches we're reverting the same thing. :-D
<cherylj> frobware: I'm on the cross team call, so that's why I'm "ignoring" you.  We're wrapping up :)
<dimitern> frobware, +1 :)
<dimitern> frobware, I figured out why the 1.25 bridge script was doing it wrong
<frobware> dimitern, and that's true on maas-spaces too?
<dimitern> frobware, http://paste.ubuntu.com/14430801/
<dimitern> frobware, the maas-spaces one does the right thing
<frobware> dimitern, just trying to see what's wrong...
<dimitern> frobware, well always using $PRIMARY_IFACE in the vlan-raw-device stanza is the cause for the errors I was seeing
<dimitern> rather than the correct vlan device, e.g. eth0.100
<frobware> dimitern, isn't that what maas-spaces is doing too?
<dimitern> frobware, check the paste - it has all details :)
<frobware> dimitern, If I grep for vlan-raw-device it is always eth0
<dimitern> frobware, what I'm seeing as a result of the maas-spaces script looks correct (although I haven't tried it live)
<frobware> dimitern, for 1.25 or maas-spaces
<frobware> dimitern, just to be clear though. In your PB I see that the vlan-raw-device is always eth0.
<jcastro> katco: I can't seem to find how to configure the lxd provider in the devel docs
<katco> jcastro: k sec
<dimitern> frobware, sorry - I meant bridge_ports most likely
<dimitern> (looking at maas-spaces bridgescript_test as well)
<katco> jcastro: https://docs.google.com/document/d/1ID-r22-UIjl00UY_URXQo_vJNdRPqmSNv7vP8HI_E5U/edit#heading=h.w4gb7d5kh9wd
<frobware> dimitern, in the 1.25 case we've omitted the iface for the underlying bridge port.
<katco> cherylj: any reason the upcoming release notes aren't checked into the devel section of the docs on jujucharms.com?
<katco> cherylj: not really sure how that whole process works
<dimitern> yeah
<frobware> dimitern, :(
<dimitern> frobware, sorry, I've just realized I need to go out and will miss the OS call
<dimitern> back later
<frobware> dimitern, k
<cherylj> katco: I'm not sure either.  sinzui? ^^
<jcastro> katco: it seems in the devel docs you'd just replace this page: https://jujucharms.com/docs/devel/config-LXC
<katco> jcastro: i'm not sure what the plans are for deprecating lxc. maybe for 2.0, but we wanted to give people time to migrate off their current workflows
<sinzui> katco: cherylj : there isn't a process to break features in release notes into spearate documetns. Some teams write add wip docs to juju docs, then write/paste the juju-doc into tje release notes
<katco> sinzui: well my questions is more specifically what is the process of getting the release notes into the docs. e.g.: https://jujucharms.com/docs/devel/reference-release-notes
<katco> sinzui: seems like the 2.0 release notes should be in devel
<sinzui> katco: we don't submit release-notes for alpha/betas.
<jcastro> hey so I thought documenting in the doc docs was the final sign off for a feature?
<cherylj> jcastro: it wouldn't prevent the feature from being merged and available in an alpha
<jcastro> cherylj: yeah I just thought there was a step in the process where it was like "check that the feature is documented"
<cherylj> jcastro: the goal is to have all of the documentation complete for all 2.0 features before the GA.  The 2.0 schedule has a documentation complete date of Mar-25
<jcastro> oh ok so it's not per-feature it's a milestone?
<cherylj> jcastro: well,  documentation is tracked per feature, but the date is the same for all features
<jcastro> ok, I'll leave some notes in this 2.0 release notes doc
<sinzui> katco: the 2.0 doc branch will be made from the master (devel) branch. The docs team would need to remove the alpha/beta information if we had added it to the the release-notes section because those version are not stable. We shouldn't add devel release-notes without the docs team agreeing that they know how to remove non-stable documentation.
<jcastro> katco: ok, good news and bad, it bootstrapped and I can go into the unit with lxc but `juju status` and `juju ssh` hang
<katco> jcastro: hm. those should work just fine
<katco> jcastro: there are instances running?
<jcastro> yes
<jcastro> the bootstrap instance is running
<jcastro> also I am on xenial, so I might be a little more bleeding than normal
<katco> jcastro: does the log provide any information when you run status?
<katco> jcastro: i don't know if anyone's tested it on xenial, but if it bootstraps it should work i think.
<jcastro> it's unclear to me where the logs are
<katco> jcastro: juju help debug-log
<jcastro> yeah that doesn't work, I suspect for the same reason ssh and status don't work
 * jcastro tries a rebootstrap
<katco> jcastro: did you freshly set up lxd or had that been installed/working already?
<jcastro> freshly installed, launching new images by hand seems to work fine
<jcastro> juju did fire up an instance and I can get to it with the lxc tools, seems like it doesn't know about it though
<katco> jcastro: what's the os for the image?
<jcastro> trusty
<katco> jcastro: ah. that's the problem.
<katco> jcastro: errr or i think it is? release notes seem to suggest that's ok
<katco> jcastro: can you try with wily?
<katco> jcastro: trusty doesn't have >= 1.3 so i don't think the tools would work
<jcastro> huh so weird, I couldn't destroy env because it was hanging
<jcastro> so I killed the instance by hand
<jcastro> and rebootstrapped
<jcastro> now it's working like a champ
<katco> jcastro: did you specify --upload-tools as well?
<jcastro> yes
<jcastro> holy shit, this is fast katco
<katco> jcastro: yeah it is
<katco> jcastro: glad it's working. wondering why it hung though.
<katco> jcastro: god knows --upload-tools is full of mystery and terror
<jcastro> I'll chalk it up to crazy guy with xenial for this one, but I'll see how the other guys get on
<katco> jcastro: could be something with thtat
<katco> jcastro: awesome ty for testing it out
<jcastro> mind if we just dump our notes in this gdoc?
<jcastro> I'd like to just tell the list to dump all their lxd provider notes in here, then we can flesh it out for the doc page?
<katco> jcastro: that's a question for cherylj since it's intended to be release notes
<jcastro> holy smokes, I know I said this before, but this is really fast katco
<katco> jcastro: notice machine 0 is just a container so you can do HA and do things like deploy openstack locally
<katco> jcastro: as well as theoretically use centos. i don't know if anyone's tried that yet. not sure if we have tools built for it
<jcastro> I'll ask bruzer to try it
 * perrito666 just deployed an openstack in lxd with lxd provider
<voidspace> frobware: dooferlad: http://reviews.vapour.ws/r/3473/
<frobware> voidspace, dooferlad, dimitern: http://reviews.vapour.ws/r/3474/ -- dimiter I reused your commit message.
<cherylj> frobware: mgz is going to test your changes to back out the devices patch
<jcastro> katco: what's the experience like if someone was all-trusty on say, their laptop?
<katco> jcastro: meaning trusty as the host?
<frobware> cherylj, mgz: magic. See the review request: http://reviews.vapour.ws/r/3474/
<jcastro> katco: yeah
<katco> jcastro: officially, that doesn't work right now because trusty only has go1.1 i think
<frobware> jcastro, I was curios about this too.
<mgz> frobware: that one, not 3472? :)
<katco> jcastro: unofficially, if you just use the ppa, it works great
<frobware> jcastro, katco: I was contemplating taking my laptop to wily but would prefer not to.
<cherylj> frobware, the DNS and default gateway bugs were caused by a commit ontop of the devices fix that didn't make it into master yet, right?
<katco> jcastro: wwitzel3 's lightning talk in seattle was to show openstack running on his trusty laptop
<cherylj> frobware: I think this one:  bug 1525280
<mup> Bug #1525280: 1.25.1 with maas 1.8: devices dns allocation uses non-unique hostname <maas-provider> <network> <regression> <juju-core:Triaged> <juju-core 1.25:Fix Committed by dimitern> <https://launchpad.net/bugs/1525280>
<katco> frobware: as a dev, you'd be fine. just use a version of juju you compile yourself or the ppa
<frobware> cherylj, correct. and I just discarded the DNS fix/review.
<katco> frobware: it's only main that wouldn't work
<cherylj> frobware: okay, when bug 1525280 does land in master, we'll need your DNS fix there as well
<jcastro> katco: ok so I tell people either wily or xenial as a minimum then?
<katco> frobware: it's because of the req. that what's in main be reproducible by building the source, and trusty only has go1.1. but since go is compiled, if you have the binary you're fine
<katco> jcastro: yes
<frobware> cherylj, yep and the one for the default gw. BTW: did you create a bug for that?
<cherylj> frobware: but I think we can get away without the default gateway fix because we're already telling people if you're using 2.0 and MAAS 1.8.3, you're gunna have a bad time.
<cherylj> frobware: unless you disagree?
<katco> jcastro: although there is movement in getting go1.5 in trusty, so that may happen at some point
<frobware> cherylj, and the bad time can include a note to set a default gw?
<katco> jcastro: at which point the next release into trusty would work
<cherylj> frobware: that was my thought
<cherylj> frobware: what do you think?
<frobware> cherylj, it's all text. :)
<cherylj> heh
<cherylj> frobware: I'll bring it up in the release call today
<jcastro> cherylj: mind if I get people dropping lxd notes in this doc or want me to split it off?
<cherylj> jcastro: if there are things that you find are preventing people from using lxd, then yes, add them do the release notes.
<frobware> mgz, 3472 vs 3474. Dimiter and I went about doing the backout in different ways and (thankfully) ended up with the same results. the git commit history in 3474 is a bit more discrete in what was done.
<cherylj> jcastro: if there are other issues that should just be included in the documentation, maybe they should be in a different doc.
<cherylj> katco: is there a separate doc you guys are working on for documenting lxd in jujucharms.com?
<katco> cherylj: no
<mgz> frobware: gotcha
<frobware> cherylj, inferring the default gw is a lower barrier to entry for the end user. I would want that.
<cherylj> frobware: if it's an easy code change, then yeah, I would agree
<cherylj> frobware: I can open the bug for you
<frobware> cherylj, please. thx.
<jcastro> cherylj: is there a reason we should keep the release notes gdoc not public?
<cherylj> jcastro: they are released publicly with the alpha / beta builds.
<frobware> cherylj, card for 1528217 forward port to master - https://canonical.leankit.com/Boards/View/101652562/119358672
<jcastro> ok so what if we want public feedback? Perhaps we should split this off?
<dooferlad> frobware, voidspace: http://reviews.vapour.ws/r/3475/
<dooferlad> will get to your reviews now.
<mup> Bug #1531932 opened: Unit agents unable to connect with MAAS 1.8 due to lack of default gateway in MAAS <juju-core:Triaged by frobware> <https://launchpad.net/bugs/1531932>
<frobware> cherylj, thanks for opening the bug
<cherylj> frobware: np :)
 * frobware EOD
<cherylj> mgz: were you able to get frobware's changes to revert 1483879 in a feature branch to test?
<mgz> cherylj: it's build-revision 3487
<mgz> various maas tests still pending
<cherylj> mgz: awesome, thanks!
<mup> Bug #1531954 opened: Trying to use Juju without a bootstrap node has a confusing error <juju-core:New> <https://launchpad.net/bugs/1531954>
<dooferlad> frobware: you have a +1 for http://reviews.vapour.ws/r/3474/
<dooferlad> frobware: I would say LGTM, but some of those reverts make me sad.
<wallyworld> katco: hey, did you want a chat?
<katco> wallyworld: hey, yep. 1:1?
<wallyworld> ok
<mup> Bug #1531995 opened: 2.0 upgrade-juju command should give exact command to run to upgrade a 1.x environment <juju-core:Triaged> <https://launchpad.net/bugs/1531995>
<natefinch> ericsnow: after the demo, we're going to have to devise a way to extract the set resource ops so we can do it in the same transaction as the rest of deploy.  But for now I'll just run setresource after deploy
<ericsnow> natefinch: we're going to do that stuff in a worker, if I remember right
<ericsnow> natefinch: so the transaction won't factor in
<natefinch> ericsnow: don't we have the resource metadata information (if not the bytes) available at deploy time? Might as well add it to the DB at that time
<ericsnow> natefinch: I'm pretty sure we don't have all the info we need, but I may be wrong
<cherylj> hey jog, I'm looking at the test run for the revert-maas-devices branch and I see that there's an OS deployer timeout
 * jog looks
<cherylj> jog: It looks like the services are still coming up.  Maybe things were slow?  I see nova-compute/0 is still installing things
<jog> cherylj, there are other lxc's still in pending, nova-compute/0's workload status is 'blocked' maybe waiting for relations to complete...
<jog> specifically LXCs on machines 5 and 6.
<jog> I'm looking to see if there is log information about what's going on there.
<cherylj> jog: If all the  clocks are synced, I see that we timed out here: 2016-01-07 21:19:34 ERROR Workloads not ready in maas-1_9-OS-deployer.
<cherylj> jog: and in the unit log for nova-compute/0:  2016-01-07 21:19:51 DEBUG juju.worker.uniter.context context.go:326 [WORKLOAD-STATUS] active: Unit is ready
<cherylj> jog: I can also look at the lxcs for machine 5 and 6
<cherylj> I see that they're "started"?
<jog> cherylj, sorry I think I was looking at the wrong one.
<cherylj> ah, phew
<jog> cherylj, the failure is due to the workload-status for all charms not setting down to 'active' or 'unknown'
<cherylj> jog: yeah, I was wondering that.  I think, however, that it might've been premature?
<jog> possibly... mgz recently added the workload verification code
<cherylj> I see that at least the nova-compute/0 unit was still chugging along after the test timed out.
<cherylj> jog: Can we queue that branch up for a retest, at least for the 1.9 os deployer test?
<jog> yeah and rabbitmq-server/0 is still running a hook
<jog> cherylj, a second run is going on now
<cherylj> jog: ah, perfect!
<jog> cherylj, hmm... we are only waiting 3 minutes
<cherylj> jog: wow, that seems short
<jog> that's after all unit agents become ready
<cherylj> oh
<cherylj> well, still seems a bit short given the size of the deployment
<jog> I'll hot fix in a longer timeout
<cherylj> thanks!
<jog> we'll need another run after this current test
<cherylj> if it fails again
<cherylj> (in the same way)
<jog> right
<jog> other Openstack tests have passed today, on master and structured-metadata
<cherylj> it really may be just a bit too short.  Like I said earlier, there were just a few seconds between the test timing out and that one unit being ready
<jog> I agree, other revisions also failed today with the same timeout waiting for workloads, then passed on a second attempt... either way is seems like a longer default timeout should be applied
<wallyworld> menn0: hey, you were going to email mario that mgopurge binary? i asked him last night and he hadn't received it yet
<menn0> I emailed it to him soon after we talked
<menn0> wallyworld:
<wallyworld> menn0: ok, i'll need to follow up with him and see why he didn't have it
<menn0> wallyworld: he sent me his email privately via IRC
<menn0> wallyworld: I just checked my sent mail and it's there
<wallyworld> menn0: ok, ty. no idea why he said he didn't get it
<menn0> wallyworld: virus scanner or spam filter?
<wallyworld> yeah, could be
<menn0> wallyworld: I'll sent it to you now as well
<wallyworld> ok, ty
<menn0> wallyworld: you should have the email now as well
<wallyworld> menn0: yep, just got it, tyvm
<menn0> cool
<mwhudson> Broadcast message from systemd-journald@DEVAC01 (Fri 2016-01-08 01:05:48 EET):
<mwhudson> [33800]: 2016-01-07 23:05:48 INFO juju.testing mgo.go:515 reset successfully reset admin password
<mwhudson> is there something i can do to shut this spam up?
<perrito666> mwhudson: soon, that is sadly something you cannot turn off in the current version of mongo, we aim to upgrade it soon
<mwhudson> ah ok
#juju-dev 2016-01-08
<axw> wallyworld: I'm ready whenever you are
<wallyworld> axw: ok, just finishing an email
<wallyworld> axw: in 1:1 now
<natefinch> ericsnow: it occurs to me that for local charms, all we have at deploy time is all we'll ever have for info on resources.
<cherylj> wallyworld: I should be able to do the aliases immediately after controller-rename lands.
<wallyworld> cherylj: :-) tyvm
<cherylj> I think there would be changes needed if I did it earlier.  I'd need to create aliases for different commands
<mup> Bug #1532063 opened: Unit seems unable to proceed after watcher died in the middle of hook execution <juju-core:New> <https://launchpad.net/bugs/1532063>
<wallyworld> cherylj: the controller rename is othogonal to the environment->model changes
<wallyworld> the controler rename is state server / jes stuff -> controller
<wallyworld> the other is environment -> model
<wallyworld> unless i'm missing something
<cherylj> yes, but the controller commands use "environment" in them
<wallyworld> ah ok
<wallyworld> axw: we can ignore bug 1531886 in getting sm to land. that issue comes from master and doesn't need fixing in the feature beanch
<mup> Bug #1531886: Status fails during upgrade <upgrade-juju> <juju-core:Triaged> <juju-core structured-metadata:In Progress by axwalk> <https://launchpad.net/bugs/1531886>
<mup> Bug #1532063 changed: Unit seems unable to proceed after watcher died in the middle of hook execution <juju-core:New> <https://launchpad.net/bugs/1532063>
<axw> wallyworld: ok
<wallyworld> it's the unit tests one that needs doing
<wallyworld> the tools not found
<axw> wallyworld: the BootstrapImage unit test fix is up for review
<wallyworld> cool, looking
<wallyworld> axw: i won't make team meeting, got things to finish, out of here soon
<axw> sure, safe travels
<axw> menn0 waigani_: I think it's just us here, I vote we cancel the team meeting
<menn0> axw: fine by me.
<waigani_> axw: okay
<axw> sold
<mup> Bug #1532063 opened: Unit seems unable to proceed after watcher died in the middle of hook execution <juju-core:New> <https://launchpad.net/bugs/1532063>
<natefinch> gah, sorry I missed the team meeting...
<natefinch> although sounds like me making it probably wouldn't have changed the outcome
<cherylj> axw: can I ask your opinion on some upgrade-juju behavior?
<axw> cherylj: certainly
<cherylj> axw: I'm trying to write the changes to allow a 2.0 client to upgrade a 1.25.2 env
<cherylj> and I'm trying to make things generic
<cherylj> so that if there comes a time for there to be a 3.0 we don't run into this again
<cherylj> we can just keep a map of major versions to the minimum version to upgrade from
<cherylj> and reference that when we do our upgrade checks
 * axw nods
<cherylj> but the problem I'm running into is we allow people using --upload-tools to put in whatever for the version
<cherylj> so we could have --upload-tools --version 4.4.4
<cherylj> and that would pretend that we're at 4.4.4
<cherylj> and we obviously don't have "4" in this map
<cherylj> so
<cherylj> either we don't check minimum levels for --upload-tools, or we don't allow those shenanigans
<cherylj> any thoughts?
<axw> cherylj: I think we should just reject the tools if they're beyond the latest major version
<cherylj> axw: yay, thanks!
<natefinch> cherylj: why do we care if people pretend they're on 4.4 if they use --upload-tools?
<cherylj> natefinch: I thought I explained it ok in the statements above.  Was it not clear?
<natefinch> cherylj: why would we stop anyone from upgrading a 1.25.2 environment?
<cherylj> natefinch: the idea was not to stop them, but to modify the 2.0 upgrade-juju command to actually do it.
<cherylj> but we need to enforce that people upgrading to 2.0 are coming from 1.25.2
<cherylj> or later
<natefinch> ahh, that's the trick
<natefinch> I would hope we could come up with a way for 2.0 that we'd never need such a restriction again
<cherylj> natefinch: I'm trying to make things generic, so we can just keep a map of major: minimum version to get there
<cherylj> like 2: 1.25.2
<cherylj> and update that when we hit 3, etc
<natefinch> I thought the reason you had to be on 1.25.2 to upgrade was because previous versions just wouldn't let you upgrade to a higher major version
<natefinch> if we just remove that code... then it should be fine
<natefinch> but I'm probably missing something :)
<cherylj> natefinch: no, we want people on 1.25.2 because we're removing upgrade steps to *get* there from 2.0
<cherylj> so 2.0 won't know what the upgrade steps are for 1.24.7 -> 1.25.2
<cherylj> I am going on the assumption that future major releases will operate the same way where we want a known place where users are coming from.
<cherylj> this "minimum upgrade version", like 1.25.2 is to 2.0
<natefinch> I guess I really hoped we could prevent people from having to upgrade twice
<cherylj> it's even more fun when you go past 2.0!
<cherylj> The decision was that you can't go from 1.25.2  -> 2.1
<cherylj> has to be 1.25.2 -> 2.0.* -> 2.1
<cherylj> 2.1 won't have *any* 1.x upgrade steps in it
<natefinch> here's my ideal scenario, though it's probably more complicated than it's worth:  you run upgrade with 2.3 on a 1.17 env, 2.3 only knows how to run an upgrade from 2.0, so it downloads 2.0, and tells it to upgrade first... 2.0 only knows how to upgrade from 1.25.2, so it downloads the 1.25.2 tools, and tells them to upgrade the environment, which works, and the rest plays out.
<cherylj> would be nice if we had the resources to put to that
<natefinch> yeah.... I guess we'll have to wait until enough people complain :)
<cherylj> no, no, that doesn't mean we need more people to do this work.  It's just means we're all shit at our jobs if we can't get it all done, right?
<cherylj> sorry, that was a bit too pessimistic :)
<natefinch> my team lost 25-33% of its developer resources and no one's changing our delivery date, which was already like a month earlier than our estimates think is likely.  So... yeah, I think you were right.
<natefinch> it's late, and dave cheney's not online to play grumpy dev, so someone's gotta pick up the slack
<cherylj> hahaha, nice!
<waigani_> heh
<natefinch> There's a station on google play music called "Code Your Face Off" which is pretty awesome, btw (and not just because of the name).  The description is "Work your magic while listening to these motivational electro-bangers -- and stop slacking off!"  https://play.google.com/music/r/m/Ltnhqsx7baik6dlsft6wdfchdly?t=Code_Your_Face_Off
<cherylj> that sounds painful
<natefinch> unrelated - anyone know if canonical pays for any of the travel to gophercon, or is it all out of pocket?  Convincing my wife to let me go away for ~4 days would be easier if it weren't also ~$1000 :)
<cherylj> natefinch: are you talking about for next year?
<cherylj> wait
<cherylj> this year
<cherylj> haha
<cherylj> in Denver
<natefinch> haha
<natefinch> yes
<cherylj> alexisb-afk: was going to work on getting sponsorship again
<natefinch> I'
<cherylj> so I'd let her know you're interested
<natefinch> will do
<natefinch> I'd love to go, last year was impossible because we already had a vacation booked that week.  Year before was impossible because we had a 2 week old.
<natefinch> (baby)
<cherylj> I figured :)
<natefinch> :)
<cherylj> aight, it's time for bed for me.  see you guys tomorrow
<natefinch> g'night
<mup> Bug #1532085 opened: Guarantee leadership revoked from departing unit before departed hooks run <juju-core:New> <https://launchpad.net/bugs/1532085>
<mup> Bug #1532085 changed: Guarantee leadership revoked from departing unit before departed hooks run <juju-core:New> <https://launchpad.net/bugs/1532085>
<mup> Bug #1532085 opened: Guarantee leadership revoked from departing unit before departed hooks run <juju-core:New> <https://launchpad.net/bugs/1532085>
<mup> Bug #1532130 opened: Config item 'version' vanishes with 1.26 <regression> <juju-core:New> <https://launchpad.net/bugs/1532130>
<mup> Bug #1532130 changed: Config item 'version' vanishes with 1.26 <regression> <juju-core:New> <https://launchpad.net/bugs/1532130>
<mup> Bug #1532130 opened: Config item 'version' vanishes with 1.26 <regression> <juju-core:New> <https://launchpad.net/bugs/1532130>
<jamespage> dimitern, hey - I'm having trouble with the MAAS spaces branch and reproducing thedac's deployments from yesterday
<jamespage> dimitern, I can't bootstrap an  environment - the machines deploy ok, but when juju configures all of the bridges post deployment, it completely hoses the network configuration on the server...
<jamespage> :(
<jamespage> dimitern, help!
<dooferlad> jamespage: we are in our standup right now. I am sure he will respond soon.
<jamespage> dooferlad, ack
<jamespage> dimitern, http://paste.ubuntu.com/14436922/
<dimitern> jamespage, uh oh .. that looks like the addresses are missing
<jamespage> dimitern, uh-oh was my reaction
<dimitern> jamespage, would you mind joining our standup to discuss this?
<jamespage> dimitern, sure
<dooferlad> https://plus.google.com/hangouts/_/canonical.com/sapphire
<dooferlad> jamespage: ^^
<frobware_> jamespage, can you paste the original /e/n/i -- juju creates a copy in /etc/network/
<jamespage> http://paste.ubuntu.com/14436933/
<jamespage> frobware, dimitern ^^
<jamespage> dooferlad, can you direct invite me from the hangout - still using iphone atm
<jamespage> http://paste.ubuntu.com/14436967/
<jamespage> dooferlad, sorry my iphone drops me from g+ now as well :(
<jamespage> thedac, when you awake ^^
<frobware> jamespage, will fix this in maas-spaces branch
<frobware> dimitern, can you raise the bug for the /e/n/i issue on 1.25 please
<frobware> dimitern, and if you have an /e/n/i that is bust can you attach it to the bug. thx.
<jamespage> frobware, ack - I think I can hack the add-bridges script for now to workaround
<jamespage> trying that now
<frobware> jamespage, http://pastebin.ubuntu.com/14437158/
<jamespage> frobware, I actuall specialized to dns-search
<frobware> jamespage, reading resolvconf and interfaces man pages indicates that dns- is not the beginning of any stanza, just options on the iface.
<jamespage> frobware, so infact those last two get appended to the last interface stanza right?
<frobware> jamespage, yep
<frobware> jamespage, otherwise you'll run into the same issue on dns-search - the script will think this a top-level stanza and break as it currently does.
<jamespage> frobware, hmm but dns-search is not part of the iface stanza's as generated right now
<jamespage> that might not matter if curtin also generates resolv.conf
<frobware> jamespage, the combination of ifup running resolvconf which plucks the dns-search from /e/n/i should end up in resolv.conf
<frobware> jamespage, I would still like to understand how/when/where the change to /e/n/i happened. Quite a few of the unit tests I have were cut+paste /e/n/i files from a deployed MAAS 1.9 node.
<jamespage> frobware, so would I - I also think the /e/n/i generated by maas/curtin is a little malformed right?
<dimitern> frobware, will do
<frobware> jamespage, the indentation and the separation of the dns-* entries confuse matters.
 * jamespage crosses fingers for network access
<jamespage> frobware, ok better - I'm only expecting a bridge for eth0 right now?
<frobware> jamespage, how many interfaces do you have? You should get a bridge per active NIC now.
<jamespage> frobware, eth0 is the only active physical nic - all others are vlans on eth1
<frobware> jamespage, can you paste your original /e/n/i
<dimitern> frobware, I've filed bug 1532167
<mup> Bug #1532167: maas bridge script handles VLAN NICs incorrectly <addressability> <maas-provider> <network> <juju-core:Triaged> <juju-core 2.0:Triaged> <https://launchpad.net/bugs/1532167>
<frobware> dimitern, thanks!
<frobware> dimitern, dooferlad, voidspace: should we propagate dns-* from the original iface to the new bridge stanza?
<dooferlad> frobware: yes
<dimitern> frobware, yeah - because the bridge is taking over the nic and all its options, just adding the bridge-utils ones to them
<frobware> dimitern, dooferlad: which is what I am/was doing, just confirming.
<dooferlad> frobware: great :-)
<frobware> dooferlad, dimitern: think this only holds true for bonds. e.g., http://pastebin.ubuntu.com/
<frobware> http://pastebin.ubuntu.com/14437347/
<dooferlad> frobware: that seems reasonable, but I don't know for sure. Since it is a rule that says "add this nameserver to resolv.conf when I am activated" as long as the rule will run under the same conditions, that is fine.
<frobware> dooferlad, otherwise we have to special case which options we carry forward. Which is not a big deal because we already do it for the bond anyway.
<frobware> dimitern, actually for the bond it doesn't take all its options. I deliberately omit anything starting with bond.
<frobware> jamespage, any joy?
<jamespage> frobware, yeah
<mup> Bug #1532167 opened: maas bridge script handles VLAN NICs incorrectly <addressability> <maas-provider> <network> <juju-core:Triaged> <juju-core 2.0:Triaged> <https://launchpad.net/bugs/1532167>
<dimitern> frobware, that's because we verified what needs to stay for the bond-bridge(-vlan) device to still work
<frobware> dimitern, I'll actively try leaving them in. I think the fewer special cases the better.
<jamespage> frobware, dimitern - ok I have a bound ceph deployment running!
<jamespage> w00t!
<frobware> dimitern, the new iface (i..e, the bridge) is not a bond so I didn't carry those options forward.
<frobware> jamespage, excellent!
<dimitern> frobware, ah, right
<frobware> dimitern, these are all just arbitrary rules. :-)
<mup> Bug #1532167 changed: maas bridge script handles VLAN NICs incorrectly <addressability> <maas-provider> <network> <juju-core:Triaged> <juju-core 2.0:Triaged> <https://launchpad.net/bugs/1532167>
<mup> Bug #1532167 opened: maas bridge script handles VLAN NICs incorrectly <addressability> <maas-provider> <network> <juju-core:Triaged> <juju-core 2.0:Triaged> <https://launchpad.net/bugs/1532167>
<dimitern> frobware, also filed bug 1532176
<mup> Bug #1532176: maas bridge script in maas-spaces feature branch handles dns-* options incorrectly <addressability> <maas-provider> <juju-core maas-spaces:Triaged> <https://launchpad.net/bugs/1532176>
<dimitern> frobware, added the expected /e/n/i in jamespage's case (AIUI)
<dimitern> jamespage, awesome!
<frobware> dimitern, do we want to raise bugs for feature branches?
<dimitern> frobware, we can - whether we should is another thing :)
<frobware> dimitern, seems overkill to me
<dimitern> frobware, but since I started composing it, esp. took care with the expected config, seemed good to have it filed
<dimitern> but I might've just as well added a card
<perrito666> dimitern: ping
<dimitern> perrito666, hey there
<perrito666> hey, ill privmsg you
<mup> Bug #1532186 opened: lxd container creation fails semi - regularly <local-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1532186>
<mup> Bug #1532186 changed: lxd container creation fails semi - regularly <local-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1532186>
<mup> Bug #1532186 opened: lxd container creation fails semi - regularly <local-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1532186>
<frobware> dimitern, dooferlad, voidspace: http://reviews.vapour.ws/r/3481/ - network/interfaces fix for demo
<dimitern> frobware, looking
<dimitern> frobware, LGTM with an extra test for the jamespage's case
<frobware> dimitern, done and thanks for the review
<dimitern> frobware, cheers
<dimitern> frobware, hmm wait a moment..
<frobware> dimitern, oh...
<dimitern> frobware, looking at the test now
<dimitern> frobware, why auto eth1.2667 and the others still remain?
<dimitern> frobware, also I can only see one bridge
<frobware> dimitern, I think we've danced this dance before.
<dimitern> frobware, either the expected is wrong or the script doesn't quite handle that as I'd have expected
<frobware> dimitern, the underlying device is not active.
<frobware> dimitern, I believe we previously said inactive devices don't get bridged.
<frobware> dimitern, however, what should the behaviour be...
<dimitern> frobware, but it's auto and it has auto children which are active
<frobware> dimitern, so the current test is... is the underlying vlan-raw-device eth1 active?
<dimitern> frobware, if this is the /e/n/i we'll use, how will multi-NIC containers work on that host node?
<frobware> dimitern, they obviously won't... :(
<dimitern> frobware, well, it looks active - "auto eth1", despite being "manual"
<dimitern> it will be up when running ip link show
<dimitern> otherwise the vlans won't work either
<frobware> dimitern, so I think we should nail down what we consider an active interface
<frobware> dimitern, btw, I don't believe this is a demo problem.
<dimitern> frobware, yeah, it's not but only because we're not using containers
<dimitern> frobware, and don't get me wrong - your fix is till valid
<frobware> dimitern, but let's fix containers whilst we're here.
<dimitern> frobware, but for the multi-NIC containers we'll need to fix that (for the os demo)
<frobware> dimitern, because some of this could go back into 1.25 too
<dimitern> frobware, I think "active" should mean either the NIC itself is up and has address (can send/receive packets) or any of it children are active
<frobware> dimitern, let's let the fix for the demo land and do another for multi-NICs
<dimitern> frobware, sgtm
<frobware> dimitern, so it's the "and has an address" - this is where we have stumbled before.
<frobware> dimitern, the eth1 in that test has no address.
<dimitern> frobware, yeah, but it has dependents which have
<frobware> dimitern, our parsing is too simplistic
<dimitern> frobware, eth1 just needs to be up
<frobware> dimitern, we don't really model it - we just do some pretty high-level text transformations.
<dimitern> frobware, yeah - one pass only
<frobware> dimitern, let me work on this now...
<dimitern> frobware, we need to get a simple model from the /e/n/i with a dict of NIC names to NIC info which includes an "active" flag and any children - similarly
<dimitern> well, it can also include all the options so you don't have to re-parse them when rendering
<frobware> dimitern, parsing is so cheap though.
<frobware> dimitern, open a file, read line by line.
<dimitern> frobware, just tried with the initial e/n/i, but changed eth1 to dhcp instead of manual and added "a" to each dns-nameservers options except the last one and this works better:http://paste.ubuntu.com/14438110/
<dimitern> frobware, what was the issue with just just always creating a bridge for each iface?
<dimitern> bonds I presume.. or boot slowdowns?
<frobware> dimitern, heh. so that's an alternative.
<frobware> dimitern, not sure there is one.
<frobware> dimitern, I was trying to cater for the case where it is unconfigured.
<frobware> dimitern, if it is unconfigured and we create a bridge, what happens for the VIP?
<dimitern> frobware, well, it needs to have "auto" at least perhaps?
<dimitern> frobware, I don't mind creating an extra bridge we'll not use, unless that causes problems
<dimitern> frobware, the VIP is not involved I think
<dimitern> frobware,do you mean the VIPs added by corosync ?
<dimitern> frobware, with or without a bridge that will still work - adding an extra IP to an interface that's not up - but it simply won't pass any traffic
<frobware> dimitern, it shouldn't matter. I was thinking of the case where we want to give back a device
<frobware> dimitern, the dellstack case --> http://pastebin.ubuntu.com/14438169/
<dimitern> frobware, in that case we'll see that e.g. eth3 on node1 is linked to a subnet but in "link_up" state only, and that's what we'll give neutron-gateway
<dimitern> (well, once network-get returns more than an address, but the full NIC info)
<frobware> dimitern, what did you think of ignoring is_active and just bridge the lot?!
<dimitern> frobware, I think we should try as the simplest thing we can do for multi-NIC containers
<dimitern> frobware, and seriously test it in an automated fashion with permutations of different NIC configs on different MAAS versions
<frobware> dimitern, my hack should you want this: http://pastebin.ubuntu.com/14438191/
<dimitern> frobware, cheers
<dimitern> frobware, I'll come up with something next week to automate setting up lxc-based maas of a given revision with a given set of networks
<frobware> dimitern, there's a certain amount of orthogonality that comes with that
<dimitern> frobware, yeah, but we need to at least make sure it works with bonds and vlans (and vlans onto a bond)
<frobware> dimitern, there are unit tests for that
<dimitern> the latter case is common in telcos
<dimitern> frobware, unit testing what we generate is a must, but live testing whether it will work (and didn't break across releases) is also needed when we can get it to work
<dimitern> (get the automated testing to work I mean)
<frobware> dimitern, sure. I tried the vlans onto a bond a little while ago, but not with all the world bridged.
<frobware> dimitern, but from a parsing and creating bridges POV I believe we have some of these permutations. what happens in the wild... stays in the wild. :-)
<dimitern> frobware, yeah, we'll just keep updating those tests as we discover issues
<frobware> dooferlad, I just dumped 1531932 your way. Happy to answer questions here, in the card or wherever works.
<dooferlad> frobware: ok
<dooferlad> I assume that is bug #1531932
<mup> Bug #1531932: Unit agents unable to connect with MAAS 1.8 due to lack of default gateway in MAAS <juju-core:Triaged by dooferlad> <https://launchpad.net/bugs/1531932>
<frobware> dooferlad, yep. Before weds next week we need to land that, #1525280. And port #1528217 to master.
<mup> Bug #1525280: 1.25.1 with maas 1.8: devices dns allocation uses non-unique hostname <maas-provider> <network> <regression> <juju-core:Triaged> <juju-core 1.25:Fix Committed by dimitern> <https://launchpad.net/bugs/1525280>
<mup> Bug #1528217: 1.25.2 doesn't set up DNS information with MAAS <blocker> <ci> <maas> <network> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1528217>
<frobware> dooferlad, to tickle the lack of gateway in 1.8 you just need to remove any entry you may have in MAAS already in the network (in my case maas-eth0).
<frobware> dimitern, if we don't have uniques hostnames on master I'm guessing we don't run into either lack of default gw and/or lack of DNS info - correct?  It looks like that patch switches the provisioner to use configType.static.
<frobware> dimitern, not sure if you saw that ^^ - formatting looks odd in my client
<dimitern> frobware, we don't have unique hostnames, but we still use static and I don't remember whether the dns and gateways were discovered when missing
<frobware> dimitern, just trying to see what's the easiest environment to initially fix this in. 1.25 is the best place today as we know it fails there.
<frobware> dooferlad, ^^
<dimitern> frobware, well, 1.25 can be fixed after the revert happens so I guess we should fix dns + gateway on master and also unique hostnames first
<frobware> dimitern, the reason why I'm asking is ... why doesn't it fail today in the CI tests.
<frobware> dimitern, if there's a default gw entry, then it will be fine. but it popped up in CI because there was no entry.
<dimitern> frobware, because there are hardly any serious tests that use containers and/or multiple envs bootstrapped concurrently
<frobware> dimitern, well 1.25.2 tripped over it as soon as we fixed dns
<frobware> dimitern, so there are some
<dimitern> frobware, and for the dns / GW - I don't believe any CI tests (except likely for dooferlad's) check containers networking accessibility etc.
<frobware> dimitern, anyway, I'm not suggesting we fix in 1.25.2 first, just that's a playground you can use today because it fails there atm
<dimitern> frobware, well, they started passing for the first time I guess :)
<dimitern> frobware, right, I see
<frobware> dimitern, was trying to lower dooferlad's entry point... :)
<mup> Bug #1528259 changed: undertakerSuite.TestAPICalls nil deref panic <ci> <i386> <panic> <regression> <test-failure> <juju-core:Fix Released by axwalk> <https://launchpad.net/bugs/1528259>
<mup> Bug #1532232 opened: maas 1.9 container networking broken <ci> <lxc> <maas-provider> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1532232>
<frobware> dimitern, ^^ LOL. sigh.
 * frobware takes up smoking...
<mup> Bug #1532232 changed: maas 1.9 container networking broken <ci> <lxc> <maas-provider> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1532232>
<mup> Bug #1528259 opened: undertakerSuite.TestAPICalls nil deref panic <ci> <i386> <panic> <regression> <test-failure> <juju-core:Fix Released by axwalk> <https://launchpad.net/bugs/1528259>
<dimitern> :)(
<frobware> sinzui, dimitern, jog: when I looked at John's email earlier today he indicated that it had worked and the failure may be intermittent. Has that changed?
<dimitern> frobware, I haven't looked into it since then - want to finish the last 2 cards around interface_set parsing
<sinzui> frobware: dimitern: I cannot say exactly what is wrong. I see that non of the containers called home, which looks like container networking.
<dimitern> sinzui, if we have the cloud-init-output.log of the containers it will tell us what went wrong
<frobware> sinzui, any of these nodes still up?
<sinzui> dimitern: It is all there http://reports.vapour.ws/releases/3487/job/maas-1_9-OS-deployer/attempt/130
<frobware> sinzui, as in can we login to the node and look at the container setup, et al?
<sinzui> frobware: no.
<dimitern> sinzui, well, notably cloud-init-output.log from the container's rootfs is not there
<dimitern> I might have reported a bug for that some time ago
<sinzui> dimitern: damn. Thzt is a bug in in CI. We need to change the rules. I think the globe is wrong
<dimitern> https://bugs.launchpad.net/juju-ci-tools/+bug/1424357
<mup> Bug #1424357: copy_local_logs should also get container logs <juju-ci-tools:Fix Released by sinzui> <https://launchpad.net/bugs/1424357>
<sinzui> frobware: structured-metadata is testing now. It is possible to re-run the failing job, or just deploy a smaller bundle that uses containers.
<dimitern> it's marked as fix released
<frobware> sinzui, can we do that next?
<mup> Bug #1528259 changed: undertakerSuite.TestAPICalls nil deref panic <ci> <i386> <panic> <regression> <test-failure> <juju-core:Fix Released by axwalk> <https://launchpad.net/bugs/1528259>
<mup> Bug #1532232 opened: maas 1.9 container networking broken <ci> <lxc> <maas-provider> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1532232>
<sinzui> dimitern: yes, but it is clear one log is missing
<dimitern> sinzui, ah, sorry - I've now actually read what I posted originally, and I don't mention to get /var/lib/lxc/juju-*/rootfs/var/log/cloud-init*.log
<dimitern> unlike with a broken kvm you can't get to, lxc rootfs is accessible from the host usually
<sinzui> dimitern: I am going to make a one line change to get the missing log, then run a simpler job to get more information
<dimitern> sinzui, thanks
<thedac> jamespage: dimitern: frobware: Odd I did not run into that problem. Do things look ok for the demo now?
<frobware> thedac, yes - I believe jamespage reported a working ceph deployment.
<thedac> ok, great.
<frobware> thedac, so it did trigger a genuine problem and we were curious as to how/why we haven't seen this to date.
<frobware> thedac, the fix has been merged into the maas-spaces branch
<thedac> Yeah, me too. I have done dozens of deplys and never saw that issue.
<thedac> s/deplys/deploys
<frobware> thedac, we were speculating that it may have been an external change in curtin/maas/trusty/...?
<thedac> hmm, well I am not going to touch those hosts in MAAS so as not to introduce new problems ;)
<sinzui> dimitern: the cloud-init-ouput.log is not in /var/lib/juju/containers. I need larger hack to get the log.
<dimitern> sinzui, yeah, it's in /var/lib/lxc/juju-*/rootfs/var/log/cloud-init-*.log
<mattyw> dimitern, ping?
<mattyw> ^^ also maybe sinzui ping?
<frobware> dooferlad, you about?
<dooferlad> frobware: yep
<frobware> dooferlad, can we do a quick HO...
<dooferlad> frobware: sure
<dimitern> frobware, dooferlad, voidspace, I'd appreciate a review on http://reviews.vapour.ws/r/3482/ when you can
<dimitern> mattyw, pong
<mattyw> dimitern, hey hey - as I understand it juju doesn't support bootstrapping on CentOs yet - is that right?
<dimitern> mattyw, I've never tried bootstrapping, but some time ago I fixed an issue with manual provisioning to a centos machine
<dimitern> mattyw, but looking at the cloudconfig code it should be possible - except if mongodb is borked there or something
<mattyw> dimitern, I'm trying to help a guy out on the list who is bootstrapping on centos, I'm sure I remember some discussion about it not working yet, but can't find any evidence yet
<mattyw> dimitern, there seem to be some selinux issues coming up
<cherylj> mattyw: see also bug 1474386
<mup> Bug #1474386: Problems bootstrapping the manual provider with CentOS <centos> <manual-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1474386>
<dimitern> mattyw, right well, sorry I can't help with that
<mattyw> dimitern, no problem - just wanted to see if you had any experience, as you were around
<cherylj> mattyw: I also recall a recent change to handle the selinux problem?  maybe?
<cherylj> sounds familiar anyway
<mattyw> cherylj, I'm not sure, I'm sure I remembered hearing something about it not being implemented ages ago, I don't think we have a ci test for it?
<mattyw> cherylj, if we think bootstrapping on centos should work I'll help the guy raise a bug for us to look at
<cherylj> mattyw: you can also attach more info that the one bug I linked to
<mattyw> cherylj, the log I've got is totally different, so I'm going to raise a new one
<cherylj> mattyw: k
<natefinch> katco, ericsnow: putting juju show-service-resources back and making juju show-charm-resources into juju resource reminds me.... I think this is exactly backwards.  I think people will use show-service-resources 1000x as often as show-charm-resources.  You'll be constantly checking your environments to make sure they have the correct resources deployed, but how often will you really want to look at what's in the charm store from the CLI?  Hardly ev
<natefinch> er?  Why not just use the nice website we've spent so much time on?
<ericsnow> natefinch: yep
<natefinch> katco, ericsnow: but hopefully thats something we can talk about during/after the demo
<natefinch> katco, ericsnow: just make john type show-service-resources like 20 times during the demo, and we'll get him on our side ;)
<mattyw> cherylj, https://bugs.launchpad.net/juju-core/+bug/1532266
<mup> Bug #1532266: bootstrapping on Centos fails with SELinux policy denies access <juju-core:New> <https://launchpad.net/bugs/1532266>
<mup> Bug #1532266 opened: bootstrapping on Centos fails with SELinux policy denies access <juju-core:Triaged> <https://launchpad.net/bugs/1532266>
<frobware> dooferlad, qemu-img create -f qcow2 -o preallocation=metadata vm-100-disk-3.qcow2  1G
<natefinch> anyone online familiar with the new charm upload/publish? I can't get my server to deploy the charm I uploaded
<natefinch> ericsnow or katco: trivial 2 line review of command rename?
<natefinch> http://reviews.vapour.ws/r/3483/diff/#
<ericsnow> natefinch: done
<natefinch> early afternoon is the best time to have a lot of stuff to land... no competition for the landing bot
<natefinch> I do wish our CI was a little more continuous than it actually is.... like telling me pre-emptively when my code won't compile after someone changes master out from under me.... ericsnow.
<natefinch> ;)
<ericsnow> natefinch: I thought I warned you :)
<natefinch> ericsnow: probably, I don't know... who can keep track?
<ericsnow> natefinch: if you have a sec, could you take a look at https://github.com/juju/charm/pull/188?
<ericsnow> natefinch: thanks!
<natefinch> ericsnow: welcome
<katco> ericsnow: just finishing up some admin things
<katco> ericsnow: and then we can pair agian
<ericsnow> katco: k
<katco> ericsnow: if you want that is
<ericsnow> katco: yep
<natefinch> ericsnow, katco: 40 lines of demo code to save resource metadata to state on deploy  (most of this will probably be reused after the demo): http://reviews.vapour.ws/r/3484/diff/#
<ericsnow> natefinch: sweet
<natefinch> wrong diff: http://reviews.vapour.ws/r/3484/diff/#
<natefinch> or not.. sorry, confusing myself between PR number and diff number
<natefinch> holy crap it works
<ericsnow> natefinch: :)
<natefinch> realized why the code wasn't working for saving resources at deploy time.... I was saving them using the serviceID for the doc, which includes the environment UUID, but we query them with the service name
<natefinch> easy fix
<katco> ericsnow: ok sorry i'm ready if you're available
<ericsnow> katco: sure
<natefinch> ericsnow, katco: very quick review of a bugfix in the output for show-service-resources? 100% relevant to the demo: http://reviews.vapour.ws/r/3485/diff/#
<natefinch> katco, ericsnow:  btw:
<natefinch> $ juju show-service-resources starsay
<natefinch> RESOURCE        ORIGIN REV USED COMMENT
<natefinch> store-resource  upload -   no   One line that is useful when operators need to push it.
<natefinch> upload-resource upload -   no   Who uses xml anymore?
<natefinch> real output stored in the database, derived from the metadata.yaml at deploy time
<natefinch> ...once my last two patches land :)
<natefinch> https://media.giphy.com/media/b3u8anVaWFQ9G/giphy.gif
<natefinch> there are a lot more hits for grep "vespene gas" in the juju codebase than I would have suspected
<natefinch> ericsnow, katco: gotta run for dinner, but I'll be back after for a few hours. Feel free to shipit and $$merge$$ my branches if you wish, otherwise I will when I get back.  I started looking at eric's patches up for review, but they're big and will take a bit.  I'll do that if there's nothing else pressing when I get back.
<katco> natefinch-afk: sorry for radio silence; ericsnow is a demanding pairing partner (sob)
<ericsnow> katco: lol
<katco> i feel so defeated
<ericsnow> katco: whatever!
<katco> natefinch-afk: thx for patches, glad to see it's working :)
<ericsnow> katco: I thought we did well :)
<katco> ericsnow: yeah jk that was awesome
<katco> ericsnow-afk: you even used emacs!
<katco> ericsnow-afk: i'm going to call you an emacs user now
<lazyPower> oh god, they're multiplying
<katco> lazyPower: you don't know the power of the emacs
<lazyPower> you're right
<katco> lazyPower: give in to your fear and hatred. let it make you stronger
<lazyPower> katco: i'll think about out... but my 8ball says the outlook is not good
<lazyPower> s/out/it/
<lazyPower> i mean, why shop at walmart (emacs) when you've got amazon (vim) :)
<perrito666> Ericsnow if you are still around pls take a look at the gce patch for storage uuid on description
<perrito666> Eow bye all
<mup> Bug #1532354 opened: Juju 1.26-Alpha4 reports incorrect public-address with MaaS 1.9 <juju-core:New> <https://launchpad.net/bugs/1532354>
#juju-dev 2016-01-09
<natefinch> katco: how goes?
#juju-dev 2016-01-10
<mup> Bug #1532629 opened: Juju 2.0 should use a different default JUJU_HOME <juju-core:Triaged by wallyworld> <https://launchpad.net/bugs/1532629>
<mup> Bug #1532629 changed: Juju 2.0 should use a different default JUJU_HOME <juju-core:Triaged by wallyworld> <https://launchpad.net/bugs/1532629>
<mup> Bug #1532629 opened: Juju 2.0 should use a different default JUJU_HOME <juju-core:Triaged by wallyworld> <https://launchpad.net/bugs/1532629>
#juju-dev 2017-01-02
<perrito666> good morning all
<rick_h> morning perrito666
<perrito666> rick_h: hey, I thought I was the only one here
<rick_h> perrito666: I'm not in officially, just saying hi :P
<perrito666> lol
<perrito666> k eod, ill be back for standup
<perrito666> cheers
#juju-dev 2017-01-03
<frobware> jam: you around today?
 * frobware tries to find a 2FA device to check mail/cal
<rogpeppe> jam: ping?
<frobware> rogpeppe: curious as to why we are merging into juju:master for zaputil
<rogpeppe> frobware: it's not juju:master - it's a new project
<rogpeppe> frobware: ... unless i've borked things up horribly :)
<rogpeppe> frobware: ah, it's juju:master because github only prints the username not the project in that place
<frobware> rogpeppe: well, possibly not. the GH page says juju:master, but I notice the popup says juju/zaputil:master... horrible UI.
<frobware> rogpeppe: apologies for the noise.
 * frobware sticks to the truth (aka the CLI)
<rogpeppe> frobware: np
<frobware> access to reviews.vapour.ws is broken. anybody around to kick it back into life?
<rick_h> frobware: wfm ?
<frobware> rick_h: hmm. I always get a 500
<rick_h> frobware: oh what page? I could login at least
<rick_h> frobware: and click on a release test result summary
<frobware> rick_h: http://reviews.vapour.ws/
<frobware> rick_h: ah.....
<rick_h> frobware: oh, that's the old reviewboard reviews site. Is it even up any more?
<rick_h> frobware: I think they took it down, or maybe they left it up for old data but no one's noticed it went boom
<frobware> rick_h: hehe. too much (little) tab completion.
<frobware> rick_h: nevermind...
 * frobware blushes
<frobware> rick_h: indeed, http://reports.vapour.ws/ wfm too. :)
<rick_h> frobware: :)
<frobware> rick_h: what doesn't work atm is dynamic bridges. :(
<frobware> rick_h: just digging through some of the changes jam made since I left
<rick_h> frobware: k
<rick_h> frobware: we talked a bit last week so I know about some of it, but not sure where it's at today
<frobware> rick_h: would be great to catch up; also trying to figure out what needs to also be in develop vis-a-vis 2.1-dynamic-bridges
<frobware> rick_h: this looks like a long list - http://reports.vapour.ws/releases/4680
<rick_h> frobware: I'm going to guess if it's a long list that there's some dirty substrate stuff with folks not around to help keep an eye on things
<rick_h> frobware: but not 100% sure
<frobware> rick_h: I'm pretty sure https://bugs.launchpad.net/juju/+bug/1652161 will break the functional-container-networking tests
<mup> Bug #1652161: juju-2.1-beta3 cannot add LXD container after host machine has rebooted <lxd> <network> <juju:New> <https://launchpad.net/bugs/1652161>
<frobware> rick_h: with or without dynamic bridging
<rick_h> frobware: k, will have to look at that. So we're not detecting an already configured lxd and trying to reconfigure it?
<perrito666> wow the channel is back alive
<rick_h> holiday is over, back to work
<voidspace> rick_h: yep :-/
<voidspace> :-)
<voidspace> rick_h: ping
<voidspace> rick_h: I'm working on bug #1631254 - lxd containers do not autostart
<mup> Bug #1631254: [2.0rc3] lxd containers do not autostart <rteam> <juju:In Progress by mfoord> <https://launchpad.net/bugs/1631254>
<voidspace> rick_h: which is marked as critical now
<voidspace> rick_h: the *specific bug*, as described, has been fixed for us by lxd
<voidspace> rick_h: https://github.com/lxc/lxd/issues/2469
<voidspace> rick_h: there is still a bug in juju that needs fixing, and I have a fix - just need tests
<voidspace> rick_h: however I can't manually test becasuse I can't repro (which is what I've spent this morning trying to do) because LXD have fixed the issue for us...
<voidspace> rick_h: I can manually verify that my fix "does the right thing" with regard to the generated config however
<voidspace> rick_h: but the bug can be downgraded from critical if it's holding anything up
<rick_h> voidspace: so can we not reproduce by setting the config value to auto start, manually shut down the lxd machine, then reboot and it does not come up before vs should with the config update?
<rick_h> voidspace: looking at the lxd issue, that was just that "if the container is running, restart it"
<rick_h> voidspace: so setting the config that it should autostart, manually turning it off, and rebooting the machine should still demonstrate the change you're making?
<voidspace> rick_h: nope, it now restarts fine even if you hard shutdown the host
<voidspace> rick_h: without my fix
<voidspace> rick_h: wait
<voidspace> rick_h: manually shutdown the lxd before hard reset
<rick_h> voidspace: I mean lxc stop the container first
<voidspace> rick_h: and with the fix we would expect an auto-restart and without we wouldn't
<rick_h> voidspace: then restart the host
<voidspace> rick_h: yep, good point
<voidspace> rick_h: thanks
<rick_h> voidspace: all good
<voidspace> I have my develop env bootstrapped, so I can try that *now*
<voidspace> rick_h: appreciated
<voidspace> rick_h: yep, on develop stopping then restarting the host does not restart the lxd container
<voidspace> rick_h: I actually wonder if my fix *does* change that....
<voidspace> I'll find out...
<rick_h> voidspace: so my understanding is that your branch fixes the configuration in lxd that controls that so I cross my fingers
<voidspace> rick_h: yep
 * voidspace lunch break
<voidspace> no school today, so no problem with second standup, beyond my normal philosophical objections to having two standups a day... :-)
<frankban> wallyworld: hey
<wallyworld> hi there
<frankban> wallyworld: I was going to ask a question but, as it happens, I just found the answer. sorry, happy new year :-)
<wallyworld> no worries. and same to you
<natefinch> sinzui: my latest merge build failed with /usr/lib/go-1.6/pkg/tool/linux_amd64/link: running gcc failed: fork/exec /usr/bin/gcc: cannot allocate memory
<natefinch> http://juju-ci.vapour.ws:8080/job/github-merge-juju/9935/console
<natefinch> abentley: ^
<perrito666> gcc?
<natefinch> hmm yeah weird
<sinzui> natefinch: perrito666 : I don;'t know why gcc is being used, but I believe it has something to do with a linker that will not actually have work to do.
<sinzui> natefinch: We can retry the merge, maybe we need more memory for the test machine
<natefinch> yeah, building Juju takes a ton of memory... I'm sure gcc is just something used in the background... we can retry, but if there's nothing in general wrong with the machine (errant processes running etc) then definitely more memory will be in order
<perrito666> sinzui: seems to be using gc-go?
<sinzui> perrito666: /usr/lib/go-1.6/pkg is no the location of gcc-go.
<natefinch> sinzui: multiple merge jobs have failed with OOM... if there's nothing actually wrong with the machine, then we need a bigger machine
<perrito666> natefinch: something might have leaked
<sinzui> perrito666: natefinch .yep. I just deleted 10 containers to reclaim memory
<perrito666> natefinch: sounds like something fixeable with a restart
<frobware> macgreagoir, voidspace: PTAL @ https://github.com/juju/juju/pull/6758 - explains some of why dynamic bridging isn't currently working.
<balloons> sinzui, natefinch, yea that job is still running on the old slave. It still does the actual merging
<natefinch> balloons: how much RAM does it have, out of curiosity?  obviously 10 errant containers are going to skew things pretty badly
<balloons> natefinch, 14 gig.
<natefinch> balloons: so is the merge job the only thing we run there? Because 14 gig should definitely be enough (again, w/o containers)
<balloons> natefinch, it is 14 gig. But I lied, I was thinking the merges where done by the old slave. But they aren't. All PR jenkins jobs are running on it
<natefinch> balloons: but serially, right?
<balloons> not entirely. The merges are, but the pre-commit jobs are open season
<balloons> natefinch, however your job was the only thing running on the box at the time. Looking back over the last month there's hardly ever more than 1 job running, so
<natefinch> so probably just the leaking containers.  maybe we need a job that runs occasionally when the machine is idle and cleans them up?
<natefinch> (obviously the best answer is not to leak containers, but that can be hard)
<alexisb> happy 2017 all!
<perrito666> alexisb: <grumpycat>NO</grumpycat?
<perrito666> :p
<redir> Good time-zone appropriate period, and happy new year juju-dev
<redir> frobware: yt?
<frobware> redir: yep
<frobware> redir: qemu and spaces?
<frobware> redir: HNY :
<frobware> :)
<redir> HNY!
<redir> curious frobware if you thought this https://github.com/juju/juju/pull/6758
<frobware> redir: that's broken for sure. Was just adding tests for that case as we speak
<redir> might be related to https://github.com/juju/juju/pull/6748#issuecomment-268935865 frobware
<frobware> redir: I confess to not trying this with KVM. I ran out of time towards the end of the year.
<redir> when that change lands I'll cherry pick and cross my fingers it puts the rabbit back in the hat
<frobware> redir: want to cherry-pick with no-commit to try out now?
<frobware> redir: not sure I'll end up with my tests completed before my EOD (soon-ish)
<redir> I s'pose that can't hurt
<frobware> redir: given this is so broken I would actually land the change as-is, with the promise to add unit tests.
<frobware> redir: this would also give the 2.1-dynamic-bridges branch a chance to get a CI run overnight.
<voidspace> alexisb: happy new year
<alexisb> same to you voidspace!  hope you had a great holiday
<voidspace> alexisb: yeah, really nice - lovely combination of busy-ness with people and relaxation with family
<voidspace> alexisb: I return to work with a renewed and invigorated hatred for my job
<voidspace> oops, didn't mean to say that
<voidspace> ;-)
<voidspace> alexisb: I hope you had a good break too, and managed to get away from work
<alexisb> voidspace, I did, disconnected for 2+ week - it was awesome
<natefinch> rick_h: you around?
<rick_h> natefinch: otp what's up?
<natefinch> rick_h: nothing super important, just wanted to talk about that deployer - to - manual email
<rick_h> natefinch: k, meet you in core
<TheMue> Happy New Year, my Juju fellows
<natefinch> TheMue: happy new year!
<TheMue> natefinch: started to work yesterday? I did, first a bit tired from vacation, but today better. got a new colleague in my team and teaching him go. :)
<natefinch> Ahh cool, must be fun teaching someone Go.
<TheMue> natefinch: yes, and he has a good feeling for it. so now we are 3 in my project.
<natefinch> TheMue: That's great
<TheMue> natefinch: absolutely, I'm happy I'm able to place Go in anew project.
<natefinch> sinzui, balloons, abentley: is the vsphere listed in cloud-city's clouds.yaml running?  Is there anything other than being on VPN that I need to do to access it?  It's listed at 10.245.0.134
 * redir lunches
<babbageclunk> Happy new year everyone!
<redir> HNY babbageclunk
<babbageclunk> perrito666: I added Cloud.Name in https://github.com/juju/juju/pull/6735 - you were right, it looks better.
<perrito666> I saw your change pass by but was in EOY mode :p
<babbageclunk> perrito666: yeah, I figured! :)
<babbageclunk> perrito666: I'll get jam to take another look today.
<perrito666> babbageclunk: looks much better indeed
<abentley> natefinch: It is running.  You probably need to be on a specific machine to access it.  I am looking that up.
<perrito666> natefinch: you need to ask IS to grant you permission to see it on the vpn
<perrito666> our vpn is quite fine grained
<perrito666> natefinch: actually someone from oil needs to open a rt for is to grant you permissions <-- abentley sinzui  babbageclunk
<perrito666> sorry that was for balloons
<perrito666> meh, I cant bootstrap with lxd because I die waiting for address (using develop) is this a problem for someone else still?
<sinzui> perrito666: I set natefinch the .ssh/config to get to the host we test with.
<perrito666> nice "hacky vpn/vpn" :p
<natefinch> sinzui: thanks
<perrito666> ok EOD until standup, see you all later
<abentley> sinzui: ping for standup
<redir> ping alexisb
<alexisb> heya redir omw
<redir> k
<alexisb> need about 5 more minutes
<natefinch> so, I apt installed golang-1.6 and .... I still have no "go" command?  WTF?
<redir> alexisb: np
<babbageclunk> wallyworld: reviewed https://github.com/juju/juju/pull/6757
<wallyworld> babbageclunk: great ty, will look
<babbageclunk> wallyworld: around for a hangout?
<wallyworld> sure
<redir> wallyworld: you're back?
<wallyworld> supposedly
<wallyworld> since yesterday
<redir> HNY wallyworld :)
<babbageclunk> wallyworld: https://hangouts.google.com/hangouts/_/canonical.com/babbageclunk
<wallyworld> same to you
<perrito666> Natefinch golang-go is the package iirc
<perrito666> meh my dns are really acting weird today
<perrito666> at least I believe its the dns
<redir> maybe it's just actin like dns
<perrito666> has anyone experienced juju develop not being able to bootstrap lxd because it nevers gets an address?
<perrito666> I ERROR failed to bootstrap model: waited for 20m0s without getting any addresses
<blahdeblah> Anyone know when we will get 1.25.9 in the stable PPA for xenial?
#juju-dev 2017-01-04
<redir> perrito666: I think a little while back I had to kill and recreate my lxd bridge. I don't recall exactly why, but not being able to deploy to lxd seems familiar.
<perrito666> redir: remember how to do that?
<redir> perrito666: well that was last year
<redir> I think something like `sudo ip link set lxdbr0 down && sudo ip link delete lxdbr0 type bridge` followed by `sudo lxd init`
<redir> but IIRC I prolly also had to remove all the lxd containers and images I had to init again
<redir> I don't recall if there's an init only bridge command
<redir> perrito666: ^
<redir> perrito666: looks like there's a non-interactive option now -- `lxd init --auto`
<redir> oh, and you prolly have to kick lxd with sysctl too
<redir> erm, systemctl
<redir> babbageclunk: were you still reviewing my pr from last year?
<babbageclunk> redir: oh sorry - I haven't picked it back up - have you solved the timing problem you were talking about in the meeting?
<redir> yep, not really timing, just order of apt installs
<redir> and updated the QA steps with a note about fixing a kvm breaking network change
<redir> for QA at least
<redir> but I need 2 +1s since is it over 500 lines
<babbageclunk> redir: ok, taking another look
<redir> babbageclunk: much  obliged
 * redir goes afk for a bit. I'll check back later
 * redir is unclear about the failures on http://juju-ci.vapour.ws/job/github-check-merge-juju/485/
<redir> if they look familiar to anyone let me know
<babbageclunk> jam: could you take another look at https://github.com/juju/juju/pull/6735?
<babbageclunk> jam: Also, happy new year!
<babbageclunk> Hmm, I'm confused - why is my irc client completing jam when he's not here?
<wallyworld> babbageclunk: ah, i just sent an email, thought you were EOD. when you start tomorrow, there's another PR to look at, sorry
<babbageclunk> wallyworld: ok, will take a look soon - still going through redir's one at the moment!
<jam> babbageclunk: I'm afk, but I have an IRC forwarder.
<jam> babbageclunk: so technically I'm always here. :)
<wallyworld> babbageclunk: ty. once we get this landed and in the hands of the GUI guys, we may see some nice progress for next week hopefully
 * wallyworld needs to do a coffee run, state of emergency here, bbiab
<babbageclunk> jam: Ah - I didn't see you in the list because you're an op!
<jam> I am OP :)
<jam> so technically I'm off this week, I'm only around because I'm sitting here doing my personal budgeting. but I can try and give it a look sometime
<jam> babbageclunk: so what is DefaultCloudName() vs Cloud.Name? It isn't quite clear from the diff why you need both.
<jam> is the latter the variable where you store the result of the former?
<babbageclunk> jam: yes
<jam> babbageclunk: I will fully admit to not doing as full of a review as last time, but LGTM
<babbageclunk> jam: good enough for me! ;)
<babbageclunk> jam: thanks - sorry to interrupt your holiday!
<mup> Bug #1653888 opened: juju-db service using 30GB+ memory <juju-core:New> <https://launchpad.net/bugs/1653888>
<voidspace> frobware: macgreagoir: if you have a chance https://github.com/juju/juju/pull/6761
<macgreagoir> voidspace: Testing it now.
<voidspace> macgreagoir: thanks
<macgreagoir> voidspace: LGTM, fwiw
<voidspace> macgreagoir: thanks
<voidspace> macgreagoir: I'm going to merge then
<macgreagoir> voidspace: ianagr :-) but OK by me.
<voidspace> macgreagoir: gah, you should be by now
<rick_h> macgreagoir: not a 'gr'?
<rick_h> oh, reviewer
<rick_h> heh
<macgreagoir> :-)
<perrito666> does anyone know how to nuke the lxd bridge and re-create it?
<macgreagoir> perrito666: Does `lxd init ...` give you what you need?
<perrito666> macgreagoir: meh, it says I have containers, which I dont
<perrito666> macgreagoir: but most likely its what I need
<macgreagoir> `brctl delbr` et cetera
<voidspace> rick_h: desparate for coffee - will be 2 mins late!
<perrito666> macgreagoir: oh I get it, It can not just run init on networking, that is kind of sad :(
<perrito666> man, apt-get purge has lost some of its touch
<natefinch> perrito666: I have a vsphere question for you
<perrito666> natefinch: oh, for me? you shouldn't have bothered
<natefinch> perrito666: well, just trying to start off the new year on the right foot ;)
<natefinch> perrito666: I am testing my provider Ping method, which is really just calling govmomi.NewClient with the given URL... but I am getting back 400 bad request for some reason
<perrito666> natefinch: well shoot, in a moment ill need to go fetch my wife's bd cake
<natefinch> perrito666: I'll try to be quick
<perrito666> you might be doing a bad request??
<natefinch> perrito666: also, happy birthday to your wife :)
<natefinch> perrito666: is there a specific path I need to give the NewClient function other than https://<someip> ?  I'm running against the vsphere that QA uses, in oil
 * perrito666 checks code
<perrito666> natefinch: first of all, remember you need to close that because they never time out
<natefinch> perrito666: yeah, doing that
<natefinch> perrito666: oh, hmm, looks like it might be /sdk
<perrito666> look for providers/vsphere/client.go line 79
<perrito666> yep
<perrito666> I wonder if that is going to answer to you without credentials
<perrito666> natefinch: on second thought, 40X is a good indicator of ping
<perrito666> it means something is up there and can tell you "no"
<natefinch> perrito666: well, almost anything will respond with 400 bad request, though
<perrito666> true, I am not sure how accurate you need your ping to be
<natefinch> perrito666: as accurate as possible.... with /sdk I get ServerFaultCode: Cannot complete login due to an incorrect user name or password.   which is probably good enough
<perrito666> k bbl
<frobware> macgreagoir, voidspace: updated https://github.com/juju/juju/pull/6758 with some unit tests. Would appreciate a look so that I can address any issues and try to get a CI run before tonight.
<frobware> rick_h: ^^
 * frobware bbiab
<mup> Bug #1651260 changed: landscape bundle error when deployed via gui (KeyError in config changed hook in haproxy charm) <matrix> <juju:New> <https://launchpad.net/bugs/1651260>
<mup> Bug #1651260 opened: landscape bundle error when deployed via gui (KeyError in config changed hook in haproxy charm) <matrix> <juju:New> <https://launchpad.net/bugs/1651260>
<mup> Bug #1651260 changed: landscape bundle error when deployed via gui (KeyError in config changed hook in haproxy charm) <matrix> <juju:New> <https://launchpad.net/bugs/1651260>
<perrito666> back
<macgreagoir> voidspace: If you have a wee sec, please: https://github.com/juju/juju/pull/6762/
<voidspace> macgreagoir: looks straightforward to me
<macgreagoir> 'Tis :-)
<voidspace> macgreagoir: LGTM
<macgreagoir> Cheers
 * voidspace lunches
<natefinch> rick_h: lol, so, I was looking at my old change, and I think I know why it's causing a failure now.... because I stopped throwing away the error that we get from lxd init
<rick_h> natefinch: heh, one step forward...
<natefinch> rick_h: so I think we were always hitting this error, we just were logging it and ignoring it before.  The easy fix is to go back to ignoring it.  The harder fix is to try to only run lxd init if it's needed (or handle the "you already ran this, dummy!" error)
<natefinch> probably #3 is easiest - still always run lxd init, but check for the "you already ran it" error and ignore just that one.  That way if it fails the first time, we actually fail early, rather than try to continue on even though there's no lxd running
<frobware> natefinch: Logging an error causes too many false positives, IMO. I've seen bug reports / mail / IRC saying, "lxd is broken, it reports an error, see this error in the logs..."
<natefinch> frobware: right, my point is, let's not log the error in this known case, since it's basically like the error when you try to create a file that already exists.
<natefinch> frobware: but for other unknown errors, let's actually return them, not just log them and effectively ignore them
<frobware> macgreagoir: ping
<macgreagoir> frobware: pong
<frobware> macgreagoir: do you have some time to look through https://github.com/juju/juju/pull/6758
<macgreagoir> frobware: I have it loaded, just kicking a couple of other things first, sorry.
<frobware> macgreagoir: ty
<redir> happy wednesday juju-dev
<frobware> redir: only 2 to go. :)
<macgreagoir> G'day redir
<macgreagoir> frobware: Review and comment. No show-stopper.
<frobware> macgreagoir: wall clock in tests...
<macgreagoir> :-)
<macgreagoir> I have one like that and I'd like to see you solve it :-D
<frobware> macgreagoir: ah, you found a bug.
<frobware> macgreagoir: all the tests pass 0 for a timeout, which means indefinite, and therefore no timer/clock is used.
<frobware> macgreagoir: there is one test that checks for timeout and passes 1s.
<frobware> macgreagoir: because of this I chose not to use the testing clock.
<macgreagoir> frobware: Maybe worth a comment so we don't scoop it up next time there's a testing sprint?
<frobware> macgreagoir: done
<frobware> voidspace: you about?
<voidspace> frobware: yep
<macgreagoir> frobware: acked in PR
<frobware> voidspace: could you also take a look over https://github.com/juju/juju/pull/6758
<voidspace> frobware: yep
<voidspace> coffee first though
<frobware> macgreagoir: you've gone all recursive on me.
<redir> sinzui: yt?
<sinzui> redir: yes
<redir> happy new year:)
<rick_h> natefinch: is it not something that we can check if lxd is configured and fail fast that way? I guess what's fail here? I mean it it's been run you're good and if it's not been run you need to run it.
<redir> sinzui: you wrote me a while back about a power8 host. Can I use that for live testing kvm? Are the details in the usual place?
<natefinch> rick_h: maybe there's a lxd command we can run that'll tell us if it is initialized?  I definitely don't want to write any heuristics outside of what lxd itself tells us though
<sinzui> redir: I don't think I can help. QA has ppc64el guests. I am not sure we can run kvm on them.
<redir> sinzui: OK. Good to know, I won't spend time trying
<sinzui> redir: well maybe. this can be done because I know a bit about borbein and its vms
 * redir backs up
<natefinch> tych0: is there an lxd command that we can run that'll tell us if we need to run lxd init?
<rick_h> natefinch: I guess the error you're getting *is* telling you that in some sense
<sinzui> redir: I can get you on to a Juju QA host and on to a charm testing host. brobein has a special kernal to support libvirt. Juju wont just work because the ubuntu kernel needs to be special
<natefinch> rick_h: yeah, just wondering if there's a better way.  Ideally something that doesn't require string matching on the lxd output for a specific error message
<sinzui> redir: So borbein is your best chance to verify that juju can drive kvm, but surely you need a cloud that is ppc64el with the right kernel to test that juju can deploy kvm containers to an instance
 * sinzui writes email with instructions.
<redir> sinzui: yeah that sounds right -- that I need both. I just had a note to ask when it came timet to live test on a different arch. I'll just work with the arm and ppc folks where they can test it.
<redir> sinzui: the code wants same host/arch to work
<sinzui> redir: I understand. QA has been asking for proper ppc64el resources for more than a year
<sinzui> redir: arm is a different, but also painful story
<redir> sinzui: tyvm
<sinzui> redir: We have a powerful arm64 host, but have never gotten libvirt working to build a vmaas. I really want to do this, but qemu/libvirt don't like the arch
<frobware> The GIL is gone: https://opensource.googleblog.com/2017/01/grumpy-go-running-python.html
<frobware> And C extensions too. But...
<perrito666> frobware: lol, yeah, I think without C extensions its a bit useless though
<perrito666> isnt ssl in python a C extension?
<frobware> perrito666: it cuts out a LOT OF STUFF
<perrito666> frobware: I presume its a "we just dont want to re-write all this code base in go"
<frobware> perrito666: I think it just shows that if you are running your own stuff on your own infra you can do whatever makes sense _for_ _you_
<perrito666> yep
<natefinch> ahh, it's a transpiler... interesting
<frobware> voidspace: any feedback. I need to EOD soon and want to merge before we scream "RC1 - ship it!"
<voidspace> frobware: looks good to me I think
<frobware> voidspace: when we rewrite the script in Go (^^) I think a lot of the testing friction will begin to disappear w.r.t. the bridge.py.
<voidspace> I like the scriptrunner
<voidspace> frobware: ok
<voidspace> frobware: I struggle to believe that Go is *really* easier to test ;-)
<voidspace> frobware: but possibly
<voidspace> the impedance mismatch goes
<frobware> voidspace: I think it's still the fact the script "does it all". Some of the invocation is in python, wrapped by Bash, invoked from Go.
<voidspace> yep
<frobware> voidspace: I would like to not expose dry-run too. :/
<voidspace> frobware: yeah, if possible
<frobware> voidspace:  that is purely an implementation detail.
<voidspace> frobware: anyway, LGTM
<frobware> voidspace: I'm happy to follow-up, but I think it's important to know get a CI test run on this _feature_ branch.
<frobware> s/know/now
<voidspace> frobware: yep, cool
<voidspace> frobware: not sure if I'll get mine in today
<voidspace> struggling
<frobware> voidspace: want a review or something else?
<voidspace> frobware: no, still trying to find the bug
<voidspace> frobware: or at least find whereabouts user config are passed into lxd
<voidspace> frobware: just getting back to it really, I'll give it a couple of hours or so before EOD
<voidspace> be nice to find itt
<frobware> SetConfig()?
<voidspace> frobware: not sure
<voidspace> I don't think so - or at least I'd like to see all the layers
<perrito666> we should get  a free day every time we change something in the output of status
 * perrito666 considers hacking his  desktop to support a 3rd monitor
<natefinch> does it not?  My laptop supports integrated screen + 2 external
<perrito666> natefinch: the furniture I meant :p
<perrito666> the computer card supports like 8 screens
<natefinch> perrito666: lol,
<natefinch> rick_h: should this fix go to develop?
<natefinch> rick_h: for the LXD init thing?
<rick_h> natefinch: into the 2.1-dynamic-bridges branch and the 2.1 and develop
<natefinch> okie
<rick_h> natefinch: actually just the 2.1-dynamic-bridges and develop
<rick_h> the 2.1 will get merged in
<natefinch> https://github.com/juju/juju/pull/6764 and https://github.com/juju/juju/pull/6765
<alexisb> babbageclunk, perrito666 is one of you around
<perrito666> alexisb: I am
<babbageclunk> alexisb: yup
<babbageclunk> alexisb: perrito666 is.
<babbageclunk> ;)
<alexisb> babbageclunk, rick_h has a request
<rick_h> babbageclunk: can you please look gat ^
<rick_h> look at that is
<rick_h> babbageclunk: and help natefinch get his PR landed
<perrito666> babbageclunk: I can do it so your day start is not so rough
<babbageclunk> rick_h: yup yup
<babbageclunk> perrito666: ok, thanks
<rick_h> natefinch: can you put a bit more background in that PR description please?
<perrito666> natefinch: that is 6764 and 6765 right?
<rick_h> perrito666: correcty
<perrito666> k
<natefinch> QA steps and description added
<rick_h> ty natefinch
<perrito666> natefinch: interesting "sudo dpkg-reconfigure -p medium lxd" <-- that error by lxd is also a lie
<perrito666> natefinch: ship them
<natefinch> merging
<rick_h> natefinch: ty, balloons ^ once that hits we can start a full test run of the feature branch please
<perrito666> since we are in it, could anyone review this? https://github.com/juju/juju/pull/6763
<natefinch> perrito666: looking
<natefinch> perrito666: why isn't this using model status
<natefinch> perrito666: surely controller upgrading should cause the model to become busy until it's finished
<perrito666> natefinch: you would think
<perrito666> but no, status actually answers even while controller is upgrading
<perrito666> I believe this patch will make us discover a whole new world of bugs where juju answers its upgrading when its not
<natefinch> right but, what I mean is, the model-status is going to still say "available" when it's really not
<natefinch> i.e. you can't juju deploy something while the controller is upgrading
<natefinch> (presumably)
<perrito666> natefinch: interesting, the rules gouvering statuses are a bit heavier than this
<natefinch> perrito666: I'm not sure that that means
<perrito666> natefinch: there is a logic behind when each status is set
<perrito666> and we dont have a status that correctly reflects "upgrading" afaik nor think to add it, also adding a status might not be backwards nice
<natefinch> perrito666: model-status is the correct place to put this.  We only recently created it, so there's no backward compatibility problem
<natefinch> perrito666: that may be why the powers that be were thinking of just adding a boolean rather than using the status we already have in place.
<natefinch> rick_h: ^
 * rick_h reads back
<perrito666> natefinch: nope, there was not much thought behind it
<natefinch> rick_h: basically... we added a new boolean to Model status, but probably should just reuse the relatively new model-status field
<rick_h> natefinch: perrito666 +1 to using the field. We don't want to make decisions based on checknig several different fields
<natefinch> re: https://github.com/juju/juju/pull/6763
<rick_h> natefinch: perrito666 adding a new "is-controller-doing-X" isn't ideal
<perrito666> mm, I think I agree with both of you
<perrito666> rick_h: what do you reckon is the status we should set? I presume I should be adding an "upgrading" message too
<alexisb> does model-status only show up in yaml?
<rick_h> natefinch: do you have a link to your PR from the sprint?
<natefinch> rick_h: I can get it
<natefinch> rick_h: https://github.com/juju/juju/pull/6661
<perrito666> natefinch: that seems to also work on json right?
<natefinch> alexisb: it's shown for juju models and juju status --format=yaml/json
<rick_h> perrito666: ^ shows how the model status was updated to be able to be used generically and might help point to where we're thinking.
<rick_h> perrito666: right, so the idea is that during an actual upgrade we'd update that as "upgrading" and so it could be "migrating" or "upgrading" or ...
<natefinch> rick_h, perrito666: gotta run for dinner.... but I recommend Status: status.Busy and StatusInfo: "controller is upgrading"
<perrito666> sounds like a fair start
<rick_h> ty natefinch
<perrito666> rick_h: ok there is some room for thought there but I believe that the rule should be something like if status != error && upgrading return  Status: status.Busy and StatusInfo: "controller is upgrading"
<rick_h> perrito666: k
<perrito666> I would not Set status for upgrading because we risk having other statuses shadowing that in the future
<perrito666> sorry for rubber ducking you btw
<rick_h> hmmm, yea thinking
<rick_h> so...you're marking it busy though right?
<perrito666> yes, but I am sort of hijacking that model.Status call and returning a status that is not the one set because the fact that an upgrade is in progress superseeds whatever model think its doing, makes sense?
<perrito666> rick_h: ^
<rick_h> perrito666: well, not really. I mean can the model be migrating and upgrading?
<rick_h> perrito666: I'm concerned that going that route leaves us open to a non-finite state machine of what is going on
<perrito666> should not
<perrito666> I mean upgrade a non concurrent task, nothing else should be happening with upgrade
<rick_h> perrito666: so I'd rather that it was actually set and that other code could rely on that information. e.g. a requested migration call should check that it's available before migrating and it then sets migrating?
<rick_h> perrito666: and the same is true of upgrading, a requested upgrade would fail if not available and if it starts it updates status and is tracked with "since..." and such
<rick_h> perrito666: like any other long running activity like that?
<perrito666> rick_h: if you request anything that makes a change on juju while upgrading juju will say no
<perrito666> not very politely
<rick_h> perrito666: right, but by setting a firm status we're allow juju to get more polite?
<perrito666> rick_h: we are, in this case we should set this status for all models before ugprading
<rick_h> perrito666: and without tracking the actual change by setting status we don't have real visiblity as to when it started/etc like we do the migration? I guess it feels like we're doing this one different than migration/etc without something I see as really different
<perrito666> then set.... mm I am not sure what to set after finishing
<rick_h> perrito666: but you can upgrade a model at a time
<rick_h> perrito666: only some models may be upgrading
<rick_h> perrito666: available?
<rick_h> since nothing else can take that while you own it
<rick_h> perrito666: I understand where you're coming from, I'm asking for a bigger promise that Juju works
<perrito666> rick_h: I am thinking in the not so long term, I mean if the model was in some other status and I mark upgrading I would be loosing that status right?
<perrito666> also, upgrade cannot happen for a model only, can it?
<perrito666> In a regular juju I mean
<perrito666> in a regular juju install upgrading upgrades the controller binary and the mongodb for all models in that controller
<rick_h> perrito666: yes, because the controller is upgraded on it's own, then the model is migrated at the same version it was, and then can be upgraded I thought
<rick_h> perrito666: but the whole rule of "the model can't be on a version > than the controller" and migrations and all this comes to play
<rick_h> nope
<voidspace> rick_h: I'm going to have to rethink my approach to bug #1631254
<mup> Bug #1631254: [2.0rc3] lxd containers do not autostart <rteam> <juju:In Progress by mfoord> <https://launchpad.net/bugs/1631254>
<voidspace> rick_h: however, given that the way to reproduce it is now to *manually* stop the container
<voidspace> rick_h: as the "hard shutdown" failure mode has been fixed in lxd
<voidspace> rick_h: I think it can be demoted from critical and need not hold up 2.1.0
<voidspace> rick_h: ok, for me to change it from critical to high?
<voidspace> rick_h: (basically we need to keep the user namespace and special case the ones that don't need it I think - which is the opposite way to how I was doing it)
<voidspace> rick_h: I haven't changed it from critical to high myself yet, and I'm now EOD (11pm) - g'night
<mup> Bug #1492237 opened: juju state server mongod uses too much disk space <canonical-bootstack> <mongodb> <oil-2.0> <uosci> <juju:Triaged> <juju-core:New> <https://launchpad.net/bugs/1492237>
<mup> Bug #1634390 opened: jujud services not starting after reboot when /var is on separate partition  <uosci> <juju:Triaged by rharding> <juju-core:New> <https://launchpad.net/bugs/1634390>
#juju-dev 2017-01-05
<alexisb> https://goo.gl/photos/h73SeP3REHgVsyyu8
<alexisb> ^^^ wallyworld, babbageclunk, redir
<wallyworld> alexisb: did you save me a piece?
<wallyworld> bring to capetown :-)
<redir> that looks awesome.
<redir> I used to go to the beach with someone that made mac-n-cheese pancakes every year.
<redir> sounds funny tastes awesome
<wallyworld> babbageclunk: the other tweak with add relation on the server side is to account for that consume could have been run first. we do check that the remote application does not exist when we go to save it, but we will need to tweak things to be a bit smarter. check early for existence, plus in the case of offers, account for that the offer may have been updated with new endpoints so a new save is indeed required
<wallyworld> that last bit with offers can be left as a todo
<babbageclunk> wallyworld: ok, i'll do that too
<wallyworld> babbageclunk: thanks, it needs a little bit of tweaking to ensure it all still makes sense once consume is added
<alexisb> veebers, heya
<alexisb> working on sending your mail
<alexisb> sorry I didnt get to it earlier
<veebers> alexisb: hey :-) Awesome thanks, no worries
<redir> what's the story with the build tests? Is windows failing a known issue?
<redir> babbageclunk: done
<babbageclunk> redir: ok looking
<redir> babbageclunk: you know the build tests story?
<babbageclunk> redir: are they broken?
<redir> it never seems to pass windows
<redir> but looks like a ci script exception
<redir> time to make dinner, bbiab to check if I can merge but EoD otherwise.
<babbageclunk> I've approved - want me to merge it?
<babbageclunk> redir ^
<babbageclunk> redir: I figured you probably would.
<redir> yay
<redir> I was going to squash it
<redir> but no big deal
<redir> me wonders if it will merge
<redir> meh it'll fail
<redir> I'll fix and merge after dinner
<babbageclunk> redir: oops sorry - should have realised you'd want to squash.
<wallyworld> babbageclunk: if you could squeeze a review for this i'd be very grateful as i can then tell the gui guys they are good to go with icons https://github.com/juju/juju/pull/6766
<babbageclunk> wallyworld: sure - looking now
<wallyworld> awesome ty
<wallyworld> babbageclunk: i realised i made a mistake in removing the "serveIcon" bool from apiserver/charms.go so I've just added it back
<wallyworld> that boolis needed for when there's no icon and we want to foerce a default
<mup> Bug #1654136 opened: Removing relationship didn't remove subordinate unit <canonical-bootstack> <juju-core:New> <https://launchpad.net/bugs/1654136>
<babbageclunk> wallyworld: lgtm
<babbageclunk> wallyworld: (sorry, computer crashed0
<babbageclunk> )
<wallyworld> babbageclunk: farking awesome, tyvm
<redir> babbageclunk: must have been tired of looking at that other PR:|
<babbageclunk> redir: :)
<jamespage> voidspace, hey - trying to remember whether network space binding of relations at add-relation time is possible or is that a deploy time binding only?
<voidspace> frobware: ^^^
<voidspace> jamespage: at the moment deploy time only I believe
<voidspace> jamespage: dynamic binding on the roadmap "imminently"
<jamespage> voidspace, ta
<voidspace> jamespage: frobware and I are in an interview right now, so will be an hour or so before we can properly discuss!
<rick_h> jamespage: deploy time only atm, we hope to add that juju bind command down the road to allow late time binding/rebinding
<perrito666> everybody, annyone has dealt with upgrades here?
<natefinch> perrito666: somewhat... what part of upgrades?
<perrito666> natefinch: I am iterating on the same issue as yesteday
<perrito666> and trying to figure out how the "upgrade by model" works but in state/upgrade.go the doc seems to state that upgrades are control level things only
<jamespage> any reason why we don't have published juju tools for yakkety/1.25.9 ?
<jamespage> rick_h, ^^ ?
<rick_h> jamespage: balloons was just saying we should have them? check the other #juju channel for conversation
<natefinch> perrito666: not sure how it works in a multi-model world, sorry
<jamespage> rick_h, ack well I think we should have them - we had .8
<rick_h> jamespage: k, will bring up with the release folks and see what's up.
<jamespage> rick_h, ta muchly
<mup> Bug #1646777 opened: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Confirmed for 1chb1n> <juju-core:Triaged by nskaggs> <https://launchpad.net/bugs/1646777>
<rick_h> jamespage: looks like the missing agents was an oopsie and the folks are on it
<jamespage> rick_h, ta
<redir> thanks for access sinzui
<redir> natefinch: you've cross compiled juju before?
<perrito666> hey people, I am not feeling very well ill be a bit unresponsive for a couple of hours
<redir> feel better perrito666
<perrito666> is that an order? :p
<perrito666> tx rick_h
<perrito666> I meant redir
<redir> who knows about the worker/provisioner container code?
<redir> natefinch  looks like a good candidate:)
<natefinch> redir: yes I've cross compiled juju
<redir> can I pick your brain for a few minutes natefinch
<natefinch> redir: sure
<redir> OK let me finish making coffee and I'll meet you in a HO
<redir> say 10 minutes?
<redir> natefinch: ^
<natefinch> redir: cool
<redir> natefinch: https://hangouts.google.com/hangouts/_/canonical.com/nate-reed
<redir> thanks natefinch
<alexisb> redir, I am doing a mini celebration for you :)
<redir>  alexisb tyvm
<redir> natefinch: you still around?
<natefinch> redir: yep
<redir> where does go build github.com/juju/juju leave the built stuff?
<natefinch> if you build a command, it leaves the command in the same place, if you build a package, it gets discarded
<natefinch> sorry.... same directory
<redir> so I want to build in appropriate directories for the binaries
<redir> I guess
<redir> or just go install and collect them from GOPATH/bin
<natefinch> go install is your friend :)
<redir> what package installs go?
<redir> golang-go?
<natefinch> yes, not sure what version that is
<natefinch> hopefully 1.6
<redir> yes
<redir> natefinch: real easy https://github.com/juju/juju/pull/6772
<redir> or babbageclunk or whomever
<natefinch> redir: ship it
<redir> tx
<babbageclunk> redir: I was too slow again!
<babbageclunk> huh - if you disable os-refresh-update it'll be faster most of the time but sometimes it'll just go into a loop trying to find things that have gone away. :(
<babbageclunk> wallyworld: ping?
<wallyworld> babbageclunk: hey
<babbageclunk> wallyworld: I'm trying out consume, and it all seems to work, except that wordpress doesn't actually work - I get a 502 bad gateway from nginx.
<babbageclunk> wallyworld: Does that sound like something obvious that I've missed?
<wallyworld> babbageclunk: this is after add relation is done?
<wallyworld> did you expose wordpress?
<babbageclunk> wallyworld: yeah, both of those
<wallyworld> hmmm. not sure off hand. that error normally means something didn;t connect. sadly it's juju debug-log time
<wallyworld> if you push your work, i can try it out as well
<babbageclunk> wallyworld: ok, thanks - just wondering whether there was something that you'd hit before.
<wallyworld> babbageclunk: i have - and the cause some "something" wrong with stuff getting wired up
<babbageclunk> wallyworld: hmm, there's no way to remove a remote application at the moment?
<wallyworld> no dont't hink so :-(
<wallyworld> on the todo list. i might add that today
<wallyworld> to debug, you can always deploy another app with a different name
<wallyworld> eg mysql2
<wallyworld> or bounce the jujud
<wallyworld> that will cause everyhthing to get wired up again in the worker
<babbageclunk> wallyworld: ah, ok - will do that instead of destroying the model next time!
<wallyworld> babbageclunk: the worker is idempotent, so when the agent restarts, it goes through the steps of noticing the remote app, creates the relation watcher, sees the relation, acts on it etc
<babbageclunk> wallyworld: the "good" news is that I get the same behaviour whether I do it as two steps consume/relate or directly in relate.
<wallyworld> hmmm
<wallyworld> babbageclunk: did you definitely let the apps finish installing etc before opening the browser? ie status shows idle for each
<wallyworld> it will show 502 if wordpress is still doing its thing
<wallyworld> wordpress is not installed when deploy is done, only when the relation is joined
<wallyworld> and the install downloads a blob and does stuff to set up which takes time
<babbageclunk> wallyworld: no, I waited until everything is idle
<wallyworld> ok, i'll run up a system and test, but it worked yesterday i swear :-)
<babbageclunk> wallyworld: :) I might have broken relate too (although I haven't really touched it) - I'll try with develop
<wallyworld> ok, i'm sure it willbe something "dumb" once we find it
<babbageclunk> can't see anything obvious in the log yet - I can see messages from juju.worker.uniter.remotestate saying "got relations change: ok=true"
#juju-dev 2017-01-06
<wallyworld> babbageclunk: i also see all the relation joined log messages, so the worker is doing its job
<wallyworld> just no hooks firing
<veebers> babbageclunk, wallyworld would either of you have a moment to help me with an error i'm seeing when deploying to GCE?
<veebers> I see an error re: no lxd socket, that seems odd right? (http://pastebin.ubuntu.com/23749591/)
<wallyworld> veebers: looks like lxd is not installed. i blame the image
<veebers> wallyworld: ah ok, who could i ping about that? sinzui would you have insight?^^ :-)
<wallyworld> veebers: although, it could be that lxd init has not been run; i can't recall the workflow there as to when that gets done. i assume juju works with lxd for other clouds
<wallyworld> there has been a couple of lxd commits lately but nothing i know of that would break this way
<sinzui> veebers: xenial always has lxd and juju does init it when it is needed I have not seen fail before. But the "lxdbr0" addresses: route ip+net: no such network interface" could happen under new juju rules where lxdbr0 will not be created until the user deploys a container to the host
<veebers> sinzui: I see this happening in my perfscale test, just after a bootstrap but before I deploy any bundle/charm (well I it happened during a deploy once, but every other time before a deploy)
<veebers> sinzui: I don't think I'm doing something I shouldn't, unless I'm missing something?
<veebers> wallyworld: I see this is syslog, does that point to the image like you suggested? juju-aa2226-0 google: No startup script found in metadata.
<wallyworld> veebers: not sure
<wallyworld> veebers: it could be that cloud init script is not being run, but it would need debugging to dig into it. it could just be a harmless warning also
<veebers> wallyworld: hmm ok, any suggestion on debugging this further?
<wallyworld> veebers: not offhand, it's just a matter of turning on debug and trawling through the logs. if juju works elsewhere with the same setup it's likely to be gce specific
<wallyworld> i've not come across the issue before so don't have any suggestions off the top of my head
<veebers> wallyworld: sweet thanks, I'll see what I can dig up
<wallyworld> babbageclunk: found the issue. it's a problem with the remote app name. i have to duck out for 15 minutes but will fix. i need to figure out how to fix as well
<babbageclunk> wallyworld: ok, good stuff
<wallyworld> babbageclunk: once fixed, i'll ping you, and also need to discuss a proper fix as we will need to extend the data model
<babbageclunk> wallyworld: ok
<wallyworld> babbageclunk: yay fixed, will write tests but after i duck out for 15 minutes
<babbageclunk> wallyworld: awesome
<mup> Bug #1484105 opened: juju upgrade-charm returns ERROR state changing too quickly; try again soon <bug-squad> <canonical-is> <upgrade-charm> <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1484105>
<blahdeblah> Anyone around who can advise on the best course of action with this bug? ^
<blahdeblah> It's affecting a production site, and juju has basically stopped in the middle of a charm upgrade, with roughly half the units upgraded, and half not.
<blahdeblah> even all-machines.log is becalmed, with only one unit logging this repeatedly: https://pastebin.canonical.com/175051/
<blahdeblah> (and not in all-machines.log, but in its own unit log)
<mup> Bug #1484105 changed: juju upgrade-charm returns ERROR state changing too quickly; try again soon <bug-squad> <canonical-is> <upgrade-charm> <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1484105>
<mup> Bug #1484105 opened: juju upgrade-charm returns ERROR state changing too quickly; try again soon <bug-squad> <canonical-is> <upgrade-charm> <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1484105>
<blahdeblah> I've done a full restart of machine 0, and am part way through a full restart of every juju agent
<blahdeblah> Is this perhaps a case when we might want to try the mongodb cleanup?
<wallyworld> babbageclunk: https://github.com/juju/juju/pull/6773 i am retesting a second time now from scratch
<wallyworld> blahdeblah: what version of juju is this?
<wallyworld> the bug talks about 1.18
<wallyworld> surely not
<babbageclunk> wallyworld: I don't quite understand the names you're using - what will the remoteApplicationName and -Alias be?
<wallyworld> babbageclunk: the name is what is is on the offering side ie the actual name. the alias is what it is known as locally in the consuming model
<babbageclunk> wallyworld: Would localAlias be a better name, since it's the local name for the remote application?
<wallyworld> sure
<babbageclunk> wallyworld: So the aliasing would be done at consume or relate time - you'd say how to refer to the remote application in the consuming model. Is that right?
<wallyworld> yes
<wallyworld> we need to add that
<wallyworld> so for now it is hard coded to user-model-appname
<babbageclunk> wallyworld: ok, got it.
<babbageclunk> wallyworld: LGTM'd
<wallyworld> awesome thanks
<blahdeblah> wallyworld: 1.25.9
<wallyworld> blahdeblah: without digging in and doing a mongo dump and/or model dump and debugging, it's hard to say off hand what the issue would be
<wallyworld> babbageclunk: looks like there's one more small issue with relation ket naming to fix (use alias vs name). i'll track that one down and land but it's tedious to debug
<babbageclunk> wallyworld: ok
<babbageclunk> wallyworld: let me know if you want a rubber ducky to talk it through with
<wallyworld> babbageclunk: it's just tedious grunt work to find the needle in the haystack sadly
<babbageclunk> wallyworld: yeah, fair enough
<wallyworld> babbageclunk: i just did 2 fresh bootstraps and deployments and everything looks ok. i must have stuffed something up when testing before. sigh. so landing now
<wallyworld> babbageclunk: if you were finished your work and posted the PR, I could review pending any final testing you may want to do
<babbageclunk> wallyworld: The tests are done, but I haven't done the fixing up you suggested in add relation yet.
<babbageclunk> wallyworld: I'll make a PR now anyway
<wallyworld> which was that again? mental blank
<babbageclunk> 13:30 <wallyworld> babbageclunk: the other tweak with add relation on the server side is to account for that consume could have been run first. we do check that the remote application does not exist when we go to save it, but we will need to tweak things to be a bit smarter. check early for existence, plus in the case of offers, account for that the offer may have been updated with new endpoints so a new save is ind
<babbageclunk> eed required
<wallyworld> babbageclunk: ah right. that can be a follow up IMO
<babbageclunk> wallyworld: ok cool
<wallyworld> that way i can grab what you have done and try it etc
<wallyworld> behind a feature flag means we can be more lenient with what we land
<wallyworld> in terms of a bit at a time with cavearts to its usage
<wallyworld> babbageclunk: landed!
<babbageclunk> wallyworld: here's my PR.
<babbageclunk> https://github.com/juju/juju/pull/6774
<babbageclunk> wallyworld: I'll rebase it and test now.
<wallyworld> ty, will look at pr
<redir> g'nite
<babbageclunk> redir: bye - if you lived here it'd be the weekend now!
<redir> they end?
<redir> have a great weekend down under and in kiwiland
<wallyworld> babbageclunk: lgtm with some nitpics, tyvm. We really should introduce  a proper Endpoint attr to the url struct
<wallyworld> babbageclunk: i've updated the working google doc with the latest todos
<babbageclunk> wallyworld: Yeah, I almost added an endpoint attr to it.
<wallyworld> i was just in a rush to get shit done
<babbageclunk> wallyworld: weird - I can't get mysql to start
<wallyworld> as in juju deploy mysql
<babbageclunk> yup - keeps dying with Fatal error: cannot allocate memory for the buffer pool
<babbageclunk> It was working this morning
<wallyworld> hmmm, there used to be an issue with lxc but not for ages
<wallyworld> reboot? :-)
<babbageclunk> yeah, I killed off lots of leaky chrome tabs, seems to have fixed it
<wallyworld> i use a suspend tabs plugin with chrome
<wallyworld> stops leaks. google docs is sweful for that
<babbageclunk> ok, finally ran through the QA steps successfully - sorry, got interrupted by the dinner/bath/bed conveyor belt.
<babbageclunk> wallyworld: is there anyway to remove a relation to a remote application? remove-relation isn't working for me.
<wallyworld> babbageclunk: yeah, all that is on the todo list. remote app and remove relation for remote. i'm 1/2 way through remove app
<wallyworld> thanks for the extra hours, much appreciated
<babbageclunk> wallyworld: no worries! Sorry it ended up being so last minute.
<wallyworld> tis ok, all par for the course
<frankban> hey wallyworld, is the branch from huw good enough for you? even without the additional information to be retrieved with appInfo call?
<wallyworld> frankban: yes, it progress :-) i haven't tried it out though as am not sure how. but if we can display *something* initially, we can iterate
<wallyworld> frankban: also found out today lightning talk got moved so we'll have a couple of days as well to get stuff tidy
<frankban> wallyworld: so yes to try it out, 1) checkout that branch, say to ~/code/juju-gui/ 2) make sysdeps 3) make dist 4) juju upgrade ~/code/juju-gui/dist/jujugui-2.2.5.tar.bz2
<wallyworld> oh easy, awesome
<frankban> wallyworld: sorry, 4) is juju upgrade-gui, not just upgrade
<wallyworld> have to pack and plane leaves in about 9 hours and i need sleep so may need to try it out once i land
<babbageclunk> wallyworld: I still get a bad gateway error from wordpress if I try to consume it using url rather than model.application.
<wallyworld> babbageclunk: hmmm, ok, i can check that, i tested the model.app method
<wallyworld> i'll fix over the weekend
<babbageclunk> wallyworld: I've been trying to work out what the problem is but I haven't got it yet.
<wallyworld> babbageclunk: tis ok, i can fix. the main use case is model.app
<wallyworld> babbageclunk: damn test sfailed
<babbageclunk> Yeah - it's legit though - the one that checks all of the commands have help?
<babbageclunk> Fixed now - I don't really get the point of the test though.
<babbageclunk> https://github.com/juju/juju/pull/6774/commits/965d0371256dbcc2436dcbc3e7cfa231b71628f1
<babbageclunk> If you've forgotten to register a command you're very unlikely to have remembered to add it to this test.
<wallyworld> babbageclunk: yeah +1 to all that
<babbageclunk> wallyworld: d'oh, missed that the list was supposed to be alphabetical.
<wallyworld> babbageclunk: so, i have code to delete remote apps.. but need to add undertaker stuff. i think that need sdoing before deleting relations. so maybe you could next week pick up the dump modle bug and app url endpoint?
<wallyworld> and local alias work
<wallyworld> i'm sure priorities wil change once sprint starts
<babbageclunk> wallyworld: I'm off for Mon-Weds next week (sorry, really should have mentioned!) but I'll pick those up after.
<babbageclunk> hang on - dump model bug?
<wallyworld> ok, enjoy
<wallyworld> yeha, if you run dump-model and there are remote relations, it fails
<babbageclunk> wallyworld: ah, makes sense
<wallyworld> ty again for ll the work
<wallyworld> i have to go pack some mor
<wallyworld> e
<babbageclunk> ok, don't forget anything!
<mup> Bug #1654528 opened: log forwarding broke between 1.25.6 and 1.25.9 on trusty <juju-core:New> <https://launchpad.net/bugs/1654528>
<rogpeppe1> is anyone around that knows how rsyslog forwarding works in juju?
<rick_h> not sure, what's up? I did see a bug go by overnight about it being broken in 1.25.9 due to a change in the tls ciphers available not being allowed any more.
<rogpeppe1> rick_h: yeah, that's what this was about.
<rogpeppe1> rick_h: i have a better idea now after rummaging around in the code a bit.
<rogpeppe1> rick_h: this should fix it: https://github.com/juju/utils/pull/257
<rogpeppe1> is there anyone in juju-core that might be able to review the above, please? voidspace, are you on-call reviewer by any chance?
<rogpeppe1> rick_h: there's some concern that the CI tests didn't find this problem before release.
<natefinch> rogpeppe1: can you put that at the bottom of the list so we don't choose it before the ecdhe ones at the bottom?
<rogpeppe1> natefinch: ok
<natefinch> rogpeppe1: otherwise lgtm
<rogpeppe1> natefinch: done
<rogpeppe1> natefinch: wanna make your mark on the PR?
<Spads> oh yes hi
<Spads> rick_h: natefinch: so this bug is important to our trusty 1.25 envs including ps45-jujucharms, but there's also likely some jujudb corruption that axino put in a bug earlier that we'll need to diagnose and sort out if juju is to do anything more in that env
<rick_h> gotcha rogpeppe1, ty for the update.
<Spads> rick_h: so can we get someone to help us diagnose the jujudb for ps45-jujucharms?
<Spads> https://bugs.launchpad.net/juju-core/+bug/1484105/comments/16 <-- the meat of the matter
<mup> Bug #1484105: juju upgrade-charm returns ERROR state changing too quickly; try again soon <bug-squad> <canonical-is> <upgrade-charm> <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1484105>
<rick_h> Spads: k, updated the bug https://bugs.launchpad.net/juju-core/+bug/1654528 with it in progress with rogpeppe1's patch
<mup> Bug #1654528: log sending broke between 1.25.6 and 1.25.9 on trusty <juju-core:In Progress by rogpeppe> <https://launchpad.net/bugs/1654528>
<Spads> rick_h: sure, but we still have what we believe is jujudb corruption that needs fixing, no?
<rick_h> Spads: looking at the other bug
<Spads> ta
<rick_h> Spads: sooo, looking at the bug and the diagnosis you all did, I don't know I have folks that know that working bits on hand. They're traveling to cape town atm :/
<Spads> yikes
<rick_h> Spads: I can see if Alexis has someone coming in that might know more, but that transaction bits can be tricky and I don't want to accidently do more damage
 * Spads nods
<Spads> yeah the transaction system kind of blows my mind
<rick_h> Spads: what's the current impact/risk level
<rick_h> Spads: e.g. is it worth playing riskier to get something done or better to play it more safe with the right heads?
<Spads> so at the moment the combination of these bugs leaves me with the perception that juju for that environment is either logjammed or fragile enough that we shouldn't touch it
 * rogpeppe1 lunches
<Spads> there's enough blocking communication from agents to controller etc that I'm less worried of something firing off and starting the avalanche
<Spads> but i could be wrong
<Spads> at the very least we should not try any changes or upgrades or anything until we have this whole mess resolved
<rick_h> Spads: +1 to the last bit, the question is can it sit until we can get 'right folks' on the ground?
<Spads> the weekend oncall IS staff will need to be briefed no matter what we do
<Spads> rick_h: when's that?  Monday?  This evening?
<rick_h> Spads: probably sunday evening or monday morning
<Spads> we have folks in the US who can take this over for their friday working hours, but I'd imagine anyone arriving in capetown is going to have other things on their mind
<Spads> yeah
<Spads> so I'm tempted to say that this can wait until Sunday night/monday morning, but I also don't know enough to really be confident of that
<rick_h> Spads: let me start an email thread and see who we can find and I'll get you better info
<Spads> part of the problem is that event logging is wedged, so this could merely be a false perception of stasis in the system
<Spads> for all I know agents are busily rm -rfing / as we type âº
<Spads> not that I think that's likely
<Spads> but the environment got opaque
<rick_h> Spads: k, email away. Give me a couple of hours for folks to get up out of bed and to read it
<Spads> rick_h: thanks.  I'll update our RT about this topic
<rogpeppe1> natefinch: do you know what the current workflow is for proposing changes to 1.2x ?
<natefinch> rogpeppe1: just pr against 1.25 branch AFAIK
<rogpeppe1> natefinch: ok, thanks
<rogpeppe1> natefinch: will do
<natefinch> rogpeppe1: I presume we'll need this for 2.x as well, since we support trusty and precise there, too
<rogpeppe1> natefinch: yeah
<rogpeppe1> natefinch: i'm a bit concerned that no-one seems to have tested this stuff
<natefinch> rogpeppe1: there's a large matrix of variables that contribute to it.  I think the use of log forwarding on precise and trusty is probably one of those corner cases that slipped by
<voidspace> rogpeppe1: sorry, missed your message
<voidspace> rogpeppe1: still need a review?
<natefinch> rogpeppe1: but yes, we should have more comprehensive testing.  I guess log forwarding must not have CI tests
<rogpeppe1> natefinch: we should get log forwarding added to the CI tests
<rogpeppe1> voidspace: thanks, but too late :)
<voidspace> rogpeppe1: ah, no - I see it's merged
<rogpeppe1> natefinch: if you didn't, could you add your actual LGTM to the issue please?
<natefinch> rogpeppe1: done
<rogpeppe1> natefinch: ta
<natefinch> her, evidently you can't technically "approve" a PR after it's merged
<Spads> so I thought that 2.x didn't rely on the same log forwarding channel for basic status stuff
<rogpeppe1> natefinch: i guess you mean target the juju-1.25-beta1 branch?
<rogpeppe1> Spads: "basic status stuff"?
<natefinch> rogpeppe1: the 1.25 branch is used to create tags the juju-* stuff is all tags
<Spads> rogpeppe1: well I was led to believe that 1.x uses log forwarding for a lot more than 2.x does is all
<Spads> and for more essential communication channels
<Spads> I may have misunderstood
<rogpeppe1> Spads: i don't think that log forwarding was ever used for juju status
<Spads> ok
<rogpeppe1> Spads: i think that the mechanisms have changed somewhat, but i don't know how
<natefinch> no no, log forwarding is new in the last year and totally external to anything juju actually does
<Spads> hm
<Spads> interesting
<rogpeppe1> natefinch: but juju 1.25 does log forwarding, right?
<natefinch> rogpeppe1: I don't actually remember
<rogpeppe1> natefinch: that is, agents send logs to the API via rsyslog
<natefinch> rogpeppe1: oh, that log forwarding
<rogpeppe1> natefinch: well, i know they do 'cos i've just looked at the code :)
<natefinch> sorry... there's a new feature for log forwarding external to juju
<natefinch> so I may have been confused about what was broken in our entire conversation here
<Spads> haha
<Spads> I believe it's the gnutls that rsyslog is linked against that's at issue here
<rogpeppe1> Spads: well that's down to the instance that's running the agent, i think
<rogpeppe1> Spads: and we want that to be fairly liberal so you can use a stock precise instance
<rogpeppe1> s/instance/image/ probably
<Spads> right
<Spads> although  really precise hasn't got many heartbeats left
<rogpeppe1> natefinch: the problem, i *think* is that the agents rely on rsyslog to send logs to the juju API server
<rogpeppe1> natefinch: and that uses the gnutls that's installed
<Spads> but I appreciate it's still supported for this code
<natefinch> rogpeppe1: we were working on that... I thought we had removed the reliance on rsyslog
<rogpeppe1> natefinch: and that didn't have any ciphersuites in common with the suites supported by juju
<rogpeppe1> natefinch: in 1.25?
<rogpeppe1> Spads: it's going to be hard moving away from precise when so many charms require it
<natefinch> rogpeppe1: I thought so.  We were having trouble with it, I thought we had switched to using go code to send the log messages.  Receiving was still done via syslog, though, IIRC, which may still cause this problem, maybe?
<Spads> rogpeppe1: You are not wrong!
<rogpeppe1> natefinch: yeah, either way is a problem
<natefinch> rogpeppe1: ok :)
<rogpeppe1> natefinch: if receiving is done by syslog, what's with the logSinkHandler?
<Spads> so part of this is that you guys wisely limited the log code to non-terrifying ciphersuites
<natefinch> heh
<Spads> so there's a valid perspective that rsyslog should really do the same
<Spads> but then you could totally end up with precise and xenial being unable to rsyslog to each other
<natefinch> or someone could update precise to use non-terrifying ciphersuites, too
<rogpeppe1> Spads: i guess the first thing to come up with is a set of suites that are a) non-terrifying and b) in common with all the platforms we care about.
<natefinch> rogpeppe1: I'm not sure what the purpose of logSinkHandler is
<Spads> rogpeppe1: we found at least one good one for trusty
<Spads> rogpeppe1: and axino pastebinned a list for precise but i haven't cross-referenced them yet
<rogpeppe1> Spads: yeah, i think that we'll be good with the one i added
<Spads> is it in the precise listing?
 * Spads closes tabs trying to find the pastebin
<Spads> https://pastebin.canonical.com/175080/ <-- that's the badger
<Spads> TLS_DHE_RSA_AES_256_CBC_SHA1                            0x00, 0x39      SSL3.0
<Spads> â¬ was that what we settled on?
<Spads> 12:14  <axino> aka TLS_RSA_AES_256_CBC_SHA1 for gnutls
<Spads> oh
<Spads> TLS_RSA_AES_256_CBC_SHA1                                0x00, 0x35      SSL3.0
<Spads> that one
<Spads> no diffie-helman exchange
<Spads> rogpeppe1: what you said about precise reminded me that there's no facility to upgrade releases in juju, and I wanted to check the status of any bugs on that even if they're wontfix, but i can't find one
<rogpeppe1> Spads: there's always a DH exchange in TLS, right? otherwise you wouldn't get a shared key AIUI. I'm presuming the DHE is another exchange for some reason.
<Spads> yeah in TLS
<Spads> hm
<Spads> I'd have to read up
<Spads> ah it's totally ephemeral vs static
<Spads> but who uses static...
 * rogpeppe1 doesn't know about ephemeral vs static DHE
<rogpeppe1> natefinch: an identical PR targetting utils 1.25 branch: https://github.com/juju/utils/pull/258
<rogpeppe1> natefinch: please rubberstamp :)
<voidspace> rick_h: I lost network, trying to rejoin now
<voidspace> rick_h: google doesn't seem keen on letting me join
<voidspace> ah, in
<rogpeppe1> Spads: ok, now i get it. ephemeral DH is definitely the thing to do.
<natefinch> rogpeppe1: LGTM
<rogpeppe1> natefinch: ta
<rogpeppe1> natefinch, Spads: this should fix https://bugs.launchpad.net/juju-core/+bug/1654528
<mup> Bug #1654528: log sending broke between 1.25.6 and 1.25.9 on trusty <juju-core:In Progress by rogpeppe> <https://launchpad.net/bugs/1654528>
<rogpeppe1> https://bugs.launchpad.net/juju-core/+bug/1654528
<natefinch> rogpeppe1: thanks for picking that up
<rogpeppe1> natefinch: np
<rogpeppe1> natefinch: any chance of a review?
<rogpeppe1> natefinch: it is trivial
<natefinch> rogpeppe1: sorry, you posted the bug link twice, didn't realize there was a PR to review
<rogpeppe1> natefinch: ha ha
<rogpeppe1> natefinch: one mo :https://github.com/juju/juju/pull/6775
<natefinch> rogpeppe1: shouldn't it just be updating the utils dep?
<rogpeppe1> natefinch: it is
<rogpeppe1> natefinch: the others are... i don't know why that happens
<rogpeppe1> natefinch: but the commits are the same
<rogpeppe1> natefinch: just the dates changed
<rogpeppe1> natefinch: i really don't understand how the same commit could get two dates
<natefinch> og yeah, huh,  missed that
<natefinch> man, who wrote this damn tool ;)
<natefinch> it's git, who really understands any of it
<rogpeppe1> natefinch: git?
<natefinch> I presume the timestamps are from git, right?
<rogpeppe1> natefinch: yeah, godeps uses git log -n 1 '--pretty=format:%H %ct' HEAD'
<rogpeppe1> natefinch: to print the commit date
<rogpeppe1> natefinch: i don't know how that could cause a different unix timestamp to be printed for the same commit
<rogpeppe1> natefinch: i have a suspicion that it might be caused by people changing dependencies.tsv manually
<natefinch> rogpeppe1: could be.  usually I edit it by hand, but just replace a whole line with a whole line generated by godeps
<rogpeppe1> natefinch: i'd recommend never editing it by hand
<rogpeppe1> natefinch: then you can't get it wrong
<natefinch> rogpeppe1: is the OS problem fixed?  used to be that godeps wouldn't show depenedencies in code that isn't compiled for the current OS
<rogpeppe1> natefinch: GOOS=windows godeps -t ./...
<rogpeppe1> natefinch: i need to work out a decent solution to the OS problem
<rogpeppe1> natefinch: and some time to implement it
<natefinch> *nod*
<natefinch> rogpeppe1: I think one run can't get all our deps, we have both linux-specific and windows-specific deps now
<rogpeppe1> natefinch: we have whole packages which are only included under linux?
<rogpeppe1> natefinch: in general i think that's a bad idea
<rogpeppe1> natefinch: but i guess i see how it can happen
<natefinch> rogpeppe1: I think the lxd stuff is only imported for linux
<natefinch> rogpeppe1: or at least transitive dependencies thereof
<rogpeppe1> natefinch: hmm, how is anyone meant to update godeps now?
<perrito666> brb lunch
<natefinch> rogpeppe1: I usually know what is changing and so I know what needs to be updated, so I just godeps -t ./... | grep thepackage
<natefinch> and then just manually replace that line in the current dependencies.tsv.... not ideal, obviously
<rogpeppe1> natefinch: i really need to fix godeps to make this work
<natefinch> rogpeppe1: one easy fix could be a wrapper around godeps that just unions two runs of godeps for win/linux
<rogpeppe1> natefinch: yeah - the solution i've seen elsewhere is to provide a list of set sets and do everything on each tag set
<natefinch> rogpeppe1: seems easy enough.  Slower than trying to do it all in a single run, but a ton easier to implement
<natefinch> and easier to know you've gotten it right
<rogpeppe1> natefinch: hmm, our old review links don't seem to work any more. http://reviews.vapour.ws/r/4396/
<natefinch> rogpeppe1: doh...
<natefinch> sinzui: is the reviewboard instance supposed to be working?
<sinzui> natefinch: It appeared to be up yesterday. We don't touch it
<natefinch> sinzui: blah
<natefinch> and this is why I'm glad we moved to github reviews
<sinzui> natefinch: I see it is angry. I will take a look at the host. I suspect disk space because that affected 2 other machines this week
<rogpeppe1> natefinch: yeah, at least one of them looks pretty clearly like a manual edit: 75dd020b936fdcf67f692eef3ee3bb74e2e248ed
<natefinch> rogpeppe1: we should send an email to juju-dev saying don't do that.  At least run godeps and copy the entire line, I think that's a reasonable enough way until we can just run godeps -t ./... > dependencies.tsv
<Spads> rogpeppe1: natefinch so that fix is much appreciated, unfortunately I need someone who understands the transactions db stuff to fix https://bugs.launchpad.net/juju-core/+bug/1484105/comments/16 before we can test it âº
<rogpeppe1> Spads: i understood (some of) the txn stuff once
<rogpeppe1> Spads: but i need to re-learn it every time i dive in
<Spads> rogpeppe1: so my plan right now is to shut down the jujuds on the controller to prevent accidental mishap over the weekend
<Spads> rogpeppe1: but if there's any read-only diagnostic work we can to to add on to junien's that may be a good thing to sort out now
<natefinch> state changing too fast means the asserts in the actual db ops don't match the coded prechecks we run before running the transaction
<rogpeppe1> Spads: the txn-revno assertion is usually to assert that nothing changed in the doc
<Spads> interesting
<natefinch> we should make a new bug each time this happens because it's almost never the same code twice
<natefinch> it's just a really uninformative error message
<Spads> a pity lp won't split bugs as easily as it merges âº
<sinzui> natefinch: reviewboard is our of disk. I am going to restatrt the host, the clear out a few G of tmp files
<Spads> natefinch: rogpeppe1: so does this look like something that would sort itself out if we quiesced the controllers, or is it something dire that needs surgery?
<rogpeppe1> Spads: i don't know without looking at it in more detail
<Spads> ok
<perrito666> sinzui: I am curious dont we have automatic checks for stats on our machines?
<natefinch> Spads: it's almost never due to actual state changing too fast
<natefinch> Spads: it's almost always a bug in the code where the prechecks in go code don't match what the mgo assertions are checking for.
<rogpeppe1> Spads: i wonder if just inserting a txn-revno:0 field into the doc might fix it
<Spads> heh
<natefinch> sinzui: thanks for the help
<Spads> so do either of you think my plan to just shut down jujuds is a good idea?
<Spads> so that we can freeze the env in time
<Spads> or should we leave it running in this state
<Spads> as it has been all day really
<rogpeppe1> Spads: this doesn't seem like a real db corruption issue to me
<rogpeppe1> Spads: just a bug in the juju transaction logic
<natefinch> +1
<Spads> oho!
<Spads> okay that's very good to hear
<rogpeppe1> Spads: i think the problem is probably that *most* places that insert a document insert a struct that has an explicit txn-revno field, so it will be inserted with the value 0
<rogpeppe1> Spads: but for settings it inserts a map
<rogpeppe1> Spads: which won't have that field
 * Spads nods
<Spads> what's the difference between a struct and a map?
<Spads> they both sound like key/value data structures to me âº
<rogpeppe1> hmm, but this is statuses not settings
<rogpeppe1> Spads: a struct is like a C struct - it has a known set of fields
<sinzui> natefinch: we don't have status checks on reviewboard or the reports site...we do on the other sites
<Spads> ah okay, so like a named tuple then
<sinzui> natefinch: reviewbaord is up
<rogpeppe1> natefinch, Spads: ah, i think the issue is that statusDoc doesn't have a txn-revno field
<rogpeppe1> Spads: yeah, kinda
<Spads> oho
<rogpeppe1> natefinch, Spads: i'm trying to work out how it could ever work
<Spads> haha
<natefinch> lol
<natefinch> I can't tell you how often I've looked at code and wondered that
<rogpeppe1> natefinch, Spads: in fact, it looks like the txn package always inserts a txn-revno field
<balloons> can someone have a quick peek at https://github.com/juju/juju/pull/6776?
<Spads> rogpeppe1: so how did it not get in?
<rogpeppe1> Spads: an excellent question
<Spads> or is it trying to insert and failing and that's the error?
<Spads> operation-ordering bug?
<natefinch> balloons: lol, I just changed this to beta4 yesterday
<natefinch> balloons: did we do what we needed to do with it at beta4?
<balloons> natefinch, lol, I know.. fun isn't it?
<balloons> natefinch, the release today is beta4, so we had to make the source reflect that for one commit to release it
<natefinch> heh
<natefinch> ok
<natefinch> balloons: lgtm
<mup> Bug #1646777 changed: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:Fix Released by nskaggs> <https://launchpad.net/bugs/1646777>
<redir> easy review https://github.com/juju/juju/pull/6777
<redir> PTAL
<natefinch> redir: ahh, so we were accidentally not building on s390x I guess?
<redir> yeah
<redir> :/
<xnox> oooooo =( please build on s390x
<natefinch> redir: yeah, architecture restrictions don't seem to make sense in go code most of the time.
<xnox> things should build there.....
<xnox> natefinch, we like use juju to deploy openstack on s390x =/
<redir> If it is not supported should it build?
<xnox> so it's not a toy port
<xnox> redir, we contractually support it
<natefinch> it's just a single file
<redir> kvm?
<natefinch> not the whole thing
<redir> right just kvm
<xnox> we support everything that works, which is lxd/local, kvm, openstack providers.
<natefinch> redir means kvm doesn't work on s390x so we were avoiding compiling some stuff for kvm, but then it broke other stuff
<xnox> there are no public clouds for s390x.
<xnox> natefinch, qemu, kvm, nova, openstack all work on s390x. there maybe some switches that are not supported by qemu-system-s390x which may be used.
<xnox> natefinch, redir - if there are any s390x specific bugs, I can look into that for you and submit merge proposals.
 * xnox had to s/pci/ccw/ in some places before
<natefinch> sorry, I think I misrepresented what redir was saying.  Anyway, the bug was that we were not compiling this one file on s390x and should have been
<xnox> ok
<redir> either this one file or the one that defines that it sin't supported
<xnox> nonetheless if s390x-arch things do creep up, feel free to ping me =)
<redir> will do xnox
<redir> xnox: to be clear it should build on s390x
<rogpeppe1> natefinch: i just did some work on making go/build work with any build tags: https://github.com/rogpeppe/godeps/pull/5
<rogpeppe1> natefinch: s:go/build/godeps/
<rogpeppe1> natefinch: feel free to review if you like
<mup> Bug #1646777 opened: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:In Progress by nskaggs> <https://launchpad.net/bugs/1646777>
<natefinch> rogpeppe1: looking
<mup> Bug #1646777 changed: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:In Progress by nskaggs> <https://launchpad.net/bugs/1646777>
<mup> Bug #1646777 opened: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:In Progress by nskaggs> <https://launchpad.net/bugs/1646777>
<natefinch> perrito666: when you get a chance: https://github.com/juju/juju/pull/6778
<perrito666> natefinch: well the power on the computer with my code is down so why not now?
<natefinch> perrito666: seems like your town needs to double down on its street drainage systems
<perrito666> natefinch: done
<perrito666> natefinch: that is just my neigbourhood, I live in a closed comunity surounded by other closed comunities so we have a shared draining system that uses, you guessed, my street, its not that deep never goes over 15cm
<perrito666> natefinch: and it drains as soon as rain stops, it all goes to a big wather buffer down my street
<natefinch> I think your definition of a drainage system needs updating if "your street" is something that can be part of it :)
<redir> closed source drainage
<perrito666> natefinch: if you lived in south america you would prefer your street to have a few cm of water instead of the possibility to have water stalling in a drainage (mosquitos)
<redir> proprietary drainage
<perrito666> natefinch: also there is no sewage here so it all goes to nearby fields where its absorbed
<redir> frobware: yt?
<redir> I'm gonna doubt it:)
 * perrito666 should really have bought that new ups this week
<perrito666> ccccccgcbucdlrnrfidiurjltlkrlkhgdhjghrlfcbgf
<natefinch> perrito666: btw, what endpoints do you think will be incorrectly handled with the code I wrote for ping?
<natefinch> perrito666: I doubt giving a gopher:// URI to the vsphere provider is going to work.
<perrito666> natefinch: oh no, I just suggest that you check for that and fail early
<perrito666> anyone needs reviews?
<TheMue> perrito666: already done some today, and my latest branch is reviewed and merged. :)
<TheMue> perrito666: establishing some good procedures of our Canonical way of work in my new team.
<perrito666> heh, I wold prefer to do juju reviews though :p
<TheMue> perrito666: none open?
<perrito666> TheMue: was looking for someone with priority since no one says anything ill default to the open ones
<TheMue> perrito666: yeah, standard business has to be done too
 * redir lunches
 * redir back
<perrito666> k, ill EOD because my laptop's battery is about to die
<perrito666> cheers
<mup> Bug #1646777 changed: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:Fix Released by nskaggs> <https://launchpad.net/bugs/1646777>
<redir> later perrito666
#juju-dev 2017-01-07
 * redir eow shortly
<redir> happy weekend
#juju-dev 2017-01-08
<redir> morning
<redir> or g'day
 * redir lunches
<redir> back
<thumper> anastasiamac, babbageclunk: morning
<anastasiamac> thumper: morning \o/
<thumper> The standup time doesn't work for me today
<thumper> and since ian, andrew and menno aren't around
<thumper> I was hoping we could go earlier
<anastasiamac> thumper: anytime works for me... not that there is much to discuss either
<thumper> yeah...
 * thumper waits for babbageclunk
 * thumper isn't 100% certain he is working
<anastasiamac> :D that'll make standup even more laconic: emails, bugs... not much else? unless u have other news?
<thumper> I'm looking into a few production bugs
<thumper> memory leaks etc
<thumper> nothing much to report
<anastasiamac> \o/
<anastasiamac> k, we can standup tomorrow, I guess
<thumper> ok
<thumper> sounds good
<redir> I've got a couple easy PRs up
<redir> hope to get one more out today.
<thumper> redir: ack
#juju-dev 2018-01-01
<gamamtz> hi guys, can i ask?
<gamamtz> does anyone know how to specify the network id in the juju deploy command for a controller in openstack with multiple networks?
<gamamtz>  i solve it, with juju model-config network=networkid
#juju-dev 2018-01-02
<axw> *chirp*
<blahdeblah> axw: was that your cricket impression? :-)
<axw> blahdeblah: yes :)
<blahdeblah> axw: If your crickets want something to do, I've just created https://bugs.launchpad.net/juju/+bug/1740815, which seems like a race condition in status retrieval.
<mup> Bug #1740815: ERROR model "modelname" has been removed from the controller, run 'juju models' and switch to one of them. <juju:New> <https://launchpad.net/bugs/1740815>
<axw> blahdeblah: ok, thanks. fixing tests atm, will take a look in a little while
<blahdeblah> axw: Thank you! :-)
<axw> blahdeblah: which version of juju is that?
<blahdeblah> 2.2.6 client, 2.2.8 agents - I'll add that to the bug
<axw> there have been some fixes in this area
<axw> ta
<blahdeblah> updated now
<thumper> morning folks
 * thumper goes through the mountain of email and creates a new year todo list
<hml> morning thumper
<thumper> hey hml
<thumper> how was your break?
<hml> thumper: good - only now itâs cold at home.  you?
<thumper> pretty good, very relaxing break away from the laptop
<thumper> now I have lots of email to look at
<hml> ha
<hml> always the problem with a break - the email
<thumper> hml: looking at the holidays in hr, not many of us working this week
<hml> thumper: half the team?
<thumper> over
<hml> :-)
<thumper> sorry, over half not working
<hml> nice easy start to the year then?
<thumper> :)
<thumper> perhaps
#juju-dev 2018-01-03
<axw> thumper: are you able to add agprado to juju-qa on launchpad or whatever is required to get him access to cloud-city?
<thumper> possibly
<thumper> is that his lp id?
<axw> thumper: yes
 * thumper headdesks
<thumper> axw: what are you up to?
<axw> thumper: standup atm. fixing bugs otherwise
<thumper> axw: I have a doozy for ya
<thumper> axw: ping me when you are done
<thumper> axw: agprado should have access now
<axw> thumper: thanks
<axw> thumper: done
<wgrant> thumper: Is this the doozy that will make me very happy? :)
<thumper> wgrant: one of them is
<thumper> axw: with you shortly, just composing an email
<axw> sure
<thumper> axw: 1:1 HO?
<axw> thumper: ok, brt
<thumper> axw: I suggest we just skip the tech board today
<axw> thumper: I had unconsciously skipped it anyway. sounds good
 * thumper nods
<hml> rick_h: are you back from vacation?
<rick_h> hml: wheee
<hml> rick_h: hope you had a good holiday!
<rick_h> hml: always good to send some time with the fam
<hml> rick_h: i have two invites for the juju show - now on competing weeks
<hml> not sure which is which :-)
<hml> wanted to forward to our new guy agprado
<rick_h> hml: my bad, I moved it to next week...poorly I see
<agprado> hello rick_h
<agprado> I hope your vacation was good!!
<rick_h> hml: hmm, I don't see two, just one.
<rick_h> agprado: what's your first/last so I can invite you via calendar?
<hml> rick_h: maybe i just didnât clean up my calendar then - so the one next week?
<hml> itâs for a full hour instead of a half?
<rick_h> hml: rgr, too short a runway this week
<rick_h> hml: meh, we aim to do 30min but sometimes it runs long so I think we just block out the time
<hml> figured
<hml> k
<rick_h> agprado: you should have an invite
<hml> thx rick_h!
<rick_h> hml: ty for the sharing
<hml> agprado: the juju show is a video chat on various juju topics - previous editions can be found on youtube -
<agprado> I've been watching the juju-show during this vacation
<agprado> rick_h: Adrian Prado. So you can add me via calendar
<rick_h> agprado: cool you should have an invite your way
<agprado> thanks rick_h
<agprado> invitation accepted!
<hml> agprado: howâs it going today?
<agprado> it's going good
<hml> :-)
<agprado> I am in the process of compiling juju
<hml> which version of go are you using?
<agprado> and writing down all the little friction points that I found in the way
<agprado> master branch head
<agprado> if you prefer I can pull the tags, and compile a specific version
<hml> shouldnât be any friction pointsâ¦.
<hml> try develop instead
<agprado> ok
<hml> for a branch - thatâs where most work is done
<hml> we donât really use master any longer
<agprado> ok
<hml> most work is in develop - unless you are backporting or on a specific bug
<hml> what sort of issues are you seeing?
<agprado> it is not related to juju directly, but when I have time, I want to create a rule for the firewall so it is possible to just uwf allow juju
<agprado> and compiling it is more a problem of my golang environment than juju itself. I am lacking packages, and dependencies
<hml> did you set JUJU_MAKE_GODEPS=true?
<agprado> nothing that should take too long to fix, and have a development state
<agprado> nope :)
<hml> what version of GO are you using?
<agprado> 1.8.3
<hml> setting JUJU_MAKE_GODEPS will help with the dependencies etc
<hml> make will use it for godeps, or you can check the Makefile for godeps
<hml> I think weâve moved to go 1.9 nowâ¦ we do hit issues with differences in the compile version from tiem to time
<agprado> ok
<agprado> I can update to 1.9
<hml> i use the go snap - itâs a little easier to find the versions you need - at least to me
<agprado> I found goenv
<agprado> a small application similar to pyenv (manages python version and virtualenvironments)
<agprado> with goenv you an install and mantain different versions and I guess that environments too
<hml> ah - weâve started to use snaps for lots of stuff - including go and juju
<agprado> snap it is :)
<hml> :-)
<rick_h> thumper: are we chatting today?
<thumper> yeah
<thumper> just coming
<rick_h> thumper: ok omw
#juju-dev 2018-01-04
<thumper> axw: morning
<axw> thumper: howdy
<thumper> axw: I threw anouther bug your way
<thumper> axw: after talking with rick_h earlier today
<axw> thumper: yep np
<rick_h> TY axw!
#juju-dev 2018-01-05
<thumper> axw: got some time?
<thumper> axw: I need to talk through some crazy
<axw> thumper: sure
<thumper> axw: https://hangouts.google.com/hangouts/_/canonical.com/broken-magic
<thumper> axw: you might be right on that stashing thing
<axw> thumper: what did you find?
<axw> I haven't found anything yet...
<thumper> I just ran again on lxd
<thumper> and before the machine could come up and write in, both had been set
<thumper> so something else must cut in for it to operate differently
<thumper> it is just a deduction thing,
<thumper> there must be something
<axw> thumper: can you pastebin that output for me?
<axw> the ops you showed me before
<thumper> axw: https://pastebin.canonical.com/206750/
<thumper> oh... yeah
<axw> ta
 * thumper gets them
<thumper> https://pastebin.canonical.com/206751/
<axw> thumper: yup, the "u" field gets decoded as a bson.M. https://paste.ubuntu.com/26323241/
<thumper> wow
<thumper> the stash'll do it then
<axw> it's a wonder it ever worked
<thumper> yeah
<thumper> probably a "feature"
<thumper> for the bson.D that is
<thumper> axw: pretty weird though that what it said it did, and what it actually did were still different.
<thumper> axw: https://github.com/juju/juju/pull/8259
<thumper> axw: mgo.v2/bson/encode.go:219
<thumper> slices of DocElem types are treated like key, value pairs of a dict
<thumper> then bson.decode gets the bson.M back out
<thumper> anyway, patch landed
<thumper> I'm at EOW
<thumper> have a good one
<axw> thumper: thanks, you too
#juju-dev 2018-01-06
<Roamer`> hmm, maybe this is a question for a support channel, not a development one, but how do I get around https://bugs.launchpad.net/ntp-charm/+bug/1736207 - 'unknown OS for series: "bionic"'?  I tried rebuilding the juju 2.3.1 package with https://github.com/juju/juju/pull/8231 and it still fails
<mup> Bug #1736207: Unknown OS for series bionic <juju:Fix Committed by thumper> <NTP Charm:Invalid> <https://launchpad.net/bugs/1736207>
<Roamer`> this is a Xenial 16.04.3 system
#juju-dev 2018-01-07
<prbS5TAB1> ââââââââââââââ LRH IS LIVE NOW!! TODAYS EDITION SLIMER GETS FUCKED IN VEGAS!! https://www.youtube.com/user/l0de/live CALL 315-505-4666 htozfv: icey cholcombe hazmat âââââââââââ
