[00:20] <wallyworld> menn0: only took 3 attempts, but fix has landed, will take a bit to wash through CI, but critical regression fix committed
[00:21] <menn0> wallyworld: glad it's in
[00:21] <wallyworld> must. not. comment.
[00:22] <wallyworld> sadly we still have work to do to make out test suite reliable
[00:27] <bodie_> rick_h_, simply that a change to the charm repo in master is causing juju core tests written against the bit that was removed, to fail
[00:28] <bodie_> it's not really critical to what I'm trying to do right now, but I want to land some changes to charm, and if v4 master isn't working perhaps I shouldn't rebase my changes on master
[00:29] <bodie_> I'll try to get ahold of rogpeppe / uros (who's that?)
[00:42] <menn0> hmmm, although LP shows no critical regressions the buildbot is still blocking merges due to bug 1388493']
[00:42] <mup> Bug #1388493: generate-tools --streams loses content of other streams <ci> <metadata> <regression> <juju-core:Fix Committed by wallyworld> <https://launchpad.net/bugs/1388493>
[00:42] <menn0> and bug 1388073
[00:42] <mup> Bug #1388073: set-env no longer accepts empty values <ci> <compatibility> <regression> <set-env> <juju-core:Fix Committed by tasdomas> <https://launchpad.net/bugs/1388073>
[00:43] <rick_h_> bodie_: urulama sorry
[00:44] <rick_h_> bodie_: roger's lead, the two have been working with charm for some charmstore stuff/etc and are up on the charm package info these days.
[00:52] <bodie_> ah, okay.  thanks rick_h_ :)
[02:58] <menn0> wallyworld_: do we officially support concurrent local provider environments on the one host machine?
[02:59] <wallyworld_> um, no
[02:59] <wallyworld_> not to my knowledge
[02:59] <wallyworld_> i'm not sure it would even work
[02:59] <davecheney> menn0: it works
[02:59] <davecheney> but you'd be bonkers to try
[02:59] <davecheney> we did it in japan
[02:59] <davecheney> 27 local providers
[02:59] <wallyworld_> \o/
[02:59] <davecheney> * footnote: that was in versino 1.14
[03:00] <davecheney> gawdknows what's changed since then
[03:00] <menn0> wallyworld_, davecheney: the reason I ask is that i've just figured out why the rsyslog worker works for all but 1 environ
[03:00] <stokachu> im going to try it
[03:01] <menn0> wallyworld_, davecheney: it writes out configuration files using different certs but all listening on the same rsyslog port
[03:01] <menn0> wallyworld_, davecheney: so only one environment has working logging
[03:01] <davecheney> menn0: rump-row
[03:01] <wallyworld_> menn0: ah, i'm currently wondering why juju debug-log is broken
[03:01] <menn0> and even if you go back to one env
[03:02] <wallyworld_> nodes are failing to send stuff to machine 0 due to certificate issues
[03:02] <menn0> the old rsyslog config doesn't appear to get removed
[03:02] <menn0> so logging is broken from then on
[03:02] <menn0> you need to clean up /etc/rsyslog.d manually
[03:02] <menn0> it's taken me quite some time to figure that out :)
[03:02] <wallyworld_> menn0: i'm seeing this
[03:02] <wallyworld_> 2014-11-03 02:56:34 ERROR juju.worker runner.go:219 exited "rsyslog": x509: certificate is valid for ip-10-46-174-79.ec2.internal, not juju-rsyslog
[03:02] <menn0> wallyworld_: that's the one
[03:03] <wallyworld_> menn0: this is a critical regression for alpha3
[03:03] <wallyworld_> as debug log is just broken
[03:03] <menn0> wallyworld_: it's only for the local provider though
[03:03] <wallyworld_> menn0: nope, also aws
[03:03] <wallyworld_> and probably evrything else too :-(
[03:03] <menn0> wallyworld_: sorry
[03:03] <wallyworld_> that line i just pasted was from an aws deploy
[03:04] <menn0> wallyworld_: I just re-read that error.
[03:04] <menn0> wallyworld_: I was seeing something different
[03:04] <wallyworld_> ah, ok
[03:04] <menn0> x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "juju-generated CA for environment \"rsyslog\"")
[03:04] <wallyworld_> it may well be the same root cause, not sure
[03:04] <menn0> wallyworld_: don't think so
[03:05] <wallyworld_> regardless rsyslog and hence debug log is stuffed
[03:05] <menn0> wallyworld_: the thing i've been investigating is definitely local provider specific when you have multiple environments set up
[03:05] <menn0> sure
[03:05] <wallyworld_> :-(
[03:08] <menn0> wallyworld_: the client side of the connection sets the expected server name to a hardcoded value of "juju-rsyslog"
[03:09] <menn0> wallyworld_: see composeTLS() in worker/rsyslog/worker.go
[03:09] <wallyworld_> menn0: yeah, i'm backing out that change to see if it works
[03:09] <wallyworld_> was changed 10th october
[03:09] <wallyworld_> so recently
[03:10]  * wallyworld_ waits imatiently for aws to start up an instance or two
[03:11] <davecheney> %  git push --set-upstream origin 100-fix-issue-69
[03:11] <davecheney> fatal: https://gopkg.in/juju/charm.v4/info/refs not valid: is this a git repository?
[03:11] <davecheney> I hate gopkg.in
[03:11] <davecheney> https://github.com/juju/charm/pull/70
[03:11] <davecheney> anyone ?
[03:14] <davecheney> wallyworld_: thanks for the review
[03:14] <davecheney> i'm going to wait for rog
[03:14] <davecheney> this is his baby
[03:14] <davecheney> and I have just slapped it, soundly
[03:15] <wallyworld_> sure, looked ok to me.... but im not 100% familiar with it
[03:18] <davecheney> wallyworld_: i'm changing the semantics of the method from what it promised, but didn't deliver
[03:19] <davecheney> to what it does
[03:19] <wallyworld_> right :-)
[03:19] <davecheney> i tried to do it the other way, ie make the methods return a *copy* of the receiver
[03:19] <davecheney> but of course all the tests expect the current behavioru
[03:19] <davecheney> so all broke
[03:19] <davecheney> so, this change renames the method to _say_ what they _do_
[03:20] <davecheney> as well as making them thread safe
[03:20]  * wallyworld_ nods
[03:24] <wallyworld_> davecheney: do you know anything about unit tags gaining a suffix? eg unit-flannel-2[2016]
[03:25] <wallyworld_> menn0: that one line change to remove "juju-rsyslog" worked
[03:25] <wallyworld_> but i'm not sure of the implications of just removing it
[03:26] <menn0> wallyworld_: good.
[03:27] <wallyworld_> the commit message says
[03:27] <wallyworld_> tls.Config clients should use a meaningful ServerName This also fixes an issue with the rsyslog tls.Config not using a ServerName at all.
[03:27] <wallyworld_> well, trouble is, if the rsyslog tls.Config uses a service name, it breaks
[03:27] <menn0> wallyworld_: the docstring for composeTLS (added in that rev) says that setting ServerName is required to support pre-1.20.9 certs
[03:28] <menn0> wallyworld_: so I suspect taking that line out is going to bring back the problem of rsyslog cert issues when upgrading from 1.20 to 1.21
[03:28] <menn0> wallyworld_: better check with wayne i guess
[03:28] <wallyworld_> yeah
[03:28] <menn0> i have to run some errands
[03:28] <wallyworld_> i'm not an expert on this stuff
[03:29] <menn0> neither am i
[03:29]  * wallyworld_ raiss a criitical bug
[03:29] <menn0> sigh
[03:29] <menn0> back in a bit
[03:36] <davecheney> wallyworld_: tag ? nope
[03:37] <davecheney> 'fraid I wasn't watching
[03:37] <davecheney> smells like an actions thing
[03:37] <wallyworld_> davecheney: np. unit agent logs now have tags like "unit-mysql-0[15930]"
[03:37] <wallyworld_> could be actions related i guess, not sure
[03:38] <wallyworld_> davecheney: a quick look at the names package doesn't seem to reveal any related changes
[03:39] <davecheney> yup
[03:40] <davecheney> doesn't look like that is whyere they are coming from
[03:40] <davecheney> it is rsyslog ?
[03:40] <davecheney> putting the PID on the end ?
[03:41] <wallyworld_> yeah, i just came to the same conclusion ook at the code, so it must be a logging artifact
[03:42] <davecheney> [$PID] is the smell
[03:44] <wallyworld_> the whle rsyslog thing has been radically changed recently
[05:50] <wallyworld_> davecheney: what's the official way to construct a name from a tag? eg if I have "unit-mysql-1", I want to get "mysql/1"
[06:58] <wallyworld_> jam: connection dropped, sorry if this comes twice, how's your TLS knowledge? i'm not sure what to do with bug 1388688 - 1.21 is broken, but apparently the change was made so that 1.20.9 could be connected to
[06:58] <mup> Bug #1388688: juju debug-log is broken <debug-log> <regression> <rsyslog> <juju-core:Triaged> <https://launchpad.net/bugs/1388688>
[07:01] <jam> wallyworld_: it didn't, looking
[07:02] <jam> wallyworld_: is this a fresh bootstrap, an upgrade, an ... ?
[07:03] <wallyworld_> jam: fresh bootstrap
[07:03] <wallyworld_> 1.21 tip
[07:03] <davecheney> wallyworld_: names.NewUnitTag(string).String()
[07:03] <davecheney> wallyworld_: it depends
[07:03] <davecheney> if you have "unit-mysql-1"
[07:03] <davecheney> where did you get it from ?
[07:03] <jam> wallyworld_: I worked with wwitzel a bit on trying to figure out what the right fix was for this.
[07:04] <wallyworld_> davecheney: it's the prefix of a log file entry
[07:04] <jam> specifically we have a variety of issues around TLS and cert names
[07:04] <jam> they want you to either put a hostname in the field
[07:04] <jam> or put IP addresses in another field
[07:04] <jam> but if we do IP addresses we have to create a new certificate anytime any IP addresses change
[07:04] <wallyworld_> for juju < 1.20.9 only?
[07:05] <jam> we *were* putting the phrase "anything" as the server hostname for API connections, but nothing for stuff like rsyslog
[07:05] <jam> and then generating certs with "Hostname: *"
[07:05] <jam> which matches "anything"
[07:05] <jam> at one point, someone switched us to start using IP addresses
[07:05] <jam> but then upgrade breaks
[07:05] <wallyworld_> oh :-(
[07:06] <jam> because it uses one cert that doesn't have IP addresses
[07:06] <jam> and when we discussed it
[07:06] <jam> it felt a lot better to use
[07:06] <jam> Hostname: *
[07:06] <jam> and meaningful "ServerName" fields
[07:06] <jam> and then maybe we could eventually migrate to Hostname: juju-meaningfulname instead of *
[07:06] <jam> now, it looks like the cert that existed actually put a DNSName into the Hostname field
[07:07] <jam> instead of Hostname: *
[07:07] <wallyworld_> and we generate the cert?
[07:07] <jam> but (a) not all clouds give you DNS names, and (b) the DNS name is specific to a machine, and not to all of the API servers in HA mode
[07:07] <jam> wallyworld_: yeah, we generate the cert and how we connect to it
[07:07] <jam> but it looks like we're having trouble because we changed it, and now we need to support all the possible ways it could have been generated.
[07:08] <jam> I thought the IP address stuff was only in the 1.21-alpha stuff
[07:08] <jam> I'd have to dig deep to know what was done when and whereb
[07:08] <jam> because we seem to have changed these things in *micro* releases, which isn't a very good practice
[07:08] <jam> wallyworld_: is it possible to wait for wwitzel, or do you need it asap ?
[07:09] <wallyworld_> jam: i've marked it as a critical regression and in my view, we cannot release alpha3 till it's fixed. we can talk to wayne when he comes online
[07:09] <wallyworld_> cause debug log is broken
[07:10] <jam> wallyworld_: to be clear, the issue you are seeing is that 1.21-alpha3 agents are unable to talk to a 1.21-alpha3 API server's rsyslog daemon, correct ?
[07:10] <wallyworld_> jam: yes, a fresh bootstrap with upload tools
[07:10] <davecheney> wallyworld_: you can use names.ParseTag(...)
[07:10] <wallyworld_> on local provider and AWS
[07:11] <wallyworld_> davecheney: thanks, will look into that
[07:23] <dimitern> morning
[08:31] <TheMue> morning
[09:06] <voidspace> morning all
[09:24] <mattyw> morning everyone
[09:29] <fwereade> tasdomas, ping
[09:30]  * fwereade waves at everybody
[09:52] <voidspace> fwereade: hah, when I started reviewing wayne's branch there were no reviews for it
[09:52] <voidspace> fwereade: when I publish my review some have appeared...
[09:52] <fwereade> voidspace, extra reviews are not in any way a problem
[09:53] <voidspace> heh :-)
[10:02] <jam> hi fwereade
[10:02] <fwereade> jam, heyhey
[10:55] <voidspace> jam: I'm there
[11:55] <mattyw> fwereade, just noticed you pinging tasdomas  - I think he's off today
[11:58] <fwereade> mattyw, cheers
[12:06] <mattyw> fwereade, do we have a way to enabling client logging to a file?
[12:07] <fwereade> mattyw, I *think* we do but I don't recall offhand
[12:23] <mattyw> fwereade, --log-file thing.log
[12:24] <fwereade> mattyw, heh, thanks
[12:40] <fwereade> bah, so, the bot's complaining about 1388073 and 1388493 -- both of which STM to be fix committed
[12:40] <fwereade> anyone?
[12:46] <jam> fwereade: bot only lets go once it is Fix Released for a while now
[12:47] <jam> fwereade: which IIRC, is Curtis seeing the actual CI test runs pass
[12:47] <fwereade> jam, ah, ok, makes sense
[12:50] <wallyworld_> fwereade: one remaining critical bug open on 1.21 is bug 1388688
[12:50] <mup> Bug #1388688: juju debug-log is broken <debug-log> <regression> <rsyslog> <juju-core:Triaged> <https://launchpad.net/bugs/1388688>
[12:50] <wallyworld_> sorta need to ask wayne about it
[12:50]  * fwereade sees `tls.Config` in the first few lines and grumbles expressively to himself
[12:51] <wallyworld_> if not for that, we could release 1.21 alpha i think/hope according to the dashboard
[12:51] <fwereade> wallyworld_, ok, does he have a mail waiting for when he wakes up?
[12:52] <wallyworld_> no, i was waiting for him to show up
[12:52] <wallyworld_> i talked to jam a little also
[12:52] <fwereade> wallyworld_, that's very good of you, but isn't it late?
[12:52] <wallyworld_> well, kinda
[12:52] <wallyworld_> i also want to talk to curtis
[12:53] <wallyworld_> to make sure everything else is ok for the release
[13:39]  * fwereade points wwitzel3 and wallyworld_ in each other's direction
[13:40] <wallyworld_> wwitzel3: bug 1388688 is a critical regression because debug log is broken, i'm not sure how to fix to make both 1.21 and 1.20 happy
[13:40] <mup> Bug #1388688: juju debug-log is broken <debug-log> <regression> <rsyslog> <juju-core:Triaged> <https://launchpad.net/bugs/1388688>
[13:40] <wwitzel3> wallyworld_: ok, taking a look
[13:40] <wallyworld_> ty
[13:42] <wallyworld_> wwitzel3: here's some discussion from earlier http://pastebin.ubuntu.com/8803081/
[13:45] <wwitzel3> wallyworld_: thank you
[13:45] <wallyworld_> np, thanks for looking
[14:10] <wwitzel3> I should of never upgraded, ugh
[14:11] <wwitzel3> my xmonand broke, my graphics is no longer using hardware rendering, my system python installation broke, I get weird kernel errors randomly, oh and the whole machine freezes up every 2-4 hours
[14:42] <sinzui> jam, alexisb can you arrange for someone to address bug 1388853. We still haven't had a single blessed revision for alpha3
[14:42] <mup> Bug #1388853: ppc64el unittests consistently fails <ci> <ppc64el> <regression> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1388853>
[14:56] <wwitzel3> wallyworld_: ok, well I have a "fix" for it based on what jam was saying.
[14:56] <wwitzel3> wallyworld_: keeps both the clients happy
[15:08] <wwitzel3> natefinch, perrito666: ping
[15:08] <natefinch> wwitzel3: coming
[15:08] <natefinch> wwitzel3: perrito666 is at ods
[15:08] <wwitzel3> thumper and me are on-call reviewers today, so will need someone to take a look at http://reviews.vapour.ws/r/324/, it fixes bug #1388688
[15:08] <mup> Bug #1388688: juju debug-log is broken <debug-log> <regression> <rsyslog> <juju-core:Triaged by wwitzel3> <https://launchpad.net/bugs/1388688>
[15:19] <wwitzel3> ping fwereade, wallyworld_, voidspace ^
[15:22] <fwereade> wwitzel3, tests?
[15:23] <fwereade> wwitzel3, (and are you sure it'll work across all the versions in play?)
[15:25] <wwitzel3> fwereade: yes, I confirmed it works for both versions
[15:25] <fwereade> wwitzel3, awesome
[15:25] <fwereade> wwitzel3, needs a unit test too, would lgtm with that
[15:25] <wwitzel3> fwereade: as for tests, no idea how we would test it
[15:26] <voidspace> wwitzel3: are you appending * inside the body of the loop?
[15:26] <fwereade> wwitzel3, and that's a good point too
[15:26] <wwitzel3> voidspace: oops, I was, need to push the updated diff, I noticed it after I initially pushed :)
[15:27] <voidspace> cool :-)
[15:28] <fwereade> wwitzel3, it looks like we're writing that stuff out somewhere
[15:28] <fwereade> wwitzel3, can the tests not read the certs back in and check their properties?
[15:28] <wwitzel3> fwereade: yep, we can do that
[15:29] <wwitzel3> fwereade: you're right, that is better than nothing
[15:29] <fwereade> wwitzel3, cool, thanks
[15:32] <alexisb> sinzui, looking
[15:37] <alexisb> sinzui, I am not seeing the attachment on that bug
[15:39] <sinzui> alexisb, :( I will try again
[15:43] <sinzui> alexisb, I added the log this time
[15:43] <alexisb> awesome, thanks sinzui
[15:48] <wwitzel3> fwereade, voidspace: added tests that check the issuer, subject, and dnsnames for the cert.
[15:50] <voidspace> wwitzel3: did you push the tests?
[15:50] <voidspace> wwitzel3: don't think they're showing on RB yet
[15:50] <voidspace> checking github
[15:50] <alexisb> all, please consider  bug 1388853 top priority
[15:50] <mup> Bug #1388853: ppc64el unittests consistently fails <ci> <ppc64el> <regression> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1388853>
[15:50] <wwitzel3> voidspace: weird, yeah, they aren't showing in RB
[15:50] <alexisb> as 1.21 is still blocked and release was targeted for last weds
[15:51] <voidspace> wwitzel3: yeah, I'm looking at it on github
[15:51] <voidspace> wwitzel3: github sometimes throttles (drops) API calls
[15:51] <mgz> alexisb: I have a plan of sorts of that bug
[15:51] <alexisb> mgz, please share
[15:52] <mgz> I'm going to change the compiler version that runs the tests
[15:52] <mgz> we fixed the juju parts of the bug on friday,
[15:53] <mgz> it's just the current version of gccgo on trusty blows up with these tests - that's a change since brussels
[15:55] <alexisb> mgz ack, how can the core team help at this stage?
[15:55] <voidspace> wwitzel3: looks good to me
[15:56] <voidspace> wwitzel3: the code adds a * after any HostPorts
[15:56] <voidspace> wwitzel3: worth having a hostport in the test, so that code is exercised?
[15:57] <mgz> alexisb: will yell if I need it
[15:57] <alexisb> mgz, ack, thanks
[16:01] <voidspace> dimitern: ping
[16:02] <dimitern> voidspace, pong
[16:02] <voidspace> dimitern: I have ec2-api-tools installed
[16:02] <voidspace> dimitern: and the right AWC environment variables set
[16:03] <dimitern> voidspace, yeah?
[16:03] <voidspace> dimitern: and I can access the account using those credentials from code (goamz)
[16:03] <voidspace> dimitern: the command line client always reports
[16:03] <voidspace> Client.Unauthorized: Not Authorized
[16:03] <voidspace> dimitern: do you have any idea why that might be?
[16:03] <voidspace> the intarwebz are not helping on this
[16:03] <voidspace> the only references I can find are to Java exceptions
[16:04] <bodie_> who should I discuss changes to charm with? rogpeppe?
[16:04] <rogpeppe> bodie_: if you like :)
[16:04] <dimitern> voidspace, not sure if this can be the cause, but I had to set a couple of additional env vars for aws:
[16:04] <voidspace> dimitern: ah
[16:04] <voidspace> maybe
[16:05] <voidspace> dimitern: you remember which ones?
[16:06] <bodie_> rogpeppe, there is a change in charm v4 that breaks juju core state/charm_test TestTestingCharm, which tests against the "metered-custom" dummy charm that was removed from charm in 190b3a5d
[16:06] <bodie_> rogpeppe, I want to make some additions to charm but I wasn't sure what to base them on since it seems like v4 is now not compatible (the commit in godeps is fine)
[16:07] <dimitern> voidspace, try these - http://paste.ubuntu.com/8804761/
[16:08] <bodie_> rogpeppe, should I just open a PR to replace the missing piece?  I think we might want to move to v5 anyway since I have some Actions related changes I want to make charm more correct.  I don't want deprecated bad bits to build up, but they are not serious... just irritating
[16:08] <rogpeppe> bodie_: are you saying that there have been some backwardly incompatible changes made to v4, or that you want to make some backwardly incompatible changes in the charm package?
[16:08] <voidspace> dimitern: that worked!
[16:08] <voidspace> dimitern: it might be worth updating the container addressability doc, which only mentions the key pair
[16:08] <voidspace> dimitern: I can do that
[16:09] <voidspace> dimitern: I think it was the EC2_KEYPAIR one which did it
[16:09] <voidspace> but I'll verify
[16:09] <bodie_> rogpeppe, both.  but the backwards incompatibility that was added to v4, can easily be reversed.  and my changes don't have to be backwardly incompatible, but I would ideally like to remove the pieces they deprecate.
[16:09] <voidspace> dimitern: nope, that on its own wasn't enough
[16:09] <bodie_> rogpeppe, does that clarify?
[16:09] <voidspace> dimitern: it was EC2_URL
[16:10] <rogpeppe> bodie_: in a call currently, will get back to you in 15 mins or so
[16:10] <bodie_> sure thing :)
[16:12] <dimitern> voidspace, I think it was one of the key vars
[16:12] <dimitern> voidspace, one name is deprecated, but accepted the other is the "new" one you're supposed to use
[16:12] <dimitern> voidspace, I have some connection issues - did you see what I sent last?
[16:12] <dimitern> voidspace, ah, ok
[16:12] <dimitern> voidspace, please update the doc then :)
[16:12] <voidspace> dimitern: no, I have the key vars in my env already - just adding EC2_URL is sufficient to make it work
[16:12] <voidspace> dimitern: done
[16:12] <dimitern> voidspace, cheers!
[16:13] <wwitzel3> voidspace: yeah, probably, it is funny .. we have NewRsyslogConfigHandler = newRsyslogConfigHandler in export_test.go, but then we never actually override or test it :P
[16:13] <voidspace> wwitzel3: hah
[16:13] <wwitzel3> voidspace: I was hoping that wasn't the case, guess I'll have to write a suite for it
[16:14] <bodie_> rogpeppe, fwiw, I think it would be good to roll forward to v5 since there are some old half-baked Actions definitions in a dummy charm in v4, that I want to remove, not deprecate.  but as long as it is clear that they are old and smelly, it doesn't change the actual behavior of charm.  it's testing-related only.
[16:14] <bodie_> same goes for the breaking commit in v4.
[16:14] <rogpeppe> bodie_: i'm not sure we need to hold to quite such a stringent backwards compatibility policy for testing-related code
[16:15] <bodie_> rogpeppe, I think TheMue had a good point: tests shouldn't be written against implementation details (dummy charms) of v4 anyway :)
[16:16] <bodie_> but, maybe that's a little bit of a funky point (dummy charms) which I'm not sure how to approach
[16:17] <bodie_> anyway, I will make my changes compatible, but there is some cruft building up and I'd like the charmers to have a proper example of what Actions should look like in a charm
[16:17] <rogpeppe> bodie_: out of the call now
[16:18] <rogpeppe> bodie_: i'm generally in agreement with TheMue's point, and i've been pondering a good direction for a while now.
[16:19] <bodie_> yeah, it is a pretty subtle and maybe small pothole
[16:19] <rogpeppe> bodie_: i recently made a backwardly incompatible change to one of the testing repo charms too
[16:19] <rogpeppe> bodie_: i don't think we need to count the testing-related packages as part of the v4 contract.
[16:20] <rogpeppe> bodie_: although it's good to keep backwardly compatible when we can, obviously
[16:21] <bodie_> my only concern is that if I want to make changes, I can't work on top of charm master if it's breaking juju master
[16:41] <voidspace> dimitern: the task breakdown and estimates in the Juju Container Addressability document are based
[16:41] <voidspace> dimitern: on AllocateAddress picking an address for us, and not on us managing addresses (and picking one from the right subnet)
[16:42] <voidspace> dimitern: I'm not sure that changes a lot regarding estimates - except it means more work in the container provisioner
[16:42] <voidspace> dimitern: unless I'm reading it wrong
[16:42] <dimitern> voidspace, that's correct, we might need to inflate the estimate there
[16:42] <voidspace> dimitern: right
[16:42] <voidspace> dimitern: there is a "Track the IP addresses we have allocated for each subnet" which could cover it
[16:43] <dimitern> voidspace, does the estimate there seem sane to you?
[16:44] <voidspace> dimitern: 2-3wks probably has enough wiggle room to cover it :-)
[16:44] <voidspace> dimitern: the other estimates seem ok to me
[16:44] <voidspace> in a hand-wavy kind of way...
[16:44] <dimitern> voidspace, :) cheers!
[16:48] <rogpeppe> bodie_: sure you can - juju master is only broken when the juju-core deps are updated, no?
[16:48] <rogpeppe> bodie_: the most important thing about compatibility is keeping go get working
[16:48] <bodie_> gotya :)
[16:50] <rogpeppe> bodie_: with the testing deps, the broken deps are usually only one level, between tests and testing package
[16:50] <rogpeppe> bodie_: so, just to summarise, your backwardly incompatible changes only involved changes to the testing charms repo?
[16:52] <bodie_> that's right, but they don't have to be backwardly incompatible.  I would just like them to be :)
[16:54] <rogpeppe> bodie_: i think you should go for it
[16:55] <rogpeppe> bodie_: for bonus points, check that the charmstore repo passes its tests with the new package
[16:56] <rogpeppe> bodie_: and we should perhaps think about moving toward entirely removing testing/repo
[16:57] <rogpeppe> bodie_: there will be places where it's still useful, but i think a better way would be to have charm testing repos inside the packages/repos that *use* the charm package
[16:58] <bodie_> rogpeppe, perhaps a good approach would be for charm to expose some methods for generating mock charms for core tests that need to integrate with charm
[16:58] <bodie_> (looking forward)
[16:59] <rogpeppe> bodie_: that's another decent approach that might be good
[16:59] <rogpeppe> bodie_: the only down side is that if every test is doing that, it can lead to a lot of disk churn
[17:00] <rogpeppe> bodie_: a lot of tests don't really care about the details of the charms - they just need *some* example charm to reploy
[17:00] <rogpeppe> deploy
[17:00] <bodie_> heh, was still mid comprehending your thoughts
[17:01]  * bodie_ scribbles "listen better" note with all the others
[17:02] <bodie_> I'm not really seeing how that solves the problem since it's like turning the deps inside out, where clients must define the interactions they expect with charm.  but I don't really have an opinion either way yet -- I was thinking of generating in-memory fake charms though, but I didn't think about how much file interaction there is with charm
[17:02] <rogpeppe> bodie_: another thing it might be interesting to do is make it easier to have an all in-memory charm
[17:03] <rogpeppe> bodie_: ha
[17:03] <bodie_> +1!  brb, meeting
[17:03] <rogpeppe> bodie_: i think it depends how one thinks of the testing charms
[17:04] <rogpeppe> bodie_: i think of them like they're testing data - a test wants to see what will happen when we do something with a given charm. it could specify that charm in the testing code, or as external data.
[17:05] <rogpeppe> bodie_: the external-data approach is arguably more readable, as the charm's in the form that it is usually specified
[17:06] <bodie_> rogpeppe, maybe it's as simple as having a function that cranks out a memory representation of the charm files, and then loads them as if they were a "real" charm file
[17:06] <rogpeppe> bodie_: but the in-testing-code is more flexible
[17:06] <bodie_> it's dumb, but it could be an easy out
[17:07] <rogpeppe> bodie_: it's easy enough to do
[17:07] <rogpeppe> bodie_: apart from the 100s of tests that would need changing in juju-core
[17:08] <rogpeppe> bodie_: and it's the thought of that that leads me to suggest a half-way house, where juju-core has its own charm testing repo
[17:08] <rogpeppe> bodie_: so then we don't need to change all those tests
[17:08] <rogpeppe> bodie_: (assuming we get juju-core to prime the charm package on the location of the juju-core charm repo)
[18:08] <wwitzel3> voidspace: updated the test to exercise the rsyslogHosts conditional logic
[18:10] <voidspace> wwitzel3: looking
[18:11] <voidspace> wwitzel3: LGTM
[18:13] <wwitzel3> voidspace: thanks
[18:13] <mgz> update on ppc blocker, bug 1388853
[18:13] <mup> Bug #1388853: ppc64el unittests consistently fails <ci> <ppc64el> <regression> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1388853>
[18:14] <mgz> the crash stuff is dealt with by having a newer compiler than is available in trusty, our test machine has this
[18:15] <mgz> if needed, ppa:ubuntu-toolchain-r/ppa can get you one as well
[18:15] <mgz> the remaining failures documented in that bug seem like ppc-specific test issues which just need fixing, naming of arch/tools issues and such like
[18:16] <alexisb> natefinch, can someone in the appropriate timezone take a look? ^^^
[18:16] <mgz> I can tell anyone who wants to take it up how to get on our actual ppc machine if needed
[18:16] <alexisb> mgz, can you make sure there is an updateint he bug
[18:17] <mgz> alexisb: yeah, I'm also going to close the previous bug as that just got very confused
[18:19] <natefinch> alexisb: yep
[18:19] <natefinch> mgz: writing up how to get on our PPC machine in a place we can easily find it would be very useful
[18:20] <mgz> natefinch: we don't have a general machine for playing on, just the one we use as the jenkins slave
[18:33] <mgz> alexisb, natefinch: added details to the bug
[18:41] <natefinch> mgz: thanks
[18:59] <voidspace> g;night all
[19:43] <katco> ericsnow: i may be looking at this wrong, but it looks like my review request didn't have a diff generated? http://reviews.vapour.ws/r/327/
[19:43] <ericsnow> katco: yeah, look like it :(
[19:43] <ericsnow> katco: that still happens infrequently
[19:44] <katco> ericsnow: did i do something wrong/different, or is that a known issue?
[19:44] <ericsnow> katco: it happens infrequently enough that I haven't looked into it
[19:44] <ericsnow> katco: not your fault
[19:44] <katco> ericsnow: k, ty sir :)
[19:45] <ericsnow> katco: you can still use rbt to post the patch
[19:45] <katco> ericsnow: righty-o
[20:17] <menn0> waigani: morning
[20:19] <waigani> menn0: morning
[20:19] <katco> am i correct in assuming landings are blocked?
[20:19] <menn0> yes they are
[20:19] <menn0> due to bug 1388853
[20:19] <mup> Bug #1388853: ppc64el unittests consistently fails <ci> <ppc64el> <regression> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1388853>
[20:19] <katco> ok, thanks menn0
[20:19] <katco> yeah, figured, but wanted to ask just to be sure
[20:19] <menn0> martin has done a good analysis. the problems looks pretty simple to fix so i'm going to have a go.
[20:20] <katco> good luck, menn0. may the source be with you.
[20:21] <mwhudson> good morning
[20:21] <menn0> mwhudson: howdy
[20:22] <mwhudson> someone put "dummy-admin dummy-admin just now just now" in the expected output of a test?
[20:22] <menn0> mwhudson: someone has
[20:23] <mwhudson> oh dear
[20:23] <mwhudson> (well, unless the test uses a synthetic time source, which it apparently doesn't...)
[20:23] <menn0> mwhudson: and it could even be thumper ...
[20:23] <menn0> mwhudson: although I think I might have reviewed that one
[20:24] <mwhudson> in a way that's good because i have no problems with telling thumper that he's done something stupid :)
[20:24] <menn0> LOL
[20:25] <mwhudson> whereas i might be a bit more circumspect with someone i didn't know so well :)
[20:25] <menn0> i'll fix that and have a look at the other problem too
[20:25] <menn0> although I can't get that to happen on my machine
[20:26] <menn0> I suspect you actually need to be on a ppc64 host
[20:26] <mgz> menn0: I can tell you how to access the jenkins build slave if that would help
[20:28] <menn0> mgz: i have a few ideas to try first but could you send me the details anyway
[20:30] <mwhudson> menn0, waigani, davecheney: email standup i guess?
[20:31] <menn0> mwhudson: unless you would prefer otherwise?
[20:31] <waigani> I'm easy either way
[20:31] <mwhudson> i am currently in a cafe on extremely slow 3g so email sounds good :)
[20:31] <waigani> email it i
[20:31] <waigani> s
[20:33] <menn0> cool
[20:40] <mgz> menn0: https://pastebin.canonical.com/119817
[20:44] <menn0> mgz: cheers
[21:24] <waigani> menn0: I may have hit a chicken and egg
[21:24] <menn0> waigani: in what way
[21:24] <menn0> ?
[21:25] <waigani> menn0: there is a settings doc which uses a global env key as it's key
[21:25] <waigani> menn0: it holds the configuration for the environment
[21:26] <waigani> menn0: the key is now perpended with the env-uuid
[21:27] <waigani> menn0: and the doc exists, with a docID _id, before the upgrade step is run
[21:29] <waigani> menn0: so during the upgrade step, it gets perpended with the env-uuid again
[21:30] <waigani> menn0: and the test also fails as len(oldDocs) != len(newDocs) - that one I can fix easy enough
[21:30] <katco> wallyworld_: noooot sure what happened, rejoining
[21:30] <menn0> waigani: give me a sec
[21:32] <menn0> waigani: ok
[21:32] <waigani> menn0: yep, went you get a sec - does it jog anything from the workarounds you had to do?
[21:33] <menn0> waigani: sounds like you might need to add another little hack
[21:33] <menn0> waigani: but i'm not clear on why the env-uuid gets added twice
[21:33] <menn0> waigani: can you explain that?
[21:34] <waigani> menn0: good question, give me a sec I'll see...
[21:36] <waigani> menn0: because I used the global env key in one of the dummy docs to migrate.
[21:37] <waigani> menn0: so if I remove that and make the test expect len(oldDocs) + 1, the test should pass
[21:39] <waigani>  or remove the envion doc from the newIDs returned...
[21:40] <waigani> ah no, okay I'll stop thinking out loud and sort this
[21:40] <menn0> waigani: ok. let me know if you need a sounding board again :)
[21:43] <waigani> menn0: thanks :)
[21:48] <menn0> This is one half of fixing the CI blocker. can someone PTAL? http://reviews.vapour.ws/r/329/
[21:49] <davecheney> menn0: can you just delete the check fo rthe duration
[21:49] <davecheney> it's unknowable
[21:50] <natefinch> +1
[21:50] <menn0> davecheney, natefinch: but I've made the test accept any possible duration. is that not ok?
[21:51] <menn0> at least then the test ensure that we're getting the right shaped output
[21:51] <davecheney> menn0: fells like solving the wrong problem
[21:51] <davecheney> now that test will fail when tim changes the printing of an _arbitrary_ duration
[21:54] <menn0> but if I do that there's nothing testing that the duration emitted by "user list" is sanely formatted at all!
[21:55] <menn0> davecheney: ^
[21:55] <menn0> davecheney: or put another way, there's nothing test that the duration formatting function is in use
[21:56] <menn0> testing even
[21:57] <natefinch> this is why I like internal tests ;)
[21:59] <menn0> davecheney, natefinch: as this is becoming yet another testing debate and I just want to get this CI blocker sorted so that I can get back to useful work I will just do as you ask
[21:59] <natefinch> menn0: for the record, I'm ok with either fix
[22:02] <davecheney> menn0: natefinch same
[22:03] <menn0> natefinch, davecheney: so what, leave the branch as it is?
[22:03] <menn0> I just started relaxing the regex further
[22:03] <natefinch> menn0: I put a ship it on it
[22:03] <menn0> ok
[22:06] <davecheney> menn0: do the absolute minimum required to get the test passing
[22:06] <davecheney> as you say
[22:06] <davecheney> this is a pointless testing debate
[22:06] <davecheney> one of N
[22:07] <menn0> thanks. i've sent that off for merging.
[22:07] <menn0> now for the other half of the ticket.
[22:23] <wallyworld_> menn0: i can pick up the ppc64 test failure
[22:23] <katco> wallyworld_: fwiw i had to clear out my cookies... looks like time-changes mess with something w/ google apps
[22:23] <wallyworld_> \o/
[22:23] <katco> wallyworld_: the time changes from brussels probably triggered it on my laptop, daylight savings time for my main comp
[22:23]  * katco shrugs
[22:24]  * wallyworld_ shrugs too
[22:24] <katco> software!
[22:24] <menn0> wallyworld_: ok. it looks related to your changes from yesterday.
[22:25] <wallyworld_> menn0: yep - i had to add extra metadata to account for 1.20 being wrong but clearly when run on non amd64, the expected list of metadata entries is different
[22:27] <menn0> wallyworld_: the fix for the other part of the ticket (TestUserList) just landed
[22:27] <wallyworld_> menn0: awesome, ty
[23:34] <lazyPower> Ran into an interesting AWS issue with a customer that I've not seen before - if anyone has a moment to take a look - https://launchpad.net/bugs/1389014
[23:34] <mup> Bug #1389014: juju AWS EC2 "unit-get public-address" gives private address  <juju-core:New> <https://launchpad.net/bugs/1389014>
[23:41] <davecheney> lazyPower: is the custoemr using vpc
[23:41] <davecheney> ?
[23:41] <davecheney> themonk@redqueen:~$ juju run -e amazon --service my-charm1 "unit-get public-address"
[23:41] <davecheney> 172.31.4.32
[23:41] <davecheney> ^ how did this work
[23:41] <lazyPower> davecheney: no idea its themonk in #juju though
[23:41] <lazyPower> exactly!
[23:42] <lazyPower> a) how do you get an IP, b) how do you get the *same* ip
[23:42] <davecheney> how was their server able to contact 172.31.4.32 ?
[23:42] <davecheney> that is a private ip
[23:42] <lazyPower> and c) how do you get a private ip
[23:42] <davecheney> lazyPower: juju just returns whatever the aws metadata services tells it
[23:42] <davecheney> so if aws says you'r ip address is 1.1.1.1
[23:42] <davecheney> that's why juju knows as well
[23:44] <lazyPower> davecheney: <themonk> lazyPower, what is 'provisioned in a VPC' ?
[23:44] <lazyPower> i'm going to say the answer is probably no...
[23:46] <lazyPower> davecheney: do you have a minute to pop in #juju and interface with themonk over his AWS issue? If not no biggie - he's got an open bug that he can track.
[23:46] <davecheney> curl http://169.254.169.254/latest/public-ipv4
[23:46] <davecheney> this is what juju does
[23:46] <davecheney> join #juju
[23:46] <davecheney> urgh
[23:46] <davecheney> sorry, i thought I was in that channel
[23:46] <davecheney> i have so many called #juju
[23:47] <lazyPower> all good davecheney :) I have the same issue between canonirc and freenode. so i know the feeling.