[00:20] menn0: only took 3 attempts, but fix has landed, will take a bit to wash through CI, but critical regression fix committed [00:21] wallyworld: glad it's in [00:21] must. not. comment. [00:22] sadly we still have work to do to make out test suite reliable [00:27] rick_h_, simply that a change to the charm repo in master is causing juju core tests written against the bit that was removed, to fail [00:28] it's not really critical to what I'm trying to do right now, but I want to land some changes to charm, and if v4 master isn't working perhaps I shouldn't rebase my changes on master [00:29] I'll try to get ahold of rogpeppe / uros (who's that?) [00:42] hmmm, although LP shows no critical regressions the buildbot is still blocking merges due to bug 1388493'] [00:42] Bug #1388493: generate-tools --streams loses content of other streams

[00:42] and bug 1388073 [00:42] Bug #1388073: set-env no longer accepts empty values

[00:43] bodie_: urulama sorry [00:44] bodie_: roger's lead, the two have been working with charm for some charmstore stuff/etc and are up on the charm package info these days. [00:52] ah, okay. thanks rick_h_ :) [02:58] wallyworld_: do we officially support concurrent local provider environments on the one host machine? [02:59] um, no [02:59] not to my knowledge [02:59] i'm not sure it would even work [02:59] menn0: it works [02:59] but you'd be bonkers to try [02:59] we did it in japan [02:59] 27 local providers [02:59] \o/ [02:59] * footnote: that was in versino 1.14 [03:00] gawdknows what's changed since then [03:00] wallyworld_, davecheney: the reason I ask is that i've just figured out why the rsyslog worker works for all but 1 environ [03:00] im going to try it [03:01] wallyworld_, davecheney: it writes out configuration files using different certs but all listening on the same rsyslog port [03:01] wallyworld_, davecheney: so only one environment has working logging [03:01] menn0: rump-row [03:01] menn0: ah, i'm currently wondering why juju debug-log is broken [03:01] and even if you go back to one env [03:02] nodes are failing to send stuff to machine 0 due to certificate issues [03:02] the old rsyslog config doesn't appear to get removed [03:02] so logging is broken from then on [03:02] you need to clean up /etc/rsyslog.d manually [03:02] it's taken me quite some time to figure that out :) [03:02] menn0: i'm seeing this [03:02] 2014-11-03 02:56:34 ERROR juju.worker runner.go:219 exited "rsyslog": x509: certificate is valid for ip-10-46-174-79.ec2.internal, not juju-rsyslog [03:02] wallyworld_: that's the one [03:03] menn0: this is a critical regression for alpha3 [03:03] as debug log is just broken [03:03] wallyworld_: it's only for the local provider though [03:03] menn0: nope, also aws [03:03] and probably evrything else too :-( [03:03] wallyworld_: sorry [03:03] that line i just pasted was from an aws deploy [03:04] wallyworld_: I just re-read that error. [03:04] wallyworld_: I was seeing something different [03:04] ah, ok [03:04] x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "juju-generated CA for environment \"rsyslog\"") [03:04] it may well be the same root cause, not sure [03:04] wallyworld_: don't think so [03:05] regardless rsyslog and hence debug log is stuffed [03:05] wallyworld_: the thing i've been investigating is definitely local provider specific when you have multiple environments set up [03:05] sure [03:05] :-( [03:08] wallyworld_: the client side of the connection sets the expected server name to a hardcoded value of "juju-rsyslog" [03:09] wallyworld_: see composeTLS() in worker/rsyslog/worker.go [03:09] menn0: yeah, i'm backing out that change to see if it works [03:09] was changed 10th october [03:09] so recently [03:10] * wallyworld_ waits imatiently for aws to start up an instance or two [03:11] % git push --set-upstream origin 100-fix-issue-69 [03:11] fatal: https://gopkg.in/juju/charm.v4/info/refs not valid: is this a git repository? [03:11] I hate gopkg.in [03:11] https://github.com/juju/charm/pull/70 [03:11] anyone ? [03:14] wallyworld_: thanks for the review [03:14] i'm going to wait for rog [03:14] this is his baby [03:14] and I have just slapped it, soundly [03:15] sure, looked ok to me.... but im not 100% familiar with it [03:18] wallyworld_: i'm changing the semantics of the method from what it promised, but didn't deliver [03:19] to what it does [03:19] right :-) [03:19] i tried to do it the other way, ie make the methods return a *copy* of the receiver [03:19] but of course all the tests expect the current behavioru [03:19] so all broke [03:19] so, this change renames the method to _say_ what they _do_ [03:20] as well as making them thread safe [03:20] * wallyworld_ nods [03:24] davecheney: do you know anything about unit tags gaining a suffix? eg unit-flannel-2[2016] [03:25] menn0: that one line change to remove "juju-rsyslog" worked [03:25] but i'm not sure of the implications of just removing it [03:26] wallyworld_: good. [03:27] the commit message says [03:27] tls.Config clients should use a meaningful ServerName This also fixes an issue with the rsyslog tls.Config not using a ServerName at all. [03:27] well, trouble is, if the rsyslog tls.Config uses a service name, it breaks [03:27] wallyworld_: the docstring for composeTLS (added in that rev) says that setting ServerName is required to support pre-1.20.9 certs [03:28] wallyworld_: so I suspect taking that line out is going to bring back the problem of rsyslog cert issues when upgrading from 1.20 to 1.21 [03:28] wallyworld_: better check with wayne i guess [03:28] yeah [03:28] i have to run some errands [03:28] i'm not an expert on this stuff [03:29] neither am i [03:29] * wallyworld_ raiss a criitical bug [03:29] sigh [03:29] back in a bit [03:36] wallyworld_: tag ? nope [03:37] 'fraid I wasn't watching [03:37] smells like an actions thing [03:37] davecheney: np. unit agent logs now have tags like "unit-mysql-0[15930]" [03:37] could be actions related i guess, not sure [03:38] davecheney: a quick look at the names package doesn't seem to reveal any related changes [03:39] yup [03:40] doesn't look like that is whyere they are coming from [03:40] it is rsyslog ? [03:40] putting the PID on the end ? [03:41] yeah, i just came to the same conclusion ook at the code, so it must be a logging artifact [03:42] [$PID] is the smell [03:44] the whle rsyslog thing has been radically changed recently === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 === kadams54 is now known as kadams54-away [05:50] davecheney: what's the official way to construct a name from a tag? eg if I have "unit-mysql-1", I want to get "mysql/1" [06:58] jam: connection dropped, sorry if this comes twice, how's your TLS knowledge? i'm not sure what to do with bug 1388688 - 1.21 is broken, but apparently the change was made so that 1.20.9 could be connected to [06:58] Bug #1388688: juju debug-log is broken

[07:01] wallyworld_: it didn't, looking [07:02] wallyworld_: is this a fresh bootstrap, an upgrade, an ... ? [07:03] jam: fresh bootstrap [07:03] 1.21 tip [07:03] wallyworld_: names.NewUnitTag(string).String() [07:03] wallyworld_: it depends [07:03] if you have "unit-mysql-1" [07:03] where did you get it from ? [07:03] wallyworld_: I worked with wwitzel a bit on trying to figure out what the right fix was for this. [07:04] davecheney: it's the prefix of a log file entry [07:04] specifically we have a variety of issues around TLS and cert names [07:04] they want you to either put a hostname in the field [07:04] or put IP addresses in another field [07:04] but if we do IP addresses we have to create a new certificate anytime any IP addresses change [07:04] for juju < 1.20.9 only? [07:05] we *were* putting the phrase "anything" as the server hostname for API connections, but nothing for stuff like rsyslog [07:05] and then generating certs with "Hostname: *" [07:05] which matches "anything" [07:05] at one point, someone switched us to start using IP addresses [07:05] but then upgrade breaks [07:05] oh :-( [07:06] because it uses one cert that doesn't have IP addresses [07:06] and when we discussed it [07:06] it felt a lot better to use [07:06] Hostname: * [07:06] and meaningful "ServerName" fields [07:06] and then maybe we could eventually migrate to Hostname: juju-meaningfulname instead of * [07:06] now, it looks like the cert that existed actually put a DNSName into the Hostname field [07:07] instead of Hostname: * [07:07] and we generate the cert? [07:07] but (a) not all clouds give you DNS names, and (b) the DNS name is specific to a machine, and not to all of the API servers in HA mode [07:07] wallyworld_: yeah, we generate the cert and how we connect to it [07:07] but it looks like we're having trouble because we changed it, and now we need to support all the possible ways it could have been generated. [07:08] I thought the IP address stuff was only in the 1.21-alpha stuff [07:08] I'd have to dig deep to know what was done when and whereb [07:08] because we seem to have changed these things in *micro* releases, which isn't a very good practice [07:08] wallyworld_: is it possible to wait for wwitzel, or do you need it asap ? [07:09] jam: i've marked it as a critical regression and in my view, we cannot release alpha3 till it's fixed. we can talk to wayne when he comes online [07:09] cause debug log is broken [07:10] wallyworld_: to be clear, the issue you are seeing is that 1.21-alpha3 agents are unable to talk to a 1.21-alpha3 API server's rsyslog daemon, correct ? [07:10] jam: yes, a fresh bootstrap with upload tools [07:10] wallyworld_: you can use names.ParseTag(...) [07:10] on local provider and AWS [07:11] davecheney: thanks, will look into that [07:23] morning [08:31] morning [09:06] morning all [09:24] morning everyone [09:29] tasdomas, ping [09:30] * fwereade waves at everybody [09:52] fwereade: hah, when I started reviewing wayne's branch there were no reviews for it [09:52] fwereade: when I publish my review some have appeared... [09:52] voidspace, extra reviews are not in any way a problem [09:53] heh :-) [10:02] hi fwereade [10:02] jam, heyhey [10:55] jam: I'm there [11:55] fwereade, just noticed you pinging tasdomas - I think he's off today [11:58] mattyw, cheers [12:06] fwereade, do we have a way to enabling client logging to a file? [12:07] mattyw, I *think* we do but I don't recall offhand [12:23] fwereade, --log-file thing.log [12:24] mattyw, heh, thanks [12:40] bah, so, the bot's complaining about 1388073 and 1388493 -- both of which STM to be fix committed [12:40] anyone? [12:46] fwereade: bot only lets go once it is Fix Released for a while now [12:47] fwereade: which IIRC, is Curtis seeing the actual CI test runs pass [12:47] jam, ah, ok, makes sense [12:50] fwereade: one remaining critical bug open on 1.21 is bug 1388688 [12:50] Bug #1388688: juju debug-log is broken

[12:50] sorta need to ask wayne about it [12:50] * fwereade sees `tls.Config` in the first few lines and grumbles expressively to himself [12:51] if not for that, we could release 1.21 alpha i think/hope according to the dashboard [12:51] wallyworld_, ok, does he have a mail waiting for when he wakes up? [12:52] no, i was waiting for him to show up [12:52] i talked to jam a little also [12:52] wallyworld_, that's very good of you, but isn't it late? [12:52] well, kinda [12:52] i also want to talk to curtis [12:53] to make sure everything else is ok for the release === lazyWeekend is now known as lazyPower === tvansteenburgh1 is now known as tvansteenburgh [13:39] * fwereade points wwitzel3 and wallyworld_ in each other's direction [13:40] wwitzel3: bug 1388688 is a critical regression because debug log is broken, i'm not sure how to fix to make both 1.21 and 1.20 happy [13:40] Bug #1388688: juju debug-log is broken

[13:40] wallyworld_: ok, taking a look [13:40] ty [13:42] wwitzel3: here's some discussion from earlier http://pastebin.ubuntu.com/8803081/ [13:45] wallyworld_: thank you [13:45] np, thanks for looking [14:10] I should of never upgraded, ugh [14:11] my xmonand broke, my graphics is no longer using hardware rendering, my system python installation broke, I get weird kernel errors randomly, oh and the whole machine freezes up every 2-4 hours === ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs: 1388853 [14:42] jam, alexisb can you arrange for someone to address bug 1388853. We still haven't had a single blessed revision for alpha3 [14:42] Bug #1388853: ppc64el unittests consistently fails

[14:56] wallyworld_: ok, well I have a "fix" for it based on what jam was saying. [14:56] wallyworld_: keeps both the clients happy [15:08] natefinch, perrito666: ping [15:08] wwitzel3: coming [15:08] wwitzel3: perrito666 is at ods [15:08] thumper and me are on-call reviewers today, so will need someone to take a look at http://reviews.vapour.ws/r/324/, it fixes bug #1388688 [15:08] Bug #1388688: juju debug-log is broken

[15:19] ping fwereade, wallyworld_, voidspace ^ [15:22] wwitzel3, tests? [15:23] wwitzel3, (and are you sure it'll work across all the versions in play?) [15:25] fwereade: yes, I confirmed it works for both versions [15:25] wwitzel3, awesome [15:25] wwitzel3, needs a unit test too, would lgtm with that [15:25] fwereade: as for tests, no idea how we would test it [15:26] wwitzel3: are you appending * inside the body of the loop? [15:26] wwitzel3, and that's a good point too [15:26] voidspace: oops, I was, need to push the updated diff, I noticed it after I initially pushed :) [15:27] cool :-) [15:28] wwitzel3, it looks like we're writing that stuff out somewhere [15:28] wwitzel3, can the tests not read the certs back in and check their properties? [15:28] fwereade: yep, we can do that [15:29] fwereade: you're right, that is better than nothing [15:29] wwitzel3, cool, thanks [15:32] sinzui, looking [15:37] sinzui, I am not seeing the attachment on that bug [15:39] alexisb, :( I will try again [15:43] alexisb, I added the log this time [15:43] awesome, thanks sinzui [15:48] fwereade, voidspace: added tests that check the issuer, subject, and dnsnames for the cert. [15:50] wwitzel3: did you push the tests? [15:50] wwitzel3: don't think they're showing on RB yet [15:50] checking github [15:50] all, please consider bug 1388853 top priority [15:50] Bug #1388853: ppc64el unittests consistently fails

[15:50] voidspace: weird, yeah, they aren't showing in RB [15:50] as 1.21 is still blocked and release was targeted for last weds [15:51] wwitzel3: yeah, I'm looking at it on github [15:51] wwitzel3: github sometimes throttles (drops) API calls [15:51] alexisb: I have a plan of sorts of that bug [15:51] mgz, please share [15:52] I'm going to change the compiler version that runs the tests [15:52] we fixed the juju parts of the bug on friday, [15:53] it's just the current version of gccgo on trusty blows up with these tests - that's a change since brussels [15:55] mgz ack, how can the core team help at this stage? [15:55] wwitzel3: looks good to me [15:56] wwitzel3: the code adds a * after any HostPorts [15:56] wwitzel3: worth having a hostport in the test, so that code is exercised? [15:57] alexisb: will yell if I need it [15:57] mgz, ack, thanks [16:01] dimitern: ping [16:02] voidspace, pong [16:02] dimitern: I have ec2-api-tools installed [16:02] dimitern: and the right AWC environment variables set [16:03] voidspace, yeah? [16:03] dimitern: and I can access the account using those credentials from code (goamz) [16:03] dimitern: the command line client always reports [16:03] Client.Unauthorized: Not Authorized [16:03] dimitern: do you have any idea why that might be? [16:03] the intarwebz are not helping on this [16:03] the only references I can find are to Java exceptions [16:04] who should I discuss changes to charm with? rogpeppe? [16:04] bodie_: if you like :) [16:04] voidspace, not sure if this can be the cause, but I had to set a couple of additional env vars for aws: [16:04] dimitern: ah [16:04] maybe [16:05] dimitern: you remember which ones? [16:06] rogpeppe, there is a change in charm v4 that breaks juju core state/charm_test TestTestingCharm, which tests against the "metered-custom" dummy charm that was removed from charm in 190b3a5d [16:06] rogpeppe, I want to make some additions to charm but I wasn't sure what to base them on since it seems like v4 is now not compatible (the commit in godeps is fine) [16:07] voidspace, try these - http://paste.ubuntu.com/8804761/ [16:08] rogpeppe, should I just open a PR to replace the missing piece? I think we might want to move to v5 anyway since I have some Actions related changes I want to make charm more correct. I don't want deprecated bad bits to build up, but they are not serious... just irritating [16:08] bodie_: are you saying that there have been some backwardly incompatible changes made to v4, or that you want to make some backwardly incompatible changes in the charm package? [16:08] dimitern: that worked! [16:08] dimitern: it might be worth updating the container addressability doc, which only mentions the key pair [16:08] dimitern: I can do that [16:09] dimitern: I think it was the EC2_KEYPAIR one which did it [16:09] but I'll verify [16:09] rogpeppe, both. but the backwards incompatibility that was added to v4, can easily be reversed. and my changes don't have to be backwardly incompatible, but I would ideally like to remove the pieces they deprecate. [16:09] dimitern: nope, that on its own wasn't enough [16:09] rogpeppe, does that clarify? [16:09] dimitern: it was EC2_URL [16:10] bodie_: in a call currently, will get back to you in 15 mins or so [16:10] sure thing :) [16:12] voidspace, I think it was one of the key vars [16:12] voidspace, one name is deprecated, but accepted the other is the "new" one you're supposed to use [16:12] voidspace, I have some connection issues - did you see what I sent last? [16:12] voidspace, ah, ok [16:12] voidspace, please update the doc then :) [16:12] dimitern: no, I have the key vars in my env already - just adding EC2_URL is sufficient to make it work [16:12] dimitern: done [16:12] voidspace, cheers! [16:13] voidspace: yeah, probably, it is funny .. we have NewRsyslogConfigHandler = newRsyslogConfigHandler in export_test.go, but then we never actually override or test it :P [16:13] wwitzel3: hah [16:13] voidspace: I was hoping that wasn't the case, guess I'll have to write a suite for it [16:14] rogpeppe, fwiw, I think it would be good to roll forward to v5 since there are some old half-baked Actions definitions in a dummy charm in v4, that I want to remove, not deprecate. but as long as it is clear that they are old and smelly, it doesn't change the actual behavior of charm. it's testing-related only. [16:14] same goes for the breaking commit in v4. [16:14] bodie_: i'm not sure we need to hold to quite such a stringent backwards compatibility policy for testing-related code [16:15] rogpeppe, I think TheMue had a good point: tests shouldn't be written against implementation details (dummy charms) of v4 anyway :) [16:16] but, maybe that's a little bit of a funky point (dummy charms) which I'm not sure how to approach [16:17] anyway, I will make my changes compatible, but there is some cruft building up and I'd like the charmers to have a proper example of what Actions should look like in a charm [16:17] bodie_: out of the call now [16:18] bodie_: i'm generally in agreement with TheMue's point, and i've been pondering a good direction for a while now. [16:19] yeah, it is a pretty subtle and maybe small pothole [16:19] bodie_: i recently made a backwardly incompatible change to one of the testing repo charms too [16:19] bodie_: i don't think we need to count the testing-related packages as part of the v4 contract. [16:20] bodie_: although it's good to keep backwardly compatible when we can, obviously [16:21] my only concern is that if I want to make changes, I can't work on top of charm master if it's breaking juju master [16:41] dimitern: the task breakdown and estimates in the Juju Container Addressability document are based [16:41] dimitern: on AllocateAddress picking an address for us, and not on us managing addresses (and picking one from the right subnet) [16:42] dimitern: I'm not sure that changes a lot regarding estimates - except it means more work in the container provisioner [16:42] dimitern: unless I'm reading it wrong [16:42] voidspace, that's correct, we might need to inflate the estimate there [16:42] dimitern: right [16:42] dimitern: there is a "Track the IP addresses we have allocated for each subnet" which could cover it [16:43] voidspace, does the estimate there seem sane to you? [16:44] dimitern: 2-3wks probably has enough wiggle room to cover it :-) [16:44] dimitern: the other estimates seem ok to me [16:44] in a hand-wavy kind of way... [16:44] voidspace, :) cheers! [16:48] bodie_: sure you can - juju master is only broken when the juju-core deps are updated, no? [16:48] bodie_: the most important thing about compatibility is keeping go get working [16:48] gotya :) [16:50] bodie_: with the testing deps, the broken deps are usually only one level, between tests and testing package [16:50] bodie_: so, just to summarise, your backwardly incompatible changes only involved changes to the testing charms repo? [16:52] that's right, but they don't have to be backwardly incompatible. I would just like them to be :) [16:54] bodie_: i think you should go for it [16:55] bodie_: for bonus points, check that the charmstore repo passes its tests with the new package === fabrice is now known as fabrice|family [16:56] bodie_: and we should perhaps think about moving toward entirely removing testing/repo [16:57] bodie_: there will be places where it's still useful, but i think a better way would be to have charm testing repos inside the packages/repos that *use* the charm package [16:58] rogpeppe, perhaps a good approach would be for charm to expose some methods for generating mock charms for core tests that need to integrate with charm [16:58] (looking forward) [16:59] bodie_: that's another decent approach that might be good [16:59] bodie_: the only down side is that if every test is doing that, it can lead to a lot of disk churn [17:00] bodie_: a lot of tests don't really care about the details of the charms - they just need *some* example charm to reploy [17:00] deploy [17:00] heh, was still mid comprehending your thoughts [17:01] * bodie_ scribbles "listen better" note with all the others [17:02] I'm not really seeing how that solves the problem since it's like turning the deps inside out, where clients must define the interactions they expect with charm. but I don't really have an opinion either way yet -- I was thinking of generating in-memory fake charms though, but I didn't think about how much file interaction there is with charm [17:02] bodie_: another thing it might be interesting to do is make it easier to have an all in-memory charm [17:03] bodie_: ha [17:03] +1! brb, meeting [17:03] bodie_: i think it depends how one thinks of the testing charms [17:04] bodie_: i think of them like they're testing data - a test wants to see what will happen when we do something with a given charm. it could specify that charm in the testing code, or as external data. [17:05] bodie_: the external-data approach is arguably more readable, as the charm's in the form that it is usually specified [17:06] rogpeppe, maybe it's as simple as having a function that cranks out a memory representation of the charm files, and then loads them as if they were a "real" charm file [17:06] bodie_: but the in-testing-code is more flexible [17:06] it's dumb, but it could be an easy out [17:07] bodie_: it's easy enough to do [17:07] bodie_: apart from the 100s of tests that would need changing in juju-core [17:08] bodie_: and it's the thought of that that leads me to suggest a half-way house, where juju-core has its own charm testing repo [17:08] bodie_: so then we don't need to change all those tests [17:08] bodie_: (assuming we get juju-core to prime the charm package on the location of the juju-core charm repo) === kadams54 is now known as kadams54-away === tvansteenburgh is now known as tvan|lunch [18:08] voidspace: updated the test to exercise the rsyslogHosts conditional logic [18:10] wwitzel3: looking [18:11] wwitzel3: LGTM [18:13] voidspace: thanks [18:13] update on ppc blocker, bug 1388853 [18:13] Bug #1388853: ppc64el unittests consistently fails