| wallyworld | menn0: only took 3 attempts, but fix has landed, will take a bit to wash through CI, but critical regression fix committed | 00:20 |
|---|---|---|
| menn0 | wallyworld: glad it's in | 00:21 |
| wallyworld | must. not. comment. | 00:21 |
| wallyworld | sadly we still have work to do to make out test suite reliable | 00:22 |
| bodie_ | rick_h_, simply that a change to the charm repo in master is causing juju core tests written against the bit that was removed, to fail | 00:27 |
| bodie_ | it's not really critical to what I'm trying to do right now, but I want to land some changes to charm, and if v4 master isn't working perhaps I shouldn't rebase my changes on master | 00:28 |
| bodie_ | I'll try to get ahold of rogpeppe / uros (who's that?) | 00:29 |
| menn0 | hmmm, although LP shows no critical regressions the buildbot is still blocking merges due to bug 1388493'] | 00:42 |
| mup | Bug #1388493: generate-tools --streams loses content of other streams <ci> <metadata> <regression> <juju-core:Fix Committed by wallyworld> <https://launchpad.net/bugs/1388493> | 00:42 |
| menn0 | and bug 1388073 | 00:42 |
| mup | Bug #1388073: set-env no longer accepts empty values <ci> <compatibility> <regression> <set-env> <juju-core:Fix Committed by tasdomas> <https://launchpad.net/bugs/1388073> | 00:42 |
| rick_h_ | bodie_: urulama sorry | 00:43 |
| rick_h_ | bodie_: roger's lead, the two have been working with charm for some charmstore stuff/etc and are up on the charm package info these days. | 00:44 |
| bodie_ | ah, okay. thanks rick_h_ :) | 00:52 |
| menn0 | wallyworld_: do we officially support concurrent local provider environments on the one host machine? | 02:58 |
| wallyworld_ | um, no | 02:59 |
| wallyworld_ | not to my knowledge | 02:59 |
| wallyworld_ | i'm not sure it would even work | 02:59 |
| davecheney | menn0: it works | 02:59 |
| davecheney | but you'd be bonkers to try | 02:59 |
| davecheney | we did it in japan | 02:59 |
| davecheney | 27 local providers | 02:59 |
| wallyworld_ | \o/ | 02:59 |
| davecheney | * footnote: that was in versino 1.14 | 02:59 |
| davecheney | gawdknows what's changed since then | 03:00 |
| menn0 | wallyworld_, davecheney: the reason I ask is that i've just figured out why the rsyslog worker works for all but 1 environ | 03:00 |
| stokachu | im going to try it | 03:00 |
| menn0 | wallyworld_, davecheney: it writes out configuration files using different certs but all listening on the same rsyslog port | 03:01 |
| menn0 | wallyworld_, davecheney: so only one environment has working logging | 03:01 |
| davecheney | menn0: rump-row | 03:01 |
| wallyworld_ | menn0: ah, i'm currently wondering why juju debug-log is broken | 03:01 |
| menn0 | and even if you go back to one env | 03:01 |
| wallyworld_ | nodes are failing to send stuff to machine 0 due to certificate issues | 03:02 |
| menn0 | the old rsyslog config doesn't appear to get removed | 03:02 |
| menn0 | so logging is broken from then on | 03:02 |
| menn0 | you need to clean up /etc/rsyslog.d manually | 03:02 |
| menn0 | it's taken me quite some time to figure that out :) | 03:02 |
| wallyworld_ | menn0: i'm seeing this | 03:02 |
| wallyworld_ | 2014-11-03 02:56:34 ERROR juju.worker runner.go:219 exited "rsyslog": x509: certificate is valid for ip-10-46-174-79.ec2.internal, not juju-rsyslog | 03:02 |
| menn0 | wallyworld_: that's the one | 03:02 |
| wallyworld_ | menn0: this is a critical regression for alpha3 | 03:03 |
| wallyworld_ | as debug log is just broken | 03:03 |
| menn0 | wallyworld_: it's only for the local provider though | 03:03 |
| wallyworld_ | menn0: nope, also aws | 03:03 |
| wallyworld_ | and probably evrything else too :-( | 03:03 |
| menn0 | wallyworld_: sorry | 03:03 |
| wallyworld_ | that line i just pasted was from an aws deploy | 03:03 |
| menn0 | wallyworld_: I just re-read that error. | 03:04 |
| menn0 | wallyworld_: I was seeing something different | 03:04 |
| wallyworld_ | ah, ok | 03:04 |
| menn0 | x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "juju-generated CA for environment \"rsyslog\"") | 03:04 |
| wallyworld_ | it may well be the same root cause, not sure | 03:04 |
| menn0 | wallyworld_: don't think so | 03:04 |
| wallyworld_ | regardless rsyslog and hence debug log is stuffed | 03:05 |
| menn0 | wallyworld_: the thing i've been investigating is definitely local provider specific when you have multiple environments set up | 03:05 |
| menn0 | sure | 03:05 |
| wallyworld_ | :-( | 03:05 |
| menn0 | wallyworld_: the client side of the connection sets the expected server name to a hardcoded value of "juju-rsyslog" | 03:08 |
| menn0 | wallyworld_: see composeTLS() in worker/rsyslog/worker.go | 03:09 |
| wallyworld_ | menn0: yeah, i'm backing out that change to see if it works | 03:09 |
| wallyworld_ | was changed 10th october | 03:09 |
| wallyworld_ | so recently | 03:09 |
| * wallyworld_ waits imatiently for aws to start up an instance or two | 03:10 | |
| davecheney | % git push --set-upstream origin 100-fix-issue-69 | 03:11 |
| davecheney | fatal: https://gopkg.in/juju/charm.v4/info/refs not valid: is this a git repository? | 03:11 |
| davecheney | I hate gopkg.in | 03:11 |
| davecheney | https://github.com/juju/charm/pull/70 | 03:11 |
| davecheney | anyone ? | 03:11 |
| davecheney | wallyworld_: thanks for the review | 03:14 |
| davecheney | i'm going to wait for rog | 03:14 |
| davecheney | this is his baby | 03:14 |
| davecheney | and I have just slapped it, soundly | 03:14 |
| wallyworld_ | sure, looked ok to me.... but im not 100% familiar with it | 03:15 |
| davecheney | wallyworld_: i'm changing the semantics of the method from what it promised, but didn't deliver | 03:18 |
| davecheney | to what it does | 03:19 |
| wallyworld_ | right :-) | 03:19 |
| davecheney | i tried to do it the other way, ie make the methods return a *copy* of the receiver | 03:19 |
| davecheney | but of course all the tests expect the current behavioru | 03:19 |
| davecheney | so all broke | 03:19 |
| davecheney | so, this change renames the method to _say_ what they _do_ | 03:19 |
| davecheney | as well as making them thread safe | 03:20 |
| * wallyworld_ nods | 03:20 | |
| wallyworld_ | davecheney: do you know anything about unit tags gaining a suffix? eg unit-flannel-2[2016] | 03:24 |
| wallyworld_ | menn0: that one line change to remove "juju-rsyslog" worked | 03:25 |
| wallyworld_ | but i'm not sure of the implications of just removing it | 03:25 |
| menn0 | wallyworld_: good. | 03:26 |
| wallyworld_ | the commit message says | 03:27 |
| wallyworld_ | tls.Config clients should use a meaningful ServerName This also fixes an issue with the rsyslog tls.Config not using a ServerName at all. | 03:27 |
| wallyworld_ | well, trouble is, if the rsyslog tls.Config uses a service name, it breaks | 03:27 |
| menn0 | wallyworld_: the docstring for composeTLS (added in that rev) says that setting ServerName is required to support pre-1.20.9 certs | 03:27 |
| menn0 | wallyworld_: so I suspect taking that line out is going to bring back the problem of rsyslog cert issues when upgrading from 1.20 to 1.21 | 03:28 |
| menn0 | wallyworld_: better check with wayne i guess | 03:28 |
| wallyworld_ | yeah | 03:28 |
| menn0 | i have to run some errands | 03:28 |
| wallyworld_ | i'm not an expert on this stuff | 03:28 |
| menn0 | neither am i | 03:29 |
| * wallyworld_ raiss a criitical bug | 03:29 | |
| menn0 | sigh | 03:29 |
| menn0 | back in a bit | 03:29 |
| davecheney | wallyworld_: tag ? nope | 03:36 |
| davecheney | 'fraid I wasn't watching | 03:37 |
| davecheney | smells like an actions thing | 03:37 |
| wallyworld_ | davecheney: np. unit agent logs now have tags like "unit-mysql-0[15930]" | 03:37 |
| wallyworld_ | could be actions related i guess, not sure | 03:37 |
| wallyworld_ | davecheney: a quick look at the names package doesn't seem to reveal any related changes | 03:38 |
| davecheney | yup | 03:39 |
| davecheney | doesn't look like that is whyere they are coming from | 03:40 |
| davecheney | it is rsyslog ? | 03:40 |
| davecheney | putting the PID on the end ? | 03:40 |
| wallyworld_ | yeah, i just came to the same conclusion ook at the code, so it must be a logging artifact | 03:41 |
| davecheney | [$PID] is the smell | 03:42 |
| wallyworld_ | the whle rsyslog thing has been radically changed recently | 03:44 |
| === kadams54 is now known as kadams54-away | ||
| === kadams54-away is now known as kadams54 | ||
| === kadams54 is now known as kadams54-away | ||
| wallyworld_ | davecheney: what's the official way to construct a name from a tag? eg if I have "unit-mysql-1", I want to get "mysql/1" | 05:50 |
| wallyworld_ | jam: connection dropped, sorry if this comes twice, how's your TLS knowledge? i'm not sure what to do with bug 1388688 - 1.21 is broken, but apparently the change was made so that 1.20.9 could be connected to | 06:58 |
| mup | Bug #1388688: juju debug-log is broken <debug-log> <regression> <rsyslog> <juju-core:Triaged> <https://launchpad.net/bugs/1388688> | 06:58 |
| jam | wallyworld_: it didn't, looking | 07:01 |
| jam | wallyworld_: is this a fresh bootstrap, an upgrade, an ... ? | 07:02 |
| wallyworld_ | jam: fresh bootstrap | 07:03 |
| wallyworld_ | 1.21 tip | 07:03 |
| davecheney | wallyworld_: names.NewUnitTag(string).String() | 07:03 |
| davecheney | wallyworld_: it depends | 07:03 |
| davecheney | if you have "unit-mysql-1" | 07:03 |
| davecheney | where did you get it from ? | 07:03 |
| jam | wallyworld_: I worked with wwitzel a bit on trying to figure out what the right fix was for this. | 07:03 |
| wallyworld_ | davecheney: it's the prefix of a log file entry | 07:04 |
| jam | specifically we have a variety of issues around TLS and cert names | 07:04 |
| jam | they want you to either put a hostname in the field | 07:04 |
| jam | or put IP addresses in another field | 07:04 |
| jam | but if we do IP addresses we have to create a new certificate anytime any IP addresses change | 07:04 |
| wallyworld_ | for juju < 1.20.9 only? | 07:04 |
| jam | we *were* putting the phrase "anything" as the server hostname for API connections, but nothing for stuff like rsyslog | 07:05 |
| jam | and then generating certs with "Hostname: *" | 07:05 |
| jam | which matches "anything" | 07:05 |
| jam | at one point, someone switched us to start using IP addresses | 07:05 |
| jam | but then upgrade breaks | 07:05 |
| wallyworld_ | oh :-( | 07:05 |
| jam | because it uses one cert that doesn't have IP addresses | 07:06 |
| jam | and when we discussed it | 07:06 |
| jam | it felt a lot better to use | 07:06 |
| jam | Hostname: * | 07:06 |
| jam | and meaningful "ServerName" fields | 07:06 |
| jam | and then maybe we could eventually migrate to Hostname: juju-meaningfulname instead of * | 07:06 |
| jam | now, it looks like the cert that existed actually put a DNSName into the Hostname field | 07:06 |
| jam | instead of Hostname: * | 07:07 |
| wallyworld_ | and we generate the cert? | 07:07 |
| jam | but (a) not all clouds give you DNS names, and (b) the DNS name is specific to a machine, and not to all of the API servers in HA mode | 07:07 |
| jam | wallyworld_: yeah, we generate the cert and how we connect to it | 07:07 |
| jam | but it looks like we're having trouble because we changed it, and now we need to support all the possible ways it could have been generated. | 07:07 |
| jam | I thought the IP address stuff was only in the 1.21-alpha stuff | 07:08 |
| jam | I'd have to dig deep to know what was done when and whereb | 07:08 |
| jam | because we seem to have changed these things in *micro* releases, which isn't a very good practice | 07:08 |
| jam | wallyworld_: is it possible to wait for wwitzel, or do you need it asap ? | 07:08 |
| wallyworld_ | jam: i've marked it as a critical regression and in my view, we cannot release alpha3 till it's fixed. we can talk to wayne when he comes online | 07:09 |
| wallyworld_ | cause debug log is broken | 07:09 |
| jam | wallyworld_: to be clear, the issue you are seeing is that 1.21-alpha3 agents are unable to talk to a 1.21-alpha3 API server's rsyslog daemon, correct ? | 07:10 |
| wallyworld_ | jam: yes, a fresh bootstrap with upload tools | 07:10 |
| davecheney | wallyworld_: you can use names.ParseTag(...) | 07:10 |
| wallyworld_ | on local provider and AWS | 07:10 |
| wallyworld_ | davecheney: thanks, will look into that | 07:11 |
| dimitern | morning | 07:23 |
| TheMue | morning | 08:31 |
| voidspace | morning all | 09:06 |
| mattyw | morning everyone | 09:24 |
| fwereade | tasdomas, ping | 09:29 |
| * fwereade waves at everybody | 09:30 | |
| voidspace | fwereade: hah, when I started reviewing wayne's branch there were no reviews for it | 09:52 |
| voidspace | fwereade: when I publish my review some have appeared... | 09:52 |
| fwereade | voidspace, extra reviews are not in any way a problem | 09:52 |
| voidspace | heh :-) | 09:53 |
| jam | hi fwereade | 10:02 |
| fwereade | jam, heyhey | 10:02 |
| voidspace | jam: I'm there | 10:55 |
| mattyw | fwereade, just noticed you pinging tasdomas - I think he's off today | 11:55 |
| fwereade | mattyw, cheers | 11:58 |
| mattyw | fwereade, do we have a way to enabling client logging to a file? | 12:06 |
| fwereade | mattyw, I *think* we do but I don't recall offhand | 12:07 |
| mattyw | fwereade, --log-file thing.log | 12:23 |
| fwereade | mattyw, heh, thanks | 12:24 |
| fwereade | bah, so, the bot's complaining about 1388073 and 1388493 -- both of which STM to be fix committed | 12:40 |
| fwereade | anyone? | 12:40 |
| jam | fwereade: bot only lets go once it is Fix Released for a while now | 12:46 |
| jam | fwereade: which IIRC, is Curtis seeing the actual CI test runs pass | 12:47 |
| fwereade | jam, ah, ok, makes sense | 12:47 |
| wallyworld_ | fwereade: one remaining critical bug open on 1.21 is bug 1388688 | 12:50 |
| mup | Bug #1388688: juju debug-log is broken <debug-log> <regression> <rsyslog> <juju-core:Triaged> <https://launchpad.net/bugs/1388688> | 12:50 |
| wallyworld_ | sorta need to ask wayne about it | 12:50 |
| * fwereade sees `tls.Config` in the first few lines and grumbles expressively to himself | 12:50 | |
| wallyworld_ | if not for that, we could release 1.21 alpha i think/hope according to the dashboard | 12:51 |
| fwereade | wallyworld_, ok, does he have a mail waiting for when he wakes up? | 12:51 |
| wallyworld_ | no, i was waiting for him to show up | 12:52 |
| wallyworld_ | i talked to jam a little also | 12:52 |
| fwereade | wallyworld_, that's very good of you, but isn't it late? | 12:52 |
| wallyworld_ | well, kinda | 12:52 |
| wallyworld_ | i also want to talk to curtis | 12:52 |
| wallyworld_ | to make sure everything else is ok for the release | 12:53 |
| === lazyWeekend is now known as lazyPower | ||
| === tvansteenburgh1 is now known as tvansteenburgh | ||
| * fwereade points wwitzel3 and wallyworld_ in each other's direction | 13:39 | |
| wallyworld_ | wwitzel3: bug 1388688 is a critical regression because debug log is broken, i'm not sure how to fix to make both 1.21 and 1.20 happy | 13:40 |
| mup | Bug #1388688: juju debug-log is broken <debug-log> <regression> <rsyslog> <juju-core:Triaged> <https://launchpad.net/bugs/1388688> | 13:40 |
| wwitzel3 | wallyworld_: ok, taking a look | 13:40 |
| wallyworld_ | ty | 13:40 |
| wallyworld_ | wwitzel3: here's some discussion from earlier http://pastebin.ubuntu.com/8803081/ | 13:42 |
| wwitzel3 | wallyworld_: thank you | 13:45 |
| wallyworld_ | np, thanks for looking | 13:45 |
| wwitzel3 | I should of never upgraded, ugh | 14:10 |
| wwitzel3 | my xmonand broke, my graphics is no longer using hardware rendering, my system python installation broke, I get weird kernel errors randomly, oh and the whole machine freezes up every 2-4 hours | 14:11 |
| === ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs: 1388853 | ||
| sinzui | jam, alexisb can you arrange for someone to address bug 1388853. We still haven't had a single blessed revision for alpha3 | 14:42 |
| mup | Bug #1388853: ppc64el unittests consistently fails <ci> <ppc64el> <regression> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1388853> | 14:42 |
| wwitzel3 | wallyworld_: ok, well I have a "fix" for it based on what jam was saying. | 14:56 |
| wwitzel3 | wallyworld_: keeps both the clients happy | 14:56 |
| wwitzel3 | natefinch, perrito666: ping | 15:08 |
| natefinch | wwitzel3: coming | 15:08 |
| natefinch | wwitzel3: perrito666 is at ods | 15:08 |
| wwitzel3 | thumper and me are on-call reviewers today, so will need someone to take a look at http://reviews.vapour.ws/r/324/, it fixes bug #1388688 | 15:08 |
| mup | Bug #1388688: juju debug-log is broken <debug-log> <regression> <rsyslog> <juju-core:Triaged by wwitzel3> <https://launchpad.net/bugs/1388688> | 15:08 |
| wwitzel3 | ping fwereade, wallyworld_, voidspace ^ | 15:19 |
| fwereade | wwitzel3, tests? | 15:22 |
| fwereade | wwitzel3, (and are you sure it'll work across all the versions in play?) | 15:23 |
| wwitzel3 | fwereade: yes, I confirmed it works for both versions | 15:25 |
| fwereade | wwitzel3, awesome | 15:25 |
| fwereade | wwitzel3, needs a unit test too, would lgtm with that | 15:25 |
| wwitzel3 | fwereade: as for tests, no idea how we would test it | 15:25 |
| voidspace | wwitzel3: are you appending * inside the body of the loop? | 15:26 |
| fwereade | wwitzel3, and that's a good point too | 15:26 |
| wwitzel3 | voidspace: oops, I was, need to push the updated diff, I noticed it after I initially pushed :) | 15:26 |
| voidspace | cool :-) | 15:27 |
| fwereade | wwitzel3, it looks like we're writing that stuff out somewhere | 15:28 |
| fwereade | wwitzel3, can the tests not read the certs back in and check their properties? | 15:28 |
| wwitzel3 | fwereade: yep, we can do that | 15:28 |
| wwitzel3 | fwereade: you're right, that is better than nothing | 15:29 |
| fwereade | wwitzel3, cool, thanks | 15:29 |
| alexisb | sinzui, looking | 15:32 |
| alexisb | sinzui, I am not seeing the attachment on that bug | 15:37 |
| sinzui | alexisb, :( I will try again | 15:39 |
| sinzui | alexisb, I added the log this time | 15:43 |
| alexisb | awesome, thanks sinzui | 15:43 |
| wwitzel3 | fwereade, voidspace: added tests that check the issuer, subject, and dnsnames for the cert. | 15:48 |
| voidspace | wwitzel3: did you push the tests? | 15:50 |
| voidspace | wwitzel3: don't think they're showing on RB yet | 15:50 |
| voidspace | checking github | 15:50 |
| alexisb | all, please consider bug 1388853 top priority | 15:50 |
| mup | Bug #1388853: ppc64el unittests consistently fails <ci> <ppc64el> <regression> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1388853> | 15:50 |
| wwitzel3 | voidspace: weird, yeah, they aren't showing in RB | 15:50 |
| alexisb | as 1.21 is still blocked and release was targeted for last weds | 15:50 |
| voidspace | wwitzel3: yeah, I'm looking at it on github | 15:51 |
| voidspace | wwitzel3: github sometimes throttles (drops) API calls | 15:51 |
| mgz | alexisb: I have a plan of sorts of that bug | 15:51 |
| alexisb | mgz, please share | 15:51 |
| mgz | I'm going to change the compiler version that runs the tests | 15:52 |
| mgz | we fixed the juju parts of the bug on friday, | 15:52 |
| mgz | it's just the current version of gccgo on trusty blows up with these tests - that's a change since brussels | 15:53 |
| alexisb | mgz ack, how can the core team help at this stage? | 15:55 |
| voidspace | wwitzel3: looks good to me | 15:55 |
| voidspace | wwitzel3: the code adds a * after any HostPorts | 15:56 |
| voidspace | wwitzel3: worth having a hostport in the test, so that code is exercised? | 15:56 |
| mgz | alexisb: will yell if I need it | 15:57 |
| alexisb | mgz, ack, thanks | 15:57 |
| voidspace | dimitern: ping | 16:01 |
| dimitern | voidspace, pong | 16:02 |
| voidspace | dimitern: I have ec2-api-tools installed | 16:02 |
| voidspace | dimitern: and the right AWC environment variables set | 16:02 |
| dimitern | voidspace, yeah? | 16:03 |
| voidspace | dimitern: and I can access the account using those credentials from code (goamz) | 16:03 |
| voidspace | dimitern: the command line client always reports | 16:03 |
| voidspace | Client.Unauthorized: Not Authorized | 16:03 |
| voidspace | dimitern: do you have any idea why that might be? | 16:03 |
| voidspace | the intarwebz are not helping on this | 16:03 |
| voidspace | the only references I can find are to Java exceptions | 16:03 |
| bodie_ | who should I discuss changes to charm with? rogpeppe? | 16:04 |
| rogpeppe | bodie_: if you like :) | 16:04 |
| dimitern | voidspace, not sure if this can be the cause, but I had to set a couple of additional env vars for aws: | 16:04 |
| voidspace | dimitern: ah | 16:04 |
| voidspace | maybe | 16:04 |
| voidspace | dimitern: you remember which ones? | 16:05 |
| bodie_ | rogpeppe, there is a change in charm v4 that breaks juju core state/charm_test TestTestingCharm, which tests against the "metered-custom" dummy charm that was removed from charm in 190b3a5d | 16:06 |
| bodie_ | rogpeppe, I want to make some additions to charm but I wasn't sure what to base them on since it seems like v4 is now not compatible (the commit in godeps is fine) | 16:06 |
| dimitern | voidspace, try these - http://paste.ubuntu.com/8804761/ | 16:07 |
| bodie_ | rogpeppe, should I just open a PR to replace the missing piece? I think we might want to move to v5 anyway since I have some Actions related changes I want to make charm more correct. I don't want deprecated bad bits to build up, but they are not serious... just irritating | 16:08 |
| rogpeppe | bodie_: are you saying that there have been some backwardly incompatible changes made to v4, or that you want to make some backwardly incompatible changes in the charm package? | 16:08 |
| voidspace | dimitern: that worked! | 16:08 |
| voidspace | dimitern: it might be worth updating the container addressability doc, which only mentions the key pair | 16:08 |
| voidspace | dimitern: I can do that | 16:08 |
| voidspace | dimitern: I think it was the EC2_KEYPAIR one which did it | 16:09 |
| voidspace | but I'll verify | 16:09 |
| bodie_ | rogpeppe, both. but the backwards incompatibility that was added to v4, can easily be reversed. and my changes don't have to be backwardly incompatible, but I would ideally like to remove the pieces they deprecate. | 16:09 |
| voidspace | dimitern: nope, that on its own wasn't enough | 16:09 |
| bodie_ | rogpeppe, does that clarify? | 16:09 |
| voidspace | dimitern: it was EC2_URL | 16:09 |
| rogpeppe | bodie_: in a call currently, will get back to you in 15 mins or so | 16:10 |
| bodie_ | sure thing :) | 16:10 |
| dimitern | voidspace, I think it was one of the key vars | 16:12 |
| dimitern | voidspace, one name is deprecated, but accepted the other is the "new" one you're supposed to use | 16:12 |
| dimitern | voidspace, I have some connection issues - did you see what I sent last? | 16:12 |
| dimitern | voidspace, ah, ok | 16:12 |
| dimitern | voidspace, please update the doc then :) | 16:12 |
| voidspace | dimitern: no, I have the key vars in my env already - just adding EC2_URL is sufficient to make it work | 16:12 |
| voidspace | dimitern: done | 16:12 |
| dimitern | voidspace, cheers! | 16:12 |
| wwitzel3 | voidspace: yeah, probably, it is funny .. we have NewRsyslogConfigHandler = newRsyslogConfigHandler in export_test.go, but then we never actually override or test it :P | 16:13 |
| voidspace | wwitzel3: hah | 16:13 |
| wwitzel3 | voidspace: I was hoping that wasn't the case, guess I'll have to write a suite for it | 16:13 |
| bodie_ | rogpeppe, fwiw, I think it would be good to roll forward to v5 since there are some old half-baked Actions definitions in a dummy charm in v4, that I want to remove, not deprecate. but as long as it is clear that they are old and smelly, it doesn't change the actual behavior of charm. it's testing-related only. | 16:14 |
| bodie_ | same goes for the breaking commit in v4. | 16:14 |
| rogpeppe | bodie_: i'm not sure we need to hold to quite such a stringent backwards compatibility policy for testing-related code | 16:14 |
| bodie_ | rogpeppe, I think TheMue had a good point: tests shouldn't be written against implementation details (dummy charms) of v4 anyway :) | 16:15 |
| bodie_ | but, maybe that's a little bit of a funky point (dummy charms) which I'm not sure how to approach | 16:16 |
| bodie_ | anyway, I will make my changes compatible, but there is some cruft building up and I'd like the charmers to have a proper example of what Actions should look like in a charm | 16:17 |
| rogpeppe | bodie_: out of the call now | 16:17 |
| rogpeppe | bodie_: i'm generally in agreement with TheMue's point, and i've been pondering a good direction for a while now. | 16:18 |
| bodie_ | yeah, it is a pretty subtle and maybe small pothole | 16:19 |
| rogpeppe | bodie_: i recently made a backwardly incompatible change to one of the testing repo charms too | 16:19 |
| rogpeppe | bodie_: i don't think we need to count the testing-related packages as part of the v4 contract. | 16:19 |
| rogpeppe | bodie_: although it's good to keep backwardly compatible when we can, obviously | 16:20 |
| bodie_ | my only concern is that if I want to make changes, I can't work on top of charm master if it's breaking juju master | 16:21 |
| voidspace | dimitern: the task breakdown and estimates in the Juju Container Addressability document are based | 16:41 |
| voidspace | dimitern: on AllocateAddress picking an address for us, and not on us managing addresses (and picking one from the right subnet) | 16:41 |
| voidspace | dimitern: I'm not sure that changes a lot regarding estimates - except it means more work in the container provisioner | 16:42 |
| voidspace | dimitern: unless I'm reading it wrong | 16:42 |
| dimitern | voidspace, that's correct, we might need to inflate the estimate there | 16:42 |
| voidspace | dimitern: right | 16:42 |
| voidspace | dimitern: there is a "Track the IP addresses we have allocated for each subnet" which could cover it | 16:42 |
| dimitern | voidspace, does the estimate there seem sane to you? | 16:43 |
| voidspace | dimitern: 2-3wks probably has enough wiggle room to cover it :-) | 16:44 |
| voidspace | dimitern: the other estimates seem ok to me | 16:44 |
| voidspace | in a hand-wavy kind of way... | 16:44 |
| dimitern | voidspace, :) cheers! | 16:44 |
| rogpeppe | bodie_: sure you can - juju master is only broken when the juju-core deps are updated, no? | 16:48 |
| rogpeppe | bodie_: the most important thing about compatibility is keeping go get working | 16:48 |
| bodie_ | gotya :) | 16:48 |
| rogpeppe | bodie_: with the testing deps, the broken deps are usually only one level, between tests and testing package | 16:50 |
| rogpeppe | bodie_: so, just to summarise, your backwardly incompatible changes only involved changes to the testing charms repo? | 16:50 |
| bodie_ | that's right, but they don't have to be backwardly incompatible. I would just like them to be :) | 16:52 |
| rogpeppe | bodie_: i think you should go for it | 16:54 |
| rogpeppe | bodie_: for bonus points, check that the charmstore repo passes its tests with the new package | 16:55 |
| === fabrice is now known as fabrice|family | ||
| rogpeppe | bodie_: and we should perhaps think about moving toward entirely removing testing/repo | 16:56 |
| rogpeppe | bodie_: there will be places where it's still useful, but i think a better way would be to have charm testing repos inside the packages/repos that *use* the charm package | 16:57 |
| bodie_ | rogpeppe, perhaps a good approach would be for charm to expose some methods for generating mock charms for core tests that need to integrate with charm | 16:58 |
| bodie_ | (looking forward) | 16:58 |
| rogpeppe | bodie_: that's another decent approach that might be good | 16:59 |
| rogpeppe | bodie_: the only down side is that if every test is doing that, it can lead to a lot of disk churn | 16:59 |
| rogpeppe | bodie_: a lot of tests don't really care about the details of the charms - they just need *some* example charm to reploy | 17:00 |
| rogpeppe | deploy | 17:00 |
| bodie_ | heh, was still mid comprehending your thoughts | 17:00 |
| * bodie_ scribbles "listen better" note with all the others | 17:01 | |
| bodie_ | I'm not really seeing how that solves the problem since it's like turning the deps inside out, where clients must define the interactions they expect with charm. but I don't really have an opinion either way yet -- I was thinking of generating in-memory fake charms though, but I didn't think about how much file interaction there is with charm | 17:02 |
| rogpeppe | bodie_: another thing it might be interesting to do is make it easier to have an all in-memory charm | 17:02 |
| rogpeppe | bodie_: ha | 17:03 |
| bodie_ | +1! brb, meeting | 17:03 |
| rogpeppe | bodie_: i think it depends how one thinks of the testing charms | 17:03 |
| rogpeppe | bodie_: i think of them like they're testing data - a test wants to see what will happen when we do something with a given charm. it could specify that charm in the testing code, or as external data. | 17:04 |
| rogpeppe | bodie_: the external-data approach is arguably more readable, as the charm's in the form that it is usually specified | 17:05 |
| bodie_ | rogpeppe, maybe it's as simple as having a function that cranks out a memory representation of the charm files, and then loads them as if they were a "real" charm file | 17:06 |
| rogpeppe | bodie_: but the in-testing-code is more flexible | 17:06 |
| bodie_ | it's dumb, but it could be an easy out | 17:06 |
| rogpeppe | bodie_: it's easy enough to do | 17:07 |
| rogpeppe | bodie_: apart from the 100s of tests that would need changing in juju-core | 17:07 |
| rogpeppe | bodie_: and it's the thought of that that leads me to suggest a half-way house, where juju-core has its own charm testing repo | 17:08 |
| rogpeppe | bodie_: so then we don't need to change all those tests | 17:08 |
| rogpeppe | bodie_: (assuming we get juju-core to prime the charm package on the location of the juju-core charm repo) | 17:08 |
| === kadams54 is now known as kadams54-away | ||
| === tvansteenburgh is now known as tvan|lunch | ||
| wwitzel3 | voidspace: updated the test to exercise the rsyslogHosts conditional logic | 18:08 |
| voidspace | wwitzel3: looking | 18:10 |
| voidspace | wwitzel3: LGTM | 18:11 |
| wwitzel3 | voidspace: thanks | 18:13 |
| mgz | update on ppc blocker, bug 1388853 | 18:13 |
| mup | Bug #1388853: ppc64el unittests consistently fails <ci> <ppc64el> <regression> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1388853> | 18:13 |
| mgz | the crash stuff is dealt with by having a newer compiler than is available in trusty, our test machine has this | 18:14 |
| mgz | if needed, ppa:ubuntu-toolchain-r/ppa can get you one as well | 18:15 |
| mgz | the remaining failures documented in that bug seem like ppc-specific test issues which just need fixing, naming of arch/tools issues and such like | 18:15 |
| alexisb | natefinch, can someone in the appropriate timezone take a look? ^^^ | 18:16 |
| mgz | I can tell anyone who wants to take it up how to get on our actual ppc machine if needed | 18:16 |
| alexisb | mgz, can you make sure there is an updateint he bug | 18:16 |
| mgz | alexisb: yeah, I'm also going to close the previous bug as that just got very confused | 18:17 |
| natefinch | alexisb: yep | 18:19 |
| natefinch | mgz: writing up how to get on our PPC machine in a place we can easily find it would be very useful | 18:19 |
| mgz | natefinch: we don't have a general machine for playing on, just the one we use as the jenkins slave | 18:20 |
| mgz | alexisb, natefinch: added details to the bug | 18:33 |
| natefinch | mgz: thanks | 18:41 |
| voidspace | g;night all | 18:59 |
| === tvan|lunch is now known as tvansteenburgh | ||
| katco | ericsnow: i may be looking at this wrong, but it looks like my review request didn't have a diff generated? http://reviews.vapour.ws/r/327/ | 19:43 |
| ericsnow | katco: yeah, look like it :( | 19:43 |
| ericsnow | katco: that still happens infrequently | 19:43 |
| katco | ericsnow: did i do something wrong/different, or is that a known issue? | 19:44 |
| ericsnow | katco: it happens infrequently enough that I haven't looked into it | 19:44 |
| ericsnow | katco: not your fault | 19:44 |
| katco | ericsnow: k, ty sir :) | 19:44 |
| ericsnow | katco: you can still use rbt to post the patch | 19:45 |
| katco | ericsnow: righty-o | 19:45 |
| === kadams54 is now known as kadams54-away | ||
| menn0 | waigani: morning | 20:17 |
| waigani | menn0: morning | 20:19 |
| katco | am i correct in assuming landings are blocked? | 20:19 |
| menn0 | yes they are | 20:19 |
| menn0 | due to bug 1388853 | 20:19 |
| mup | Bug #1388853: ppc64el unittests consistently fails <ci> <ppc64el> <regression> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1388853> | 20:19 |
| katco | ok, thanks menn0 | 20:19 |
| katco | yeah, figured, but wanted to ask just to be sure | 20:19 |
| menn0 | martin has done a good analysis. the problems looks pretty simple to fix so i'm going to have a go. | 20:19 |
| katco | good luck, menn0. may the source be with you. | 20:20 |
| mwhudson | good morning | 20:21 |
| menn0 | mwhudson: howdy | 20:21 |
| mwhudson | someone put "dummy-admin dummy-admin just now just now" in the expected output of a test? | 20:22 |
| menn0 | mwhudson: someone has | 20:22 |
| mwhudson | oh dear | 20:23 |
| mwhudson | (well, unless the test uses a synthetic time source, which it apparently doesn't...) | 20:23 |
| menn0 | mwhudson: and it could even be thumper ... | 20:23 |
| menn0 | mwhudson: although I think I might have reviewed that one | 20:23 |
| mwhudson | in a way that's good because i have no problems with telling thumper that he's done something stupid :) | 20:24 |
| menn0 | LOL | 20:24 |
| mwhudson | whereas i might be a bit more circumspect with someone i didn't know so well :) | 20:25 |
| menn0 | i'll fix that and have a look at the other problem too | 20:25 |
| menn0 | although I can't get that to happen on my machine | 20:25 |
| menn0 | I suspect you actually need to be on a ppc64 host | 20:26 |
| mgz | menn0: I can tell you how to access the jenkins build slave if that would help | 20:26 |
| menn0 | mgz: i have a few ideas to try first but could you send me the details anyway | 20:28 |
| mwhudson | menn0, waigani, davecheney: email standup i guess? | 20:30 |
| menn0 | mwhudson: unless you would prefer otherwise? | 20:31 |
| waigani | I'm easy either way | 20:31 |
| mwhudson | i am currently in a cafe on extremely slow 3g so email sounds good :) | 20:31 |
| waigani | email it i | 20:31 |
| waigani | s | 20:31 |
| menn0 | cool | 20:33 |
| === kadams54-away is now known as kadams54 | ||
| mgz | menn0: https://pastebin.canonical.com/119817 | 20:40 |
| menn0 | mgz: cheers | 20:44 |
| waigani | menn0: I may have hit a chicken and egg | 21:24 |
| menn0 | waigani: in what way | 21:24 |
| menn0 | ? | 21:24 |
| waigani | menn0: there is a settings doc which uses a global env key as it's key | 21:25 |
| waigani | menn0: it holds the configuration for the environment | 21:25 |
| waigani | menn0: the key is now perpended with the env-uuid | 21:26 |
| waigani | menn0: and the doc exists, with a docID _id, before the upgrade step is run | 21:27 |
| waigani | menn0: so during the upgrade step, it gets perpended with the env-uuid again | 21:29 |
| waigani | menn0: and the test also fails as len(oldDocs) != len(newDocs) - that one I can fix easy enough | 21:30 |
| katco | wallyworld_: noooot sure what happened, rejoining | 21:30 |
| menn0 | waigani: give me a sec | 21:30 |
| menn0 | waigani: ok | 21:32 |
| waigani | menn0: yep, went you get a sec - does it jog anything from the workarounds you had to do? | 21:32 |
| menn0 | waigani: sounds like you might need to add another little hack | 21:33 |
| menn0 | waigani: but i'm not clear on why the env-uuid gets added twice | 21:33 |
| menn0 | waigani: can you explain that? | 21:33 |
| waigani | menn0: good question, give me a sec I'll see... | 21:34 |
| waigani | menn0: because I used the global env key in one of the dummy docs to migrate. | 21:36 |
| waigani | menn0: so if I remove that and make the test expect len(oldDocs) + 1, the test should pass | 21:37 |
| waigani | or remove the envion doc from the newIDs returned... | 21:39 |
| waigani | ah no, okay I'll stop thinking out loud and sort this | 21:40 |
| menn0 | waigani: ok. let me know if you need a sounding board again :) | 21:40 |
| waigani | menn0: thanks :) | 21:43 |
| menn0 | This is one half of fixing the CI blocker. can someone PTAL? http://reviews.vapour.ws/r/329/ | 21:48 |
| davecheney | menn0: can you just delete the check fo rthe duration | 21:49 |
| davecheney | it's unknowable | 21:49 |
| natefinch | +1 | 21:50 |
| menn0 | davecheney, natefinch: but I've made the test accept any possible duration. is that not ok? | 21:50 |
| menn0 | at least then the test ensure that we're getting the right shaped output | 21:51 |
| davecheney | menn0: fells like solving the wrong problem | 21:51 |
| davecheney | now that test will fail when tim changes the printing of an _arbitrary_ duration | 21:51 |
| menn0 | but if I do that there's nothing testing that the duration emitted by "user list" is sanely formatted at all! | 21:54 |
| menn0 | davecheney: ^ | 21:55 |
| menn0 | davecheney: or put another way, there's nothing test that the duration formatting function is in use | 21:55 |
| menn0 | testing even | 21:56 |
| natefinch | this is why I like internal tests ;) | 21:57 |
| menn0 | davecheney, natefinch: as this is becoming yet another testing debate and I just want to get this CI blocker sorted so that I can get back to useful work I will just do as you ask | 21:59 |
| natefinch | menn0: for the record, I'm ok with either fix | 21:59 |
| davecheney | menn0: natefinch same | 22:02 |
| menn0 | natefinch, davecheney: so what, leave the branch as it is? | 22:03 |
| menn0 | I just started relaxing the regex further | 22:03 |
| natefinch | menn0: I put a ship it on it | 22:03 |
| menn0 | ok | 22:03 |
| davecheney | menn0: do the absolute minimum required to get the test passing | 22:06 |
| davecheney | as you say | 22:06 |
| davecheney | this is a pointless testing debate | 22:06 |
| davecheney | one of N | 22:06 |
| menn0 | thanks. i've sent that off for merging. | 22:07 |
| menn0 | now for the other half of the ticket. | 22:07 |
| wallyworld_ | menn0: i can pick up the ppc64 test failure | 22:23 |
| katco | wallyworld_: fwiw i had to clear out my cookies... looks like time-changes mess with something w/ google apps | 22:23 |
| wallyworld_ | \o/ | 22:23 |
| katco | wallyworld_: the time changes from brussels probably triggered it on my laptop, daylight savings time for my main comp | 22:23 |
| * katco shrugs | 22:23 | |
| * wallyworld_ shrugs too | 22:24 | |
| katco | software! | 22:24 |
| menn0 | wallyworld_: ok. it looks related to your changes from yesterday. | 22:24 |
| wallyworld_ | menn0: yep - i had to add extra metadata to account for 1.20 being wrong but clearly when run on non amd64, the expected list of metadata entries is different | 22:25 |
| menn0 | wallyworld_: the fix for the other part of the ticket (TestUserList) just landed | 22:27 |
| wallyworld_ | menn0: awesome, ty | 22:27 |
| === kadams54 is now known as kadams54-away | ||
| === kadams54-away is now known as kadams54 | ||
| lazyPower | Ran into an interesting AWS issue with a customer that I've not seen before - if anyone has a moment to take a look - https://launchpad.net/bugs/1389014 | 23:34 |
| mup | Bug #1389014: juju AWS EC2 "unit-get public-address" gives private address <juju-core:New> <https://launchpad.net/bugs/1389014> | 23:34 |
| davecheney | lazyPower: is the custoemr using vpc | 23:41 |
| davecheney | ? | 23:41 |
| davecheney | themonk@redqueen:~$ juju run -e amazon --service my-charm1 "unit-get public-address" | 23:41 |
| davecheney | 172.31.4.32 | 23:41 |
| davecheney | ^ how did this work | 23:41 |
| lazyPower | davecheney: no idea its themonk in #juju though | 23:41 |
| lazyPower | exactly! | 23:41 |
| lazyPower | a) how do you get an IP, b) how do you get the *same* ip | 23:42 |
| davecheney | how was their server able to contact 172.31.4.32 ? | 23:42 |
| davecheney | that is a private ip | 23:42 |
| lazyPower | and c) how do you get a private ip | 23:42 |
| davecheney | lazyPower: juju just returns whatever the aws metadata services tells it | 23:42 |
| davecheney | so if aws says you'r ip address is 1.1.1.1 | 23:42 |
| davecheney | that's why juju knows as well | 23:42 |
| lazyPower | davecheney: <themonk> lazyPower, what is 'provisioned in a VPC' ? | 23:44 |
| lazyPower | i'm going to say the answer is probably no... | 23:44 |
| lazyPower | davecheney: do you have a minute to pop in #juju and interface with themonk over his AWS issue? If not no biggie - he's got an open bug that he can track. | 23:46 |
| davecheney | curl http://169.254.169.254/latest/public-ipv4 | 23:46 |
| davecheney | this is what juju does | 23:46 |
| davecheney | join #juju | 23:46 |
| === kadams54 is now known as kadams54-away | ||
| davecheney | urgh | 23:46 |
| davecheney | sorry, i thought I was in that channel | 23:46 |
| davecheney | i have so many called #juju | 23:46 |
| lazyPower | all good davecheney :) I have the same issue between canonirc and freenode. so i know the feeling. | 23:47 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!