/srv/irclogs.ubuntu.com/2016/09/29/#juju-dev.txt

axwperrito666: I'm not sure what your message was in reply to ("I believe so too...")00:24
* axw goes to get breakfast00:24
alexisbalrighty all I am off for the night, see everyone tomorrow01:12
thumperaxw: good point about not showing for only one unit01:40
thumperI'll look into it01:40
axwthumper: thanks. one complication would be when you specify status filtering01:41
axwi.e. you might just be showing a subset of units, and there may be more than what you're showing01:41
thumperyeah...01:41
thumperalso, what about showing leader in yaml / json?01:42
thumperkeep there even if only one?01:42
axwthumper: IMO it's fine to have it in there01:42
thumperok01:43
* thumper will think on it01:43
axwthumper: seems I led you astray on the openstack storage fix. https://bugs.launchpad.net/juju/+bug/161509503:12
mupBug #1615095: storage: volumes not supported <landscape> <juju:In Progress by axwalk> <https://launchpad.net/bugs/1615095>03:12
thumperoh?03:12
thumperdamn03:12
axwthumper: you know how we're looking for the volume endpoint in SetConfig?03:12
axwthumper: doesn't work, because the client isn't authenticated yet03:13
thumperah03:13
thumperoops03:13
axwwe don't authenticate to keep Open fast03:13
axwthumper: gtg out, will fix it tomorrow03:13
thumperack03:13
=== frankban|afk is now known as frankban
rogpeppe1axw: thanks a lot for the review of my letsencrypt branch08:13
dimiternrogpeppe1: hey, I'm trying to figure out why the api server started to throw errors like this: ERROR juju.worker runner.go:210 exited "apiserver": cannot start api server worker: crypto/tls: private key does not match public key08:38
dimiternrogpeppe1: it seems to happen frequently after the controller machine was restarted, on CI08:39
rogpeppe1dimitern: interesting08:39
=== rogpeppe1 is now known as rogpeppe
rogpeppedimitern: which version of juju are you using?08:39
dimiternrogpeppe: what I found so far seems to indicate the key file was corrupted somehow08:40
rogpeppedimitern: wouldn't it be nice if debugging output printed the whole error stack?08:41
dimiternrogpeppe: 2.0, on a feature branch master-lp162703708:41
rogpeppedimitern: so have you seen this issue on master?08:41
dimiternrogpeppe: yes, 2 days ago08:42
dimiternrogpeppe: http://reports.vapour.ws/releases/4429/job/functional-container-networking-maas-2-0/attempt/109308:42
dimiternrogpeppe: and the last occurrence is http://reports.vapour.ws/releases/4436/job/functional-container-networking-maas-2-0/attempt/110808:42
dimiternrogpeppe: the errors are visible in machine-0.log on the controller, after it has been rebooted - it seems intermittent though08:43
rogpeppedimitern: ah, so the reports.vapour.ws log doesn't have that error in08:44
dimiternrogpeppe: it just shows it failed to connect to the apiserver after 10m of trying08:44
dimiternrogpeppe: but the machine-0.log shows the apiserver keeps restarting every 3s with that error08:44
rogpeppedimitern: i don't see any occurrence of "does not match" in http://data.vapour.ws/juju-ci/products/version-4429/functional-container-networking-maas-2-0/build-1093/controller/machine-0/machine-0.log.gz08:45
dimiternrogpeppe: sorry, so the one on master shows a different error, but still related: 2016/09/27 13:24:45 http: TLS handshake error from 10.0.30.40:43974: remote error: bad certificate08:46
dimiternor maybe not related ... /me is getting confused :/08:47
rogpeppedimitern: it seems like it might well be related08:47
dimiternrogpeppe: I know some things around tls / certs have changed lately08:47
rogpeppedimitern: they have?08:47
rogpeppedimitern: have you got a link to a log that contains the "private key does not match" error?08:48
dimiternrogpeppe: might have.. not sure - I know you added tests, but that shouldn't have caused such things08:48
dimiternyeah, just a sec08:48
rogpeppedimitern: yeah, actually, i did change some stuff, it's true, i'd forgotten that08:49
dimiternrogpeppe: here's the log http://data.vapour.ws/juju-ci/products/version-4436/functional-container-networking-maas-2-0/build-1107/machine-0.log.gz08:49
rogpeppedimitern: are all the failures since that commit (42d0c9c07cffbc5075cb05add9e1398056f0d890) ?08:50
dimiternrogpeppe: it's from the feature branch CI run, but the branch itself does not have anything to do with tls or certs - just changes in provider/maas08:50
dimiternrogpeppe: let me check08:50
rogpeppedimitern: does that feature branch include commit 42d0c9c07cffbc5075cb05add9e1398056f0d890 ?08:50
rogpeppedimitern: does the report (http://reports.vapour.ws/releases/4429/job/functional-container-networking-maas-2-0/attempt/1093) mention the commit id anywhere? i can't find it currently.08:52
dimiternrogpeppe: I can't seem to find that commit on the fbranch or master08:52
rogpeppedimitern: i don't know what "revision 4429" means in a git context08:52
dimiternrogpeppe: check the top of the report - has a link to the commit hash tested and it links to github08:52
rogpeppedimitern: what's the text of the link?08:53
dimiternrogpeppe: ah, no actually - the08:53
dimiternJenkins link links to the job which mentions the commit id08:53
dimiterne.g. http://juju-ci.vapour.ws:8080/job/functional-container-networking-maas-2-0/1107/08:53
dimiterngitbranch:master-lp1627037:github.com/juju/juju bbd844f08:54
dimitern 08:54
rogpeppedimitern: i get a 404 from the jenkins link08:54
dimiternrogpeppe: ah, looking at that job's build history I can confirm the commit  ​42d0c9c08:54
rogpeppedimitern: how do you find the build history?08:54
dimiternrogpeppe: failed08:54
rogpeppedimitern: failed what?08:55
dimiternrogpeppe: can you open http://juju-ci.vapour.ws:8080/job/functional-container-networking-maas-2-0/ for example?08:55
rogpeppedimitern: nope08:55
rogpeppedimitern: 40408:55
voidspacemgz: ping08:55
dimiternrogpeppe: try logging in and then the link above?08:57
rogpeppedimitern: ok, works now, thanks08:57
rogpeppedimitern: one might've thought the 404 would contain a login link08:57
anastasiamacrogpeppe: try to login ;) this 404 is misleading... it really means that u r not authenticated...08:57
anastasiamac:)08:57
voidspacemgz: unping :-)08:57
rogpeppeanastasiamac: yeah, i hate that :)08:57
dimiternrogpeppe: yeah, it's not *that* helpful..08:57
anastasiamacrogpeppe: \o/ keeping us on our toes08:58
rogpeppedimitern: so it looks like http://juju-ci.vapour.ws:8080/job/functional-container-networking-maas-2-0/1093/console doesn't include the commit we're thinking of (42d0c9c)09:01
rogpeppedimitern: so perhaps we shouldn't level the blame at that :)09:01
rogpeppedimitern: assuming the "REVISION_ID=a5606e7126c0ee5b816b3c52e85f5c77635b5ce3" holds the revision being tested09:03
dimiternrogpeppe: well, 42d0c9c did fail the job on master though...09:07
dimiternrogpeppe: http://juju-ci.vapour.ws:8080/job/functional-container-networking-maas-2-0/1098/09:08
rogpeppedimitern: and that was the first time it failed like that?09:08
dimiternrogpeppe: it failed before, but not like this I think.. checking earlier logs09:08
dimiternrogpeppe: it failed before because the substrate was unclean - no machine matches constraints09:09
dimiternrogpeppe: but interestingly, the very next run on 42d0c9c passed ok..09:11
dimiternmight be just flaky.. or misconfigured maas node09:11
dooferladdimitern, frobware: So, this keeps happening:09:11
dooferladMODEL  CONTROLLER  CLOUD/REGION  VERSION09:12
dooferladfoo    maas        maas          2.0-rc2.109:12
dooferladAPP  VERSION  STATUS  SCALE  CHARM  STORE  REV  OS  NOTES09:12
dooferladUNIT  WORKLOAD  AGENT  MACHINE  PUBLIC-ADDRESS  PORTS  MESSAGE09:12
dooferladMACHINE  STATE    DNS            INS-ID                                                          SERIES  AZ09:12
dooferlad0        started  192.168.1.101  /MAAS/api/1.0/nodes/node-67b68b08-1452-11e6-9228-54a050d5d9eb/  xenial  default09:12
dooferlad0/lxd/0  started  10.0.0.199     juju-df0cd5-0-lxd-0                                             xenial09:12
dooferlad1        started  192.168.1.102  /MAAS/api/1.0/nodes/node-7b5b54e0-1452-11e6-9228-54a050d5d9eb/  xenial  default09:12
dooferlad1/lxd/0  started  192.168.1.103  juju-df0cd5-1-lxd-0                                             xenial09:12
dooferladoops, that should have gone to pastebin09:12
dooferladhang on09:12
dimiternyeah09:12
dimitern:)09:12
dooferladhttps://www.irccloud.com/pastebin/PrlicZuK/09:12
dooferladI have a smart IRC client, dumb user.09:12
rogpeppedimitern: which is the juju report associated with http://juju-ci.vapour.ws:8080/job/functional-container-networking-maas-2-0/1098/ ?09:12
frobwaredooferlad: on the 0/lxd/0 machine can you cat:09:12
frobware/var/lib/cloud/seed/nocloud-net/network-config09:13
dooferladfrobware: on it09:13
rogpeppedimitern: i'd like to see the machine-0.log for it09:13
dooferladhttps://www.irccloud.com/pastebin/5d4y61hL/09:14
dooferladfrobware: ^^09:14
dimiternrogpeppe: it should say somewhere.. looking09:14
dooferladfrobware: it is the same as on the LXC that worked09:15
dimiternrogpeppe: http://reports.vapour.ws/releases/443109:15
frobwaredooferlad: MAAS?09:16
dooferladfrobware: yes09:16
dooferladfrobware: 1.909:16
frobwaredooferlad: this look like you need mick's fix09:17
rogpeppedimitern: but which link on that page has the actual test run that contains the machine-0.log artifact for that run?09:18
frobwaredooferlad: https://github.com/juju/juju/pull/627609:18
frobwaredooferlad: I just ran into this too.09:18
dooferladfrobware: it was doing this yesterday too though, before that landed (I think)09:18
frobwaredooferlad: Just checking that it has landed...09:19
dimiternrogpeppe: scroll down09:19
frobwaredooferlad: so it has.09:19
rogpeppedimitern: i see hundreds of links but no artifacts09:19
dimiternrogpeppe: and search for the job name - on the build number to the right (hovering) you'll see the logs09:19
rogpeppedimitern: what job name am i looking for?09:19
frobwaredooferlad: 17 hours ago - is that before or after the CI job?09:20
dimiternrogpeppe: it's frustrating how it overlaps the logs, but if you hover on functional-container-networking-maas-2-0 | Succeeded | >1099<09:21
dooferladfrobware: it is showing up in the history, so it must have merged, right?09:21
rogpeppedimitern: but that's a success - i thought this was meant to have failed09:21
frobwaredooferlad: agreed. but pull and check09:21
dimiternrogpeppe: the list that appears has 1098 ... but unfortunately http://reports.vapour.ws/releases/4431/job/functional-container-networking-maas-2-0/attempt/1098 does no appear to have the machine-0 log09:21
dooferladfrobware: or is this the 'new process' stuff that I have been ignoring09:22
frobwaredooferlad: that was going through my head09:22
dimiterndamn :/ hate it when that happens!09:22
dooferladfrobware: yes, it is there in my build09:22
rogpeppedimitern: so you're not sure whether 42d0c9c failed because of the issue you've described?09:23
dooferladfrobware: and it wasn't yesterday when I was running into it the first time09:25
rogpeppedimitern: looking at the changes i made in that branch, i'd find it very unlikely that they could cause an extra error case when starting the api server09:25
rogpeppedimitern: the changes were all about how changed certificates were handled09:26
rogpeppedimitern: i think the issue probably arises because a duff cert/key pair is being passed into the api server09:27
rogpeppedimitern: it's difficult to say without being able to reproduce the issue09:28
rogpeppedimitern: perhaps add some debugging log statements that might help if the issue happens again09:28
rogpeppedimitern: for example if the cert is wrong, log it and the key09:28
dimiternrogpeppe: well, I'll let you know if we repro it again, and thanks for looking into it!09:29
rogpeppedimitern: np09:29
frobwaredooferlad: did you draw any conclusion?09:41
dooferladfrobware: no. Got stuck in other email. Will take another look in a moment or 1009:41
frobwaredooferlad: pulling that change on top of what I'm doing fixed that lxd/dhcp/eth0 case for me09:46
dooferladfrobware: frustrating that it didn't help me then :-|09:51
frankbanhey, I need a review for https://github.com/juju/juju/pull/6352 anyone available? thanks!09:52
* dooferlad tries again in case of user error09:52
rogpeppejam: i've replied to https://github.com/juju/juju/pull/6345 and made one or two changes. do you think it's good to land?10:06
dimiternvoidspace: your PR fails make check on trusty btw10:09
dimiternvoidspace: you can see which failed in trusty.out log - GetServerAddrs or something like this10:10
dimiternvoidspace: http://juju-ci.vapour.ws:8080/job/github-merge-juju/9362/artifact/artifacts/trusty-out.log10:10
dimiternvoidspace: ah, you've updated that I guess it will pass now - sorry for the noise :)10:14
frankbanaxw: could you please take a look at https://github.com/juju/juju/pull/6352 when you have time? thanks10:16
voidspacedimitern: I know, already fixed10:16
voidspacedimitern: but thanks10:16
dooferladok, latest master still has the random not getting the right address problem. No question. Yay. Starting more digging...10:23
dooferladfrobware, dimitern: the answer is in /var/log/lxd/<name>/lxc.conf -- the container that doesn't end up on the right subnet gets:10:30
dooferladlxc.network.type = veth10:30
dooferladlxc.network.flags = up10:30
dooferladlxc.network.link = lxdbr010:30
dooferladlxc.network.hwaddr = 00:16:3e:4c:3d:b310:30
dooferladlxc.network.name = eth010:30
dooferladThe right thing would be more like:10:30
dooferladlxc.network.type = veth10:30
dooferladlxc.network.flags = up10:30
dooferladlxc.network.link = br-eth010:30
dooferladlxc.network.hwaddr = 00:16:3e:ca:f3:6c10:30
dooferladlxc.network.mtu = 150010:30
dimiterndooferlad: what's in the machine-0.log around PrepareContainerInterfaceInfo API call?10:30
dooferladlxc.network.name = eth010:30
dooferladlxc.network.type = veth10:30
dooferladlxc.network.flags = up10:30
dooferladlxc.network.link = br-eth110:30
dooferladlxc.network.hwaddr = 00:16:3e:40:36:2110:30
dooferladlxc.network.mtu = 150010:30
dooferladlxc.network.name = eth110:30
dimiterndooferlad: any errors10:30
dimiternaaaaah stop it:)10:30
frobwaredooferlad: what does: lxc config show <container-name> show?10:31
anastasiamacdooferlad: \o/ i second dimitern :D10:31
dooferladbah10:31
frobwaredooferlad: humbug?10:32
dooferladdimitern: logs https://www.irccloud.com/pastebin/4xfeEDhy/10:33
dimiterndooferlad: I can't see PrepareContainerInterfaceInfo response there?10:37
dimiterndooferlad: it should be a bit earlier in the contoller10:38
dimiterndooferlad: controller's machine log10:38
frobwaredimitern, dooferlad, voidspace, babbageclunk: I concluded my manual testing on https://github.com/juju/juju/pull/634210:39
frobwaredimitern, dooferlad, voidspace, babbageclunk: want to land this but looking for final review/approval now10:40
dimiternfrobware: http://pasteboard.co/8ThbB7FWl.png10:42
rogpeppedimitern: do you know what the set-numa-control-policy setting does, by any chance?10:45
rogpeppedimitern: i was mucking around in controller/config.go and came across: // NumaControlPolicyKey stores the value for this setting.10:45
rogpeppedimitern: which is possibly the most uninformative doc comment I have ever come across10:46
rogpeppethe above is a question for anyone else too, BTW.10:46
dooferladdimitern: sorry, my fingers typed 'juju ssh 0', not 'juju ssh -m controller 0'10:46
* dooferlad curses CLI changes10:46
anastasiamacrogpeppe: sounds like my thing :D off memory, it was done as a fix for https://bugs.launchpad.net/juju-core/+bug/135033710:48
mupBug #1350337: Juju DB should use numactl when running mongo on multi-socket nodes <canonical-bootstack> <hours> <maas> <mongodb> <juju-core:Fix Released by anastasia-macmood> <https://launchpad.net/bugs/1350337>10:48
rogpeppethis is also hilarious: // DefaultNUMAControlPolicy should not be used by default.10:48
anastasiamacrogpeppe: it's a setting specific for NUMA machines. r u using NUMA?10:48
rogpeppeanastasiamac: no, but i think that every attribute should be adequately documented10:49
anastasiamacrogpeppe: mongo needs to have flag setup to run on NUMA. Hence, the setting10:49
dimitern:D10:49
dimiternit is10:49
dimiternhilarious10:49
frobwaredimitern: guestmount -a /var/lib/libvirt/images/maas19-node6.qcow2 -o ro -m /dev/sda1 /mnt10:49
anastasiamacrogpeppe: agreed :) it was my ealry days on juju and of course, as a dev, it was obvious to me at the time :) sorry10:49
rogpeppeanastasiamac: ok, well it should be documented that it's about running mongo with numactl for a start10:49
anastasiamacrogpeppe: agreed, feel free to clarify as an external force that is now aware :-P10:50
anastasiamacrogpeppe: especially, since u r in the area :)10:50
rogpeppeanastasiamac: do you know what a "multi-socket server" is?10:51
anastasiamacrogpeppe: :D not sure where u r reading "server"... i thought it was about nodes :)10:52
rogpeppeanastasiamac: BTW shouldn't this be done by some code on the system to test if it's NUMA rather than with a config setting?10:52
rogpeppeanastasiamac: from the bug report you linked "When running Juju on multi-socket servers I see this in the mongo log:"10:52
rogpeppeanastasiamac: perhaps it's talking about physical CPU sockets10:53
anastasiamacrogpeppe: haha ;) no, i do not know... can't really cast my mind that far back: feels like another lifetime \o/10:54
rogpeppeanastasiamac: how about this for a doc comment?10:54
rogpeppe// NUMAControlPolicyKey specifies whether the MongoDB10:54
rogpeppe// instance on the controller nodes should be run under numctl.10:54
rogpeppe// This should be set if the controller will run on NUMA hardware.10:54
anastasiamacrogpeppe: \o/ sounds perfect :D10:56
dooferladdimitern, frobware: got it:11:00
dooferlad77427b4d-a1e4-4659-8b34-fc17ed10ac2b machine-0: 2016-09-29 10:10:35 DEBUG juju.apiserver request_notifier.go:140 -> [3C2] machine-0 694.756636ms {"request-id":107,"response":"'body redacted'"} Provisioner[""].PrepareContainerInterfaceInfo11:00
dooferlade6b42b7c-5cb7-47e2-8e4d-27040bdc810b machine-0: 2016-09-29 10:10:35 WARNING juju.provisioner lxd-broker.go:62 failed to prepare container "0/lxd/0" network config: creating device interface: ServerError: 400 BAD REQUEST ({"vlan": ["This field is required."]})11:00
dooferladmissing vlan field11:00
frobwaredooferlad: what does the setup look like to get into this state?11:03
dooferladfrobware: could you be more specific?11:03
frobwaredooferlad: :) MAAS setup / node config / interface config on the nodes11:04
dooferladfrobware: ip addr https://www.irccloud.com/pastebin/mxvcDvoP/11:05
dooferladfrobware: sorry, ignore that11:05
dooferladfrobware: this is the right machine https://www.irccloud.com/pastebin/XrgdgkDx/11:06
dooferladthat machine has 3 NICs with eth1 assigned an address11:07
dooferladno VLAN tags11:07
dooferladI really don't want to see if it works if I change the NIC with an address to eth0. It would make me throw things if that was it.11:07
dooferladfrobware: I need to go make lunch for older daughter. Back in a few minutes.11:09
perrito666axw: still here?11:16
rogpeppea large-but-mechanical change to use consistent spellings for API and NUMA throughout Juju. review appreciated, thanks. https://github.com/juju/juju/pull/635311:22
hoenirhttps://bugs.launchpad.net/juju-core/+bug/1173122/+index?ss=111:34
mupBug #1173122: API server should not log passwords <logging> <security> <tech-debt> <juju-core:Triaged> <https://launchpad.net/bugs/1173122>11:34
hoenirwhy log passwords? and why not review the patch sended? (check diff).11:34
hoeniranyone?11:47
rick_h_hoenir: sorry, I've no idea why this never got looked at. It's from over 2 years ago on the juju-core release in maintenance mode because we've focused on juju 2.0 in launchpad.net/juju and since the code's moved to github we've not looked at code reviews in LP for some time11:49
hoenirrick_h_, thanks for making it clear.11:50
hoenirrick_h_, I makes me wander... why we keep them if their are invalid? Why not close all bugs/PR's on bzr/lanchpad and add into the project desction smth like "Juju dosen't recive any more commits on launchpad, we moved to github" ?11:54
rick_h_hoenir: all I can say is that I've never even looked to be honest.11:55
rick_h_hoenir: there's no reason not to11:55
frobwaredooferlad: sorry, was sidetracked. will look in a bit.11:56
frobwarerick_h_: we have not landed the patch (yet!). We ran into an issue with aliases when testing this morning.11:57
hoenirI think we will make our lives better if we will take some time cleaning the project issues/bugs... It's kind of a mess.11:57
rick_h_frobware: k, please keep us up to speed on that.11:58
frobwarerick_h_: we should sync (now?)11:58
rick_h_frobware: k, meet you in standup11:58
frobwarerick_h_: omw11:58
anastasiamachoenir: "project issues/bugs".. are u refereing to launchpad?12:04
hoeniranastasiamac, yeah12:06
frobwaredooferlad, babbageclunk, voidspace, macgreagoir: PTAL @ https://github.com/juju/juju/pull/6342 and rubber stamp an approval if you're happy with it.12:07
hoeniranastasiamac,  lately the review system is chaged or I'm undertanding it wrong? We still use http://reviews.vapour.ws or the github one?12:07
anastasiamachoenir: we r experimenting with doing reviews on github12:08
anastasiamachoenir: ur patch in launchpad, is it possible for u to re-propose agaisnt github?12:09
Andrew_jediHello guys, I was wondering whether this bug fix was included in the openstack oslo messaging charms for Liberty? https://bugs.launchpad.net/oslo.service/+bug/152490712:09
mupBug #1524907: [SRU] Race condition in SIGTERM signal handler <sts> <sts-sru> <Ubuntu Cloud Archive:Fix Released> <Ubuntu Cloud Archive liberty:In Progress by hopem> <oslo.service:Fix Released> <python-oslo.service (Ubuntu):Fix Released> <python-oslo.service (Ubuntu Wily):Won't Fix>12:09
mup<python-oslo.service (Ubuntu Xenial):Fix Released> <python-oslo.service (Ubuntu Yakkety):Fix Released> <https://launchpad.net/bugs/1524907>12:09
anastasiamacAndrew_jedi: this is mostly juju-dev channel, u might get more info in #juju - where charmers are \o/12:11
Andrew_jedianastasiamac: thanks!12:12
hoeniranastasiamac, could you reformulate the last question again?12:12
hoeniranastasiamac, the patch in the launchpad It's not mine.12:13
anastasiamachoenir: oh k.. i wonder if we have adressed it in codebase in github already12:13
rick_h_dooferlad: are you still reviewing the bridge changes pr?12:15
hoenirYeah that's why I'm suggesting to clean up a bit the project tracker issue.12:15
rick_h_mgz: heads up, might be a coulpe of min late to our call this morning. Gotten sidetracked this morning12:16
mgzrick_h_: no problem12:17
jambabbageclunk: I was playing around with your tests again, and if I run with '-v' I do see some errors here and there about "http: TLS handshake error from..." I wonder if the mongo driver is assuming ssl?12:29
jamand periodically it is trying to poll the replicaset but failing because of TLS stuff, which isn't failing the primary test12:29
babbageclunkjam: I think we see that occasionally in normal test runs anyway? Maybe.12:30
jambabbageclunk: it certainly sounds like something is genuinely wrong and we just haven't been noticing.12:30
jamI'll try running with pure trunk and see if I still see it12:30
babbageclunkjam: Yeah, could be.12:31
rogpeppejam: have you got a final opinion on https://github.com/juju/juju/pull/6345 (your review didn't approve or not-approve) ?12:31
jamrogpeppe: I haven't seen your response yet, will try and look again12:31
rogpeppejam: ta12:32
rick_h_mgz: omw12:35
mgzrick_h_: I await12:35
jamrogpeppe: I had some comments, but I think LGTM12:41
rogpeppejam: thanks. i responded to "I would guess the minimum is still how long it takes for DNS to update."12:45
rogpeppejam: it's independent of DNS updates12:45
jamrogpeppe: so its not entirely independent. if you run 2 controllers at the same IP address, then sure, they both get a signed cert.12:46
jambut if you are doing "juju bootstrap" you then need to go update your IP record12:46
jamand then it has to propagate12:46
jamand *then* you can get a signed cert.12:46
rogpeppejam: you do, but that's independent of letsencrypt12:46
rogpeppejam: ideally I think we'd provide a way for Juju to automatically update its own DNS record12:47
rogpeppejam: and then of course it would take a while for the record in the IP address to propagate12:47
rogpeppejam: one other thing, kinda trivial: I've been wanting to standardise on Api vs API spelling for ages (we're inconsistent all over), and this PR does that. Do you think it's a good idea? https://github.com/juju/juju/pull/635312:48
jamrogpeppe: you didn't address Http while you were at it?12:49
rick_h_rogpeppe: jam my one concern would be if things like deployer/etc break due to api changes to them ^12:49
jamI'm +1 on the general concept of preferring API12:49
rogpeppejam: good point, that would be good too12:49
rogpepperick_h_: and +1 on that12:49
rogpepperick_h_: i'll double check we're not changing any API calls12:50
jamthe only thing I worry about with a global search/replace is things that we've exposed as part of a public interface.12:50
jamas rick_h_ pointed out as well.12:50
rogpeppejam: looks like no API calls are affected12:52
rogpeppejam: or by Http vs HTTP either, which I'll do too12:54
rick_h_rogpeppe: <3 ty12:54
jambabbageclunk: looks like you're right, the TLS stuff is pre-existing.13:15
jamI wonder if something is trying to update to an old address that got torn down13:15
jam(my quick hack of apiserver.go to drop TLS shows about 10 tests that actually need ssl cause they are testing that we expose SSL endpoints, and 54s vs 58s runtime)13:16
jamso another 10%-ish, but quite a bit more actual work to get all of those to actually run.13:16
jamrogpeppe: looks like you missed a legacy_test.go http://juju-ci.vapour.ws:8080/job/github-merge-juju/9366/artifact/artifacts/trusty-err.log13:33
mupBug #1173122 changed: API server should not log passwords <logging> <security> <tech-debt> <juju-core:Fix Released> <https://launchpad.net/bugs/1173122>13:34
rogpeppejam: darn, i thought i'd run all the tests locally13:38
rogpeppejam: ah, it's not surprising - that code has only just landed in master13:39
* rogpeppe should've rebased before $$merge$$13:40
rogpeppejam: BTW I changed https://github.com/juju/juju/pull/6353 to do all of: API, NUMA, HTTPS, HTTP, FTP and URL.13:41
jamrogpeppe: so I'm +1 on theory, but I'd like someone to really go through it carefully to watch out for any actual public API changes.13:43
rogpeppejam: so I did a grep for all those names inside apiserver/... and there are no methods that have changed name; and there are no fields in apiserver/params without JSON annotations that have changed name either13:44
rick_h_frobware: dimitern ty for getting that in and landed! please enjoy the rest of the sprint!13:44
dimiternrick_h_: thank you ;) I'm glad it landed13:45
rogpeppejam: i'd really like it if we had an automatic API compatibility checking tool that checked for any type and name changes. obviously it wouldn't be sufficient, but it would be a good sanity check to have.13:46
rogpeppejam: i started doing one when we had "friday labs" but no time now13:46
dimiternoh the good ol' days..13:49
dimitern:D13:49
rogpeppejam: here are all the differences that are inside apiserver/... that aren't in _test.go files: http://paste.ubuntu.com/23251153/13:49
rogpeppedimitern: :)13:49
mupBug #1173122 opened: API server should not log passwords <logging> <security> <tech-debt> <juju-core:Fix Released> <https://launchpad.net/bugs/1173122>13:49
mupBug #1173122 changed: API server should not log passwords <logging> <security> <tech-debt> <juju-core:Fix Released> <https://launchpad.net/bugs/1173122>13:55
natefinchrick_h_, voidspace, dooferlad, macgreagoir: ping for standup14:02
voidspacenatefinch: omw :-)14:02
voidspacenatefinch: thanks14:02
anastasiamacnatefinch: macgreagoir is at sprint :) r u just missing Mick or do u need to update him? :D14:03
macgreagoirnatefinch: Aye, sorry, I'm sprinting :-)14:04
natefinchmacgreagoir: forgot, nevermind :)14:05
babbageclunkjam: Sorry, have you already done the no-ssl-api-connection hack?14:16
jambabbageclunk: only in one package and a very dirty dirty hack without making all the tests work.14:16
jamwith failing tests its hard to say if its faster because of the SSL or because of the tests stopping early14:17
babbageclunkjam: ah, cool - I'm finding it pretty slow going/very whack-a-mole14:17
jamI just did "apiserver/" itself.14:17
jamI have the feeling that the difficulty you're seeing in implementing is enough of an answer in itself.14:18
jamnot worth it for the time to whack all the moles14:18
babbageclunkjam: yeah, I'm thinking that too14:18
jambabbageclunk: because to do it *cleanly* you still have to whack those moles14:19
frankbannatefinch, axw: I have two branch up for review when you have time: https://github.com/juju/juju/pull/6352 and https://github.com/juju/juju/pull/6354 thanks!14:39
dimiternmacgreagoir, anastasiamac: PTAL https://github.com/juju/juju/pull/6355 - small time-related state improvements14:39
natefinchfrankban: will take a look14:40
frankbannatefinch: ty!14:41
anastasiamacdimitern: lgtm14:45
rick_h_natefinch: ping, sorry forgot to stay on14:45
rick_h_natefinch: can you meet me back in the standup?14:45
anastasiamacjam: tell user when operations are blocked \o/ https://github.com/juju/juju/pull/635614:46
natefinchrick_h_: sure14:46
rick_h_dooferlad: re: https://bugs.launchpad.net/juju/+bug/1623480 did you run frobware's test on azure and that's corrected?14:58
mupBug #1623480: Cannot resolve own hostname in LXD container <lxd> <network> <juju:Triaged by rharding> <https://launchpad.net/bugs/1623480>14:58
dooferladrick_h_: no.14:59
dooferladrick_h_: probably should!14:59
katco`rick_h_: hey what was the consensus on the new charm URL format. is the change from cs:<series>/<charm>-<rev> to cs:<series>/<charm>/<rev> intended?15:00
rick_h_dooferlad: ty15:00
rick_h_katco`: consensus was "watch out! it's a swamp!"15:00
katco`lol15:00
katco`rick_h_: but i'm OK to correct unit tests to new format? i'm not just fixing by coincidence?15:01
rick_h_katco`: I've got a reply on the latest there I need to go process.15:01
rick_h_katco`: i'm not sure tbh15:01
katco`rick_h_: ok, well lmk; no rush15:01
rick_h_katco`: the quick test is can you use that url in trunk atm?15:01
katco`rick_h_: looks like it locates it at least15:02
rick_h_katco`: ok, I know not all the url space works so I think it might be in an in-between land15:03
rick_h_katco`: so we have to run with what we've got as to get it all working seems like it's beyond the 2.0 scope atm15:03
katco`rick_h_: although it's cs:<charm>/<series>/<rev>15:03
rick_h_katco`: right, a lovely mongrel15:03
rick_h_although actually that's good charm/series/rev, wonder if charm/rev works15:04
* rick_h_ needs to update trunk and tinker15:04
katco`rick_h_: doesn't look like it15:04
katco`rick_h_: if you're going to specify a rev, looks like you need a series. which doesn't make any sense with multi-series charms15:05
rick_h_katco`: right :/15:05
dooferladfrobware, dimitern: found it https://bugs.launchpad.net/juju/+bug/162897315:25
mupBug #1628973: maas provider uses 'primary interface' logic - allocateContainerAddresses1 fails when interface 0 doesn't have a VLAN <juju:Confirmed> <https://launchpad.net/bugs/1628973>15:25
dimiterndooferlad: looking15:25
dimiterndooferlad: there is in fact such a thing as primary nic for a device15:27
dimiterndooferlad: it's the only one maas creates along with the device when you pass it a mac address15:27
dooferladdimitern, frobware: that whole function is suspect; looking at a subnet to find what VLAN an interface should use? You can have the same subnet used by two different VLANs - kind of the point.15:27
dooferladdimitern: no, there isn't.15:28
dooferladdimitern: that logic uses 'primary' to mean 'first in the list'15:28
dimiterndooferlad: no, there is ever only one vlan for any subnet in maas15:28
dimiternin fact subnets are contained within a vlan15:28
dooferladdimitern: that is a current MAAS limitation, not a network limitation.15:28
rogpeppedimitern: BTW that doc comment on NowToTheSecond is misleading - mongo stores time in millisecond resolution, not second resolution15:28
dimiternand I'm talking about the maas vlan db entity, not a VLAN with tag 123415:29
dimiterndooferlad: I agree it's maas specific, but that's what the code is handling there15:29
dimiterndooferlad: ok the name might be changed to "first" vs "primary"15:30
dooferladdimitern: I suggest comments. They are useful :-)15:30
dimiternmaas db triggers ensure every device has at least 1 NIC, but it doesn't link it to the correct vlan (i.e. uses the "default vlan" for it)15:31
dooferladthe problem with that statement is that 'default vlan' could be the absence of a VLAN in terms of the network, right?15:32
dimiterndooferlad: I'm not saying they aren't useful ;) I'm thankful15:32
dimiterndooferlad: it's a vlan used when maas cannot figure out which vlan to use - it's in fact hardcoded in maas src with id=500115:32
dimiternhorrible, horrible stuff15:33
dooferladyikes!15:33
dooferladdo you know what happens if we don't specify the VLAN field in the API request? It seems like that is the right thing to do in this case.15:33
dooferladI need to look at the API docs (and hope they are correct)15:34
dimiternI know yes - it fails, as for a physical nic vlan is required15:34
dimiternand all device nics are physical15:34
dimiternthat chunk of code tries to satisfy the requirement15:35
dooferladoh my15:35
dooferladI think we need to just use id=5001 when it isn't set then :-(15:36
dimiternyou can easily check: maas <profile> devices create parent=xyz mac_addresses=aa:bb:cc:dd:ee:f0 ; maas <profile> interfaces create-physical <dev-id> [no vlan=]15:36
dimiternmight be, but if it happens not to match the vlan used by the host bridge, you'll get issues15:37
dimiternfortunately, figuring out which subnet the host bridge is on (and the vlan of that subnet) is easier now, since we only bridge host interfaces with ip addresses15:40
dimiternit was much harder previously, as the host bridge might be without address (hence subnet)15:41
dooferladdimitern: in allocateContainerAddresses1 that isn't the case. On my hardware br-eth0 and br-eth2 don't have addresses, so we shouldn't be trying to get lxd/0 eth0 or eth2 addresses. That code even uses the horror of if nic.InterfaceName == "eth0".15:47
dooferladit will fail if an interface called eth0 isn't configured to use as a set of defaults.15:47
dimiterndooferlad: are you using tip of master? how come those address-less br-* get created?15:48
dooferladdimitern: yes, using the tip of master15:48
dooferladdimitern: as of this morning that is.15:49
dimiterndooferlad: yeah, well the bridge-only-configured (master-lp1627073 ?) PR landed, so now you shouldn't be seeing such bridges anymore15:50
dooferladdimitern: so your changes will have stopped br-eth0 from turning up, but this code still looks for an eth0 (from MAAS?)15:50
dooferladso it doesn't make any difference15:50
dooferlad...maybe15:50
dimiterndooferlad: eth0 is inside the device15:50
dooferladwill it be linked to br-eth0?15:50
dimiterndooferlad: it will be linked to the first bridge on the host15:50
dimiterndooferlad: which has an address - might be br-ens5 or br-eth0 (depends on the host node network config)15:51
dooferladso, it will be linked to br-eth1 in this case. OK.15:51
dimiternyeah15:51
dimiterndooferlad: also maas names the first device nic "eth0", fwiw15:52
dooferladdimitern: but first != special15:52
dooferladdimitern: and untagged is a magic API value15:52
dimiterndooferlad: it kinda is, because it's always there15:52
dooferlad(5001)15:52
dimiterndooferlad: so we can't create it, just find it and update it (if needed)15:53
dimiternmaas auto-creates it using the mac address passed to create device15:53
dooferladso after the message "NIC %v has no subnet - setting to manual and using untagged VLAN", shouldn't we assign the value "5001" to match the API?15:53
dimiterndooferlad: I think that's just dead code after that PR landed :/15:54
dimiterndooferlad: since it will have a subnet always15:54
dooferladdimitern: I am all for deleting it and replacing it with an error!15:54
dimiterndooferlad: if you're willing to go there, please do - I'll happily review the changes15:55
dimitern(now rc2's been cut off anyway)15:55
dooferladdimitern: I think we need to add a cleanup card to schedule the work. That function is over 100 lines long without any comments and I think has bigger problems than the dead code we just identified.15:58
katco`p15:58
dimiterndooferlad: sgtm15:58
=== frankban is now known as frankban|afk
dooferladdimitern: confirmed the problem has gone with your latest changes :-)16:32
dimiterndooferlad: \o/ !! :)16:32
natefinchrick_h_: so, authentication with apikey is going to be more difficult than I thought.  It's special for rackspace, not using the openstack standards, and since our rackspace code just reuses all that... it would take some work to extract the authentication code into something rackspace can override17:01
rick_h_natefinch: yea, that's what I was thinking17:03
rick_h_natefinch: so it'd be good to drop any notes into the bug, rename it to "add support for api-key auth to the rackspace provider" and 2.1 it17:03
natefinchrick_h_: yep17:03
natefinchrick_h_: it's unfortunate... I can change the names we call them, but then deep in the goose code, it has hard coded expectations of what the values will be called, and those aren't what rackspace expects.17:04
rick_h_natefinch: yea, that's what I expected17:04
natefinchrick_h_: oh well.  I can disable access-key type authentication easily enougj17:05
rick_h_natefinch: ty17:06
natefinchSimple review anyone? +25 -17: https://github.com/juju/juju/pull/635717:51
natefinchrick_h_: ^ "fixed" :)17:53
rick_h_natefinch: ty17:53
hatchwhen connecting to a controller via the API the user-info returns two fields 'controller-access' and 'model-access' - why would these values differ from what is returned via the cli for the same user?18:00
hatchthey are both "" when the cli reports that it should have addModel18:01
rick_h_natefinch: so when you add credential there's no option right?18:01
rick_h_natefinch: as far as which type to use?18:01
hatchrick_h_: via the api you can specify for certain clouds18:02
natefinchrick_h_: correct.  it just says 'Using auth-type "userpass"'18:02
natefinch$ juju add-credential rackspace18:02
natefinchEnter credential name: bar18:02
natefinchUsing auth-type "userpass".18:02
natefinchEnter username: foo18:02
natefinchEnter password:18:02
natefinchEnter tenant-name: a18:02
natefinchCredentials added for cloud rackspace.18:02
rick_h_natefinch: <3 ty18:03
natefinchrick_h_: d-(^_^)-b18:04
hatchok let me rephrase my question, when logging into the controller should the acl data be returned in the response, or do I need to make another request for the data?18:07
rick_h_perrito666: ^18:08
katco`https://github.com/juju/juju/pull/6348 could use another look. warning: scripted renames ahead18:12
rick_h_natefinch: katco` can you all swap for review then please?18:13
natefinchyep yep18:16
katco`yep18:19
natefinchkatco`: is this correct? cs:~a-user/trusty/spam-5  is now   cs:trusty/spam/~a-user/5  ?18:20
katco`natefinch: i think i got the bargain in this transaction: lgtm18:21
natefinchkatco`: lol I was gonna say.... :)18:21
katco`natefinch: sigh no that's not correct18:21
natefinchkatco`: well, at least that's fairly easy to find with a regex18:23
katco`yeah18:23
katco`i have a feeling i probably missed some cases too18:23
natefinchI presume /2 instead of -2 is the new correct way?18:24
katco`yeah18:24
katco`read commit message18:24
natefinchahh yeah, missed that, but figured it from context18:25
katco`i actually don't know how it's supposed to look for a user url18:25
natefinchI presume ~user/charmname/series/revision ..... anything else would be kinda wacky18:26
natefinch(which, yes, means the old way was wacky ;)18:26
katco`natefinch: cs:charm/series/rev means is kinda wacky for multiseries charms too18:26
natefinchkatco`: very true18:28
natefinchkatco`: if I ran the zoo, all charms would be multiseries... it's silly to have different charms when the charm format is cross platform.18:34
perrito666hatch: to be honest, I dont remember18:45
katco`what was the consensus on squashing commits with reviews?18:48
natefinchkatco`: do it, I believe was the consensus18:48
katco`natefinch: squashed with fix for user urls18:49
hatchperrito666: heh ok18:51
rick_h_perrito666: can you please help hatch out with the gui using ACL bits. There's a possiblity of a bug that we're about to head out in rc2 and I want to make sure it's ok18:56
perrito666rick_h_: sure18:57
hatchthanks all18:58
hatchyou rock18:58
perrito666yes we do18:58
natefinchkatco`: I like the way you show the regex you used as if I can somehow tell if it's right or wrong ;)19:02
katco`natefinch: lol, well it's mainly there in case we need it again or it's screwed something up :)19:04
katco`natefinch: just good to pair the command used with the commit19:04
natefinchkatco`: lgtm.  I have a quibble with the logging, but meh, it's not that important19:04
katco`natefinch: yeah i am considering your comments19:04
katco`natefinch: i usually have a rather black and white view on the matter; i.e. it's not the place of a function to determine what happens with its errors. i'm weighing your opinion trying to decide if it's not always so19:05
katco`natefinch: you are correct in that we only ever log it; the case i consider is the future. it's much easier to change things on the edges, and when the handling is done on the bottom-edge, it couples all callers together with their handling19:06
natefinchwell, either senderror has to handle its own errors, or the http handler function has to handle their own errrors19:06
katco`natefinch: right, and i always prefer the top-edge (i.e. http handler)19:07
katco`pushing decisions down into your tree makes code rigid, but easier to read/use19:07
natefinchI guess I see senderror as just a shortcut instead of copying and pasting that whole thing into every http handler19:07
katco`natefinch: i believe sometimes it goes through other functions before making its way up to the http handler?19:08
katco`natefinch: at any rate, i am still mulling over your comments. i'm going to $$merge$$ so that the tests i undoubtedly missed fail and take a late lunch19:09
natefinchheh good idea19:09
natefinchif the merge goes through, like I said, it's mostly a philosophical difference.  It's obviously not incorrect as it is.19:10
natefinchkatco`: btw, it looks like sendError *used* to get called from lower in helper functions, and I agree that would be bad... but now those helper functions (correctly) just return errors, so sendError is only ever called from the top level http handler functions.19:11
katco`natefinch: ah19:12
katco`natefinch: i have been told to wait on lunch in case my wife needs help stuffing our cat into the crate for the vet19:12
natefinchhaha, I know how that goes19:13
natefinchone of our cats is like "No!!! DEATH FIRST!"  And the other one is like "hey, what's this, a nice little place to curl up? don't mind if I do"19:13
katco`lol yep. we have those same 2 cats19:14
=== frankban|afk is now known as frankban
balloonsredir, you about/19:33
=== frankban is now known as frankban|afk
alexisbballoons, nope he wont be19:40
alexisbballoons, something I can help you with?19:40
alexisbperrito666, ping19:40
balloonsalexisb, just looking for someone to +1 this so we can merge it ;-) https://github.com/juju/juju/pull/635819:41
natefinchI'm on call reviewer19:41
balloonsahh, I misread.. howdy natefinch19:41
balloonsyou've been through the drill before19:42
natefinchballoons: lgtm19:42
alexisbnatefinch, beat me to it :)19:42
perrito666alexisb: pong (sorry was having afternoon tea, yes we do that here)19:45
alexisb:) np19:46
=== mup_ is now known as mup
=== mup_ is now known as mup
redirballoons: all set?19:59
balloonsredir, yeppers!19:59
* redir nods19:59
natefinchanyone remember if facades are supposed to start at version 0 or 1?20:09
=== mup_ is now known as mup
=== mup_ is now known as mup
=== mup_ is now known as mup
=== mup_ is now known as mup
katco`natefinch: 120:39
katco`natefinch: bc 0 is the default value and is equivalent to forgetting about the version20:39
veebersHas this been discussed before? During a bootstrap when an 'apt-get' fails juju continues to then try install some packages and doesn't timeout at all.20:40
=== mup_ is now known as mup
=== mup_ is now known as mup
perrito666axw: ?22:00
=== mup_ is now known as mup
=== mup_ is now known as mup
=== mup_ is now known as mup
thumpermorning22:09
katco`heya thumper22:14
axwperrito666: awake now, but I'm heading into a meeting shortly22:15
=== mup_ is now known as mup
=== mup_ is now known as mup
=== mup_ is now known as mup

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!