| wallyworld | babbageclunk: i also see all the relation joined log messages, so the worker is doing its job | 00:04 |
|---|---|---|
| wallyworld | just no hooks firing | 00:04 |
| veebers | babbageclunk, wallyworld would either of you have a moment to help me with an error i'm seeing when deploying to GCE? | 00:25 |
| veebers | I see an error re: no lxd socket, that seems odd right? (http://pastebin.ubuntu.com/23749591/) | 00:26 |
| wallyworld | veebers: looks like lxd is not installed. i blame the image | 00:30 |
| veebers | wallyworld: ah ok, who could i ping about that? sinzui would you have insight?^^ :-) | 00:31 |
| wallyworld | veebers: although, it could be that lxd init has not been run; i can't recall the workflow there as to when that gets done. i assume juju works with lxd for other clouds | 00:33 |
| wallyworld | there has been a couple of lxd commits lately but nothing i know of that would break this way | 00:33 |
| sinzui | veebers: xenial always has lxd and juju does init it when it is needed I have not seen fail before. But the "lxdbr0" addresses: route ip+net: no such network interface" could happen under new juju rules where lxdbr0 will not be created until the user deploys a container to the host | 00:34 |
| veebers | sinzui: I see this happening in my perfscale test, just after a bootstrap but before I deploy any bundle/charm (well I it happened during a deploy once, but every other time before a deploy) | 00:39 |
| veebers | sinzui: I don't think I'm doing something I shouldn't, unless I'm missing something? | 00:39 |
| veebers | wallyworld: I see this is syslog, does that point to the image like you suggested? juju-aa2226-0 google: No startup script found in metadata. | 00:40 |
| wallyworld | veebers: not sure | 00:44 |
| wallyworld | veebers: it could be that cloud init script is not being run, but it would need debugging to dig into it. it could just be a harmless warning also | 00:45 |
| veebers | wallyworld: hmm ok, any suggestion on debugging this further? | 00:58 |
| wallyworld | veebers: not offhand, it's just a matter of turning on debug and trawling through the logs. if juju works elsewhere with the same setup it's likely to be gce specific | 01:02 |
| wallyworld | i've not come across the issue before so don't have any suggestions off the top of my head | 01:02 |
| veebers | wallyworld: sweet thanks, I'll see what I can dig up | 01:04 |
| === tasdomas` is now known as tasdomas | ||
| wallyworld | babbageclunk: found the issue. it's a problem with the remote app name. i have to duck out for 15 minutes but will fix. i need to figure out how to fix as well | 01:15 |
| === chaology is now known as Guest71584 | ||
| babbageclunk | wallyworld: ok, good stuff | 01:25 |
| wallyworld | babbageclunk: once fixed, i'll ping you, and also need to discuss a proper fix as we will need to extend the data model | 01:25 |
| babbageclunk | wallyworld: ok | 01:31 |
| wallyworld | babbageclunk: yay fixed, will write tests but after i duck out for 15 minutes | 01:37 |
| babbageclunk | wallyworld: awesome | 01:56 |
| mup | Bug #1484105 opened: juju upgrade-charm returns ERROR state changing too quickly; try again soon <bug-squad> <canonical-is> <upgrade-charm> <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1484105> | 02:02 |
| blahdeblah | Anyone around who can advise on the best course of action with this bug? ^ | 02:04 |
| blahdeblah | It's affecting a production site, and juju has basically stopped in the middle of a charm upgrade, with roughly half the units upgraded, and half not. | 02:05 |
| blahdeblah | even all-machines.log is becalmed, with only one unit logging this repeatedly: https://pastebin.canonical.com/175051/ | 02:05 |
| blahdeblah | (and not in all-machines.log, but in its own unit log) | 02:06 |
| mup | Bug #1484105 changed: juju upgrade-charm returns ERROR state changing too quickly; try again soon <bug-squad> <canonical-is> <upgrade-charm> <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1484105> | 02:08 |
| mup | Bug #1484105 opened: juju upgrade-charm returns ERROR state changing too quickly; try again soon <bug-squad> <canonical-is> <upgrade-charm> <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1484105> | 02:11 |
| blahdeblah | I've done a full restart of machine 0, and am part way through a full restart of every juju agent | 02:24 |
| blahdeblah | Is this perhaps a case when we might want to try the mongodb cleanup? | 02:25 |
| wallyworld | babbageclunk: https://github.com/juju/juju/pull/6773 i am retesting a second time now from scratch | 02:35 |
| wallyworld | blahdeblah: what version of juju is this? | 02:37 |
| wallyworld | the bug talks about 1.18 | 02:37 |
| wallyworld | surely not | 02:37 |
| babbageclunk | wallyworld: I don't quite understand the names you're using - what will the remoteApplicationName and -Alias be? | 02:45 |
| wallyworld | babbageclunk: the name is what is is on the offering side ie the actual name. the alias is what it is known as locally in the consuming model | 02:46 |
| babbageclunk | wallyworld: Would localAlias be a better name, since it's the local name for the remote application? | 02:46 |
| wallyworld | sure | 02:46 |
| babbageclunk | wallyworld: So the aliasing would be done at consume or relate time - you'd say how to refer to the remote application in the consuming model. Is that right? | 02:48 |
| wallyworld | yes | 02:48 |
| wallyworld | we need to add that | 02:48 |
| wallyworld | so for now it is hard coded to user-model-appname | 02:48 |
| babbageclunk | wallyworld: ok, got it. | 02:49 |
| babbageclunk | wallyworld: LGTM'd | 02:56 |
| wallyworld | awesome thanks | 02:56 |
| blahdeblah | wallyworld: 1.25.9 | 02:57 |
| wallyworld | blahdeblah: without digging in and doing a mongo dump and/or model dump and debugging, it's hard to say off hand what the issue would be | 02:58 |
| wallyworld | babbageclunk: looks like there's one more small issue with relation ket naming to fix (use alias vs name). i'll track that one down and land but it's tedious to debug | 03:28 |
| babbageclunk | wallyworld: ok | 03:29 |
| babbageclunk | wallyworld: let me know if you want a rubber ducky to talk it through with | 03:29 |
| wallyworld | babbageclunk: it's just tedious grunt work to find the needle in the haystack sadly | 03:34 |
| babbageclunk | wallyworld: yeah, fair enough | 03:50 |
| wallyworld | babbageclunk: i just did 2 fresh bootstraps and deployments and everything looks ok. i must have stuffed something up when testing before. sigh. so landing now | 04:22 |
| wallyworld | babbageclunk: if you were finished your work and posted the PR, I could review pending any final testing you may want to do | 04:28 |
| babbageclunk | wallyworld: The tests are done, but I haven't done the fixing up you suggested in add relation yet. | 04:29 |
| babbageclunk | wallyworld: I'll make a PR now anyway | 04:30 |
| wallyworld | which was that again? mental blank | 04:30 |
| babbageclunk | 13:30 <wallyworld> babbageclunk: the other tweak with add relation on the server side is to account for that consume could have been run first. we do check that the remote application does not exist when we go to save it, but we will need to tweak things to be a bit smarter. check early for existence, plus in the case of offers, account for that the offer may have been updated with new endpoints so a new save is ind | 04:30 |
| babbageclunk | eed required | 04:30 |
| wallyworld | babbageclunk: ah right. that can be a follow up IMO | 04:31 |
| babbageclunk | wallyworld: ok cool | 04:31 |
| wallyworld | that way i can grab what you have done and try it etc | 04:31 |
| wallyworld | behind a feature flag means we can be more lenient with what we land | 04:32 |
| wallyworld | in terms of a bit at a time with cavearts to its usage | 04:32 |
| wallyworld | babbageclunk: landed! | 04:48 |
| babbageclunk | wallyworld: here's my PR. | 04:49 |
| babbageclunk | https://github.com/juju/juju/pull/6774 | 04:50 |
| babbageclunk | wallyworld: I'll rebase it and test now. | 04:50 |
| wallyworld | ty, will look at pr | 04:50 |
| redir | g'nite | 04:53 |
| babbageclunk | redir: bye - if you lived here it'd be the weekend now! | 04:56 |
| redir | they end? | 04:57 |
| redir | have a great weekend down under and in kiwiland | 04:58 |
| wallyworld | babbageclunk: lgtm with some nitpics, tyvm. We really should introduce a proper Endpoint attr to the url struct | 05:04 |
| wallyworld | babbageclunk: i've updated the working google doc with the latest todos | 05:05 |
| babbageclunk | wallyworld: Yeah, I almost added an endpoint attr to it. | 05:06 |
| wallyworld | i was just in a rush to get shit done | 05:06 |
| === mup_ is now known as mup | ||
| babbageclunk | wallyworld: weird - I can't get mysql to start | 08:09 |
| wallyworld | as in juju deploy mysql | 08:09 |
| babbageclunk | yup - keeps dying with Fatal error: cannot allocate memory for the buffer pool | 08:09 |
| babbageclunk | It was working this morning | 08:10 |
| wallyworld | hmmm, there used to be an issue with lxc but not for ages | 08:10 |
| wallyworld | reboot? :-) | 08:10 |
| babbageclunk | yeah, I killed off lots of leaky chrome tabs, seems to have fixed it | 08:11 |
| wallyworld | i use a suspend tabs plugin with chrome | 08:12 |
| wallyworld | stops leaks. google docs is sweful for that | 08:12 |
| babbageclunk | ok, finally ran through the QA steps successfully - sorry, got interrupted by the dinner/bath/bed conveyor belt. | 08:14 |
| babbageclunk | wallyworld: is there anyway to remove a relation to a remote application? remove-relation isn't working for me. | 08:17 |
| wallyworld | babbageclunk: yeah, all that is on the todo list. remote app and remove relation for remote. i'm 1/2 way through remove app | 08:17 |
| wallyworld | thanks for the extra hours, much appreciated | 08:18 |
| babbageclunk | wallyworld: no worries! Sorry it ended up being so last minute. | 08:18 |
| wallyworld | tis ok, all par for the course | 08:18 |
| === frankban|afk is now known as frankban | ||
| frankban | hey wallyworld, is the branch from huw good enough for you? even without the additional information to be retrieved with appInfo call? | 08:24 |
| wallyworld | frankban: yes, it progress :-) i haven't tried it out though as am not sure how. but if we can display *something* initially, we can iterate | 08:25 |
| wallyworld | frankban: also found out today lightning talk got moved so we'll have a couple of days as well to get stuff tidy | 08:26 |
| frankban | wallyworld: so yes to try it out, 1) checkout that branch, say to ~/code/juju-gui/ 2) make sysdeps 3) make dist 4) juju upgrade ~/code/juju-gui/dist/jujugui-2.2.5.tar.bz2 | 08:27 |
| wallyworld | oh easy, awesome | 08:27 |
| frankban | wallyworld: sorry, 4) is juju upgrade-gui, not just upgrade | 08:27 |
| wallyworld | have to pack and plane leaves in about 9 hours and i need sleep so may need to try it out once i land | 08:28 |
| === ahasenack is now known as Guest99073 | ||
| babbageclunk | wallyworld: I still get a bad gateway error from wordpress if I try to consume it using url rather than model.application. | 08:51 |
| wallyworld | babbageclunk: hmmm, ok, i can check that, i tested the model.app method | 09:35 |
| wallyworld | i'll fix over the weekend | 09:35 |
| babbageclunk | wallyworld: I've been trying to work out what the problem is but I haven't got it yet. | 09:46 |
| wallyworld | babbageclunk: tis ok, i can fix. the main use case is model.app | 09:47 |
| wallyworld | babbageclunk: damn test sfailed | 09:47 |
| babbageclunk | Yeah - it's legit though - the one that checks all of the commands have help? | 09:49 |
| babbageclunk | Fixed now - I don't really get the point of the test though. | 09:49 |
| babbageclunk | https://github.com/juju/juju/pull/6774/commits/965d0371256dbcc2436dcbc3e7cfa231b71628f1 | 09:49 |
| babbageclunk | If you've forgotten to register a command you're very unlikely to have remembered to add it to this test. | 09:51 |
| wallyworld | babbageclunk: yeah +1 to all that | 09:51 |
| babbageclunk | wallyworld: d'oh, missed that the list was supposed to be alphabetical. | 09:52 |
| wallyworld | babbageclunk: so, i have code to delete remote apps.. but need to add undertaker stuff. i think that need sdoing before deleting relations. so maybe you could next week pick up the dump modle bug and app url endpoint? | 09:53 |
| wallyworld | and local alias work | 09:53 |
| wallyworld | i'm sure priorities wil change once sprint starts | 09:53 |
| babbageclunk | wallyworld: I'm off for Mon-Weds next week (sorry, really should have mentioned!) but I'll pick those up after. | 09:54 |
| babbageclunk | hang on - dump model bug? | 09:55 |
| wallyworld | ok, enjoy | 09:55 |
| wallyworld | yeha, if you run dump-model and there are remote relations, it fails | 09:55 |
| babbageclunk | wallyworld: ah, makes sense | 09:55 |
| wallyworld | ty again for ll the work | 09:55 |
| wallyworld | i have to go pack some mor | 09:55 |
| wallyworld | e | 09:55 |
| babbageclunk | ok, don't forget anything! | 09:56 |
| === jamespag` is now known as jamespage | ||
| === mup_ is now known as mup | ||
| mup | Bug #1654528 opened: log forwarding broke between 1.25.6 and 1.25.9 on trusty <juju-core:New> <https://launchpad.net/bugs/1654528> | 10:43 |
| === Guest99073 is now known as ahasenack | ||
| === ahasenack is now known as Guest76687 | ||
| === perrito667 is now known as perrito666 | ||
| rogpeppe1 | is anyone around that knows how rsyslog forwarding works in juju? | 11:33 |
| === Guest76687 is now known as ahasenack | ||
| rick_h | not sure, what's up? I did see a bug go by overnight about it being broken in 1.25.9 due to a change in the tls ciphers available not being allowed any more. | 12:36 |
| === freyes__ is now known as freyes | ||
| rogpeppe1 | rick_h: yeah, that's what this was about. | 13:06 |
| rogpeppe1 | rick_h: i have a better idea now after rummaging around in the code a bit. | 13:06 |
| rogpeppe1 | rick_h: this should fix it: https://github.com/juju/utils/pull/257 | 13:06 |
| rogpeppe1 | is there anyone in juju-core that might be able to review the above, please? voidspace, are you on-call reviewer by any chance? | 13:08 |
| rogpeppe1 | rick_h: there's some concern that the CI tests didn't find this problem before release. | 13:09 |
| natefinch | rogpeppe1: can you put that at the bottom of the list so we don't choose it before the ecdhe ones at the bottom? | 13:12 |
| rogpeppe1 | natefinch: ok | 13:12 |
| natefinch | rogpeppe1: otherwise lgtm | 13:12 |
| rogpeppe1 | natefinch: done | 13:13 |
| rogpeppe1 | natefinch: wanna make your mark on the PR? | 13:13 |
| Spads | oh yes hi | 13:17 |
| Spads | rick_h: natefinch: so this bug is important to our trusty 1.25 envs including ps45-jujucharms, but there's also likely some jujudb corruption that axino put in a bug earlier that we'll need to diagnose and sort out if juju is to do anything more in that env | 13:18 |
| rick_h | gotcha rogpeppe1, ty for the update. | 13:20 |
| Spads | rick_h: so can we get someone to help us diagnose the jujudb for ps45-jujucharms? | 13:24 |
| Spads | https://bugs.launchpad.net/juju-core/+bug/1484105/comments/16 <-- the meat of the matter | 13:24 |
| mup | Bug #1484105: juju upgrade-charm returns ERROR state changing too quickly; try again soon <bug-squad> <canonical-is> <upgrade-charm> <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1484105> | 13:24 |
| rick_h | Spads: k, updated the bug https://bugs.launchpad.net/juju-core/+bug/1654528 with it in progress with rogpeppe1's patch | 13:27 |
| mup | Bug #1654528: log sending broke between 1.25.6 and 1.25.9 on trusty <juju-core:In Progress by rogpeppe> <https://launchpad.net/bugs/1654528> | 13:27 |
| Spads | rick_h: sure, but we still have what we believe is jujudb corruption that needs fixing, no? | 13:28 |
| rick_h | Spads: looking at the other bug | 13:28 |
| Spads | ta | 13:28 |
| rick_h | Spads: sooo, looking at the bug and the diagnosis you all did, I don't know I have folks that know that working bits on hand. They're traveling to cape town atm :/ | 13:34 |
| Spads | yikes | 13:34 |
| rick_h | Spads: I can see if Alexis has someone coming in that might know more, but that transaction bits can be tricky and I don't want to accidently do more damage | 13:35 |
| * Spads nods | 13:35 | |
| Spads | yeah the transaction system kind of blows my mind | 13:35 |
| rick_h | Spads: what's the current impact/risk level | 13:35 |
| rick_h | Spads: e.g. is it worth playing riskier to get something done or better to play it more safe with the right heads? | 13:35 |
| Spads | so at the moment the combination of these bugs leaves me with the perception that juju for that environment is either logjammed or fragile enough that we shouldn't touch it | 13:35 |
| * rogpeppe1 lunches | 13:36 | |
| Spads | there's enough blocking communication from agents to controller etc that I'm less worried of something firing off and starting the avalanche | 13:36 |
| Spads | but i could be wrong | 13:36 |
| Spads | at the very least we should not try any changes or upgrades or anything until we have this whole mess resolved | 13:37 |
| rick_h | Spads: +1 to the last bit, the question is can it sit until we can get 'right folks' on the ground? | 13:37 |
| Spads | the weekend oncall IS staff will need to be briefed no matter what we do | 13:37 |
| Spads | rick_h: when's that? Monday? This evening? | 13:37 |
| rick_h | Spads: probably sunday evening or monday morning | 13:38 |
| Spads | we have folks in the US who can take this over for their friday working hours, but I'd imagine anyone arriving in capetown is going to have other things on their mind | 13:38 |
| Spads | yeah | 13:38 |
| Spads | so I'm tempted to say that this can wait until Sunday night/monday morning, but I also don't know enough to really be confident of that | 13:38 |
| rick_h | Spads: let me start an email thread and see who we can find and I'll get you better info | 13:38 |
| Spads | part of the problem is that event logging is wedged, so this could merely be a false perception of stasis in the system | 13:39 |
| Spads | for all I know agents are busily rm -rfing / as we type ☺ | 13:39 |
| Spads | not that I think that's likely | 13:39 |
| Spads | but the environment got opaque | 13:39 |
| rick_h | Spads: k, email away. Give me a couple of hours for folks to get up out of bed and to read it | 13:43 |
| Spads | rick_h: thanks. I'll update our RT about this topic | 13:43 |
| rogpeppe1 | natefinch: do you know what the current workflow is for proposing changes to 1.2x ? | 14:06 |
| natefinch | rogpeppe1: just pr against 1.25 branch AFAIK | 14:30 |
| rogpeppe1 | natefinch: ok, thanks | 14:30 |
| rogpeppe1 | natefinch: will do | 14:30 |
| natefinch | rogpeppe1: I presume we'll need this for 2.x as well, since we support trusty and precise there, too | 14:31 |
| rogpeppe1 | natefinch: yeah | 14:32 |
| rogpeppe1 | natefinch: i'm a bit concerned that no-one seems to have tested this stuff | 14:32 |
| natefinch | rogpeppe1: there's a large matrix of variables that contribute to it. I think the use of log forwarding on precise and trusty is probably one of those corner cases that slipped by | 14:34 |
| voidspace | rogpeppe1: sorry, missed your message | 14:34 |
| voidspace | rogpeppe1: still need a review? | 14:35 |
| natefinch | rogpeppe1: but yes, we should have more comprehensive testing. I guess log forwarding must not have CI tests | 14:35 |
| rogpeppe1 | natefinch: we should get log forwarding added to the CI tests | 14:35 |
| rogpeppe1 | voidspace: thanks, but too late :) | 14:35 |
| voidspace | rogpeppe1: ah, no - I see it's merged | 14:35 |
| rogpeppe1 | natefinch: if you didn't, could you add your actual LGTM to the issue please? | 14:35 |
| natefinch | rogpeppe1: done | 14:36 |
| rogpeppe1 | natefinch: ta | 14:36 |
| natefinch | her, evidently you can't technically "approve" a PR after it's merged | 14:36 |
| Spads | so I thought that 2.x didn't rely on the same log forwarding channel for basic status stuff | 14:36 |
| rogpeppe1 | natefinch: i guess you mean target the juju-1.25-beta1 branch? | 14:37 |
| rogpeppe1 | Spads: "basic status stuff"? | 14:37 |
| natefinch | rogpeppe1: the 1.25 branch is used to create tags the juju-* stuff is all tags | 14:38 |
| Spads | rogpeppe1: well I was led to believe that 1.x uses log forwarding for a lot more than 2.x does is all | 14:38 |
| Spads | and for more essential communication channels | 14:38 |
| Spads | I may have misunderstood | 14:38 |
| rogpeppe1 | Spads: i don't think that log forwarding was ever used for juju status | 14:38 |
| Spads | ok | 14:38 |
| rogpeppe1 | Spads: i think that the mechanisms have changed somewhat, but i don't know how | 14:39 |
| natefinch | no no, log forwarding is new in the last year and totally external to anything juju actually does | 14:39 |
| Spads | hm | 14:39 |
| Spads | interesting | 14:39 |
| rogpeppe1 | natefinch: but juju 1.25 does log forwarding, right? | 14:39 |
| natefinch | rogpeppe1: I don't actually remember | 14:39 |
| rogpeppe1 | natefinch: that is, agents send logs to the API via rsyslog | 14:40 |
| natefinch | rogpeppe1: oh, that log forwarding | 14:40 |
| rogpeppe1 | natefinch: well, i know they do 'cos i've just looked at the code :) | 14:40 |
| natefinch | sorry... there's a new feature for log forwarding external to juju | 14:40 |
| natefinch | so I may have been confused about what was broken in our entire conversation here | 14:41 |
| Spads | haha | 14:41 |
| Spads | I believe it's the gnutls that rsyslog is linked against that's at issue here | 14:41 |
| rogpeppe1 | Spads: well that's down to the instance that's running the agent, i think | 14:42 |
| rogpeppe1 | Spads: and we want that to be fairly liberal so you can use a stock precise instance | 14:42 |
| rogpeppe1 | s/instance/image/ probably | 14:42 |
| Spads | right | 14:43 |
| Spads | although really precise hasn't got many heartbeats left | 14:43 |
| rogpeppe1 | natefinch: the problem, i *think* is that the agents rely on rsyslog to send logs to the juju API server | 14:43 |
| rogpeppe1 | natefinch: and that uses the gnutls that's installed | 14:44 |
| Spads | but I appreciate it's still supported for this code | 14:44 |
| natefinch | rogpeppe1: we were working on that... I thought we had removed the reliance on rsyslog | 14:44 |
| rogpeppe1 | natefinch: and that didn't have any ciphersuites in common with the suites supported by juju | 14:44 |
| rogpeppe1 | natefinch: in 1.25? | 14:44 |
| rogpeppe1 | Spads: it's going to be hard moving away from precise when so many charms require it | 14:44 |
| natefinch | rogpeppe1: I thought so. We were having trouble with it, I thought we had switched to using go code to send the log messages. Receiving was still done via syslog, though, IIRC, which may still cause this problem, maybe? | 14:45 |
| Spads | rogpeppe1: You are not wrong! | 14:45 |
| rogpeppe1 | natefinch: yeah, either way is a problem | 14:45 |
| natefinch | rogpeppe1: ok :) | 14:46 |
| rogpeppe1 | natefinch: if receiving is done by syslog, what's with the logSinkHandler? | 14:46 |
| Spads | so part of this is that you guys wisely limited the log code to non-terrifying ciphersuites | 14:47 |
| natefinch | heh | 14:47 |
| Spads | so there's a valid perspective that rsyslog should really do the same | 14:47 |
| Spads | but then you could totally end up with precise and xenial being unable to rsyslog to each other | 14:48 |
| natefinch | or someone could update precise to use non-terrifying ciphersuites, too | 14:48 |
| rogpeppe1 | Spads: i guess the first thing to come up with is a set of suites that are a) non-terrifying and b) in common with all the platforms we care about. | 14:48 |
| natefinch | rogpeppe1: I'm not sure what the purpose of logSinkHandler is | 14:51 |
| Spads | rogpeppe1: we found at least one good one for trusty | 14:52 |
| Spads | rogpeppe1: and axino pastebinned a list for precise but i haven't cross-referenced them yet | 14:52 |
| rogpeppe1 | Spads: yeah, i think that we'll be good with the one i added | 14:53 |
| Spads | is it in the precise listing? | 14:54 |
| * Spads closes tabs trying to find the pastebin | 14:54 | |
| Spads | https://pastebin.canonical.com/175080/ <-- that's the badger | 14:54 |
| Spads | TLS_DHE_RSA_AES_256_CBC_SHA1 0x00, 0x39 SSL3.0 | 14:54 |
| Spads | ⬑ was that what we settled on? | 14:55 |
| Spads | 12:14 <axino> aka TLS_RSA_AES_256_CBC_SHA1 for gnutls | 14:57 |
| Spads | oh | 14:57 |
| Spads | TLS_RSA_AES_256_CBC_SHA1 0x00, 0x35 SSL3.0 | 14:57 |
| Spads | that one | 14:57 |
| Spads | no diffie-helman exchange | 14:57 |
| Spads | rogpeppe1: what you said about precise reminded me that there's no facility to upgrade releases in juju, and I wanted to check the status of any bugs on that even if they're wontfix, but i can't find one | 14:59 |
| rogpeppe1 | Spads: there's always a DH exchange in TLS, right? otherwise you wouldn't get a shared key AIUI. I'm presuming the DHE is another exchange for some reason. | 15:01 |
| Spads | yeah in TLS | 15:01 |
| Spads | hm | 15:01 |
| Spads | I'd have to read up | 15:01 |
| Spads | ah it's totally ephemeral vs static | 15:01 |
| Spads | but who uses static... | 15:03 |
| * rogpeppe1 doesn't know about ephemeral vs static DHE | 15:05 | |
| rogpeppe1 | natefinch: an identical PR targetting utils 1.25 branch: https://github.com/juju/utils/pull/258 | 15:05 |
| rogpeppe1 | natefinch: please rubberstamp :) | 15:05 |
| voidspace | rick_h: I lost network, trying to rejoin now | 15:08 |
| voidspace | rick_h: google doesn't seem keen on letting me join | 15:08 |
| voidspace | ah, in | 15:08 |
| rogpeppe1 | Spads: ok, now i get it. ephemeral DH is definitely the thing to do. | 15:08 |
| natefinch | rogpeppe1: LGTM | 15:11 |
| rogpeppe1 | natefinch: ta | 15:12 |
| rogpeppe1 | natefinch, Spads: this should fix https://bugs.launchpad.net/juju-core/+bug/1654528 | 15:25 |
| mup | Bug #1654528: log sending broke between 1.25.6 and 1.25.9 on trusty <juju-core:In Progress by rogpeppe> <https://launchpad.net/bugs/1654528> | 15:25 |
| rogpeppe1 | https://bugs.launchpad.net/juju-core/+bug/1654528 | 15:25 |
| natefinch | rogpeppe1: thanks for picking that up | 15:25 |
| rogpeppe1 | natefinch: np | 15:26 |
| rogpeppe1 | natefinch: any chance of a review? | 15:33 |
| rogpeppe1 | natefinch: it is trivial | 15:33 |
| natefinch | rogpeppe1: sorry, you posted the bug link twice, didn't realize there was a PR to review | 15:34 |
| rogpeppe1 | natefinch: ha ha | 15:34 |
| rogpeppe1 | natefinch: one mo :https://github.com/juju/juju/pull/6775 | 15:35 |
| natefinch | rogpeppe1: shouldn't it just be updating the utils dep? | 15:36 |
| rogpeppe1 | natefinch: it is | 15:36 |
| rogpeppe1 | natefinch: the others are... i don't know why that happens | 15:36 |
| rogpeppe1 | natefinch: but the commits are the same | 15:36 |
| rogpeppe1 | natefinch: just the dates changed | 15:36 |
| rogpeppe1 | natefinch: i really don't understand how the same commit could get two dates | 15:37 |
| natefinch | og yeah, huh, missed that | 15:37 |
| natefinch | man, who wrote this damn tool ;) | 15:37 |
| natefinch | it's git, who really understands any of it | 15:37 |
| rogpeppe1 | natefinch: git? | 15:38 |
| natefinch | I presume the timestamps are from git, right? | 15:38 |
| rogpeppe1 | natefinch: yeah, godeps uses git log -n 1 '--pretty=format:%H %ct' HEAD' | 15:39 |
| rogpeppe1 | natefinch: to print the commit date | 15:39 |
| rogpeppe1 | natefinch: i don't know how that could cause a different unix timestamp to be printed for the same commit | 15:40 |
| rogpeppe1 | natefinch: i have a suspicion that it might be caused by people changing dependencies.tsv manually | 15:41 |
| natefinch | rogpeppe1: could be. usually I edit it by hand, but just replace a whole line with a whole line generated by godeps | 15:43 |
| rogpeppe1 | natefinch: i'd recommend never editing it by hand | 15:43 |
| rogpeppe1 | natefinch: then you can't get it wrong | 15:43 |
| natefinch | rogpeppe1: is the OS problem fixed? used to be that godeps wouldn't show depenedencies in code that isn't compiled for the current OS | 15:44 |
| rogpeppe1 | natefinch: GOOS=windows godeps -t ./... | 15:45 |
| rogpeppe1 | natefinch: i need to work out a decent solution to the OS problem | 15:45 |
| rogpeppe1 | natefinch: and some time to implement it | 15:45 |
| natefinch | *nod* | 15:45 |
| natefinch | rogpeppe1: I think one run can't get all our deps, we have both linux-specific and windows-specific deps now | 15:49 |
| rogpeppe1 | natefinch: we have whole packages which are only included under linux? | 15:49 |
| rogpeppe1 | natefinch: in general i think that's a bad idea | 15:50 |
| rogpeppe1 | natefinch: but i guess i see how it can happen | 15:50 |
| natefinch | rogpeppe1: I think the lxd stuff is only imported for linux | 15:50 |
| natefinch | rogpeppe1: or at least transitive dependencies thereof | 15:51 |
| rogpeppe1 | natefinch: hmm, how is anyone meant to update godeps now? | 15:51 |
| perrito666 | brb lunch | 15:52 |
| natefinch | rogpeppe1: I usually know what is changing and so I know what needs to be updated, so I just godeps -t ./... | grep thepackage | 15:53 |
| natefinch | and then just manually replace that line in the current dependencies.tsv.... not ideal, obviously | 15:54 |
| rogpeppe1 | natefinch: i really need to fix godeps to make this work | 15:55 |
| natefinch | rogpeppe1: one easy fix could be a wrapper around godeps that just unions two runs of godeps for win/linux | 15:55 |
| rogpeppe1 | natefinch: yeah - the solution i've seen elsewhere is to provide a list of set sets and do everything on each tag set | 15:56 |
| natefinch | rogpeppe1: seems easy enough. Slower than trying to do it all in a single run, but a ton easier to implement | 15:57 |
| natefinch | and easier to know you've gotten it right | 15:58 |
| rogpeppe1 | natefinch: hmm, our old review links don't seem to work any more. http://reviews.vapour.ws/r/4396/ | 15:59 |
| natefinch | rogpeppe1: doh... | 16:00 |
| natefinch | sinzui: is the reviewboard instance supposed to be working? | 16:00 |
| sinzui | natefinch: It appeared to be up yesterday. We don't touch it | 16:00 |
| natefinch | sinzui: blah | 16:01 |
| natefinch | and this is why I'm glad we moved to github reviews | 16:01 |
| sinzui | natefinch: I see it is angry. I will take a look at the host. I suspect disk space because that affected 2 other machines this week | 16:01 |
| rogpeppe1 | natefinch: yeah, at least one of them looks pretty clearly like a manual edit: 75dd020b936fdcf67f692eef3ee3bb74e2e248ed | 16:01 |
| natefinch | rogpeppe1: we should send an email to juju-dev saying don't do that. At least run godeps and copy the entire line, I think that's a reasonable enough way until we can just run godeps -t ./... > dependencies.tsv | 16:03 |
| Spads | rogpeppe1: natefinch so that fix is much appreciated, unfortunately I need someone who understands the transactions db stuff to fix https://bugs.launchpad.net/juju-core/+bug/1484105/comments/16 before we can test it ☺ | 16:04 |
| rogpeppe1 | Spads: i understood (some of) the txn stuff once | 16:05 |
| rogpeppe1 | Spads: but i need to re-learn it every time i dive in | 16:05 |
| Spads | rogpeppe1: so my plan right now is to shut down the jujuds on the controller to prevent accidental mishap over the weekend | 16:05 |
| Spads | rogpeppe1: but if there's any read-only diagnostic work we can to to add on to junien's that may be a good thing to sort out now | 16:06 |
| natefinch | state changing too fast means the asserts in the actual db ops don't match the coded prechecks we run before running the transaction | 16:06 |
| rogpeppe1 | Spads: the txn-revno assertion is usually to assert that nothing changed in the doc | 16:06 |
| Spads | interesting | 16:06 |
| natefinch | we should make a new bug each time this happens because it's almost never the same code twice | 16:07 |
| natefinch | it's just a really uninformative error message | 16:07 |
| Spads | a pity lp won't split bugs as easily as it merges ☺ | 16:07 |
| sinzui | natefinch: reviewboard is our of disk. I am going to restatrt the host, the clear out a few G of tmp files | 16:09 |
| Spads | natefinch: rogpeppe1: so does this look like something that would sort itself out if we quiesced the controllers, or is it something dire that needs surgery? | 16:10 |
| rogpeppe1 | Spads: i don't know without looking at it in more detail | 16:10 |
| Spads | ok | 16:11 |
| perrito666 | sinzui: I am curious dont we have automatic checks for stats on our machines? | 16:11 |
| natefinch | Spads: it's almost never due to actual state changing too fast | 16:11 |
| natefinch | Spads: it's almost always a bug in the code where the prechecks in go code don't match what the mgo assertions are checking for. | 16:12 |
| rogpeppe1 | Spads: i wonder if just inserting a txn-revno:0 field into the doc might fix it | 16:12 |
| Spads | heh | 16:13 |
| natefinch | sinzui: thanks for the help | 16:13 |
| Spads | so do either of you think my plan to just shut down jujuds is a good idea? | 16:14 |
| Spads | so that we can freeze the env in time | 16:14 |
| Spads | or should we leave it running in this state | 16:14 |
| Spads | as it has been all day really | 16:14 |
| rogpeppe1 | Spads: this doesn't seem like a real db corruption issue to me | 16:14 |
| rogpeppe1 | Spads: just a bug in the juju transaction logic | 16:14 |
| natefinch | +1 | 16:14 |
| Spads | oho! | 16:15 |
| Spads | okay that's very good to hear | 16:15 |
| rogpeppe1 | Spads: i think the problem is probably that *most* places that insert a document insert a struct that has an explicit txn-revno field, so it will be inserted with the value 0 | 16:16 |
| rogpeppe1 | Spads: but for settings it inserts a map | 16:16 |
| rogpeppe1 | Spads: which won't have that field | 16:16 |
| * Spads nods | 16:17 | |
| Spads | what's the difference between a struct and a map? | 16:17 |
| Spads | they both sound like key/value data structures to me ☺ | 16:17 |
| rogpeppe1 | hmm, but this is statuses not settings | 16:17 |
| rogpeppe1 | Spads: a struct is like a C struct - it has a known set of fields | 16:18 |
| sinzui | natefinch: we don't have status checks on reviewboard or the reports site...we do on the other sites | 16:18 |
| Spads | ah okay, so like a named tuple then | 16:18 |
| sinzui | natefinch: reviewbaord is up | 16:18 |
| rogpeppe1 | natefinch, Spads: ah, i think the issue is that statusDoc doesn't have a txn-revno field | 16:19 |
| rogpeppe1 | Spads: yeah, kinda | 16:19 |
| Spads | oho | 16:19 |
| rogpeppe1 | natefinch, Spads: i'm trying to work out how it could ever work | 16:20 |
| Spads | haha | 16:20 |
| natefinch | lol | 16:20 |
| natefinch | I can't tell you how often I've looked at code and wondered that | 16:20 |
| rogpeppe1 | natefinch, Spads: in fact, it looks like the txn package always inserts a txn-revno field | 16:44 |
| balloons | can someone have a quick peek at https://github.com/juju/juju/pull/6776? | 16:44 |
| Spads | rogpeppe1: so how did it not get in? | 16:44 |
| rogpeppe1 | Spads: an excellent question | 16:44 |
| Spads | or is it trying to insert and failing and that's the error? | 16:44 |
| Spads | operation-ordering bug? | 16:44 |
| natefinch | balloons: lol, I just changed this to beta4 yesterday | 16:46 |
| natefinch | balloons: did we do what we needed to do with it at beta4? | 16:46 |
| balloons | natefinch, lol, I know.. fun isn't it? | 16:46 |
| balloons | natefinch, the release today is beta4, so we had to make the source reflect that for one commit to release it | 16:47 |
| natefinch | heh | 16:47 |
| natefinch | ok | 16:47 |
| natefinch | balloons: lgtm | 16:47 |
| mup | Bug #1646777 changed: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:Fix Released by nskaggs> <https://launchpad.net/bugs/1646777> | 17:11 |
| redir | easy review https://github.com/juju/juju/pull/6777 | 17:18 |
| redir | PTAL | 17:18 |
| natefinch | redir: ahh, so we were accidentally not building on s390x I guess? | 17:35 |
| redir | yeah | 17:35 |
| redir | :/ | 17:35 |
| xnox | oooooo =( please build on s390x | 17:35 |
| natefinch | redir: yeah, architecture restrictions don't seem to make sense in go code most of the time. | 17:35 |
| xnox | things should build there..... | 17:35 |
| xnox | natefinch, we like use juju to deploy openstack on s390x =/ | 17:36 |
| redir | If it is not supported should it build? | 17:36 |
| xnox | so it's not a toy port | 17:36 |
| xnox | redir, we contractually support it | 17:36 |
| natefinch | it's just a single file | 17:36 |
| redir | kvm? | 17:36 |
| natefinch | not the whole thing | 17:36 |
| redir | right just kvm | 17:37 |
| xnox | we support everything that works, which is lxd/local, kvm, openstack providers. | 17:37 |
| natefinch | redir means kvm doesn't work on s390x so we were avoiding compiling some stuff for kvm, but then it broke other stuff | 17:37 |
| xnox | there are no public clouds for s390x. | 17:37 |
| xnox | natefinch, qemu, kvm, nova, openstack all work on s390x. there maybe some switches that are not supported by qemu-system-s390x which may be used. | 17:38 |
| xnox | natefinch, redir - if there are any s390x specific bugs, I can look into that for you and submit merge proposals. | 17:38 |
| * xnox had to s/pci/ccw/ in some places before | 17:38 | |
| natefinch | sorry, I think I misrepresented what redir was saying. Anyway, the bug was that we were not compiling this one file on s390x and should have been | 17:38 |
| xnox | ok | 17:39 |
| redir | either this one file or the one that defines that it sin't supported | 17:39 |
| xnox | nonetheless if s390x-arch things do creep up, feel free to ping me =) | 17:39 |
| redir | will do xnox | 17:39 |
| redir | xnox: to be clear it should build on s390x | 17:49 |
| rogpeppe1 | natefinch: i just did some work on making go/build work with any build tags: https://github.com/rogpeppe/godeps/pull/5 | 18:26 |
| rogpeppe1 | natefinch: s:go/build/godeps/ | 18:26 |
| rogpeppe1 | natefinch: feel free to review if you like | 18:26 |
| mup | Bug #1646777 opened: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:In Progress by nskaggs> <https://launchpad.net/bugs/1646777> | 18:41 |
| natefinch | rogpeppe1: looking | 18:54 |
| mup | Bug #1646777 changed: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:In Progress by nskaggs> <https://launchpad.net/bugs/1646777> | 18:59 |
| mup | Bug #1646777 opened: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:In Progress by nskaggs> <https://launchpad.net/bugs/1646777> | 19:05 |
| natefinch | perrito666: when you get a chance: https://github.com/juju/juju/pull/6778 | 19:34 |
| perrito666 | natefinch: well the power on the computer with my code is down so why not now? | 19:34 |
| natefinch | perrito666: seems like your town needs to double down on its street drainage systems | 19:39 |
| perrito666 | natefinch: done | 19:42 |
| perrito666 | natefinch: that is just my neigbourhood, I live in a closed comunity surounded by other closed comunities so we have a shared draining system that uses, you guessed, my street, its not that deep never goes over 15cm | 19:43 |
| perrito666 | natefinch: and it drains as soon as rain stops, it all goes to a big wather buffer down my street | 19:43 |
| natefinch | I think your definition of a drainage system needs updating if "your street" is something that can be part of it :) | 19:44 |
| redir | closed source drainage | 19:44 |
| perrito666 | natefinch: if you lived in south america you would prefer your street to have a few cm of water instead of the possibility to have water stalling in a drainage (mosquitos) | 19:45 |
| redir | proprietary drainage | 19:45 |
| perrito666 | natefinch: also there is no sewage here so it all goes to nearby fields where its absorbed | 19:45 |
| redir | frobware: yt? | 19:54 |
| redir | I'm gonna doubt it:) | 19:54 |
| * perrito666 should really have bought that new ups this week | 19:55 | |
| perrito666 | ccccccgcbucdlrnrfidiurjltlkrlkhgdhjghrlfcbgf | 19:59 |
| natefinch | perrito666: btw, what endpoints do you think will be incorrectly handled with the code I wrote for ping? | 20:00 |
| natefinch | perrito666: I doubt giving a gopher:// URI to the vsphere provider is going to work. | 20:00 |
| perrito666 | natefinch: oh no, I just suggest that you check for that and fail early | 20:01 |
| perrito666 | anyone needs reviews? | 20:45 |
| TheMue | perrito666: already done some today, and my latest branch is reviewed and merged. :) | 20:46 |
| TheMue | perrito666: establishing some good procedures of our Canonical way of work in my new team. | 20:47 |
| perrito666 | heh, I wold prefer to do juju reviews though :p | 20:47 |
| TheMue | perrito666: none open? | 20:47 |
| perrito666 | TheMue: was looking for someone with priority since no one says anything ill default to the open ones | 20:49 |
| TheMue | perrito666: yeah, standard business has to be done too | 20:49 |
| * redir lunches | 21:13 | |
| * redir back | 21:32 | |
| === frankban is now known as frankban|afk | ||
| perrito666 | k, ill EOD because my laptop's battery is about to die | 21:39 |
| perrito666 | cheers | 21:39 |
| mup | Bug #1646777 changed: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:Fix Released by nskaggs> <https://launchpad.net/bugs/1646777> | 21:41 |
| redir | later perrito666 | 21:46 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!