wallyworldbabbageclunk: i also see all the relation joined log messages, so the worker is doing its job00:04
wallyworldjust no hooks firing00:04
veebersbabbageclunk, wallyworld would either of you have a moment to help me with an error i'm seeing when deploying to GCE?00:25
veebersI see an error re: no lxd socket, that seems odd right? (http://pastebin.ubuntu.com/23749591/)00:26
wallyworldveebers: looks like lxd is not installed. i blame the image00:30
veeberswallyworld: ah ok, who could i ping about that? sinzui would you have insight?^^ :-)00:31
wallyworldveebers: although, it could be that lxd init has not been run; i can't recall the workflow there as to when that gets done. i assume juju works with lxd for other clouds00:33
wallyworldthere has been a couple of lxd commits lately but nothing i know of that would break this way00:33
sinzuiveebers: xenial always has lxd and juju does init it when it is needed I have not seen fail before. But the "lxdbr0" addresses: route ip+net: no such network interface" could happen under new juju rules where lxdbr0 will not be created until the user deploys a container to the host00:34
veeberssinzui: I see this happening in my perfscale test, just after a bootstrap but before I deploy any bundle/charm (well I it happened during a deploy once, but every other time before a deploy)00:39
veeberssinzui: I don't think I'm doing something I shouldn't, unless I'm missing something?00:39
veeberswallyworld: I see this is syslog, does that point to the image like you suggested? juju-aa2226-0 google: No startup script found in metadata.00:40
wallyworldveebers: not sure00:44
wallyworldveebers: it could be that cloud init script is not being run, but it would need debugging to dig into it. it could just be a harmless warning also00:45
veeberswallyworld: hmm ok, any suggestion on debugging this further?00:58
wallyworldveebers: not offhand, it's just a matter of turning on debug and trawling through the logs. if juju works elsewhere with the same setup it's likely to be gce specific01:02
wallyworldi've not come across the issue before so don't have any suggestions off the top of my head01:02
veeberswallyworld: sweet thanks, I'll see what I can dig up01:04
=== tasdomas` is now known as tasdomas
wallyworldbabbageclunk: found the issue. it's a problem with the remote app name. i have to duck out for 15 minutes but will fix. i need to figure out how to fix as well01:15
=== chaology is now known as Guest71584
babbageclunkwallyworld: ok, good stuff01:25
wallyworldbabbageclunk: once fixed, i'll ping you, and also need to discuss a proper fix as we will need to extend the data model01:25
babbageclunkwallyworld: ok01:31
wallyworldbabbageclunk: yay fixed, will write tests but after i duck out for 15 minutes01:37
babbageclunkwallyworld: awesome01:56
mupBug #1484105 opened: juju upgrade-charm returns ERROR state changing too quickly; try again soon <bug-squad> <canonical-is> <upgrade-charm> <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1484105>02:02
blahdeblahAnyone around who can advise on the best course of action with this bug? ^02:04
blahdeblahIt's affecting a production site, and juju has basically stopped in the middle of a charm upgrade, with roughly half the units upgraded, and half not.02:05
blahdeblaheven all-machines.log is becalmed, with only one unit logging this repeatedly: https://pastebin.canonical.com/175051/02:05
blahdeblah(and not in all-machines.log, but in its own unit log)02:06
mupBug #1484105 changed: juju upgrade-charm returns ERROR state changing too quickly; try again soon <bug-squad> <canonical-is> <upgrade-charm> <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1484105>02:08
mupBug #1484105 opened: juju upgrade-charm returns ERROR state changing too quickly; try again soon <bug-squad> <canonical-is> <upgrade-charm> <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1484105>02:11
blahdeblahI've done a full restart of machine 0, and am part way through a full restart of every juju agent02:24
blahdeblahIs this perhaps a case when we might want to try the mongodb cleanup?02:25
wallyworldbabbageclunk: https://github.com/juju/juju/pull/6773 i am retesting a second time now from scratch02:35
wallyworldblahdeblah: what version of juju is this?02:37
wallyworldthe bug talks about 1.1802:37
wallyworldsurely not02:37
babbageclunkwallyworld: I don't quite understand the names you're using - what will the remoteApplicationName and -Alias be?02:45
wallyworldbabbageclunk: the name is what is is on the offering side ie the actual name. the alias is what it is known as locally in the consuming model02:46
babbageclunkwallyworld: Would localAlias be a better name, since it's the local name for the remote application?02:46
babbageclunkwallyworld: So the aliasing would be done at consume or relate time - you'd say how to refer to the remote application in the consuming model. Is that right?02:48
wallyworldwe need to add that02:48
wallyworldso for now it is hard coded to user-model-appname02:48
babbageclunkwallyworld: ok, got it.02:49
babbageclunkwallyworld: LGTM'd02:56
wallyworldawesome thanks02:56
blahdeblahwallyworld: 1.25.902:57
wallyworldblahdeblah: without digging in and doing a mongo dump and/or model dump and debugging, it's hard to say off hand what the issue would be02:58
wallyworldbabbageclunk: looks like there's one more small issue with relation ket naming to fix (use alias vs name). i'll track that one down and land but it's tedious to debug03:28
babbageclunkwallyworld: ok03:29
babbageclunkwallyworld: let me know if you want a rubber ducky to talk it through with03:29
wallyworldbabbageclunk: it's just tedious grunt work to find the needle in the haystack sadly03:34
babbageclunkwallyworld: yeah, fair enough03:50
wallyworldbabbageclunk: i just did 2 fresh bootstraps and deployments and everything looks ok. i must have stuffed something up when testing before. sigh. so landing now04:22
wallyworldbabbageclunk: if you were finished your work and posted the PR, I could review pending any final testing you may want to do04:28
babbageclunkwallyworld: The tests are done, but I haven't done the fixing up you suggested in add relation yet.04:29
babbageclunkwallyworld: I'll make a PR now anyway04:30
wallyworldwhich was that again? mental blank04:30
babbageclunk13:30 <wallyworld> babbageclunk: the other tweak with add relation on the server side is to account for that consume could have been run first. we do check that the remote application does not exist when we go to save it, but we will need to tweak things to be a bit smarter. check early for existence, plus in the case of offers, account for that the offer may have been updated with new endpoints so a new save is ind04:30
babbageclunkeed required04:30
wallyworldbabbageclunk: ah right. that can be a follow up IMO04:31
babbageclunkwallyworld: ok cool04:31
wallyworldthat way i can grab what you have done and try it etc04:31
wallyworldbehind a feature flag means we can be more lenient with what we land04:32
wallyworldin terms of a bit at a time with cavearts to its usage04:32
wallyworldbabbageclunk: landed!04:48
babbageclunkwallyworld: here's my PR.04:49
babbageclunkwallyworld: I'll rebase it and test now.04:50
wallyworldty, will look at pr04:50
babbageclunkredir: bye - if you lived here it'd be the weekend now!04:56
redirthey end?04:57
redirhave a great weekend down under and in kiwiland04:58
wallyworldbabbageclunk: lgtm with some nitpics, tyvm. We really should introduce  a proper Endpoint attr to the url struct05:04
wallyworldbabbageclunk: i've updated the working google doc with the latest todos05:05
babbageclunkwallyworld: Yeah, I almost added an endpoint attr to it.05:06
wallyworldi was just in a rush to get shit done05:06
=== mup_ is now known as mup
babbageclunkwallyworld: weird - I can't get mysql to start08:09
wallyworldas in juju deploy mysql08:09
babbageclunkyup - keeps dying with Fatal error: cannot allocate memory for the buffer pool08:09
babbageclunkIt was working this morning08:10
wallyworldhmmm, there used to be an issue with lxc but not for ages08:10
wallyworldreboot? :-)08:10
babbageclunkyeah, I killed off lots of leaky chrome tabs, seems to have fixed it08:11
wallyworldi use a suspend tabs plugin with chrome08:12
wallyworldstops leaks. google docs is sweful for that08:12
babbageclunkok, finally ran through the QA steps successfully - sorry, got interrupted by the dinner/bath/bed conveyor belt.08:14
babbageclunkwallyworld: is there anyway to remove a relation to a remote application? remove-relation isn't working for me.08:17
wallyworldbabbageclunk: yeah, all that is on the todo list. remote app and remove relation for remote. i'm 1/2 way through remove app08:17
wallyworldthanks for the extra hours, much appreciated08:18
babbageclunkwallyworld: no worries! Sorry it ended up being so last minute.08:18
wallyworldtis ok, all par for the course08:18
=== frankban|afk is now known as frankban
frankbanhey wallyworld, is the branch from huw good enough for you? even without the additional information to be retrieved with appInfo call?08:24
wallyworldfrankban: yes, it progress :-) i haven't tried it out though as am not sure how. but if we can display *something* initially, we can iterate08:25
wallyworldfrankban: also found out today lightning talk got moved so we'll have a couple of days as well to get stuff tidy08:26
frankbanwallyworld: so yes to try it out, 1) checkout that branch, say to ~/code/juju-gui/ 2) make sysdeps 3) make dist 4) juju upgrade ~/code/juju-gui/dist/jujugui-2.2.5.tar.bz208:27
wallyworldoh easy, awesome08:27
frankbanwallyworld: sorry, 4) is juju upgrade-gui, not just upgrade08:27
wallyworldhave to pack and plane leaves in about 9 hours and i need sleep so may need to try it out once i land08:28
=== ahasenack is now known as Guest99073
babbageclunkwallyworld: I still get a bad gateway error from wordpress if I try to consume it using url rather than model.application.08:51
wallyworldbabbageclunk: hmmm, ok, i can check that, i tested the model.app method09:35
wallyworldi'll fix over the weekend09:35
babbageclunkwallyworld: I've been trying to work out what the problem is but I haven't got it yet.09:46
wallyworldbabbageclunk: tis ok, i can fix. the main use case is model.app09:47
wallyworldbabbageclunk: damn test sfailed09:47
babbageclunkYeah - it's legit though - the one that checks all of the commands have help?09:49
babbageclunkFixed now - I don't really get the point of the test though.09:49
babbageclunkIf you've forgotten to register a command you're very unlikely to have remembered to add it to this test.09:51
wallyworldbabbageclunk: yeah +1 to all that09:51
babbageclunkwallyworld: d'oh, missed that the list was supposed to be alphabetical.09:52
wallyworldbabbageclunk: so, i have code to delete remote apps.. but need to add undertaker stuff. i think that need sdoing before deleting relations. so maybe you could next week pick up the dump modle bug and app url endpoint?09:53
wallyworldand local alias work09:53
wallyworldi'm sure priorities wil change once sprint starts09:53
babbageclunkwallyworld: I'm off for Mon-Weds next week (sorry, really should have mentioned!) but I'll pick those up after.09:54
babbageclunkhang on - dump model bug?09:55
wallyworldok, enjoy09:55
wallyworldyeha, if you run dump-model and there are remote relations, it fails09:55
babbageclunkwallyworld: ah, makes sense09:55
wallyworldty again for ll the work09:55
wallyworldi have to go pack some mor09:55
babbageclunkok, don't forget anything!09:56
=== jamespag` is now known as jamespage
=== mup_ is now known as mup
mupBug #1654528 opened: log forwarding broke between 1.25.6 and 1.25.9 on trusty <juju-core:New> <https://launchpad.net/bugs/1654528>10:43
=== Guest99073 is now known as ahasenack
=== ahasenack is now known as Guest76687
=== perrito667 is now known as perrito666
rogpeppe1is anyone around that knows how rsyslog forwarding works in juju?11:33
=== Guest76687 is now known as ahasenack
rick_hnot sure, what's up? I did see a bug go by overnight about it being broken in 1.25.9 due to a change in the tls ciphers available not being allowed any more.12:36
=== freyes__ is now known as freyes
rogpeppe1rick_h: yeah, that's what this was about.13:06
rogpeppe1rick_h: i have a better idea now after rummaging around in the code a bit.13:06
rogpeppe1rick_h: this should fix it: https://github.com/juju/utils/pull/25713:06
rogpeppe1is there anyone in juju-core that might be able to review the above, please? voidspace, are you on-call reviewer by any chance?13:08
rogpeppe1rick_h: there's some concern that the CI tests didn't find this problem before release.13:09
natefinchrogpeppe1: can you put that at the bottom of the list so we don't choose it before the ecdhe ones at the bottom?13:12
rogpeppe1natefinch: ok13:12
natefinchrogpeppe1: otherwise lgtm13:12
rogpeppe1natefinch: done13:13
rogpeppe1natefinch: wanna make your mark on the PR?13:13
Spadsoh yes hi13:17
Spadsrick_h: natefinch: so this bug is important to our trusty 1.25 envs including ps45-jujucharms, but there's also likely some jujudb corruption that axino put in a bug earlier that we'll need to diagnose and sort out if juju is to do anything more in that env13:18
rick_hgotcha rogpeppe1, ty for the update.13:20
Spadsrick_h: so can we get someone to help us diagnose the jujudb for ps45-jujucharms?13:24
Spadshttps://bugs.launchpad.net/juju-core/+bug/1484105/comments/16 <-- the meat of the matter13:24
mupBug #1484105: juju upgrade-charm returns ERROR state changing too quickly; try again soon <bug-squad> <canonical-is> <upgrade-charm> <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1484105>13:24
rick_hSpads: k, updated the bug https://bugs.launchpad.net/juju-core/+bug/1654528 with it in progress with rogpeppe1's patch13:27
mupBug #1654528: log sending broke between 1.25.6 and 1.25.9 on trusty <juju-core:In Progress by rogpeppe> <https://launchpad.net/bugs/1654528>13:27
Spadsrick_h: sure, but we still have what we believe is jujudb corruption that needs fixing, no?13:28
rick_hSpads: looking at the other bug13:28
rick_hSpads: sooo, looking at the bug and the diagnosis you all did, I don't know I have folks that know that working bits on hand. They're traveling to cape town atm :/13:34
rick_hSpads: I can see if Alexis has someone coming in that might know more, but that transaction bits can be tricky and I don't want to accidently do more damage13:35
* Spads nods13:35
Spadsyeah the transaction system kind of blows my mind13:35
rick_hSpads: what's the current impact/risk level13:35
rick_hSpads: e.g. is it worth playing riskier to get something done or better to play it more safe with the right heads?13:35
Spadsso at the moment the combination of these bugs leaves me with the perception that juju for that environment is either logjammed or fragile enough that we shouldn't touch it13:35
* rogpeppe1 lunches13:36
Spadsthere's enough blocking communication from agents to controller etc that I'm less worried of something firing off and starting the avalanche13:36
Spadsbut i could be wrong13:36
Spadsat the very least we should not try any changes or upgrades or anything until we have this whole mess resolved13:37
rick_hSpads: +1 to the last bit, the question is can it sit until we can get 'right folks' on the ground?13:37
Spadsthe weekend oncall IS staff will need to be briefed no matter what we do13:37
Spadsrick_h: when's that?  Monday?  This evening?13:37
rick_hSpads: probably sunday evening or monday morning13:38
Spadswe have folks in the US who can take this over for their friday working hours, but I'd imagine anyone arriving in capetown is going to have other things on their mind13:38
Spadsso I'm tempted to say that this can wait until Sunday night/monday morning, but I also don't know enough to really be confident of that13:38
rick_hSpads: let me start an email thread and see who we can find and I'll get you better info13:38
Spadspart of the problem is that event logging is wedged, so this could merely be a false perception of stasis in the system13:39
Spadsfor all I know agents are busily rm -rfing / as we type ☺13:39
Spadsnot that I think that's likely13:39
Spadsbut the environment got opaque13:39
rick_hSpads: k, email away. Give me a couple of hours for folks to get up out of bed and to read it13:43
Spadsrick_h: thanks.  I'll update our RT about this topic13:43
rogpeppe1natefinch: do you know what the current workflow is for proposing changes to 1.2x ?14:06
natefinchrogpeppe1: just pr against 1.25 branch AFAIK14:30
rogpeppe1natefinch: ok, thanks14:30
rogpeppe1natefinch: will do14:30
natefinchrogpeppe1: I presume we'll need this for 2.x as well, since we support trusty and precise there, too14:31
rogpeppe1natefinch: yeah14:32
rogpeppe1natefinch: i'm a bit concerned that no-one seems to have tested this stuff14:32
natefinchrogpeppe1: there's a large matrix of variables that contribute to it.  I think the use of log forwarding on precise and trusty is probably one of those corner cases that slipped by14:34
voidspacerogpeppe1: sorry, missed your message14:34
voidspacerogpeppe1: still need a review?14:35
natefinchrogpeppe1: but yes, we should have more comprehensive testing.  I guess log forwarding must not have CI tests14:35
rogpeppe1natefinch: we should get log forwarding added to the CI tests14:35
rogpeppe1voidspace: thanks, but too late :)14:35
voidspacerogpeppe1: ah, no - I see it's merged14:35
rogpeppe1natefinch: if you didn't, could you add your actual LGTM to the issue please?14:35
natefinchrogpeppe1: done14:36
rogpeppe1natefinch: ta14:36
natefinchher, evidently you can't technically "approve" a PR after it's merged14:36
Spadsso I thought that 2.x didn't rely on the same log forwarding channel for basic status stuff14:36
rogpeppe1natefinch: i guess you mean target the juju-1.25-beta1 branch?14:37
rogpeppe1Spads: "basic status stuff"?14:37
natefinchrogpeppe1: the 1.25 branch is used to create tags the juju-* stuff is all tags14:38
Spadsrogpeppe1: well I was led to believe that 1.x uses log forwarding for a lot more than 2.x does is all14:38
Spadsand for more essential communication channels14:38
SpadsI may have misunderstood14:38
rogpeppe1Spads: i don't think that log forwarding was ever used for juju status14:38
rogpeppe1Spads: i think that the mechanisms have changed somewhat, but i don't know how14:39
natefinchno no, log forwarding is new in the last year and totally external to anything juju actually does14:39
rogpeppe1natefinch: but juju 1.25 does log forwarding, right?14:39
natefinchrogpeppe1: I don't actually remember14:39
rogpeppe1natefinch: that is, agents send logs to the API via rsyslog14:40
natefinchrogpeppe1: oh, that log forwarding14:40
rogpeppe1natefinch: well, i know they do 'cos i've just looked at the code :)14:40
natefinchsorry... there's a new feature for log forwarding external to juju14:40
natefinchso I may have been confused about what was broken in our entire conversation here14:41
SpadsI believe it's the gnutls that rsyslog is linked against that's at issue here14:41
rogpeppe1Spads: well that's down to the instance that's running the agent, i think14:42
rogpeppe1Spads: and we want that to be fairly liberal so you can use a stock precise instance14:42
rogpeppe1s/instance/image/ probably14:42
Spadsalthough  really precise hasn't got many heartbeats left14:43
rogpeppe1natefinch: the problem, i *think* is that the agents rely on rsyslog to send logs to the juju API server14:43
rogpeppe1natefinch: and that uses the gnutls that's installed14:44
Spadsbut I appreciate it's still supported for this code14:44
natefinchrogpeppe1: we were working on that... I thought we had removed the reliance on rsyslog14:44
rogpeppe1natefinch: and that didn't have any ciphersuites in common with the suites supported by juju14:44
rogpeppe1natefinch: in 1.25?14:44
rogpeppe1Spads: it's going to be hard moving away from precise when so many charms require it14:44
natefinchrogpeppe1: I thought so.  We were having trouble with it, I thought we had switched to using go code to send the log messages.  Receiving was still done via syslog, though, IIRC, which may still cause this problem, maybe?14:45
Spadsrogpeppe1: You are not wrong!14:45
rogpeppe1natefinch: yeah, either way is a problem14:45
natefinchrogpeppe1: ok :)14:46
rogpeppe1natefinch: if receiving is done by syslog, what's with the logSinkHandler?14:46
Spadsso part of this is that you guys wisely limited the log code to non-terrifying ciphersuites14:47
Spadsso there's a valid perspective that rsyslog should really do the same14:47
Spadsbut then you could totally end up with precise and xenial being unable to rsyslog to each other14:48
natefinchor someone could update precise to use non-terrifying ciphersuites, too14:48
rogpeppe1Spads: i guess the first thing to come up with is a set of suites that are a) non-terrifying and b) in common with all the platforms we care about.14:48
natefinchrogpeppe1: I'm not sure what the purpose of logSinkHandler is14:51
Spadsrogpeppe1: we found at least one good one for trusty14:52
Spadsrogpeppe1: and axino pastebinned a list for precise but i haven't cross-referenced them yet14:52
rogpeppe1Spads: yeah, i think that we'll be good with the one i added14:53
Spadsis it in the precise listing?14:54
* Spads closes tabs trying to find the pastebin14:54
Spadshttps://pastebin.canonical.com/175080/ <-- that's the badger14:54
SpadsTLS_DHE_RSA_AES_256_CBC_SHA1                            0x00, 0x39      SSL3.014:54
Spads⬑ was that what we settled on?14:55
Spads12:14  <axino> aka TLS_RSA_AES_256_CBC_SHA1 for gnutls14:57
SpadsTLS_RSA_AES_256_CBC_SHA1                                0x00, 0x35      SSL3.014:57
Spadsthat one14:57
Spadsno diffie-helman exchange14:57
Spadsrogpeppe1: what you said about precise reminded me that there's no facility to upgrade releases in juju, and I wanted to check the status of any bugs on that even if they're wontfix, but i can't find one14:59
rogpeppe1Spads: there's always a DH exchange in TLS, right? otherwise you wouldn't get a shared key AIUI. I'm presuming the DHE is another exchange for some reason.15:01
Spadsyeah in TLS15:01
SpadsI'd have to read up15:01
Spadsah it's totally ephemeral vs static15:01
Spadsbut who uses static...15:03
* rogpeppe1 doesn't know about ephemeral vs static DHE15:05
rogpeppe1natefinch: an identical PR targetting utils 1.25 branch: https://github.com/juju/utils/pull/25815:05
rogpeppe1natefinch: please rubberstamp :)15:05
voidspacerick_h: I lost network, trying to rejoin now15:08
voidspacerick_h: google doesn't seem keen on letting me join15:08
voidspaceah, in15:08
rogpeppe1Spads: ok, now i get it. ephemeral DH is definitely the thing to do.15:08
natefinchrogpeppe1: LGTM15:11
rogpeppe1natefinch: ta15:12
rogpeppe1natefinch, Spads: this should fix https://bugs.launchpad.net/juju-core/+bug/165452815:25
mupBug #1654528: log sending broke between 1.25.6 and 1.25.9 on trusty <juju-core:In Progress by rogpeppe> <https://launchpad.net/bugs/1654528>15:25
natefinchrogpeppe1: thanks for picking that up15:25
rogpeppe1natefinch: np15:26
rogpeppe1natefinch: any chance of a review?15:33
rogpeppe1natefinch: it is trivial15:33
natefinchrogpeppe1: sorry, you posted the bug link twice, didn't realize there was a PR to review15:34
rogpeppe1natefinch: ha ha15:34
rogpeppe1natefinch: one mo :https://github.com/juju/juju/pull/677515:35
natefinchrogpeppe1: shouldn't it just be updating the utils dep?15:36
rogpeppe1natefinch: it is15:36
rogpeppe1natefinch: the others are... i don't know why that happens15:36
rogpeppe1natefinch: but the commits are the same15:36
rogpeppe1natefinch: just the dates changed15:36
rogpeppe1natefinch: i really don't understand how the same commit could get two dates15:37
natefinchog yeah, huh,  missed that15:37
natefinchman, who wrote this damn tool ;)15:37
natefinchit's git, who really understands any of it15:37
rogpeppe1natefinch: git?15:38
natefinchI presume the timestamps are from git, right?15:38
rogpeppe1natefinch: yeah, godeps uses git log -n 1 '--pretty=format:%H %ct' HEAD'15:39
rogpeppe1natefinch: to print the commit date15:39
rogpeppe1natefinch: i don't know how that could cause a different unix timestamp to be printed for the same commit15:40
rogpeppe1natefinch: i have a suspicion that it might be caused by people changing dependencies.tsv manually15:41
natefinchrogpeppe1: could be.  usually I edit it by hand, but just replace a whole line with a whole line generated by godeps15:43
rogpeppe1natefinch: i'd recommend never editing it by hand15:43
rogpeppe1natefinch: then you can't get it wrong15:43
natefinchrogpeppe1: is the OS problem fixed?  used to be that godeps wouldn't show depenedencies in code that isn't compiled for the current OS15:44
rogpeppe1natefinch: GOOS=windows godeps -t ./...15:45
rogpeppe1natefinch: i need to work out a decent solution to the OS problem15:45
rogpeppe1natefinch: and some time to implement it15:45
natefinchrogpeppe1: I think one run can't get all our deps, we have both linux-specific and windows-specific deps now15:49
rogpeppe1natefinch: we have whole packages which are only included under linux?15:49
rogpeppe1natefinch: in general i think that's a bad idea15:50
rogpeppe1natefinch: but i guess i see how it can happen15:50
natefinchrogpeppe1: I think the lxd stuff is only imported for linux15:50
natefinchrogpeppe1: or at least transitive dependencies thereof15:51
rogpeppe1natefinch: hmm, how is anyone meant to update godeps now?15:51
perrito666brb lunch15:52
natefinchrogpeppe1: I usually know what is changing and so I know what needs to be updated, so I just godeps -t ./... | grep thepackage15:53
natefinchand then just manually replace that line in the current dependencies.tsv.... not ideal, obviously15:54
rogpeppe1natefinch: i really need to fix godeps to make this work15:55
natefinchrogpeppe1: one easy fix could be a wrapper around godeps that just unions two runs of godeps for win/linux15:55
rogpeppe1natefinch: yeah - the solution i've seen elsewhere is to provide a list of set sets and do everything on each tag set15:56
natefinchrogpeppe1: seems easy enough.  Slower than trying to do it all in a single run, but a ton easier to implement15:57
natefinchand easier to know you've gotten it right15:58
rogpeppe1natefinch: hmm, our old review links don't seem to work any more. http://reviews.vapour.ws/r/4396/15:59
natefinchrogpeppe1: doh...16:00
natefinchsinzui: is the reviewboard instance supposed to be working?16:00
sinzuinatefinch: It appeared to be up yesterday. We don't touch it16:00
natefinchsinzui: blah16:01
natefinchand this is why I'm glad we moved to github reviews16:01
sinzuinatefinch: I see it is angry. I will take a look at the host. I suspect disk space because that affected 2 other machines this week16:01
rogpeppe1natefinch: yeah, at least one of them looks pretty clearly like a manual edit: 75dd020b936fdcf67f692eef3ee3bb74e2e248ed16:01
natefinchrogpeppe1: we should send an email to juju-dev saying don't do that.  At least run godeps and copy the entire line, I think that's a reasonable enough way until we can just run godeps -t ./... > dependencies.tsv16:03
Spadsrogpeppe1: natefinch so that fix is much appreciated, unfortunately I need someone who understands the transactions db stuff to fix https://bugs.launchpad.net/juju-core/+bug/1484105/comments/16 before we can test it ☺16:04
rogpeppe1Spads: i understood (some of) the txn stuff once16:05
rogpeppe1Spads: but i need to re-learn it every time i dive in16:05
Spadsrogpeppe1: so my plan right now is to shut down the jujuds on the controller to prevent accidental mishap over the weekend16:05
Spadsrogpeppe1: but if there's any read-only diagnostic work we can to to add on to junien's that may be a good thing to sort out now16:06
natefinchstate changing too fast means the asserts in the actual db ops don't match the coded prechecks we run before running the transaction16:06
rogpeppe1Spads: the txn-revno assertion is usually to assert that nothing changed in the doc16:06
natefinchwe should make a new bug each time this happens because it's almost never the same code twice16:07
natefinchit's just a really uninformative error message16:07
Spadsa pity lp won't split bugs as easily as it merges ☺16:07
sinzuinatefinch: reviewboard is our of disk. I am going to restatrt the host, the clear out a few G of tmp files16:09
Spadsnatefinch: rogpeppe1: so does this look like something that would sort itself out if we quiesced the controllers, or is it something dire that needs surgery?16:10
rogpeppe1Spads: i don't know without looking at it in more detail16:10
perrito666sinzui: I am curious dont we have automatic checks for stats on our machines?16:11
natefinchSpads: it's almost never due to actual state changing too fast16:11
natefinchSpads: it's almost always a bug in the code where the prechecks in go code don't match what the mgo assertions are checking for.16:12
rogpeppe1Spads: i wonder if just inserting a txn-revno:0 field into the doc might fix it16:12
natefinchsinzui: thanks for the help16:13
Spadsso do either of you think my plan to just shut down jujuds is a good idea?16:14
Spadsso that we can freeze the env in time16:14
Spadsor should we leave it running in this state16:14
Spadsas it has been all day really16:14
rogpeppe1Spads: this doesn't seem like a real db corruption issue to me16:14
rogpeppe1Spads: just a bug in the juju transaction logic16:14
Spadsokay that's very good to hear16:15
rogpeppe1Spads: i think the problem is probably that *most* places that insert a document insert a struct that has an explicit txn-revno field, so it will be inserted with the value 016:16
rogpeppe1Spads: but for settings it inserts a map16:16
rogpeppe1Spads: which won't have that field16:16
* Spads nods16:17
Spadswhat's the difference between a struct and a map?16:17
Spadsthey both sound like key/value data structures to me ☺16:17
rogpeppe1hmm, but this is statuses not settings16:17
rogpeppe1Spads: a struct is like a C struct - it has a known set of fields16:18
sinzuinatefinch: we don't have status checks on reviewboard or the reports site...we do on the other sites16:18
Spadsah okay, so like a named tuple then16:18
sinzuinatefinch: reviewbaord is up16:18
rogpeppe1natefinch, Spads: ah, i think the issue is that statusDoc doesn't have a txn-revno field16:19
rogpeppe1Spads: yeah, kinda16:19
rogpeppe1natefinch, Spads: i'm trying to work out how it could ever work16:20
natefinchI can't tell you how often I've looked at code and wondered that16:20
rogpeppe1natefinch, Spads: in fact, it looks like the txn package always inserts a txn-revno field16:44
balloonscan someone have a quick peek at https://github.com/juju/juju/pull/6776?16:44
Spadsrogpeppe1: so how did it not get in?16:44
rogpeppe1Spads: an excellent question16:44
Spadsor is it trying to insert and failing and that's the error?16:44
Spadsoperation-ordering bug?16:44
natefinchballoons: lol, I just changed this to beta4 yesterday16:46
natefinchballoons: did we do what we needed to do with it at beta4?16:46
balloonsnatefinch, lol, I know.. fun isn't it?16:46
balloonsnatefinch, the release today is beta4, so we had to make the source reflect that for one commit to release it16:47
natefinchballoons: lgtm16:47
mupBug #1646777 changed: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:Fix Released by nskaggs> <https://launchpad.net/bugs/1646777>17:11
redireasy review https://github.com/juju/juju/pull/677717:18
natefinchredir: ahh, so we were accidentally not building on s390x I guess?17:35
xnoxoooooo =( please build on s390x17:35
natefinchredir: yeah, architecture restrictions don't seem to make sense in go code most of the time.17:35
xnoxthings should build there.....17:35
xnoxnatefinch, we like use juju to deploy openstack on s390x =/17:36
redirIf it is not supported should it build?17:36
xnoxso it's not a toy port17:36
xnoxredir, we contractually support it17:36
natefinchit's just a single file17:36
natefinchnot the whole thing17:36
redirright just kvm17:37
xnoxwe support everything that works, which is lxd/local, kvm, openstack providers.17:37
natefinchredir means kvm doesn't work on s390x so we were avoiding compiling some stuff for kvm, but then it broke other stuff17:37
xnoxthere are no public clouds for s390x.17:37
xnoxnatefinch, qemu, kvm, nova, openstack all work on s390x. there maybe some switches that are not supported by qemu-system-s390x which may be used.17:38
xnoxnatefinch, redir - if there are any s390x specific bugs, I can look into that for you and submit merge proposals.17:38
* xnox had to s/pci/ccw/ in some places before17:38
natefinchsorry, I think I misrepresented what redir was saying.  Anyway, the bug was that we were not compiling this one file on s390x and should have been17:38
redireither this one file or the one that defines that it sin't supported17:39
xnoxnonetheless if s390x-arch things do creep up, feel free to ping me =)17:39
redirwill do xnox17:39
redirxnox: to be clear it should build on s390x17:49
rogpeppe1natefinch: i just did some work on making go/build work with any build tags: https://github.com/rogpeppe/godeps/pull/518:26
rogpeppe1natefinch: s:go/build/godeps/18:26
rogpeppe1natefinch: feel free to review if you like18:26
mupBug #1646777 opened: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:In Progress by nskaggs> <https://launchpad.net/bugs/1646777>18:41
natefinchrogpeppe1: looking18:54
mupBug #1646777 changed: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:In Progress by nskaggs> <https://launchpad.net/bugs/1646777>18:59
mupBug #1646777 opened: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:In Progress by nskaggs> <https://launchpad.net/bugs/1646777>19:05
natefinchperrito666: when you get a chance: https://github.com/juju/juju/pull/677819:34
perrito666natefinch: well the power on the computer with my code is down so why not now?19:34
natefinchperrito666: seems like your town needs to double down on its street drainage systems19:39
perrito666natefinch: done19:42
perrito666natefinch: that is just my neigbourhood, I live in a closed comunity surounded by other closed comunities so we have a shared draining system that uses, you guessed, my street, its not that deep never goes over 15cm19:43
perrito666natefinch: and it drains as soon as rain stops, it all goes to a big wather buffer down my street19:43
natefinchI think your definition of a drainage system needs updating if "your street" is something that can be part of it :)19:44
redirclosed source drainage19:44
perrito666natefinch: if you lived in south america you would prefer your street to have a few cm of water instead of the possibility to have water stalling in a drainage (mosquitos)19:45
redirproprietary drainage19:45
perrito666natefinch: also there is no sewage here so it all goes to nearby fields where its absorbed19:45
redirfrobware: yt?19:54
redirI'm gonna doubt it:)19:54
* perrito666 should really have bought that new ups this week19:55
natefinchperrito666: btw, what endpoints do you think will be incorrectly handled with the code I wrote for ping?20:00
natefinchperrito666: I doubt giving a gopher:// URI to the vsphere provider is going to work.20:00
perrito666natefinch: oh no, I just suggest that you check for that and fail early20:01
perrito666anyone needs reviews?20:45
TheMueperrito666: already done some today, and my latest branch is reviewed and merged. :)20:46
TheMueperrito666: establishing some good procedures of our Canonical way of work in my new team.20:47
perrito666heh, I wold prefer to do juju reviews though :p20:47
TheMueperrito666: none open?20:47
perrito666TheMue: was looking for someone with priority since no one says anything ill default to the open ones20:49
TheMueperrito666: yeah, standard business has to be done too20:49
* redir lunches21:13
* redir back21:32
=== frankban is now known as frankban|afk
perrito666k, ill EOD because my laptop's battery is about to die21:39
mupBug #1646777 changed: juju 1.25.9 yakkety deploys failing 'agent-state-info: no matching tools available' <OpenStack Charm Test Infra:Invalid by 1chb1n> <juju-core:Fix Released by nskaggs> <https://launchpad.net/bugs/1646777>21:41
redirlater perrito66621:46

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!