[00:12] update github.com/juju/names failed; trying to fetch newer version [00:12] godeps: cannot update "/home/dfc/src/github.com/juju/names": fatal: reference is not a tree: 4bd61d19a7fce663e2821fe05ddf69f774d444da [00:12] another day, another godeps head desk [00:51] anyone able to help with a really weird juju deployment issue? juju deployer is barfing out a traceback at me [00:52] wallyworld_: is anyone looking at bug 1396981? that's actually the CI blocker, not the other critical one which i've been looking at. [00:52] Bug #1396981: Upgrade fails with tools sha mismatch

[00:52] oh, i didn't see that one [00:53] bradm: I can try and help [00:53] i'll read the bug [00:53] bradm: what's the issue [00:55] https://pastebin.canonical.com/121262/ is the output [00:56] basically doing a juju deploy of ubuntu charm (calling it infra) is failing at the add unit stage, and then every run there after fails with a 'ERROR cannot add service "infra": service already exists' [00:57] this is using maas with physical nodes, so the deploy will take some time [00:57] juju version 1.20.11 [01:00] any info missing from that? [01:01] bradm: that looks like you're using the python deployer, right? [01:02] wallyworld_: yup [01:02] wallyworld_: this has worked fine on other deployment stacks, fwiw [01:02] i've not seen any of that code sadly, it's all been developed outside of core [01:03] do you normally get all those errors? [01:03] the charm url or branch? yeah, thats just the way we've had to get the charms into place [01:03] hmm, ik [01:04] there's probably a fix for it, but you know customer deadlines. [01:04] it looks like there's a mismatch between the model knowm by the deployer and that of juju [01:04] the deployer doesn't seem to know about the service [01:04] it kicks off the initial deploy ok, I can see it in juju status [01:05] so juju status shows an infra service all deployed ok [01:05] not deployed ok, they're in pending [01:06] but it does eventually return ok [01:06] bradm, wallyworld_: could there be a race... where at the time status was checked the service wasn't there yet? [01:06] oddly if I use juju-deployer find service thing, it gives me the right info [01:07] i think you really need to talk to someobe who wrote the deployer code [01:07] we can speculate, but that won't lead to much [01:07] probably :( [01:08] was it kapil who wrote it? [01:08] https://pastebin.canonical.com/121264/ is kind of interesting [01:08] the juju deploy says the service already exists [01:09] which, well, it does. [01:09] so that implies a scripting error [01:09] its just in pending [01:09] if you want to add the same charm twice, you need to use a different service name' [01:09] doesn't matter if it's in pending or not [01:10] sure [01:10] I have no idea why its trying to do that, though. [01:10] and, yeah, pretty sure its kapil who wrote the code [01:10] i think it's past his bedtime now [01:11] menn0: that critical bug - sadly there's not enough to go on in the bug report - they really needed to attach all the simplestreams metadata files [01:11] yeah, I've tried chasing him down before, I don't think I have any overlap [01:11] and it being turkey day doesn't help [01:11] maybe rick_h_ knows [01:11] wallyworld_: so what can we do to unblock CI [01:12] menn0: i'm thinking we just need to revert [01:12] but that's a very big hammer approach [01:12] i just wish they would attach the relevant info to the bug report so we can diagnose properly [01:13] on the surface, having 2 different lots of metadata files, old and new, shouldn't matter [01:14] wallyworld_: maybe rick_h_ knows what? [01:14] hey rick [01:14] the deployer is acting up and we here on core know nothing about it [01:15] its super confusing that I can ask juju deployer to tell me where the servic e is deployed, but a deployment run tries to do a fresh deploy [01:16] rick_h_: I'm having issues with juju deployer barfing with https://pastebin.canonical.com/121262/ on the first run, and then every run after complains about https://pastebin.canonical.com/121264/ [01:17] yet a juju-deployer -f infra tells me the right location (even if it is still pending) [01:17] bradm: is this specific to this bundle? Or does it do it on any deployment? [01:17] bradm: have the deployer file you're using? [01:17] rick_h_: its specific to this exact deployment of bootstack, we've got multiple other instance that works fine [01:19] rick_h_: the full deployer file? or just the infra bits? [01:20] * menn0 goes for lunch [01:20] bradm: both? [01:20] rick_h_: its a bit complex for the full thing, its a HA juju environment with openstack deployed to it [01:20] rick_h_: including landscape, ksplice, nagios, etcetc [01:21] https://pastebin.canonical.com/121265/ is the infra bit, its about as simple as you can get [01:26] the stacks that work have different physical hardware underlying it, but as far as I can tell its the same config. [01:27] same version of juju-core, minor differences in juju-deployer version, so I downreved the one that wasn't working to be the same [01:28] bradm: yea, so looking the code in the traceback makes sense. What does juju status have? I'm guessing infra isn't in there? [01:28] bradm: but yea, got nothing sorry. I didn't even realize that the deployer did local charms for the local:ubuntu stuff. [01:29] rick_h_: the infra charm is in state pending, since it takes a while to install [01:31] the infra charm is ok now, but I still get the "service already exists" bit [01:31] so for some reason, juju-deployer isn't finding the service there, and trying to deploy it [01:32] bradm: hmm, race condition then? env_status['services'][svc.name] is failing [01:32] rick_h_: I think we're hitting two seperate issues - one race condition for the initial deploy, and one bug because it doesn't find the existing service after its deployed. [01:33] bradm: right, I'm guessing it doesn't finish writing out/dealing with info on the first pass [01:33] bradm: so the second pass is always going to fail with it in some sort of incomplete state [01:33] rick_h_: yeah, it takes a good 5 - 10 minutes or more for the deploy [01:33] rick_h_: but surely it should see that the infra service exists, and not try to redeploy it [01:35] rick_h_: WTH? [01:35] rick_h_: I don't even... [01:35] rick_h_: why are you here on thanksgiving? [01:35] rick_h_: looking at the code, it means that env_status['services'] isn't in the list of services [01:35] bradm: why you do this to him? [01:36] er, I mean, infra isn't in that list [01:36] bradm: right, the best I can do is to offer frankban can look at if you have it running/give him access to the env tomorrow in EU time [01:36] waigani: when you've made the tweak and it is ready for review, ping me [01:36] thumper: because my wife is watching a stupid movie, sent the family home, and wallyworld mentioned me so got curious [01:36] rick_h_: what movie? [01:36] rick_h_: its a customer deploy so I don't know about getting access to it, but we can certainly do some debugging if he's around [01:36] thumper: going to bail out though :) too much wine to debug deployer :) [01:36] thumper: 22 jump street [01:36] rick_h_: this is why I log out of IRC [01:37] rick_h_: thank you :-) [01:37] yep that is stupid [01:37] rick_h_: many thanks for the help :) [01:37] thumper: got distracted, I'll do that now [01:37] the only reason I watched it was because I was on a plane [01:37] waigani: and you too, thanks :) [01:37] er [01:37] wallyworld_ even [01:37] bradm: yea sorry. The only thing I can think to do is to run the deployer and pdb and check what's in the list of services [01:37] stupid tab complete and me not looking closely :) [01:37] bradm: welcome ;) [01:37] AGHH!!!! [01:37] dog just farted [01:37] bradm: so it'd basically be running it, stepping through it, trying to see wtf it's doing, and debugging the actual data there. [01:37] * thumper coughs [01:37] lol [01:38] rick_h_: no need to be sorry, any help has been useful, even if just show I'm not missing anything obvious [01:42] thumper: so i don't know what to do about bug 1396981. there really isn't enough attached to the bug to prove the root cause is the suspected pull request, and tracking it down will take time, but i'm loath to revert without more proof [01:42] Bug #1396981: Upgrade fails with tools sha mismatch

[01:42] would you revert anyway? [01:42] I've not seen that bug... [01:43] it could be a scripting issue in the tests [01:43] for example [01:43] thumper: so the pr was merged 3 days ago - you've upgraded from 1.20 since then right? [01:44] only with uploading tools [01:45] wallyworld_: is the filename inludec in the hash calculation? [01:45] if so, it would be a reason [01:45] nope [01:45] the filename is the json metadata filenakjme [01:45] the hash is calculated on the contents of the tools tarball [01:46] thumper: extractPortsIdPart is gone but I've left extractPortsIdParts as the openedPortsWatcher uses it to do some funky tansformId() stuff on the doc id: http://reviews.vapour.ws/r/496/ [01:47] and the upgrade step uses it [01:48] wallyworld_: I'm not sure that reverting the patch would fix the problem [01:48] wallyworld_: it should be easy enough to test locally though, no? [01:48] i don't think so either [01:48] yes, but i'm in the middlw of something, sigh [02:36] thumper, wallyworld_: shall I have a go at reproing bug 1396981? [02:36] Bug #1396981: Upgrade fails with tools sha mismatch

[02:36] menn0: if you want, i was about to start looking, just finishing up some other work and about to stash [02:37] wallyworld_: I don't mind who does it. I just want CI unblocked :) [02:37] yeah, sorry. i'll start looking [02:41] wallyworld_: kk. I suspect you'll get there faster. Let me know if I can help. [02:41] sure, ty [03:18] wallyworld_: I've possibly been making a mountain out of a molehill with identifying disks. if MAAS creates a physical volume on the disk, then we can create a logical one and that'll get a stable UUID. we would only hand the logical volume off to the charm, so assuming the charm doesn't go OOB and touch the disk itself, it should be fine to use a device name [03:18] I'll need to think a bit more... [03:19] sure [03:43] menn0: thumper: I've marked the bug as Incomplete with an explanation, not sure if that unblocks landings or not [03:43] wallyworld_: ok. I have something to land so let me try. [03:44] you may need a JFDI [03:44] if the first try doesn't work [03:52] wallyworld_: seems to have worked... the tests are running [03:52] \o/ [04:17] waigani: finally reviewing your statusDoc changes [04:18] menn0: thanks. When you have a moment, can we talk over how to test the allwatcher branch? [04:21] waigani: review done. i think you missed one. [04:21] menn0: the update one? [04:22] waigani: yeah. [04:22] waigani: doesn't it replace the document? [04:22] menn0: that would already have an env-uuid right? [04:22] waigani: don't think so. "$set" will replace the existing document with the new one. [04:23] waigani: or maybe i'm wrong [04:23] * menn0 checks docs [04:24] waigani: ignore me. I misunderstood what $set does [04:24] * menn0 updates review [04:24] menn0: okay [04:25] waigani: chat in standup channel regarding allwatcher [04:25] ? [04:25] menn0: that would be great [05:27] axw: much thanks for the review and comments! [05:28] * jw4 goes back to thanksgiving dinner guests [05:32] jw4: no worries. happy thinksgiving [07:13] fwereade_: you free for a question? === urulama_ is now known as urulama [08:02] wallyworld_, sorry, here [08:27] fwereade_, hey [08:27] dimitern, heyhey [08:27] fwereade_, I didn't get a reply yet from jamespage about the meeting btw [08:27] dimitern, bah [08:28] fwereade_, you should've got a copy though [08:29] fwereade_, I'll ping him if I see him, or I'll resend it on monday [08:30] fwereade_, do you know if gustavo is taking time off? [08:30] dimitern, yeah, I think he's off until december [08:30] fwereade_, ah, good - just in time for me to propose a few more goamz MPs :) [08:31] dimitern, he's been off for a while I think [08:31] yeah, it seems so [08:54] morning all [08:56] morning mattyw [09:10] fwereade_: you free now? === liam_ is now known as Guest93453 [09:13] wallyworld_, yeah [09:13] wallyworld_, sorry, were you out before? [09:13] fwereade_: quick chat in our 1:1? [09:13] yeah was afk for a bit [09:27] morning [09:29] dimitern, sorry [09:30] jamespage, it's ok, I'm sure you've been pretty busy [09:31] dimitern, book me a slot for today if you like :-) [09:31] jamespage, sure, I'll send an invite [09:32] fwereade_, when is a good time for a chat today? [10:04] voidspace, since we're the only ones here - a quick standup? :) [10:04] dimitern: omw [10:06] dimitern: https://bugs.launchpad.net/juju-core/+bug/1396981 [10:06] Bug #1396981: Upgrade fails with tools sha mismatch

[10:06] voidspace, lp:~dimitern/goamz/update-aws-api-version-to-latest [10:07] https://code.launchpad.net/~dimitern/goamz/modifysubnetattribute/+merge/243128 [10:14] dimitern, oops, sorry, missed that -- I can come anytime basically [10:15] * fwereade_ has a naming problem /grrmbl [10:15] fwereade_, jamespage, ok, how about 12 UTC for 1h? [10:15] dimitern, fine with me - please invite gnuoy as well [10:15] dimitern, sgtm [10:16] jamespage, fwereade_, ok, I'll send an invite now, thanks [10:33] fwereade_, jamespage, gnuoy, invites sent [10:34] ta [10:36] dimitern: lGTM [10:36] dimitern: ModifySubnetAttribute that is [10:36] voidspace, thanks! === rogpeppe1 is now known as rogpeppe [11:00] so, my naming problem [11:00] given that uniter/context is now uniter/runner [11:01] and context.Factory is now runner.Factory [11:01] which produces Runners [11:01] my answer is marathon [11:01] what's the question? [11:01] things like NewHookRunner still produce Runners [11:01] which have methods like RunHook, RunAction, RunCommands [11:02] the factory really ought to be producing things with just a Run method [11:02] so the obvious name for that is a Runner [11:02] ie type Runner interface { Run() error } [11:02] what then do I call the varying [11:02] oh wait [11:03] ok, I think we have Runner as defined above [11:03] but no [11:04] ok, so the existing Runner type [11:04] still needs to exist [11:04] there's enough behaviour shared between hook/action/command running that it shouldn't be broken up [11:04] but what do I call that internal type? [11:05] fwereade_: hurdler? [11:05] maybe that's `coreRunner` without a Run method, with `hookRunner`, `actionRunner`, `commandRunner`, each of which implement Runner [11:06] testing it nicely is maybe yucky [11:06] that seems reasonable, from a code sharing perspecitve [11:07] testing should just be on those seperate objects, right? so you expose to tests and exercise, the fact they share coreRunner is just an implementation detail no? [11:08] mgz, mmmmaybe [11:08] mgz, I will kick it around and see what I can do [11:08] mgz, anastasiamac, thanks for listening to my ramblings [11:09] fwereade_: have fun ;-) my next suggestion was going to br a "racer" but implications r not pleasant. I like ur suggestion beta ;p [11:10] mgz, the main thing is that I think it'd be nicer to test the various Runner implementations against a mocked-out coreRunner, and keep the coreRunner tests as they are as much as possible [11:10] fwereade_: hmmm [11:10] so I guess I export_test type CoreRunner struct {*coreRunner} [11:10] that's an interesting idea, couldn't do it with just struct containment [11:10] or something [11:11] today is a holiday in the US right? [11:11] or maybe I should get my thesaurus out and call the CoreRunner thing an Invoker or something [11:11] perrito666: rite [11:11] not sure that makes anyones lives much easier though [11:11] ok so I am teamless for a day [11:12] perrito666: if it's still thusday [11:12] perrito666: i belive some ppl ar taking Friday [11:12] perrito666, you may officially have a team but most of them have probably taken friday too [11:12] * perrito666 thinks that the choice of returning from vacations on a friday might not have been the best [11:12] perrito666: teamless but not alone;-) [11:13] fwereade_: I don't really think new names for different levels of runners would make it clearer :) [11:13] mgz, indeed [11:13] mgz, but having the *same* names for two different levels may be *even less* clear [11:14] * fwereade_ grumbles to himself a bit [11:14] I can think of lots of terrible suggestions :) [11:15] haha [11:15] I think I'm going to get a sandwich and see if inspiration strikes [11:15] like the old add-more-er-s one [11:15] lol [11:16] but it's *obvious* that an Ererer is a factory for Erers, which are objects allowing one to express uncertainty [11:16] :D [11:16] import ererererest [11:16] * fwereade_ twitches [11:17] * fwereade_ goes to get bread [11:18] fwereade_: do u need to identify "two different levels" as runners? shouldn't only a "coreRunner" b identfied as such and all others [hookRunner-commandRunner] as smth else? [11:19] anastasiamac, the urge to call the thing with a Run method Runner is strong, though [11:20] fwereade_: :-p the same as "executor" for execute? [11:20] * fwereade_ looks shamefaced because he has an Executor with a Run method in uniter/operation [11:21] it was execute for a while [11:21] * fwereade_ should s/Operation.Execute/Operation.Run/ and s/Executor.Run/Executor.Execute/, shouldn't he [11:22] * fwereade_ sandwich, anyway [11:33] fwereade_: was lighthearted ;D no sinister intention behind (m guilty of naming too many executors in past life) [11:33] fwereade_: looking at thesaurus for runner/run is amusing ;-) [11:43] :q [11:43] wrong window... [11:43] anastasiamac: o/ [11:43] anastasiamac: how's life - late for you isn't it? [11:45] voidspace: life is hot and stormy ;-) [11:45] anastasiamac: just how I like life [11:45] voidspace: m usually online at this hour - kids r asleep: so blissful :D [11:45] anastasiamac: unfortunately mine is cold and greay [11:46] anastasiamac: ah, nice :-) [11:46] *grey [11:46] voidspace: yesterday had hail the size of cricket balls in cbd [11:46] even more cold than usual as our bathroom window is being replaces [11:46] anastasiamac: yow [11:46] voidspace: glass shattered - cars, buildings, roofs [11:46] anastasiamac: cbd? [11:46] anastasiamac: wow, I bet [11:46] voidspace: central business district [11:47] ah [11:47] voidspace: we were lucky: just lost power [11:47] voidspace: were running essentials on generator [11:47] voidspace: can u believe that internet is not essential in some households? [11:47] anastasiamac: that makes our village seem tame by comparison [11:48] anastasiamac: to be fair it is pretty tame [11:48] anastasiamac, that's crazy talk [11:48] anastasiamac: very crazy [11:48] voidspace: fwereade_: yes, apparently ppl prefer fridge to router ;p [11:48] totally bizarre priorities [11:49] * perrito666 merges his branch after one week and it still compiles... success [11:49] perrito666: \o/ [11:51] voidspace: interesting time to replace bathroom window.. isn't it almost winter? [11:51] voidspace: like one day away? [11:52] anastasiamac: we're renting - and the landlord offered to replace the single glazed wooden one with a double glazed upvc one [11:52] anastasiamac: we didn't say no :-) [11:52] anastasiamac: although we're looking at buying a house in the same village soon anyway [11:53] anastasiamac: but yeah, pretty wintery here [11:53] I'm hoping we get snow this winter [11:53] we didn't last year [11:53] total waste of a winter [11:55] ah how I miss winter (which most likely looks like your spring voidspace ) [11:55] i love the idea of snow but it usually comes with cold which I am not big fan of ;-) [11:56] perrito666: heh [11:56] altho - winter clothes m fan of [11:56] anastasiamac: working from home is great :-) [11:56] I am still waiting on the delivery of my AC it is becoming annoying apparently they had a cyber monday sales spike and they are behind on delivery [11:56] anastasiamac: although no "snow days", so long as the internet works so do we... [11:56] perrito666: so no AC at the moment? [11:56] voidspace: only in the bedroom [11:57] voidspace: the good news is that here climate has a cycle of 40C, rain, 40C, rain and I am in the rain part atm [11:57] hehe [11:57] I do love a hot climate. Snow is the *only* redeeming feature of winter here. [11:58] Well, it makes me appreciate the summer more I guess. [11:58] perrito666: in our parts rain brings little relief [11:58] voidspace: I do prefer cold climate, With cold you can pile up clothes on you until it recedes, with hot summer there is only so much you can remove :p [11:58] anastasiamac: where are you? [11:58] voidspace: really? snowball fights, sleigh riding, etc - must b gr8 ;D [11:59] perrito666: BrisbaneAustralia [11:59] anastasiamac: I keep wanting to know australia, seems so nice [11:59] perrito666: like u - in the middel of summer, hot/wet/hot/wet cycle :) [11:59] anastasiamac: right, but they all come from snow... [11:59] anastasiamac: mostly winter is just cold and wet [11:59] sledging is *awesome* [12:00] lots of great fields round here for sledging [12:00] fwereade_, gnuoy, meeting? [12:00] voidspace: i dont remember winter much ;( [12:00] voidspace: well in here its hot and wet, you feel like steamed rice all the time [12:00] perrito666: :-) [12:00] perrito666: u r in Argentina? [12:03] anastasiamac: yup [12:19] wallyworld_: hiya [12:19] wallyworld_: just say your email [12:19] s/say/saw/ [12:19] hi [12:19] hope it made sense [12:20] wallyworld_: when you were doing your secure wget, were you adding the root cert as a trusted cert somehow? [12:21] i was using wget --ca-certificate blah.pem https://ssipaddress:17070/.... [12:22] where blah.pem is the ca cert [12:22] wallyworld_: right, i see. yeah, i can see the issue. [12:23] rsyslog cert had the same issue [12:23] wallyworld_: it's a pity that it's not possible to specify a genuinely wildcard domain name [12:23] yeah [12:23] using ip addresses is insecure sadly also, but we don't have dns names to use [12:24] wallyworld_: tbh there's no security issue here - we don't rely on the ip address or host name for security at all [12:24] wallyworld_: it's just that we need to work around the dubious x509 model [12:25] rogpeppe: anastasiamac tells me it's insecure, i know not that much about security [12:25] wallyworld_: it would be insecure if we weren't using a self-signed root cert [12:25] ah ok [12:25] wallyworld_: which we're explicitly trusting [12:26] wallyworld_: so i guess you'd generate the certificate with all the possible ip addresses of all the state servers [12:27] i generate the cert for each state server separately, with just that state server's ip addresses [12:27] wallyworld_: and all the DNS names too - basically everything from the server addresses [12:27] wallyworld_: i don't think that's a great idea [12:27] since each state server runs its own https sservice [12:27] wallyworld_: we need to be able to connect to any of the servers with the same cert [12:27] wallyworld_: or... [12:27] we can i think [12:27] wallyworld_: i see [12:28] wallyworld_: so each state server has the root cert [12:28] wallyworld_: and generates its own certificate for its own addresses [12:28] yes [12:28] the same root cert [12:28] seems to work anyway [12:28] wallyworld_: yeah, that seems like a good way to do it [12:29] rogpeppe: wallyworld_: no opinion on security in our case - I was refereing to Department of Defence, Australian Signals Directorate (http://www.asd.gov.au/publications/csocprotect/dns_security.htm) [12:29] rogpeppe: cool, so the issue is should i store the ca cert private key in agent conf on the state server [12:29] it was discarded but is now needed [12:30] anastasiamac: i have no trust in modern web "security" tbh [12:30] we already store the server cert private key there [12:30] anastasiamac: (hiya, BTW) [12:30] rogpeppe: o/ [12:31] rogpeppe: i don't trust either - this is an explanantion of what might happen and how it the risks can be mitigated [12:31] wallyworld_: yeah, i think just use the ca cert everywhere we were using the server cert before. [12:31] wallyworld_: except when actually starting a server [12:31] rogpeppe: actually, trust is a bad word for it - all "security" is only "securing" to a degree... [12:31] anastasiamac: yup [12:32] anastasiamac: but this is all so obviously broken... [12:32] rogpeppe: :-( yes it is but wallyworld_ is on it! it'll b unbroken soon :D [12:33] thanks rogpeppe , i'll tidy up my branch next week [12:33]