[00:00] http://paste.ubuntu.com/6840930/ [00:00] great, how am I supposed to shut down this environment ? [00:18] i would kill the machines by hand and delete the control container, then delete the jenv === thumper-afk is now known as thumper [00:21] davecheney: really, WTF? [00:22] how do I remove the govet check? [00:22] wallyworld_: I upgraded to trusty last night [00:22] trunk starts the local provider [00:22] hmmm [00:22] although my http-proxy wasn't on the right port [00:23] trunk mind, not release [00:23] release is all fubared [00:23] when i tried on friday, the agent ran, but no lxc container would start [00:23] for some reason I've lost the all-machines.log [00:23] yeah, me too [00:23] need to figure that out [00:23] I have a branch to submit for destroy [00:23] \o/ [00:24] if I can get lbox shite working [00:24] thumper: with go vet, i installed from source and got it to work that way [00:24] wallyworld_: you installed go from source? [00:24] * wallyworld_ tries to remember [00:24] go get -u ...something [00:24] let me see if my history shows it [00:25] i think the error message gave a hint [00:25] it tries to open the /usr/lib/go pkg dir [00:25] for some reason it isn't honouring the GOPATH [00:26] davecheney: can I just change an environment variable? [00:26] not sure if this was the attempt that worked [00:26] sudo bash -c "export GOPATH=/usr/lib/go; go get code.google.com/p/go.tools/cmd/vet" [00:26] something like that was necessary [00:26] i just assumed it was my fucked up set up [00:26] didn't realise it was a general issue for trusty [00:27] hmm... [00:28] yeah, that looks weird to the extent I don't want to do that [00:28] i may have had to do it with a GOPATH pointing to a dir off my home, not usr/lib [00:28] but i recall sudo and messing with GOPATH was necessary [00:28] a fuck it... [00:30] nah, that doesn't work [00:31] hmmm sorry. i can't recall exactly what worked. i *think* i had to point GOPATH to a local src dir, but use sudo. not sure thpugh [00:33] * thumper stabs .lbox.check [00:34] FARK [00:34] can't propose with a dirty branch [00:35] so either I commit the lack of vet [00:35] or don't use lbox [00:35] * thumper decides to skip lbox [00:36] \o/ [00:39] wallyworld_: https://code.launchpad.net/~thumper/juju-core/fix-lxc-destroy/+merge/203861 [00:39] looking [00:42] thumper: done. btw did you see google offloaded motorola to lenovo. those lenovo guys are really going on a shopping spree [00:42] no I didn't know that [00:43] wallyworld_: did google keep all the patents? [00:43] yep :-) [00:43] of course [00:43] heh [00:43] they licenced them to lenovo [00:43] wallyworld_: I can't land it yet, it didn't fix the problem [00:43] so they paied $12bil, sold for $3bil [00:46] ah... [00:46] hang on [00:46] I see what's happened now [00:46] I have a running juju machine agent for zero [00:46] rebuilding a local juju doesn't help destroy environment [00:47] because the environment is already running [00:47] and using an old jujud [00:47] which doesn't have my fix [00:47] FFS [00:52] manual hackery FTW [00:53] i've done the same thing before too [00:57] ok, at least that is the local provider starting and stopping in trusty [00:57] now to poke and see why we have no all-machines.log [00:58] hmm... I think I may see [01:00] thumper: I looked into that the other day, it was permissions [01:00] yeah, I saw that too [01:01] axw: have you a fix ready? [01:01] if not, I'll do it [01:01] thumper: I think we may have to do something yucky, like symlinking to /var/log/juju [01:01] no [01:01] it passed out of my memory on the flight back [01:07] wallyworld_: I'm witting in our weekly hangout... [01:07] oh yeah [01:12] thumper, how's proxy support coming? [01:12] hazmat: trunk is almost feature complete [01:13] there are a few things outstanding [01:13] - remove old apt proxy cruft in container setup [01:13] - make sure containers get the proxy info for the container providers [01:13] - some upgrade code [01:16] hmm.. that sounds pretty good... does that include things like the cli respecting std http_proxy env vars for tool lookup? [01:16] hazmat: in a hangout just now [01:16] hazmat: how long have you got? [01:17] thumper, i'll make some time.. i'm pushing through on a late night [01:32] waigani: hangout? standup [01:32] yep [01:33] cap [01:33] crap [01:33] now destroy-environment cannot save me [01:33] https://bugs.launchpad.net/juju-core/+bug/1274355 [02:01] school run [02:01] bbs [02:04] * axw rejoices and promptly stops thinking about windows cli [02:05] davecheney: try destroy-environment --force [02:05] davecheney: actually... what version are you on? [02:05] trunk [02:06] axw: yup [02:06] that worked [02:06] davecheney: destroy-environment now goes and talks to the API server to destroy the environment "cleanly" [02:06] if it's borked, --force is required [02:07] we just had a chat on our scrum, and figured we should probably tell that to the user if it fails and --force wasn't specified [02:13] couldn't hurt [02:51] * thumper sighs [02:52] * thumper created a broken local provider [02:52] * thumper deletes things [03:31] fark!!! [03:31] I think it is apparmour [03:32] thumper: doing what? [03:32] stopping writing to ~/.juju/log/all-machines.log [03:32] hm ok [03:33] * thumper pokes and tweaks [03:36] * thumper does something icky === wedgwood is now known as Guest44693 [03:46] * axw goes out for lunch [03:46] bbl [03:47] if I don't return, the heat has claimed me [03:47] wallyworld_: ping [03:47] wallyworld_: who owns your /var/log/syslog file? [03:48] thumper: mine is syslog:adm [03:48] hmm... [03:48] mine is messagebus:adm [03:48] no idea where that came from [04:14] thumper: um, let me check [04:14] wallyworld_: that's ok [04:15] i don't have one right now [04:15] all these permission problems are due to rsyslog apparmor profiles [04:15] ah [04:15] I've got a work-around [04:15] but I need to go make dinner [04:15] so it'll have to wait [04:15] f*cking apparmor [04:15] :-) [04:15] waigani: is there a bug for the "no all-machines.log for the local provider"? [04:16] thumper: I have not created on [04:16] *one === thumper is now known as thumper-afk [06:16] wallyworld_: when you're not busy, can you please poke the bot? [06:16] sure [06:16] will do it now [06:17] axw: poked [06:17] thanks [06:38] wallyworld_: I thought there was a reason we needed FullPath for the manual bootstrap - the reason eludes me though [06:42] axw: FullPath is filled in when the particular tools are read from simplestreams. but is was unnecessary to fill in that value when writing metadata files caused it is ignored then [06:43] we just store ralative path in the json [06:43] and construct fullpath when needed as the metadata is loaded [06:43] yeah, I thought it was used in-memory... my memory may just be faulty though [06:44] welp, seems to work anyway [06:44] yay [06:53] wallyworld_: in provider/maas/storage.go, there's talk about 404 and IsNotFoundError [06:53] does this not work? [06:53] um [06:54] that works [06:54] but the issues is URL() [06:54] ah sorry [06:54] URL, not Get [06:54] yeah, maas is the only one that can't give a url for a non existent fle [06:54] so i changed dummy storagr to match that [06:54] wallyworld_: ahh, so MAAS 404s even for URL [06:55] got it [06:55] yeah :-( [06:55] sadly yes [07:00] wallyworld_: why do we no longer set imagemetadata.DefaultBaseURL in bootstrap? and does tools.DefaultBaseURL continue to be updated? [07:00] (sorry for dumb questions, just trying to be thorough...) [07:00] axw: np. the image metadata is uploaded from a local dir to cloud storage [07:00] it was a mistake to set th default url [07:00] in the cloud storage, the uploaded metadata is in the search path [07:01] and keeping default url unchanged means we fall back [07:01] if no images found in user metadata [07:01] ok [07:01] whereas if we change it, it'll try to look in cloud storage and in the local dir [07:02] right? [07:02] yeah, but it will error out later in the boot process [07:02] i didn't look into why [07:02] mk [07:02] i decided it was wrong to change the default url [07:03] tools url is changed cause that's what sync tools needs [07:03] end result - tools and image metadata is uploaded to cloud storage if local dir is used [07:04] thanks [07:04] and cloud storage is in simplestreams search path, behind config urs and ahead of default locations [07:04] got it [07:04] I remember why FullPath/URL was required before: we used to do a URL fetch, rather than go through storage [07:05] so we previously had to keep the URL in memory to pass along to the merging bit [07:32] axw: just to check - i'm pretty sure that we don't need full url now right [07:32] wallyworld_: right [07:33] cool, that matches my understanding too [07:33] since we use the path & storage now [07:33] yep [07:33] just wanted to be 1000% sure :-) [07:33] :) [07:34] axw: with the abs(), i thought it better to report the whole dir to the user to avoid confusion [07:35] okey dokey [08:41] axw: do we actually have a solution for people trying to use juju-core with Canonistack and setting up 'sshuttle' while they are trying to bootstrap? [08:42] or does bootstrap itself complete because it uses the SSH redirection, (I thought one of the steps was to directly connect to the API to know that it is up, so now that step will fail) [08:42] jam: bootstrap works because it goes through SSH, sshuttle is still required as it was before post-bootstrap [08:43] there's no API interaction during bootstrap [08:43] axw: so "before" you could use sshuttle to the bootstrap node once it came up, and then everything else would go direct to the API, that stil works? [08:43] we did talk about that during SFO - juju status at the end - but it hasn't been implemented [08:43] ah, k [08:43] jam: yep [08:44] sshuttle -r ubuntu@bootstrap-node subnet/cidr [08:45] I would kinda of like it if API connections did some magical tunnelling as a fallback, but it's probably too messy [08:46] axw: It would be helpful for people at Canonical, I'm not sure that it is a general solution. [08:47] Saying "you need public access to the cloud machines you are using" doesn't seem like a particularly bad thing to me. [08:47] yeah I guess so. I don't really know how many of our private cloud users will have the same issues [08:48] probably not that many [08:49] morning [08:49] morning mgz [09:01] axw: I think you'll generally need some-sort-of-VPN but we actually have 2 different ones inside Canonical. (the Server Stack needs OpenVPN to access, and doesn't work via SSH at all) [09:02] true, though our bootstrap won't even work without ssh access [09:03] I was just thinking it'd be nice to have guaranteed API access if bootstrap succeeds [09:03] it's not a big deal to have to sshuttle though, really [09:06] I may be a min or two late for meeting later [09:33] wallyworld_: ping [09:36] axw: you reviewed this branch, i think: https://codereview.appspot.com/58510044 [09:37] axw: i'm having difficulty understanding how it works - did you work it out? === thumper-afk is now known as thumper [09:52] rogpeppe: sorry, which part? [09:53] doh, good catch on the dummy storage boolean... [09:54] rogpeppe: re "How does the URL get into the metadata now?" -- simplestreams metadata doesn't encode the URL [09:55] we used to store it in the in-memory object when tools metadat was fetched by URL [09:55] but now we use the path, and go through storage [09:56] rogpeppe: and the fact taht we were calling the environment storage'URL [09:56] oops [09:56] storage's URL method, caused an error wtih MAAS [09:59] where's wallyworld_ [09:59] here [10:04] on the juju call. yay. will look in a bit. i'm also 3/4 drunk :-D [10:05] wallyworld_: ROFL [10:41] wallyworld_: 3/4 drunk and working on maas fixes \o/ [10:41] no, i'm ok now :-) [10:41] haha [10:41] it's been an hour or more since my last wine, i was exagerating :-) [10:42] a man can drink a glass of wine with dinner :-) [10:42] digging more like [10:42] a single glass has been known to get you blotto [11:03] right, I'm not interviewing horacio right now because we need to turn the power off here to investigate scary burning smells [11:03] hopefully I will be back shortly [11:04] fwereade: good luck [11:04] cheers [11:08] mgz: are you available to just hang out? [11:12] axw, wallyworld_: what's the purpose of ToolsMetadata.FullPath? are we actually setting it now? [11:12] ah! it's ignored [11:12] rogpeppe: it's used when wget fetches the tools [11:13] it's ignore writing out [11:13] wallyworld_: ok [11:13] it is composed from the relative path stored in the json [11:13] composed when needed [11:15] rogpeppe: is trunk working? [11:15] wallyworld_: i'm going to try it in a mo [11:15] * wallyworld_ crosses his fingers [11:16] jam: sure [11:16] mgz: I'm just going to grab a snack, mumble or g+? [11:17] lets try mumble first, I'll be on there [11:26] yeah, I can't understand you at all, which means it is messed up, I'll reconnect 1 more time [11:27] jam: we'll have to just fall back to hangout [11:27] use the daily standup one? [11:29] wallyworld_: sorry, it appears that there are no spare boxes currently, so testing it will have to wait for a little bit [11:29] ok, np [11:29] mgz: now ff is freaking out again... just a sec [11:29] :) [11:30] rogpeppe: i'm quietly confident it will work :-) [11:30] mgz: I'm in the "juju core team" one from earlier today [11:30] wallyworld_: did you see my remarks on your CL? [11:31] no, i'll look [11:35] rogpeppe: tomorrow i'll fix the remaining issues, thanks [11:35] wallyworld_: thanks [11:36] rogpeppe: it seems that we're still not quite universally consistent with converting relative paths to absolute when running commands [11:36] wallyworld_: yeah, we should probably use Context.AbsPath throughout [11:37] yeah, i didn't realise we had that method [12:07] rogpeppe, i'm trying to debug this missing GOPATH issue that cause uniter tests to fail in an install hook [12:07] rogpeppe, so i'm using your debug.Callers() thing to dump the stack when I detect GOPATH is empty in testing/charms.go:init() [12:08] dimitern: ok [12:08] rogpeppe, but what I'm getting is a bunch of files, and the line number for each one is always the last line in the file [12:08] rogpeppe, what does that mean? [12:08] dimitern: paste? [12:08] rogpeppe: so if you *don't* supply -test.timeout, then when it times out it just says "test took to long" if you *do* supply a value (like 10s in my testing) then you get a panic [12:09] rogpeppe, http://paste.ubuntu.com/6843438/ [12:10] rogpeppe, and my changes to init(): http://paste.ubuntu.com/6843442/ [12:11] rogpeppe, for some reason during the install hook the environment gets reset or something and GOPATH is lost [12:15] rogpeppe: it looks like '-test.timeout' is for each individual test triggering a panic, vs the global timeout being run by the meta-process when running multiple packages [12:19] jam: interesting [12:20] dimitern: sorry, i got waylaid by insanity. i have to go to lunch now, sorry, otherwise i won't make it to the discussion in 40 mins. will look when i can. [12:21] rogpeppe, ok === BradCrittenden is now known as bac === gary_poster|away is now known as gary_poster [13:06] rogpeppe, I found a solution: http://paste.ubuntu.com/6843674/ adding GOPATH to the list of env vars a hook gets [13:06] dimitern: yeah, i thought that might be the problem [13:07] rogpeppe, what i don't get is why it wasn't a problem before? [13:07] axw: are you around? [13:12] dimitern: agreed - i'm not sure either [13:12] dimitern: there's a perhaps-better solution [13:12] rogpeppe, what's that? [13:12] dimitern: just make it so init doesn't panic [13:13] rogpeppe, but it won't solve the issue - it won't find the repo path still [13:13] dimitern: it doesn't need the repo path [13:13] dimitern: it's just re-running the test binary as a hook, right? [13:13] rogpeppe, why don't remove init() altogether then? [13:15] rogpeppe, it seems better to get the absolute path of the charms.go in testing and infer the path from there [13:15] rogpeppe, is that possible? [13:16] dimitern: how would you do that? [13:16] dimitern: i think that removing the init altogether is reasonable though [13:16] rogpeppe, well, in python i'll do it with os.path.realpath(".") or something [13:16] dimitern: the current directory isn't related to where charms.go is [13:17] rogpeppe, no, sorry - i'll use the __file__ magic const that contains the full path to the current file [13:17] rogpeppe, kind of like $0 in bash [13:17] dimitern: define testing.Charms as a function, say func() *charms.Repo [13:17] dimitern: no such thing in Go [13:17] there is some 'build' runtime magic stuff [13:18] dimitern: (and probably a good thing too, as we'd be tempted to do this, and it would be a bad idea :-]) [13:18] jam: we're already doing that [13:18] we use it in testing/charms.go ? something ilke that [13:18] jam: but GOPATH isn't set if we re-exec the binary [13:20] dimitern: something like this perhaps? http://paste.ubuntu.com/6843741/ [13:21] dimitern: that actually makes tests that don't use testing charms slightly faster [13:21] s/tests/suites/ [13:21] rogpeppe: sort of around, what's up? [13:22] axw: just wondering whether it's currently possible to manually bootstrap on a node, but not use the manual/null provider [13:23] rogpeppe: bootstrap or "provision"? [13:23] add-machine? [13:23] axw: bootstrap [13:23] no [13:23] rogpeppe: interested in reviewing https://codereview.appspot.com/58510045/? it still misses tests (have to think about it and how to do best), but it works live. [13:23] axw: right, i thought so [13:23] axw: but needed to check [13:23] rogpeppe: nps. who wants that and why? [13:24] axw: there are currently some people who have duplicated the entire bootstrap logic in shell (and it works... kinda) [13:24] axw: just to get this behaviour [13:24] axw: the specific reason is that they want to bootstrap juju onto a maas controller node [13:24] rogpeppe, it doesn't work [13:24] oh, the installer thing [13:24] rogpeppe, it still fails if GOPATH is not set [13:25] dimitern: doesn't matter [13:25] dimitern: why would that hook code be calling testing.Repo ? [13:25] rogpeppe: why do they not just use the manual provider...? [13:25] no ssh? [13:25] axw: because they want the environment to be using the maas provider [13:25] oh yeah, right [13:26] rogpeppe, it doesn't, it just imports stuff from testing [13:26] dimitern: right, so it's ok then [13:27] rogpeppe, adding GOPATH to that list of vars fixes the issue, i don't want to make it any more intrusive, and i want to land this already [13:27] dimitern: that actually makes it *more* intrusive, no? [13:27] dimitern: because that list of vars is in production code, no? [13:28] rogpeppe, so what - in production it will be empty if you don't have go installed, and even if you do, it won't affect anything [13:29] dimitern: we should not be changing production code for this kind of reason - it's a testing bug and it should be fixed in the testing code [13:29] dimitern: there might even be a less intrusive way, one mo [13:31] rogpeppe, what you proposed with making Repo a func doesn't work, because it's a type and it's exported [13:31] rogpeppe, I have to refactor the whole thing [13:32] dimitern: sorry, i meant Charms [13:32] dimitern: but, i'm pretty sure there's a much less intrusive way [13:32] dimitern: just checking [13:33] rogpeppe, in every test that uses testing.Charms.xxx I have to go and change it so it reads testing.Charms().xxx [13:34] dimitern: sure, but that's one trivial global change: gofmt -r -w 'testing.Charms -> testing.Charms()' [13:34] dimitern: but, again, i think there's a better way [13:34] rogpeppe, I'm listening [13:34] dimitern: i'm still testing whether it's viable [13:40] dimitern: something like this: http://paste.ubuntu.com/6843805/ [13:40] dimitern: those are the only changes necessary, i think [13:41] rogpeppe, I'll try it [13:41] rogpeppe: your last hint to me works fine and can be reviewed ;) [13:41] TheMue: i'll have a look [13:42] rogpeppe: thx. unit tests are still missing, have to think how to do it best === natefinch is now known as natefinch-afk [13:43] TheMue: what's the backward compatibility issue you mention w.r.t. environtag/ [13:43] ? [13:44] TheMue: oh, i see [13:44] TheMue: why not just do that behaviour server side, i.e. no tags means everything? [13:44] rogpeppe: ah, fine. yes, today the command doesn't need an argument. william wanted, that always a tag has to be passed [13:44] TheMue: then you won't need any special casing for the environment tag at all [13:45] TheMue: really? [13:45] rogpeppe: but i have to add an error replay from server-side w/o a tag [13:45] TheMue: seems odd to me [13:45] rogpeppe: yes, really ;) [13:45] TheMue: i don't understand the "error replay" thing [13:45] rogpeppe: I could live with a default choosing the environment too, would make it simpler [13:45] rogpeppe: eh, reply [13:46] TheMue: yeah [13:46] rogpeppe: if you initially call w/o entities or later set empty entities [13:46] TheMue: it simplifies things [13:46] rogpeppe, this works yes, tyvm [13:46] dimitern: np [13:46] rogpeppe: definitely, please add a comment regarding this point [13:47] dimitern: sorry for the pushback, but the GOPATH thing sounded like creeping dependencies, fixing things far away from where the actual problem was [13:47] TheMue: will do [13:47] rogpeppe, yeah, I agree - sorry, but I was trying to land this for a day now and getting really frustrated [13:47] rogpeppe: but filtering is really nice ;) tested it with one and multiple tags [13:47] TheMue: cool [13:48] dimitern: i know the feeling [13:53] rogpeppe: we had a gecko show up inside, which my wife made quite clear that it had to go :) [13:53] jam: lol [13:54] jam: from a position firmly on top of a table? [13:54] they're a bit harder to deal with than mice, as mice don't climb 2m up the wall [13:54] rogpeppe: not quite so bad, but certainly back off a ways and quite vocal [13:54] geckos sound fun [13:54] mgz: +1 [13:56] mgz: they're actually really quite cute (IMO) but I can understand not wanting to catch unexpected motion out of the corner of your eye. It can be a bit disturbing. [13:57] okay, lunch for me, hae proposed a much simplified https://codereview.appspot.com/56560043 along the lines you suggested jam (previous rev has the real cert stuff in) [14:18] so anybody got an update for me on the 1.17.2 world [14:19] as in, are we able to do a 1.17.2 release anytime soon (as measured in hours) [14:19] sinzui: ^^^? [14:21] mgz: are you still going to be able to make it to cfgmgmt camp? [14:22] marcoceppi: yeah, I'll give you and rbasak my travel details [14:22] mgz: cool, robbie will be at fossdem but not cfgmgmt camp, that'll just be us [14:26] arosales: you around? [14:26] mramm, local provider is very broken in trunk. It is not releasable. [14:26] mramm, but when we have a blessed revision, a release takes 4 hours [14:27] juju dev team, is there a way to get just 1.17.1 plus the MAAS bootstrap fix as a branch that we could release as 1.17.2? [14:27] sinzui: so, in the meeting this morning our antipodes were under the impression the only remaining broken things were misunderstandings/script things to fix [14:27] sinzui: from the mailing list thread, I'm not clear what's still borked [14:29] mgz, maybe one thing needs to be fixed. local deploys of 1.17.1/2 have not worked since last Friday, so I have nothing to give me confidence that there is only one thing blocking the fix [14:31] mgz, though We can see that changes to trunk are fixing things. Since yesterday, local deploy fails because wordpress fails to start: http://162.213.35.54:8080/job/local-deploy/ [14:32] We also see this in some of the failing azure tests [14:34]