[02:03] <wallyworld_> thumper: if you have a moment, i'd love a review on https://codereview.appspot.com/16550046/ so i can land it for curtis. no real hurry or anything
[02:13]  * thumper looks at wallyworld_'s work
[02:13] <wallyworld_> \o/ thanks
[02:20] <thumper> wallyworld_: I suggest we skip the package review today
[02:20] <thumper> nothing has been suggested and axwalk isn't around
[02:20] <wallyworld_> sounds good to me
[02:20] <thumper> cool
[02:20]  * thumper goes back to writing emails
[02:21] <wallyworld_> thanks for review
[02:21] <thumper> np
[02:21] <thumper> sometimes I feel that if I don't suggest anything, you'll think I didn't read it
[02:36] <sodre> hi guys.. a quick question...
[02:36] <sodre> maybe not so quick.
[02:37] <sodre> I had a juju debug-hooks session going before I left the office.
[02:37] <sodre> That session got killed somehow. When I tried to start a new debug-hooks session from home, juju complains with the following error:
[02:38] <sodre> duplicate session: nova-cloud-controller/0
[02:38] <sodre> Connection to m1basic-07.vm.draco.metal.as closed.
[02:38] <sodre> ERROR exit status 1
[02:38] <sodre> is there a way to reset that status (possibly without reboot) ?
[02:39] <hazmat> sodre, you can login into the machine and kill the tmux session or reattach
[02:39] <sodre> that was my first thought :)
[02:39] <hazmat> tmux list-sessions shows nothing?
[02:40] <sodre> aha... :) my bad. I looked for byobu.
[02:40] <sodre> okay.. now I'm back! Thanks hazmat!
[02:41] <hazmat> np
[03:09] <bigjools> thumper: are you dealing with vlan stuff on the juju side?
[03:09] <bigjools> I have a vague plan of how I want to do this and need to talk to you guys
[03:17] <hazmat> bigjools, vlans as tag constraints or something more intrisinic?
[03:17] <hazmat> afaicr part of the issue is that provider specific tags are all exposed globally atm
[03:17] <hazmat> with something more intrinsic and specific to maas
[03:18] <hazmat> though i guess could just continue to stick more one use globals on.
[03:18] <bigjools> hazmat: it's all up for discussion. I don't necessarily like tags being used for everything. Not sure it'll work here either.
[03:18] <sodre> guys, is there a good starting point for deploying juju on a Private openstack install? I am having trouble with juju bootstrap
[03:18] <bigjools> hazmat: point is - nodes may be on multiple vlans and juju can ask for a specific vlan
[03:18] <hazmat> bigjools, you can allocate a machine on multiple vlans?
[03:18] <hazmat> gotcha
[03:19] <bigjools> hazmat: only if the machine has multiple network interfaces
[03:19] <hazmat> sodre, let's switch out to #juju and pastebin your $ juju bootstrap -v --debug
[03:19] <sodre> got it
[03:19] <hazmat> sodre, likely its an issue around syncing tools (juju sync-tools)..
[03:19] <sodre> could be.
[03:20] <hazmat> bigjools, right, but its a possibility, why would tags be a bad match here outside of the reuse of the semantic .. the and logic seems similiar
[03:21] <bigjools> hazmat: it could work for sure, we just need to discuss options and see if that is indeed the right way. I hate to jump on the first idea that comes up :)
[03:23] <hazmat> fair enough
[03:23]  * bigjools lunches
[03:25] <hazmat> thumper, kinda of cool albeit not working, local provider and add-machine --to=lxc:1 ... the sub container doesn't  seem to be functional sadly.
[03:26] <hazmat> er.. not really create it appears
[03:46] <hazmat> any openstack/goose devs around?
[03:55] <thumper> hazmat: agreed, we don't support nested containers yet
[03:56] <thumper> bigjools: not sure who will be dealing with it yet
[03:56] <thumper> bigjools: we have to deliver a big hunk of stuff prior to network aware juju
[03:56] <thumper> bigjools: but I do have a big picture understanding of it
[03:57] <thumper> hazmat: please stop telling people to use "-v", if using --debug, verbose is implied
[03:57] <hazmat> really need some keyword calls for help here.. sodre is on #juju but having some issues getting a private openstack installation going .. looks like tool and image metadata issues.. but also notable as an annoyance is the upload of 34mb of tools every bootstrap and the failure of sync-tools
[03:57] <hazmat> thumper, sorry habbit
[03:57] <thumper> :)
[04:00] <bigjools> thumper: ok thanks
[04:00]  * thumper looks at wallyworld_ for synctools problems that hazmat is watching someone have
[04:00] <wallyworld_> hmmm?
[04:01] <wallyworld_> k, /me read #juju
[04:01] <hazmat> wallyworld_, http://pastebin.com/t0H8DVVG
[04:02] <hazmat> wallyworld_, why is it doing uploads like that every bootstrap?
[04:04] <wallyworld_> hazmat: it uploads when it can't find tools
[04:05] <wallyworld_> iow it does a --upload-tools for you
[04:05] <wallyworld_> so the user doesn't have to type it
[04:05] <hazmat> wallyworld_, sure but every bootstrap?
[04:05] <wallyworld_> but if tools are there, it won't upload anything
[04:05] <wallyworld_> only if tools are missing
[04:05] <wallyworld_> once set up correctly, it won't do it
[04:05] <hazmat> umm.. he's done multiple bootstraps.. that's just his most recent run
[04:06] <wallyworld_> in that case, theres a problem finding tools
[04:06] <hazmat> yeah..
[04:06] <wallyworld_> otherwise it wouldn't upload
[04:06] <hazmat> wallyworld_, interestingly sync-tools through an error.. http://pastebin.com/BJJUrasS
[04:06] <hazmat> wallyworld_, way past EOD here.. hopefully you can help him out.. going to try and finish up some ODS demo work.
[04:07] <wallyworld_> huh, never seen that eror before
[04:07] <wallyworld_> ok, no problem
[04:07] <wallyworld_> thanks for helping
[04:21] <jam> wallyworld_: I'm off for a run, but the "invalid literal t" stuff looks like a bug in goose, search for something about json marshalling
[04:21] <jam> specificalyl, we make a list request
[04:21] <jam> and expect to get JSON back
[04:22] <jam> but end up getting just a plain listing
[04:22] <jam> which doesn't JSON.Unmarshal well
[04:23] <wallyworld_> oh, ok
[05:37] <bigjools> rvba: can you think of why the response to Azure's GetOSImages would get truncated inside the Go code? curl works ok.
[05:38] <bigjools> rvba: wallyworld_ is debugging and it only reads ~16k of a 159k response
[05:39] <rvba> bigjools: sounds fishy… I don't recall seeing that… let me have a quick look at the code and see if anything comes back…
[05:39] <wallyworld_> i blame ms... or go
[05:39] <jam> rvba: sounds a lot like an File.Read() rather than ioutil.ReadAll()
[05:40] <wallyworld_> jam: ioutil.ReadAll() fails the same way, so i have done my own read loop to try and see where it gets to
[05:41] <jam> wallyworld_: I wonder if it is getting something like EINTR and not retrying
[05:41] <wallyworld_> jam: could be. all i know is that reading from response.Body fails
[05:41] <wallyworld_> it is expecting 160kb response (which is absurd in itself but that's another story)
[05:42] <jam> wallyworld_: sure. I haven't seen it myself, I just know that it is easy to do a object.Read() when you actually want ioutil.ReadAll()
[05:42] <wallyworld_> but the connection is closed after reading 16k
[05:42] <jam> wallyworld_: there can certainly be a lot of OS Images ...
[05:42] <wallyworld_> jam: have you seen the xml. sooooo verbose
[05:42] <wallyworld_> all sorts of crap in there besides image details
[05:46] <rvba> wallyworld_: could this be related somehow to the change made in revision 226? (the "request.Close = true" thingy)?
[05:46] <wallyworld_> rvba: no, that change would have improve the situation if anything afaik
[05:46] <wallyworld_> cause it would have stopped go from reusing connections which may have been closed
[05:47] <rvba> I see.
[05:56] <wallyworld_> time to pick up the kid from school. will have to continue debugging when i get back
[08:30] <rogpeppe> mornin' all
[10:07] <dimitern> morning all
[10:08] <mgz> mornin'
[10:14] <dimitern> mgz, did we change the standup time to be 1h earlier? or my tz is wrong again in the calendar
[10:15] <mgz> hm, we just switched to summer time
[10:17] <mgz> so, I think it should have moved an hour forwards for both of us
[10:19] <dimitern> what's your local time now?
[10:19] <mgz> 10:20
[10:20] <dimitern> hmm, so the clock seems right
[10:20] <dimitern> but on the calendar I can see the standup at 11:45, whereas before it was at 12:45
[10:21] <mgz> right, I think the event is pinned on gmt, so, moves an hour with summer time ending
[10:21] <wallyworld> rvba: you around?
[10:21] <rvba> wallyworld: yep
[10:22] <wallyworld> rvba: running the azure test example utility, any idea why i get "The versioning header is not specified or was specified incorrectly"
[10:22] <wallyworld> gwacl/example/management/run.go
[10:23] <wallyworld> i had to hack out the retrier to stop the network connection close issue
[10:23] <rvba> wallyworld: hum, IIRC it should "just work", let me have a look.
[10:23] <rvba> wallyworld: what did you have to do?
[10:23] <wallyworld> instead of using the retrier: retrier := session.retryPolicy.getForkedHttpRetrier(session.client)
[10:24] <wallyworld> i just did a session.client.Do(req) directly
[10:24] <wallyworld> that allows the images to be listed ok
[10:24] <rvba> Weird.
[10:24] <wallyworld> there's a separate issue with the conection getting closed
[10:24] <wallyworld> so that hack just gets past that bit
[10:25] <wallyworld> i think there may be a real issue with the header version
[10:25] <wallyworld> since juju bootstrap also fails with that error on azure all of a sudden
[10:25] <rvba> Maybe the version we use is not supported anymore…
[10:25] <rvba> That would be weird… but anything is possible.
[10:25] <wallyworld> yeah, maybe
[10:25] <wallyworld> thanks for looking :-)
[10:25] <rvba> I'm still looking :)
[10:26] <wallyworld> so we use "x-ms-version", "2013-03-01"
[10:26] <wallyworld> is that the header it is refering to?
[10:26] <rvba> Yes
[10:26] <wallyworld> ok, maybe there's anew one? where do we look to find out?
[10:27] <jam> mgz, dimitern: Generally we've left the meeting at GMT so that when we have 5 different daylight switchover times triggering asynchronously we don't have to bump it around. But we can discuss changing the time if it works better for people.
[10:27] <rvba> Somewhere in the documentation.  But the point of using version headers like this is exactly to avoid trouble when the API changes :).
[10:27] <jam> My guess is that when Nate changes TZ in a couple weeks we might all want to move earlier
[10:27] <wallyworld> yeah, i would have thought so too
[10:27] <jam> but we'll have to check with wallyworld if that is going to work
[10:28] <wallyworld> might work :-)
[10:28] <jam> wallyworld: don't you switch on to daylight savings time? or is Queensland DST agnostic
[10:28] <wallyworld> sadly QLD is retarded. no DST here :-(
[10:29] <rvba> http://msdn.microsoft.com/en-us/library/windowsazure/gg592580.aspx says "2013-03-01" is still the most recent version.
[10:29] <wallyworld> hmmm.
[10:29] <wallyworld> maybe there's an azure bug
[10:29] <dimitern> jam, nah, it's alright I just have to remember it's 1h later
[10:29] <jam> rvba: it also says "last updated November 19, 2012"
[10:29] <rvba> jam: indeed :)
[10:29] <dimitern> so are we using git now?
[10:29] <rvba> Let me try to run the example script…
[10:30] <jam> dimitern: not as in "today we commit with git" but as in "we'll be working on migrating now"
[10:30] <wallyworld> rvba: ok. let me know if you have to hack out the retrier
[10:30] <dimitern> jam, i see
[10:30] <jam> I'm trying to decide my personal involvement.
[10:30] <wallyworld> rvba: my command line was: go run -a run.go -cert=/home/ian/.juju/azure.pem -location="West US" -subscriptionid="<blah>" -pause=true
[10:30] <dimitern> it would be useful to have a "crash course" comparison between common bzr and git commands
[10:30] <jam> dimitern: http://www.git-tower.com/blog/git-cheat-sheet-detail/
[10:31] <dimitern> jam, cheers
[10:31] <rvba> wallyworld: ta
[10:31] <jam> dimitern: http://doc.bazaar.canonical.com/migration/en/survival/bzr-for-git-users.html is the reverse
[10:31] <jam> but it probably has the correlations
[10:31] <wallyworld> jam: i had a quick look at git today - it is weird. git pull doesn't actually update your branch
[10:31] <wallyworld> and you get the whole repo whether you want it or not
[10:32] <jam> wallyworld: so git has a concept of a bunch of local branches that track remote branches
[10:32] <jam> wallyworld: and it doesn't support remote ops like "log"
[10:32] <jam> so you *have* to pull locally
[10:32] <jam> to be able to see the history
[10:32] <wallyworld> :-(
[10:32] <jam> wallyworld: and under that circumstance, it doesn't make sense to always merged
[10:32] <jam> merge
[10:32] <wallyworld> and you need to checkout after pull to see the changes?
[10:32] <jam> wallyworld: "git merge" will also "fast forward" like "bzr pull' unless you pass "--no-ff" or configure it to disable that.
[10:32] <jam> wallyworld: well, you can "git log" locally
[10:33] <jam> there is probably a "bzr cat" equivalent.
[10:33] <rvba> wallyworld: the command just hangs, no output, no nothing :/
[10:33] <wallyworld> i wonder how to set up a lightwieght checkout scenario
[10:33] <wallyworld> rvba: really? hmmmm
[10:33] <rvba> Oh wait, it crashed after a while
[10:33] <dimitern> so nothing in git like qbzr?
[10:33] <jam> wallyworld: each "git repository" is a working tree + repository + multiple branches
[10:34] <rvba> With "The versioning header is not specified or was specified incorrectly."
[10:34] <jam> so you generally just have a repo and switch between the branches in there
[10:34] <jam> essentially the colocated version of bzr by default
[10:34] <wallyworld> rvba: right, that's what i see, except i have the conn closed issue too
[10:34] <rvba> Gosh, what a mess.
[10:34] <wallyworld> yeah :-(
[10:34] <natefinch> morning all
[10:35] <jam> wallyworld: I know it is possible to play with ENV args like GIT_REPO to allow you to say the history is "over there". And I've seen comments about other bits of sharing histories between working trees.
[10:35] <jam> wallyworld: but AFAICT the *common* behavior is that you have just one tree that you switch between branches and they all share the same repo
[10:35] <wallyworld> jam: ok. seems like we need to work out best practice for bzr refugees
[10:36] <wallyworld> from what i understand, seems like a shame though it forces you to get the *whole* repo
[10:36] <jam> wallyworld: well, you close/archive/rename branches when you don't need them anymore
[10:36] <jam> just like in bzr, they are just pointers into the revision graph, so they are "cheap"
[10:36] <jam> (gits actual objects are cheaper than bzr's because of extra files bzr keeps around for each branch)
[10:37] <wallyworld> but unlike bzr, you get the whole repo and potentially waste lots of disk space if there are lots of blos etc
[10:37] <wallyworld> i wonder if that can be avoided
[10:37] <jam> wallyworld: bzr pull gets the whole history
[10:37] <jam> not sure what you mean about waste
[10:37] <wallyworld> for that branch
[10:37] <wallyworld> easier to explain verbally :-)
[10:37] <jam> wallyworld: so again, usually you would work in the same "working tree" and just pull the branches next to yours
[10:38] <wallyworld> right. but i was told if you get a working tree, you also get all other working tree for all other branches in the repo regardless if you want them or not
[10:39] <jam> wallyworld: "git clone" defaults to hardlinking the archive
[10:39] <natefinch> wallyworld: yes, you do.  But you generally wouldn't check out multiple copies of the same project
[10:39] <jam> wallyworld: no other working trees
[10:39] <jam> just branch pointers
[10:39] <wallyworld> right. but if a repo contains project foo and project bar and you just want foo, you are forced to get bar too
[10:40] <wallyworld> i was hoping to avoid that
[10:40] <natefinch> wallyworld: yes, well, don't put two projects in the same repo :/
[10:40] <wallyworld> ok. seems like many people do though. let's hope we don't
[10:40] <jam> wallyworld: sure. though as nate says, they don't generally share repositories for projects that don't share ancestry.
[10:40] <jam> wallyworld: there is "git clone --shared"
[10:40] <jam> and --local, --bare, --mirror
[10:40] <wallyworld> i'm just going y what i've read/seen
[10:40] <jam> and shallow clones with --depth
[10:40] <jam> and
[10:41] <jam>  --separate-git-dir
[10:41] <rvba> wallyworld: what was the symptom of the other issue?  "panic: use of closed network connection"?
[10:41] <jam> wallyworld: so you can probably do whatever the fsck you want
[10:41] <wallyworld> rvba: yes
[10:41] <wallyworld> jam: great :-)
[10:41] <wallyworld> jam: still trying to grok the best approach to suit my workflow
[10:41] <natefinch> jam, wallyworld : except most of the time it's best to just get the whole repo and switch branches as you need to.  That actually works perfectly with Gopath, that assumes one spot on your hard drive for a project
[10:42] <jam> wallyworld: though as you mention, clone copies all branches by default, so mixed project repositories will generally cause clone to copy multiple ancestries. In *git's* world view there is just one global ancestry that happens to have disjoint pieces
[10:42] <rvba> Oh wait, it crashed after a while
[10:42] <rvba> wallyworld: I don't see it with revision 225.  With revision 226, I get that "use of closed network connection" error: http://paste.ubuntu.com/6329291/
[10:42] <jam> natefinch: to be fair, the gopath model is particularly bad, so I'm not super keen that another tool does it, too :)
[10:43] <wallyworld> rvba: that is interesting indeed
[10:43] <jam> natefinch: prior to this, I always had 3 checkouts of a project I was working on. 1 was pristine trunk, 1 was the thing I was actively working on, and 1 was for code review that came back that needed tweaks without wanting to interrupt my current work
[10:43] <wallyworld> rvba: there's another way to try and work around Go's braindead http issue. i might try that
[10:43] <jam> natefinch: it is nice to be able to just fire up your text editor to see how it was done in trunk
[10:44] <natefinch> jam: you can still do that. They're just different branches, and switching between them is super fast in git
[10:44] <wallyworld> rvba: so that still leaves the versioning issue :-(
[10:44] <jam> natefinch: it is fast in bzr, too, it is *still* nice to not step on what I'm working on to see what it is like in another branch
[10:44] <jam> natefinch: note that I said *3* not N for all the branches I work on.
[10:44] <rvba> wallyworld: yep.  Let's see if the request looks sane.
[10:45] <jam> anyway, stand up time for those who are joining
[10:45] <jam> https://plus.google.com/hangouts/_/calendar/am9obi5tZWluZWxAY2Fub25pY2FsLmNvbQ.mf0d8r5pfb44m16v9b2n5i29ig
[10:45] <jam> rogpeppe, dimitern, wallyworld, natefinch: ^^
[10:45] <jam> mgz: ^^
[10:45] <mgz> ta
[10:45] <wallyworld> rvba: thanks so much for looking. i'll check back after our standup
[10:45] <dimitern> so it is 1h earlier then
[10:45] <rvba> wallyworld: np
[10:53] <wallyworld> sinzui: not sure if you are online. if you are, i have some info for you
[11:03] <rvba> wallyworld: it's the deletion of the affinity group request that fails…
[11:03] <wallyworld> rvba: ok. do we do that during bootstrap also?
[11:04] <rvba> wallyworld: probably not
[11:04] <wallyworld> bootstrap fails with the "same" error
[11:04] <wallyworld> well, it did for me and curtis
[11:05] <rvba> wallyworld: well, maybe bootstrap does something similar (for some definition of "similar")
[11:05] <wallyworld> likely
[11:05] <wallyworld> i wonder why that particular api call fails
[11:05] <rvba> wallyworld: it would be worth knowing which API call fails exactly when running bootstrap.
[11:05] <rvba> Might help us pinpoint the problem.
[11:06] <rvba> Because the request seems okay to me, the api header is set.
[11:06] <wallyworld> rvba: i'll try and find out. but likely not tonight. too tired
[11:06] <wallyworld> rvba: unless ms changed something at their end?
[11:06] <wallyworld> wrt versioning?
[11:07] <wallyworld> nothing would surprise me
[11:07] <rvba> wallyworld: I tried using another version for that request, no dice
[11:07] <wallyworld> sigh
[11:07] <wallyworld> it shouldn't be this hard :-(
[11:34] <jam> natefinch: just wanted to poke, you'll be going of DST in another couple weeks, right? Is that going to impact you for the standup?
[11:35] <jam> wallyworld: Replacing the ginger beer with ginger wine leads to a particularly  lethal drink known as a Turbo Dark 'N' Stormy; a favourite tipple of Pembroke College Boat Club (Oxford)
[11:35]  * wallyworld makes a note to try that tomorrow
[11:35] <natefinch> jam: yeah, I was going to bring that up.  The current standup time will soon be 5:45am for me, which is somewhat less than awesome
[11:35] <jam> natefinch: sounds pretty awesome for us. You were complaining that your daughter was waking up before you
[11:36] <jam> this way we get focused Nate time :)
[11:36] <natefinch> jam: ha yeah
[11:36] <jam> natefinch: so what time will work for you?
[11:37] <natefinch> jam: we could do a half hour earlier, that would be 6:15 my time, whcih is enough time for me to get up and make coffee, which is really all I need before the standup :)  Depends on what works for other people
[11:37] <jam> 'earlier' ?
[11:37] <natefinch> er later
[11:38] <wallyworld> i'll be totally drunk by then :-)
[11:38] <jam> wallyworld: not sure if that's a problem :)
[11:38] <wallyworld> that's 21:15 for me, but should work
[11:38] <wallyworld> jam: yeah, how would you know the difference, right?
[11:39] <jam> wallyworld: whmaobaou whaoeu baoeuwhaoeunwhou, might give it away
[11:39] <wallyworld> maybe :-)
[11:39] <jam> wallyworld: though, I guess when you got tipsy in SFO you didn't seem to mumble
[11:40] <wallyworld> i only had a couple of drinks, i'm not a 2 pot screamer :-)
[11:40] <wallyworld> hmmm. do you know what a 2 pot screamer is in the US?
[11:40] <wallyworld> maybe that's an australian expression
[11:41] <wallyworld> a "pot" in australia is a standard 285ml glass of beer
[11:41] <natefinch> wallyworld: never heard of a 2 pot screamer
[11:41] <wallyworld> maybe it makes sense now that i've explained what a "pot" is ?
[11:42] <natefinch> wallyworld: I inferred what it meant from context :)
[11:42] <jam> wallyworld: essentially a "cheap date" :)
[11:42] <wallyworld> yeah :-)
[11:42] <jam> natefinch: so if we chip in to buy you a coffee maker with built in grinder and timer we can keep the current time ? :)
[11:42] <natefinch> jam: sure, but I get to pick the model ;)
[11:42] <wallyworld> i don't mind if it is later
[11:42] <jam> natefinch: I think we can probably move it by 30min. It may mean that I get interrupted more when my son gets home, but I think we can work it out.
[11:43] <jam> natefinch: my wife and I had one, the main problem we had was that because the grinder is exposed (to let the grinds into the brewing section), it was *really* loud, and the grinding area needed to be cleaned after every run because the steam would get in there and you end up with soggy grounds
[11:43] <natefinch> jam: we can give it a try and change things if it's not working.   5:45 is less of a horrible time for me than most people I think.
[11:44] <jam> natefinch: sure, though it is the *daily* standup ,so I'd like to be at a reasonable time as much as possible
[11:45] <natefinch> jam: we could also move it even later, though that would probably mean Ian can't go
[11:45] <jam> natefinch: so we should definitely do it, you say ? :)
[11:46] <natefinch> jam: haha... it would mean about 50% less whining during the standup ;)
[11:47] <mgz> he doesn't drink that much wine
[12:55] <natefinch> interesting - https://github.com/juju
[13:02] <sinzui> Thank you for the email wallyworld
[13:30] <hazmat> jam, awesome stuff re scale testing.. have always enjoyed these write ups (+maas, +bzr back in the day)
[13:44] <hazmat> one curious bit was the 5s pump on the unit agent api for watch events, was wondering where that came from
[13:45] <hazmat> the all watcher client impl has something similiar but thats based on a periodic update and push, is that the same behavior/impl for the agent api?
[13:53] <sinzui> Hi devs, we have a critical bug in progress that I want you all to be aware of. Any insight into this issue will be appreciated. I currently think this issue is more than just blocking my 1.16.1 release, I think it is a bug that needs fixing for 1.16.1: https://bugs.launchpad.net/juju-core/+bug/1246320
[13:53] <_mup_> Bug #1246320: Azure bootstrap fails: versioning header is not specified <azure> <bootstrap> <juju-core:In Progress> <https://launchpad.net/bugs/1246320>
[14:05] <rogpeppe> lunch
[14:10] <rogpeppe> hazmat: all watchers use the same underlying poller (implemented by the state/watcher package)
[14:15] <jamespage> sinzui, this might sound like an odd question but the 1.16.1 release will just be juju-core changes right?
[14:15] <jamespage> not a re-sync of trunk of deps as well
[14:19] <sinzui> jamespage, https://launchpad.net/juju-core/+milestone/1.16.1is just bug fixes that are not entangled with other libraries. The fixes were backported from trunk
[14:20] <jamespage> sinzui, goody
[14:20] <jamespage> just making sure
[14:20] <jamespage> sinzui, dependency management needs to be a key focus for 14.04 if we want to get juju-core into main
[14:22] <sinzui> jamespage, understood.
[14:31] <jamespage> sinzui, I'm going to take a look at the state of the golang tooling for packaging early in cycle
[14:31] <jamespage> if its good enough I'd like to push out dependencies into their own packages
[14:46] <sinzui> jamespage, I am considering releases doe gwacl, goose, and gomaasapi. Would you want them as packages in 14.04?
[14:50] <natefinch> sinzui, jamespage: I would find it odd to have random Go packages as packages for apt-get.  They're basically useless on their own, and don't represent an actual usable application, unlike juju-core.  But maybe I'm misunderstanding what's being proposed.
[14:52] <sinzui> natefinch, it is odd. We tend to bend things like java to debian package management. debian maven for example strives to collect java libs in a tree that is shared for debian package management
[14:53] <sinzui> natefinch, I don't know the right path for go, but since it can wrap c libs, it cannot assume all packages are the same
[14:54] <jamespage> sinzui, not yet
[14:55] <sinzui> PS I had zero success with debian and maven for 13.04 and don't want to try anything like that again
[14:58] <arosales> mgz, if you have some time today could you take a look at the merge request @ https://code.launchpad.net/~dstroppa/juju-core/add-joyent-provider/+merge/189360
[14:59] <arosales> mgz, or possibly get it to the correct person for review
[15:01] <mgz> arosales: sure, that sounds reasonable
[15:01] <mgz> ah, whoops, I fixed bug 1243861 but didn't link the bug
[15:01] <_mup_> Bug #1243861: juju should add the cloud-archive repository differently <juju-core (Ubuntu):Confirmed> <https://launchpad.net/bugs/1243861>
[15:02] <arosales> mgz, thank you sir
[15:02] <mgz> ah, it's an ubuntu bug, that's why I missed it
[15:03] <sinzui> mgz, we have a fix for that bug going in 1.17 ?
[15:03] <mgz> sinzui: it's in 1.16.1 too
[15:03] <sinzui> fab.
[15:03] <mgz> linking up now
[15:04] <mgz> er... or, actually not merged on 1.16 branch yet, just proposed
[15:07] <mgz> submitted.
[15:08] <mgz> knew I was forgetting some branch as well as the openstack one
[15:40] <sinzui> Hi juju dev's I have updated https://bugs.launchpad.net/juju-core/+bug/1246320 . juju 1.14.1 is also broken with the same error. I think azure or streams has changed. I am looking for insight into the root cause
[15:40] <_mup_> Bug #1246320: Azure bootstrap fails: versioning header is not specified <azure> <bootstrap> <juju-core:In Progress> <https://launchpad.net/bugs/1246320>
[15:45] <mgz> does seem like an azure change, we discussed it briefly in the standup
[15:46] <mgz> I guess the next step is fiddling with the version info gwacl sends and seeing what happens
[15:46] <mgz> or just escalating to microsoft and asking what changed
[15:55] <arosales> +1 on the importance of sinzui's bug
[15:55] <arosales> bug 1246320
[15:55] <_mup_> Bug #1246320: Azure bootstrap fails: versioning header is not specified <azure> <bootstrap> <juju-core:In Progress> <https://launchpad.net/bugs/1246320>
[15:55] <mgz> arosales: have you got anyone on the azure end you can bother?
[15:55] <arosales> mgz, sure if its a platform issue I can ping microsoft
[15:56] <mgz> arosales: given it's failing on 1.14, 1.16, and trunk, when the exact same code was working before, something their side has changed
[15:57] <mgz> it may be that we now need to do something different, but it would be good to find out what they changed
[15:57] <arosales> mgz, if you can point me to an error message I can pass it their way
[15:57] <arosales> error message that would indicate azure is failing as they can still provision instances
[15:58] <arosales> be right back
[16:00] <mgz> is's basically MissingOrIncorrectVersionHeader, but we may still need to trace exactly what we're sending in the case it fails
[16:01] <mgz> hm... I have an idea
[16:18] <sinzui> mgz, I discussed azure APIVersion with utlemming in #juju.
[16:19] <sinzui> mgz, I just tried a global change of the APIVersion is gwacl by setting the date to today. All I accomplished was making bootstrap fail faster
[16:19] <rogpeppe2> dimitern: how's your knowledge of charm upgrade semantics?
[16:19] <sinzui> I will try some alternate dates
[16:20] <mgz> sinzui: too many jujus. I see that log now. did you also catch the earlier discussion here on it between wallyworld and rvba?
[16:20] <sinzui> I read it without full comprehension
[16:21] <mgz> well, rvba tried an alternate date and it didn't help
[16:23] <sinzui> mgz, fab, I will stop this line of inquiry
[16:24] <mgz> I do still worry about the whole on-some-requests-but-not-others aspects
[16:25] <mgz> could still be our http stack screwing up I guess, would be nice to wireshark and confirm the exact bits we send for the working vs non-working api calls
[17:00] <dimitern> rogpeppe, good
[17:01] <rogpeppe> dimitern: i'm trying to upgrade a charm that's in an error state, but it doesn't seem to be upgrading
[17:01] <rogpeppe> dimitern: from my reading of the logic in ModeHookError, it should allow upgrading in that case
[17:18] <rogpeppe> dimitern: ah, i get it now - i didn't use the upgrade-charm --force flag
[17:18] <dimitern> rogpeppe, right!
[17:19] <dimitern> rogpeppe, also, there's resolve --retry on the unit that's in an error state
[17:19] <rogpeppe> dimitern: yeah, i knew about that
[17:19] <rogpeppe> dimitern: i was just confused by the fact that the new charm had been downloaded, but the unit wasn't using it
[17:26] <rogpeppe> cool, it all works now.
[18:15] <natefinch> rogpeppe, mgz, dimitern:  my instance type constraint: https://codereview.appspot.com/14523052/
[18:47] <rogpeppe> natefinch: sorry, i'll have a look in the morning'
[18:47] <natefinch> rogpeppe: no problem
[18:48] <rogpeppe> natefinch: you might be interested to have a look at http://godoc.org/launchpad.net/juju-utils/cmd/gocharm
[18:48] <rogpeppe> natefinch: it's in an early stage - there are a few things that i know will change
[18:48] <natefinch> rogpeppe: nice, I'll definitely check it out
[18:48] <rogpeppe> natefinch: but the basic structure seems ok, i think
[18:49] <rogpeppe> natefinch: you'll also want to look at http://godoc.org/launchpad.net/juju-utils/hook
[18:49] <rogpeppe> natefinch: and there's an example charm in launchpad.net/juju-utils/cmd/gocharm/example-charms
[18:49]  * rogpeppe is now done for the day
[18:49] <natefinch> rogpeppe: that's awesome.  Nice work
[18:51] <rogpeppe> natefinch: ta
[18:51] <rogpeppe> g'night all
[19:18] <bac> sinzui: would you have a moment for a hangout?
[19:19] <sinzui> I do
[19:26] <sinzui> arosales, juju-devs, I am contemplating freezing 1.16.1 and doing a release. I don't know how long it will take to sort out the azure issue, but I would rather do a 1.16.2 tomorrow then delay the maas and lxc fixes
[19:28] <arosales> sinzui, and it is looking like the azure issues may be platform dependent
[19:28] <arosales> sinzui, thanks for getting the xml rpc capture to utlemming.
[19:29] <arosales> sinzui, +1 on 1.16.1 from me and if needed a 1.16.2 for any azure fixes _if_  it is not the platform
[19:29] <sinzui> okay. I will proceed with the tests I setup yesterday, then deliver a tarball.
[19:36] <mgz> sinzui: going ahead with 1.16.1 now seems reasonable to me
[20:08] <thumper> morning folks
[21:30] <wallyworld_> sinzui: hi, so just to confirm, someone is talking to microsoft about the azure api issue?
[21:34] <sinzui> wallyworld_, I believe utlemming will
[21:34] <wallyworld_> ok, great
[21:35] <sinzui> He was in was delivering new images for them tomorrow
[21:35] <wallyworld_> i'll try and ensure the http connection issue (if i can confirm it for sure) is sorted
[21:35] <wallyworld_> btw, i'm almost 100% sure there's no streams issue here at all
[21:55] <thumper> finally wrote up loggo http://how-bazaar.blogspot.co.nz/2013/10/loggo-hierarchical-loggers-for-go.html
[22:06] <hazmat> thumper, ping..
[22:06] <thumper> hazmat: otp now
[22:06] <hazmat> thumper, any idea why a machine agent would just ignore containers deployed to it..
[22:07] <thumper> got more info?
[22:07] <hazmat> nothing in the machine log of note.. and a bunch of containers attached which are pending
[22:07] <hazmat> thumper switching channels
[22:12] <mramm> thumper: your blog post is now up on hacker news: https://news.ycombinator.com/item?id=6643805
[22:50]  * thumper -> gym
[22:56] <hazmat> jujucore.. there's a panic dump in cloudinit from sodre on #juju
[22:56] <hazmat> setting mongodb password
[22:57] <hazmat> for posterity http://paste.ubuntu.com/6332775/
[23:04] <wallyworld__> hazmat: is there a bug raised?
[23:05] <hazmat> wallyworld__, dunno.. wasn't there some size issue with password and mongodb?
[23:06] <wallyworld__> seems so, i was just wondering if a bug was raised so we could track it. i'll search and raise one if not done already
[23:07] <hazmat> cool
[23:12] <sodre> for the record, when that ocurred the admin-secret was set to : 'super-secret'
[23:13] <hazmat> solid!
[23:21] <wallyworld__> bug 1246517 fwiw
[23:21] <_mup_> Bug #1246517: user edited admin-secret causes panic in cloud-init <bootstrap> <juju-core:Triaged> <https://launchpad.net/bugs/1246517>
[23:24] <sinzui> thumper, where is the equivalent of the all-machine-log for local-provider environments?
[23:30] <sodre> _mup_ even with a long password it still crashes for me.
[23:31] <hazmat> sinzui, its in $juj_home
[23:38] <wallyworld__> sinzui: btw, juju-core trunk has the --public option now for tools sync. so everything should be in place for metadata generation with mirrors
[23:40] <sinzui> okay, thank you wallyworld__
[23:41] <wallyworld__> let me know if i can do anything else, or if there are issues etc etc
[23:41] <sinzui> wallyworld__, azure did change API yesterday. They believe it is backwards compatible...
[23:41] <wallyworld__> not for us it seems?
[23:41] <sinzui> our apps would disagree
[23:41] <wallyworld__> yep
[23:41] <wallyworld__> do we have details?
[23:41] <wallyworld__> is there a fix i can work on?
[23:41] <bigjools> \o/
[23:41] <hazmat> so getting reports of lxc broken on maas in two different envs..
[23:42] <hazmat> if anyone can help debug .. see adam_g on #juju/can
[23:43] <sodre> hazmat, do you mean trying to add an lxc unit to a MAAS provisioned machine ?
[23:44] <bigjools> thumper did all the work for that
[23:44] <sinzui> thank you hazmat
[23:45] <sinzui> wallyworld__, they need URI, HTTP headers and Request body. my gwacl hack captured the body. I will hack gwacl some more to provide the rest
[23:46] <sinzui> wallyworld__, I need to finish 1.16.1 tarball first
[23:46] <wallyworld__> or i could help
[23:46] <sinzui> I cannot get local provider to work. I don't see any difference between 1.16 and 1.16.1
[23:46] <wallyworld__> what symptoms?
[23:46] <sinzui> wallyworld__, I will show you the patch
[23:49] <hazmat> sinzui, machine log?
[23:49] <hazmat> sinzui, lxc-ls output
[23:49] <hazmat> sodre, yes
[23:49] <hazmat> sodre, does it work for you?
[23:49] <sinzui> wallyworld__, http://pastebin.ubuntu.com/6332990/
[23:49] <sodre> nope. When I try to add an lxc unit it just hangs.
[23:49] <sodre> well, it keeps waiting for them to come up.
[23:50] <wallyworld_> sinzui: ok, i can get the extra info and put in bug report
[23:51] <sinzui> hazmat, The machine log : http://pastebin.ubuntu.com/6332998/
[23:51] <sinzui> hazmat, lxc-ls: curtis-local-machine-1  curtis-local-machine-2  lts-p
[23:53] <sinzui> hazmat, also, I did do an lxc-destroy for the two machines from previous tests because destroy-environment left them behind in /var/lib/lxc
[23:56] <hazmat> sinzui, its been working for me.. bbiam
[23:58] <hazmat> sinzui, lxc-destroy directly should be avoided if possible.. local provider leaaves some turds in /etc/lxc/autostart and /var/lib/juju otherwise
[23:58] <hazmat> not sure if it bookeeps enough to be smart about that
[23:59] <hazmat> sinzui, so lxc-ls --fancy shows them as running?
[23:59] <hazmat> sinzui, i assume there stuck in pending in status?
[23:59] <hazmat> sinzui, so if their running from --fancy.. log into one and pastebin /var/log/cloudinit-output.log