[01:03] <alexisb> hey wallyworld dont mind me messing with the api doc :)
[01:03] <wallyworld> alexisb: sure, go for it :-) it's still very much WIP
[01:03] <alexisb> yep, just reviewing things before I start linking them back
[01:04] <wallyworld> alexisb: we'll want to add a server person for cloud init work
[01:04] <alexisb> wallyworld, ack
[01:04] <alexisb> what about landscape?
[01:04] <alexisb> should I add dean
[01:05] <wallyworld> yeah, still figuring out exactly what we want in that area, but we should make sure there's an overall consensus
[01:05] <alexisb> ack
[07:56] <voidspace> morning all
[07:56] <voidspace> morning all
[09:37] <perrito666> morning
[09:58] <aznashwan> hey; could I please get a quick review on http://reviews.vapour.ws/r/1486/?
[10:01] <aznashwan> note that Gabriel does a good job of explaining the reasoning behind the package's addition in response to Casey's comment (first one).
[10:03] <natefinch> aznashwan: hey, that's a really neat and useful package for windows devs
[10:04] <natefinch> aznashwan: I guess there wasn't one already written anywhere?
[10:05] <aznashwan> natefinch: I had that written a while back but we only just got round to refactoring the Windows service package and its usecase popped up there...
[10:31] <mup> Bug #1195757 changed: Package Juju for Windows <improvement> <windows> <juju-core:Fix Released> <https://launchpad.net/bugs/1195757>
[14:01] <katco> wwitzel3: natefinch: stand up
[14:11] <mup> Bug #1449050 was opened: Juju get returns a yaml file that can't be passed to set <juju-core:New> <https://launchpad.net/bugs/1449050>
[14:35] <mup> Bug #1449054 was opened: Intermittent panic: rescanned document <ci> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1449054>
[14:51] <aznashwan> natefinch: ping
[15:14] <aznashwan> alexisb: ping
[15:14] <alexisb> hey aznashwan whats up?
[15:15] <aznashwan> alexisb: fresh out of holiday; thanks
[15:15] <aznashwan> alexisb: just wanted to point out this thing: http://reviews.vapour.ws/r/1486/
[15:16] <aznashwan> alexisb: @natefinch pointed out that we should put this in a separate package
[15:17] <aznashwan> alexisb: I need to know who'd be the person who I'd best talk to about it...
[15:19] <alexisb> aznashwan, are you in malta?
[15:19] <aznashwan> alexisb: nope; back home.
[15:19] <alexisb> ack
[15:19] <alexisb> natefinch should be able to help
[15:20] <alexisb> aznashwan, I can follow up with him when he is back online
[15:20] <aznashwan> alexisb: much obliged :D
[15:58] <natefinch> alexisb. aznashwan: back
[16:02] <alexisb> natefinch, aznashwan is looking for guidance with separating windows securestrings into its own package (based on you suggestion)
[16:02] <natefinch> alexisb: I agree with my suggestion ;)
[16:02] <alexisb> lol
[16:03] <alexisb> I trust you can help guide him as well :)
[16:03] <natefinch> alexisb: sure thing
[16:03] <alexisb> thanks natefinch
[16:09] <aznashwan> natefinch: ping
[16:09] <natefinch> aznashwan: howdy
[16:10] <aznashwan> natefinch: so; any idea how we're going to proceed?
[16:11] <mup> Bug #1449123 was opened: EnvJujuClient24._shell_environ does not correctly handle juju_home <juju-core:Triaged> <https://launchpad.net/bugs/1449123>
[16:11] <aznashwan> natefinch: The thing itself is really tiny; so I don't personally see it as a worthy package in it of itself...
[16:12] <natefinch> aznashwan: it's not about the size, it's about a unit of functionality
[16:13] <aznashwan> natefinch: Gabriel at one time suggested I either keep it on me (currently it's been made private; vanity imports aren't that cool); or put it in on Cloudbase
[16:13] <natefinch> aznashwan: anything which is worthy of its own package is worthy of its own repo, if you want other people to use it.  And we definitely want other people to use it, because open source etc.
[16:14] <natefinch> aznashwan: did you build it on your own time, or on cloudbase's time?
[16:14] <aznashwan> cloudbase's time/my own I'm not sure I was an unpaid intern :))
[16:14] <natefinch> aznashwan: if you did it while "at work" then it belongs to cloudbase (assuming they do the same thing most companies do)
[16:15] <aznashwan> natefinch: as you guys will
[16:15] <natefinch> aznashwan: then it's up to you and cloudbase to decide where you want to release it.  I don't really care if it's under github.com/juju or github.com/cloudbase or wherever.  But I do think it deserves its own top level repo.
[16:16] <aznashwan> natefinch: thanks for the insights
[16:16] <natefinch> aznashwan: in theory it might work inside a more general windows-specific repo, but juju doesn't have one, and utils is too much of a jumble of stuff to really count.
[16:16] <aznashwan> natefinch: that it is; sadly
[16:17] <natefinch> aznashwan: ultimately, the point is to make it available and usable to the Go community in general.  If its in its own top level repo, that has the least barrier of entry.  It doesn't matter that it's small. There's plenty of packages in the standard library that are similar size.
[16:18] <aznashwan> natefinch: also; while we're on the subject of making stuff repos of their own; Gabriel and I have been meaning to push for a separate version package for a while now
[16:18] <natefinch> aznashwan: actually, small is nice, because it means you can be very precise about only importing what you really need, instead of importing the 40-million-windows-things package.
[16:18] <natefinch> aznashwan: ahh yes, definitely
[16:19] <natefinch> aznashwan: after looking at the problems with the packaging command stuff, I was convinced of the need for that as well.
[16:19] <aznashwan> natefinch: I mean utils alone has a handful of places version detection is required beyond what simple build constraints can account for...
[16:23] <aznashwan> natefinch: thanks a lot for all the support
[16:24] <natefinch> aznashwan: welcome :)
[16:41] <mup> Bug #1449123 changed: EnvJujuClient24._shell_environ does not correctly handle juju_home <juju-ci-tools:In Progress by abentley> <https://launchpad.net/bugs/1449123>
[17:21]  * wwitzel3 shakes his fists at the sky
[18:11] <mup> Bug #1449050 changed: Juju get returns a yaml file that can't be passed to set <config> <juju-core:New> <https://launchpad.net/bugs/1449050>
[18:58] <natefinch> did we change what's required in environments.yaml?  Because after updating to master, when I do juju status on my amazon environment I get "invalid EC2 provider config: control-bucket: expected string, got nothing"
[19:05] <mup> Bug #1447392 changed: ssh args list too long when bootstrapping <bootstrap> <regression> <juju-core:Fix Released by bteleaga> <https://launchpad.net/bugs/1447392>
[19:05] <mup> Bug #1447846 changed: Hooks don't fire after upgrade 1.23.0 <hooks> <regression> <upgrade-juju> <juju-core:Fix Released by menno.smits> <juju-core 1.23:Fix Committed by menno.smits> <https://launchpad.net/bugs/1447846>
[20:02] <natefinch> cmars: you around?
[20:38] <cmars> natefinch, what's up?
[20:40] <natefinch> I updated the review you looked at on friday: http://reviews.vapour.ws/r/1487/diff/#
[20:40] <natefinch> cmars: it includes a test that I noticed was skipped on windows, even though it was testing the windows-only code :/  So I fixed the test.
[20:46] <natefinch> wwitzel3: if you do juju status on an unbootstrapped amazon environment, do you get "invalid EC2 provider config: control-bucket: expected string, got nothing" ?  That seems like a new error, and incorrect, to boot.
[20:48] <wwitzel3> natefinch: you asking me to verify that locally? or is this a change I landed?
[20:48] <wwitzel3> natefinch: I agree that seems like the wrong error
[20:49] <natefinch> wwitzel3: asking you to  try it out if it's not too much trouble to make sure I'm not crazy :)
[20:49] <natefinch> wwitzel3: I don't have any reason to believe you're responsible, I Just happened to know you were at your keyboard ;)
[20:49] <wwitzel3> natefinch: yep, same error in the Error detail
[20:49] <wwitzel3> natefinch: :)
[20:50] <perrito666> wow, adding a simple hook ends up being 10 lines of code :|
[20:50] <natefinch> perrito666: I agree, that's pretty good ;)
[20:51] <perrito666> ill make sure I add that to a document so it is also easy to find which are those 10 lines
[20:52] <natefinch> perrito666: 10 lines is fine if they're all together.  If they're spread across 10 files, that's bad.
[20:52] <perrito666> 5 files plus charms package
[20:52] <natefinch> yeah that sucks
[20:53] <perrito666> the charm package part will be fun
[20:53]  * natefinch coughs  *code generation*
[20:53] <perrito666> this is only valid for a very stupid hook
[20:54] <perrito666> but the fact that we need to add that info to a separate repo raises all sort of doubts
[20:56] <natefinch> yup
[21:42] <mup> Bug #1449277 was opened: juju environment create fails on aws: invalid config <juju-core:Triaged by waigani> <https://launchpad.net/bugs/1449277>
[21:51] <katco> wallyworld: want to do our 1:1 early?
[21:51] <wallyworld> yup
[21:51] <katco> wallyworld: cool... grabbing a drink and brt
[21:59] <perrito666> anyone would be so kind? https://github.com/juju/charm/pull/122
[22:01] <ericsnow> menn0: ping
[22:13] <menn0> ericsnow: hi
[22:15] <ericsnow> menn0: I landed your hooks/upgrade fix and wanted to make sure I didn't introduce a DB-related bug :)
[22:15] <menn0> ericsnow: ok... what's happening?
[22:15] <ericsnow> menn0: see #1449054 (incl. failures that predate my patch)
[22:15] <mup> Bug #1449054: Intermittent panic: rescanned document <ci> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1449054>
[22:17] <ericsnow> menn0: I don't think I introduced anything new, but got that exact same panic the first time I tried to merge, so wanted to be sure it wasn't something I broke :)
[22:17] <ericsnow> s/anything new/any new bugs/
[22:18] <menn0> ericsnow: I remember looking at this issue when it happened way back in 1.21. that time it was caused by a bug in mgo/txn which was fixed.
[22:19] <ericsnow> menn0: yep
[22:20] <menn0> ericsnow: given what was in the fix you committed and given that it was happening also before that fix I don't think it's that
[22:20] <ericsnow> menn0: so either it regressed or we are hitting some corner case in DB-related code that was added relatively recently (1.23?)
[22:20] <menn0> ericsnow: yep
[22:20] <menn0> ericsnow: let me take a look at what was committed anyway
[22:21] <ericsnow> menn0: k, thanks
[22:21] <menn0> ericsnow: I can't see the PR on GH under my name or yours
[22:22] <ericsnow> menn0: https://github.com/juju/juju/pull/2127
[22:23] <menn0> ericsnow: thanks... I skipped straight over it when looking at your PRs :)
[22:23] <ericsnow> :)
[22:27] <menn0> ericsnow: the PR looks good and I don't think there's any way this upgrade step could cause that panic in the apiserver tests
[22:27] <menn0> ericsnow: I think the 2 aren't related
[22:27] <ericsnow> menn0: cool, that's what I figured
[22:28] <ericsnow> menn0: thanks for taking a look
[22:28] <menn0> ericsnow: now the fun part: figuring out what IS causing that panic.
[22:29] <ericsnow> menn0: "is fun for you but we have to clean up after"
[22:29] <ericsnow> (from "the man who knew too little")
[22:29] <ericsnow> :)
[22:30] <mgz> ericsnow, menn0: bug 1449054
[22:30] <mup> Bug #1449054: Intermittent panic: rescanned document <ci> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1449054>
[22:31] <menn0> mgz: yep, had seen that
[22:31] <mgz> william and I looked at the code again today, pretty clearly not from that change, and I went over the CI history and filed that bug for it
[22:31] <menn0> mgz: I had just commented on it
[22:32] <mgz> I think from the three cases where we caught it in CI, juju is failing to react well to an ill machine underneath it
[22:32] <menn0> mgz: given that it happened before the fix for bug 1447846 went in it's pretty unlikely to be that
[22:32] <mup> Bug #1447846: Hooks don't fire after upgrade 1.23.0 <hooks> <regression> <upgrade-juju> <juju-core:Fix Released by menno.smits> <juju-core 1.23:Fix Committed by menno.smits> <https://launchpad.net/bugs/1447846>
[22:33] <menn0> mgz: it still shouldn't happen obviously.
[22:33] <menn0> mgz: finding a link between a sick machine and problems with the txn layer will be fun
[22:33] <mgz> indeed, that's why I tracked down it happening in 1.23-alpha1
[22:34]  * menn0 checks something
[22:37] <menn0> mgz: what was the date we first saw this problem in recent times?
[22:41] <mgz> menn0: Feb 12th, rev 4bfd0056afb508b41927d871bde0604fdfb665b1
[22:43] <menn0> mgz: thanks. the last dep update for the mgo repo was well before that so it's unlikely to be a mgo change
[22:44] <menn0> mgz: last time we saw this issue it was a bug in mgo/txn
[22:46] <menn0> mgz: what's the best way to find the console logs of all recent fails due to this panic?
[22:46] <mgz> menn0: I linked the entire set in the bug
[22:46] <mgz> it's just the three
[22:46] <mgz> also, looking at the surrounding failures is informative
[22:46] <menn0> mgz: ok cool. I wasn't sure if that was the whole set or not.
[22:46] <mgz> the utopic machine was certainly not well on Friday
[22:48] <mgz> my current guess is it's our driver or mongo not reacting well to a disk read error or similar and dropping data
[22:48] <mgz> then being surprised when it checks later and finds its not there
[22:48] <mgz> *write
[22:48] <menn0> mgz: that sounds plausible
[22:49] <mgz> anyway, the kind of thing where the whole machine is imminently likely to die anyway
[22:49] <mgz> so failing better would be nice, but not a high priority
[22:49] <menn0> mgz: where is that machine hosted?
[22:50] <mgz> menn0: ec2, us-east-1
[22:51] <menn0> mgz: so could we just blow it away and start again?
[22:51] <wallyworld> menn0: can you remind me, that rsyslog issue we discussed last week, i forget the reason you gave for hitting that
[22:51] <mgz> menn0: yup
[22:52] <menn0> wallyworld: the rsyslog-gnutls package isn't installed if the os update/upgrade options are turned off
[22:52] <menn0> wallyworld: and without that the rsyslog config that juju wants to use can't work
[23:00] <menn0> wallyworld: it's bug 1424892
[23:00] <mup> Bug #1424892: rsyslog-gnutls is not installed when enable-os-refresh-update is false <cloud-init> <logging> <juju-core:In Progress by natefinch> <https://launchpad.net/bugs/1424892>
[23:01] <wallyworld> menn0: ah yes, thank you
[23:01] <menn0> wallyworld: were you offline just before when I wrote to you?
[23:01] <wallyworld> i remember now
[23:01] <wallyworld> menn0: my stupid irc connection keeps dropping
[23:01] <wallyworld> NFI why
[23:02] <menn0> wallyworld: looks like nate and katco have been looking at the rsyslog issue
[23:02] <wallyworld> menn0: yes, moonstone is all doing bugs atm
[23:02] <wallyworld> while waiting for feature clarification
[23:15] <axw> wallyworld: seems there's a problem with my webcam, trying to fix now
[23:16] <anastasiamac> wallyworld: standup?
[23:16] <wallyworld> axw: google hates me, be there in a bit
[23:36] <mup> Bug #1449301 was opened: storage: storage cannot be destroyed <storage> <juju-core:Triaged by axwalk> <https://launchpad.net/bugs/1449301>
[23:36] <mup> Bug #1449302 was opened: upgrades: old machines need block devices document <storage> <juju-core:Triaged> <https://launchpad.net/bugs/1449302>
[23:48] <mup> Bug #1449301 changed: storage: storage cannot be destroyed <storage> <juju-core:Triaged by axwalk> <https://launchpad.net/bugs/1449301>
[23:48] <mup> Bug #1449302 changed: upgrades: old machines need block devices document <storage> <juju-core:Triaged> <https://launchpad.net/bugs/1449302>
[23:54] <mup> Bug #1449301 was opened: storage: storage cannot be destroyed <storage> <juju-core:Triaged by axwalk> <https://launchpad.net/bugs/1449301>
[23:54] <mup> Bug #1449302 was opened: upgrades: old machines need block devices document <storage> <juju-core:Triaged> <https://launchpad.net/bugs/1449302>
[23:59] <menn0> wallyworld: if you have a moment, could you pls create a feature branch on the juju repo for me called "db-log"?
[23:59] <wallyworld> sure