[02:47] <menn0> axw: how's things?
[02:49] <axw> menn0: heya. not bad, yourself?
[02:49] <menn0> axw: yeah not too bad. super tired after 2 terrible nights in a row with the youngest.
[02:50] <menn0> axw: did you catch up with what happened while you were on leave?
[02:50] <menn0> I reverted that Safe.J change as it made ensure-availability very unhappy
[02:50] <menn0> I have no idea why but it was that change that did it
[02:50] <axw> menn0: I did see, yes, thanks for fixing
[02:51] <axw> menn0: ok. it's weird, can't see why it'd break anything
[02:51] <axw> but I think it shouldn't be too hard to work around
[02:54] <menn0> axw: ok cool
[02:56] <menn0> axw: also, regarding bug 1350983. is this something you can confirm as being fixed?
[02:56] <menn0> needs testing against azure to be sure.
[02:57] <menn0> axw: actually I just saw that Curtis says the problem is still happening even after the Safe.J change was reverted.
[02:57] <menn0> axw: so it might be something else
[02:58] <axw> yeah, I don't think that bug is related to Safe
[06:02] <axw> anyone around to do a review? https://github.com/juju/juju/pull/483
[07:15] <TheMue> morning
[07:38] <dimitern> waigani, hey, are your still around?
[07:38] <dimitern> TheMue, morning
[07:38] <waigani> dimitern: hello, yes :)
[07:39] <dimitern> waigani, i'm sending you a quick write-up how I use git rebase workflow with my scripts/aliases
[07:39] <dimitern> waigani, and I'd appreciate feedback on how it works for you :)
[07:39] <waigani> dimitern: thank you!
[07:40] <dimitern> waigani, np, i'm glad somebody else will try it out!
[07:47]  * TheMue is setting up a test environment with nested lxc containers
[08:03] <voidspac_> axw: ping
[08:03] <voidspac_> are you still awake?
[08:03] <axw> voidspac_: awake? it's 4pm :)
[08:03] <voidspac_> axw: oh!
[08:04] <voidspac_> I always assume that it's the middle of the night for you when it's day for me :-)
[08:04] <voidspac_> axw: anyway, so on my PR to fix a local provider bug you complain that it doesn't fix the manual provider too...
[08:04] <voidspac_> axw: :-p
[08:04]  * axw complains a lot
[08:05] <axw> I do indeed, because that's the way it screwed up my laptop
[08:05] <voidspac_> axw: heh, so report your own damn bugs...
[08:05] <voidspac_> axw: seriously though - do we want the same fix for the manual provider
[08:05] <axw> voidspac_: pretty sure the original one was related to manual, but yeah, I should have
[08:06] <voidspac_> do we not want jujud to change networking for manually provisioned machines
[08:06] <axw> I think so
[08:06] <axw> umm
[08:06] <voidspac_> https://bugs.launchpad.net/juju-core/+bug/1349635
[08:06] <axw> well, what are the networks for a manual environment?
[08:06] <voidspac_> "local environment"
[08:06] <voidspac_> although, hmmm
[08:06] <axw> the summary was changed
[08:07] <axw> the title I mean
[08:07] <axw> anyway
[08:07] <voidspac_> sure
[08:07] <voidspac_> so in the fix I have we don't touch networks at all for local provider
[08:07] <voidspac_> "apply" just short-circuits and bails
[08:08] <voidspac_> you suggest having the networker "know" if the provider supports networking or not
[08:08] <voidspac_> axw: what specifically are you suggesting?
[08:08] <axw> voidspac_: right. and this is the sort of thing fwereade prefers; that we operate in terms of environment/provider capabilities, rather than coding to specific providers
[08:08]  * axw thinks
[08:09] <voidspac_> axw: ok, but local provider machine 0 *is* a special snowflake
[08:09] <voidspac_> unfortunately
[08:09] <voidspac_> so it's not *just* provider, it's the bootstrap machine on the local provider
[08:10] <voidspac_> not touching networking config at all for manual provider also makes sense - so long as that's what we want
[08:10] <voidspac_> (and is easy enough to add a check for)
[08:12] <axw> voidspac_: so what *I* would prefer is that the networker do nothing for any envrionment whose SupportsNetworks method returns false
[08:12] <voidspac_> axw: when I grep for SupportsNetworks I get no results
[08:12] <axw> lemme check spelling
[08:13] <axw> voidspac_: sorry, no first "s": SupportNetworks
[08:13] <voidspac_> axw: that would be easy enough if it fixes the issue
[08:14] <voidspac_> axw: I'll look into that (I notice you did mention it in your comment - sorry)
[08:14] <voidspac_> axw: thanks
[08:14] <axw> voidspac_: atm, only the MAAS provider returns true (depending on MAAS version)
[08:14] <voidspac_> axw: we *need* the networking stuff for local provider machine != 0
[08:14] <voidspac_> axw: it sets up something we need
[08:14] <axw> oh, we do? what's that?
[08:15] <voidspac_> not sure, but local provider breaks without it
[08:15] <axw> it's relatively new, and used just for MAAS I thought
[08:15] <voidspac_> that was my first approach
[08:15] <voidspac_> dimitern: ^^^^
[08:15] <axw> buh
[08:15] <voidspac_> axw: let me try again
[08:15] <voidspac_> (once a test run finishes)
[08:15] <axw> okey dokey
[08:16] <axw> dimitern: is it safe/sensible to disable the networker worker if the environment's SupportNetworks method returns false at machine agent startup?
[08:16] <voidspac_> if you have openstack environment variables set there's a juju-metadata test that always fails
[08:16] <voidspac_> it doesn't get the error it expects
[08:16] <axw> wonderful
[08:16] <axw> that means it's not embedding the base suite I guess
[08:18] <TheMue> $ remove => not found / $ delete => not found / $ destroy => destroyed *aaaaaaah*
[08:20] <dimitern> axw, unfortunately, no
[08:20] <dimitern> voidspac_, axw, supportnetworks capability is supposed to be used by providers to tell juju if *any* networking is supported, but for the local/manual case this is not enough
[08:22] <dimitern> voidspac_, axw, it seems to me we need another capability or something for this
[08:22] <axw> dimitern: I don't quite understand. local and manual both currently return false for SupportNetworks, so why isn't that enough to go by?
[08:22] <axw> if that means "no networking is supported"
[08:23] <dimitern> it's not necessarily an environment-wide setting (for local it only needs to apply to the bootstrap node; for manual too I guess, but how about manually provisioned machines?)
[08:23] <dimitern> axw, because I haven't got that far yet - they will support *some* sort of networking
[08:24] <axw> ok, I see
[08:24] <axw> so it'd work today, but not for long
[08:24] <dimitern> yeah
[08:24] <axw> well then, whatever we do is going to be a stop-gap
[08:25] <axw> voidspac_: ^^ maybe don't invest too much time if it's all going to change soon
[08:25] <dimitern> voidspac_, axw, how about having a capability called ManageBootstrapNetworking bool ?
[08:26] <dimitern> it can be true for all but the local and manual provider
[08:26] <voidspac_> dimitern: that is then only used by local provider?
[08:26] <axw> dimitern: I don't know that that's right either. Do we really want to be managing networking on manually provisioned machines? They won't ever know about networks, right?
[08:27] <voidspac_> ManageNetworking that is always false on manual provider and false for local bootstrap machine
[08:27] <dimitern> axw, they will at some point, at least units deployed on such machines
[08:27] <dimitern> voidspac_, alas it can't be per-machine, it's an environ capability.. although I guess we can make it take a machine id and check..
[08:29] <voidspac_> dimitern: well, the environ is running on a specific machine
[08:30] <voidspac_> but if that's innappropriate for an environ capabilitiy then maybe that's not where this belongs
[08:32] <dimitern> voidspac_, you know, I'd rather have this critical bug fixed now, and have a TODO + tech dept bug about improving the way we decide the value of the writeNetworkConfig flag for the networker
[08:32] <voidspac_> dimitern: well, the PR is ready to merge
[08:32] <voidspac_> dimitern: *but*
[08:32] <voidspac_> dimitern: axw says the original report was about manually provisioned machines, not just the local provider
[08:32] <voidspac_> and my PR does not address the manual provider at all
[08:33] <voidspac_> I can add a TODO for sure
[08:33] <axw> can we just make it != "maas" for now, and TODO FIXME
[08:33] <voidspac_> axw: I will check if that works
[08:33] <voidspac_> axw: I have test failures though :-/
[08:33] <voidspac_> which I suspect are sporadic
[08:33] <voidspac_> but checking
[08:34] <axw> :(
[08:34] <voidspac_> I need more coffee for this
[08:36] <dimitern> axw, type != "maas" is a really dirty hack :) but I can live with it for a short while, until we find a better way
[08:37] <axw> I agree, I didn't really want to go there
[08:37] <axw> seems better to fix now and do properly when networking evolves tho
[08:42] <axw> TheMue: it would appear you're the lucky OCR. can you please take a look at https://github.com/juju/juju/pull/483 and https://github.com/juju/juju/pull/484? these fix 2/3 critical bugs on 1.21
[08:42] <TheMue> Btw, reading the notes of Stéphane and Serge *before* installing nested containers helps .. *blush*
[08:42] <voidspac_> if you do a bootstrap (local provider), immediately followed by a deploy
[08:42] <voidspac_> you get:
[08:42] <voidspac_> ERROR upgrade in progress - Juju functionality is limited
[08:42] <TheMue> axw: I’m OCR, oh!
[08:43] <axw> oops
[08:43] <axw> TheMue: no you're not
[08:43] <axw> sorry
[08:43] <axw> spreadsheet hadn't updated yet
[08:43] <TheMue> No, Nate and Wayne. *phew*
[08:43] <TheMue> But still can take a look.
[08:43] <axw> thanks :)
[08:46] <TheMue> axw: 483 LGTM
[08:46] <axw> TheMue: thanks
[08:54] <voidspac_> axw: dimitern: latest revision uses the dirty hack
[08:54] <voidspac_> https://github.com/juju/juju/pull/477
[08:54] <voidspac_> axw: dimitern: only writes network config changes for maas
[08:55] <TheMue> axw: 484 LGTM too with one note
[08:55] <voidspac_> works for local provider (tested "manually")
[08:56] <axw> TheMue: thanks
[08:57] <TheMue> axw: yw
[09:00] <axw> voidspac_: I'd prefer if we just disabled it altogether if != maas, but I'll defer to dimitern since I don't know enough about the networker
[09:02] <voidspac_> it would simplify this code a great deal to just only start the networker for maas
[09:03] <dimitern> axw, voidspac_, please, let's not disable the networker just for that
[09:04] <dimitern> voidspac_, LGTM
[09:48] <voidspac_> dimitern: might be late to standup by the way
[09:48] <voidspac_> only by ten minutes or so
[09:48] <voidspac_> dimitern: lp bug created
[09:48] <voidspac_> by the way
[09:48] <voidspac_> and linked in comment
[09:48] <dimitern> voidspac_, no worries, cheers
[09:49] <voidspac_> even though this is a critical bug it doesn't seem to be one of the blocking bugs, so I'm not sure if this can be merged yet
[09:49] <voidspac_> we'll see
[09:54] <rogpeppe1> dimitern: are merges to juju-core still blocked on critical bugs?
[09:55] <dimitern> rogpeppe1, I'm not aware of that, if it's the case
[09:56] <rogpeppe1> dimitern: ah, ok. i still see two critical bugs open, and i thought that all merges into juju-core were blocked if that was the case, unless they directly addressed those issues
[09:56] <rogpeppe1> dimitern: is that not the case?
[09:56] <rogpeppe1> dimitern: (i've had a branch sitting for weeks waiting for things to be unblocked)
[09:56] <dimitern> rogpeppe1, one of the critical bugs has an approved fix pending merging, i need to see about the other one
[09:57] <rogpeppe1> dimitern: i guess i'll try to merge again then
[09:59] <dimitern> rogpeppe1, sgtm
[10:20] <TheMue> aaaaah, after two failed approaches my nested containers are now running (ok, currently only, but the way how to do it has been important)
[10:26] <TheMue> hehe, running the mac with a test trusty in a vm running a trusty lxc container running two trusty containers
[10:26] <rogpeppe1> dimitern: hmm, looks like merges *are* still blocked: http://juju-ci.vapour.ws:8080/job/github-merge-juju/236/console
[10:28] <dimitern> rogpeppe1, that's michael's fix - we'll sort it out on the standup shortly
[10:28] <rogpeppe1> dimitern: cool. i'll keep my fingers crossed. could you ping me when merges are finally unblocked, please?
[10:29] <dimitern> rogpeppe1, sure thing
[10:29] <rogpeppe1> dimitern: ta!
[10:52] <perrito666> axw: hey thanks for the patch
[10:53] <perrito666> axw: the other bug seems to have been fixed after your other patch was reverted
[10:53] <perrito666> but I was awaiting confirmation from sin
[10:53] <perrito666> zui
[11:03] <dimitern> voidspace, are you back?
[11:06] <dimitern> mgz, ping
[11:11] <voidspace> dimitern: yes, gimme a second
[11:15] <voidspace> dimitern: TheMue: I guess I missed standup, sorry
[11:15] <natefinch> people keep inviting me to meetings in Nuremberf and not actually giving me a way to participate :/
[11:16] <natefinch> s/Nuremberf/Nuremberg/
[11:16] <dimitern> voidspace, np, we can have a call or chat here if you like
[11:16] <voidspace> dimitern: chat here is fine
[11:17] <voidspace> dimitern: I guess I need something new, other than "generally explore ipv6"
[11:17] <voidspace> dimitern: my PR is blocked, even though it fixes a critical bug
[11:17] <voidspace> I guess it doesn't fix a critical bug on the right branch :-)
[11:18] <voidspace> dimitern: I still have an outstanding task to close the database port as an upgrade step, so I can revert to that
[11:21] <dimitern> voidspace, btw I had a couple of suggestions to your PR - did you see them?
[11:22] <voidspace> dimitern: yes, I addressed them
[11:22] <voidspace> dimitern: create LP bug and link in TODO, get rid of loop
[11:22] <dimitern> voidspace, right, cheers
[11:22] <voidspace> dimitern: and fix a comment
[11:23] <dimitern> voidspace, it seems the bot blocked it because it has conflicting "fixes-###" comments
[11:23] <dimitern> voidspace, I've never seen this before, might be a new thing
[11:23] <voidspace> dimitern: that's the standard blocked message isn't it?
[11:23] <voidspace>  Does not match ['fixes-1350983', 'fixes-1353443']
[11:24] <voidspace> so, only a fix for issue 1350983 or 1353443 will be allowed
[11:24] <voidspace> and mine fixes 1349635
[11:24] <voidspace> dimitern: see the same message here https://github.com/juju/juju/pull/449
[11:25] <voidspace> well, not the same message precisely but the same format
[11:25] <dimitern> right
[11:25] <dimitern> voidspace, we need some guys with strong bot-fu :) mgz perhaps, others?
[11:27] <dimitern> voidspace, we need this fix to land, as rogpeppe1 is blocked because of that
[11:27] <voidspace> dimitern: but trunk is blocked on those other issues
[11:27] <dimitern> voidspace, you mean bug 1350983 ?
[11:28] <dimitern> mup, bug 1350983
[11:28] <voidspace> dimitern: yep and 1353443
[11:28] <mup> dimitern: Bug #1350983: Cannot bootstrap on azure: cannot create log collection <azure-provider> <bootstrap> <ci> <regression> <juju-core:In Progress by hduran-8> <https://launchpad.net/bugs/1350983>
[11:28] <voidspace> according to the message
[11:28] <voidspace> mup bug 1353443
[11:28] <dimitern> perrito666, hey, any update on ^^ ?
[11:28] <voidspace> mup, bug 1353443
[11:28] <mup> voidspace: Bug #1353443: Install hook error due to modprobe 8021q failure <ci> <local-provider> <networker> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1353443>
[11:29] <perrito666> dimitern: sinzui was going to make a few runs on CI but as of yesterday It seemd to be fixed, let me mark fix committed
[11:29] <perrito666> I tested it a LOT of times without a failure
[11:29] <dimitern> voidspace, I commented on that ^ bug earlier; the fix is not the same as for the networker bug screwing the local provider networking
[11:29] <dimitern> perrito666, but has it landed on trunk?
[11:30] <perrito666> dimitern: yes
[11:31] <voidspace> dimitern: disabling the networker except for maas would fix it though....
[11:31] <perrito666> ow fff is failing again
[11:32] <voidspace> dimitern: does the networker do anything useful right now if you're not using maas?
[11:32] <voidspace> perrito666: fff?
[11:32] <perrito666> voidspace: I try not to say profanities on work channels
[11:32] <dimitern> voidspace, :) well, not really
[11:32] <dimitern> voidspace, we'll still have issues with lxc containers in maas
[11:33] <perrito666> voidspace: if you where here you could have heard that pronounced out loud
[11:33] <voidspace> dimitern: right
[11:33] <voidspace> perrito666: oh :-D
[11:33] <voidspace> perrito666: I thought fff was the thing that was failing and wondered what it was
[11:33] <dimitern> voidspace, that's why I think the fix should involve making the networker aware whether it's running in an LXC container or not, and not try to modprobe anything
[11:34] <dimitern> voidspace, it does enable/disable NICs (or it should at least)
[11:36] <voidspace> dimitern: is that done in Execute ?
[11:36] <voidspace> dimitern: because I disable that too
[11:37] <perrito666> dimitern: oh it is failing with timeout io which is fixed by the next merge to land in CI
[11:38] <dimitern> voidspace, ah
[11:38] <dimitern> voidspace, i'll have to pull your branch again and do some more testing on my maas
[11:39] <voidspace> dimitern: it's not disabled for maas
[11:39] <voidspace> dimitern: it is for the other providers
[11:40] <dimitern> voidspace, yes, but frankly I haven't tested the networker on maas myself so far :/
[11:40] <voidspace> right
[11:40] <dimitern> (i.e. it's main job about managing interfaces)
[11:40] <dimitern> its
[11:41] <dimitern> ok, brb for a while
[11:41]  * dimitern lunches
[11:52] <menn0> voidspace: you know how to make your PR merge right?
[11:54] <menn0> voidspace: you need a comment of "__fixes-1349635__" and then also "$$whatever$$" in the same or another comment
[11:54] <menn0> $$fixes-1349635$$ won't cut it
[11:54] <menn0> I suspect "$$__fixes-1349635__$$" might work as a 2-in-1 though :)
[11:56] <menn0> #1349635
[11:56] <menn0> (that was me testing mup)
[11:56] <menn0> ANYWAY... bed time!
[12:28] <voidspace> mgz: are you tinkering?
[12:34] <mgz> voidspace: not tinkering, just needed to apply a hammer to get your change landing
[12:34] <wwitzel3> natefinch: so if we rotate purely on size, rsyslog can do it. But it has no concept of a daily. Do you think that is acceptable? Or does it need to be a daily rotation?
[12:34] <voidspace> mgz: thanks
[12:45] <katco> wwitzel3: dropping in on this convo: size of log = what, daily = how. i always prefer the what if possible.
[12:45] <katco> e.g.: a log could grow too large w/in a day, or not grow enough in a month
[12:45] <mgz> voidspace: do you know if there's any hope of bug 1353443 being got to today?
[12:46] <mgz> it's currently unassigned, but blocks master and you and dimitern seem to know what's up with it
[12:47] <wwitzel3> katco: I agree, the open bug is about the size, but I think we did log ration for units as a daily rotation, so having a different rotation scheme for the all-machines might be weird
[12:48] <katco> wwitzel3: yeah that would be confusing as a user
[12:52] <wwitzel3> katco: though if it was clearly documented, that might be ok
[12:54] <katco> wwitzel3: docs? who reads the friendly docs? :)
[12:59] <natefinch> wwitzel3, katco: size.   Daily rotation is useless.   You run the risk of rotating an empty log, or filling up the disk when you get 10 gigs of logs in a really busy couple hours.
[13:00] <natefinch> wwitzel3, katco: I went through that with lumberjack, at first I supported rotation by time, but then I realized that was just a bad idea
[13:00] <natefinch> wwitzel3, katco: the code I wrote for rotating the unit and machine logs just rotates based on size
[13:00] <katco> natefinch: hey i saw a js logging package earlier this morning: sherlog. :p
[13:00] <natefinch> katco: nice
[13:15] <voidspace> mgz: I don't know
[13:15] <voidspace> mgz: not by me I'm afraid as I have to finish a bit early, off up a mountain later
[13:16] <voidspace> mgz: and I'm not sure what the best fix is - "not modprobe inside an lxc container" is most of the answer, but the right way for the networker to know whether or not it is contained is trickier
[13:16] <voidspace> and dimitern should probably answer that
[13:18] <dimitern> voidspace, we can tell whether we're in a lxc container or not easily from the agent config - machine tag will be something like machine-0-lxc-0 instead of machine-0
[13:18] <dimitern> voidspace, we shouldn't disable modprobe for kvm containers though, only for lxc
[13:18] <mgz> dimitern: I think this is the only remaining issue blocking landing
[13:18] <voidspace> dimitern: so, "if strings.Contains(agent.Tag(), "lxc")...
[13:19] <voidspace> it's pretty horrible
[13:20] <dimitern> voidspace, no, there's a names.ParseTag
[13:23] <voidspace> dimitern: do you think that is the right fix, protect modprobe with a check for lxc by parsing the tag?
[13:24] <dimitern> voidspace, I think it's the right fix to disable modprobe altogether for machine agents running in lxc containers
[13:25] <dimitern> voidspace, whether I think parsing the tag to get the container type is good enough - not really :)
[13:25] <voidspace> right
[13:25] <voidspace> that seems yucky
[13:25] <dimitern> voidspace, we should have a way to tell what's the machine
[13:28] <dimitern> voidspace, but nevertheless having a common way to tell if a machine is a container and what type is useful, and should be in state and the api
[13:29] <voidspace> in the api?
[13:29] <voidspace> a machine agent shouldn't have to make an external call to find out whether it's running in a container or not
[13:29] <voidspace> or maybe it should, if that lives in the state
[13:29] <voidspace> seems like the agent should know though
[13:30] <perrito666> natefinch: lemme know if we do the 1:1 so I actually get onto the call
[13:32] <perrito666> aghhh so close
[13:32] <natefinch> prein a minute
[13:32] <natefinch> perrito666:  in a minute
[13:38] <dimitern> voidspace, by the api i meant "extend the agent api as needed so that info is available", no need to make a separate call (like with Life())
[13:46] <natefinch> dimitern, voidspace:  do you guys think you can get #1353443 fixed today?  I'm anxious to get master open to commits again
[13:47] <natefinch> mup is failing me  - https://bugs.launchpad.net/juju-core/+bug/1353443
[13:47] <voidspace> natefinch: dimitern: I'm leaving in 15minutes I'm afraid...
[13:48] <dimitern> natefinch, voidspace, I think it just got merged, no?
[13:49] <voidspace> dimitern: no, the other one
[13:49] <dimitern> oh, bugger
[13:50] <voidspace> This one just got merged
[13:50] <voidspace> https://bugs.launchpad.net/juju-core/+bug/1349635
[13:50] <dimitern> natefinch, voidspace, I'll take it and try to do a quick fix
[13:50] <dimitern>  #1353443 I mean
[13:50] <voidspace> dimitern: thanks
[17:23] <natefinch> be back in a couple hours.... taking my wife to her midwife appointment.
[23:30] <perrito666> anyone alive here/
[23:30] <perrito666> ?