[02:47] axw: how's things? [02:49] menn0: heya. not bad, yourself? [02:49] axw: yeah not too bad. super tired after 2 terrible nights in a row with the youngest. [02:50] axw: did you catch up with what happened while you were on leave? [02:50] I reverted that Safe.J change as it made ensure-availability very unhappy [02:50] I have no idea why but it was that change that did it [02:50] menn0: I did see, yes, thanks for fixing [02:51] menn0: ok. it's weird, can't see why it'd break anything [02:51] but I think it shouldn't be too hard to work around [02:54] axw: ok cool [02:56] axw: also, regarding bug 1350983. is this something you can confirm as being fixed? [02:56] needs testing against azure to be sure. [02:57] axw: actually I just saw that Curtis says the problem is still happening even after the Safe.J change was reverted. [02:57] axw: so it might be something else [02:58] yeah, I don't think that bug is related to Safe [06:02] anyone around to do a review? https://github.com/juju/juju/pull/483 [07:15] morning [07:38] waigani, hey, are your still around? [07:38] TheMue, morning [07:38] dimitern: hello, yes :) [07:39] waigani, i'm sending you a quick write-up how I use git rebase workflow with my scripts/aliases [07:39] waigani, and I'd appreciate feedback on how it works for you :) [07:39] dimitern: thank you! [07:40] waigani, np, i'm glad somebody else will try it out! [07:47] * TheMue is setting up a test environment with nested lxc containers [08:03] axw: ping [08:03] are you still awake? [08:03] voidspac_: awake? it's 4pm :) [08:03] axw: oh! [08:04] I always assume that it's the middle of the night for you when it's day for me :-) [08:04] axw: anyway, so on my PR to fix a local provider bug you complain that it doesn't fix the manual provider too... [08:04] axw: :-p [08:04] * axw complains a lot [08:05] I do indeed, because that's the way it screwed up my laptop [08:05] axw: heh, so report your own damn bugs... [08:05] axw: seriously though - do we want the same fix for the manual provider [08:05] voidspac_: pretty sure the original one was related to manual, but yeah, I should have [08:06] do we not want jujud to change networking for manually provisioned machines [08:06] I think so [08:06] umm [08:06] https://bugs.launchpad.net/juju-core/+bug/1349635 [08:06] well, what are the networks for a manual environment? [08:06] "local environment" [08:06] although, hmmm [08:06] the summary was changed [08:07] the title I mean [08:07] anyway [08:07] sure [08:07] so in the fix I have we don't touch networks at all for local provider [08:07] "apply" just short-circuits and bails [08:08] you suggest having the networker "know" if the provider supports networking or not [08:08] axw: what specifically are you suggesting? [08:08] voidspac_: right. and this is the sort of thing fwereade prefers; that we operate in terms of environment/provider capabilities, rather than coding to specific providers [08:08] * axw thinks [08:09] axw: ok, but local provider machine 0 *is* a special snowflake [08:09] unfortunately [08:09] so it's not *just* provider, it's the bootstrap machine on the local provider [08:10] not touching networking config at all for manual provider also makes sense - so long as that's what we want [08:10] (and is easy enough to add a check for) [08:12] voidspac_: so what *I* would prefer is that the networker do nothing for any envrionment whose SupportsNetworks method returns false [08:12] axw: when I grep for SupportsNetworks I get no results [08:12] lemme check spelling [08:13] voidspac_: sorry, no first "s": SupportNetworks [08:13] axw: that would be easy enough if it fixes the issue [08:14] axw: I'll look into that (I notice you did mention it in your comment - sorry) [08:14] axw: thanks [08:14] voidspac_: atm, only the MAAS provider returns true (depending on MAAS version) [08:14] axw: we *need* the networking stuff for local provider machine != 0 [08:14] axw: it sets up something we need [08:14] oh, we do? what's that? [08:15] not sure, but local provider breaks without it [08:15] it's relatively new, and used just for MAAS I thought [08:15] that was my first approach [08:15] dimitern: ^^^^ [08:15] buh [08:15] axw: let me try again [08:15] (once a test run finishes) [08:15] okey dokey [08:16] dimitern: is it safe/sensible to disable the networker worker if the environment's SupportNetworks method returns false at machine agent startup? [08:16] if you have openstack environment variables set there's a juju-metadata test that always fails [08:16] it doesn't get the error it expects [08:16] wonderful [08:16] that means it's not embedding the base suite I guess [08:18] $ remove => not found / $ delete => not found / $ destroy => destroyed *aaaaaaah* === psivaa-afk-bbl is now known as psivaa [08:20] axw, unfortunately, no [08:20] voidspac_, axw, supportnetworks capability is supposed to be used by providers to tell juju if *any* networking is supported, but for the local/manual case this is not enough [08:22] voidspac_, axw, it seems to me we need another capability or something for this [08:22] dimitern: I don't quite understand. local and manual both currently return false for SupportNetworks, so why isn't that enough to go by? [08:22] if that means "no networking is supported" [08:23] it's not necessarily an environment-wide setting (for local it only needs to apply to the bootstrap node; for manual too I guess, but how about manually provisioned machines?) [08:23] axw, because I haven't got that far yet - they will support *some* sort of networking [08:24] ok, I see [08:24] so it'd work today, but not for long [08:24] yeah [08:24] well then, whatever we do is going to be a stop-gap [08:25] voidspac_: ^^ maybe don't invest too much time if it's all going to change soon [08:25] voidspac_, axw, how about having a capability called ManageBootstrapNetworking bool ? [08:26] it can be true for all but the local and manual provider [08:26] dimitern: that is then only used by local provider? [08:26] dimitern: I don't know that that's right either. Do we really want to be managing networking on manually provisioned machines? They won't ever know about networks, right? [08:27] ManageNetworking that is always false on manual provider and false for local bootstrap machine [08:27] axw, they will at some point, at least units deployed on such machines [08:27] voidspac_, alas it can't be per-machine, it's an environ capability.. although I guess we can make it take a machine id and check.. [08:29] dimitern: well, the environ is running on a specific machine [08:30] but if that's innappropriate for an environ capabilitiy then maybe that's not where this belongs [08:32] voidspac_, you know, I'd rather have this critical bug fixed now, and have a TODO + tech dept bug about improving the way we decide the value of the writeNetworkConfig flag for the networker [08:32] dimitern: well, the PR is ready to merge [08:32] dimitern: *but* [08:32] dimitern: axw says the original report was about manually provisioned machines, not just the local provider [08:32] and my PR does not address the manual provider at all [08:33] I can add a TODO for sure [08:33] can we just make it != "maas" for now, and TODO FIXME [08:33] axw: I will check if that works [08:33] axw: I have test failures though :-/ [08:33] which I suspect are sporadic [08:33] but checking [08:34] :( [08:34] I need more coffee for this [08:36] axw, type != "maas" is a really dirty hack :) but I can live with it for a short while, until we find a better way [08:37] I agree, I didn't really want to go there [08:37] seems better to fix now and do properly when networking evolves tho [08:42] TheMue: it would appear you're the lucky OCR. can you please take a look at https://github.com/juju/juju/pull/483 and https://github.com/juju/juju/pull/484? these fix 2/3 critical bugs on 1.21 [08:42] Btw, reading the notes of Stéphane and Serge *before* installing nested containers helps .. *blush* [08:42] if you do a bootstrap (local provider), immediately followed by a deploy [08:42] you get: [08:42] ERROR upgrade in progress - Juju functionality is limited [08:42] axw: I’m OCR, oh! [08:43] oops [08:43] TheMue: no you're not [08:43] sorry [08:43] spreadsheet hadn't updated yet [08:43] No, Nate and Wayne. *phew* [08:43] But still can take a look. [08:43] thanks :) [08:46] axw: 483 LGTM [08:46] TheMue: thanks [08:54] axw: dimitern: latest revision uses the dirty hack [08:54] https://github.com/juju/juju/pull/477 [08:54] axw: dimitern: only writes network config changes for maas [08:55] axw: 484 LGTM too with one note [08:55] works for local provider (tested "manually") [08:56] TheMue: thanks [08:57] axw: yw [09:00] voidspac_: I'd prefer if we just disabled it altogether if != maas, but I'll defer to dimitern since I don't know enough about the networker [09:02] it would simplify this code a great deal to just only start the networker for maas [09:03] axw, voidspac_, please, let's not disable the networker just for that [09:04] voidspac_, LGTM [09:48] dimitern: might be late to standup by the way [09:48] only by ten minutes or so [09:48] dimitern: lp bug created [09:48] by the way [09:48] and linked in comment [09:48] voidspac_, no worries, cheers [09:49] even though this is a critical bug it doesn't seem to be one of the blocking bugs, so I'm not sure if this can be merged yet [09:49] we'll see [09:54] dimitern: are merges to juju-core still blocked on critical bugs? [09:55] rogpeppe1, I'm not aware of that, if it's the case [09:56] dimitern: ah, ok. i still see two critical bugs open, and i thought that all merges into juju-core were blocked if that was the case, unless they directly addressed those issues [09:56] dimitern: is that not the case? [09:56] dimitern: (i've had a branch sitting for weeks waiting for things to be unblocked) [09:56] rogpeppe1, one of the critical bugs has an approved fix pending merging, i need to see about the other one [09:57] dimitern: i guess i'll try to merge again then [09:59] rogpeppe1, sgtm [10:20] aaaaah, after two failed approaches my nested containers are now running (ok, currently only, but the way how to do it has been important) [10:26] hehe, running the mac with a test trusty in a vm running a trusty lxc container running two trusty containers [10:26] dimitern: hmm, looks like merges *are* still blocked: http://juju-ci.vapour.ws:8080/job/github-merge-juju/236/console [10:28] rogpeppe1, that's michael's fix - we'll sort it out on the standup shortly [10:28] dimitern: cool. i'll keep my fingers crossed. could you ping me when merges are finally unblocked, please? [10:29] rogpeppe1, sure thing [10:29] dimitern: ta! === jamespag` is now known as jamespage [10:52] axw: hey thanks for the patch [10:53] axw: the other bug seems to have been fixed after your other patch was reverted [10:53] but I was awaiting confirmation from sin [10:53] zui [11:03] voidspace, are you back? [11:06] mgz, ping [11:11] dimitern: yes, gimme a second [11:15] dimitern: TheMue: I guess I missed standup, sorry [11:15] people keep inviting me to meetings in Nuremberf and not actually giving me a way to participate :/ [11:16] s/Nuremberf/Nuremberg/ [11:16] voidspace, np, we can have a call or chat here if you like [11:16] dimitern: chat here is fine [11:17] dimitern: I guess I need something new, other than "generally explore ipv6" [11:17] dimitern: my PR is blocked, even though it fixes a critical bug [11:17] I guess it doesn't fix a critical bug on the right branch :-) [11:18] dimitern: I still have an outstanding task to close the database port as an upgrade step, so I can revert to that [11:21] voidspace, btw I had a couple of suggestions to your PR - did you see them? [11:22] dimitern: yes, I addressed them [11:22] dimitern: create LP bug and link in TODO, get rid of loop [11:22] voidspace, right, cheers [11:22] dimitern: and fix a comment [11:23] voidspace, it seems the bot blocked it because it has conflicting "fixes-###" comments [11:23] voidspace, I've never seen this before, might be a new thing [11:23] dimitern: that's the standard blocked message isn't it? [11:23] Does not match ['fixes-1350983', 'fixes-1353443'] [11:24] so, only a fix for issue 1350983 or 1353443 will be allowed [11:24] and mine fixes 1349635 [11:24] dimitern: see the same message here https://github.com/juju/juju/pull/449 [11:25] well, not the same message precisely but the same format [11:25] right [11:25] voidspace, we need some guys with strong bot-fu :) mgz perhaps, others? [11:27] voidspace, we need this fix to land, as rogpeppe1 is blocked because of that [11:27] dimitern: but trunk is blocked on those other issues [11:27] voidspace, you mean bug 1350983 ? [11:28] mup, bug 1350983 [11:28] dimitern: yep and 1353443 [11:28] dimitern: Bug #1350983: Cannot bootstrap on azure: cannot create log collection

[11:28] according to the message [11:28] mup bug 1353443 [11:28] perrito666, hey, any update on ^^ ? [11:28] mup, bug 1353443 [11:28] voidspace: Bug #1353443: Install hook error due to modprobe 8021q failure

[11:29] dimitern: sinzui was going to make a few runs on CI but as of yesterday It seemd to be fixed, let me mark fix committed [11:29] I tested it a LOT of times without a failure [11:29] voidspace, I commented on that ^ bug earlier; the fix is not the same as for the networker bug screwing the local provider networking [11:29] perrito666, but has it landed on trunk? [11:30] dimitern: yes [11:31] dimitern: disabling the networker except for maas would fix it though.... [11:31] ow fff is failing again [11:32] dimitern: does the networker do anything useful right now if you're not using maas? [11:32] perrito666: fff? [11:32] voidspace: I try not to say profanities on work channels [11:32] voidspace, :) well, not really [11:32] voidspace, we'll still have issues with lxc containers in maas [11:33] voidspace: if you where here you could have heard that pronounced out loud [11:33] dimitern: right [11:33] perrito666: oh :-D [11:33] perrito666: I thought fff was the thing that was failing and wondered what it was [11:33] voidspace, that's why I think the fix should involve making the networker aware whether it's running in an LXC container or not, and not try to modprobe anything [11:34] voidspace, it does enable/disable NICs (or it should at least) [11:36] dimitern: is that done in Execute ? [11:36] dimitern: because I disable that too [11:37] dimitern: oh it is failing with timeout io which is fixed by the next merge to land in CI [11:38] voidspace, ah [11:38] voidspace, i'll have to pull your branch again and do some more testing on my maas [11:39] dimitern: it's not disabled for maas [11:39] dimitern: it is for the other providers [11:40] voidspace, yes, but frankly I haven't tested the networker on maas myself so far :/ [11:40] right [11:40] (i.e. it's main job about managing interfaces) [11:40] its [11:41] ok, brb for a while [11:41] * dimitern lunches [11:52] voidspace: you know how to make your PR merge right? [11:54] voidspace: you need a comment of "__fixes-1349635__" and then also "$$whatever$$" in the same or another comment [11:54] $$fixes-1349635$$ won't cut it [11:54] I suspect "$$__fixes-1349635__$$" might work as a 2-in-1 though :) [11:56] #1349635 [11:56] (that was me testing mup) [11:56] ANYWAY... bed time! [12:28] mgz: are you tinkering? [12:34] voidspace: not tinkering, just needed to apply a hammer to get your change landing [12:34] natefinch: so if we rotate purely on size, rsyslog can do it. But it has no concept of a daily. Do you think that is acceptable? Or does it need to be a daily rotation? [12:34] mgz: thanks [12:45] wwitzel3: dropping in on this convo: size of log = what, daily = how. i always prefer the what if possible. [12:45] e.g.: a log could grow too large w/in a day, or not grow enough in a month [12:45] voidspace: do you know if there's any hope of bug 1353443 being got to today? [12:46] it's currently unassigned, but blocks master and you and dimitern seem to know what's up with it [12:47] katco: I agree, the open bug is about the size, but I think we did log ration for units as a daily rotation, so having a different rotation scheme for the all-machines might be weird [12:48] wwitzel3: yeah that would be confusing as a user [12:52] katco: though if it was clearly documented, that might be ok [12:54] wwitzel3: docs? who reads the friendly docs? :) [12:59] wwitzel3, katco: size. Daily rotation is useless. You run the risk of rotating an empty log, or filling up the disk when you get 10 gigs of logs in a really busy couple hours. [13:00] wwitzel3, katco: I went through that with lumberjack, at first I supported rotation by time, but then I realized that was just a bad idea [13:00] wwitzel3, katco: the code I wrote for rotating the unit and machine logs just rotates based on size [13:00] natefinch: hey i saw a js logging package earlier this morning: sherlog. :p [13:00] katco: nice [13:15] mgz: I don't know [13:15] mgz: not by me I'm afraid as I have to finish a bit early, off up a mountain later [13:16] mgz: and I'm not sure what the best fix is - "not modprobe inside an lxc container" is most of the answer, but the right way for the networker to know whether or not it is contained is trickier [13:16] and dimitern should probably answer that [13:18] voidspace, we can tell whether we're in a lxc container or not easily from the agent config - machine tag will be something like machine-0-lxc-0 instead of machine-0 [13:18] voidspace, we shouldn't disable modprobe for kvm containers though, only for lxc [13:18] dimitern: I think this is the only remaining issue blocking landing [13:18] dimitern: so, "if strings.Contains(agent.Tag(), "lxc")... [13:19] it's pretty horrible [13:20] voidspace, no, there's a names.ParseTag [13:23] dimitern: do you think that is the right fix, protect modprobe with a check for lxc by parsing the tag? [13:24] voidspace, I think it's the right fix to disable modprobe altogether for machine agents running in lxc containers [13:25] voidspace, whether I think parsing the tag to get the container type is good enough - not really :) [13:25] right [13:25] that seems yucky [13:25] voidspace, we should have a way to tell what's the machine [13:28] voidspace, but nevertheless having a common way to tell if a machine is a container and what type is useful, and should be in state and the api [13:29] in the api? [13:29] a machine agent shouldn't have to make an external call to find out whether it's running in a container or not [13:29] or maybe it should, if that lives in the state [13:29] seems like the agent should know though [13:30] natefinch: lemme know if we do the 1:1 so I actually get onto the call [13:32] aghhh so close [13:32] prein a minute [13:32] perrito666: in a minute [13:38] voidspace, by the api i meant "extend the agent api as needed so that info is available", no need to make a separate call (like with Life()) [13:46] dimitern, voidspace: do you guys think you can get #1353443 fixed today? I'm anxious to get master open to commits again [13:47] mup is failing me - https://bugs.launchpad.net/juju-core/+bug/1353443 [13:47] natefinch: dimitern: I'm leaving in 15minutes I'm afraid... [13:48] natefinch, voidspace, I think it just got merged, no? [13:49] dimitern: no, the other one [13:49] oh, bugger [13:50] This one just got merged [13:50] https://bugs.launchpad.net/juju-core/+bug/1349635 [13:50] natefinch, voidspace, I'll take it and try to do a quick fix [13:50] #1353443 I mean [13:50] dimitern: thanks [17:23] be back in a couple hours.... taking my wife to her midwife appointment. === natefinch is now known as natefinch-afk === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha [23:30] anyone alive here/ [23:30] ?