| axw | waigani: #1270466 can be closed now? | 01:37 |
|---|---|---|
| axw | hm, mup doesn't recognise LP bug IDs anymore? :( | 01:37 |
| waigani | axw: closed | 01:38 |
| axw | cool | 01:39 |
| waigani | axw: so getting api endpoints at end of bootstrap. I can get machine 0's public address. But what about HA? I see a machine has a DistributionGroup. Should I use that to see if we are in HA and get the endpoints of the others? I think api.Client.APIHostPorts() would get them all, but that func is not allowed while 'in upgrade' | 01:42 |
| axw | waigani: there's only one machine during bootstrap. bootstrap already grabs that machine's address so it can SSH in | 01:43 |
| axw | waigani: see waitSSH in provider/common/bootstrap.go | 01:45 |
| waigani | axw: will do, cheers | 01:45 |
| axw | waigani: once Bootstrap has returned successfully, you should be able to assume that the instance has addresses, so you don't need the complicated logic from there | 01:45 |
| axw | you may just want to return the addresses from there tho | 01:46 |
| waigani | axw: ah, perfect. I assumed something like that was going on, as bootstrap finished but common.ProviderStateInstances was not finding any instances. I'll use waitSSH to wait | 01:47 |
| axw | waigani: just to be clear, waitSSH is already used by bootstrap. So you should just be able to bubble up the addresses it calculates | 01:49 |
| waigani | axw: I have not been able to do that so far | 01:50 |
| waigani | axw: but I'll have a look at how waitSSH is used and see what I'm missing | 01:51 |
| axw | waigani: provider/common.Bootstrap should return addresses. it should get them from FinishBootstrap, which should get them from waitSSH | 01:51 |
| waigani | axw: ah, the dummy bootstrap does not use common.Bootstrap | 01:52 |
| axw | waigani: you'll need to change Environ.Bootstrap's signature to return hostports, and dummy/manual/local will need different logic as they don't use common.Bootstrap | 01:53 |
| axw | not that straightforward; maybe you do want to make an API call for the first cut | 01:53 |
| axw | waigani: if you make an API connection, *any* method (assuming Login passes validation), then you | 01:54 |
| axw | 'll get back the hostports | 01:54 |
| axw | cached by juju.NewAPIConn | 01:54 |
| waigani | axw: sorry otp, reading now | 01:57 |
| waigani | axw: yeah, so I that is where I've ended up, dummy and local don't use common.Bootstrap, so I'm doing an API call to get endpoints (EnvCommandBase.NewAPIClient()). But client.APIHostPorts() is not allowed, because we are in 'upgrade' at the end of bootstrap. | 02:05 |
| waigani | axw: so I can only (so far) get the public address of machine 0 | 02:06 |
| axw | waigani: (you could just return localhost for local, and any old thing for dummy...) -- but you don't actually need to call APIHostPorts, you could just call, say, client.Status | 02:06 |
| axw | that is allowed during upgrade | 02:07 |
| axw | the act of making a method call will cache the addresses | 02:07 |
| axw | you don't need to explicitly do anything with that | 02:07 |
| axw | waigani: actually, I don't think you even need to make a method call. just open and close the API connection | 02:08 |
| waigani | axw: that was my first thought but thumper was adamant that just calling status does not work, he said he tried. I'll try myself (to be sure) | 02:08 |
| axw | waigani: you can see how it works in juju/api.go | 02:09 |
| axw | at the bottom of newAPIFromStore | 02:09 |
| waigani | axw: grrrr, just calling state bloody works. I'll try just connecting to the api | 02:14 |
| axw | waigani: just calling status you mean? | 02:15 |
| waigani | axw: yes sorry | 02:15 |
| axw | sure, just checking I understand | 02:15 |
| waigani | yep, that works - sigh | 02:19 |
| waigani | thanks axw | 02:19 |
| axw | waigani: cool, nps | 02:19 |
| bigjools | how much use do you guys make of syslog? I got the impression from the ML that you're thinking of stopping using it? | 02:20 |
| axw | bigjools: we don't use syslog directly, we write our logs to a file and have rsyslog tail it | 02:20 |
| axw | then they all send to a centralised rsyslog to aggregate | 02:21 |
| bigjools | ok | 02:21 |
| bigjools | In maas we're moving towards just sending everything to syslog and letting it do the heavy lifting | 02:21 |
| axw | bigjools: does that get sent anywhere remote? aggregated? | 02:22 |
| bigjools | the plan is to aggregate somewhere, yes, not sure where yet | 02:23 |
| axw | mmk | 02:23 |
| bigjools | well we don't have a "central" place | 02:23 |
| bigjools | so the decision is not easy :) | 02:24 |
| axw | when I said centralised, I actually meant to all of the juju state servers | 02:24 |
| axw | so it's still HA | 02:24 |
| * axw doesn't know MAAS well enough to draw parallels | 02:25 | |
| bigjools | right | 02:25 |
| bigjools | we might do something similar and sync to all appservers | 02:25 |
| === rogpeppe1 is now known as rogpeppe | ||
| dimitern | morning all | 07:17 |
| voidspace | yay, online | 07:31 |
| voidspace | morning everyone | 07:31 |
| TheMue | morning | 07:48 |
| tasdomas | could someone take a look at https://github.com/juju/juju/pull/464 ? | 08:26 |
| === rvba` is now known as rvba | ||
| TheMue | tasdomas: yep, will do. | 08:34 |
| tasdomas | TheMue, thanks! | 08:37 |
| fwereade | tasdomas, LGTM with a followup, I think the problems may run a little deeper | 08:37 |
| TheMue | fwereade: just seen you’ve LGTMed the PR | 08:38 |
| voidspace | hmm... when I try to submit a "national holiday" to hr.canonical.com it asks me to attach supporting evidence to the request... | 08:45 |
| mattyw | rogpeppe, would you be able to take another look at https://github.com/juju/juju/pull/469 | 08:56 |
| rogpeppe | mattyw: looking | 08:56 |
| mattyw | voidspace, https://www.gov.uk/bank-holidays | 08:57 |
| mattyw | voidspace, I think Auntie Liz has to approve bank holiday's each year so I did a quick google to see if I could find a pdf of her signature | 08:58 |
| rogpeppe | mattyw: i'm mostly +1 on fwereade's suggestion | 08:59 |
| rogpeppe | mattyw: except that i'd use Reset rather than creating a new timer each time | 09:00 |
| mattyw | rogpeppe, that is what's i'm doing in the latest code | 09:01 |
| rogpeppe | mattyw: oh, weird. i never quite get how github comments work | 09:02 |
| mattyw | rogpeppe, oh right - fwereade commented 4 minutes ago | 09:02 |
| mattyw | rogpeppe, so you were refering to a comment I hadn't seen yet :s - concurrency on a monday morning | 09:02 |
| rogpeppe | mattyw: his code is pretty much functionally equivalent, but slightly neater and fits with a common worker pattern | 09:03 |
| voidspace | mattyw: hehe, | 09:12 |
| * fwereade bbiab | 09:23 | |
| voidspace | TheMue: dimitern: internet has recovered, so I may make standup after all | 10:46 |
| voidspace | but first, coffee... | 10:46 |
| dimitern | voidspace, great :) see you there | 10:46 |
| perrito666 | morning | 11:27 |
| === Guest42485 is now known as wallyworld | ||
| tasdomas | could somebody take a look at https://github.com/juju/juju/pull/455 ? | 11:39 |
| wallyworld | katco: mgz: standup? | 12:02 |
| mgz | no katco yet alas. | 12:12 |
| tasdomas | I'm seeing a failure on master in the test suite for utils/ssh : http://paste.ubuntu.com/8016636/ - is anybody else seeing something like this? | 12:13 |
| mgz | tasdomas: nope, what does `ssh -V` say for you? | 12:15 |
| mgz | ...not that that should matter actualyl | 12:16 |
| tasdomas | mgz, ssh: Could not resolve hostname hostname: Name or service not known | 12:17 |
| mgz | tasdomas: your ssh looks broken then, check yuor network config? | 12:17 |
| tasdomas | mgz, that's the error I get when running the ssh line in terminal | 12:18 |
| tasdomas | mgz, ssh -V : OpenSSH_6.6.1p1 Ubuntu-2ubuntu2, OpenSSL 1.0.1f 6 Jan 2014 | 12:18 |
| mgz | oh, I was right, the ssh should in fact not be used | 12:20 |
| mgz | tasdomas: try `BASH_ENV="" go test`? | 12:23 |
| tasdomas | mgz, seems like it was a godeps issue - forgot to run godeps -u | 12:24 |
| mgz | fair enough. | 12:25 |
| tasdomas | mgz, and I need to update my vm | 12:25 |
| tasdomas | mgz, thanks | 12:28 |
| natefinch | fwereade: you around? | 13:49 |
| fwereade | natefinch, heyhey | 13:50 |
| fwereade | natefinch, how's it going? | 13:50 |
| natefinch | fwereade: not bad. Well rested from my week of not traveling ;) | 13:50 |
| wwitzel3 | lol, too soon | 13:52 |
| natefinch | fwereade: I'm a bit stuck on what to do about log rotation. There's a few different bad options, and I'm not sure how to choose between them | 13:52 |
| natefinch | wwitzel3: heh | 13:52 |
| fwereade | natefinch, haha | 13:52 |
| wwitzel3 | natefinch: I'm working on tht now, with rsyslog, should I not be, or you talking about Windows? | 13:53 |
| fwereade | natefinch, wwitzel3: windows is also interesting, gsamfira certainly has relevant input there | 13:53 |
| natefinch | wwitzel3: you're doing all-machines, right? | 13:54 |
| fwereade | natefinch, wwitzel3: I would hope that log rotation would encompass *all* our logs, even if the work came in a few stages | 13:54 |
| natefinch | fwereade: yes of course. There's really only a couple spots to hit - all-machines is a different beast than the machine-n and unit-n logs | 13:55 |
| fwereade | natefinch, cool | 13:55 |
| wwitzel3 | natefinch: yeah, all-machine | 13:56 |
| wwitzel3 | s | 13:56 |
| natefinch | wwitzel3: cool, yes, there's no real option for anything but logrotate / rsyslog rotation there. | 13:56 |
| gsamfira | not even if we log directly to rsyslog from juju? in the case of all-machines.log I mean | 13:57 |
| gsamfira | ? | 13:57 |
| natefinch | gsamfira: the rotation has to be on the state machine, which is listening using rsyslog | 13:57 |
| gsamfira | that's fine. That should be easily doable using standard logrotate | 13:58 |
| gsamfira | and for all other juju logs, lumberjack can take care of the rotation | 13:58 |
| gsamfira | no? | 13:58 |
| natefinch | gsamfira: yes, sorry if I wasn't clear. That's what wwitzel3 is doing | 13:58 |
| gsamfira | ahh | 13:58 |
| wwitzel3 | yep :) | 13:59 |
| gsamfira | my mistake. Read diagonally | 13:59 |
| natefinch | the thing that I'm talking about is rotating the unit-n and machine-n logs. Some people have questioned why we don't just use logrotate for those as well, and only make windows use something else. | 13:59 |
| natefinch | my preference would be that if we can do something the same everywhere, to do it the same, that way we don't need to maintain different codepaths depending on OS | 13:59 |
| gsamfira | well, as you pointed out on the mailing list, its better to have one config that works across platforms then 2-3 separate ones :) | 14:00 |
| natefinch | yes | 14:00 |
| gsamfira | yep | 14:00 |
| gsamfira | :) | 14:00 |
| gsamfira | the more os specific dependencies you can remove, the better | 14:00 |
| natefinch | I believe that as well | 14:00 |
| natefinch | fwereade: the question is - how far do we go? Do we write directly to syslog from inside juju, thereby bypassing rsyslog and not needing to worry about whether it misses log messages due to rotation? Or do we continue with rsyslog and figure out how to make that work with rotation? Or do you disagree with me. and think we should use logrotate on linux? | 14:02 |
| gsamfira | implementing syslog using Go, should be simple enough. I will cleanup the fork I made of the standard syslog package, and I can have a shot at implementing it in Juju by tomorrow. | 14:02 |
| fwereade | natefinch, so, if we can write directly to syslog, I would like that very much | 14:03 |
| fwereade | natefinch, I think thumper was going to reply to your post explaining the problems he'd found with go syslog | 14:03 |
| natefinch | fwereade: ahh, that would be very good to hear about. | 14:03 |
| fwereade | natefinch, apart from handwaving about how it outputs stuff is in some way inconvenient I can't actually remember the issue with it myself | 14:03 |
| gsamfira | hmm. Syslog implements io.Writer. In theory, it should be like writting to a file. The only thing is that you need to set a prefix, which you would normally do via the rsyslog config | 14:04 |
| gsamfira | but nay formatting should be done via loggo | 14:04 |
| natefinch | gsamfira: as long as loggo does it's output in a single write, that's ok. If it batches them or breaks a single line into multiple writes, it can be a problem. the syslog package expects each write to be a single log message. | 14:07 |
| gsamfira | I would like to try and tackle any issues that might have stopped us from using this. Until better log aggregation is found at least :) | 14:07 |
| natefinch | gsamfira: I think that would be great | 14:07 |
| wwitzel3 | natefinch: standup :) | 14:08 |
| gsamfira | natefinch: Hopefully loggo adheres to that expectation as well :). I'll have a look | 14:10 |
| katco | good morning all | 14:32 |
| katco | mgz: wallyworld: sorry about missing the standup | 14:33 |
| perrito666 | natefinch: wwitzel3 ericsnow suddenly the page died on my and I can no longer connect sorry | 14:33 |
| mgz | katco: no problem, ian will catch us up on last week tomorrow | 14:33 |
| katco | mgz: ok cool | 14:34 |
| natefinch | perrito666: np | 14:35 |
| mgz | natefinch: have you seen bug 1355219? | 14:48 |
| bodie_ | morning all | 14:51 |
| natefinch | mgz: go get gopkg.in/natefinch/npipe.v2 works for me | 14:58 |
| natefinch | mgz: It looks like we assume juju-core_1.21-alpha1.tar.gz has it in there.... we should look at how that's generated and why it's not getting juju's dependencies. | 15:01 |
| natefinch | mgz: my guess is that it solely uses godeps and someone either messed up godeps or forgot to update it. | 15:02 |
| natefinch | brb | 15:02 |
| dimitern | anybody willing to review a quick fix https://github.com/juju/juju/pull/491 for bug https://bugs.launchpad.net/juju-core/+bug/1353443 ? | 15:28 |
| dimitern | TheMue, rogpeppe, voidspace, fwereade ^^ | 15:28 |
| rogpeppe | dimitern: looking | 15:28 |
| dimitern | rogpeppe, ta! | 15:29 |
| rogpeppe | dimitern: what if we're running inside kvm inside lxc? should that trigger the same logic? | 15:29 |
| dimitern | rogpeppe, so the kvm is the host, and the lxc is where the MA+networker runs? | 15:30 |
| rogpeppe | dimitern: i was thinking the other way around actually | 15:30 |
| dimitern | rogpeppe, i don't think you can run kvm inside lxc | 15:31 |
| rogpeppe | dimitern: you sure about that? | 15:31 |
| dimitern | rogpeppe, 99% | 15:31 |
| dimitern | rogpeppe, it doesn't make sense, as kvm needs full access to hardware virtualization, which an lxc host cannot provide | 15:32 |
| rogpeppe | dimitern: i'm still a bit uncomfortable about assuming that there's no container type that can run inside lxc | 15:33 |
| rogpeppe | dimitern: now and for the indefinite future | 15:33 |
| dimitern | rogpeppe, lxc can run inside lxc | 15:33 |
| rogpeppe | dimitern: ok, no non-lxc container type :-) | 15:34 |
| rogpeppe | dimitern: i'd be happier if the logic looked at the last container type | 15:34 |
| rogpeppe | dimitern: because that's what it actually cares about, no? | 15:34 |
| dimitern | rogpeppe, you mean the logic about parsing the machine id ? | 15:35 |
| rogpeppe | dimitern: yes | 15:35 |
| rogpeppe | dimitern: specifically the fact you use strings.Contains | 15:36 |
| natefinch | sinzui: you around? | 15:36 |
| dimitern | rogpeppe, well, my idea there is that if lxc is involved at any level of nesting, we shouldn't modprobe | 15:37 |
| natefinch | mgz: I presume that 1.21 build is running off of trunk? | 15:38 |
| mgz | natefinch: yup | 15:38 |
| voidspace | dimitern: that's a big test for a little bit of code | 15:38 |
| natefinch | mgz: k thanks | 15:38 |
| dimitern | voidspace, it's a fairly involved worker for all it does :) | 15:39 |
| voidspace | heh, right | 15:39 |
| dimitern | voidspace, we need to simplify it | 15:39 |
| voidspace | "we" | 15:39 |
| voidspace | :-p | 15:39 |
| voidspace | I'm going afk for a while (two hours) | 15:40 |
| rogpeppe | dimitern: i have difficulty working out if that's a valid decision or not | 15:41 |
| rogpeppe | dimitern: because it's not clear to me that you might not be able to nest some container type that *does* allow modprobe | 15:41 |
| dimitern | rogpeppe, i'm chatting with jamespage and hallyn at #server (@c) about this actually | 15:41 |
| jamespage | dimitern, rogpeppe: lets talk here | 15:41 |
| dimitern | jamespage, right, sorry about the context switching :) | 15:43 |
| jamespage | dimitern, np | 15:43 |
| dimitern | jamespage, so, is my idea about not allowing modprobe if lxc is involved at any level of nesting, not only the last level? | 15:44 |
| jamespage | dimitern, rogpeppe: its possible to hack an LXC container sufficiently so it can run a KVM instance - but this still runs within the host kernel | 15:44 |
| dimitern | is it a correct approach to the problem, i mean | 15:44 |
| jamespage | dimitern, a KVM instance will always support modprobe as it has its own kernel etc... | 15:45 |
| jamespage | dimitern, LXC will not and should not | 15:45 |
| jamespage | dimitern, this is why its possible to run the nova-compute charm on KVM, but not LXC | 15:46 |
| dimitern | jamespage, so you can modprobe 8021q for example inside the kvm, hosted inside an lxc without 8021q on its respective host? | 15:46 |
| jamespage | dimitern, yes | 15:46 |
| jamespage | dimitern, in the very hyperthetical situation that is true | 15:46 |
| dimitern | jamespage, ok, so to be absolutely correct, we should check what rogpeppe suggests - only the last-level-of-nesting container, which runs the agent should not be lxc | 15:47 |
| rogpeppe | i think we should allow that case, even if hypothetical | 15:47 |
| jamespage | dimitern, yes | 15:47 |
| dimitern | jamespage, rogpeppe, ok, fair enough, thanks for the feedback | 15:48 |
| rogpeppe | dimitern: another approach might be just to ignore that error from modprobe | 15:48 |
| rogpeppe | dimitern: that way if someone fixes lxc, the code will just work again | 15:49 |
| dimitern | rogpeppe, I don't really like this | 15:49 |
| dimitern | rogpeppe, because it's hard enough to deal with the output lsmod and modprobe give anyway | 15:50 |
| rogpeppe | dimitern: yeah | 15:50 |
| rogpeppe | dimitern: another thought is to put the "isRunningInLXC" logic inside (or called from) ensureVLANModule | 15:51 |
| rogpeppe | dimitern: because at the moment it's really not clear why we call isRunningInLXC before calling ensureVLANModule | 15:51 |
| rogpeppe | dimitern: but if we passed the machine id into ensureVLANModule and did the check in there, the connection is obvious (and the comments would read nicely too) | 15:52 |
| dimitern | rogpeppe, ensureVLANModule should not be called like this, but this is another issue - we should only call it, if there are any VLANs to setup | 15:53 |
| jcw4 | Fix for LP-1355319 : https://github.com/juju/juju/pull/493 | 15:59 |
| dimitern | rogpeppe, updated isRunningInLXC a bit to check the last level container type | 16:00 |
| rogpeppe | dimitern: looking | 16:01 |
| rogpeppe | dimitern: i don't think that will actually work, will it? | 16:01 |
| natefinch | I hate it when I know I'm doing something stupidly wrong.... but still can't figure out what it is | 16:02 |
| dimitern | rogpeppe, why? | 16:02 |
| rogpeppe | dimitern: don't you want something like: if len(parts) > 2 && parts[len(parts)-2] == "lxc" { ? | 16:02 |
| rogpeppe | dimitern: because AFAICS the logic you've got now checks whether *any* of the splits holds the string "lxc" | 16:03 |
| dimitern | rogpeppe, ah, because it could be "0", yes ofc | 16:03 |
| dimitern | rogpeppe, it checks the last split first going forward | 16:03 |
| rogpeppe | dimitern: better test case required :-) | 16:03 |
| rogpeppe | dimitern: i'd write a test case for isRunningInLXC directly | 16:03 |
| dimitern | rogpeppe, you mean add a test for isRunningInLXC specifically? | 16:03 |
| dimitern | ok | 16:03 |
| rogpeppe | dimitern: it would seem worth it, yes | 16:04 |
| rogpeppe | dimitern: your current logic doesn't seem to test the last split moving forward... | 16:05 |
| jcw4 | wallyworld, perrito666 since you're on call reviewers :) PTAL https://github.com/juju/juju/pull/493 | 16:05 |
| jcw4 | trivial dependencies.tsv fix | 16:06 |
| perrito666 | jcw4: certainly | 16:06 |
| jcw4 | perrito666: tx | 16:06 |
| jcw4 | natefinch: shall I just close https://github.com/juju/juju/pull/493 ? (perrito666) | 16:08 |
| natefinch | jcw4: leave it open for now, if I can't get my fix in before yours, we can just use yours. But I'd like to see if I can get mine in so godeps produces the correct result | 16:09 |
| jcw4 | natefinch: +1 :) | 16:10 |
| perrito666 | jcw4: I was really expecting a larger patch | 16:12 |
| jcw4 | perrito666: :) | 16:12 |
| rogpeppe | dimitern: what calls networker.SetUp ? | 16:12 |
| rogpeppe | dimitern: ah, it's StringsWorker | 16:13 |
| rogpeppe | dimitern: why are privateInterface and priviateBridge global variables? | 16:13 |
| rogpeppe | dimitern: and unguarded by a mutex | 16:13 |
| rogpeppe | dimitern: that gets my trouble whiskers twitching | 16:14 |
| dimitern | rogpeppe, that's all vlad's code, and needs some work | 16:15 |
| rogpeppe | dimitern: but it's landed in core? hmm. | 16:15 |
| dimitern | rogpeppe, yes, but it's hardly ever used for now - only with maas and only when there are networks defined | 16:16 |
| rogpeppe | dimitern: yeah but... sigh | 16:16 |
| dimitern | rogpeppe, I agree it needs refactoring, there are tech dept bugs for some things already | 16:18 |
| dimitern | rogpeppe, thanks for bearing with me :) | 16:19 |
| dimitern | rogpeppe, updated yet again - isrunninginlxc fixed + added a test | 16:20 |
| dimitern | rogpeppe, all comments addressed or replied to, can I get LGTM? :) | 16:29 |
| katco | is there a way to do an in-place branch in bazaar? creating a branch for some code changes is messing up the imports since it creates a new directory. | 16:29 |
| rogpeppe | dimitern: i'm still trying to grok the test | 16:29 |
| dimitern | rogpeppe, so there's a fair amount of boilerplate setup code in there | 16:31 |
| dimitern | rogpeppe, the main thing is patching the executeCommands and monitoring modprobe does not happen before ifup commands | 16:32 |
| dimitern | rogpeppe, I tested if fails when I comment out the && !isRunningInLXC part | 16:32 |
| jcw4 | natefinch: the more I think about it the more I think I should just close my PR and let you do the full fix. If we're going to fix this bug we should fix it right and your plan sounds right :) | 16:39 |
| natefinch | jcw4: I have mine ready to propose anyway | 16:40 |
| jcw4 | natefinch: perfect; thx | 16:40 |
| rogpeppe | dimitern: review done | 16:41 |
| === Ursinha is now known as Ursinha-afk | ||
| dimitern | rogpeppe, thanks! | 16:42 |
| natefinch | rogpeppe: quick review? https://github.com/juju/juju/pull/494 | 16:44 |
| natefinch | jcw4: ^^ | 16:47 |
| rogpeppe | natefinch: i hope you mean "godeps -t ./..." | 16:47 |
| natefinch | oh oops | 16:48 |
| natefinch | rogpeppe: no, but they apparently produce the same results on github.com/juju/juju | 16:49 |
| natefinch | (currently) | 16:50 |
| rogpeppe | natefinch: i guess that's not actually too surprising | 16:50 |
| rogpeppe | natefinch: because of the testing packages | 16:50 |
| dimitern | wtf is this?! "Build failed: Does not match ['fixes-1355219', 'fixes-1347715']\n build url: http://juju-ci.vapour.ws:8080/job/github-merge-juju/261\n" | 16:50 |
| natefinch | rogpeppe: yep, was thinking that | 16:50 |
| dimitern | neither of those were mentioned in the PR at all | 16:50 |
| rogpeppe | dimitern: you *need* to mention those in the PR | 16:51 |
| rogpeppe | dimitern: because they're the critical bugs that currently need fixing | 16:51 |
| natefinch | dimitern: master is locked | 16:51 |
| dimitern | natefinch, rogpeppe, I see, ok | 16:51 |
| natefinch | dimitern: I have this labeled as "CI blockers" in my bookmarks bar: https://bugs.launchpad.net/juju-core/+bugs?field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.importance%3Alist=CRITICAL&field.tag=ci+regression+&field.tags_combinator=ALL | 16:51 |
| dimitern | nothing to worry about then, it will get picked up by the bot when trunk is unblocked? | 16:51 |
| natefinch | that I'm not sure about | 16:52 |
| natefinch | you might need to repropose | 16:52 |
| * rogpeppe adds that to his bookmarks bar too | 16:52 | |
| rogpeppe | i actually want a status constantly showing whether juju-core is currently blocked | 16:52 |
| * dimitern as well, thanks! | 16:52 | |
| natefinch | (that link is from curtis) | 16:52 |
| natefinch | rogpeppe: challenge: add it to your shell prompt ;) | 16:53 |
| rogpeppe | natefinch: i don't believe in dynamic shell prompts | 16:53 |
| natefinch | rogpeppe: ha ok | 16:53 |
| rogpeppe | natefinch: too much noise | 16:53 |
| rogpeppe | natefinch: i just have a single character for my prompt | 16:54 |
| natefinch | rogpeppe: I find it handy for seeing the current git branch, but yeah, I get that it can be noisy | 16:54 |
| natefinch | rogpeppe: where would you want to see the notification that master is blocked? | 16:54 |
| rogpeppe | natefinch: in the (largely blank) notification bar at the top of my screen | 16:55 |
| dimitern | rogpeppe, you won't like my shell prompt then: [maas-trusty] (lp-1353443-dont-modprobe-in-lxc) dimitern@kubrik:~/work/juju-core$ | 16:55 |
| dimitern | :) | 16:55 |
| natefinch | rogpeppe: fair enough. I hhave no idea how to do that. | 16:56 |
| rogpeppe | dimitern: i'm surprised you've got any room to type a command there! :-) | 16:56 |
| dimitern | rogpeppe, like most problems in the world, this as well can be solved with a bigger screen :D | 16:56 |
| rogpeppe | for many of the shell commands i execute, i don't have a command prompt at all :-) | 16:56 |
| rogpeppe | dimitern: yeah, i could do with a 6 foot by 6 foot screen :-) | 16:57 |
| rogpeppe | natefinch: why does your PR update the mgo dependency too? | 16:58 |
| dimitern | rogpeppe, but then again... who couldn't! | 16:58 |
| * dimitern needs to stop for today | 16:58 | |
| dimitern | DmC hack&slash awaits! | 16:59 |
| natefinch | rogpeppe: it's the same commit, it was just in the wrong (not alphabetical) spot | 16:59 |
| rogpeppe | natefinch: ha, good point :-) | 17:00 |
| rogpeppe | dimitern: enjoy! | 17:00 |
| natefinch | rogpeppe: but thanks for checking... I realized I hadn't actually double checked that the commit numbers were the same. | 17:01 |
| rogpeppe | natefinch: reviewed | 17:01 |
| natefinch | rogpeppe: what portable API could it export? | 17:02 |
| rogpeppe | natefinch: a named pipe implementation that works on all platforms? | 17:03 |
| perrito666 | rogpeppe: that is a bit overkill | 17:04 |
| gsamfira | named pipes on windows are a bit different then on linux. They are more akin to unix sockets. | 17:04 |
| natefinch | rogpeppe: it's just an implementation for net.Listener and net.Conn | 17:05 |
| natefinch | (etc) | 17:05 |
| === jog_ is now known as jog | ||
| rogpeppe | natefinch: the portable API could be almost exactly that exported by the juju/sockets package | 17:07 |
| rogpeppe | natefinch: (speaking of which, why is that package under juju at all?) | 17:08 |
| natefinch | rogpeppe: that's where a bunch of our OS-specific abstractions live... probably just by accident, because thats where osenv lived first | 17:09 |
| rogpeppe | natefinch: osenv *is* very juju specific | 17:10 |
| rogpeppe | natefinch: sockets is not at all juju specific | 17:10 |
| rogpeppe | natefinch: FWIW, i'm not really sure why we're using named pipes at all under windows. we could presumably just use localhost sockets instead (the current port could be stored in the unit agent directory) | 17:13 |
| rogpeppe | natefinch: hmm, just thought: does juju-run mean that any process on the host can get root privileges? | 17:14 |
| sinzui | hi natefinch | 17:15 |
| natefinch | sinzui: hey | 17:15 |
| gsamfira | rogpeppe: that's how it was implemented in the PoC (tcp sockets bound to localhost). The run.socket actually held 127.0.0.1:<PORT> :) | 17:16 |
| rogpeppe | gsamfira: PoC ? | 17:16 |
| natefinch | proof of concept | 17:16 |
| gsamfira | Proof of Concept | 17:16 |
| rogpeppe | gsamfira: ah. that seems fine to me. apart from you can't hide the socket inside a root-accessible-only directory, i guess. which i *hope* we're doing currently. | 17:17 |
| gsamfira | TCP sockets are slower though. It was noticeable when running juju commands. | 17:17 |
| gsamfira | well, on windows at least, named pipes are not represented on disk | 17:18 |
| rogpeppe | gsamfira: *really*? the sockets are that high-bandwidth? | 17:18 |
| gsamfira | its not bandwidth...but the whole protocol stack init process and round trip | 17:18 |
| gsamfira | named pipes are recommended for Inter-process communication on windows. Chrome/Firefox use them | 17:19 |
| gsamfira | overhead is minimal | 17:19 |
| gsamfira | and they behave like unix sockets | 17:19 |
| gsamfira | but with a file API | 17:19 |
| gsamfira | so you treat them as you would any file | 17:19 |
| rogpeppe | gsamfira: i guess that a hook will make quite a lot of these connections | 17:19 |
| gsamfira | but it allows bidirectional communication | 17:19 |
| gsamfira | yes | 17:19 |
| gsamfira | most do | 17:19 |
| gsamfira | even juju-run was slower then with named pipes | 17:20 |
| gsamfira | the biggest issue was allocating ports though | 17:20 |
| rogpeppe | gsamfira: that was slow? | 17:20 |
| gsamfira | because there is no database locally, you needed to find a free one | 17:20 |
| gsamfira | and each time you deploy a new charm, you need to make sure you don't overlap | 17:21 |
| gsamfira | a simple directory listing via juju-run would take about a second | 17:21 |
| gsamfira | :) | 17:21 |
| gsamfira | so in terms of allocating a named pipe, its simple. you have `\\.\pipe\unit-tag-run` and `\\.\pipe\unit-tag-agent` | 17:22 |
| gsamfira | ohh, and it was easier for an installed application (by a charm for example) to take over the port you allocated. So when a juju command would try to run, it would fail to bind on that port. | 17:24 |
| rogpeppe | gsamfira: couldn't you just let the OS allocate one for you? | 17:25 |
| rogpeppe | gsamfira: i guess that last is a potential problem though | 17:26 |
| gsamfira | you could in theory allocate a different port each time. | 17:26 |
| gsamfira | but in the end it was simpler just to use nate's named pipes module :) | 17:26 |
| rogpeppe | gsamfira: :-) | 17:26 |
| gsamfira | named pipes is an unfortunate name | 17:27 |
| rogpeppe | gsamfira: yeah. | 17:27 |
| gsamfira | especially considering that on unix it has such different behavior | 17:27 |
| natefinch | namespaces are important :) | 17:28 |
| rogpeppe | gsamfira: a better name might be fsnet, i suppose (for "file-system network") | 17:28 |
| natefinch | windows.NamedPipe != linux.NamedPipe | 17:28 |
| rogpeppe | natefinch: you couldn't possibly know how much i agree with you there :-) | 17:28 |
| rogpeppe | natefinch: (about namespaces) | 17:29 |
| gsamfira | yep :) | 17:29 |
| rogpeppe | natefinch: i still think that npipe should export a portable interface | 17:29 |
| gsamfira | well, its kind of hard to do | 17:29 |
| rogpeppe | natefinch: so that anyone that wants a filesystem based listener can use it, and their code will work regardless of whether it's on windows or not | 17:29 |
| gsamfira | named pipes on windows allows bi-directional communication, whereas on linux it does not | 17:29 |
| natefinch | rogpeppe: it's specifically a windows implementation... I think it would be a good idea to provide a different package that uses it to produce a platform-agnostic API | 17:30 |
| natefinch | rogpeppe: maybe the problem is just that it should be called winnpipe or something | 17:30 |
| rogpeppe | natefinch: what is in the npipe API that's windows-specific? | 17:31 |
| gsamfira | whatever it ends up being called, I would love to see it in the standard "net" package :) | 17:31 |
| perrito666 | natefinch: why dont you call the thing winPipes :p | 17:31 |
| rogpeppe | natefinch: i know you've written it as a window-specific package, but i don't see why it has to be. by making it portable, you make it more useful. | 17:32 |
| rogpeppe | s/window/windows/ | 17:32 |
| rogpeppe | natefinch: i guess the name space argument looks quite different | 17:33 |
| gsamfira | rogpeppe: I'm not sure named pipes on Linux can be used in the same way | 17:33 |
| rogpeppe | gsamfira: it wouldn't use named pipes on linux | 17:33 |
| rogpeppe | gsamfira: it would use unix-domain sockets | 17:33 |
| gsamfira | hmm, but the net package already does that | 17:33 |
| rogpeppe | gsamfira: sure. but not for windows. | 17:33 |
| === ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs 1355320, 1347715, 1355324 | ||
| gsamfira | if nate's implementation makes it into the stdlib, we should be all set :) | 17:33 |
| rogpeppe | gsamfira: so this would be a portable way of getting the same functionality using either net directly or the windows named pipe stuff | 17:34 |
| === Ursinha-afk is now known as Ursinha | ||
| gsamfira | my hope is that at some point we could do a rpc.Dial("local", path) and have the rpc package use named pipes on windows | 17:36 |
| gsamfira | and unix sockets in linux | 17:36 |
| gsamfira | without any 3rd party package | 17:36 |
| gsamfira | but I see what you are saying | 17:36 |
| gsamfira | a helper package would be ok until that happens | 17:37 |
| rogpeppe | gsamfira: rpc doesn't have a Dial function :-) | 17:38 |
| rogpeppe | gsamfira: well, the juju one anyway | 17:38 |
| natefinch | rogpeppe: yeah, the real problem is that net.Listen("namedpipe", "\\.\foo") doesn't call my code on windows :) | 17:38 |
| natefinch | rogpeppe: that's the cross-platform API | 17:38 |
| rogpeppe | natefinch: we should always try to make cross-platform APIs when possible, in my view | 17:39 |
| natefinch | rogpeppe, gsamfira: I'd love to get the package into the stdlib... I don't know how likely that is, though. | 17:39 |
| rogpeppe | natefinch: platoform-specific code should be pushed down to as low a level as possible | 17:39 |
| natefinch | rogpeppe: I agree completely | 17:39 |
| natefinch | rogpeppe: like I said, it should live in platform-specific code inside net, if that were possible | 17:40 |
| rogpeppe | natefinch: sure, but given that it can't why not provide an interface similar to what you'd hope the net package *should* provide | 17:40 |
| rogpeppe | ? | 17:40 |
| rogpeppe | natefinch: so that folks can use the abstraction your package provides without needing to write windows-specific shims like our juju/sockets package | 17:41 |
| gsamfira | I guess it should be a simple addition. something like fsnet.Listen(path) and fsnet.Dial(path). Basically what the sockets package does :) | 17:42 |
| gsamfira | but if you plan on proposing it for stdlib, I can see why you would not want this | 17:42 |
| natefinch | rogpeppe: I dunno, I prefer just having a package that is compatible with the net package. I don't think it's really that bad, and it keeps my package more simple | 17:42 |
| rogpeppe | natefinch: for instance, i think it's really crappy that the highest level code in juju (cmd/jujud) needs to define RunCommand.winSockPath | 17:42 |
| natefinch | rogpeppe: that could live somewhere less prominent | 17:43 |
| rogpeppe | natefinch: it's only compatible with the net package on windows. | 17:43 |
| natefinch | rogpeppe: it only need to be compatible with the net package on windows. It can't run on linux. | 17:43 |
| rogpeppe | natefinch: if the abstraction was right, there would be no need for two separate paths | 17:43 |
| rogpeppe | natefinch: the abstraction can surely be implemented under linux | 17:43 |
| gsamfira | that's my fault. I'll have a go at cleaning that up as soon as the final bits of the windows support is in. apologies | 17:45 |
| natefinch | rogpeppe: both sides have to agree to use named pipes. You can't have both sides randomly picking some transport and assume they'll line up. | 17:46 |
| natefinch | rogpeppe: so, at some point, you have to say "hey, used named pipes", and that's only possible on windows | 17:46 |
| rogpeppe | natefinch: sure. you'd need to document that it use unix sockets under linux and named pipes under windows. | 17:46 |
| natefinch | rogpeppe: named pipes work across the network too. You can't connect to a named pipe on another machine from a unix socket on this machine | 17:47 |
| rogpeppe | natefinch: and you'd need to document how the mapping is made between dial/listen addresses and the underlying named pipe/socket addresses | 17:47 |
| rogpeppe | natefinch: that's fine - it's an implementation restriction. | 17:47 |
| natefinch | rogpeppe: I don't like letting someone do a listen on one machine and a dial on the other using the same code and then have it not work because one is windows and one is linux. | 17:48 |
| rogpeppe | natefinch: obviously you can't accomplish the impossible. | 17:49 |
| rogpeppe | natefinch: if you defined things right, it would be obvious when you were dialling or listening in an OS-specific name space | 17:50 |
| natefinch | rogpeppe: I think that sometimes making it ugly and obvious is better than making it pretty and obscured. The fact that both the client and server need to be on the same OS makes me want to make sure it's super obvious this is windows-only, and you can't just listen from linux and have it work. | 18:04 |
| rogpeppe | natefinch: if i was doing it, i might make a package interface something like this: http://paste.ubuntu.com/8019246/ | 18:04 |
| rogpeppe | natefinch: i think the fact that the paths have backslashes in is a pretty good indication of that | 18:05 |
| rogpeppe | natefinch: i suspect that most people will be using named pipes for intra-machine communication | 18:05 |
| rogpeppe | natefinch: (we certainly are) | 18:05 |
| natefinch | rogpeppe: we are, and that's why we can safely just pick whichever transport and know the other side will work. But I don't want my very specific package to encourage generic use without people thinking about it. | 18:08 |
| rogpeppe | natefinch: why would anyone use your package unless they knew what they needed it for? | 18:09 |
| rogpeppe | natefinch: i just hate to see OS-specific stuff spread throughout juju-core unnecessarily | 18:10 |
| rogpeppe | anyway, gotta go | 18:10 |
| rogpeppe | natefinch: ttfn | 18:10 |
| natefinch | rogpeppe: see ya | 18:12 |
| natefinch | rogpeppe: FWIW, I don't disagree. an OS-agnostic wrapper around npipe is certainly welcome, I just don't think it belongs *in* npipe :) | 18:12 |
| natefinch | CI blockers are like a hydra... fix one and two more appear | 18:36 |
| perrito666 | natefinch: what broke now? | 18:38 |
| natefinch | perrito666: HA, what else? | 18:39 |
| perrito666 | ... | 18:39 |
| perrito666 | could you pass me the search link again? | 18:39 |
| natefinch | perrito666: https://bugs.launchpad.net/juju-core/+bugs?field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.importance%3Alist=CRITICAL&field.tag=ci+regression+&field.tags_combinator=ALL | 18:39 |
| natefinch | perrito666: you should bookmark it :) | 18:40 |
| natefinch | perrito666: oh yeah, and restore :/ | 18:40 |
| perrito666 | natefinch: I bookmarked it on the other computer sorry | 18:40 |
| natefinch | perrito666: heh it's ok :) | 18:40 |
| natefinch | perrito666: chrome sync would fix that if you were using it :) | 18:40 |
| perrito666 | natefinch: I am using chrome, which is odd, I guess when the computer froze during the hangout it caused it not to sync | 18:41 |
| natefinch | ah, huh. Wonder if it was an in-memory cache that didn't get written out | 18:42 |
| perrito666 | natefinch: dunno, should be ok now | 18:42 |
| perrito666 | mmm, restore breakage makes no sense | 18:47 |
| sinzui | perrito666, I am going to try changing the restore command in the test to use --show-log to get more info | 19:29 |
| perrito666 | sinzui: thank you | 19:29 |
| natefinch | heh, git diff <commit> HEAD makes a lot more sense than git diff HEAD <commit> .... the later makes it look like we removed a whole bunch of functionality, stopped using gopkg.in, etc | 19:31 |
| === Ursinha is now known as Ursinha-afk | ||
| sinzui | perrito666, --show-log isn't giving more information. I will add an attempt to get logs from the machine that added | 20:16 |
| perrito666 | sinzui: --debug perhaps? | 20:17 |
| perrito666 | if not I can reproduce this test in a moment | 20:17 |
| perrito666 | I have a doctor appointment but will be back later | 20:17 |
| dpb1 | Hi -- testing 1.20.4 -- I'm having trouble getting this openstack cloud to bootstrap: http://paste.ubuntu.com/8020244/ -- I suspect that the cloud api endpoint is not getting written out correctly somewhere. Anyone seen something like this? | 20:18 |
| sinzui | perrito666, Somone need to ensure --debug stops leaking credentials. really! no person can post a log with --debug because the whole work gets the keys to use their account | 20:18 |
| * perrito666 agrees with sinzui | 20:18 | |
| perrito666 | sinzui: what are you running the tests with, jenkins or your box? | 20:19 |
| perrito666 | if its your own box you could just mail me the results, I do have access to these credentials anyway, I think | 20:19 |
| perrito666 | now fine people if you excuse me, I really, really need to the doc | 20:20 |
| perrito666 | might be back online from the waiting room | 20:20 |
| sinzui | perrito666, I may do. I am trying to create a special log when a failure is detected and the env is still up | 20:20 |
| perrito666 | I usually do that with the restore test by getting the logs before the destroy in finally, even when the tests dond fail | 20:22 |
| sinzui | perrito666, I am trying to capture the new ip of the machine. It is in the string of the error message. I hope to get the log a the moment before destroy is called | 20:25 |
| dpb1 | n/m, I've sorted it | 20:35 |
| === Ursinha-afk is now known as Ursinha | ||
| dpb1 | sinzui: https://bugs.launchpad.net/juju-deployer/+bug/1243827 -- are the juju tools compiled against an old goyaml by any chance? | 21:21 |
| dpb1 | sinzui: I'm getting something very similar when the tools jujud reads in a yaml with an underscore in the openstack password. | 21:22 |
| sinzui | dpb1, The tools are from the debs we build. the debs used the tarball. The tarball pinned the deps to the version specified by juju-core | 21:22 |
| dpb1 | sinzui: ok. I guess I'll file a new bug then? That same symptom seems to be happening with jujud from 1.20.4. jujud reads in the environment on the bootstrap node, and if the password contains an '_', it gets stripped. | 21:25 |
| sinzui | dpb1, both devel and stable are 1 version behind | 21:25 |
| sinzui | dpb1, I see the fix was made...but the engineer never updated juju to use it :( | 21:28 |
| dpb1 | oh dear | 21:28 |
| dpb1 | :( | 21:28 |
| sinzui | dpb1, https://bugs.launchpad.net/juju-deployer/+bug/1243827 | 21:29 |
| sinzui | ^ the bug wrongly says it is invalid for juju-core. | 21:29 |
| dpb1 | ohhh | 21:29 |
| sinzui | I will correct the bug and target it to 1.20 now | 21:30 |
| dpb1 | sinzui: thanks! I'll add our tags so we can track. appreciate it | 21:30 |
| menn0 | sinzui: have you had a chance to look at those CI log archiving changes I sent you? | 21:39 |
| menn0 | sinzui: they could be useful in figuring out the current blockers | 21:39 |
| sinzui | menn0, I was looking for them and couldn't find them. I thought you forgot to request a merge | 21:39 |
| menn0 | sinzui: I don't think I'm allowed to request merges for that project or can I? | 21:40 |
| menn0 | I sent you the changes in an email | 21:40 |
| menn0 | as a bzr merge directive | 21:40 |
| sinzui | menn0, https://code.launchpad.net/juju-ci-tools is a public project | 21:40 |
| menn0 | you can bzr pull the changes straight out | 21:40 |
| menn0 | sinzui: ok. let me try again. | 21:41 |
| sinzui | thank you menn0 | 21:55 |
| menn0 | sinzui: https://code.launchpad.net/~menno.smits/juju-ci-tools/log-dumping-improvements/+merge/230398 | 21:55 |
| menn0 | sinzui: I blame my complete inexperience with Launchpad | 21:55 |
| sinzui | menn0, hmm, at first glance, this wont help the current bug because the bootstrap host may not exist after the restore was attempted | 21:58 |
| menn0 | sinzui: yep you're right. dump_env_logs assumes the bootstrap node is available. | 21:59 |
| menn0 | it should help with the HA blockers though | 21:59 |
| sinzui | menn0, okay that means my current branch is still valid. I am searching for the address of the new instance. I will base my branch on yours though. | 22:00 |
| menn0 | sinzui: where are you searching for the address? | 22:01 |
| sinzui | menn0, the address might be in the output of error that is raised. I can see the "attempting to connect (.*)" lines, so I could use the last one as an address to ssh to | 22:02 |
| menn0 | sinzui: that could work | 22:03 |
| sinzui | menn0, does dump_euca_console work with openstack. that is where the tests can be run | 22:06 |
| sinzui | and joyent | 22:06 |
| sinzui | and azure | 22:06 |
| menn0 | sinzui: apart from being in a new function, that code hasn't changed | 22:07 |
| menn0 | dump_euca_console doesn't do anything unless the host_id is set | 22:07 |
| sinzui | menn0, I see. That code is very naive | 22:08 |
| menn0 | waigani: could the failures at the bottom of bug #1347715 be related to your recent work? | 22:20 |
| sinzui | menn0, I merged your branch. I ponder one more change though | 22:39 |
| sinzui | menn0, We really shouldn't try to get bootstrap_host (ip) after is was destroyed. that undermines getting all the logs from all the machines. | 22:40 |
| sinzui | menn0, I think we should set bootstrap_host to None to encourage your new functions to get everything it can from what ever exists in the env | 22:40 |
| waigani | menn0: I've read through bug and back through my possibly related change sets, nothing jumps out. were you thinking of anything in particular? | 22:41 |
| menn0 | waigani: just that it appears to be SSH related | 22:59 |
| menn0 | waigani: it might be a completely unrelated problem | 22:59 |
| menn0 | waigani: so feel free to ignore me | 22:59 |
| menn0 | sinzui: the idea behind passing in the bootstrap host address is that it allows retrieval of at least some logs even if "juju status" is no longer functioning | 23:05 |
| menn0 | sinzui: generally tests seem to be remembering the bootstrap host address before doing something drastic | 23:05 |
| menn0 | sinzui: thinking out loud... perhaps a better approach might be to ask the underlying provider infrastructure for machine addresses if "juju status" fails. | 23:07 |
| menn0 | sinzui: we could then remove the bootstrap_host arg to dump_env_logs | 23:08 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!