[00:08] top - 11:08:40 up 16 days, 1:34, 9 users, load average: 5.09, 2.39, 1.02 [00:27] davecheney: is it fixed already? [00:44] i have a partial fix that fixes the link error during compilation [00:45] i'm going (20 mins so far ...) [00:45] doing a [00:45] full test run to see where we're at [00:45] the fix isn't suitable for upstream, but it gives me a starting point [00:45] thumper: http://paste.ubuntu.com/7154127/ [00:45] * thumper nods [00:46] that isn't looking to bad [00:46] the fix is basically to reverse the order of the .a files presented to the linker [00:46] but a more sophisticated fix will probably be needed [00:46] looks like an ordering issue for those failures [00:47] yeah, i suspect that once the tests build [00:47] the fixes to the failed test cases will be fairly straight forward [00:54] wallyworld: https://codereview.appspot.com/22320043/ [00:54] wallyworld: resurrecting a 4 month old branch [00:54] ok [00:54] because I want it for my plugin [00:54] davecheney: agreed [01:09] thumper: email sent [01:09] davecheney: oh... kay... [01:09] as i said in there, i'm working on a less horrible fix [01:10] thumper: you'll see, don't panic [01:13] * thumper isn't panicing [01:31] thumper: standup [01:31] thumper: wallyworld axw alexisb waigani are al waiting for you to grace us with your presence [02:03] thumper: did you say you don't know what the minimum version of lxc we require is [02:03] ? [02:03] axw: correct... [02:03] ok [02:03] you should ask hallyn [02:04] mk [02:04] really, it is the one that makes lxc-start move the container into the STARTING state [02:21] wallyworld: so the user has to resolve provisioning errors, the provisioner won't just retry? [02:21] like resolving charm errors? [02:21] axw: for now yes. the provisioner can't retry without extra logic, since if it is a rate limit issue, it would just compound the problem [02:22] we need to add extra logic to be smart about how and when auto retry happens [02:22] yeah okay. it'd need exponential backoff and so on [02:22] thanks [02:22] np [02:22] so first cut is just to provide the machinery and a manual trigger [02:33] wallyworld: does it not make sense to have the machine watcher get notified when the machine's status changes? [02:34] wallyworld: which would make the MachinesWithTransientErrors API redundant I think [02:35] axw: sorta. but i wanted to keep the implementations separate because 1. we are close to release and i didn't want to dick with that logic too much, 2. we may well want to have separate queues / processing requirements for handling machines with errors [02:36] ok [02:36] it's all a a bit rushed sadly [02:37] well it's not being used, so if it turns out it can be done without the API then we can just not use it. but at least it'll be there if we do need it [02:37] perhaps we can revert to using the normal machine watcher, but i didn't want to risk breaking existing stuff what was expecting certain semantics for that watcher [02:37] yep fair call [02:37] axw: it's used in the branch i'm about to propose [02:37] wallyworld: ok [02:38] by the provisioner task [02:38] it's internal anyway [02:38] but we can easily change that in future if we want to do it differently as it's an internal api [02:38] yrp [02:38] the joys of trying to get stuff into a release [02:39] maybe i'm being overly cautious, but i do like having this stuff go in as an orthogonal implementation to what's there [02:39] to reduce risk [02:44] thumper: https://codereview.appspot.com/80300043/ [02:44] fix proposed for the gccgo link order bug [02:44] now to make some spaghetti and see if I can roll a custom version of gccgo-go with that fix [02:45] davecheney: ack [02:47] davecheney: seems reasonable [02:47] davecheney: except for this 'a.p.' malarky [02:47] :-) [03:01] thumper: good question about the race condition - I honestly don't know, but I think you're right and it could race. I'll use locals for those vars updated in the goroutine [03:15] axw: ok [03:16] wallyworld: got a minute for a hangout? [03:16] sure [03:18] thumper: you setting one up or me? [03:19] I'll do it [03:19] typical of me [03:19] ask a question... [03:19] then "oh, look, shiney" [03:19] https://plus.google.com/hangouts/_/72cpjm79q3a5ke7h17c6in5qas?hl=en [04:03] axw: what are you looking at now? [04:03] axw: because two things leap to mind... [04:03] thumper: was going to do some more azure stuff, but I can shelve that again [04:04] axw: the more urgent one is "warn about empty .jenv file" as opposed to being too dumb [04:04] axw: second one is 'juju unset-env' [04:04] thumper: warn about empty file and then act as if it didn't exist? [04:05] umm... I'm not sure [04:05] I'm tempted to just tell them it is there, and stop [04:05] the whole reason it is created empty is to grab the namespace [04:05] it is only weird edge cases where it would be empty and there [04:06] yeah true. I guess we'll need a utility to help people clean stuff up when we actually do have the config hosted [04:06] * thumper nods [04:06] I'll just have it exit for now with a message pointing at the location [04:07] I think that is reasonable [04:08] it should (fingers crossed) no longer happen [04:16] thumper: was just looking at IRC logs and looks like mgz is still working on it. I'll do unset-env for now, if you have no objections [04:16] ok [04:20] axw: I've emailed fwereade and jam about it [04:20] thumper: okey dokey [04:31] derp [04:31] thumper: I mucked up the Infof in cmd/juju/common.go [04:31] it says bootstrap fails even when it doesn't [04:31] hmm... [04:31] * axw fixes [04:32] oops [04:34] davecheney, ping? [04:41] wallyworld: quick review please? https://codereview.appspot.com/80330043 [04:41] sure [04:41] wallyworld: thanks for updating the client thing - will take another look in a bit [04:41] np, i was glad to get a 2nd opinion [04:46] mattyw: pong [04:57] wallyworld: reviewed again - sorry, missed something before [04:57] sure, looking [06:38] wallyworld: I can't think of a way to solve the attribute error thing without a tonne of extra work, so if you don't mind I'll land this and we can iterate later [06:38] at least then the API will be in === vladk|offline is now known as vladk === vladk is now known as vladk|offline [06:58] jam: the go.crypto/ssh ProxyCommand doesn't shell out to "ssh", it shells out to "juju ssh" [06:58] so effectively calling back into the same go.crypto/ssh code [07:17] axw: ok [07:49] mornin' all [09:03] morning rogpeppe [09:06] jam: morning [09:07] jam: are you around for a quick chat? [09:31] morning all [09:31] voidspace: hiya === vladk|offline is now known as vladk [10:03] * fwereade is just about to do some reviews, but would like it if someone would take a quick look at https://codereview.appspot.com/80540043/ [10:05] good morning eeryone [10:09] fwereade: would you be available for a quick chat? [10:12] fwereade: looks good altough you might want to have opinion from a more expert dev [10:22] fwereade: what problems did the tests find? [10:22] rogpeppe, sure [10:23] rogpeppe, just a couple of things like Checking an error and not returning [10:23] rogpeppe, so a subsequent look at a FileInfo would panic [10:23] rogpeppe, I had it in one place and not another [10:23] rogpeppe, dirs weren't checking perms or something? [10:24] https://plus.google.com/hangouts/_/canonical.com/juju-core?authuser=1 [10:33] if someone has a moment, I would appreciate this to be looked at https://codereview.appspot.com/80560043/patch/1/10002 [10:36] perrito666, looking [10:38] perrito666, reviewed [10:38] dimitern: thank yu [10:46] perrito666, i forgot to mention that this won't work on windows, but that's ok I guess [10:47] perrito666, or perhaps it will.. not sure (testing needed) [10:47] dimitern: utils ssh lacks an implementation for windows? (I guess that would depend on cygwin or smth on that area) [10:48] perrito666, no, but on windows because we're not using openssh's scp no extra arguments can be passed to the command (not sure if that's relevant) [10:48] call is a bit bad for me this morning... [11:03] dimitern: fixed [11:05] perrito666, cheers [11:05] fwereade, rogpeppe, mgz, https://codereview.appspot.com/76910044/ is ready to land - want a last look? [11:08] dimitern: sure [11:17] anyone else wants to take a look at? https://codereview.appspot.com/80560043/patch/1/10002 it is fairly short [11:21] dimitern: extra tx if you paste the links here [11:22] :) different computer [11:28] perrito666, sure: https://wiki.edubuntu.org/SecurityTeam/TestingMAAS http://marcoceppi.com/2012/05/juju-maas-virtualbox/ [11:29] dimitern: sory, I use another machine to be able to hold hangouts that long [11:29] perrito666, I should start following your lead in that soon :) [11:30] I use a mac, which has hardware accel for that, yet it gets kind of hot, I guess I should get an android tablet or similar [11:40] marcoceppi, nice article btw :) ^^ [11:41] dimitern: thanks, it's too bad virtualbox and maas don't play together nicely [11:41] marcoceppi, yeah, I tried myself twice before deploying maas + vbox (unsuccessful) [11:42] dimitern: it's much better to use maas + qemu/kvm [11:43] marcoceppi, I tried that yesterday and almost got it working - pxe boot working (with both cluster controller vm and a blank node using single NIC bridged to virbr0), but after commissioning the outbound networking on the new node is not working [11:43] brb [11:44] dimitern: how is your maas setup? does it own dchp, etc? [11:45] marcoceppi, yeah, it has 2 nic (lo and eth0 - the bridged one; also managed in maas for dhcp + dns) [11:45] marcoceppi, had to set the default domain config from local to "", because it seems avahi boot is not working [11:45] dimitern: is ip_forwarding setup properly with the masquard rules, etc? [11:46] dimitern: it took me a long time to get maas-master networking configured correctly [11:46] marcoceppi, and I also set the iptables rules according to the TestingMAAS article [11:46] * marcoceppi reads article [11:46] dimitern: there are some scripts that might help you setup the networking, actually [11:46] marcoceppi, ip_fwd on cluster vm or on the host (the latter is on) [11:46] latter [11:47] marcoceppi, so pxe boot works, but as the new node starts apt-getting stuff it fails to resolve archive.ubuntu.com urls [11:47] the fact that this is saying cobbler convinces me it's out of date. I don't think maas uses cobbler anymore [11:48] yeah, it doesn't [11:48] but I used the alternative (installing maas-dhcp and configuring it through the web ui) [11:48] dimitern: well, there should be a squid-deb-proxy running on maas master, you should be able to set the apt-mirror url in maas to piont to the maas master's internal network ip [11:48] but that won't solve needed to resolve addresses outside of the machine [11:49] marcoceppi, so just install squid-deb-proxy on cluster controller, add a new archive from the web ui? [11:49] dimitern: also, is it not connecting to the outside or is it a dns lookup issue? [11:50] marcoceppi, not sure if it's just dns resolving issue or outbound networking in general, need to look into it [11:50] dimitern: if you can get to a node in maas, you should be able to ping a network address, and if you can, you'll just need to set up the proper DNS forwarding route for the maas dns server [11:50] marcoceppi, the frustrating thing is that the guide wasn't specific on how to setup the new nodes' networking when creating the vm [11:51] dimitern: just DHCP should be sufficent [11:51] marcoceppi, but following yours, i decided all vms have nic bridged to the virbr0 [11:51] dimitern: I had to set the external DNS server for my nodes to resolve archive.ubuntu [11:51] marcoceppi, so the node appears and stays in commissioning (in some cases even ready), but the fqdn is something like ";; cannot resolve xyz ;;" [11:52] wwitzel3, with libvirt? [11:52] well as an alternative, you can use virtulbox in nat mode and then do a reverse ssh tunnel from within the maas machine [11:52] the master that is [11:53] dimitern: yes, in MAAS/settings [11:53] dimitern: there is an Upstream DNS [11:53] wwitzel3, ah, ok, I'll try that [11:53] dimitern: also, I set the default domain to some fake internal domain, like my.maas and added the maas controller to my resolv.conf as the nameserver for that domain [11:54] wwitzel3, cheers [11:56] rogpeppe: want to rejoin the hangout? [11:56] rogpeppe: it's empty [11:56] well, except for me... [11:57] voidspace: will do [11:59] dimitern: could you ack the chanes, just to be sure? https://codereview.appspot.com/80560043/patch/20001/30002 [11:59] rogpeppe: https://codereview.appspot.com/72500043/ [11:59] * perrito666 has the button hanging over the submit [12:00] natefinch: would you be able to join us in the hangout? [12:00] natefinch: (https://plus.google.com/hangouts/_/canonical.com/juju-core?v=1395698400&authuser=1) [12:04] morning all [12:09] perrito666, looking [12:10] perrito666, reviewed, just one minor thing [12:10] wwitzel3: was it you that asked me how to make gofmt go trough vim? [12:10] fwereade, rogpeppe, mgz, poke about https://codereview.appspot.com/76910044/ [12:11] perrito666: well I can make it automatically run, I just can't get it to highlight the lines with errors [12:11] wwitzel3: ah I got it to jut fix the file :p [12:11] which is faster [12:11] perrito666: well mine does that too as long as there aren't any synxtax errors [12:12] perrito666: but if there are syntax errors, it just outputs "go fmt returned an error" and doesn't tell me what line, so I have to drop to the command line and run go fmt to see where the error actually is. [12:13] wwitzel3: i think I can add that to fmt.vim :) ill try on the weekend [12:14] dimitern: going over it now [12:20] dimitern: lgtm [12:20] mgz, ta! [12:21] dimitern: fixed, merging then :) [12:21] dimitern: thank you very much [12:21] perrito666, go for it [12:26] mm, apparently has a conflict with my other branch, how should I fix that? [12:29] pull trunk, merge trunk in to the branch you're trying to land, [12:30] ack [12:30] perrito666, switch to trunk, pull it in, then switch to your other branch and do "bzr merge :parent" [12:30] resolve the conflicts, commit, push, then tell the bot to try again [12:30] :) [12:30] rogpeppe, do you have a minute? [12:31] rogpeppe, I'm wondering why this fails http://play.golang.org/p/VPC3P2nGYQ ? [12:31] dimitern: gimme 5 mins [12:31] rogpeppe, sure [12:32] perrito666, ah, one thing you might need - after merging, resolving, committing and pushing, you'll probably need to add a comment on the MP to self-approve the changes [12:38] after merging and committing I du another propose right? [12:41] just the landing step [12:41] no really need to lbox [12:41] flip the approved on the mp is all you really need [12:41] ok, I just bzr push then go to the mp in launchpad and hit approved [12:41] yup [12:42] ok, makes sense [12:46] jam: fwereade: i added support for multiple machines also [12:57] http://en.wikipedia.org/wiki/Toast_sandwich [13:03] rogpeppe, I'm really banging my head here on this one [13:03] dimitern: ok, looking [13:04] dimitern: what does jc.DeepEquals print? [13:04] rogpeppe, mismatch at ["services"]["network-service"]: validity mismatch; obtained map[interface {}]interface {}{"charm":"cs:quantal/dummy-1", "exposed":false, "networks":map[interface {}]interface {}{"disabled":[]interface {}{"net3", "net4"}, "enabled":[]interface {}{"net1", "net2"}}}; expected [13:04] rogpeppe, expected is actually not nil, but IsValid == false [13:05] rogpeppe, here's the full output with some added logging: http://paste.ubuntu.com/7156594/ [13:05] dimitern: the second one has "networks-service" not "network-service" [13:05] dimitern: http://play.golang.org/p/BWtpWAaiEf [13:06] rogpeppe, ah! [13:06] rogpeppe, tyvm [13:06] dimitern: np. i've been that same situation *so* many times! [13:06] dimitern: it was worse without jc.DeepEquals... [13:07] rogpeppe, indeed :) at least it tell you at which point it fails [13:07] darn plurals :) [13:17] fwereade, mgz, updated https://codereview.appspot.com/76400049/ - now we show networks for services, not machines (ignore the branch name, don't know how to rename it) [13:17] fwereade, I'd especially like your comments due to your not lgtm [13:24] wwitzel3, rogpeppe: back in action, going to do some cleanup of the proposal branch [13:25] voidspace: http://paste.ubuntu.com/7156661/ [13:26] dimitern: are you free for a quick chat? [13:26] natefinch: ok, cool. i'm afraid i haven't got any further with my review [13:26] rogpeppe, sure [13:26] rogpeppe: no problem [13:26] rogpeppe, mumble? [13:27] dimitern: https://plus.google.com/hangouts/_/canonical.com/juju-core?v=1395698400&authuser=1 ? [13:27] dimitern: (i've forgotten how to use mumble...) [13:27] rogpeppe, ok [13:28] sinzui: hey, with the last landing you should be able to run the test for https://bugs.launchpad.net/juju-core/+bug/1291022 [13:28] <_mup_> Bug #1291022: Cannot restore a state-server on ec2 and openstack [13:29] perrito666, fab, the test is being run with every revision. I will look for a pass with r2488 [13:33] sinzui: plz keep me posted, I ran it successfully by integrating my branches, but ymmv [13:33] HA! I love this [13:33] https://github.com/bradfitz/iter [13:39] bodie_: I just saw that the other day, pretty great [13:41] natefinch: can I call? [13:41] wwitzel3: sure [14:17] perrito666: I ended up using the syntastic bundle for vim and it does proper line highlighting or errors [14:17] of [14:17] :) that is very good news [14:18] perrito666: yeah, it hilights the offending line and provideds the message [14:20] looking for a quick leg up understanding /state/api/apiclient.go struct State [14:20] if anyone has a sec, pm? [14:21] thanks in advance :) [14:30] sigh, *of course* mongodb charm doesn't implement any nagios relations [14:31] alexisb: ping [14:31] oops, that was meant for #juju === hatch__ is now known as hatch [14:35] bodie_, api.State is the client-facing api facade, i.e. it emulates state.State, but through the api (it contains the underlying api connection as well) [14:36] yeah, I saw that -- wasn't aware that it is the client frontend for State.state though. (useful!) [14:37] I was going on the assumption that the API is an http endpoint but I'm thinking that is wrong. correct? [14:42] is JUJU_HOME ~user or ~user/.juju ? [14:42] it's the latter [14:46] bodie_, strictly speaking it's a front-end of state.State as seen through the API only [14:47] by API you mean the actual programmatic API, right? Not the HTTP API [14:48] I was under the impression the gui communicates with the program via HTTP API but now I'm recalling hearing that it uses websockets [14:48] bodie_, the API endpoint itself is either https-based or websocket-based (the former is only used for uploading charms and fetching stuff from charms, the latter exposes the rest of the api) [14:48] I see [14:48] so if I want to add verbs to the API, I don't need to define them in the HTTPS frontend [14:49] bodie_, if by verbs you mean api methods that can be used by clients (cli, gui, deployer, etc.), then you need to add them to api.Client as methods which call their apiserver.Client counterparts [14:50] fwereade, rogpeppe, mgz, juju deploy --networks/--no-networks support, PTAL https://codereview.appspot.com/80600044/ [14:51] OK, I think that makes sense :) we had talked about doing it via hooks but I'm not positive that's the right way [14:51] or at least, it gives me something to go on [14:51] thanks for the help [14:51] bodie_, what do you have in mind? === vladk is now known as vladk|offline [14:52] bodie_, things hooks can do are very limited compared to the api [14:52] service actions such as mysql snapshot [14:53] there are a few of us who got roped in from the outside, trying to put it together over the next few weeks [14:53] bodie_, what will that do? please, expand a bit. [14:53] I think it's either an experiment to see how frustrating it is for newbies to dive into the deep end unguided, or we're on reality tv [14:53] but here I am haha [14:54] :) [14:54] basically, a few Action words added to the command interface [14:54] juju do, juju queue, juju cancel, and a couple of others [14:54] the flow is like [14:55] the api code can be daunting at first, but it's really straightforward once you grok the way things pass through client->api->websocket->apiserver->state [14:55] yeah, I'm kind of briding the first bit of that right now [14:55] s/briding/bridging/ [14:55] ah, so juju can be added either in cmd/juju or as plugins named "juju-" [14:56] the flow is like -- juju do [14:56] sounds like you could use a plugin that calls juju run internally [14:56] yeah, I think I get how to add a command now, I just want to make sure I understand what needs to happen once it gets triggered [14:56] hm [14:57] well juju run gets called by the command line already right? regardless of argument [14:57] so we would just add a file to /cmd/juju/ [14:57] juju run allows you to run scripts remotely on a unit or all machines inside the same context (env vars etc.) that a hook runs [14:57] okay [14:57] yeah, that's what I've been digging through over the last two days [14:58] either in cmd/juju/mycmd.go (for example) or as an executable juju-mycmd in $PATH - both will give you the ability to do "juju mycmd" [14:59] right... and then I'll have to --upload-tools to test it (right?) [14:59] no [14:59] ok [14:59] so let's say I want to have arguments to that command trigger stuff on the remote [14:59] e.g. mysql dump [15:00] the "run" on the node gets triggered by uh (checks notes0 [15:00] ) [15:00] --upload-tools is needed in general to have the latest jujud binaries in the cloud; for client-side stuff (on your machine) you just need to rebuild cmd/juju or whatever other plugin/script you're using [15:00] apiClient in /cmd/juju/run.go Run function [15:00] triggers execution on the remote via rpc [15:01] or -- triggers the remote function rather [15:01] which is in /cmd/juju [15:01] the simplest way to do that is to first check if you can simulate what you need via juju run (or other commands) [15:01] implementing RunCommand [15:01] * bodie_ reads [15:01] if it is sufficient, you can put them in a script and use that [15:02] juju run is quite versatile - check its help [15:02] ok [15:05] bodie_, juju ssh and juju scp are also useful to run stuff remotely or copy files across local/remote machines (but these do not give you hook contexts like run) [15:06] OK, so maybe what we're actually doing is trying to make a Run alias that knows how to do certain things based on what's in the respective charm [15:06] rogpeppe, got a moment? [15:06] mattyw: sure [15:06] mattyw: hangout? [15:06] rogpeppe, we can do - irc is probably fine for this [15:07] unless you want to hear my voice ;) [15:07] mattyw: that's cool too, though i ache for the melodious timbre of your voice, obviously [15:08] rogpeppe, you're only human [15:08] how poetic [15:08] rogpeppe, this review from fwereade https://codereview.appspot.com/75600044/diff/20001/state/user.go#newcode32 [15:08] admin user being allowed an empty password [15:09] bodie_, i wouldn't call it "alias" (in CLI terms only actual commands can have aliases), but I got you - it's just a script that calls one or more times juju run and can take arguments [15:09] I'm sure there was a reason - but can't remember what/ if there was - do you have any wisdom to impart? [15:10] mattyw: i'm not really sure why we consider "admin" special in any way actually [15:10] mattyw: apart from for mongo access, which isn't relevant to this [15:11] mattyw: i'd prefer to avoid any name special-casing [15:11] rogpeppe, I'm starting to wonder if I snuck that code in so I didn't have to change loads of tests [15:11] mattyw: ha ha [15:12] rogpeppe, I'll try removing it and see what happens [15:12] rogpeppe, if you don't think it should be there either then I'll just have to bullet the bite [15:12] mattyw: looks like there are in fact a fair number of tests that call AddUser with an empty password [15:12] I looking for reviews on https://codereview.appspot.com/76400049/ and https://codereview.appspot.com/80600044/ please [15:12] mattyw: why don't we just allow the empty password? [15:13] rogpeppe, I've changed a number of tests in that cl aready - you mean for all users? [15:13] mattyw: yeah [15:14] rogpeppe, I suppose so that makes sense [15:14] mattyw: given that a user can change a password to "password" or "a", i don't really see what preventing a blank password gives us [15:14] rogpeppe, very true [15:15] rogpeppe, fwereade I'm going to change the mp to allow the empty password for all users for now then if that's ok? [15:19] rogpeppe, mattyw: that sounds fair to me [15:20] rogpeppe, mattyw: state is probably not the place for that sort of policy enforcement anyway [15:20] fwereade: +1 [15:22] fwereade, rogpeppe where would be? (out of interest) [15:22] mattyw: if we wanted to, probably in the API [15:22] mattyw, rogpeppe: API layer IMO [15:23] nice when we concur :) [15:23] fwereade, got some time to review my CLs? [15:41] dimitern, first one LGTM [15:41] fwereade, \o/ [15:46] is there a way I can get a list of the charms loaded into the juju env? (uploaded and not necessarily deployed) [15:46] maybe poking a db or something? [15:47] rogpeppe: why is go test -c so dang slow? [15:47] hatch, you could check the storage [15:47] natefinch: it's quicker if you do go test -i first [15:48] dimitern cool thanks [15:48] dimitern, couple of thoughts on the second, let me know when you've read it [15:49] fwereade, cheers [15:50] rogpeppe: that doesn't seem to make a noticeable difference. It's just, I can compile in .2 seconds, but test -c takes 1.2-1.5 seconds for the same package [15:51] rogpeppe: the reason I care is that I want to be able to find compile errors in tests when I make changes to code, and not have to actually run the tests to check. I can recursively call go test -c in all subdirectories, but it takes forever [15:51] fwereade, right now we can't verify if the provider supports networks, but we can do it later if we have a way to introspect the provider capabilities reliably === vladk|offline is now known as vladk [17:55] alexisb: here [17:55] err [17:55] sorry [17:55] allenap: here [17:55] mgz: Can I borrow your brain? roaksoax (or one of his team) is seeing http://paste.ubuntu.com/7153235/ with Juju 1.5 but not 1.4. I can see what’s going on in MAAS, but I can’t figure out what has happened to get to this point. [17:56] mgz: It’s like Juju is trying to start a node that it doesn’t own. [17:56] Or that isn’t allocated. [17:57] maas 1.5 [17:57] withjuju 1.6 [17:57] roaksoax: Is this after an upgrade from precise to trusty? [17:57] Oops, of course. [17:57] allenap: looking :0 [17:57] mgz: Thanks! [17:57] allenap: this is maas in trusty [17:58] rogpeppe: are you around? [17:59] perrito666: yup [17:59] rogpeppe: I have an issue and mgz said youmight have an answer http://pastebin.ubuntu.com/7158113/ [17:59] I want to do an instance of the struct as marked in the pastebin [17:59] but GetField returns value, err [18:00] do you know any elegant way to do it? [18:00] perrito666: i might do. what the type that massNet.GetField returns? [18:00] allenap: so, is this from nodes/ acquire, or later? [18:01] string, err [18:01] we pretty much just ask maas to give us suitable things, regardless of juju version [18:01] roaksoax: ^ [18:02] rogpeppe: my python insst [18:03] *instinct is to have a map of struct params to values, iterate over and use setattr, but reflection in go is much more heavyweight [18:04] allenap: I take it maas isn't logging the actual api call that was the trigger for the eventual perm error? it's a bit hard to see from the traceback, but I assume acquire, as that's when it'd start [18:06] mgz: Request logs should show the URL, but not the body. Logging in MAAS is not quite joined-up enough to do this easily. [18:06] roaksoax: can you pull out the url of the request from those logs too? [18:08] mgz: i don't think we can [18:08] perrito666: here's one of 3 possibilities: http://play.golang.org/p/fYV5AdaU_P [18:08] roaksoax: Are you able to reproduce this and do further diagnosis? [18:08] mgz: so something I had seen long ago is for example, there are 10 nodes available in MAAS< then when you bootstrap, you have 9 available to use. SO if you add a extra server into MAAS, to have 10 available, juju would only know about 9 free and not 10 [18:08] perrito666: well 3 that i could think of straight off [18:08] could this be somehow related [18:09] rogpeppe: that's neat [18:09] roaksoax: Woah, that’s weird! [18:09] rogpeppe: thank you [18:10] mgz, roaksoax: I’m really sorry, but I have to go. I’ll be back in <2 hours, once the kids are in bed. [18:10] rharper: ^^ [18:10] rharper: mgz is requesting some type of logs for the failures you've seen [18:11] sure [18:11] so, we saw this in the juju status output: http://paste.ubuntu.com/7153156/ [18:11] looking on maas server logs, we saw this: http://paste.ubuntu.com/7153235/ [18:12] but the fundamental question is, can we bootstrap in parallel against a maas env; we have 4 jenkins instances, each one will bootstrap potentially in parallel, but never consume more hardware that we have available [18:13] perrito666, mgz: here's another, slightly more straightforward approach: http://paste.ubuntu.com/7158165/ [18:13] (but somewhat less performant too) [18:15] rharper: the short answer is yes, with a current version of maas, provided you're using a seperate user account for each [18:15] in older versions, parallel use was borked [18:16] mgz: so, we're on precise -- the clients use juju-core from precise, so 1.16 -- [18:16] we have separpate maas user for each jenkins slave [18:16] so no one is using the same maas user [18:16] perrito666, mgz: here's a third possibility: http://paste.ubuntu.com/7158198/ [18:17] rharper: then this should certainly work. so you also have four juju users with the different maas user creds? [18:17] mgz: yep [18:18] and the maas is the one from the precise cloud archive? [18:18] FWIW, in practice it runs fine -- I posted above a strange juju status output where we din't have a dns value for one of the nodes; and that 403 error [18:18] roaksoax: what's the maas? 1.5 on trusty IIRC [18:19] yup [18:19] maas 1.5 on trusty [18:20] mgz: ^^ [18:23] roaksoax: I can't see anything juju specific, though having an older client work when a newer one doesn't my be a helpful clue === vladk is now known as vladk|offline [18:24] mgz: any idea what that gomaasapi 403 would be about? or the permission stack trace from maas ? is it even related to parallel stuff at all ? [18:25] * rogpeppe has to run [18:25] natefinch, wwitzel3: perhaps you could push your branch so that i can take a look in the morning [18:26] g'night all [18:26] rogpeppe: sure [18:26] rogpeppe: night [18:28] rharper: so, the 403 is because of the perm error [18:28] why the perm error... is mysterious, it could be some scrambling from the accounts getting mixed somehow, but it could be all kinds of other things as well [18:29] hey I am suddenly getting error: flag provided but not defined: --instance-id when running bootstrap, I am using trunk and boostraping in amazon [18:29] yeah, there was a claim of an edit -- but I can't figure out why anything would be issuing an edit [18:29] it's basically a juju-deployer run of an openstack install [18:29] anyone bumped into thi? [18:29] natefinch: can you hear me? [18:29] nope [18:29] mgz: and only this one of the 7 machines that deployed and came up fine [18:29] for this juju env [19:15] right, g'night all [19:15] EOD [20:08] morning [20:08] fwereade: around? [20:08] thumper, heyhey, been a while [20:08] morning thumper [20:08] sure has [20:08] o/ wwitzel3 [20:08] fwereade: got a few minutes for a hangout? [20:08] o/ natefinch [20:08] thumper, ofc [20:09] fwereade: https://plus.google.com/hangouts/_/7ecpisoldu4ri6vlrus3kt5tu8?hl=en [20:09] morning thumper [20:18] natefinch: you back? [20:18] wwitzel3: not really. It's an online errand ;) [20:18] natefinch: ahh ok :) [20:28] wwitzel3: you have set up vmaas, yes? [20:28] wwitzel3: can you look at https://bugs.launchpad.net/juju-core/+bug/1297899 ?\ [20:28] <_mup_> Bug #1297899: Libvirt machines deployed with MAAS and Precise is only able to successfully boot once [20:34] thumper: test's running on ppc!! [20:34] \o/ [20:35] thumper: nice! [20:35] davecheney: so https://bugs.launchpad.net/juju-core/+bug/1289067 fixed? [20:35] <_mup_> Bug #1289067: arm64 multiple definition of `launchpad.net_juju_core_cmd._.0 [20:35] thumper: i fixed it upstream yesterday [20:35] make sure you have gccgo-go~1.2.1 [20:35] released 8 hours ago [20:40] thumper: natefinch http://paste.ubuntu.com/7158930/ [20:40] actual ppc test results [20:40] sinzui: http://paste.ubuntu.com/7158930/ [20:40] ^ real honest to god test results on ppc [20:40] just filtering out all the [LOG] lines now [20:40] davecheney: sweeeet [20:41] davecheney: https://github.com/natefinch/nolog [20:41] \o/ [20:42] thumper: a lot of those failures are simple slice ordering issues [20:42] davecheney, I just updated wolfe-02. Just the one lib because eco are working on the machine. [20:42] probably from stuff that is stored in a map [20:42] then we are returning the map keys [20:43] I expect great things in the next CI run [20:43] sinzui: sure, you just need gccgo-go 1.2.1 [20:43] yep, that is what I got [20:43] thumper: what is the 'official' fix to that class of test failures ? [20:43] should I sort the results before comparing [20:43] davecheney: jc.SameContents [20:43] or is there a gocheck Checker for that ? [20:43] natefinch: thanks, that was the tool I was thinking of [20:45] davecheney: welcome [20:46] davecheney, thumper. CI is already running (well walking is a better verb) the arm64 unit tests. We are hours from seeing the result if the instance doesn't die of exhaustion: http://ec2-54-84-137-170.compute-1.amazonaws.com:8080/job/walk-unit-tests-arm64-trusty/51/console [20:46] sinzui: cool [20:47] ha [20:47] sinzui: is that qemu, or the fast model simulator ? [20:48] davecheney, I think qemu. It is very slow. [20:48] I haven't been able to keep an instance up for more than 18 hours too [20:49] wwitzel3: So, I don't know that I need to get back into the hangout for the last 12 minutes of the day. Have you made any progress? [20:49] wwitzel3: not sure when your EOD is today [20:59] sinzui: the fast model is an oxymoron [21:00] sinzui: thumper natefinch http://paste.ubuntu.com/7158982/ [21:00] cleaner results [21:00] so there are a bunch of simple problems like using the wrong checker [21:01] there is something in one package that causes a panic on ppc64 [21:01] repo_test.go:237: c.Assert(err, gc.ErrorMatches, expect) [21:01] ... error string = "invalid character '<' looking for beginning of value" [21:01] ... regex string = "Cannot access the charm store. Are you connected to the internet. Error details:.*" [21:01] we have a test that looks like it's not expecting to run behind a firewall [21:02] lots of tool errors [21:03] EOD for me [21:03] Started service at: https://127.0.0.1:33899 [21:03] Set tools-metadata-url="https://127.0.0.1:33899/v1/1/juju-dist-test/tools" [21:03] Set image-metadata-url="https://127.0.0.1:33899/v1/1/juju-dist-test" [21:03] caused by: Resource at https://127.0.0.1:33899/v1/1/juju-dist-test/tools/streams/v1/index.sjson not found [21:03] caused by: request (https://127.0.0.1:33899/v1/1/juju-dist-test/tools/streams/v1/index.sjson) returned unexpected status: 404; error info: 404 Not Found [21:03] The resource could not be found. [21:03] looks like tool error again [21:04] the serious error is in RPC [21:04] which is quiet serious 'cos that is what powers the api [21:08] ok, a binch of errors because i didn't run godeps recently [21:08] sinzui: what are your 1.17.7 plans? [21:08] sinzui: do you want [21:08] a. individual bugs for these test faliures [21:08] b. one bigass bug for it all [21:09] c. no bugs, just fixes [21:10] wwitzel3: are you still around? [21:14] davecheney, I actually like individual bugs. Each is something valuable to say we fixed [21:14] sinzui: right ok [21:14] i might do it a bit arse backwards and raise a bug when I raise a mp [21:14] i don't think it is super valuable to do a pass of all the things that re not passing [21:14] we have more than enough bugs [21:15] davecheney, I have done the same when my head is working on this fix [21:15] and they will possibly be dupes [21:15] and finally, we have the juju CI for the overall status which is reported to our betters [21:15] davecheney, We can dedupe. We always look better when we can say, we fixed more than we planned [21:16] sinzui: ok, i'll use my judgement [21:16] (everyone backs slowly away) [21:17] thumper I really like the state of Juju. I will release 1.17.7 tomorrow. I will pick a rev that CI likes. Well I will caution that canonistack is sick and I might make those tests optional [21:17] sinzui: what's up with canonistack? [21:17] ah, abentley has already made them optional [21:18] thumper, The cloud is suffering very poor performance. Tests that took 5 minutes last month now take 30. I can complete tests in azure fater [21:18] faster [21:18] omg [21:18] that is terrible [21:19] canonistack, now slower than azure [21:19] not something I want on a bumper sticker [21:19] haha [21:19] thumper, The slowness may also be exacerbating issues were juju status returns an error. Aaron is adding a retry to the rules. Retrying status for 10 minutes is faster than starting the test over for 30 minutes [21:19] canonistack, it's got electrolites [21:21] canonistack, shut her down, she's sucking mud! [22:52] thumper: ffs, lachie has left some school books at home, i need to go and take them to the school, bbiab [23:23] niemeyer: you around ? [23:23] davecheney: yo [23:23] niemeyer: http://paste.ubuntu.com/7159729/ [23:23] davecheney: Hmm [23:23] davecheney: What happened to the entries? [23:24] this is on ppc64 [23:24] built under gccgo [23:24] the syscall pacakge is different there [23:24] davecheney: Slightly awkward in this case, I suppose [23:24] niemeyer: is there a build tag for the compiler ? [23:24] davecheney: This is an ioctl for the terminal device [23:24] // +build gc ? [23:25] the constants will be the same, it is linux after all [23:25] davecheney: It's not clear why gccgo and ppc aren't happy about it [23:25] davecheney: Right [23:25] davecheney: Also, FWIW, this is stolen from the stock go.crypto ssh [23:26] niemeyer: yeah, that also doesn't work on ppc :* [23:26] :( [23:26] davecheney: But it's a pretty minimal part of it [23:26] niemeyer: if I did up a patch that use a build tag on the compiler [23:26] davecheney: This is just grabbing the username and disabling echo for the password [23:26] to either import the constants frmo syscall if the compiler is gc [23:26] or define them locally if it's gccgo ? [23:27] davecheney: I suggest just defining them locally all the time.. if that makes ppc happy, we can just put a sad-face comment above it, and ask somebody to fix it for us eventually [23:27] niemeyer: SGTM [23:28] this is in the _linux.go file [23:28] and those constants do not change [23:28] niemeyer: will propose a patch real soon [23:28] Thanks! [23:39] niemeyer: https://code.launchpad.net/~dave-cheney/goetveld/003-fix-syscall-constants-for-gccgo/+merge/212961 [23:39] did this on my local machine [23:39] as lbox isn't working on ppc [23:39] testing on there now [23:41] ubuntu@winton-02:~/src/launchpad.net/lbox$ lbox [23:41] Usage: lbox [options] [23:41] yup, now it works [23:50] davecheney: Have you tried an interaction that requires the terminal stuff? [23:54] niemeyer: yup, works fine [23:54] axw https://bugs.launchpad.net/juju-core/+bug/1298115 [23:54] <_mup_> Bug #1298115: utils/ssh: test failures [23:58] niemeyer: thanks! [23:59] davecheney: No problem, and thanks for the fx [23:59] fx [23:59] fix [23:59] Thinkpad's are not the same anymore