[00:00] davecheney: I'm new as I know, so I'm asking to understand [00:00] s/i/you [00:00] sure [00:00] basically the rule is [00:00] if a function/method returns an error [00:01] you generally cannot make any assetions abvout the state of any other values it returns or the instance itself [00:01] *generally* [00:01] there are exceptions [00:01] so, there is no point in cleaning up the internal state of that instance as it is broke [00:02] there is value in cleaning up the files we poo'd on disk [00:02] because that is something we can do to make it possible to run the test again later [00:02] ah okay [00:13] waigani: this ticket, https://bugs.launchpad.net/juju-core/+bug/1299969 [00:13] <_mup_> Bug #1299969: launchpad.net/juju-core/provider/manual: ssh tests are not properly isolated

[00:14] is a blocker on getting the common and manual provider tests working [00:14] do you think you could add it to your todo list ? [00:15] davecheney: sure. I actaully hit that problem and thought I was being stupid because I did not have keys on the vm [00:16] waigani: right, but the solution is not to add keys to the vm [00:17] that will just push the problem off onto someone elses' plate [00:17] davecheney: yeah I get that now :) [00:17] the goal is we want to run these tests during the build on the lp build servers [00:17] which are scrupiolously clean [00:17] right [00:17] so any screwing with the environment beforehand will have to be expunged [00:28] davecheney: provider/local is now passing :) proposing my branch now [00:32] sinzui: nite, question, am I supposed/allowed to clos the bug and also, could you open the new one with the new bug info? (such as the exit of jenkins) [00:32] waigani: sweet [00:48] perrito666, I will do that [00:50] thumper: waigani wallyworld http://paste.ubuntu.com/7219526/ [00:50] i [00:50] i'd like to log a bug about this [00:51] perrito666, The bug is fix committed. It cannot be closed until the milestone is released/a package is distributed to the users. [00:51] i see far to many of these errors in the log when working with the local provider [00:51] i think its more than a harmless warning [00:51] * sinzui reports the new bug [00:51] davecheney: which bit exactly? [00:51] sinzui: tx [00:51] yeah, doesn't seem right [00:52] thumper: 2014-04-08 00:35:07 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "=DEBUG" to "=WARNING;unit=DEBUG" [00:52] ^ what generates this line [00:52] davecheney: the logging config worker [00:53] davecheney: that is the default logging level [00:53] thumper: http://paste.ubuntu.com/7219530/ [00:53] that is the logging level specified in state [00:53] thumper: this is the repro [00:53] ok [00:53] /dev/sda 9.9G 7.6G 1.9G 81% / [00:53] 10gb vm isn't really enough to debug on ... [00:55] * davecheney waves to axw [00:55] welcome to daylights saving [00:55] davecheney: do you have a question for me? [00:55] davecheney: howdy [00:56] no daylight savings over here [00:56] thumper: how can I set the logging back to debug everywhere ? [00:57] i'm trying to get more details on issue 1303787 [00:57] davecheney: "juju bootstrap --debug" or "juju bootstrap --logging-config='juju=debug'" [00:57] davecheney: or... if you want more [00:57] thumper: cna't change it after the fact ? [00:57] davecheney: yes [00:58] config-set ? [00:58] juju set-env logging-config=juju=debug [00:58] oh ffs unity [00:58] thumper: thans [00:58] thanks [01:00] ok, environmnet setup, now apparently we wait 7 hours ... [01:01] davecheney: I have been talking about that issue [01:01] davecheney: how much memory does your VM have? [01:01] 8 gb [01:01] ok... perhaps that'll be fine [01:01] 24 was bad, 16 was good [01:01] hmm, that is odd [01:01] I'm wondering if there is a fundamental issue elsewhere [01:01] as in, not us [01:01] more likely [01:02] but yes, keep an eye on it [01:02] i don't doubt there are bugs in the compiler [01:02] that is why i need machine-0.log [01:02] all machines via rsyslog is eating the stack trace [01:02] thumper: could you make sure if thre is any conversation about that issue that you cc me [01:02] i've been trying to get everyone to communicate via the isse [01:02] for the original machine that was a problem has now got 16gig of ram and a new kernel [01:02] davecheney: it was all voice conversation [01:03] but am currently managing 3 indepedentant email threads by three separate groups working the problem [01:03] I was chatting with hazmat about it [01:03] huh... [01:03] that isn't fun [01:05] thumper: i've done more funner tings [01:05] things [01:05] :) [01:05] i know this issue is critical [01:05] but saying it's critical isn't enough [01:05] we need details [01:09] fairly trivial review anyone? deleting 1.16 compat code: https://codereview.appspot.com/84520046/ [01:17] davecheney: re http://pastebin.ubuntu.com/7219597/ [01:17] (re https://bugs.launchpad.net/juju-core/+bug/1303787) [01:17] <_mup_> Bug #1303787: hook failures - nil pointer dereference

[01:17] i'm guessing that doesn't help much :P [01:18] tvansteenburgh: have you encountered issues since the vm was resized? [01:18] tvansteenburgh: the command you want is [01:18] dpkg -l | grep gccgo [01:18] sorry [01:19] dpkg -l | grep gccgo [01:20] thumper: honestly i don't know for sure as i haven't been on the machine today, but from convos on irc i've inferred that there are still probs [01:20] tvansteenburgh: can you get me the machine-0.log file from the machine [01:20] tvansteenburgh: or [01:20] better [01:20] can you do ssh-import-id dave-cheney [01:20] and I can do it myself [01:20] davecheney: dpkg -l | grep gccgo prints nothing [01:21] davecheney: imported your key [01:21] tvansteenburgh: is this on a ppc64el machine ? [01:21] yes [01:21] tvansteenburgh: this is very worrying [01:21] what you are teling me doesn't add up [01:22] the example fro the weekend showed that juju was compiled locally [01:22] but you're saying that there is no compiler on this machine [01:22] don't shoot the newbie :) [01:22] tvansteenburgh: soryr mate [01:22] i didn't install juju on this machine so i'm not sure what to tell you [01:22] but keep in mind that juju was upgraded since the original bug report [01:23] tvansteenburgh: ok [01:23] so it's possible that it was from source before, and now pre-compiled binary [01:23] i'll check out the machine myself [01:23] tvansteenburgh: are you going to be doing hte demo, or hazmat ? [01:23] hazmat [01:24] roger [01:24] davecheney: no, hazmat ;-P [01:24] * thumper snickers at his joke [01:24] * hazmat hides [01:26] tvansteenburgh: what is the internal ip of wolfe-01 [01:26] i have the wrong config [01:28] 10.245.66.193 [01:28] ta [01:30] tvansteenburgh: thanks, i'm grabbing the log files now [01:34] thumper, local provider detects btrfs and just uses it if its at /var/lib/lxc ? [01:34] yup [01:34] thumper, awesome [01:35] * hazmat gives it a whirl [01:35] hazmat: will still take time to create the template first time around [01:37] wallyworld: got 10 minutes? I need help solving a tools issue [01:37] sure [01:37] * thumper starts a hangout [01:38] https://plus.google.com/hangouts/_/7acpj3v9ejfcghmeigfq41jfc4?hl=en [01:40] thumper, understood, i updated the lxc cache image directly as well [01:42] well that's cute.. juju set-env random-string="abc" just works.. [01:42] and is retrievable via juju get-env [01:48] http://paste.ubuntu.com/7219672/ [01:48] grep -c panic [01:48] eeerk [01:51] sinzui: where is the packaging branch for juju-core ? [01:54] tvansteenburgh: i'd take a punt and say wolfe-01 got nuked and recreated [01:55] that is why there is no compiler [01:55] nuke; recreate; [sudo] apt-get install juju-core [01:57] davecheney: yeah i think that /is/ what happened [01:57] tvansteenburgh: that makes sense tehn [01:57] then [02:03] jamespage: where is the packaging branch for juju-core ? === arosales_ is now known as arosales [02:38] thumper: wallyworld: axw: my internet connection is crapping out on m [02:38] e [02:39] yay for NZ inyternet [02:44] works for me [03:24] thumper, davecheney so good news.. i've run through the demo about a dozen times.. zero panics [03:24] \o/ [03:24] thumper, and using btrfs :-) [03:24] on power? [03:24] thumper, yup [03:24] awesome [03:39] how do I find out which package provides an executable again? [03:45] wallyworld https://codereview.appspot.com/85220043 [03:45] looking [03:45] now this works [03:45] for some value of works [03:45] it tries to create a trusty container and fails [03:46] for other reasons [03:46] I'm asking on #ubuntu-server for answers to those reasons [04:17] ubuntu@winton-02:~$ juju status [04:17] ERROR failed verification of local provider prerequisites: exec: "mongod": executable file not found in $PATH [04:17] MongoDB server must be installed to enable the local provider: [04:17] sudo apt-get install mongodb-server [04:17] still telling me to instal lmongodb-server [04:17] did that branch land ? [04:19] not yet [04:20] ok [05:13] https://codereview.appspot.com/85150044 [05:13] trivial fix [05:19] davecheney: LGTM, all the other files in that directory use the bson: syntax [05:21] jam: that is why i feel pretty confident that the fix is ok [05:21] davecheney: me too [05:38] ├─jujud─┬─lxc-create───lxc-ubuntu-clou───ubuntu-cloudimg───wget [05:39] │ └─11*[{jujud}] [05:39] what the balls is it downloading [05:39] and is there a way to speed it up === vladk|offline is now known as vladk [05:53] davecheney: if you are using LXC, it has to download the Ubuntu image for the LXC instance [05:53] it's hung up trying to download some small the releases text file from cdimages [05:55] ubuntu@ip-10-251-8-60:~$ juju deploy ubuntu --debug [05:55] 2014-04-08 05:54:20 INFO juju.cmd supercommand.go:296 running juju-1.19.0-trusty-amd64 [gccgo] [05:55] 2014-04-08 05:54:20 DEBUG juju api.go:189 trying cached API connection settings [05:55] 2014-04-08 05:54:20 INFO juju api.go:259 connecting to API addresses: [localhost:17070 10.0.3.1:17070] [05:55] 2014-04-08 05:54:20 INFO juju.state.api apiclient.go:194 dialing "wss://localhost:17070/" [05:55] 2014-04-08 05:54:20 INFO juju.state.api apiclient.go:141 connection established to "wss://localhost:17070/" [05:55] ^ hun [05:55] hung [05:55] hang on [05:55] why is it talking to localhost ? [05:55] the api server is running on 10.3.0.1 [05:57] axw: wasn't there a bug logged about this over the weekend ? [05:57] * axw looks up [05:58] um [05:58] davecheney: localhost is the "public address" for the local provider's machine-0 [05:58] 10.0.3.1 is the internal [05:59] davecheney: is there a problem with that? === vladk is now known as vladk|offline [06:02] axw: not sure [06:02] there will be when proxies are involved [06:02] we always say [06:02] no_proxy="10.0.3.1" [06:02] not no_proxy="localhost" [06:03] no_proxy=localhost only makes sense if you're outside the environment [06:03] i.e. using the CLI [06:03] I suppose we could manage the CLI's environment specifically for the local provider [06:04] axw: ok [06:04] nm [06:04] probably not the problem at the moment [06:24] 2014-04-08 06:07:56 ERROR juju.provisioner provisioner_task.go:438 cannot start instance for machine "3": error executing "lxc-start": command get_cgroup failed to receive response [06:24] any ideas ? [06:25] I wonder if it is related to https://bugs.launchpad.net/bugs/1304167 [06:25] <_mup_> Bug #1304167: syntax error, trusty beta-2 cloud image === vladk|offline is now known as vladk [06:32] davecheney: that sounds like nested-lxc issues, but I'm guessing you're not nesting? [06:33] nup [06:33] just reboote [06:33] thre was an apparmor snafu installing the lxc package [06:33] maybe that was the cause [06:36] http://paste.ubuntu.com/7220297/ [06:36] nope, lxc is broken [06:47] mornin' all [06:47] i was just looking at this change: https://github.com/juju/testing/pull/3/files [06:47] is there any way i can see the whole file diff on github? [06:48] or do i have to pull it down to do that? [07:13] jamespage, jam: so, I think I've tracked down that relations-on-upgrade bug -- but I think (1) it only hits relations without other members, and (2) it's always existed [07:13] jamespage, were there any other units in that peer relation? [07:14] jamespage, jam: that's bug 1303697 ^^ [07:14] <_mup_> Bug #1303697: peer relation disappears during upgrade of juju

[07:32] fwereade: I'm creating an instance.Placement structure for placement directives, but I'd like to leave SSH out of it because it only makes sense in add-machine (for now at least) [07:32] is that okay with you? [07:33] fwereade: http://paste.ubuntu.com/7220424/ -- currently adding a new method to state to add a machine with one of these [07:35] axw, I feel it's mildly suboptimal, because I think it *should* be reasonable to deploy --to ssh:blah -- but I can see how it would pull in a lot more work, so, fair enough [07:36] axw, about the kv pairs in environment placement... upsides and downsides [07:36] I'm pretty sure we can add it later if necessary [07:36] axw, and, hmm, I don't quite love the separation between ContainerPlacement/EnvironmentPlacement [07:37] fwereade: the idea was that a container placement can encapsulate any other, the others being terminals [07:37] axw, (1) I think the conception of lxc/kvm as pseudo-providers is quite nice; (2) I think that --to us-east-1a is nice, but that's not a kv pair [07:37] fwereade: so you could do lxc:maas:name=abc [07:38] axw, ha, that's an interesting thought [07:39] axw, I'm a little bit -1 on it -- I think it's ok that the domain of the lxc/kvm providers is restricted to existing machines [07:40] axw, I can see nice things about it but it also makes my spidey sense tingle a little [07:40] okay, I'll collapse them for now [07:40] fwereade: so just some kind of scope/provider and a value, and leave it to the environ to parse k=v if it wants to? [07:41] axw, yeah, I think that's the right separation of concerns [07:41] ok [07:41] axw, and it does leave the providers free to have us-east-1a for now, and later to allow zone=us-east-1a,somethingelse=somethingelse [07:42] yep, fair enough [07:42] axw, fwiw I think scope is a nice term for it [07:44] okey dokey well that simplifies the changes needed to state then [07:44] * fwereade is getting really quite grumpy with the half-assed attempts to make the uniter work on multiple goroutines [07:54] axw, I don't suppose you know why we call u.proxy.SetEnvironmentValues() in updatePackageProxy, do you? [07:54] * axw looks [07:55] fwereade: I think so the hooks run with the proxy env vars set? [07:55] axw, we explicitly pass the proxy settings in when we create a hook context [07:56] sorry, not really sure tbh. better off asking thumper - he implemented that I think [07:56] axw, np, cheers [08:04] fwereade: is this better? http://paste.ubuntu.com/7220551/ [08:05] machine-id is a special case, because there isn't really a scope/provider/whatever [08:05] for empty scope, "juju add-machine" will fill it in with the current environment [08:06] so "juju add-machine --to maas-node-123" will implicitly have the current env's name as the placement scope [08:08] axw, I'm wondering whether it would be excessively evil to consider "0" or whatever to be in some special scope itself [08:09] axw, "--to 0" == --to "machine:0" as it were [08:10] axw, just trying to figure out the consequences of a collision between DWIM vs law of least surprise [08:10] axw, or juju:0, or something [08:10] fwereade: I suppose we could just have an instance.MachineScope constant, = "-" or some character that is not valid in an environment name [08:11] fwereade, hey [08:11] dimitern, heyhey [08:11] fwereade, https://codereview.appspot.com/85060043/diff/1/provider/maas/environ_whitebox_test.go#newcode537 re that comment [08:11] dimitern, oh yes [08:11] fwereade, what exported method did you mean? [08:11] dimitern, StartInstance [08:12] fwereade, but to do that we need to land a few things in gomaasapi [08:12] dimitern, expand please? sorry I'm not up on all the context there [08:13] fwereade, well, to test the new features we need them to be supported by the test server of gomaasapi [08:14] fwereade, that's lshw machine details and list_connected_macs for a network [08:15] fwereade, i wanted to save some time and land these changes (after testing them live on my local maas ofc) so we can have MVP, and later polish it [08:15] dimitern, ok, I see -- as long as there's a card for it, then :) [08:15] fwereade, sure, will add one now [08:16] axw, yeah, something like that maybe [08:17] fwereade: I'll go with that and see how it turns out. [08:18] axw, cheers [08:38] dimitern: morning [08:38] vladk, hey morning [08:40] dimitern: I am working on linking GetNetworkInfo and extractInterfaces [08:43] dimiternЖ I do not want both of us do the same changes [08:47] anyone: if I do juju destroy-environment, does this remove /var/log/juju-$ENV ? [08:48] davecheney: i'm not sure it does. there might have been a discussion about this recently. jam? fwereade? [08:49] davecheney: thumper explicitly designed it so that destroy-env does *not* delete /var/log/juju* (though it does delete /var/lib). [08:49] var/log will be deleted on another bootstrap [08:49] axw: i am thinking that "mongodb" in utils/apt.go, cloudArchivePackages, should be "mongodb-server". does that seem right to you? [08:49] vladk, ah, sorry, I misunderstood you perhaps [08:49] jam: hang on [08:49] you just said two things [08:50] and they conflicted [08:50] vladk, I have the branch that returns []NetworkInfo proposed already [08:50] you said /var/log/juju-ubuntu-local would not be deleted [08:50] then you said it would [08:50] any else know about what package we should be installing for mongodb? [08:50] davecheney: /var/log is not deleted during destroy, but /var/lib is [08:50] vladk, but we do need some changes in gomaasapi [08:50] davecheney: during bootstrap /var/log is cleaned up [08:50] jam: /var/log/juju-ubuntu-local is truncated [08:50] s/any else/anyone else/ [08:50] so the all machines log of my environment is only 5 mins long [08:51] vladk, to extend the test server, so it supports /version/ (capabilities), /network//?op=details (for the XML lshw data) [08:51] rogpeppe: sorry, just a moment [08:51] dimitern: I see your last code review, do you have something behind that? [08:52] vladk, I have 2 more - the provisioner to take []NetworkInfo and add NICs/networks in state [08:52] rogpeppe: yes, I think you're right [08:52] vladk, and the last one is the cloudinit scripts, where I believe you did some work on and we can do it together? [08:52] axw: but i don't see mongodb-server mentioned in http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/cloud-tools_versions.html [08:53] jam: ok, thanks for confirming [08:53] arosales: you online ? [08:53] axw: but... it looks like that's what we're using in current bootstrap, so i guess it should work [08:53] rogpeppe: well, mongodb depends on mongodb-server. so we get what we need either way [08:53] axw: ah [08:54] axw: ok, trying that [08:54] axw: i've added mongodb-server to cloudArchivePackages [08:54] cool [08:57] dimitern: we need one more change to gomaasapi, to not panic in testing mode, what bootstrap a node without networks [08:59] dimitern: how can I see your code? [09:01] vladk, yeah, that's why we need to change gomaasapi [09:02] vladk, what code specifically? [09:02] fwereade, I replied to your comment https://codereview.appspot.com/85060043/ does it make sense? [09:04] dimitern, yeah, sgtm [09:04] fwereade, cool, submitting then [09:17] dimitern: I can't work on cloudinit script before you land your changes on NIC->VLAN mapping [09:19] vladk, it's approved and on its way to land now [09:24] morning all [09:27] axw: thanks for doing some testing on my branch. Don't need to apologize that it works ;) [09:30] hmm, it occurs to me that since we don't need the tools until we connect to the instance, we could start the instance first, then concurrently upload the tools. that would make bootstrap --upload-tools about twice as quick for me. [09:30] as i wait for yet another bootstrap [10:02] fwereade: where does the machiner get the addresses from now? [10:03] rogpeppe, net.InterfaceAddrs [10:03] rogpeppe, and it sets them all with NetworkUnknown [10:03] fwereade: as an alternative, perhaps the uniter should wait until its machine has an address before starting [10:03] rogpeppe, I'm not sure exactly what handwaving we do with the local provider to make those addresses show up usefully [10:04] rogpeppe, yeah, maybe that's the way -- ask repeatedly for your own address until you have one, then carry on [10:04] rogpeppe, and I think we can do that with existing apis [10:04] fwereade: yeah [10:05] axw: not yet... [10:05] fwereade: istm that if you're the uniter, you really don't want to start until other units can talk to you [10:05] natefinch: cat ~/.juju/local/cloud-init-output.log and see if there's anyhting interesting there [10:06] rogpeppe, yeah -- and that used to work, but we moved stuff around [10:06] rogpeppe, although, hmm [10:06] rogpeppe, unaddressable containers should be fine, this might render them less fine [10:06] * fwereade goes to dig and find out exactly how we collate addresses [10:06] fwereade: hmm, interesting [10:07] natefinch: also, you do have juju-mongodb installed right? [10:07] fwereade: i think we need to be able to explicitly mark a container as unaddress [10:07] able [10:07] rogpeppe, yeah [10:08] fwereade: some special form of address might work [10:08] rogpeppe, and add a bunch of consistency stuff so we don't accidentally put workloads that need to relate into them etc etc [10:08] fwereade: yeah [10:08] rogpeppe, well, they'll still have some sort of address on the machine network at least [10:08] axw: yeah, there's nothing interesting in the cloud init output.... last thing is just "opening environment local" [10:08] rogpeppe, that'll all come if/when we do explicit proxy charm support I think [10:09] axw: I have mongodb built with SSL in the correct spot, but it's not specifically juju-mongodb [10:09] natefinch: what are you having problems with? [10:10] natefinch: mk. is mongo even running? [10:10] axw: mongo runs, yeah [10:10] rogpeppe: bootstrapping local [10:11] axw: maybe I should just install juju-local [10:11] wouldn't hurt :) [10:11] :) [10:11] natefinch: do you have an http proxy set? just wondering if that breaks something... [10:11] otherwise I am out of ideas [10:11] no proxy [10:12] natefinch: try putting some more debug statements in the dial code [10:12] gotta make dinner, bbl === axw is now known as axw-afk [10:18] well, installing juju-local let me bootstrap, but I can't destroy the environment now. Sigh [10:19] wallyworld, I think we'll need to TOPIC this, but let me read further [10:19] sure [10:20] wallyworld, yeah, I think we need to make the fallback logic provider-specific too [10:21] wallyworld, consider env:arch=i386, service:instance-type=t1.micro [10:21] fwereade: i think it is, or? [10:21] all providers except ec2 and openstack just use the current WithFallbacks as they don't yet support instance-typ [10:22] ec2 and openstack have specific logic [10:22] wallyworld, ah! ok, I misread something, let me keep going [10:22] ok, no rush [10:22] wallyworld, we should probably move WithFallbacks off the Constraints type though [10:22] sure [10:23] that's easily done [10:23] wallyworld, and I think there's a subtlety wrt which values need to be masked when [10:23]