[04:13] <davecheney> thumper: State.StateServingInfo checks for a StateServingInfo doc
[04:13] <davecheney> or more specifically a doc that will marshal into that type
[04:13] <davecheney> then checks if doc.Port == 0 and takes that as a not found signal
[04:13] <davecheney> it does this because state.Open inserts an empty document into that collection ...
[04:14] <thumper> right ...
[04:15] <davecheney> thumper: can you think of a reason not to change this to
[04:15] <davecheney> not insert the doc if there is nothing to insert
[04:15] <davecheney> then use ErrNotFound as the sentital for the document not being found ?
[04:16]  * thumper looks at it now...
[04:18] <thumper> I'm guessing because it is always expected to be there (for some value of there)
[04:18] <thumper> makes it easy to set by just calling update
[04:18] <thumper> isn't there an Upsert?
[04:27] <thumper> davecheney: I think with a change to use upsert, I think the behaviour could be changed
[04:27] <davecheney> fuk this nosql bullshit
[04:29] <davecheney> ok, that'll have to stay as broken as it is today
[04:29] <davecheney> func (st *State) SetStateServingInfo(info StateServingInfo) error {
[04:29] <davecheney>         if info.StatePort == 0 || info.APIPort == 0 ||
[04:29] <davecheney>                 info.Cert == "" || info.PrivateKey == "" {
[04:29] <davecheney> ^ we only consider these four fields to be important
[04:29] <davecheney> any others can be nil and are not consdiered 'empty'
[04:32] <thumper> :)
[04:32] <thumper> davecheney: perhaps #juju-dev a better channel... users probably don't care :-)
[04:37] <davecheney> whoops
[04:38] <davecheney> i'm in there wrong channel
[10:22] <pitti> hello all
[10:25] <pitti> did anyone succeed with using juju-local in utopic? it was my third attempt this morning, and now it fails to build containers
[10:25] <pitti> I filed bug 1363832 this morning
[10:25] <mup> Bug #1363832: [utopic] fails to build containers <amd64> <apport-bug> <utopic> <juju-core (Ubuntu):New> <https://launchpad.net/bugs/1363832>
[10:25] <pitti> I wonder if it's generally broken, or something on my system is special
[12:05] <bloodearnest> has any one else observed juju invoking config-changed at random on a deployed environment?
[12:06] <bloodearnest> We only noticed because it fails as a configured auth token has expired
[14:03] <pitti> did anyone succeed with using juju-local in utopic? it was my third attempt this morning, and now it fails to build containers
[14:03] <pitti> I filed bug 1363832 this morning
[14:03] <mup> Bug #1363832: [utopic] fails to build containers <amd64> <apport-bug> <utopic> <juju-core (Ubuntu):New> <https://launchpad.net/bugs/1363832>
[14:03] <pitti> I wonder if it's generally broken, or something on my system is special
[14:05] <html> juju? i cant even figure out how to get it to work-ish
[14:44] <hazmat> bloodearnest, at random?.. it gets called in a  few places
[14:44] <hazmat> bloodearnest, it called before start, and after upgrade, and whenever the config changes, and if the machine is rebooted.
[14:44] <bloodearnest> hazmat: as in, after a long period of no juju activity, with no human stimulus
[14:45] <hazmat> fwereade, ^
[14:45] <bloodearnest> hazmat: havn't been able to reliably reproduce
[14:45] <bloodearnest> hazmat: but I've seen it on local provider, and jjo's seen it a few times in prodstack
[14:46] <hazmat> pitti, i haven't seen that.. i filed a separate issue about the default containers defaulting to trusty (or more specifically latest lts) instead of precise.
[14:47] <hazmat> pitti, are there any other containers on the machine from juju? its hard to tell if its having issues starting the template container or the actual container
[14:48] <pitti> hazmat: no, before there were only a few completely unrelated containers
[14:48] <pitti> hazmat: it tried to create a juju-trusty-lxc-template but failed
[14:48] <pitti> I lxc-destroy'ed that after each failed attempt
[14:50] <hazmat> i should really polish up my juju lxc scripts.. simplifies the interaction much more (nothing on the host).
[14:50] <hazmat> pitti, could you manually start the template container? ie lxc-start -d -n juju-trusty-lxc-template
[14:51] <pitti> hazmat: well no, there is nothing in that container
[14:51] <pitti> hazmat: it created a /var/log/juju/ (empty), and that's it
[14:51] <hazmat> oh..
[14:52] <hazmat> pitti, that's strange.. the cloud image template might have some cache issues as well, if it also fails if you manually do lxc-create -t ubuntu-cloud -- -r trusty .. i'd try to clearing out the appropriate series dir in /var/cache/lxc
[14:53] <pitti> hazmat: there is no other juju related contianer
[14:53] <hazmat> pitti, and re env settings.. you have juju get-env | grep lxc shows -> lxc-clone-aufs: false ?
[14:53] <pitti> hazmat: so if juju deploy is supposed to merely clone an existing container, that would explain why it fails
[14:54] <pitti> hazmat: but then bootstrap ought to have failed
[14:54] <hazmat> pitti, yup.. it setups a template container, and clones for the actual unit
[14:54] <pitti> (I pointed that out in the bug report -- bootstrap does *not* create any container)
[14:54] <hazmat> if your on btrfs.. it uses a snapshot automatically if /var/lib/lxc is btrfs
[14:54] <pitti> no, plain ext4
[14:54] <hazmat> pitti, yeah.. bootstrap does nothing
[14:54] <pitti> so where does the template container come from?
[14:54] <pitti> if bootstrap doesn't create it, and deploy expects it?
[14:54] <hazmat> pitti, its all async the first time a machine is requested needed
[14:54] <hazmat> pitti, deploy will do it inline
[14:55] <hazmat> this is a long standing issue imo cause it has to download.. a full image and its doing it without giving feedback
[14:55] <hazmat> ie. at min we should be showing more info on status when its doing stuff in the background for the image dl
[14:55] <pitti> well, a "full image" is something like 60 MB or so?
[14:56] <pitti> at least that's the rough size of stgraber's pre-built LXC templates
[14:56] <hazmat> bootstrap can't quite do it, cause we don't know what series will need to be created
[14:56] <pitti> that should download in a minute; I waited > 10 mins
[14:56] <pitti> hazmat: ok, thanks for explaining the workflow
[14:56] <pitti> I suppose I'll set it up again and this time just let it sit there for really long
[14:56] <pitti> but it showed a lot of error messages and there was no active process, it seemed quite dead/failed to me
[14:56] <hazmat> pitti, no.. its bigger.. ~220mb
[14:57] <pitti> uh, wow
[14:57] <hazmat> pitti, its the image off cloud-images.ubuntu.com its downloading .. not the mystery images off linux-containers.org
[14:57] <pitti> ah, that; but even that just takes 3 mins or so for me
[14:58] <hazmat> pitti, you can see the image it downloads  @ /var/cache/lxc/cloud-$series per the ubuntu-cloud-template source
[14:58] <hazmat> pitti, the actual error message is the failure to start the container, so i'm wondering if you manually create a container does it work
[14:59] <hazmat> with the ubuntu-cloud lxc template
[14:59] <pitti> ok, /scratch/juju set up again, bootstrap is done
[14:59] <pitti> $ juju get-env | grep lxc
[14:59] <pitti> container: lxc
[14:59] <pitti> lxc-clone-aufs: false
[14:59] <pitti> hazmat: ^ FTR
[14:59] <hazmat> cool
[14:59] <pitti> hazmat: I just changed root-dir and default-release, nothing else
[15:00] <hazmat> if the manual creating an lxc with the cloud template doesn't work, that's the primary issue and i'd suggest wiping /var/cache/lxc/cloud* as a likely cause.
[15:00] <pitti> so I don't have anything juju-ish in /scratch/lxc/ which
[15:00] <pitti> just my unrelated other containers
[15:00] <hazmat> k
[15:00]  * pitti runs juju deploy juju-gui
[15:01] <pitti> /var/cache/lxc/cloud-trusty/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
[15:01] <pitti> that exists and doesn't change size (185 MB)
[15:01] <pitti> /var/cache/lxc/cloud-precise/ubuntu-12.04-server-cloudimg-amd64-root.tar.gz also exists and doesn't change size (238 MB)
[15:02] <fwereade> bloodearnest, we do call config-changed on somewhat slim pretexts, but not *at random*
[15:02] <pitti> hazmat: tail -f /scratch/juju/log/* doesn't show anything either
[15:03] <pitti> and ps aux no processses that are more recent than 2 hours ago
[15:03] <pitti> so something in the template container creation goes pear-shaped
[15:03] <hazmat> pitti, to sanity check can you manually create a container using the cloud template?
[15:04] <pitti> hazmat: well, I'd need a cloud template container first :) (that's the very bug)
[15:04] <pitti> how do I craete that?
[15:04] <pitti> from /var/cache/lxc/cloud-trusty/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
[15:04] <hazmat> sudo lxc-create -n myprecise -t ubuntu-cloud -- -r precise -S ~/.ssh/id_rsa.pub
[15:04] <hazmat> sudo lxc-start -d -n myprecise
[15:05] <pitti> hazmat: no, see above -- there *is* no ubuntu-cloud container -- creating that is the thing that fails
[15:05] <hazmat> pitti, lxc has different templates for creating a container
[15:05] <pitti> the only thing that was created in all this was the empty juju-trusty-lxc-template
[15:05] <hazmat> pitti, per contents of /usr/share/lxc/templates/
[15:05] <pitti> hazmat: ah sorry, ignore me
[15:05] <pitti> create -t, not clone
[15:06] <pitti> failed to get https://cloud-images.ubuntu.com/query/precise/server/daily-dl.current.txt
[15:06] <pitti> There is no download available for release=precise, stream=daily, arch=amd64
[15:07] <pitti> hazmat: that smells like something then
[15:07] <hazmat> that's wierd that link resolves
[15:08] <hazmat> and it should be defaulting to release stream not daily
[15:09] <bloodearnest> fwereade: at random == our perception :)
[15:10] <hazmat> oh.. i wonder if it goes to daily cause your on an unreleased.. smarts of some form
[15:11] <hazmat> pitti, it sounds like the bug may apply to the lxc templates then if the underlying lxc tools aren't working
[15:11] <bloodearnest> fwereade: other than after start/upgrade/reboot, or an actual honest-to-goodness config-change, are there any more scenarios that it might be?
[15:15] <pitti> hazmat: reaading /usr/share/lxc/templates/lxc-ubuntu-cloud it seems to default to "tryreleased" and falls back to "daily" if that fails
[15:16] <pitti> $ ubuntu-cloudimg-query trusty released amd64
[15:16] <pitti> failed to get https://cloud-images.ubuntu.com/query/trusty/server/released-dl.current.txt
[15:16] <pitti> hazmat: ^ that might explain it further
[15:16] <hazmat> yeah
[15:16] <pitti> so that fails, and it falls back to "daily"
[15:16] <hazmat> pitti, that link resolves though..
[15:17] <hazmat> pitti, that link works for me
[15:17] <hazmat> pitti, as does the cli
[15:17] <pitti> hazmat: yes, for me too (in the browser)
[15:18]  * pitti adds a cloud-utils task then
[15:18] <hazmat> cool
[15:19] <pitti> hazmat: does wget work for you?
[15:19] <pitti> hazmat: it fails because the certificate can't be checked
[15:19] <pitti> and it's calling wget without --no-check-certificate (quite rightly so)
[15:20] <fwereade> bloodearnest, I *think* those should be the only cases (plus agent restart, not quite a reboot necessarily)
[15:20] <fwereade> bloodearnest, is it possible something's bouncing the agent?
[15:20] <pitti> hazmat: ok, thanks muchly for your help! I'll go on prod our cloud folks
[15:22] <bloodearnest> fwereade: that sounds a likely candidate, will try to check next time it happens
[15:23] <pitti> hazmat: if wget works for you on that URL, it'd be interesting if you are running trusty or utopic
[15:24] <pitti> hazmat: wget does work for me in trusty, but apparently got stricter in utopic
[15:24] <bloodearnest> fwereade: what's the rationale behind running config-changed after reboot/restart?
[15:24] <fwereade> bloodearnest, heh
[15:25] <fwereade> bloodearnest, we don't track what version of the config you've seen
[15:25] <bloodearnest> ahh
[15:25] <fwereade> bloodearnest, I have no idea what inspired that approach
[15:25] <fwereade> bloodearnest, but fixing it has not been especially high on my list
[15:26] <hazmat> pitti, cert issues on trusty then?
[15:26] <hazmat> or perhaps tls negotiation
[15:26] <pitti> hazmat: or wget just simply didn't default to checking the cert for https:// on trusty yet
[15:26] <pitti> which is more likely
[15:26] <fwereade> bloodearnest, (and since we *do* run it on reboot I worry a bit that not doing so will subtly break some charms that have accidentally come to depend on it)
[15:26] <pitti> hazmat: I pinged smoser about it in #u-devel
[15:27] <bloodearnest> fwereade: ack, it was a bug in our charm that made us discover it, just wanted to know why it was happening
[15:27] <pitti> hazmat: I hacked ubuntu-cloudimg-query and will now re-try juju
[15:32]  * pitti also adds --no-check-certificate to the ubuntu-cloud LXC template and tries again
[15:35] <pitti> hazmat: ah, now I have a juju-trusty-lxc-template, a running martin-local-machine-1, and we are as far as agent-state-info: 'hook failed: "install"'
[15:35] <pitti> hazmat: that's a whole lot of progress :)
[15:36] <hazmat> pitti, awesome
[15:37] <pitti> apt-get install failed
[15:43] <hazmat> pitti, ugh.. dns issues in container?
[15:43] <hazmat> pitti, or are you getting checksum mismatch?
[15:43] <pitti> I wish it would contain apt's error messages
[15:44] <hazmat> pitti, lxc-attach -n your-container-name  && and apt-get install
[15:44] <pitti> hazmat: yep, that's my next step
[15:44] <hazmat> pitti, actually better alternative
[15:44] <hazmat> is juju debug-hooks unit-name
[15:44] <hazmat> and then juju resolved --retry
[15:44] <hazmat> + unit-name
[15:45] <pitti> aah
[15:45] <hazmat> pitti, if you like your containers in clouds.. with across host networking.. this is something fun i put together over the weekend. http://bazaar.launchpad.net/~hazmat/charms/trusty/rudder/trunk/view/head:/readme.txt
[15:45] <pitti> # cat /etc/apt/apt.conf.d/42-juju-proxy-settings
[15:45] <pitti> Acquire::http::Proxy "http://127.0.0.1:3142";
[15:46] <pitti> now, that wouldn't work, my dear juju
[15:46] <hazmat> wtf
[15:46] <hazmat> it should be pointing to the bridge address
[15:46] <hazmat> 10.0.3.1
[15:47] <hazmat> and now we're back to juju bugs ;-)
[15:48] <pitti> hazmat: thanks for pointing out teh debug stuff
[15:48] <hazmat> pitti, np
[15:49] <pitti> hazmat: filed and updated bug 1364069
[15:49] <mup> Bug #1364069: local provider must transform localhost in apt proxy address <amd64> <apport-bug> <utopic> <juju-core (Ubuntu):New> <https://launchpad.net/bugs/1364069>
[15:49] <pitti> resolved --retry still fails, but I need to run now; more tomorrow :)
[15:50] <hazmat> thats the one downside of using my own juju lxc scripts.. i don't get to experience the pain of local provider
[15:50] <hazmat> pitti, cheers