[00:16] <wallyworld> axw: ah, i think i understand what you might be saying - snap lxd daemon picks up the request from juju rather than deb lxd daemon and hence the wrong lxd get used. i thought setting LXD_DIR was supposed to cause the right lxd to get used (at least that's what nicholas thought).
[00:22] <menn0> wallyworld: I just realised that I hit the same thing with the conjure-up snap last week. it ships with its own version of juju and has LXD issues.
[00:22] <wallyworld> menn0: yeah. adam and nick know about it. not sure exactly what their solution is. my system is still somewhat screwed even after removing lxd snap
[00:23] <wallyworld> need to look at it again today
[00:23] <menn0> wallyworld: do you need to run sudo lxd init again?
[00:23] <wallyworld> that's my last resort, yeah
[01:25] <axw> wallyworld: PR to update azure regions: https://github.com/juju/juju/pull/6969. seeing as we need to get the public-clouds.yaml updated anyway, figured we may as well try and get this one in too?
[01:25] <wallyworld> sure
[01:45] <wallyworld> axw: sorry about delay. lgtm but there's a block on landing it due to current QA policy. there's a meeting tomorrow where that will be fixed
[01:46] <axw> wallyworld: thanks
[03:13] <menn0> wallyworld: were you running into this at bootstrap? 2017-02-13 03:05:33 ERROR cmd supercommand.go:458 new environ: Get https://10.0.8.1:8443/1.0: x509: certificate has expired or is not yet valid
[03:13] <menn0> anastasiamac: or is this what you saw? ^^
[03:13] <wallyworld> menn0: similar, yeah, can't recall exact wording
[03:14] <menn0> its happening for me r
[03:14] <menn0> too
[03:14] <menn0> axw: could ^^^ be related to the lxd cert caching change?
[03:14] <anastasiamac> menn0: i did not see this. wallyworld may have... just fixed my lxd to work over the weekend. my failures were related to lxc profile misconfiguration
[03:15] <wallyworld> menn0: i've had to totally purge lxd, manually remove neworks and ip links, lxd init again etc. still not there though - juju can't talk to container
[03:15] <axw> menn0: maybe. or the snap. are you using the snap?
[03:15] <menn0> no snaps involved
[03:16] <axw> menn0: you haven't installed the juju or lxd snaps?
[03:16] <menn0> not on this machine
[03:17] <axw> menn0: ok. seems most likely related to my changes then. it doesn't happen to me though. can you try and isolate it?
[03:17] <menn0> axw: sure. where's the cache file?
[03:18] <axw> menn0: juju will pull certs out of either ~/.config/lxc or ~/.local/share/juju/lxd
[03:19] <menn0> axw: I don't have ~/.local/share/juju/lxd
[03:19] <axw> menn0: either/or. if that one doesn't exist, it'll look in ~/.config/lxc
[03:19] <axw> (I don't have ~/.local/share/juju/lxd either)
[03:20] <menn0> axw: lxc is working fine by itself
[03:20] <menn0> axw: I just launched a container with "lxc launch"
[03:21] <axw> menn0: lxc will use the unix socket locally
[03:21] <axw> menn0: so HTTPS won't factor at all
[03:21] <menn0> axw: ok right
[03:31] <menn0> axw: moving the .config/lxc directory out of the way doesn't fix things
[03:31] <jam> thumper: menn0: coming by to say hi?
[03:32] <thumper> jam: yeah
[03:32] <menn0> jam: coming
[03:35] <axw> menn0: can you try adding some logging to finalizeLocalCertificateCredential in provider/lxd/credentials.go? we should be generating a new cert, uploading it to lxd, and then using that
[03:41] <axw> wallyworld: can you pastebin the output of "lxc config trust list" for me?
[03:43] <axw> wallyworld menn0: one possibility is that the certs in ~/.config/lxc are expired. mine expire in 2026...
[03:43] <menn0> axw: but I moved them out of the way and the symptoms stayed the same?
[03:43] <axw> menn0: ok, weird
[03:45] <axw> menn0: even that doesn't repro for me
[03:45] <axw> menn0: which version of ubuntu, lxd?
[03:47] <menn0> axw: well those certs *have* just expired
[03:47] <menn0>         Validity
[03:47] <menn0>             Not Before: Feb 10 02:21:23 2016 GMT
[03:47] <menn0>             Not After : Feb  9 02:21:23 2017 GMT
[03:47] <menn0> lxd 2.0.8, xenial
[03:49] <axw> menn0: did you add a lxd cert credential to credentials.yaml? by autoload-credentials perhaps?
[03:50] <menn0> axw: no lxd creds in credentials.yaml
[03:51] <axw> menn0: well if you moved the certs, I can't see how their expiry matters :/
[03:53] <menn0> axw: unless there's something inside the lxd daemon?
[03:54] <axw> menn0: maybe the server cert is the thing that has expired?
[03:55] <axw> menn0: the client credential changes could be a coincidence. seeing as the client certs you had have expired, possibly the server cert has too
[03:56] <menn0> axw: I think you're right. the problem happens with juju 2.0.3 too
[03:57] <axw> menn0: ok, cool
[03:57]  * axw wonders how to regen
[03:59] <axw> menn0: looks like if you delete /var/lib/lxd/server.{crt,key}, lxd will recreate them on startup
[04:00] <axw> wallyworld: ^^
[04:00] <menn0> axw: they've expired too. that's got to be the problem.
[04:00] <axw> menn0: terribly confusing coindidence :)
[04:10] <wallyworld> axw: yeah, i did that earlier, full lxd reinstall and init was also needed for me
[04:10] <wallyworld> due to wierd network issues
[12:29] <jam> wallyworld: ping if you're around
[12:29] <wallyworld> hey
[12:30] <perrito666> morning you two
[12:30] <wallyworld> evening here :-)
[12:33] <perrito666> wallyworld: any urgent bug that was left last night? otherwise ill just pick from the pile
[12:35] <wallyworld> perrito666: this one is important https://bugs.launchpad.net/juju/+bug/1623217
[12:35] <mup> Bug #1623217: juju bundles should be able to reference local resources <juju:Triaged> <https://launchpad.net/bugs/1623217>
[12:36] <wallyworld> not sure if it can be done in time
[12:36] <wallyworld> ie not sure of the scope of any change, haven't looked into it
[12:40] <wallyworld> jam: can i help with something?
[12:41] <jam> on bug #1577556 they just mentioned that they saw a 'statuses' doc that didn't have a txn-revno. I was a bit confused but it looks like statuseshistory is where we don't use TXNs but 'statuses' we *should* be using txns, right?
[12:41] <mup> Bug #1577556: unit failing to get unit-get private-address in the install hook <intermittent-failure> <network> <uosci> <OpenStack Charm Test Infra:Confirmed> <juju:Triaged>
 <juju-core 1.25:Triaged> <ubuntu-openstack-ci:Triaged> <mysql (Juju Charms Collection):Fix Released> <https://launchpad.net/bugs/1577556>
[12:43] <jam> wallyworld: sorry, wrong bug. actual is bug #1484105
[12:43] <mup> Bug #1484105: juju upgrade-charm returns ERROR state changing too quickly; try again soon <bug-squad> <canonical-is> <upgrade-charm> <upgrade-juju> <juju-core:Fix Released> <https://launchpad.net/bugs/1484105>
[12:44] <perrito666> wallyworld: tx
[12:44] <wallyworld> jam: yeah, status hisotry avoids txns. but the status doc itself in the status collection should use txn
[12:45] <perrito666> afaik the regular status collection does use txns
[12:46] <wallyworld> jam: all usages of statusDoc should be in the context of a txn.Op  slice
[12:47] <jam> perrito666: right, its supposed to, I was worried we had a case where sometimes we were and sometimes we weren't.
[12:47] <jam> but they are reliably (?) hitting a case where the statuses docs are missing the txn-revno, which is bad mojo for the TXN logic
[12:49] <perrito666> jam: odd, I wonder if someone confused status with status history
[12:49] <perrito666> jam: the latest status added was model status iirc
[12:49] <wallyworld> jam: i just did a (quick) code search and can't see anywhere where we are writing to the status collection not using txns
[12:50] <jam> perrito666: wallyworld: yeah, I grepped the code as well. It does mention an assert that "txn-revno == 0" which would indicate we tried to read it, got back 0 and then said "well, its gotta stay 0 for this updaet"
[12:51] <wallyworld> jam:  you talking about assert := bson.D{{"txn-revno", txnRevno}} in statusSetOps() I assume
[12:51] <jam> wallyworld: comment #16 on the bug
[12:52] <jam> there is an enttry in txn queue that says: "a": { "txn-revno": NumberLong(0) }
[12:52] <jam> my guess is that it read the doc, saw there was no txn-queue so got the 'zero' value and then put that back into the assert.
[12:52] <jam> So I don't think that is what *caused* the problem in the first place, just those txns all fall over because the data is bad.
[12:53] <jam> we don't really know what caused it to be bad in the beginning.
[12:56] <wallyworld> jam: juju uses the txn-revno assert in a set status function (this pattern isused elsewhere too, eg updating ports). but i think that relies on a create being done first to insert the original doc and set the txnrev-no field. and we do have a createStatusOp.  i can't see how we would attempt to set a  status value on a doc with a given doc id without having done a create first
[12:56] <wallyworld> and that create should insert the txnrev-no field
[12:57] <jam> wallyworld: (a) race? (b) if we're creating the doc with an upsert, we're still using  the txn logic, and the mgo/txns is the thing that keeps txn-revno correct
[12:57] <jam> we're just using an assert to say "if this doc is changed underneath us, ignore this change"
[12:58] <wallyworld> that matches my understanding
[12:58] <perrito666> jam: just a wild idea, do you think this could be caused by mgopurge?
[12:59] <wallyworld> we create the doc with an insert and a doc not exists assert
[12:59] <perrito666> wallyworld: https://bugs.launchpad.net/juju/+bug/1623217 seems like it will need some spec-ing and agreement from stakeholders
[12:59] <mup> Bug #1623217: juju bundles should be able to reference local resources <juju:Triaged> <https://launchpad.net/bugs/1623217>
[13:00] <wallyworld> probably, i just saw it and it looked important
[13:00] <wallyworld> should move to 2.2
[13:00] <perrito666> wallyworld: it has been like that since december, I dont think it is that urgent that we can skip proper procedure :)
[13:01] <perrito666> we should get someone started on that spec though
[13:01] <wallyworld> fair enough, i just skimmed it
[13:01] <perrito666> true, was just a comment not a rant
[13:01] <rick_h> perrito666: the goal there is just to allow local file paths like local charms
[13:01] <rick_h> perrito666: so hopefully a short spec
[13:02] <perrito666> rick_h: yes
[13:09] <jam> rick_h: perrito666: not sure how much of a spec it should have, vs you already have a syntax for 'use this local charm' ./path, we just support that for a resource blob as well
[13:11] <rick_h> jam: +1
[13:11] <jam> perrito666: short enough spec for you ? :)
[13:13] <perrito666> jam: sure it is, if annyone comes asking ill post that line :p
[13:38] <jam> perrito666: I didn't think that was actually going into 2.1, and doing it post rc feels a bit late, but if it isn't terribly hard, it is a nice quality-of-life for a bunch of people
[13:48] <perrito666> jam: I dont think ill be able to fit this into 2.1 most likely tonight ill move that to 2.2
[18:49] <Dmitrii-Sh> https://github.com/cloud-green/juju-relation-mongodb/pull/6 cmars, mattyw - JFYI: sent out a couple of patches for the mongodb interface
[19:19] <thumper> morning folks
[19:25]  * perrito666 touches tip of hat
[19:25] <mattyw> Dmitrii-Sh, hey there, thanks very much, will take a look
[19:45] <tasdomas> could there be a reason why juju 1.25.6 fails to bootstrap an aws model with authentication failure, while juju 2.0.0 works fine (with the same credentials)
[19:48] <thumper> tasdomas: no idea sorry
[20:27] <mup> Bug #1664359 opened: Authentication fails for juju 1.25.6 on aws <juju-core:New> <https://launchpad.net/bugs/1664359>
[23:09] <wallyworld> perrito666: thumper: IMO the proposed test fix for that status history test is too lenient - it throws away the notion of the order of the expected status messges
[23:10] <wallyworld> the fix should have been to be lenient with the expected count of messages, not the order
[23:15] <perrito666> wallyworld: I can very well send a follow up, since the test belo does test the order I thought it was not really that important
[23:17] <wallyworld> it does but the two tests are ostensibly similar - one uses a filter, the other doesn't. they should essentially test the same things the same way
[23:17] <wallyworld> just with nd without  filter
[23:19] <perrito666> yes, but there is a race there, that infrastructure was built by william but he did not notice that by inserting the records with identical dates the ordering is non deterministic
[23:20] <perrito666> two items with the same date can be returned in any given order, and that is ok
[23:20] <perrito666> because in reality it will never happen that two items have the same date to the nanosecond
[23:24] <wallyworld> ah, i see, that helps explain it a bit