[01:06] <axw> babbageclunk: you said in standup you wanted some advice on something?
[04:07] <blahdeblah> Anyone started looking at lp:1618212 ?  Just got it after upgrading from 2.0.3 to 2.1.2, which means that I can't upgrade any of the models, because I can't list them.
[04:14] <anastasiamac> blahdeblah: m looking at a bug that feels related
[04:14] <blahdeblah> Well, let me know if you need to look at anything in this env.
[04:14] <anastasiamac> blahdeblah: but this bug is about model destruction not listing...
[04:15] <anastasiamac> blahdeblah: logs r always helpful, mayb even a db dump (altho with 40+models sounds scary...)
[04:16] <blahdeblah> anastasiamac: I just took a backup, so you can have that if needed.  (This is the same env which led me to create lp:1680683.)
[04:16] <blahdeblah> The backup is 6 GB
[04:17] <anastasiamac> blahdeblah: m saying that the bug u referenced above is about destruction, not listing... altho frankly i am ware and kind of looking at both issues - the destruction of the model as well as lsiting of models...
[04:17] <anastasiamac> aware even
[04:18] <blahdeblah> anastasiamac: you must be reading a different bug to me; 1618212 is definitely about listing models
[04:19] <anastasiamac> blahdeblah: the title says "juju models fails during model destruction"
[04:19] <blahdeblah> anastasiamac: Every commenter on that bug says they're getting the error from "juju models".
[04:20] <blahdeblah> It might be related to destruction in some way under the covers, but where it shows up in the user experience is listing models.
[04:21] <blahdeblah> And I've just encountered it while listing models after an upgrade, no destruction involved.
[04:21] <anastasiamac> blahdeblah: i think the problems r related - we cannot cleanly destroy models under some circumstance, and list models gets confused
[04:21] <anastasiamac> blahdeblah: specifically for list models, I have: https://bugs.launchpad.net/juju/+bug/1674805 and https://bugs.launchpad.net/juju/+bug/1676279
[04:21] <mup> Bug #1674805: list-models fails after model is destroyed after model migration <ci> <intermittent-failure> <list-models> <model-migration> <juju:Triaged by anastasia-macmood> <https://launchpad.net/bugs/1674805>
[04:21] <mup> Bug #1676279: juju 2.1.1 controller not removing models correctly (list-models showing "info: settings not found") <destroy-model> <list-models> <juju:Triaged> <https://launchpad.net/bugs/1676279>
[04:22] <blahdeblah> maybe one of those will match my problem better
[04:22]  * blahdeblah looks
[04:22] <anastasiamac> blahdeblah: definitely worth a comment on any related bug that matches ur case ;)...
[04:23] <blahdeblah> 1676279 seems like a dup of 1618212 to me
[04:24] <blahdeblah> so does 1674805, TBH
[04:26] <blahdeblah> But neither matches better, so 1618212 will do for now
[04:26] <blahdeblah> I think, however, that destroy-models might be a distraction, since I got the error message without trying to destroy anything.
[04:27] <anastasiamac> blahdeblah: yeah... i think the underlying cause is the same but all are coming from slightly different angles.. yes, there is an issue in both destroy and list - slightly different issue...
[04:28] <anastasiamac> blahdeblah: hence, i said that the bug u referenced was for destroy... ppl just started putting list models comments into it
[04:28] <anastasiamac> blahdeblah: at the end of the day, both issues needs to b fixed so feel free to update whichever bugs feels closer to u :D
[04:29] <blahdeblah> Well, you know the code better than I
[04:29]  * anastasiamac laughs
[04:29] <anastasiamac> blahdeblah: u made my day, really
[04:30] <blahdeblah> There was no model destruction involved in my case, yet I ended up with the same error message.  So if you want a new bug that says "juju models fails after juju-upgrade", I'm happy to supply one.
[04:31] <blahdeblah> Number of anastasiamac PRs on juju > number of blahdeblah PRs on juju; hence, you know the code better than I. ;-)
[04:31] <blahdeblah> (where number of blahdeblah PRs on juju == 0)
[04:39] <anastasiamac> blahdeblah: my first, almost knee-jerk reaction is "no! i don't want another bug"
[04:39] <blahdeblah> I figured ;-)
[04:39] <anastasiamac> blahdeblah: but i think it'd b better if u did report it separately... besides i do want to see ur logs and db...
[04:40] <anastasiamac> blahdeblah: coz we have not heard (I *think*) of probelms when upgrading...
[04:41] <anastasiamac> blahdeblah: *after* upgrading
[04:41] <blahdeblah> So, is that a "yes, I do want another bug"?
[04:41] <anastasiamac> blahdeblah: and m pretty sure we test upgrades extensively...
[04:41] <blahdeblah> anastasiamac: Personally, I don't think it would achieve much
[04:42] <anastasiamac> blahdeblah: no, let's not for now :)
[04:42] <blahdeblah> I think what's happening is there were models which were marked as destroying when I started the upgrade.
[04:43] <anastasiamac> blahdeblah: yeah, that's the symptoms m seeing in all other similar failures
[04:43] <anastasiamac> lingering/ghosting models
[04:45] <blahdeblah> yeah - given axino's comments in 1676279, I think once you fix one, you'll fix them all
[04:46] <anastasiamac> blahdeblah: i wonder if his db surgery on the models marked as 'destroying' will get u unblocked.. at least while we r sorting the issue..
[04:47] <blahdeblah> I still disagree that this isn't a bug in listing; regardless of whether a previous destroy left some models lingering, "juju models" should keep working.
[04:47] <anastasiamac> blahdeblah: but since u r dealing with production as well, mayb not the most desirable way forward...
[04:47] <anastasiamac> blahdeblah: there is nothing to disagree - I agree that listing has a bug
[04:48] <blahdeblah> No, not the most desirable way forward, but if we can't list models, JAAS will probably be a bit dead in the water, so it may be necessary.
[05:33] <blahdeblah> urulama: ^ See discussion above; looks like there are at least 3 bugs about destroyed/ing models staying in DB
[05:33] <urulama> juju bugs for breakfast \o/
[05:33] <urulama> :)
[06:14] <blahdeblah> anastasiamac: How likely is it that those model listing/destroying bugs will be fixed in a 2.1.x update?
[06:15] <anastasiamac> blahdeblah: i thought u said that u r seeing in after the upgrade to 2.1.x already... 2.1.2 specifically?..
[06:16] <blahdeblah> Yep
[06:16] <blahdeblah> So how likely is it that we'll see a fix in 2.1.[3-5]?
[06:16] <anastasiamac> blahdeblah: we have not planning another 2.1.x... most likely in 2.2...? unless the sky falls?...
[06:17] <blahdeblah> I thought that might be your answer.  :-)
[06:17] <blahdeblah> thanks
[06:18] <anastasiamac> blahdeblah: just for reference, m still trying to figure out what goes wrong... i can reproduce destroy failure... i also *know* what's causing issues on list... but until i can nail it down, i don't know if it is easy to backport and release another 2.1.x...
[06:19] <blahdeblah> No worries - just trying to work out best course of action for this controller on which models can't be listed.
[06:19] <blahdeblah> urulama: ^ You mind checking whether us-east-1 is still functional from your perspective?
[06:19] <anastasiamac> blahdeblah: m guessing that u have tried the standard?
[06:20] <blahdeblah> anastasiamac: "the standard"?
[06:20] <blahdeblah> not as yet, but I will
[06:21] <urulama> blahdeblah: that was aws, right?
[06:21] <blahdeblah> yep
[06:22] <urulama> ok, looks fine, but i'll check with some big users as well
[10:25] <wpk> https://drive.google.com/open?id=0BwYF43OiGvHjWkFnLVJCVlFaRFk
[10:26] <wpk> ^^^ that's a dependency graph for juju/juju/api/
[11:04] <rogpeppe> anyone around to review this large but mechanical code-move branch: https://github.com/juju/juju/pull/7213
[18:27] <cmars> anyone available this friday afternoon for a quick review?
[18:28] <cmars> i've got https://github.com/juju/juju/pull/7214 which fixes critical bug https://bugs.launchpad.net/juju/+bug/1680883