[01:06] babbageclunk: you said in standup you wanted some advice on something? [04:07] Anyone started looking at lp:1618212 ? Just got it after upgrading from 2.0.3 to 2.1.2, which means that I can't upgrade any of the models, because I can't list them. [04:14] blahdeblah: m looking at a bug that feels related [04:14] Well, let me know if you need to look at anything in this env. [04:14] blahdeblah: but this bug is about model destruction not listing... [04:15] blahdeblah: logs r always helpful, mayb even a db dump (altho with 40+models sounds scary...) [04:16] anastasiamac: I just took a backup, so you can have that if needed. (This is the same env which led me to create lp:1680683.) [04:16] The backup is 6 GB [04:17] blahdeblah: m saying that the bug u referenced above is about destruction, not listing... altho frankly i am ware and kind of looking at both issues - the destruction of the model as well as lsiting of models... [04:17] aware even [04:18] anastasiamac: you must be reading a different bug to me; 1618212 is definitely about listing models [04:19] blahdeblah: the title says "juju models fails during model destruction" [04:19] anastasiamac: Every commenter on that bug says they're getting the error from "juju models". [04:20] It might be related to destruction in some way under the covers, but where it shows up in the user experience is listing models. [04:21] And I've just encountered it while listing models after an upgrade, no destruction involved. [04:21] blahdeblah: i think the problems r related - we cannot cleanly destroy models under some circumstance, and list models gets confused [04:21] blahdeblah: specifically for list models, I have: https://bugs.launchpad.net/juju/+bug/1674805 and https://bugs.launchpad.net/juju/+bug/1676279 [04:21] Bug #1674805: list-models fails after model is destroyed after model migration

[04:21] Bug #1676279: juju 2.1.1 controller not removing models correctly (list-models showing "info: settings not found")

[04:22] maybe one of those will match my problem better [04:22] * blahdeblah looks [04:22] blahdeblah: definitely worth a comment on any related bug that matches ur case ;)... [04:23] 1676279 seems like a dup of 1618212 to me [04:24] so does 1674805, TBH [04:26] But neither matches better, so 1618212 will do for now [04:26] I think, however, that destroy-models might be a distraction, since I got the error message without trying to destroy anything. [04:27] blahdeblah: yeah... i think the underlying cause is the same but all are coming from slightly different angles.. yes, there is an issue in both destroy and list - slightly different issue... [04:28] blahdeblah: hence, i said that the bug u referenced was for destroy... ppl just started putting list models comments into it [04:28] blahdeblah: at the end of the day, both issues needs to b fixed so feel free to update whichever bugs feels closer to u :D [04:29] Well, you know the code better than I [04:29] * anastasiamac laughs [04:29] blahdeblah: u made my day, really [04:30] There was no model destruction involved in my case, yet I ended up with the same error message. So if you want a new bug that says "juju models fails after juju-upgrade", I'm happy to supply one. [04:31] Number of anastasiamac PRs on juju > number of blahdeblah PRs on juju; hence, you know the code better than I. ;-) [04:31] (where number of blahdeblah PRs on juju == 0) [04:39] blahdeblah: my first, almost knee-jerk reaction is "no! i don't want another bug" [04:39] I figured ;-) [04:39] blahdeblah: but i think it'd b better if u did report it separately... besides i do want to see ur logs and db... [04:40] blahdeblah: coz we have not heard (I *think*) of probelms when upgrading... [04:41] blahdeblah: *after* upgrading [04:41] So, is that a "yes, I do want another bug"? [04:41] blahdeblah: and m pretty sure we test upgrades extensively... [04:41] anastasiamac: Personally, I don't think it would achieve much [04:42] blahdeblah: no, let's not for now :) [04:42] I think what's happening is there were models which were marked as destroying when I started the upgrade. [04:43] blahdeblah: yeah, that's the symptoms m seeing in all other similar failures [04:43] lingering/ghosting models [04:45] yeah - given axino's comments in 1676279, I think once you fix one, you'll fix them all [04:46] blahdeblah: i wonder if his db surgery on the models marked as 'destroying' will get u unblocked.. at least while we r sorting the issue.. [04:47] I still disagree that this isn't a bug in listing; regardless of whether a previous destroy left some models lingering, "juju models" should keep working. [04:47] blahdeblah: but since u r dealing with production as well, mayb not the most desirable way forward... [04:47] blahdeblah: there is nothing to disagree - I agree that listing has a bug [04:48] No, not the most desirable way forward, but if we can't list models, JAAS will probably be a bit dead in the water, so it may be necessary. [05:33] urulama: ^ See discussion above; looks like there are at least 3 bugs about destroyed/ing models staying in DB [05:33] juju bugs for breakfast \o/ [05:33] :) [06:14] anastasiamac: How likely is it that those model listing/destroying bugs will be fixed in a 2.1.x update? [06:15] blahdeblah: i thought u said that u r seeing in after the upgrade to 2.1.x already... 2.1.2 specifically?.. [06:16] Yep [06:16] So how likely is it that we'll see a fix in 2.1.[3-5]? [06:16] blahdeblah: we have not planning another 2.1.x... most likely in 2.2...? unless the sky falls?... [06:17] I thought that might be your answer. :-) [06:17] thanks [06:18] blahdeblah: just for reference, m still trying to figure out what goes wrong... i can reproduce destroy failure... i also *know* what's causing issues on list... but until i can nail it down, i don't know if it is easy to backport and release another 2.1.x... [06:19] No worries - just trying to work out best course of action for this controller on which models can't be listed. [06:19] urulama: ^ You mind checking whether us-east-1 is still functional from your perspective? [06:19] blahdeblah: m guessing that u have tried the standard? [06:20] anastasiamac: "the standard"? [06:20] not as yet, but I will [06:21] blahdeblah: that was aws, right? [06:21] yep [06:22] ok, looks fine, but i'll check with some big users as well === frankban|afk is now known as frankban [10:25] https://drive.google.com/open?id=0BwYF43OiGvHjWkFnLVJCVlFaRFk [10:26] ^^^ that's a dependency graph for juju/juju/api/ [11:04] anyone around to review this large but mechanical code-move branch: https://github.com/juju/juju/pull/7213 === frankban is now known as frankban|afk [18:27] anyone available this friday afternoon for a quick review? [18:28] i've got https://github.com/juju/juju/pull/7214 which fixes critical bug https://bugs.launchpad.net/juju/+bug/1680883