[01:36] any idea how to fix this warning when building juju "failed to update distro info: open /usr/share/distro-info/ubuntu.csv: no such file or directory" [01:42] tlm: update deps, there was a bug in juju/os [01:42] that or install distro-info dep [01:42] ah sweet [01:47] tlm: stickupkid: I think landed a fix to develop [01:47] on k8s we don't have the apt package installed [01:47] so the error was happening [01:48] also on centos [03:36] thumper: were you looking for a new 2.7 sha for QA testing? [03:37] wallyworld_: why did we need another? rick_h_ said he had passed on the previous one [03:37] thumper: you asked for the race fix to be targetted to 2.7 [03:38] wallyworld_: ah, no [03:38] we don't need that for 2.7.5 [03:38] just that if we are going to fix it, should do it on that branch first [03:38] since we know the problem is there too [03:38] yup, no worries, just wanted to clarify [03:38] i was hoping we would not do a new sha [04:26] Review for anyone that feels the need: https://github.com/juju/juju/pull/11350 [04:26] or desire [04:26] or whatever [04:26] ok [04:27] wallyworld_: thanks [04:28] wallyworld_: it may be easier to review the commits one by one [04:28] I made sure they made sense individually [04:28] because you care [04:28] yes, I care [04:28] they even have decent commit messages [04:28] IMO [04:29] not too shabby [04:39] wallyworld_: found the bug [04:39] you approved it :) [04:40] oh :-( [04:40] :( [04:40] I'm assuming daemonset support was only in develop? [04:40] or did it get backported to 2.7? [04:40] yeah [04:40] only develop [04:41] sweet, unreleased bug then [04:41] what was it? [04:41] ensureStatefulSet was missing else { return nil } on the create [04:41] so it would create [04:41] then immediately update it [04:42] bollocks [04:42] I was this "" close to thinking it was a k8s bug [04:43] so that explains all the symptoms? [05:06] wallyworld_: at least the deployment failure [05:13] thumper: lgtm with a few comments [05:21] thanks [05:21] I'm EODing, and away tomorrow, will look Thursday [05:21] enjoy the drive [06:42] jam: did you want to land your jujuc PR and we can pick it up? easier than forking and re-proposing [06:43] we can do the common socket code refactor etc [06:43] wallyworld[m]: sure [06:43] ty [07:11] Hi all [07:12] how do I see all permissions for a user? I have two superuser but only one can perform system calls, the other one receives permission denied... [07:39] flxfoo: juju show-model will show the users with acess to that model and what permissions they have [07:39] juju show-user will also show a specific user [07:43] wallyworld_: thanks [07:46] wallyworld_: I tried to automate the backup (percona-cluster run-action backup) of the percona cluster, with a cronjob.. at the end user needs to be superuser+admin ... any other idea to do that? [07:49] flxfoo: i can't recall what level of permission is needed to run an action. i can try it [07:54] flxfoo: i just tested a user with write access to a model containing a deployed application, and that user could run an action on that app [09:08] jam: I pushed the PR for relation-created. Can you take a look? https://github.com/juju/juju/pull/11341 [09:09] manadart: if curious how I worked around that interface limitation that we were chatting about you might also want to take a look at ^ [09:16] achilleasa: Ack, will take a look. [09:29] wallyworld_: mmm interesting... and just calling `run()` ? [09:31] flxfoo: "juju run-action" on a client [10:00] hpidcock: lgtm with a question about cm generations [10:00] i don't quite get how we handle a mismatch [10:22] manadart, is anyone calling UpdateSpace, from my initial grepping, it seems like update-space and UpdateSpace method do nothing, is that correct? [10:23] manadart, seems like I could propose a PR to remove old stuff at the very least [10:24] stickupkid: Yes, I believe there are some commands not implemented that date from the spaces MVP. [10:24] manadart, let me do that first, it's very confusing to have UpdateSpace that will panic [10:31] manadart, https://github.com/juju/juju/pull/11353 [10:42] manadart, I've got some other dead code to remove, if you've got a sec [10:43] stickupkid: Yep. [11:02] manadart, https://github.com/juju/juju/pull/11353 <- removed - I have a question about this though (https://github.com/juju/juju/pull/11353/files#diff-bfb621f2a13c2cf9efe1d5eaee1a31c7), everything else came out clean [11:06] stickupkid: I'd remove that method from API too. [11:06] manadart, agreed, just wanted another set of eyes [11:08] manadart, done [11:22] wallyworld_: no sorry I meant, if I `run('ls -l') on remote machine, I have permission denied [11:59] stickupkid: https://github.com/juju/juju/pull/11354. [12:00] manadart, you can delete the checklist from the PR, it's just away to make you remember :D [12:01] stickupkid: [12:01] Done. [12:05] manadart, done [12:11] stickupkid: any chance you could help me understand an issue with compiling Juju for windows? [12:11] jam, of course [12:11] daily? [12:11] sure [12:54] manadart, do we want `juju move-to-space db-space 172.31.1.0/28 172.31.16.0/20` or `juju move-to-space db-space id1 id2`? [13:08] jam, "success" [13:32] stickupkid: indeed :0 [14:53] rick_h_: this is how the who-departed will work: https://paste.ubuntu.com/p/KVFDYrhnXQ/ did I miss anything? [15:06] manadart, if I use SubnetsByCIDR to gather the tags for calling MoveSubnets, what's a tag in the params.SubnetsResults? [15:06] manadart, the ID? [15:06] stickupkid: Yes. [15:06] yolo [15:06] that's not confusing, tags are IDs [15:06] :| [15:07] achilleasa: looking [15:08] achilleasa: is that first number the relation id? [15:08] yes this is a debug log entry [15:08] achilleasa: that looks right to me [15:08] the plan is to expose the departing unit to charms via the JUJU_DEPARTING_UNIT envvar [15:09] (only present when running relation-departed hooks) [15:10] achilleasa: +1 [15:19] rick_h_: is this the bug we are targeting or is there a newer one out there? https://bugs.launchpad.net/juju-core/+bug/1417874 [15:19] Bug #1417874: [RFE] Impossible to cleanly remove a unit from a relation

[15:21] achilleasa: let me check. I think I added the one to the checklist on the main card [15:21] achilleasa: yea, that's the right looking one [15:22] rick_h_: cool; I will mark that as in-progress [15:23] achilleasa: woot woot [15:33] manadart, why do these exist here? https://github.com/juju/juju/blob/develop/core/network/space.go#L26-L49 [15:37] stickupkid: Not sure where those are used in conversion, but they might be better closer to usage... [15:37] manadart, yeah, these aren't core, esp. if we want to rip out the CLI, thise makes it super hard [15:38] oh god, they're like fucking view models and core models - yikes [15:39] manadart, got a sec [15:39] ? [15:40] stickupkid: Yep. [16:08] hi all, [16:08] we have an issue with rackspace, when trying to deploy a new unit we have "cannot run instance: failed to run a server with nova.RunServerOpts{Name:"juju-00f9f6-v5-prod-50", FlavorId:"general1-2", ImageId:"6e731e8e-ff3d-4e27-be6c-147fde688b7c", UserData:[]uint8{0x1" [16:09] where the imageID is no longer available [16:09] support told me to use another image ID [16:09] but I don't know how to tell juju to use that new image ID... [16:09] any idea? [17:16] I think I just found the function with the longest signature in juju! https://github.com/juju/juju/blob/develop/worker/uniter/runner/context/export_test.go#L156 [17:16] * rick_h_ dies just a little more inside [17:16] wow, that...is impressive [17:17] Its call-site is even more impressive: https://github.com/juju/juju/blob/develop/worker/uniter/runner/context/env_test.go#L113 [17:18] I will ... structurify it :D [17:19] hah, good luck! [17:19] I need to add an extra field there so... [20:10] hi all [20:10] howdy flxfoo [20:10] I find the reference if the "bad" image Id in /var/lib/juju/bootstrap-params file in the controller instance [20:11] refered as a custom-image-metada [20:11] can I remove or change that value? [20:13] flxfoo: normally that's set via controller-config I believe. Is this a running controller? [20:16] rick_h_: yeah running controller [20:17] flxfoo: ok, let me dbl check if it's model-config or controller-config you'd set to change that value [20:18] rick_h_: I have neither that key in controller-config nor model-config :( [20:19] or I don't look at the right place? [20:20] flxfoo: yea, refreshing. I think you're into https://discourse.jujucharms.com/t/cloud-image-metadata/1137 [20:20] flxfoo: which is using the metadata-source arg on bootstrap. [20:21] flxfoo: and you can see some metadata related config in the model-config `juju model-config | grep meta` [20:21] flxfoo: I'd be curious what those are currently set to? [20:22] rick_h_: agent-metadata-url defalt "" [20:22] rick_h_: agent-metadata-url default "" [20:22] rick_h_: container-image-metadata-url default "" [20:23] rick_h_: image-metadata-url default "" [20:23] that's what I got [20:23] I think the issue is related to that thread https://discourse.jujucharms.com/t/rackspace-cloud-london-region-missing-images/1339/5 [20:24] where had been applied a "temporary" fix... using simplestream... and I think it should revert... but I don't know how... :( [20:24] flxfoo: oh ok, good question. [20:24] this `custom-image-metadata should probably be empty or so.. [20:25] rick_h_: I suppose if I change that `bootstrap-params` file that won't change anything... right? [20:29] flxfoo: yea looking. I'd not expect that to do anything now that bootstrap is over. Since it's a bootstrap arg, and not config, it's not really a thing that's got a knob for "changing it" [20:29] flxfoo: so trying to validate if there's a clear path to changing it or not. I'd hate to bork things even more. [20:30] rick_h_:ok thanks... do you a location where i could check that as well? or is it source code only? [20:31] flxfoo: yea in this case I'm just poking at source trying to see [20:31] rick_h_: k [20:31] flxfoo: honestly might have to wait and ask wallyworld for his expertise in this area [20:31] and he is the one who was appearing in the thread... so... [20:32] rick_h_: is he around sometime... better tomorrow? [20:32] flxfoo: he'll be online in the next couple of hours [20:33] ok will try to wait a little then... thanks for your time... I though i was going crazy... (not to mention the quarantine) [20:33] maybe you could answer that following question though... [20:34] so i have this `custom-image-metadata` with at least a image id which is causng trouble because not theire anymore... [20:34] then the following lines defines the regions etc... [20:35] controller-cloud: name:rackspace. etc... regions etc... [20:35] right, that's the general cloud definition baked in. It tells the contrller things like api endpoints to use/etc [20:36] I suppose that custom-image-metadata is taking precedence on the that following list... am I making a correct assumption? [20:37] flxfoo: well it's filling in the images specific data and not everything there. It can't refer to a region that the cloud list doesn't know about, for instance [20:37] flxfoo: but it's tied together for sure [20:38] custom-image-metadata: '[{"id":"6e731e8e-ff3d-4e27-be6c-147fde688b7c","virt":"pvhvm","arch":"amd64","version":"18.04","crsn":"LON","region":"LON","endpoint":"https://lon.identity.api.rackspacecloud.com/v2.0/"}]' [20:39] here it would tell to use that particular image in case of region LON endpoint reached... [20:39] flxfoo: right so that's saying that image is avilable in that region [20:39] ok [20:39] flxfoo: there could be more images with different architectures/etc as well. [20:39] flxfoo: in particular different ubuntu series like 14.04, 16.04, 18.04 but we know not all regions have the same things, etc [20:40] rick_h_: that's my point, the URL (thread) I posted was saying that for some time images were not available for LON region... so a custom bootstrap was made... then the images are then available... but the controller does not know about it [20:41] I am curious what would look like a setting when images are provided by region... [20:42] flxfoo: right, so this gets interesting as the use case for htis is more bootstrapping on openstack and defifining things for that. I'm not sure how it's mutated/managed over time [20:42] flxfoo: in theory all this data should be pulled from an online stream url. If the stream is valid, and Juju is looking at that url, it might just be a cause of removing this. [20:43] flxfoo: but in looking at the code a metadata directory is setup on the machine and such so I think there's more to it than that [20:43] so that's where I don't want to mess you up and wait for wallyworld [20:43] rick_h_: exactly... obviouslly it is not manageable, because now that rackspace upgraded their images (That I suppose is my issue) then my model is just no its own, I can not deploy or up.dwon scale anything [20:44] flxfoo: yea :( maybe you can use that model config to set the image-metadata-url to the official one? [20:44] rick_h_: ok thanks... [20:44] flxfoo: but it's not clear who wins priority atm [20:45] rick_h_: right, well I have to check... would it be the images endpoint url? [20:46] rick_h_: (if I cURL rackspace api I can get a list of available images) [20:46] flxfoo: so I was thinking the model-config parameter image-metadata-url [20:46] flxfoo: we can test this by adding a new model if you can? [20:46] and then set that value, and see if you can `juju add-machine` [20:47] rick_h_: I suppose I won't interfere with the current model? [20:47] rick_h_: sorry I am a bit new to juju... [20:47] rick_h_: what are the risks? :p [20:48] flxfoo: right, but doing it in a new model we're reducing any risk [20:48] flxfoo: once we're done with the test we can just remove the model [20:49] rick_h_: sure I got that, not playing with the current model but a new one... [20:51] rick_h_: I don't have any config to pass to a new model though [20:55] flxfoo: do you know the url that rackspace hosts for the image data normally? [20:57] http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:rax.json could it be this? [20:57] naaaa [20:57] sorry [20:59] https://lon.images.api.rackspacecloud.com/v2//images [21:00] through their api... [21:00] so you got a json list of image definitions with other details... [21:00] hmm, so the question is can you set that in the new model using `juju model-config image-metadata-url="https://lon.images.api.rackspacecloud.com/v2//images"` [21:04] rick_h_: as the controller has the `tenant-name` + the identity endpoint defined by region , i suppose it should be just https;//lon.images.api.rackspace.com/v2 [21:05] flxfoo: yea, might be. That I'm not 100% sure on. one thing we can do is make sure to crank up logging and watch for what it's doing in the actual requests. [21:06] flxfoo: juju model-config logging-config="=DEBUG;unit=TRACE" [21:06] well probably not unit-trace [21:06] flxfoo: juju model-config logging-config="=DEBUG;" [21:10] rick_h_: can that debug configuration load machines? [21:11] progress report for last week is live https://discourse.jujucharms.com/2807 [21:12] flxfoo: what do you mean by "load machines"? [21:12] flxfoo: it'll make all logging at debug level until it's set back to info [21:14] ok [21:15] I will try now to see if that gives something [21:17] rick_h_: should I add a charm like memcached or something small? [21:19] flxfoo: just `juju add-machine` [21:19] flxfoo: and see if you can get a new machine to come up [21:22] ok I have something different [21:22] hah, not sure if that's a good sign or a bad one [21:23] failed to start machine 0 (multiple networks with label "": [5836eb06-731c-4653-87ac-bebcd97f1a18 dd3389c6-cc70-43bf-a8ec-489be09855cb [21:23] :) [21:24] To resolve this error, set a value for "network" in model-config or model-defaults; [21:24] or supply it via --config when creating a new model), retrying in 10s (10 more attempts) [21:25] flxfoo: ok, so have to set the model value for that [21:25] flxfoo: exactly [21:25] flxfoo: so have to add that to the model-config for our test model [21:25] flxfoo: juju model-config network=XXXXXX (not sure which one you want) [21:27] rick_h_: wait not sure what I am suppose to put here [21:29] flxfoo: one of those uuids from above [21:30] flxfoo: we might have to check and see if there's a value in your other working model? [21:30] flxfoo: I would assume that was needed there as well [21:30] flxfoo: you can get the current config values by just doing `juju model-config network` [21:45] rick_h_: ok passes, but still have the original error line from custom-metadata-url content [21:45] :( [21:50] rick_h_: so I suppose the custom-metadata-url gets precedence [22:12] rick_h_: which code did you look at for `custom-metadata-url`? [22:18] hey timClicks, how should I fix a bug in the docs? [22:20] flxfoo: if you bootstrapped with --metadata-source arg, then that custom image metadata is stored in mongo and takes precidence. you'll need to delete those image metadata records. to do that you'll need to turn on the image-metadata feature flag on the controller and on the client. to set on the controller, "juju controller-config features=[image-metadata]". you might need to bounce the controller agent systemd, not sure. then [22:20] on the client, export JUJU_DEV_FEATURE_FLAGS=image-metadata. after that, you can juju metadata delete-image . you can also juju metadata list-images [22:21] wallyworld: ok, thanks... I will need to read that several times... :) [22:22] sure, it's a bit messy because image metadata manipulation is not something normally exposed out of the box - you need a feature flag enabled and if not done at bootstrap, you need to turn it on afterwards. if anything is unclear, just ask. i would expect there will be questions [22:23] wallyworld: thanks a lot... give me a bit time :) [22:23] i assume you had custom jsom image metadata at bootstrap time? and you bootstrapped with --metadata-source [22:24] in any case, once the feature flag is enabled, you'll have access to add/list/delete custom image metadata commands [22:24] to see what your controller is using [22:29] wallyworld: correct bootstrapped with --metadata-source (I did not do that personally but the person in charge at the time) [22:32] wallyworld: so I understood properly, we will tell the juju client to use the `image-metadata` with the JUJU_DEV_FEATURE_FLAGS env var. Then by setting the related controller config entry `features`... restart the controller agent (which I need to check which process it is)... did I get things right? [22:33] wallyworld: I mean , If you undrestand what I am writing... I look tired :) [22:33] wallyworld: I think I got the picture... [22:33] wallyworld: are you around about that time usually? [22:36] wallyworld: the fact that I modified the `image-metadata-url` would have an incidence? [22:37] flxfoo: that sounds right to me - I think wallyworld might have afk'd for a moment [22:40] flxfoo: you can bounce the juju agent on the controller with `sudo systemctl restart jujud-machine-0.service` [22:40] (assuming this is just a one-machine controller?) [22:40] babbageclunk: yes [22:41] babbageclunk: thanks [22:41] flxfoo: sorry, was caught up with something else. yeah, summary is correct. first step is to get the feature flag enabled and use the list-images command to see what the controller is using [22:41] wallyworld: nm [22:41] tz here is GMT+10 [22:41] wallyworld: k [22:43] to restart the controller agent, ssh into the controller machine (ususally juju ssh -m controller 0) and then restart the juju-machine-0 systemd service [22:43] wallyworld: I will try that tomorrow morning.. and will let you know... I think things/issues come from there https://discourse.jujucharms.com/t/rackspace-cloud-london-region-missing-images/1339/5 [22:44] wallyworld: many thanks, I would not have found that by myself for sure... :) [22:45] np, here to help you get this sorted. good that rackspace has complete image metadata but a mess to clean up for sure [22:45] wallyworld: and they are not helping... you are... thanks guys [22:46] I am out... good night/day :) [22:46] ttyl :-)