[02:13] kelvinliu: tlm: hpidcock: to allow for juju to know whether a k8s charm wants to be treated as an operator, i've added a "mode" attribute to the charm deployment metadata. when juju deploys such an operator charm, it will not expect there to be a separate workload image deployed and will treat the charm operator pod itself as a unit as far as the juju model is concerned. https://github.com/juju/charm/pull/306 [02:14] see if you think that makes sense [02:14] will take a look [02:17] lgtm wallyworld [02:18] ty. still need a bit of thought about other things like stateful vs stateless [02:42] wallyworld: if mode was set, service and type should not set, right? [02:44] so probably add a validation for this check [02:44] kelvinliu: not sure, we could still allow them to control how the operator is deployed instead of juju choosing [02:45] ie does it have to be a stateful set [02:45] maybe not [02:45] there is no workload in this case [02:45] so no deployment.type [02:45] the workload is the operator - that may need to be a stateful set [02:46] isn't it up to the user to decide? [02:46] are we talking about charm as an operator? [02:46] yes [02:46] HO? [02:46] sure [06:46] wait wallyworld why is the operator deployed as a deployment rather than a statefulset? [06:48] that doesn't make sense, as a deployment doesn't have to ensure there is at most 1 pod [06:48] I think we want statefulsets for operators [06:52] we do hpidcock ? [06:53] uhhh yeah [06:53] if we have two uniters for a unit things will get weird [06:55] yep good point [06:55] I thought we where talking normal kube operators [07:03] hi guys [07:04] I did what wallyworld propose... I don't see the `list-images` command though, `cached-images` etc... return an empty list still [07:09] `juju metadata list-images` returns ERROR unknown object type "ImageMetadataManager" (not implemented) [07:09] so what version am I suppose to run? [07:44] the `ImageMetadataManager` not implemented, sounds like the "feature" image-metadata is not enabled no? [07:44] the controller-config is listing it though... [07:45] I check if the changes were effecive... looks like... a typo? [08:21] flxfoo: that error means that the controller is not using the feature flag. yuo definitey did [08:21] juju controller-config features=[image-metadata] [08:21] ? [08:23] achilleasa: a trivial charm.v6 PR if you have a moment https://github.com/juju/charm/pull/307 will be used in a juju feature so will be tested end-end as part of that work [08:46] wallyworld: yes I did... sorry was away [08:47] juju controller-config|grep metadata , returns features - image-metadata [08:47] I restarted the agent on the controller as well [08:48] yeah it seems not to use that features config... [08:48] wallyworld: maybe something else is missing... can I paste the controller config somewhere? [08:55] wallyworld, flxfoo : isn't that a env var feature flag not a controller-config one? (I thought it predated controller-config) [08:57] wallyworld: approved [09:03] wallyworld: sorry not folliwing that [09:05] wallyworld: I did `juju controller-config features=[image-metadata] and the entry appears when displaying `juju controller-config` with `features - image-metadata` [09:06] and I set the env as well (export [09:06] export JUJU_DEV_FEATURE_FLAGS=image-metadata [09:09] wallyworld: juju version is 2.6.8 (if it is needed)... If I read the code properly, that is not a version issue [09:12] wallyworld: are you saying that only the env var is necessary? [09:19] wallyworld: https://pastebin.com/ZyUxSeXs [09:40] jam: ah you might be right; i thought the feature flags from the controller config were universally used. are you saying the systemd.conf file needs editing to adjust the env var? [09:41] not sure. might just need agent.conf to be updated. i haven't worked with them in a while [09:41] me either obviously :-( [09:44] flxfoo: based on the above, you may need to set JUJU_ENV_FEATURE_FLAGS env var in maybe the systemd conf file- the need is to get it exposed to the running process [09:46] wallyworld: JUJU_ENV_FEATURE_FLAGS and not JUJU_DEV_FEATURE_FLAGS right? [09:47] flxfoo: right, JUJU_DEV_FEATURE_FLAGS [09:47] sorry [09:48] wallyworld: ok let me try [09:50] wallyworld: ERROR unrecognized command: metadata list-images [09:53] wallyworld: with JUJU_DEV_FEATURE_FLAGS, the error is different, ERROR unknown object type "ImageMetadataManager" (not implemented) [09:53] wallyworld: so that might be JUJU_DEV_FEATURE_FLAGS... but the controller is not responding to the call? [09:58] wallyworld: I don't see any JUJU_ENV_FEATURE_FLAGS in the juju code... but JUJU_DEV_FEATURE_FLAGS yes [10:00] wallyworld: I think the client is fine, but suspect that the controller is not "reacting" maybe it needs a "reload"? [10:01] wallyworld: could it be something like `juju controller-config features=[image-metadata-manager]' ? [10:03] flxfoo: the existence of the command is controlled by setting the feature flag via the env var on your client machine. the ability of the controller to accept the command is via setting the feature flag on the server [10:04] export JUJU_DEV_FEATURE_FLAGS=image-metadata [10:04] on the client should allow the comand to run [10:05] setting the env var of the jujud process on the controller is needed to make the controller aware of it [10:05] i thought setting the controller-config would be enough but it seems that it's an old feature flag so env var is needed [10:06] wallyworld: aaahaaaa sorry [10:09] wallyworld: so modifying that script /lib/systemd/system/jujud-machine-0/exec-start.sh ? [10:10] flxfoo: sounds about right, i don't have a running system in front of me to check [10:14] I will try in to set it better in `/etc/environment` file [10:15] and restart the controller [10:16] manadart: did you see my reply in 11341? Also, jam did you get a chance to look at this PR? I have another PR based off this one and I am trying to get this one landed so I can rebase the next one [10:18] achilleasa: Yep. All good. [10:18] wallyworld: ok I patch the script (for testing) and restarted the jujud agent process [10:20] achilleasa, I haven't looked through the code yet, though I did read the coverletter. [10:21] wallyworld: .... I added `export JUJU_ENV_FEATURE_FLAGS=image-metadata` before the line `'/var/lib/juju/tools/machine-0/jujud' machine --data-dir '/var/lib/juju' --machine-id 0 --debug` [10:22] wallyworld: restarted the agent [10:22] nothing new , same error (not implemented) [10:23] wallyworld: ok got it [10:24] same variable name seem to be used both sides `JUJU_DEV_ENV_FEATURE_FLAGS` [10:24] yes :-) [10:24] wallyworld: sorry, I misunderstood ... [10:24] sorry if i mistyped it [10:25] :D [10:25] ok, now I can see the custom imaged listed [10:25] awesome [10:25] now you can delete image id records as needed [10:25] but only this one... do I have to use `image-metadata-url` with rackspace url to images? [10:25] o ok sure [10:26] i thought the images were published at cloud-images.ubuntu.com, but i'd need to check [10:26] wallyworld: ok deleted [10:26] or if you have a place where the image metadata lives you can set the image-metadata-url [10:27] rakspace images are there i think [10:27] http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:rax.json [10:27] so it should just work hopefully [10:28] we really should update the feature flag on the controller so that setting it via controller config works [10:28] flxfoo: late here, i'm tired, so i might leave you to it. one of the other folks can help if needed [10:28] wallyworld: can I help with that? [10:29] flxfoo: it's a simple fix, i can try and do it nex tweek [10:29] it's not something people really use a lot [10:29] wallyworld: of course :p... you are far... I will try to get images back in... maybe using the api endpoint as well? [10:29] if the publisehd image metadata works, no need to set anythg else really [10:30] *published [10:30] wallyworld: have a nice eve, keep you posted... [10:30] awesome, yeah, let me know how yu go [10:31] wallyworld: sorry to ask last little one, should I update `image-metadata-url` or not? because right now the list-images is empty... [10:36] flxfoo: list images is only for custom metadata [10:36] the metadata published at the default location as posted above should be sufficient [10:37] the image-metadata-url is only for if you are hosting your own metadata somewhere and not using custom image metadata in the db [10:37] so there's 3 places image metadata can come from [10:37] you only need to have it in one place [10:38] (usually the default place at cloud-images.ubuntu.com) works [10:40] wallyworld: so I empty it ... [10:41] wallyworld: ERROR matching cloud image metadata not found (not found) [10:42] wallyworld: will look deeper, if I find something [10:48] flxfoo: that mean that what's been published doesn't match your cloud endpoint / region. you can look at the json metadata (index file and data file at cloud-images) to see what's been published [10:52] wallyworld: ok thanks [10:57] wallyworld: well the difference with the other providers in that `products` is empty for rax [11:21] achilleasa, still reading through your very thorough QA steps. a couple questions [11:23] 1) do we know why we end up seeing double 'relation-changed' for back to back for things like wordpress monitoring mysql [11:26] jam: not sure; I believe this is not related to my changes but it will be pretty easy to verify with develop HEAD; let me try it out [11:26] 2) Have we checked whether you can do relation-get/set --app during relation-created and/or leader-elected after this change? [11:28] jam: 2) scenario 4 tests relation-set --app in relation-created (and relation-get after leader-elected). Do you have a different scenario in mind? [11:28] achilleasa, I think it is because relation-joined forces a relation-changed event, and then mysql actually updates its contents after seeing wordpress. [11:29] achilleasa, probably just haven't read that far yet. [13:42] achilleasa: i get tired on ! checks sometimes. especially when it becomes a double negative, though not the case here. will change. [13:42] achilleasa: is there anything else on the PR? what is left to move to approved? [13:43] hml: just that. I will go ahead and approve so you can get it landed [13:46] achilleasa: thank you. i’ll make the change, then squash the commits. before merging [14:23] rick_h_: I think a per-mddel limit is do-able with a little bit of effort although it will not be strict; a write by a unit might exceed the limit (slightly) but future writes will immediately fail unless the limit is raised. If that sounds good I will try to cobble together a PoC [14:26] Because of the way that the total size will be calculated we can probably be even more granular and have a limit for charm state and a separate limit for uniter state [14:34] achilleasa: right, that's the expectation I think is ok that the last one will succeeed and overwrite it [14:35] or overspend it I guess [14:48] manadart, did you test the spaces with no VPC? [14:49] stickupkid: As in with a legacy AWS account? I have not done that. [14:49] manadart, I just happened to bootstrap onto jams aws, and zero subnets show up [14:49] stickupkid: Then you don't get spaces :) [14:57] rick_h_: so, while there isn't a clean way to get the stats, I discovered 'Object.bsonsize' which gives you the size of a doc. Next logical step is obviously to map-reduce and sum the sizes :D https://paste.ubuntu.com/p/trWRwCPfQ9/ [14:57] the above snippet can be modified to aggregate sizes per-field so we get a charm vs non-charm state sum [15:01] manadart, quick question, when moving the last subnet(s) to another space, shouldn't we clean or hide that space [15:01] ? [15:02] manadart, this looks funky [15:02] https://paste.ubuntu.com/p/DgZgjDfpjM/ [15:02] stickupkid: No. The user can delete it if desired. [15:02] booo [15:02] stick funky looking [15:02] s/stick/still/ [15:05] lovely it works really nicely [15:36] achilleasa: well, do we need to? What is the scope of a single doc? [15:36] achilleasa: a model level summary was just something that was possibly easy, if each doc is unit sized then we can just make the config a per-unit level? [15:39] rick_h_: the idea is to calculate the total size for all state-docs for a particular model uuid and use it as an early cutoff for writes. However we still need to persist the total size and update it as part of our txn logic to avoid a race [15:40] where N units try to update the state at the same time and each one things they are within the range but once you commit the txn you may be well above the limit === narindergupta is now known as narinderguptamac [15:47] achilleasa: right but my point is that the idea of a model-level rule was arbitrary on our end [15:47] achilleasa: so if unit-level is easier/makes more sense we can ignore the model level aggregation issue? [15:56] rick_h_: can you join us in daily? [15:57] achilleasa: omw [16:31] rick_h_: per-unit limiting is easier to implement but does not guarantee an upper bound on the data stored per model (i.e. I can add an arbitrary number of units). It all depends on what we want to achieve. In my view per-model limiting makes more sense if we want to provide max-space guarantees [16:34] hi [16:52] achilleasa: I think we just need a knob. A per-unit seems to make sense because it doesn't block up everyone but only limits the effect on a single unit [16:52] mordkoff: howdy [16:55] achilleasa: if you have time let's catch up or wait until tomorrow. I think if per-unit is easier then we should go down that road but we can go through the different merits [16:55] rick_h_: hmmm... but I thought the requirement was to put a configurable upper bound on storage/model. If we are fine with a per-unit limit I can prototype that instead [16:55] rick_h_: daily? [16:55] achilleasa: just needs some limit. At the time I thought we'd have to do it model-wide due ot mongodb namespacing [16:55] achilleasa: omw [17:29] rick_h_: regarding the bundlechanges fix, I don't think it is possible to define an offer without having a valid application block in your bundle due to validation rules. I will leave the present-in-bundle AND controller logic or get an error in for the fix but I cannot come up with a test case to trigger the error [17:33] hi all (again) :) [17:33] juts q quick one [17:34] after enabling the `image-metadata` feature, now in some other scripts (python) I have some errors with `unknown facade ImageMetadataManager` [17:35] flxfoo, pylibjuju? [17:35] along with KeyError: 'ImageMetadataManager' [17:36] stickupkid: I suppose so... coming along with python3.6/site-packages/juju/client/connection.py [17:36] I suppose that is pylibjuju yeah [17:37] flxfoo, so this have been fixed in the follow PR https://github.com/juju/python-libjuju/pull/382, but hasn't been released yet [17:37] o [17:37] k [17:37] flxfoo, can you by anychange use the head of master [17:37] ? [17:38] flxfoo, sorry for the breakages, but this was done to help ensure we didn't break pylibjuju when using different facade versions. [17:38] stickupkid: well, I will keep that i mind... still I can not get the OS image list from rax... so maybe I will have to go back to the custom-image-metadata... not sure how everything will fit together honestly [17:39] stickupkid: no pb... I think the situation with rackspace + some juju changes those days are putting the platform in an unstable position... [17:39] flxfoo, be mindful that pylibjuju doesn't know about custom-image-metadata, so you'll have to use juju CLI directly (shell out etc) to use that feature [17:40] stickupkid: the thing is , before with the custom-image-metadata setting, the scripts works perfectly fine (tested one only, actually used for deployment) [17:41] I start to think that we are too much coupled with the tools... then... for about a few minutes I could not even deploy anything (web app) nor scaling up/down the platform... that's pretty horrible situation [17:41] :( [17:42] or we completely missed something on the way... [17:42] I don't know if other people here are rackspace customers? [17:42] flxfoo, have you seen this https://discourse.jujucharms.com/t/juju-office-hours-2020-03-26/2819 [17:43] flxfoo, ;-) [17:43] stickupkid: quick PR for you https://github.com/juju/bundlechanges/pull/60 [17:44] stickupkid, interesting thanks [17:45] stickupkid: thanks anyway :p ... my daughter needs attention... :) [17:46] achilleasa, "if we are abusing [17:46] // bundle deployments to do upgrades" [17:46] HAHA [17:46] stickupkid: you know... for future code archeologists... [17:47] achilleasa, why this line? https://github.com/juju/bundlechanges/pull/60/files#diff-31a4a7824225629df071a1cba8b785d5R346 [17:48] why not `var reqs []string` [17:48] flxfoo: yea sorry, not many rackspace folks and they were killing off their public cloud last I was aware so not a lot of love in there [17:48] stickupkid: just reuses the existing slice capacity between loop iterations; pretty much equivalent [17:49] achilleasa, pretty much hard to read [17:49] stickupkid: np, let me change it to use a var [17:49] stickupkid: definitely, we will probably move asap... [18:14] hml: can you take a look at https://github.com/juju/juju/pull/11358 and land it if it works for you as I am hitting EOD? I will forward-port it to develop in the morn [18:14] achilleasa: sure [20:01] sorry to disturb again [20:02] I would like the time being to use the custom-image-metadata with new values, I generated the files... how can I insert in the mongodb? [20:04] maybe `juju metadata add-image" ? [20:04] if yes what should I put for storage-size and storage-type? [20:28] flxfoo: is this for openstack? [20:28] * thumper is wondering where our docs are falling down [20:28] well that was bad timing [20:59] Can I bootstrap a running controller? (probably stupid question) [21:01] hi flxfoo, can you explain what you mean? [21:01] bootstrapping is the process of starting a controller [21:02] if you have one running, what are you trying to do? [21:02] @thumper: ok, that what I though... to use `metadata-source` again [21:02] flxfoo: can you bring up a new controller? [21:02] flxfoo: and use juju migrations to move your model over to the new one? [21:03] rick_h_: wait wait [21:03] flxfoo: I asked this question just before you dropped from the channel before... [21:03] ;) [21:03] flxfoo: is this for openstack? [21:03] it is link to rackspace [21:03] they remove a os image [21:04] that my colleague bootsrapped our controller using that image [21:04] now [21:04] flxfoo: so if you're able to bootstrap a fresh controller juju supports migrating models over https://discourse.jujucharms.com/t/migrating-models/1152 [21:04] that was some time ago, I was not there [21:05] now talking with wallyworld yesterday, we enable the `image-metadata` feature in order to remove the "custom" image... [21:06] unfornutely back to "default" no rackspace os images appear to be available [21:06] flxfoo: is this on their public cloud? [21:06] so what I try to achieve is to use that `custom-image-metadata` with a new image id rackspace gave me [21:06] rick_h_: guess so yes... [21:07] I managed (about half an hour ago) to add an image vis `juju metadata add-aimge` [21:08] but when I use it (juju add-machine` juju loop creating instances , that I can see in rackspace *all* as building state stuck at 90% [21:21] flxfoo: so you got a new image id added and juju attempts to use that image to provision a machine but the provisioning fails? that sounds like an image / cloud issue that needs to be investigated [21:30] wallyworld: hi there... well It is actually very strange [21:31] so first as I said juju just looped saying that the machine is state BUILT, so tryinh again... and now I have 18 instances in rackspace in build state... :( [21:31] and [21:31] I tried to put some debu on... but then my image first disappear from list images [21:32] when remove-machaine --force would that remove an image as well eventually? [21:34] I am not getting what is going on... [21:34] anyway... [21:34] wallyworld: how could I set the `custom-image-metadata` with "new" values? is that possible? [21:38] flxfoo: it may be that the custom image metadata added via add-image has an expiry, i'd need to check. you may need to re-run add-image. i have no idea why the machine gets stuck in building. that seems like a cloud issue. in any case, you'll need to use the dashboard or something to inspect the state of the instance [21:39] it the cutom image metadata does have an expiry, we may be able to work around that, but the first issue is to see why the chosen image is not working [21:41] wallyworld: but I did not set any expire time when using add-image. juju metadata add-image 498bfa38-d288-4e37-b975-a4acb77f8557 --series bionic --region LON --virt-type pvhvm [21:42] flxfoo: no, but juju may be setting an expiry, i'll need to check [21:42] maybe I should just put bionic and region [21:42] wallyworld: a ok ok [21:42] wallyworld: if it does , the expire time is pretty short [21:43] 6 hours i think. but this stuff was done ages ago so i'm not sure [21:48] wallyworld: the image disappear after a few minutes [21:48] wallyworld: I added it already 4 times in the last couple of hours or so [21:49] wallyworld: just disappear, I added it about 10 minutes ago [21:50] wallyworld: storage-type what could be a valis value for that one in add-image? [21:57] flxfoo: storage-type is cloud dependent. i am not sure what's needed for rackspace. i think that was added more for aws. rackspace doesn't use it i don't think. i am not sure why the iamge metadata is not sticking until i get a chance to look. the fact that the image is being discovered and used to provision a machine but getting stuck is the first thing to solve. and that needs to be done using hte cloud tooling, dashboard, [21:57] ssh into image etc [21:59] wallyworld: right... did not find a way to get in yet... as "building" there is no obvious way to get in... but will dig into that thanks [22:00] flxfoo: good luck. once that is figured out we can look at the other things [22:57] cool lxc petname: infinite-sailfish [23:20] corona-prince would be a good one