/srv/irclogs.ubuntu.com/2020/03/25/#juju.txt

wallyworldkelvinliu: tlm: hpidcock: to allow for juju to know whether a k8s charm wants to be treated as an operator, i've added a "mode" attribute to the charm deployment metadata. when juju deploys such an operator charm, it will not expect there to be a separate workload image deployed and will treat the charm operator pod itself as a unit as far as the juju model is concerned.  https://github.com/juju/charm/pull/30602:13
wallyworldsee if you think that makes sense02:14
tlmwill take a look02:14
tlmlgtm wallyworld02:17
wallyworldty. still need a bit of thought about other things like stateful vs stateless02:18
kelvinliuwallyworld: if mode was set, service and type should not set, right?02:42
kelvinliuso probably add a validation for this check02:44
wallyworldkelvinliu: not sure, we could still allow them to control how the operator is deployed instead of juju choosing02:44
wallyworldie does it have to be a stateful set02:45
wallyworldmaybe not02:45
kelvinliuthere is no workload in this case02:45
kelvinliuso no deployment.type02:45
wallyworldthe workload is the operator - that may need to be a stateful set02:45
wallyworldisn't it up to the user to decide?02:46
kelvinliuare we talking about charm as an operator?02:46
wallyworldyes02:46
kelvinliuHO?02:46
wallyworldsure02:46
hpidcockwait wallyworld why is the operator deployed as a deployment rather than a statefulset?06:46
hpidcockthat doesn't make sense, as a deployment doesn't have to ensure there is at most 1 pod06:48
hpidcockI think we want statefulsets for operators06:48
tlmwe do hpidcock ?06:52
hpidcockuhhh yeah06:53
hpidcockif we have two uniters for a unit things will get weird06:53
tlmyep good point06:55
tlmI thought we where talking normal kube operators06:55
flxfoohi guys07:03
flxfooI did what wallyworld propose... I don't see the `list-images` command though, `cached-images` etc... return an empty list still07:04
flxfoo`juju metadata list-images` returns ERROR unknown object type "ImageMetadataManager" (not implemented)07:09
flxfooso what version am I suppose to run?07:09
flxfoothe `ImageMetadataManager` not implemented, sounds like the "feature" image-metadata is not enabled no?07:44
flxfoothe  controller-config is listing it though...07:44
flxfooI check if the changes were effecive... looks like... a typo?07:45
wallyworldflxfoo: that error means that the controller is not using the feature flag. yuo definitey did08:21
wallyworldjuju controller-config features=[image-metadata]08:21
wallyworld?08:21
wallyworldachilleasa: a trivial charm.v6 PR if you have a moment https://github.com/juju/charm/pull/307 will be used in a juju feature so will be tested end-end as part of that work08:23
flxfoowallyworld: yes I did... sorry was away08:46
flxfoojuju controller-config|grep metadata , returns features                  - image-metadata08:47
flxfooI restarted the agent on the controller as well08:47
flxfooyeah it seems not to use that features config...08:48
flxfoowallyworld: maybe something else is missing... can I paste the controller config somewhere?08:48
jamwallyworld, flxfoo : isn't that a env var feature flag not a controller-config one? (I thought it predated controller-config)08:55
achilleasawallyworld: approved08:57
flxfoowallyworld: sorry not folliwing that09:03
flxfoowallyworld: I did `juju controller-config features=[image-metadata] and the entry appears when displaying `juju controller-config` with `features - image-metadata`09:05
flxfooand I set the env as well (export09:06
flxfooexport JUJU_DEV_FEATURE_FLAGS=image-metadata09:06
flxfoowallyworld: juju version is 2.6.8 (if it is needed)... If I read the code properly, that is not a version issue09:09
flxfoowallyworld: are you saying that only the env var is necessary?09:12
flxfoowallyworld: https://pastebin.com/ZyUxSeXs09:19
wallyworldjam: ah you might be right; i thought the feature flags from the controller config were universally used. are you saying the systemd.conf file needs editing to adjust the env var?09:40
jamnot sure. might just need agent.conf to be updated. i haven't worked with them in a while09:41
wallyworldme either obviously :-(09:41
wallyworldflxfoo: based on the above, you may need to set JUJU_ENV_FEATURE_FLAGS env var in maybe the systemd conf file- the need is to get it exposed to the running process09:44
flxfoowallyworld: JUJU_ENV_FEATURE_FLAGS and not JUJU_DEV_FEATURE_FLAGS right?09:46
wallyworldflxfoo: right, JUJU_DEV_FEATURE_FLAGS09:47
wallyworldsorry09:47
flxfoowallyworld: ok let me try09:48
flxfoowallyworld: ERROR unrecognized command: metadata list-images09:50
flxfoowallyworld: with JUJU_DEV_FEATURE_FLAGS, the error is different, ERROR unknown object type "ImageMetadataManager" (not implemented)09:53
flxfoowallyworld: so that might be JUJU_DEV_FEATURE_FLAGS... but the controller is not responding to the call?09:53
flxfoowallyworld: I don't see any JUJU_ENV_FEATURE_FLAGS in the juju code... but JUJU_DEV_FEATURE_FLAGS yes09:58
flxfoowallyworld: I think the client is fine, but suspect that the controller is not "reacting" maybe it needs a "reload"?10:00
flxfoowallyworld: could it be something like `juju controller-config features=[image-metadata-manager]' ?10:01
wallyworldflxfoo: the existence of the command is controlled by setting the feature flag via the env var on your client machine. the ability of the controller to accept the command is via setting the feature flag on the server10:03
wallyworldexport JUJU_DEV_FEATURE_FLAGS=image-metadata10:04
wallyworldon the client should allow the comand to run10:04
wallyworldsetting the env var of the jujud process on the controller is needed to make the controller aware of it10:05
wallyworldi thought setting the controller-config would be enough but it seems that it's an old feature flag so env var is needed10:05
flxfoowallyworld: aaahaaaa sorry10:06
flxfoowallyworld: so modifying that script /lib/systemd/system/jujud-machine-0/exec-start.sh ?10:09
wallyworldflxfoo: sounds about right, i don't have a running system in front of me to check10:10
flxfooI will try in to set it better in `/etc/environment` file10:14
flxfooand restart the controller10:15
achilleasamanadart: did you see my reply in 11341? Also, jam did you get a chance to look at this PR? I have another PR based off this one and I am trying to get this one landed so I can rebase the next one10:16
manadartachilleasa: Yep. All good.10:18
flxfoowallyworld: ok I patch the script (for testing) and restarted the jujud agent process10:18
jamachilleasa, I haven't looked through the code yet, though I did read the coverletter.10:20
flxfoowallyworld: .... I added `export JUJU_ENV_FEATURE_FLAGS=image-metadata` before the line `'/var/lib/juju/tools/machine-0/jujud' machine --data-dir '/var/lib/juju' --machine-id 0 --debug`10:21
flxfoowallyworld: restarted the agent10:22
flxfoonothing new , same error (not implemented)10:22
flxfoowallyworld: ok got it10:23
flxfoosame variable name seem to be used both sides `JUJU_DEV_ENV_FEATURE_FLAGS`10:24
wallyworldyes :-)10:24
flxfoowallyworld: sorry, I misunderstood ...10:24
wallyworldsorry if i mistyped it10:24
flxfoo:D10:25
flxfoook, now I can see the custom imaged listed10:25
wallyworldawesome10:25
wallyworldnow you can delete image id records as needed10:25
flxfoobut only this one... do I have to use `image-metadata-url` with rackspace url to images?10:25
flxfooo ok sure10:25
wallyworldi thought the images were published at cloud-images.ubuntu.com, but i'd need to check10:26
flxfoowallyworld:  ok deleted10:26
wallyworldor if you have a place where the image metadata lives you can set the image-metadata-url10:26
wallyworldrakspace images are there i think10:27
wallyworldhttp://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:rax.json10:27
wallyworldso it should just work hopefully10:27
wallyworldwe really should update the feature flag on the controller so that setting it via controller config works10:28
wallyworldflxfoo: late here, i'm tired, so i might leave you to it. one of the other folks can help if needed10:28
flxfoowallyworld: can I help with that?10:28
wallyworldflxfoo: it's a simple fix, i can try and do it nex tweek10:29
wallyworldit's not something people really use a lot10:29
flxfoowallyworld: of course :p... you are far... I will try to get images back in... maybe using the api endpoint as well?10:29
wallyworldif the publisehd image metadata works, no need to set anythg else really10:29
wallyworld*published10:30
flxfoowallyworld: have a nice eve, keep you posted...10:30
wallyworldawesome, yeah, let me know how yu go10:30
flxfoowallyworld: sorry to ask last little one, should I update `image-metadata-url` or not? because right now the list-images is empty...10:31
wallyworldflxfoo: list images is only for custom metadata10:36
wallyworldthe metadata published at the default location as posted above should be sufficient10:36
wallyworldthe image-metadata-url is only for if you are hosting your own metadata somewhere and not using custom image metadata in the db10:37
wallyworldso there's 3 places image metadata can come from10:37
wallyworldyou only need to have it in one place10:37
wallyworld(usually the default place at cloud-images.ubuntu.com) works10:38
flxfoowallyworld: so I empty it ...10:40
flxfoowallyworld: ERROR matching cloud image metadata not found (not found)10:41
flxfoowallyworld: will look deeper, if I find something10:42
wallyworldflxfoo: that mean that what's been published doesn't match your cloud endpoint / region. you can look at the json metadata (index file and data file at cloud-images) to see what's been published10:48
flxfoowallyworld: ok thanks10:52
flxfoowallyworld: well the difference with the other providers in that `products` is empty for rax10:57
jamachilleasa, still reading through your very thorough QA steps. a couple questions11:21
jam1) do we know why we end up seeing double 'relation-changed' for back to back for things like wordpress monitoring mysql11:23
achilleasajam: not sure; I believe this is not related to my changes but it will be pretty easy to verify with develop HEAD; let me try it out11:26
jam2) Have we checked whether you can do relation-get/set --app during relation-created and/or leader-elected after this change?11:26
achilleasajam: 2) scenario 4 tests relation-set --app in relation-created (and relation-get after leader-elected). Do you have a different scenario in mind?11:28
jamachilleasa, I think it is because relation-joined forces a relation-changed event, and then mysql actually updates its contents after seeing wordpress.11:28
jamachilleasa, probably just haven't read that far yet.11:29
hmlachilleasa: i get tired on ! checks sometimes.  especially when it becomes a double negative, though not the case here.  will change.13:42
hmlachilleasa:  is there anything else on the PR?  what is left to move to approved?13:42
achilleasahml: just that. I will go ahead and approve so you can get it landed13:43
hmlachilleasa:  thank you.  i’ll make the change, then squash the commits. before merging13:46
achilleasarick_h_: I think a per-mddel limit is do-able with a little bit of effort although it will not be strict; a write by a unit might exceed the limit (slightly) but future writes will immediately fail unless the limit is raised. If that sounds good I will try to cobble together a PoC14:23
achilleasaBecause of the way that the total size will be calculated we can probably be even more granular and have a limit for charm state and a separate limit for uniter state14:26
rick_h_achilleasa:  right, that's the expectation I think is ok that the last one will succeeed and overwrite it14:34
rick_h_or overspend it I guess14:35
stickupkidmanadart, did you test the spaces with no VPC?14:48
manadartstickupkid: As in with a legacy AWS account? I have not done that.14:49
stickupkidmanadart, I just happened to bootstrap onto jams aws, and zero subnets show up14:49
manadartstickupkid: Then you don't get spaces :)14:49
achilleasarick_h_: so, while there isn't a clean way to get the stats, I discovered 'Object.bsonsize' which gives you the size of a doc. Next logical step is obviously to map-reduce and sum the sizes :D https://paste.ubuntu.com/p/trWRwCPfQ9/14:57
achilleasathe above snippet can be modified to aggregate sizes per-field so we get a charm vs non-charm state sum14:57
stickupkidmanadart, quick question, when moving the last subnet(s) to another space, shouldn't we clean or hide that space15:01
stickupkid?15:01
stickupkidmanadart, this looks funky15:02
stickupkidhttps://paste.ubuntu.com/p/DgZgjDfpjM/15:02
manadartstickupkid: No. The user can delete it if desired.15:02
stickupkidbooo15:02
stickupkidstick funky looking15:02
stickupkids/stick/still/15:02
stickupkidlovely it works really nicely15:05
rick_h_achilleasa:  well, do we need to? What is the scope of a single doc?15:36
rick_h_achilleasa:  a model level summary was just something that was possibly easy, if each doc is unit sized then we can just make the config a per-unit level?15:36
achilleasarick_h_: the idea is to calculate the total size for all state-docs for a particular model uuid and use it as an early cutoff for writes. However we still need to persist the total size and update it as part of our txn logic to avoid a race15:39
achilleasawhere N units try to update the state at the same time and each one things they are within the range but once you commit the txn you may be well above the limit15:40
=== narindergupta is now known as narinderguptamac
rick_h_achilleasa:  right but my point is that the idea of a model-level rule was arbitrary on our end15:47
rick_h_achilleasa:  so if unit-level is easier/makes more sense we can ignore the model level aggregation issue?15:47
achilleasarick_h_: can you join us in daily?15:56
rick_h_achilleasa:  omw15:57
achilleasarick_h_: per-unit limiting is easier to implement but does not guarantee an upper bound on the data stored per model (i.e. I can add an arbitrary number of units). It all depends on what we want to achieve. In my view per-model limiting makes more sense if we want to provide max-space guarantees16:31
mordkoffhi16:34
rick_h_achilleasa:  I think we just need a knob. A per-unit seems to make sense because it doesn't block up everyone but only limits the effect on a single unit16:52
rick_h_mordkoff:  howdy16:52
rick_h_achilleasa:  if you have time let's catch up or wait until tomorrow. I think if per-unit is easier then we should go down that road but we can go through the different merits16:55
achilleasarick_h_: hmmm... but I thought the requirement was to put a configurable upper bound on storage/model. If we are fine with a per-unit limit I can prototype that instead16:55
achilleasarick_h_: daily?16:55
rick_h_achilleasa:  just needs some limit. At the time I thought we'd have to do it model-wide due ot mongodb namespacing16:55
rick_h_achilleasa:  omw16:55
achilleasarick_h_: regarding the bundlechanges fix, I don't think it is possible to define an offer without having a valid application block in your bundle due to validation rules. I will leave the present-in-bundle AND controller logic or get an error in for the fix but I cannot come up with a test case to trigger the error17:29
flxfoohi all (again) :)17:33
flxfoojuts q quick one17:33
flxfooafter enabling the `image-metadata` feature, now in some other scripts (python) I have some errors with `unknown facade ImageMetadataManager`17:34
stickupkidflxfoo, pylibjuju?17:35
flxfooalong with KeyError: 'ImageMetadataManager'17:35
flxfoostickupkid: I suppose so... coming along with python3.6/site-packages/juju/client/connection.py17:36
flxfooI suppose that is pylibjuju yeah17:36
stickupkidflxfoo, so this have been fixed in the follow PR https://github.com/juju/python-libjuju/pull/382, but hasn't been released yet17:37
flxfooo17:37
flxfook17:37
stickupkidflxfoo, can you by anychange use the head of master17:37
stickupkid?17:37
stickupkidflxfoo, sorry for the breakages, but this was done to help ensure we didn't break pylibjuju when using different facade versions.17:38
flxfoostickupkid: well, I will keep that i mind... still I can not get the OS image list from rax... so maybe I will have to go back to the custom-image-metadata... not sure how everything will fit together honestly17:38
flxfoostickupkid: no pb... I think the situation with rackspace + some juju changes those days are putting the platform in an unstable position...17:39
stickupkidflxfoo, be mindful that pylibjuju doesn't know about custom-image-metadata, so you'll have to use juju CLI directly (shell out etc) to use that feature17:39
flxfoostickupkid: the thing is , before with the custom-image-metadata setting, the scripts works perfectly fine (tested one only, actually used for deployment)17:40
flxfooI start to think that we are too much coupled with the tools... then... for about a few minutes I could not even deploy anything (web app) nor scaling up/down the platform... that's pretty horrible situation17:41
flxfoo:(17:41
flxfooor we completely missed something on the way...17:42
flxfooI don't know if other people here are rackspace customers?17:42
stickupkidflxfoo, have you seen this https://discourse.jujucharms.com/t/juju-office-hours-2020-03-26/281917:42
stickupkidflxfoo, ;-)17:43
achilleasastickupkid: quick PR for you https://github.com/juju/bundlechanges/pull/6017:43
flxfoo stickupkid, interesting thanks17:44
flxfoostickupkid: thanks anyway :p ... my daughter needs attention... :)17:45
stickupkidachilleasa, "if we are abusing17:46
stickupkid// bundle deployments to do upgrades"17:46
stickupkidHAHA17:46
achilleasastickupkid: you know... for future code archeologists...17:46
stickupkidachilleasa, why this line? https://github.com/juju/bundlechanges/pull/60/files#diff-31a4a7824225629df071a1cba8b785d5R34617:47
stickupkidwhy not `var reqs []string`17:48
rick_h_flxfoo:  yea sorry, not many rackspace folks and they were killing off their public cloud last I was aware so not a lot of love in there17:48
achilleasastickupkid: just reuses the existing slice capacity between loop iterations; pretty much equivalent17:48
stickupkidachilleasa, pretty much hard to read17:49
achilleasastickupkid: np, let me change it to use a var17:49
flxfoostickupkid: definitely, we will probably move asap...17:49
achilleasahml: can you take a look at https://github.com/juju/juju/pull/11358 and land it if it works for you as I am hitting EOD? I will forward-port it to develop in the morn18:14
hmlachilleasa:  sure18:14
flxfoosorry to disturb again20:01
flxfooI would like the time being to use the custom-image-metadata with new values, I generated the files... how can I insert in the mongodb?20:02
flxfoomaybe `juju metadata add-image" ?20:04
flxfooif yes what should I put for storage-size and storage-type?20:04
thumperflxfoo: is this for openstack?20:28
* thumper is wondering where our docs are falling down20:28
thumperwell that was bad timing20:28
flxfooCan I bootstrap a running controller? (probably stupid question)20:59
thumperhi flxfoo, can you explain what you mean?21:01
thumperbootstrapping is the process of starting a controller21:01
thumperif you have one running, what are you trying to do?21:02
flxfoo@thumper: ok, that what I though... to use `metadata-source` again21:02
rick_h_flxfoo:  can you bring up a new controller?21:02
rick_h_flxfoo:  and use juju migrations to move your model over to the new one?21:02
flxfoorick_h_: wait wait21:03
thumperflxfoo: I asked this question just before you dropped from the channel before...21:03
flxfoo;)21:03
thumperflxfoo: is this for openstack?21:03
flxfooit is link to rackspace21:03
flxfoothey remove a os image21:03
flxfoothat my colleague bootsrapped our controller using that image21:04
flxfoonow21:04
rick_h_flxfoo:  so if you're able to bootstrap a fresh controller juju supports migrating models over https://discourse.jujucharms.com/t/migrating-models/115221:04
flxfoothat was some time ago, I was not there21:04
flxfoonow talking with wallyworld yesterday, we enable the `image-metadata` feature in order to remove the "custom" image...21:05
flxfoounfornutely back to "default" no rackspace os images appear to be available21:06
rick_h_flxfoo:  is this on their public cloud?21:06
flxfooso what I try to achieve is to use that `custom-image-metadata` with a new image id rackspace gave me21:06
flxfoorick_h_: guess so yes...21:06
flxfooI managed (about half an hour ago) to add an image vis `juju metadata add-aimge`21:07
flxfoobut when I use it (juju add-machine` juju loop creating instances , that I can see in rackspace *all* as building state stuck at 90%21:08
wallyworldflxfoo: so you got a new image id added and juju attempts to use that image to provision a machine but the provisioning fails? that sounds like an image / cloud issue that needs to be investigated21:21
flxfoowallyworld: hi there... well It is actually very strange21:30
flxfooso first as I said juju just looped saying that the machine is state BUILT, so tryinh again... and now I have 18 instances in rackspace in build state... :(21:31
flxfooand21:31
flxfooI tried to put some debu on... but then my image first disappear from list images21:31
flxfoowhen remove-machaine --force would that remove an image as well eventually?21:32
flxfooI am not getting what is going on...21:34
flxfooanyway...21:34
flxfoowallyworld: how could I set the `custom-image-metadata` with "new" values? is that possible?21:34
wallyworldflxfoo: it may be that the custom image metadata added via add-image has an expiry, i'd need to check. you may need to re-run add-image. i have no idea why the machine gets stuck in building. that seems like a cloud issue. in any case, you'll need to use the dashboard or something to inspect the state of the instance21:38
wallyworldit the cutom image metadata does have an expiry, we may be able to work around that, but the first issue is to see why the chosen image is not working21:39
flxfoowallyworld: but I did not set any expire time when using add-image. juju metadata add-image 498bfa38-d288-4e37-b975-a4acb77f8557 --series bionic --region LON --virt-type pvhvm21:41
wallyworldflxfoo: no, but juju may be setting an expiry, i'll need to check21:42
flxfoomaybe I should just put bionic and region21:42
flxfoowallyworld: a ok ok21:42
flxfoowallyworld: if it does , the expire time is pretty short21:42
wallyworld6 hours i think. but this stuff was done ages ago so i'm not sure21:43
flxfoowallyworld: the image disappear after a few minutes21:48
flxfoowallyworld: I added it already 4 times in the last couple of hours or so21:48
flxfoowallyworld: just disappear, I added it about 10 minutes ago21:49
flxfoowallyworld: storage-type what could be a valis value for that one in add-image?21:50
wallyworldflxfoo: storage-type is cloud dependent. i am not sure what's needed for rackspace. i think that was added more for aws. rackspace doesn't use it i don't think. i am not sure why the iamge metadata is not sticking until i get a chance to look. the fact that the image is being discovered and used to provision a machine but getting stuck is the first thing to solve. and that needs to be done using hte cloud tooling, dashboard,21:57
wallyworldssh into image etc21:57
flxfoowallyworld: right... did not find a way to get in yet... as "building" there is no obvious way to get in... but will dig into that thanks21:59
wallyworldflxfoo: good luck. once that is figured out we can look at the other things22:00
babbageclunkcool lxc petname: infinite-sailfish22:57
wallyworldcorona-prince would be a good one23:20

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!