pmatulisec0, can't you use an overlay?00:18
ec0would overlays still require stub applications to be present for relations to applications which exist in the model, which do not exist in the overlay?01:12
pmatulissearch for 'rabbit' here: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-manila-ganesha.html01:42
wallyworldkelvinliu: last core network piece https://github.com/juju/juju/pull/1201104:21
kelvinliuwallyworld: lgtm ty04:37
kelvinliuwallyworld: tlm hpidcock  any of u free to take a look this pr for adding service updating stuff in broker level plz  https://github.com/juju/juju/pull/12013  thank you06:19
hpidcockkelvinliu: on it06:19
kelvinliuthis one is the 1/3 split from the big PR for svc stuff06:20
hpidcockkelvinliu: looks good, just one comment about container ports06:29
kelvinliuhpidcock: tyrv06:40
kelvinliucurrently we define the svc ports in container in podspec, which means we always expose the pod in cluster if there is svc06:41
hpidcockkelvinliu: "List of ports to expose from the container. Exposing a port here gives the system additional information about the network connections a container uses, but is primarily informational. Not specifying a port here DOES NOT prevent that port from being exposed. Any port which is listening on the default "" address inside a container will be accessible from the network."06:44
hpidcockonly time I think it's used is when the podspec has hostNetworking set to true "Host networking requested for this pod. Use the host's network namespace. If this option is set, the ports that will be used must be specified."06:47
kelvinliuand it's bad to update podspec when we have a port mapping change as well06:53
kelvinliuhpidcock: wallyworld tlm could anyone free to take a look this pr for adding facades for the new caasfirewaller worker plz https://github.com/juju/juju/pull/12014  thanks07:09
achilleasastickupkid: got two small PRs for you: https://github.com/juju/juju/pull/12005 and https://github.com/juju/bundlechanges/pull/6808:00
stickupkidflxfoo, you around?08:54
achilleasastickupkid: we seem to be using dashes for other changes (e.g. RelationsDiff) so I will make them all use dashes09:53
achilleasastickupkid: pushed a commit to rename num_units; can you take another look?09:56
achilleasastickupkid: ok, reverted last commit and changed the dash to _ (retained it for the ExposedEndpointDiff struct fields though since we use the same pattern for other structs)11:25
stickupkidyeah tick11:25
jammanadart, I did a bit of digging on https://bugs.launchpad.net/juju/+bug/189595411:26
mupBug #1895954: SEGV when jujud restarts after upgrade-controller 2.8.1 -> 2.8.2 <juju:In Progress by manadart> <https://launchpad.net/bugs/1895954>11:26
jama quick fix would just be to trap for charmDoc.Meta is nil11:26
jamI think11:26
manadartjam: Ack.11:27
stickupkidjam, that doc is a requirement11:27
stickupkidjam, we should pretty much die there11:27
jamstickupkid, we have a clear database that failed with a panic nil dereference during upgrade11:27
jamstickupkid, failing during upgrade is *not* the place to fail, because then you can't get out of the failure11:28
stickupkidsure sure, ok fair11:30
jammanadart, a thought. instead of trying to get him to 2.8.1 just get his charm.meta data fixd12:01
jamand then get a patch into 2.8.312:01
manadartjam: Yes, pursuing this line presently.12:27
routergodhi! looking for help with [Bug 1895954] Re: SEGV when jujud restarts after upgrade-controller 2.8.1 -> 2.8.212:29
mupBug #1895954: SEGV when jujud restarts after upgrade-controller 2.8.1 -> 2.8.2 <juju:In Progress by manadart> <https://launchpad.net/bugs/1895954>12:29
routergodthat is me12:30
manadartroutergod: Hey. Need any help with running the query I posted in Discourse?12:30
routergodah no I missed that...12:31
routergodoh that produces a lot of docs (one per application?)12:38
routergodall have         "meta" : null,12:38
manadartCan give me a couple of examples of charms that you have deployed - just the names will do.12:40
routergodthere are three models in this controller, this all seems to relate to one of them12:45
routergodit is one i didn't touch for a while - an early focal-ussuri openstack-base12:46
manadartroutergod: Indeed. Is that model (0ea37312-4df6-49a6-8137-f6b872def5dd) functional?12:54
manadartIn any case, I think running this will allow the upgrade to proceed: `db.charms.update({meta: null}, { $set: {"meta": {}} }, false, true)`13:00
routergodthe model machines are running and they look normal. i don't care about that model though.13:03
routergodlet me try that, it will only impact that one model either way13:03
jamroutergod, can you confirm if you have a significant number of entries that *aren't* null? I'm wondering if "the things that are working" are the ones with valid data, and the "things that are nil" are just cruft that we failed to prune correctly.13:09
routergodhow do i say !null to mongo?13:10
routergod@mandart sorry i am a mongo idiot. are the backticks part of the command?13:12
jamroutergod, {$exists: true}13:12
manadartroutergod: No, just for code formatting :)13:12
jamah sorry, {$ne: null}13:12
routergodthat explains something... :-)13:12
routergodok it is doing something other than crashing now13:14
jamroutergod, do you feel comfortable giving just all the contents of that collection? We have the feeling that these are charms that you started deploying, but it failed to get from the store, and then you would have a later charm that was properly populated13:14
jamand thus these could just be removed from the DB13:14
routergodyes I can share just tell me the command please13:15
jamroutergod, ^13:15
jamIf we want filtered, then it would be: db.charms.find({meta: {$ne: null}, "model-uuid" : "0ea37312-4df6-49a6-8137-f6b872def5dd"}).pretty()13:16
soumplis_manadart: Just tried your suggestion and it seems that the controllers come up again13:19
manadartsoumplis_: Great news. We have a fix, but are still strategising around re-release/pull.13:20
routergoddo you want it filtered? it is quite large13:21
routergodnah i'll post the lot :-)13:21
routergodbrilliant, controller seems back up here now13:26
routergodgetting a lot of this in the machine log though;13:26
routergod2020-09-17 13:24:58 ERROR juju.core.raftlease store.go:265 timeout waiting for Command(ver: 1, op: extend, ns: application-leadership, model: 0ea373, lease: glance, holder: glance/0) to be processed13:26
routergod`routergod@juju:~/.local/share/juju$ juju switch controller13:29
routergod14:28:26+01:00  upgrade in progress since "2020-09-17T13:13:43Z"13:29
routergodhow to sync the other databases? do I just repeat that command there?13:30
manadartroutergod: That will have happened automatically. Writes will have been redirected to the primary, and the replicas sync'd.13:31
manadartControllers being down should not affect that.13:31
routergodyeah that what i expected. but the other two are like this in the machine log13:34
routergod`2020-09-17 13:31:26 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [b5fb8e] "machine-2" cannot open api: unable to connect to API: dial tcp connect: connection refused`13:34
routergodlooks like the upgrade is complete on one controller but not the other two13:34
jammanadart, https://github.com/juju/juju/pull/1201613:38
routergodmanadart if I repeat your original mongo foo on one of the 'down' controllers i get this;13:40
routergod`juju:SECONDARY> db.charms.find({meta: null}).pretty()13:40
routergod"signature" : {13:40
manadartjam: Commented on it.13:44
jamroutergod, it looks like you got rate limited13:44
routergodoh :-(13:48
routergodmanadart I tried your mongo stanza on one of the 'down' controllers and got an "errmsg" : "not master and slaveOk=false"13:50
routergodjujud is not starting on those two13:50
manadartroutergod: What is in /var/log/juju/machine-x.log on those?13:51
routergod2020-09-17 13:31:26 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [b5fb8e] "machine-2" cannot open api: unable to connect to API: dial tcp connect: connection refused13:51
routergodmanadart 2020-09-17 13:31:26 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [b5fb8e] "machine-2" cannot open api: unable to connect to API: dial tcp connect: connection refused13:52
jammanadart, updated PR13:56
soumplis_routergod: had the same issue with mongo. try to stop all but one mongos, and then start them again. this seems to let me execute the command13:57
jamroutergod, there should be an earlier failure about the apiserver not starting (that is the other side saying it is failing to talk to itself). We could also look at "juju_engine_report" but that would need pastebin13:58
soumplis_routergod: after you apply the mongo "hack" you have to restart jujud-machineX13:58
routergodsoumplis_ many thanks that works after restart. manadart , others many thanks for your help!14:13
jamroutergod, your second paste of "everything" doesn't have the ones that the first past with "meta: null" contained14:16
routergodjam err.. ?14:20
routergodjam I can't repeat that original because we changed the values to something else, but I will search for the ids or something...14:24
jamhttps://pastebin.ubuntu.com/p/j6JJ39Q4WN/ has "0ea37312-4df6-49a6-8137-f6b872def5dd:cs:ntp-41" but https://pastebin.ubuntu.com/p/ydyp3smGMY/ doesn't seem to have the string "ntp-41" anywhere14:25
jamroutergod, it *does* have ntp-39 with valid metadata14:25
stickupkidmanadart, so this doesn't fix the potential issue with the model cache, but does fix pylibjuju https://github.com/juju/juju/pull/1201714:26
routergodjam did I screw up the paste?14:39
jamroutergod, ok, I see ntp-41 in your latest paste14:40
routergodjam that just that specific model uuid14:40
jamsure. I see a ntp-39 that is all happy, and a ntp-41 which is still a placeholder14:41
routergodjam juju status show model 0ea37312-4df6-49a6-8137-f6b872def5dd running ntp-3914:45
stickupkidmanadart, also this fixes the issue where we have a custom value in application status https://github.com/juju/python-libjuju/pull/44414:46
stickupkidmanadart, walk through it quickly, before you EOD?14:46
manadartstickupkid: OK. Tally-ho.14:47
jamroutergod, sounds like an "juju upgrade-charm" that never completed because it couldn't get the charm from the store.14:48
routergodjam i will try upgrade-charm in that model to see what happens14:49
jamThere does appear to be an NTP 41 available: https://jaas.ai/ntp14:49
routergodyeah i am running that everywhere else14:49
hmlstickupkid:  do you have time to review 12008 ad 12009?14:54
jampetevg, would you be able to move the existing bugs targeting 2.8.3 to a 2.8.4 milestone except for the one we are doing the release for/14:57
jamroutergod, can you confirm that it gets to ntp-41 correctly?14:58
petevg@jam: ha! That sounds fun. Sure. I'll go ahead and do that.14:58
jammanadart, how comfortable are you with suggesting the db.update() workaround that you pasted above?15:01
routergod@jam i started with upgrade-model there because it was on 2.8.0. this has not completed, i will try to find what is wrong...15:01
routergodthere is one unit in blocked in the model15:02
routergodactually that might be it15:02
jamroutergod, that shouldn't prevent a model upgrade (IIRC)15:02
petevgjam: now I just have to remember how to create a milestone in launchpad. It's more annoying/obscure than it should be.15:02
jampetevg, I start here: https://launchpad.net/juju/15:03
jamclick on the 2.8 series link in the "Series and milestones"15:03
jamthen on https://launchpad.net/juju/2.815:03
jamCreate milestone15:03
jam@petevg, but we shouldn't close the 2.8.3 milestone (if you did)15:04
jamSince I still need to target the SEGV bug to 2.8.315:04
jamah, maybe you renamed 2.8.3 to 2.8.4 and are creating a  new 2.8.3 that would work15:05
manadartjam: Reasonably :) It seems to me such records are generally going to be orphans.15:05
petevgjam: that is exactly what I'm doing :-)15:05
manadartIf they were place-holders awaiting fulfilment, I assume you'd have other things in flux preventing an upgrade.15:05
routergodjam where do i look for logs about an upgrade-model command please?15:12
jamroutergod, I would look at machine-X.log on the controllers, or at least start there15:20
stickupkidhml, looking now15:21
jamcan someone validate my proposed steps: https://bugs.launchpad.net/juju/+bug/1895954/comments/715:22
mupBug #1895954: SEGV when jujud restarts after upgrade-controller 2.8.1 -> 2.8.2 <juju:In Progress by jameinel> <juju 2.8:In Progress by jameinel> <https://launchpad.net/bugs/1895954>15:22
jamI have done them here, but I want to make sure it isn't JustMe15:22
jamI also added a comment about stop/start of the controllers15:23
stickupkidjam, workaround is good15:34
jamstickupkid, can you try actually copy & paste the commands with a test environ?15:42
stickupkidjam, I can confirm, I inserted a null meta charm and ran the commands15:48
stickupkidjam, WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })15:48
jamstickupkid, thanks15:48
routergodjam I had neglected to restart jujud on that one controller. After restart the upgrade-model works15:48
jamroutergod, yay. ok, so can you upgrade the charm as wel?15:48
routergodjam upgrade-charm ntp seemed to work ok, now on ntp-4115:50
routergodmany thanks!15:50
jamhml, petevg I have kicked off $$merge$$ and need to run some errands for the family. I will be back to check on it. I will be available on Telegram/Mattermost if it is urgent.15:59
petevgjam: sounds good. I'm gonna step away for lunch, and plan on starting a release when I get back.16:00
jampetevg, looks like it merged successfully, so ab69570b38fbc746e54184e4c3274612bcbb8327 should be the hash16:52
petevgjam: cool cool. I'm gonna start the release process.17:09

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!