[00:16] Gotta step away for the minutes... build looks to be going fine [00:19] externalreality: aye, just waiting for ppa publish so should be finished shortly :-) [00:19] veebers, ack [00:26] is anyone else seeing an issue with dns not working with juju instances, containers or otherwise? [00:27] scratch that, just containers inside of machines [00:27] this is so weird [00:28] how can it both work, work with ips, but not work? [00:28] elaboration, machine can ping dns and ip, containers can ping ip, but not dns [00:31] veebers, done [00:31] moving on to manual [00:47] fallenour: which provider? [00:48] fallenour: seems like the resolver in the container isn't set up right [00:48] thumper: maas [00:48] thumper: since u were in juju/errors, did u see there were some new PRs? [00:49] anastasiamac: that is why I was there [00:49] \o/ [00:49] thumper: wait, what do you mean the resolver isnt setup right? it should inherit it just like the machine does through dhcp like its host machine does via maas [00:50] thumper: do juju containers inherit dns settings differently? [00:50] fallenour: I'm not entirely sure, but there was a bug around containers on maas missing some search domains [00:50] I don't recall exactly [00:50] which versions of maas and juju? [00:51] thumper: 2.4.0-beta2 for maas, and 2.3.7 [00:51] system is an upgrade from xenial to bionic [00:52] when did the upgrade happen? [00:52] thumper: today o.o [00:53] technically yesterday [00:53] was the machine in a juju model before? [00:53] issues started yesterday as well [00:53] yea, it was juju before, its with the same maas system as before [00:53] the only thing I can think of is maybe 8.8.8.8 isnt a dns server? which would be insane [00:54] seems like a weird interaction between the upgrade and the containers [00:54] are the continers upgraded or just the host? [00:55] thumper: the whole thing is a rebuild [00:55] thumper: machine instances are bionic [00:56] the machine was rebuilt by maas rather than upgraded in place? [00:56] thumper: charm was initially the telemetry openstack deployment, then I switched to openstack-base after it kept failing [00:56] thumper: no, I wiped the entire controller environment, and started from a new controller with maas cloud [00:57] fallenour: unfortunately you have now exceeded my knowledge in this area [01:27] thumper: saadness [01:34] thumper: do you think it would be smarter if I added dns settings to the configurations manually [01:39] ok so now its getting even weirder, the containers have an ip configured in the resolv.conf file, and it resolves. now im just lost x...x [01:42] ok so I think I might have found it, but I needto understand how resolv.conf works better. can anyone explain to me what it means by search [01:47] ok, so I checked into that, figured out that I understand even less now x...x [01:49] thumper: one thing I did just notice is the /etc/apt/sources.list file is different between machine and container, not that that would have any bearing on host resolution. [01:57] fallenour: no, I don't think that would make any difference [01:58] thumper: ok so I did notice something odd. All the machines can resolve to just about any host I can think of via dns, with the exception of the maas system [01:59] fallenour: I think perhaps asking on #maas (assuming that channel exists) [02:13] wallyworld: I need your brain, I've introduced a json/yaml mismatch which is a pain https://github.com/juju/juju/blob/develop/core/resources/resources.go#L13 I think the easiest way forward would be to nicely ask charmstore team to alter the json presented in the api response? (or we can change the yaml side to be the camelcase bits) [02:21] kelvin__: query: I've juju deployed the k8s cluster, how would I use kubectrl to describe or log any of the pods/container details? [02:24] veebers, first, u need scp the kubeconfig file from master node to ur local at ~/.kube/config, then kubectl -n modelname logs podName [-f] or kubectl -n modelname describe podName [02:24] kelvin__: awesome, it was the ~/.kube/config part I was missing, thanks! [02:24] veebers, np [02:40] veebers: what's the charn store producing? "registrypath"? [02:41] wallyworld: no the json on, ImageName [02:41] it ultimately needs to produce a split repo/path [02:42] we might need to do a patch for that [02:43] vino: canoical-kubernetes bundle is significantly diofferent to the kubernetes-core bundle [02:43] wallyworld: ack, we could converge the json/yaml tags then [02:44] wallyworld: we should also consider upgrade testing with caas charms deployed [02:44] at some point yeah [02:44] 2.5.x upgrades work fine [02:45] wallyworld: i used canonical-kubernetes bundle [02:45] both gui and exportbundle [02:46] wallyworld: I know jam was looking at some 2.3 resource update transaction stuff which may prove interesting [02:46] ok, that's good because it adds more complexity compared to the normal bundle [02:46] veebers: interesting for what? [02:46] wallyworld: i do see issues while deploying the bundle created by export-bundle command. [02:46] k8s was only a soft laucnh in 2.4 [02:47] but the one exported gui gets deployed properly. [02:47] wallyworld: if we're doing resource update transactions wrong now, may mess with upgrades later. Although that being said I'm not sure if the bug jam was looking into was upgrade or just general missing transactions etc. [02:47] vino: that sounds like a bug to fix - the yaml should be identical, what's the difference? [02:48] There is additional bindings for each app which is not actually issues. [02:48] veebers: it was just a general crappy bug to do with writing resource changes [02:48] and not relevant here [02:48] wallyworld: ack, afaict a fix/something only exists in 2.3 [02:48] wallyworld: the machine constraints. [02:48] let me pastebin [02:49] ok, i'll need to see the yaml for each [02:49] if the gui export is differnet to the CLI one the CLI one needs to be fixed [02:52] thumper: ok so I found this: https://bugs.launchpad.net/juju/+bug/1764317 , and I pulled my package info for juju with dpkg -l, and now im totally confused. According to that, there was a fix release from the 10th, which says 2.4, but according to my package release for bionic, mine is up to date, but according to my dpkg output, im still on 2.3.7-16.04, which is a whole version behind. Why is that? [02:52] Bug #1764317: bionic LXD containers on bionic hosts get incorrect /etc/resolve.conf files

[02:52] wallyworld: the gui yaml is - https://pastebin.ubuntu.com/p/BdHVy3jdWX/ [02:53] and the cmdline generated : https://pastebin.ubuntu.com/p/Y2PTHBh7K4/ [02:54] machine constraints difference - the commandline is missing the additional machine details. [02:55] the way i get is different in my code. [02:56] vino: looks like the CLI one is missing the constraints [02:56] those will need to be added [02:57] wallyworld: yes. this is not case with other simple bundles. [02:58] i do see the in machines category generated by CLI is set for each application in the gui yaml file [02:58] depends if the bundle includes constrtaints or not. you need to look at the bundle data struct and ensure all necessary content is queired from state [02:58] ok. [02:58] its good we verified this way. [02:58] yes :-) [03:01] vino: the relations block looks different as well [03:01] thats what the error i do see. [03:01] when i am trying to deploy [03:01] there are 24 relations [03:03] all the entries are there which are found in gui [03:04] vino: the yaml looks diffenre though, notice the extra "-" [03:04] yes thats the difference i notice. [03:05] wallyworld: i am not sure what it means here. :( the error i am getting here is the bundle has additional endpoints. [03:06] vino: it does indeed have extra endpoints in it compared to gui. i'd need to see exact message [03:09] https://pastebin.ubuntu.com/p/qmpFk3FvFB/ [03:10] both has 24 [03:10] the ordering is different. [03:15] And also, the machine constraints are there in CLI but that resides in every app in the gui yaml. And ofcourse there are additonal mac constraints - This needs to fix as well. But i need to understand the difference between these 2 sets of machine constraints. [03:35] fallenour: 2.3.7 is the latest xenial release, but snaps are now the preferred way to get juju client updates [03:46] wallyworld: are we wanting to make the --resource fix and the use of a file for secrets one PR, or just fix for now, cloud/creds split then circle back to secrets inclusion? [03:47] wallyworld: this issue with relation is not just with kubernetes bundle. this issue i do see with mediawiki. [03:48] vino: sorry, was in meeting. that error in the pastebin is exactly because the relations are messed up in the CLI export [03:48] they should be a list of 2 item tuples [03:49] also, the machine hardware constraints from the provisoning info need to be exported to macth the gui output [03:49] veebers: i am happy to have the --resource fixes in the one pr [03:50] wallyworld: there are 2 sets of constraints. One in Application and other in machine itself. [03:50] wallyworld: ack, just looking at extending a test for the fix [03:50] CLI export is setting the Application constraints in machine struct. [03:51] vino: i think the gui javascript reads and exports the machine hardware provisioning info and adds that to each machine record (but you'd need to check the javascript) [03:51] but yes. Relations require fix. [03:51] also, for now we can leave off endpoint bindings [03:51] ok. [03:51] we just want gui output = cli output [03:51] to start with [03:52] there will be some small differences in annotations [03:52] but we should be able to replicate what the javascript produces [03:53] then we can start fixing the gaps, eg recording placement intent etc [03:57] wallyworld: i mis understood b/w he constraints in app and mac and implemented the logic for one for other. [03:57] i will fix both. [03:58] sounds good, thank you [03:58] not a lot to fix, just a few little tewaks [03:58] *tweaks [04:01] sure. I am able to understnad better only when i am testing these bundle. The ones i verified were easy and starightforward. Although i missed mediawiki. I did verify mediawiki single which worked perfectly. to take and deploy. [04:01] Iam sur that the ubuntu bundle issue i mentioned to u this morning cud be becuase of my ocde issue only. [04:01] not the bundle :( [04:02] could be. the main thing at the moment is to ensure the relations are expored correctly [04:03] as a list of (2 element) list. each item is a list of 2 endpoints [04:12] yup got it. [04:41] wallyworld: FYI https://github.com/juju/juju/pull/8979 (when you have a moment) [04:45] wallyworld: oh, I'll fix up https://github.com/juju/juju/pull/8968 as per your comments then hit the cloud/creds bits [04:52] thumper: Should I land https://github.com/juju/juju/pull/8954, or do you want me to hold off? [05:02] veebers: done [05:05] * thumper looks at babbageclunk's pr [05:05] thumper: thanks! [05:06] wallyworld: I may have misunderstood you earlier, you're keen for the --resource fix + the addition to take in a file path in this PR or hold off on the file path and just have the imagepath string support that currently exists? [05:10] babbageclunk: land away [05:11] babbageclunk: I only had one comment about error types [05:11] but they could be changed later [05:11] thumper: ok, checking now [05:11] I just landed a change to juju/errors from an external contributer that added a timeout type error [05:11] and we already have invalid types [05:12] veebers: since it's your EOD we can land as is and follow up tomorrow, will only be a small addition [05:12] wallyworld: ok, I'll tweak as per PR comments [05:13] thumper: ah - ok, so it's a historical quirk that core/lease has its own ErrInvalid? Yeah, I'll harmonise those at some point in the future. [05:13] * thumper nods [05:13] babbageclunk: also... [05:13] I'd like to start moving loggers into the worker objects themselves [05:13] rather than package level loggers [05:14] to facilitate getting the logs into the right model's logs [05:14] thumper: oh, yes - that's something I've been doing with my own new workers [05:14] thumper: but I didn't do that with the lease manager. [05:14] that's fine, it was like that befor [05:14] cool, I'll make a note of that too. [05:14] no doubt we will do a pass through at some stage once we have things configured properly [05:15] yeah, I'd say so [05:19] oh, come on! my merge failed with `fatal: unable to access 'https://github.com/alecthomas/gometalinter/': Could not resolve host: github.com` [05:20] babbageclunk: hit up IS, it's not the first time we've had odd access issues like that :-| [05:22] I mean, it resolved it plenty of times to get all the source code, right. [05:25] babbageclunk: right, seems like a really odd (and annoying) issue. [05:25] veebers: I'm just retrying the merge before bothering IS [05:25] I wonder if it's something obsure as a bad port on a machine or switch, although I imagine something like that might be self-diagnosable? [05:29] * babbageclunk shrugs [05:44] wallyworld: if a docker resource excludes a registry, should we just assume it's docker.io or is that assuming too much? (we currently assume it is that and add it). It means the [05:44] * veebers actually thinks when that might actually happen. [05:45] veebers: the behaviour should be the same as docker pull etc; docker.io is the default IIANM [05:45] It'll only happen when someone uses --resource I think, if I attach a publicly available image to a charm it'll be a registry.charmstore. . . [05:46] it will happen if the user overrides the resource yes [05:46] we can leave it empty [05:46] and let k8s/docker do its thing [05:46] that would be best [05:46] wallyworld: I think blank would be best [05:46] wallyworld: hah aye, what you said :-) [05:46] yep [05:47] * veebers makes the change (and fixes a crap test he wrote) === frankban|afk is now known as frankban [10:39] wallyworld: are u still ard ? [10:40] a bit [10:41] :) [10:41] ok. [10:42] i wud like to discuss abt the comment u have provided. "provisioned machine hardware". [10:43] my understanding is it is done for now. parity with gui [10:43] there's a method on machine to get it i think. my understanding is that's what the gui javascript does [10:44] HardwareCharacteristics() or something [10:46] Aint i doing the same. [10:46] ? [10:46] i am addressing that in the bundle.go constarints.go [10:46] arch/cpucore/cpupower/mem [10:47] I always wanted to ask abt another aspect : availabilityZone which currently i am not addressing. [10:47] Plus, cud u please suggest me some more complex bundles for manual verification. [10:47] currently the kube is in parity with gui. [10:49] I have provided the pastebin link for both in the PR [11:23] vino: sorry, something came up, will look tomorrow [11:47] is anyone aware of what the latest up date version of juju is, the snap version? [11:51] fallenour_: should be 2.4.1 hopefully. It was in the process of being released yesterday. /me checks with snap info juju [11:52] woot! stable: 2.4.1 [11:53] rick_h_: ok, so thats a concern for me then. According to both snap refresh, as well as dist-upgrade, mine is the latest, and thats at 2.3.7, which has issues. [11:54] rick_h_: I need to move to 2.4 or greater because of the current maas issue with bionic [11:55] fallenour_: ? snap refresh juju gets you 2.3.7? what does the output of snap info juju show you? [11:55] fallenour_: maybe you need to make sure to track the stable channel? [11:55] rick_h_: snap "juju" has no updates available [11:56] rick_h_: if Im not mistaken, I am tracking stable channel. [11:56] fallenour_: right, so I want to make sure snap info juju shows 2.4.1. In fact I bet you haveboth installed? [11:56] fallenour_: what does "which juju" show you? [11:56] and what does "/snap/bin/juju --version" do [11:57] rick_h_: ok, so this is gonna be fun [11:57] it says I have 2.4.1 installed. dkpg says I have 2.3.7 installed [11:57] snap bin says I have 2.4.1 installed [11:57] now, my brain is the sad [12:04] rick_h_: ok, so assuming juju is working just fine. That means maas isnt working. I assume their team will fix it eventually, but I dontwant to wait for them to fix the problem. DNS names arent resolving, /etc/resolv.conf is the best way to do this? And if so, would simply explicitly naming the name server should fix the issue with resolving to the package repos, which is currently hindering all of my charm installs [12:05] fallenour_: so the key thing is you need to bootstrap using the /snap/bin/juju so that the controller is 2.4 which has bionic fixes [12:06] rick_h_: is that sound logic? and if so, sed -i to deliver the changes via juju run machines --all sed -i "nameserver " /etc/resolv.conf [12:07] rick_h_: to rebootstrap? Im confused [12:07] re-bootstrap* [12:26] bdx: kwmonroe tvansteenburgh jamespage cory_fu and anyone else. First spec on discourse for your lxd profile enjoyment. Please feel free to read/comment/etc. https://discourse.jujucharms.com/t/wip-specification-for-lxd-profile-updates-permitted-by-charms/78 === blahdeblah_ is now known as blahdeblah === Guest14217 is now known as zeus === frankban is now known as frankban|afk [20:30] Morning o/ [20:45] So with lxc clustering, can I setup a machine in my basement as a 'cluster' and bootstrap to it from my laptop (from the comfort of my office)? [20:50] veebers: i think so, i’ve been told that there isn’t a difference between remote and cluster lxc [20:51] nice, that would free up my laptop a wee bit [20:51] veebers: i was going to try with one of the nucs on a maas later [20:52] veebers: or tomorrow [20:52] veebers: i’ve been told that the video linked to the discourse on lxc cluster is amazing at explaining [20:57] hml oh I missed that, I'll take a look thanks! [20:58] hah, I should have read the post that I opened but hadn't read yet ^_^ [20:58] :-D [21:27] veebers: yes you can. You can have a one node cluster and do that for sure [21:27] veebers: let us know how it goes success/fail in that discourse post to generate conversation [21:28] rick_h_: very nice, I might try set something up this weekend if I can find a moment [21:28] rick_h_: will do, it won't be with maas though, just a machine with ubuntu installed on it [21:31] veebers: yea, maas is just a handy way to have some extra machine around [21:31] veebers: so the only issue you'll have to make sure is to manually run the lxc bridge setup command in that doc [21:31] veebers: after that it's an add-cloud, add-credential, bootstrap [21:33] rick_h_: sweet, can't wait to deploy the k8s bundle to it and free up my laptop :-) [21:48] why do you call it a one-node cluster? juju is just connecting to a remote LXD host [22:19] huh, true :-) [22:22] if I setup clustering from the get-go then I can start adding hardware as I retire it from general use (laptops etc.) [23:06] wallyworld: are there some good examples of k8s configs that I could take a look at? want to confirm my understandings [23:07] what do you mean by configs? [23:07] wallyworld: i.e. it's possible that a config will define a couple of clusters, and a user and that user has access to all those clusters? [23:07] wallyworld: i.e. what add-k8s parses [23:07] i don't have any aprt from what we get by deploying cdk [23:07] well... there we go [23:08] mailing lists are closed [23:09] thumper: end of an era! [23:09] wallyworld: ack, does what I say make sense? (re: users -> clusters)? [23:10] I think so. i think a config can define more than one cluster. not sure about user access [23:12] anastasiamac: that it is [23:12] all hail our new discourse overlord [23:50] wallyworld: we might have a caas issue where you can't teardown if a deploy fails. I'll dig in and come up with some actually useful data, just FYI at this stage :-) [23:50] veebers: that's an issue for iaas as well [23:50] wallyworld: ah ok [23:58] wallyworld: unless I've got the wrong end of the stick I don't think that we can just go through and add a cluster endpoint + server to each user cred, the config field context defines what cluster a user has access too, so looks like we'll need to parse the cluster details and conditionally create a user cred for that cluster based on the context