externalrealityGotta step away for the minutes... build looks to be going fine00:16
veebersexternalreality: aye, just waiting for ppa publish so should be finished shortly :-)00:19
externalrealityveebers, ack00:19
fallenour_is anyone else seeing an issue with dns not working with juju instances, containers or otherwise?00:26
fallenour_scratch that, just containers inside of machines00:27
fallenour_this is so weird00:27
fallenour_how can it both work, work with ips, but not work?00:28
fallenour_elaboration, machine can ping dns and ip, containers can ping ip, but not dns00:28
externalrealityveebers, done00:31
externalrealitymoving on to manual00:31
thumperfallenour: which provider?00:47
thumperfallenour: seems like the resolver in the container isn't set up right00:48
fallenour_thumper: maas00:48
anastasiamacthumper: since u were in juju/errors, did u see there were some new PRs?00:48
thumperanastasiamac: that is why I was there00:49
fallenour_thumper: wait, what do you mean the resolver isnt setup right? it should inherit it just like the machine does through dhcp like its host machine does via maas00:49
fallenour_thumper: do juju containers inherit dns settings differently?00:50
thumperfallenour: I'm not entirely sure, but there was a bug around containers on maas missing some search domains00:50
thumperI don't recall exactly00:50
thumperwhich versions of maas and juju?00:50
fallenour_thumper: 2.4.0-beta2 for maas, and 2.3.700:51
fallenour_system is an upgrade from xenial to bionic00:51
thumperwhen did the upgrade happen?00:52
fallenour_thumper: today o.o00:52
fallenour_technically yesterday00:53
thumperwas the machine in a juju model before?00:53
fallenour_issues started yesterday as well00:53
fallenour_yea, it was juju before, its with the same maas system as before00:53
fallenour_the only thing I can think of is maybe isnt a dns server? which would be insane00:53
thumperseems like a weird interaction between the upgrade and the containers00:54
thumperare the continers upgraded or just the host?00:54
fallenour_thumper: the whole thing is a rebuild00:55
fallenour_thumper: machine instances are bionic00:55
thumperthe machine was rebuilt by maas rather than upgraded in place?00:56
fallenour_thumper: charm was initially the telemetry openstack deployment, then I switched to openstack-base after it kept failing00:56
fallenour_thumper: no, I wiped the entire controller environment, and started from a new controller with maas cloud00:56
thumperfallenour: unfortunately you have now exceeded my knowledge in this area00:57
fallenour_thumper: saadness01:27
fallenour_thumper: do you think it would be smarter if I added dns settings to the configurations manually01:34
fallenour_ok so now its getting even weirder, the containers have an ip configured in the resolv.conf file, and it resolves. now im just lost x...x01:39
fallenour_ok so I think I might have found it, but I needto understand how resolv.conf works better. can anyone explain to me what it means by search <domain name> <systemname>01:42
fallenour_ok, so I checked into that, figured out that I understand even less now x...x01:47
fallenour_thumper: one thing I did just notice is the /etc/apt/sources.list file is different between machine and container, not that that would have any bearing on host resolution.01:49
thumperfallenour: no, I don't think that would make any difference01:57
fallenour_thumper: ok so I did notice something odd. All the machines can resolve to just about any host I can think of via dns, with the exception of the maas system01:58
thumperfallenour: I think perhaps asking on #maas (assuming that channel exists)01:59
veeberswallyworld: I need your brain, I've introduced a json/yaml mismatch which is a pain https://github.com/juju/juju/blob/develop/core/resources/resources.go#L13 I think the easiest way forward would be to nicely ask charmstore team to alter the json presented in the api response? (or we can change the yaml side to  be the camelcase bits)02:13
veeberskelvin__: query: I've juju deployed the k8s cluster, how would I use kubectrl to describe or log any of the pods/container details?02:21
kelvin__veebers, first, u need scp the kubeconfig file from master node to ur local at ~/.kube/config, then kubectl -n modelname logs podName [-f]  or kubectl -n modelname describe podName02:24
veeberskelvin__: awesome, it was the ~/.kube/config part I was missing, thanks!02:24
kelvin__veebers, np02:24
wallyworldveebers: what's the charn store producing? "registrypath"?02:40
veeberswallyworld: no the json on, ImageName02:41
wallyworldit ultimately needs to produce  a split repo/path02:41
wallyworldwe might need to do a patch for that02:42
wallyworldvino: canoical-kubernetes bundle is significantly diofferent to the kubernetes-core bundle02:43
veeberswallyworld: ack, we could converge the json/yaml tags then02:43
veeberswallyworld: we should also consider upgrade testing with caas charms deployed02:44
wallyworldat some point yeah02:44
wallyworld2.5.x upgrades work fine02:44
vinowallyworld: i used canonical-kubernetes bundle02:45
vinoboth gui and exportbundle02:45
veeberswallyworld: I know jam was looking at some 2.3 resource update transaction stuff which may prove interesting02:46
wallyworldok, that's good because it adds more complexity compared to the normal bundle02:46
wallyworldveebers: interesting for what?02:46
vinowallyworld: i do see issues while deploying the bundle created by export-bundle command.02:46
wallyworldk8s was only a soft laucnh in 2.402:46
vinobut the one exported gui gets deployed properly.02:47
veeberswallyworld:  if we're doing resource update transactions wrong now, may mess with upgrades later. Although that being said I'm not sure if the bug jam was looking into was upgrade or just general missing transactions etc.02:47
wallyworldvino:  that sounds like a bug to fix - the yaml should be identical, what's the difference?02:47
vinoThere is additional bindings for each app which is not actually issues.02:48
wallyworldveebers: it was just a general crappy bug to do with writing resource changes02:48
wallyworldand not relevant here02:48
veeberswallyworld: ack, afaict a fix/something only exists in 2.302:48
vinowallyworld: the machine constraints.02:48
vinolet me pastebin02:48
wallyworldok, i'll need to see the yaml for each02:49
wallyworldif the gui export is differnet to the CLI one the CLI one needs to be fixed02:49
fallenour_thumper: ok so I found this: https://bugs.launchpad.net/juju/+bug/1764317 , and I pulled my package info for juju with dpkg -l, and now im totally confused. According to that, there was a fix release from the 10th, which says 2.4, but according to my package release for bionic, mine is up to date, but according to my dpkg output, im still on 2.3.7-16.04, which is a whole version behind. Why is that?02:52
mupBug #1764317: bionic LXD containers on bionic hosts get incorrect /etc/resolve.conf files <bionic> <cdo-qa> <cdo-qa-blocker> <foundations-engine> <kvm> <lxd> <network> <uosci> <juju:Fix Released by ecjones> <juju 2.3:Fix Released by ecjones> <https://launchpad.net/bugs/1764317>02:52
vinowallyworld: the gui yaml is - https://pastebin.ubuntu.com/p/BdHVy3jdWX/02:52
vinoand the cmdline generated : https://pastebin.ubuntu.com/p/Y2PTHBh7K4/02:53
vinomachine constraints difference - the commandline is missing the additional machine details.02:54
vinothe way i get is different in my code.02:55
wallyworldvino: looks like the CLI one is missing the constraints02:56
wallyworldthose will need to be added02:56
vinowallyworld: yes. this is not case with other simple bundles.02:57
vinoi do see the in machines category generated by CLI is set for each application in the gui yaml file02:58
wallyworlddepends if the bundle includes constrtaints or not. you need to look at the bundle data struct and ensure all necessary content is queired from state02:58
vinoits good we verified this way.02:58
wallyworldyes :-)02:58
wallyworldvino: the relations block looks different as well03:01
vinothats what the error i do see.03:01
vinowhen i am trying to deploy03:01
vinothere are 24 relations03:01
vinoall the entries are there which are found in gui03:03
wallyworldvino: the yaml looks diffenre though, notice the extra "-"03:04
vinoyes thats the difference i notice.03:04
vinowallyworld: i am not sure what it means here. :( the error i am getting here is the bundle has additional endpoints.03:05
wallyworldvino: it does indeed have extra endpoints in it compared to gui. i'd need to see exact message03:06
vinoboth has 2403:10
vinothe ordering is different.03:10
vinoAnd also, the machine constraints are there in CLI but that resides in every app in the gui yaml. And ofcourse there are additonal mac constraints - This needs to fix as well. But i need to understand the difference between these 2 sets of machine constraints.03:15
thumperfallenour: 2.3.7 is the latest xenial release, but snaps are now the preferred way to get juju client updates03:35
veeberswallyworld: are we wanting to make the --resource <docker image> fix and the use of a file for secrets one PR, or just fix for now, cloud/creds split then circle back to secrets inclusion?03:46
vinowallyworld: this issue with relation is not just with kubernetes bundle. this issue i do see with mediawiki.03:47
wallyworldvino: sorry, was in meeting. that error in the pastebin is exactly because the relations are messed up in the CLI export03:48
wallyworldthey should be a list of 2 item tuples03:48
wallyworldalso, the machine hardware constraints from the provisoning info need to be exported to macth the gui output03:49
wallyworldveebers: i am happy to have the --resource fixes in the one pr03:49
vinowallyworld: there are 2 sets of constraints. One in Application and other in machine itself.03:50
veeberswallyworld: ack, just looking at extending a test for the fix03:50
vinoCLI export is setting the Application constraints in machine struct.03:50
wallyworldvino: i think the gui javascript reads and exports the machine hardware provisioning info and adds that to each machine record (but you'd need to check the javascript)03:51
vinobut yes. Relations require fix.03:51
wallyworldalso, for now we can leave off endpoint bindings03:51
wallyworldwe just want gui output = cli output03:51
wallyworldto start with03:51
wallyworldthere will be some small differences in annotations03:52
wallyworldbut we should be able to replicate what the javascript produces03:52
wallyworldthen we can start fixing the gaps, eg recording placement intent etc03:53
vinowallyworld: i mis understood b/w he constraints in app and mac and implemented the logic for one for other.03:57
vinoi will fix both.03:57
wallyworldsounds good, thank you03:58
wallyworldnot a lot to fix, just a few little tewaks03:58
vinosure. I am able to understnad better only when i am testing these bundle. The ones i verified were easy and starightforward. Although i missed mediawiki. I did verify mediawiki single which worked perfectly. to take and deploy.04:01
vinoIam sur that the ubuntu bundle issue i mentioned to u this morning cud be becuase of my ocde issue only.04:01
vinonot the bundle :(04:01
wallyworldcould be. the main thing at the moment is to ensure the relations are expored correctly04:02
wallyworldas a list of (2 element) list. each item is a list of 2 endpoints04:03
vinoyup got it.04:12
veeberswallyworld: FYI https://github.com/juju/juju/pull/8979 (when you have a moment)04:41
veeberswallyworld: oh, I'll fix up https://github.com/juju/juju/pull/8968 as per your comments then hit the cloud/creds bits04:45
babbageclunkthumper: Should I land https://github.com/juju/juju/pull/8954, or do you want me to hold off?04:52
wallyworldveebers: done05:02
* thumper looks at babbageclunk's pr05:05
babbageclunkthumper: thanks!05:05
veeberswallyworld: I may have misunderstood you earlier, you're keen for the --resource fix + the addition to take in a file path in this PR or hold off on the file path and just have the imagepath string support that currently exists?05:06
thumperbabbageclunk: land away05:10
thumperbabbageclunk: I only had one comment about error types05:11
thumperbut they could be changed later05:11
babbageclunkthumper: ok, checking now05:11
thumperI just landed a change to juju/errors from an external contributer that added a timeout type error05:11
thumperand we already have invalid types05:11
wallyworldveebers: since it's your EOD we can land as is and follow up tomorrow, will only be a small addition05:12
veeberswallyworld: ok, I'll tweak as per PR comments05:12
babbageclunkthumper: ah - ok, so it's a historical quirk that core/lease has its own ErrInvalid? Yeah, I'll harmonise those at some point in the future.05:13
* thumper nods05:13
thumperbabbageclunk: also...05:13
thumperI'd like to start moving loggers into the worker objects themselves05:13
thumperrather than package level loggers05:13
thumperto facilitate getting the logs into the right model's logs05:14
babbageclunkthumper: oh, yes - that's something I've been doing with my own new workers05:14
babbageclunkthumper: but I didn't do that with the lease manager.05:14
thumperthat's fine, it was like that befor05:14
babbageclunkcool, I'll make a note of that too.05:14
thumperno doubt we will do a pass through at some stage once we have things configured properly05:14
babbageclunkyeah, I'd say so05:15
babbageclunkoh, come on! my merge failed with `fatal: unable to access 'https://github.com/alecthomas/gometalinter/': Could not resolve host: github.com`05:19
veebersbabbageclunk: hit up IS, it's not the first time we've had odd access issues like that :-|05:20
babbageclunkI mean, it resolved it plenty of times to get all the source code, right.05:22
veebersbabbageclunk: right, seems like a really odd (and annoying) issue.05:25
babbageclunkveebers: I'm just retrying the merge before bothering IS05:25
veebersI wonder if it's something obsure as a bad port on a machine or switch, although I imagine something like that might be self-diagnosable?05:25
* babbageclunk shrugs05:29
veeberswallyworld: if a docker resource excludes a registry, should we just assume it's docker.io or is that assuming too much? (we currently assume it is that and add it). It means the05:44
* veebers actually thinks when that might actually happen.05:44
wallyworldveebers: the behaviour should be the same as docker pull etc; docker.io is the default IIANM05:45
veebersIt'll only happen when someone uses --resource I think, if I attach a publicly available image to a charm it'll be a registry.charmstore. . .05:45
wallyworldit will happen if the user overrides the resource yes05:46
wallyworldwe can leave it empty05:46
wallyworldand let k8s/docker do its thing05:46
wallyworldthat would be best05:46
veeberswallyworld: I think blank would be best05:46
veeberswallyworld: hah aye, what you said :-)05:46
* veebers makes the change (and fixes a crap test he wrote)05:47
=== frankban|afk is now known as frankban
vinowallyworld: are u still ard ?10:39
wallyworlda bit10:40
vinoi wud like to discuss abt the comment u have provided. "provisioned machine hardware".10:42
vinomy understanding is it is done for now. parity with gui10:43
wallyworldthere's a method on machine to get it i think. my understanding is that's what the gui javascript does10:43
wallyworldHardwareCharacteristics() or something10:44
vinoAint i doing the same.10:46
vinoi am addressing that in the bundle.go constarints.go10:46
vinoI always wanted to ask abt another aspect : availabilityZone which currently i am not addressing.10:47
vinoPlus, cud u please suggest me some more complex bundles for manual verification.10:47
vinocurrently the kube is in parity with gui.10:47
vinoI have provided the pastebin link for both in the PR10:49
wallyworldvino: sorry, something came up, will look tomorrow11:23
fallenour_is anyone aware of what the latest up date version of juju is, the snap version?11:47
rick_h_fallenour_: should be 2.4.1 hopefully. It was in the process of being released yesterday. /me checks with snap info juju11:51
rick_h_woot!   stable:        2.4.111:52
fallenour_rick_h_: ok, so thats a concern for me then. According to both snap refresh, as well as dist-upgrade, mine is the latest, and thats at 2.3.7, which has issues.11:53
fallenour_rick_h_: I need to move to 2.4 or greater because of the current maas issue with bionic11:54
rick_h_fallenour_: ? snap refresh juju gets you 2.3.7? what does the output of snap info juju show you?11:55
rick_h_fallenour_: maybe you need to make sure to track the stable channel?11:55
fallenour_rick_h_: snap "juju" has no updates available11:55
fallenour_rick_h_: if Im not mistaken, I am tracking stable channel.11:56
rick_h_fallenour_: right, so I want to make sure snap info juju shows 2.4.1. In fact I bet you haveboth installed?11:56
rick_h_fallenour_: what does "which juju" show you?11:56
rick_h_and what does "/snap/bin/juju --version" do11:56
fallenour_rick_h_: ok, so this is gonna be fun11:57
fallenour_it says I have 2.4.1 installed. dkpg says I have 2.3.7 installed11:57
fallenour_snap bin says I have 2.4.1 installed11:57
fallenour_now, my brain is the sad11:57
fallenour_rick_h_: ok, so assuming juju is working just fine. That means maas isnt working. I assume their team will fix it eventually, but I dontwant to wait for them to fix the problem. DNS names arent resolving, /etc/resolv.conf is the best way to do this? And if so, would simply explicitly naming the name server should fix the issue with resolving to the package repos, which is currently hindering all of my charm installs12:04
rick_h_fallenour_: so the key thing is you need to bootstrap using the /snap/bin/juju so that the controller is 2.4 which has bionic fixes12:05
fallenour_rick_h_: is that sound logic? and if so, sed -i to deliver the changes via juju run machines --all sed -i "nameserver <nameserverip>" /etc/resolv.conf12:06
fallenour_rick_h_: to rebootstrap? Im confused12:07
rick_h_bdx: kwmonroe tvansteenburgh jamespage cory_fu and anyone else. First spec on discourse for your lxd profile enjoyment. Please feel free to read/comment/etc. https://discourse.jujucharms.com/t/wip-specification-for-lxd-profile-updates-permitted-by-charms/7812:26
=== blahdeblah_ is now known as blahdeblah
=== Guest14217 is now known as zeus
=== frankban is now known as frankban|afk
veebersMorning o/20:30
veebersSo with lxc clustering, can I setup a machine in my basement as a 'cluster' and bootstrap to it from my laptop (from the comfort of my office)?20:45
hmlveebers: i think so, i’ve been told that there isn’t a difference between  remote and cluster lxc20:50
veebersnice, that would free up my laptop a wee bit20:51
hmlveebers: i was going to try with one of the nucs on a maas later20:51
hmlveebers: or tomorrow20:52
hmlveebers: i’ve been told that the video linked to the discourse on lxc cluster is amazing at explaining20:52
veebershml oh I missed that, I'll take a look thanks!20:57
veebershah, I should have read the post that I opened but hadn't read yet ^_^20:58
rick_h_veebers: yes you can. You can have a one node cluster and do that for sure21:27
rick_h_veebers: let us know how it goes success/fail in that discourse post to generate conversation21:27
veebersrick_h_: very nice, I might try set something up this weekend if I can find a moment21:28
veebersrick_h_: will do, it won't be with maas though, just a machine with ubuntu installed on it21:28
rick_h_veebers: yea, maas is just a handy way to have some extra machine around21:31
rick_h_veebers: so the only issue you'll have to make sure is to manually run the lxc bridge setup command in that doc21:31
rick_h_veebers: after that it's an add-cloud, add-credential, bootstrap21:31
veebersrick_h_: sweet, can't wait to deploy the k8s bundle to it and free up my laptop :-)21:33
pmatuliswhy do you call it a one-node cluster? juju is just connecting to a remote LXD host21:48
veebershuh, true :-)22:19
veebersif I setup clustering from the get-go then I can start adding hardware as I retire it from general use (laptops etc.)22:22
veeberswallyworld: are there some good examples of k8s configs that I could take a look at? want to confirm my understandings23:06
wallyworldwhat do you mean by configs?23:07
veeberswallyworld: i.e. it's possible that a config will define a couple of clusters, and a user and that user has access to all those clusters?23:07
veeberswallyworld: i.e. what add-k8s parses23:07
wallyworldi don't have any aprt from what we get by deploying cdk23:07
thumperwell... there we go23:07
thumpermailing lists are closed23:08
anastasiamacthumper: end of an era!23:09
veeberswallyworld: ack, does what I say make sense? (re: users -> clusters)?23:09
wallyworldI think so. i think a config can define more than one cluster. not sure about user access23:10
thumperanastasiamac: that it is23:12
thumperall hail our new discourse overlord23:12
veeberswallyworld: we might have a caas issue where you can't teardown if a deploy fails. I'll dig in and come up with some actually useful data, just FYI at this stage :-)23:50
wallyworldveebers: that's an issue for iaas as well23:50
veeberswallyworld: ah ok23:50
veeberswallyworld: unless I've got the wrong end of the stick I don't think that we can just go through and add a cluster endpoint + server to each user cred, the config field context defines what cluster a user has access too, so looks like we'll need to parse the cluster details and conditionally create a user cred for that cluster based on the context23:58

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!