/srv/irclogs.ubuntu.com/2019/10/07/#juju.txt

thumperbabbageclunk: https://github.com/juju/juju/pull/10686 for the httpserver test race00:54
babbageclunklooking00:54
thumperbabbageclunk: noticed that I hadn't pushed last commit00:55
babbageclunkok00:55
thumperbabbageclunk: good now00:55
babbageclunkthumper: why'd you change the quick request loop from a timeout based to just trying 5 times? Won't that lead to more failures?00:59
thumperno, because it normally catches it immediately, the first time through01:00
thumperand if not the first. definitely the second01:00
thumperthe loop is just there for paranoia01:00
babbageclunkI don't see how making it max of 5 rather than up to LongAttempt is more paranoid though01:02
thumperI could rewrite it to end up at LongWait, but the attempt code is deprecated, and shouldn't be used01:03
thumperit introduces double sleeps through the loop for no benefit01:03
babbageclunkah, ok01:03
* thumper reworks01:04
thumperrunning tests01:04
babbageclunkthumper: so I should be waiting? I might just approve with a comment on that line01:10
thumperbabbageclunk: pushing change now01:11
thumperwas looping the tests to double check01:11
thumperbabbageclunk: it's all there now01:11
babbageclunkthumper: cool approved01:14
thumperta01:14
thumperanother worker updated : https://github.com/juju/juju/pull/1068704:44
FallenourGood morning everyone!09:47
FallenourIs anyone else having issues with containers deploying with juju? im still having this issue, and I either need to get this resolved, or I need to pull juju out of production entirely and explore alternative solutions.09:48
stickupkidFallenour, any more information about that?09:57
Fallenouryea I keep getting DNS error messages on my lxd containers not deploying. MaaS/Juju deployment, servers deploy images just fine, image deploys just fine, and the OS can ping google.com, so its not a DNS issue. From what Ive seen its an ongoing issue from as far back as 2016, and its impacting all of juju it seems. I cant pitch a solution that doesnt work in my datacenter to someone elses.10:00
Fallenour@stickupkid, http://paste.ubuntu.com/p/DBbGM7Ksdx/10:07
Fallenour for the juju status. Just rechecked by sshing into machine 0, pinging google, it works10:07
stickupkidmanadart, do you have any thoughts about this?10:11
stickupkid"no matching image found", seems familar, but could be wrong10:12
FallenourIm testing something right now as a LDE @stickupkid10:20
FallenourIs there a way to add just a container, an empty container via juju?10:20
manadartstickupkid Fallenour: 2.4.7 is quite old.10:20
FallenourI think I know what the issue might be. Its a BIG MAYBE10:20
Fallenour2.4.7 for what @manadart10:21
manadartFallenour: "juju add-machine lxd:x"10:21
stickupkidFallenour, juju10:21
FallenourIm on 2.6.9 @manadart stickupkid10:21
Fallenourbionic-amd6410:21
FallenourMy controllers are on 2.4.7 though I just saw. Should I update those, and if so, how?10:22
stickupkidFallenour, not according to the status output10:22
Fallenourhttp://paste.ubuntu.com/p/kpKXyQYfYp/10:22
stickupkidFallenour, upgrading can be found here https://jaas.ai/docs/upgrading-models10:23
FallenourJust an FYI, thats how long Ive been fighting to get this to work, and even since before 2.4.710:23
FallenourSo do keep in mind, this is not from a lack of effort o.o10:23
FallenourI really have been fighting for years to get this thing to work for me. I r eally do want it in production10:23
stickupkidFallenour, i think upgrading your controllers would help, some bug fixes around this have landed since 2.4.710:23
Fallenouralright, I just ran upgrade model10:24
Fallenouron the controllers10:24
Fallenourits installing 2.5.810:25
Fallenourim also concurrently installing the latest images for 18.04 on bare machines, as well as the upgrade on the controllers10:25
FallenourIm gonna be so sad if it was a simple upgrade on the controllers @___@10:26
Fallenour2-3 years of fighting, all because of an update x...x10:26
Fallenour@stickupkid, @manadart I upgraded the controllers, as well as ran upgrade-model on the model itself, but its still failing to install an image. I did notice that when I run juju-status that its still showing that the model is on 2.4.7. Is there anything else i need to run?10:49
stickupkidFallenour, you need to run it on the controller, `juju upgrade-mode -m controller`10:50
FallenourI ran it in the controller model10:50
Fallenour@stickupkid,10:50
FallenourIt shows 2.5.8 in the controller model10:51
Fallenourbut not in cloud-000-000001, the current model im testing with10:51
Fallenour@stickupkid, @manadart I went ahead and destroyed the model and redeployed it. Its showing 2.5.8. Ill let you knwo what it does once I get the machines deployed10:53
stickupkidachilleasa, nammn_de - whilst I'm fighting with a lxd container in jenkins, I've brought the shell check stuff upto parity with the older pre-check tests https://github.com/juju/juju/pull/1068910:59
stickupkidachilleasa, nammn_de if anyone finds a way to speed up `go vet` I'm all ears, there is no way it should take 180s, that's crazy11:00
Fallenour@stickupkid, not sure what you are building, but ramfs is a good start for dramatic performance improvements. thats one fo the things Ive been working with for a while now. ive noticed substantial performance improvements when using it, especailyl in combination with ceph with juju11:01
stickupkidFallenour, don't tempt me with ramfs :D11:05
Fallenour@stickupkid, loool, its one of the things Im hoping to bottle and ship as part of the cloud Ive created, which uses juju as a machine mechanism. ceph is another major component. Once i get this working, I plan on shipping alpha for people to test. Ill probably need you guys help  with getting it setup right, but lawd if it works, its gonna be a monster.11:06
FallenourI was able to build an entire hybrid platform, complete with a centralized UI, and self-healing services, fully compliant out of the box, but its all in pieces until this works.11:07
stickupkidFallenour, yeah, you should bring it up here - https://discourse.jujucharms.com/ people would be very interested!11:07
Fallenour@stickupkid, Oh I cant wait! Its just insane. ive written over 615,000+ lines of code for it so far. Its a beast.11:08
FallenourNOOOOOOOOOOOOooooooooooooooo11:08
Fallenourthe same issue still!11:08
FallenourD;11:08
Fallenour@stickupkid, @manadart whats the current version for controllers?11:23
rick_hFallenour:  long time no see, how goes?11:25
Fallenour@rick_h, ! Its been a while! Im doing amazingly well! We are borderline like...3-4 weeks of getting funded! Super awesome things happening11:26
FallenourHow are you doing? How have you been?11:26
rick_hFallenour:  whoa! awesome on the funding news11:26
FallenourHow are you doing? How have you been?11:26
Fallenourooops! Wrong window XD11:26
rick_hFallenour:  partying hard you know :) just plugging away at the Juju fun stuff11:27
rick_hlol11:27
Fallenouryea its super cool! Im really excited! OOH! Even bigger news! I got NASA to sponsor the project! I will be able to test the entire cloud platform Ive been workign on for ages with them.11:27
Fallenourrick_h, Oh I wish! Ive still smashing my face to keyboard trying to get it working.11:27
rick_hFallenour:  wow, that's crazy. You run across magicaltrout in your travels there?11:27
rick_hFallenour:  ouch on the face smashing, I'd suggest less of that.11:27
Fallenourrick_h, not yet, but Im expecting Ill hear from him the moment he realizes juju is coming DoD and fedspace wide.11:27
rick_hnever really seems to work on the good bugs anyway11:27
rick_hlol11:27
Fallenourrick_h, Im having issues with lxd deploying now, of all the things, images where never a problem. Its trading one fire for antoher I suppose.11:28
rick_hFallenour:  :/ that's odd. What LXD issues. I haven't seen anything in the recent fires.11:28
* rick_h is still catching up on weekend emails and now is scared to look at the bugs section of email11:28
Fallenourrick_h, yea they think it might be a controller version issue thatll solve itself once updates are done, so Im moving from 2.4.7 to whatever current is.11:29
stickupkidcan we tripple check by doing "lxc launch ubuntu:18.04 bionic"11:29
rick_hFallenour:  oh ouch, yea 2.4's been a bit11:29
Fallenourstickupkid, yea Im just waiting for the 2.5.8 to 2.6.X to occur. its requiring I update twice.11:29
stickupkidand what does "lxc image list"11:30
rick_hstickupkid:  Fallenour yea my thought exactly. Did 2.4 know bionic existed to be able to use?11:30
stickupkidprint out11:30
stickupkidrick_h, errrr11:30
stickupkidmaybe11:30
Fallenourrick_h, yea I was really hesistant to change anything until I got everything working, and then I was gonna merge it all together, and then upgrade. Seems like upgrade is happening now though :P11:30
rick_hFallenour:  wheeee11:30
Fallenourrick_h, surprisingly, yea, it worked11:30
rick_hFallenour:  umm, congrats? :P11:30
FallenourI was running trusty, xenial, and bionic, side by side. was mazballs tbh11:31
rick_hhah, well that's ok and good. We definitely support each of those LTS's11:31
Fallenourrick_h, security would have had a seizure though, so definitely not recommended for prod, but it demosntrates the capacity for juju to support a dramatic difference in OS, whcih was a plus11:31
rick_hFallenour:  just have to get on that UA train and have the ESM for trusty :P11:32
* rick_h takes off the sales hat11:32
Fallenourrick_h, loool. I was keeping it up for minecraft modules. The target for that one was gaming servers. Ill be able to move it to bionic once I start pushing modules out for software support with bionic11:33
FallenourI didnt have a chance to tell you, but I finished carousel, so now I can mass produce software support once i figure out how to build charms.11:33
rick_hFallenour:  awesome11:35
Fallenourit doesnt seem to be a version problem, Ive updated everything to 2.6.9, and its still occuring @rick_h @stickupkid @manadart11:51
rick_hFallenour:  ok, what's the issue/log output?11:51
Fallenourmachine-0: 07:46:30 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=WARNING;unit=DEBUG"11:51
Fallenourmachine-0: 07:46:30 ERROR juju.worker.dependency "broker-tracker" manifold worker returned unexpected error: no container types determined11:51
Fallenourmachine-0: 07:50:03 WARNING juju.container.broker no name servers supplied by provider, using host's name servers.11:51
Fallenourmachine-0: 07:50:03 WARNING juju.container.broker no search domains supplied by provider, using host's search domains.11:51
Fallenourmachine-0: 07:50:03 WARNING juju.container.broker incomplete DNS config found, discovering host's DNS config11:51
Fallenourmachine-0: 07:50:45 WARNING juju.worker.provisioner failed to start machine 0/lxd/0 (acquiring LXD image: no matching image found), retrying in 10s (10 more attempts)11:51
Fallenourmachine-0: 07:51:02 WARNING juju.container.broker no name servers supplied by provider, using host's name servers.11:52
Fallenourmachine-0: 07:51:02 WARNING juju.container.broker no search domains supplied by provider, using host's search domains.11:52
Fallenourmachine-0: 07:51:02 WARNING juju.container.broker incomplete DNS config found, discovering host's DNS config11:52
FallenourSorry for the wall of text everyone11:52
rick_hhah ok sec...processing11:52
rick_hpastebin ftw :)11:52
Fallenouryea I wasnt sure if itd let me pastebinit since its an ongoing write11:52
rick_hFallenour:  this is on MAAS?11:52
Fallenouryea11:53
rick_hFallenour:  what version of MAAS and are the DNS servers setup in the MAAS config?11:53
manadartFallenour: Can you leave the root log level at DEBUG for a re-run, so we can see what the container manager is outputting?11:57
Fallenourrick_h, its 2.4.2, and DNS is configured and working. I can ssh into a machien deployed and ping google.11:59
Fallenourmanadart, how would i do that/11:59
rick_hFallenour:  hmmm, this is from the machine you want the container on?12:00
rick_hFallenour:  e.g. machine 0 in this case it looks like12:00
Fallenourrick_h, yea12:00
manadartFallenour: According to the first line of the output^ you have model config to use WARNING by default except for units.12:01
Fallenourrick_h, when I ssh into the machines, I can ping external DNS sources, but for some reason its giving me the image cannot be foudn error. I even went out of my way tod ownload all the images just to make them available, but that didnt work. I also added additional DNS servers, to include a root dns server, but that didnt resolve the issue either.12:01
manadartFallenour: try: juju model-config logging-config="<root>=DEBUG"12:01
Fallenourmanadart, ok ill try that12:01
rick_hFallenour:  was the machine manually added? https://bugs.launchpad.net/juju/+bug/182171412:02
mupBug #1821714: Container on sshprovided machine: incorrect DNS <canonical-bootstack> <juju:Triaged> <https://launchpad.net/bugs/1821714>12:02
Fallenourrick_h, I have this issue whether I use juju deploy, conjure up, or manual12:03
Fallenourone thing I have been looking into, is juju using snap as the default lxd or /bin/ ?12:03
rick_hFallenour:  hmm, yea I see https://bugs.launchpad.net/juju/+bug/1826203 is the same as well :/12:03
mupBug #1826203: deploy openstack base bundle failed with lxd error: incomplete DNS config found, discovering host's DNS config <debug-log> <juju-release-support> <maas> <openstack-provider> <juju:New> <https://launchpad.net/bugs/1826203>12:03
rick_hFallenour: the deb bin vs the snap though it detects if the snap is on the system12:04
Fallenourmy thoughts are if the lxd container system use isnt matching, it would fail12:04
Fallenourrick_h, ok, so that wouldnt matter to juju?12:04
rick_hFallenour: I don't think so12:04
Fallenourrick_h, I didnt either, but Im willing to try it.12:05
* rick_h has to take the dog to the vet, will look when I get back.12:07
rick_hFallenour:  it might be worth hopping onto https://bugs.launchpad.net/juju/+bug/1826203 as that seems the same vein and means it's not just you :)12:07
mupBug #1826203: deploy openstack base bundle failed with lxd error: incomplete DNS config found, discovering host's DNS config <debug-log> <juju-release-support> <maas> <openstack-provider> <juju:New> <https://launchpad.net/bugs/1826203>12:07
rick_hFallenour:  do you have any http proxies or anything in play in the controller/model?12:39
Fallenourrick_h, not that Im aware12:41
FallenourIm currently playing with wiping out the dns config entirely, and seeing what happens12:41
Fallenourbuilding a new machine, whcih will deplyo the new config. Will see what happens @rick_h12:41
rick_hFallenour:  ok, still looking12:41
Fallenourif everything works, and I get this working, like fully working, i expect free tacos & energy drinks for life from Canonical12:42
Fallenourand some kinda plushy animal. Suse gave me an iguana o.o12:43
Fallenourrick_h, well...at least I know that its not maas now12:46
rick_hFallenour:  ?12:46
Fallenourrick_h, unless...does juju store the dns configurations for the model, or per machine when using maas?12:46
rick_hFallenour:  so the thing is that I don't think Juju deals with DNS and just relies on the machines at play.12:46
rick_hFallenour:  e.g. dhcp/etc12:47
Fallenourrick_h, wiped the dns config entirely from maas, so its not a bad config from maas. maas is fully up to date.12:47
rick_hFallenour:  I'm guessing there's something up wtih the dhcp setup and dns but not sure12:47
Fallenourrick_h, thats the thing though,t he machines can ping externally12:47
Fallenourrick_h, if its dns, it doesnt make much sense, becuase it can ping dns names, and resolve them.12:47
Fallenourrick_h, whats the address of the system it gets images from?12:48
rick_hFallenour:  right, but according to https://github.com/juju/juju/blob/085584f255f6d66530da25947544ca33418d0675/container/broker/broker.go#L76 we look at the interfaces on the host12:48
Fallenourrick_h, which is what doesnt make sense. The host can reach the internet and resolve addresses12:48
Fallenourrick_h, whtas the dns name of the source it draws the images from?12:48
rick_hFallenour: I *think* https://us.images.linuxcontainers.org/12:51
Fallenourrick_h, OOOO12:52
FallenourOOOOH BOIII!12:52
FallenourI found something here!12:52
rick_hFallenour:  I'm not going to get my hopes up yet...12:52
Fallenourrick_h, it cant ping the address for the images12:55
rick_hFallenour:  :(12:55
Fallenourrick_h, it cna ping ubuntu though. hmmm12:55
Fallenourrick_h, and it can pull down packages12:56
rick_hFallenour:  https vs non https?12:56
Fallenourrick_h, stickupkid manadart is there a command I can use to test to see if I can download an image?12:56
rick_hFallenour:  lxc launch12:57
rick_hFallenour:  from that host machine12:57
Fallenourrick_h, I tried building a container from the local machine, but its failing12:58
rick_hFallenour:  there's your problem?12:58
Fallenourrick_h, yea Im thinking its something to do with reaching the images12:58
rick_hif you can't do it Juju certainly can't12:58
Fallenourrick_h, mmmmm12:58
Fallenourrick_h, have you guys seen this issue before? this doesnt make any sense logically12:58
rick_hFallenour:  no, my brain is hurting trying to think it through12:59
stickupkidlxc launch ubuntu:18.04 bionic12:59
Fallenourif I can ping the parent domain, it doesnt make sense that I could....FIREWALL PFSENSE!!!12:59
rick_hFallenour:  I'm getting a setup going on my maas to see if I can run logs and see where it falls over here12:59
stickupkidthat's the exact command12:59
rick_hFallenour:  is there's pfsense on the maas nodes?12:59
Fallenourrick_h, no, but there is a firewall inline13:00
Fallenourrick_h, im testing that now by disabling all block ru...no that isnt it either.13:00
Fallenourrick_h, I disabled all the block rules13:00
manadartFallenour: Does resolve.conf on the MAAS hosts have the MAAS server as the nameserver?13:01
Fallenourmanadart, yea it says search maas13:02
Fallenourrick_h, manadart which if Im not mistaken, if the gateway is the firewall, would forward it to the firewall if it doesnt know it right?13:02
Fallenourrick_h, manadart would...would ubuntu be blocking the request because its behind a firewall? because its port is 443. ssl issue?13:03
Fallenourrick_h, I have an idea13:03
Fallenourbtw I just want to say in advance I really do appreciate you guys and whta you do for the community13:05
Fallenourrick_h, its gotta be something inline13:05
Fallenourrick_h, I just pulled down an image after testing to see if Ic na pull it down inline, and it failed, then unplugged, went wireless, and it worked13:05
rick_hFallenour:  yea, it's hard because firewalls like to drop things without ack/etc so you don't know how to blame13:06
Fallenourrick_h, manadart ok, so we know its inline, but whats the best way to troubleshoot which device it is? theres only two things in line that could do this, the maas system, and the firewall.13:06
Fallenourrick_h, the thing with the firewall though is I disabled the block rules13:06
rick_hFallenour: check the logs on those boxes looking for someone running a firewall and dropping the requests13:06
Fallenourrick_h, i run both boxes. Whats the best way(s) to do that?13:07
Fallenourrick_h, should I start with the firewall, pfsense, or the maas box?13:07
Fallenourboth are latest and up to date.13:07
Fallenourrick_h, man this is something...else? Ive never seen this kind of problem before. im in the logs now, and its not even showing the IP address for anything ubuntu related.13:10
rick_hFallenour:  but the url is linuxcontainers.org vs ubuntu13:10
Fallenourrick_h, but its trying to resolve to cloud-images.ubuntu.com:44313:12
Fallenourrick_h, does it hit that address, and then pull from somewhere else?13:12
rick_hFallenour:  oh maybe that's good then.13:13
Fallenourrick_h, ok, I need to make this really noisy so I see it for sure13:13
rick_hFallenour:  maybe see if you can create a container on the maas host?13:14
Fallenourrick_h, ok so now this really doesnt make sense. Im on the same network, behind the same infra, and one system allows me to pull, and the other doesnt?13:15
Fallenourrick_h, but none of the infra is specifically isolated. Where does the metadata come from for the images for lxd?13:16
Fallenourrick_h, manadart ok so this is just weird. Maas wont let me test it. it keep saying lxdbr0 exists, but it doesnt? so I cant use sudo lxd init --auto to test13:20
rick_hFallenour:  ? on the maas node?14:20
rick_hsorry, maas host I should say14:20
fallenour1@rick_h, stickupkid manadart ok lets try this again14:39
fallenour1OOOOOOOOOOOOOOOooooooooooooooooOOOOOOOOOOOOO!!!!!!!!!!!14:39
fallenour1GOOOOOOOALLL!!!14:39
rick_hfallenour1:  ?! found it?14:40
fallenour1rick_h, DAMN YOU SNORT! I DID!14:40
manadartfallenour1: \o/ So what is it.14:40
* rick_h drumrolls14:40
fallenour1rick_h, It was snortc2 rule set that was blocking a lot of things unreasonably14:40
fallenour1rick_h, So waht I did collectively: updated juju from 2.4.7 > 2.6.X, Updated maas packages, wiped DNS setting in maas, removed squid, squid proxy, snort, and pfblockerng rules, and it works!14:41
rick_hfallenour1:  lol14:41
rick_hfallenour1:  "small tweaks"14:41
fallenour1rick_h, LOL14:42
fallenour1rick_h, alright! lets DO THIS!14:42
fallenour1ooo you GOTTA BE KIDDING ME14:42
fallenour1power went out Q____Q14:42
pmatulisrick_h, here it is: https://bugs.launchpad.net/juju/+bug/184712817:33
mupBug #1847128: [2.7] ceph-osd stuck in "agent initializing" <juju:New> <https://launchpad.net/bugs/1847128>17:33
rick_hpmatulis:  ty19:00
rick_hhml:  did that help any with the model-config test?20:00
hmlrick_h: i’m in debugger land, trying to understand what’s going wrong.  helped to point in a better direction.20:01
rick_hhml:  ok, let me know if you could use extra eyeballs20:01
hmlrick_h: rgr20:01
bdxheyya22:10
bdxwhen I add `series: kubernetes` to my bundle, it does not want to deploy22:11
bdxthis bundle https://api.jujucharms.com/charmstore/v5/~omnivector/bundle/slurm-core-k8s-2/archive/bundle.yaml22:11
bdxgive me https://paste.ubuntu.com/p/Bh6Hwy54zq/22:13
bdxbut the bundle shows the k8s tag in the charmstore, all looks good there22:13
bdxto get the k8s bundle to deploy I need to change `series: kubernetes` to `bundle: kubernetes` liek so https://paste.ubuntu.com/p/WQ7xmF4Gw7/22:14
bdxafter I make this change the bundle can deploy, but doesnt show the k8s tag in the charmstore22:15
bdxare people aleady aware of this?22:15
thumperseries: kubernetes is just wrong22:25
thumperbecause kubernetes isn't a series22:25
thumperthe bundle: keyword was added to handle this22:25
thumperif the charmstore is parsing 'series: kubernetes' it is doing it wrong22:26

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!