| === xavpaice_ is now known as xavpaice | ||
| === rogpeppe1 is now known as rogpeppe | ||
| === frankban|afk is now known as frankban | ||
| dakj | icey: hi icey, are you here? | 07:30 |
|---|---|---|
| icey | Indeed dakj | 07:32 |
| dakj | icey: do you have time to help me to resolve the issue with Ceph-Mon? | 07:36 |
| icey | dakj: it looked like the mons were clustered happily, except one was showing stale status | 07:36 |
| icey | dakj: did updating your ceph-osd configuration get disks working? | 07:36 |
| dakj | icey: on ceph-osd in old-devices there is /dev/vdb, while before the commit it was /dev/sdb. Units ceph-osd/12, ceph-osd13, and ceph-osd14 result blocked. On each node the command fdisk reports that https://paste.ubuntu.com/24591531/. Its juju status is here https://paste.ubuntu.com/24591542/ | 07:45 |
| icey | dakj: can you log into one of the ceph-mon units (`juju ssh ceph-mon/12`) and run `sudo ceph -s` for me? | 07:46 |
| Alex_____ | team, i am getting this while running yum command http://pastebin.ubuntu.com/24588230/ in centos | 07:55 |
| Alex_____ | any idea how to resolved this | 07:55 |
| anrah | Alex_____: are you bootstrapping controller? | 08:01 |
| Alex_____ | @anrah | 08:01 |
| Alex_____ | i was trying to run yum command giving some error | 08:02 |
| Alex_____ | anrah: | 08:02 |
| Alex_____ | anrah: i am getting this inside the vm | 08:03 |
| anrah | Alex_____: I mean how this is related to juju? And it seems like you are using Vagrant? You must configure your Vagrant box to have access to Internet | 08:05 |
| Alex_____ | anrah: i just did vagrant config for some testing purpose.. and by default if i go inside the centos vm box i am not able to run the yum command itself.. can you help me on this | 08:07 |
| dakj | icey: here is it https://paste.ubuntu.com/24591604/ | 08:07 |
| icey | dakj: so that answers part of it, the /12 machine is definitely not clustered :) could you also run that from one of the other 2 machines (ceph-mon/13 for example)? | 08:08 |
| dakj | Icey: here is it https://paste.ubuntu.com/24591622/ | 08:11 |
| icey | dakj: can your ceph* nodes talk to each other on the network? it looks like it has failed to bring one of the mon nodes up, and liek the OSD nodes have never managed to register themselves with the mons | 08:12 |
| jamespage | icey, dakj: has the path between the units been verified for network MTU configuration etc... using iperf? | 08:27 |
| jamespage | feels like the mon is trying to bootstrap, but failing due to some external reason | 08:27 |
| jamespage | suspicion would be packet gfrag but I may be wrong | 08:27 |
| icey | jamespage: that's what it looks like to me, it seems like 2 of the mons were fine (and successful), one of the mons and the OSDs are left in the cold | 08:27 |
| dakj | Icey: I'm making ping between the lxd node. All node response | 08:27 |
| jamespage | dakj: ping won't tell you the right things | 08:28 |
| jamespage | use iperf with the mtu flag set | 08:28 |
| jamespage | it will give you perf data and validate the mtu settings | 08:28 |
| dakj | Icey: here is the paste of the ping between lxc machine https://paste.ubuntu.com/24591663/ | 08:29 |
| kjackal | Good morning juju world! | 08:32 |
| dakj | Jamespage: I can try that, but if the issue is MTU I think that also other lxc machine had to have the same issue. | 08:33 |
| jamespage | maybe | 08:33 |
| jamespage | lets see | 08:33 |
| dakj | James-age: where do I must to install that? | 08:33 |
| dakj | Ice, James-age: now the situation of juju status is changing https://paste.ubuntu.com/24591673/ | 08:35 |
| dakj | Now all ceph-mon and ceph-osd are in blocked | 08:35 |
| jamespage | dakj: you'll need to install iperf on the machines you want to test connectivity between | 08:40 |
| jamespage | dakj: and then on one run iperf -s -m and from another do iperf -c <IP of first machine> -m | 08:41 |
| jamespage | and then vica versa | 08:41 |
| jamespage | network problems can be uni-directional | 08:41 |
| dakj | I've to install that on 2 o more LXC machine and test the MTU status, isn't it? | 08:42 |
| dakj | James-age & icey: the juju status is return to original status (https://paste.ubuntu.com/24591699/) | 08:44 |
| dakj | James-age, icey: the result between the VM with MAAS and the LXC vm of openstack-dashboard/4 is here https://paste.ubuntu.com/24591716/ | 08:50 |
| dakj | Here is between open openstack-dashboard/4 and ceph-mom/13 https://paste.ubuntu.com/24591721/ | 08:55 |
| dakj | Ice, Jamespage: from the result I don't think the issue is about that.....I hope of having read that well. | 09:03 |
| kklimonda | is there a way to override how juju installs lxd and lxcfs? | 09:14 |
| dakj | Icey, Jamespage: any idea? | 09:17 |
| === salmankhan1 is now known as salmankhan | ||
| jamespage | dakj: not sure - can you do the tests between the ceph-mon units please | 09:30 |
| dakj | Jamespage: any response between ceph-mon/12 and ceph-mon/13 or /14, otherwise between /13 and /14 is fine https://paste.ubuntu.com/24591867/ | 09:48 |
| === vds_ is now known as vds | ||
| jamespage | dakj: all three machines need to be able to communicate with each other | 09:49 |
| dakj | I think that issue with /12 because is in maintenance | 09:49 |
| dakj | Then osd/12, osd/13 and osd/14 are in blocked | 09:50 |
| dakj | Juju status https://paste.ubuntu.com/24591878/ | 09:51 |
| anrah | Is there a way for amulet to load charm configuration file from file? | 10:06 |
| kklimonda | how does jujucharms.com versioning work? is it set in stone, and are there guarantees that a) charm will never be deleted and b) no two releases will have the same version? If so, does this hold for the charms not in the "main" namespace, for example cs:~sdn-charmers/keystone-0 ? If not, what's the correct way to guarantee charm version over the life of the | 10:06 |
| kklimonda | deployment? | 10:06 |
| dakj | Jamespage: I was thinking and if the issue is on ceph-osd/12,ceph-osd/13, and ceph-osd/14?? Why their status on juju is in blocked and on gui have not error? | 10:11 |
| dakj | the hypothesis | 10:11 |
| stub | kklimonda: It is guaranteed that no two releases will have the same version. It is possible for someone to revoke access to a charm, effectively deleting it (and maybe you can really delete it) | 11:08 |
| stub | kklimonda: This goes for both namespaced charms and the top level namespace | 11:08 |
| stub | kklimonda: If you want to pin a particular version, deploy cs:~sdn-charmers/keystone-0 and never run upgrade-charm | 11:09 |
| stub | kklimonda: If you want to upgrade to a particular version, use the --switch argument to juju upgrade-charm | 11:10 |
| kklimonda | stub: is keystone-0 always point to the same version of the charm, assuming maintainer doesn't do anything stupid like deleting and uploading it again with diffent code? | 11:10 |
| stub | kklimonda: But most deployments want latest stable, which would be cs:~sdn-charmers/keystone (ie. no revision, default channel) | 11:11 |
| stub | kklimonda: It will always point to the same version of the charm. If the maintainer deleted it and uploaded it again, it would get a new revision | 11:11 |
| kklimonda | stub: I assume most larger deployments want to have a tested version of the charm, to avoid any drift in a longer period | 11:12 |
| kklimonda | especially if they have more than one deployment (for example testing and prod) | 11:13 |
| stub | If they are doing their own testing, yes. They will deploy the last known good revision. | 11:14 |
| kklimonda | my only concern is that someone can revoke access to the charm, that sounds like a npm left-pad nightmare. | 11:14 |
| kklimonda | I'll have to reconsider maintaining local copy of all charms | 11:15 |
| stub | I don't think it would happen in the curated top level namespace, and may not be possible. | 11:15 |
| stub | The eco system team or charm store team would know more | 11:15 |
| kklimonda | mhm | 11:15 |
| stub | I think mortals can only control access to their own namespace. The charms promulgated to the top level namespace are maintained by the ~charmers team. | 11:16 |
| stub | People in the US timezone will know more | 11:17 |
| stub | Nothing to stop you maintaining your own forks though if that is what you prefer. | 11:18 |
| stub | You can even use the charm store to do it ;) | 11:19 |
| stub | We actually deploy most of our stuff from a local copy (which we pull from the charmstore), because we need to inject site specific hooks before we deploy. | 11:20 |
| dakj | Jamespage: is there any other task I can make to resolve that? thanks | 11:23 |
| jamespage | dakj: tbh without understanding why you're hitting the problem you have, I don't have a step forward for you atm | 11:24 |
| jamespage | stub: actually "The charms promulgated to the top level namespace are maintained by the ~charmers team" is not 100% true any longer | 11:24 |
| jamespage | its possible for any team to own promulgated charms (see ~openstack-charmers or ~ganglia-charmers for examples) | 11:25 |
| stub | I've only worked with my local namespaces, and had them magically promulgate to the top level. | 11:26 |
| jamespage | dakj: you could strace the ceph-mon on the unit that's failing to join - might give a sniff of what's up | 11:26 |
| jamespage | dakj: hmm just noticed this | 11:28 |
| jamespage | [ 4] 0.0-19.1 sec 525 MBytes 231 Mbits/sec | 11:28 |
| jamespage | that appears out of sync with the other two metrics you recorded | 11:28 |
| dakj | James-age: do you think it's a network issue? | 11:32 |
| dakj | And why the other lxc vm work well??? the issue is only on lxc that have ceph-mon........ | 11:34 |
| dakj | James-age: do you know the credential for the login on Openstack dashboard? | 12:32 |
| jamespage | dakj: you need to prefix messages to me with 'jamespage' otherwise I don't get notifications | 12:46 |
| jamespage | dakj: if you're working from the openstack-base bundle then the username and password are | 12:46 |
| jamespage | admin/openstack | 12:47 |
| dakj | jamespage: ok, sorry it's the autocorrection that change your nick. | 12:48 |
| jamespage | dakj: that's entirely likely | 12:51 |
| dakj | jamespage: I send you a private text | 12:53 |
| jamespage | dakj: your deployment is still not complete; I'm pretty sure that you have some sort of networky issue in your virtual lab setup, but its hard to be specific as to what exactly that is | 12:53 |
| jamespage | the fact that hooks are still trying to run to complete the deployment sniffs like packet fragmentation type issues | 12:53 |
| jamespage | dakj: you might want to try to validate you virtual lab independently of trying to deploy openstack itself | 12:55 |
| jamespage | dakj: https://jujucharms.com/u/admcleod/magpie is useful here - you can deploy that charm to both physical machines and to lxd containers on the machines and it will check and report on performance and mtu configuration/mismatches | 12:56 |
| jamespage | dakj: fwiw this is what I do on hardware prior to even trying to deploy openstack | 12:58 |
| jamespage | as it flushed out issues that are hard to diagnose later when you see things behaving oddly | 12:59 |
| jamespage | dakj: ultimately I think MAAS will grow this type of feature, but for now its easy todo with a charm | 12:59 |
| jamespage | dakj: that last status output you pasted - some units had completely disappeared | 13:00 |
| cnf | \o jamespage | 13:02 |
| cnf | i am in london atm | 13:02 |
| jamespage | hey cnf | 13:02 |
| jamespage | enjoy the city ;) | 13:02 |
| cnf | eh, meeting / demos all day | 13:03 |
| dakj | jamespage: my lab is based on a IBM server with ESX as hypervisor and it's connected directly to firewall via switch. On ESX then I've created all environment MAAS, Landscape, Juju and Openstack all of them are vm. | 13:12 |
| dakj | MAAS has been configured with 2 vnet (https://paste.ubuntu.com/24592671/) the first one is for DNS/DHCP service the other one for public as suggest here https://jujucharms.com/openstack-base/ | 13:17 |
| jamespage | dakj: all one physical server? | 13:18 |
| dakj | yes | 13:19 |
| jamespage | dajk: what type of vmware switch are you using? | 13:19 |
| jwd | hello | 13:20 |
| dakj | I've an IBM System x3650 M4 with 64GB of RAM and 4tb of storage | 13:20 |
| jwd | just doing my first experiments with juju here | 13:21 |
| rick_h | jwd: welcome | 13:23 |
| randomhack | having a problem bootstrapping to openstack if the keystone endpoint doesn't fit this pattern https://host:port/version - my keystone is on https://host/keystone/version | 13:23 |
| jwd | rhx | 13:23 |
| jwd | thx | 13:23 |
| randomhack | ERROR cannot set config: cannot create a client: invalid major version number /keystone/v3: strconv.Atoi: parsing "/keystone/v3": invalid syntax | 13:23 |
| dakj | jamespage: yes it's a only physical server (I've an IBM System x3650 M4 with 64GB of RAM and 4tb of storage) | 13:24 |
| jwd | from watching the chat here most ppl seem to use it to deploy an openstack environment? | 13:24 |
| jamespage | dakj: yeah - I was really after the vmware virtual switch type and configuration being used | 13:25 |
| jamespage | jwd: agreed alot of the conversation would lead you towards that conclusion, but it does alot of other things as well | 13:26 |
| dakj | jamespage: I've 1 vSwitch with 2 vnet (10.20.81.0 'n 10.20.82.0) both are in Promiscuous mode actived | 13:27 |
| jamespage | dakj: that last nugget was what I was looking for | 13:28 |
| jamespage | thanks for confirming | 13:28 |
| * jamespage remembered something from way back about having to have that enabled, otherwise the LXD containers never get network access | 13:28 | |
| SimonKLB | is it possible to register a localhost/lxd controller on a remote client? | 13:46 |
| SimonKLB | im able to access the gui, but when trying to register gives me: | 13:47 |
| SimonKLB | ERROR unable to connect to API: x509: certificate is valid for anything, juju-apiserver, juju-mongodb, localhost, not [dns] | 13:47 |
| rick_h | SimonKLB: no, not at this time. There's work going on to build on lxd to make it more cloud-like but it's in flight | 13:51 |
| SimonKLB | rick_h: got it! thanks | 13:51 |
| lazyPower | #TIL juju status now reports the progress of a lxd image import | 13:59 |
| lazyPower | \o/ | 13:59 |
| dakj | jamespage: but MAAS subnets has to be configured in this way eth0(https://pasteboard.co/7oi5M27zS.png) and eth1 (https://pasteboard.co/7oitlHdJm.png) | 14:31 |
| freyes | hi marcoceppi , could you take a look to this PR when you have some time? https://github.com/marcoceppi/charm-ubuntu/pull/5 | 14:31 |
| magicaltrout | lazyPower: should i be able to update from CDK deployed a few months ago to current? | 14:32 |
| magicaltrout | its not a trick question, just want to know if i can run juju update-charm or whatever | 14:32 |
| jwd | can a controller be deleted somehow? | 14:36 |
| lazyPower | magicaltrout: yes, there's upgrade instructions on the k8s docs on how to do an in place upgrade | 14:38 |
| jwd | ah found it | 14:38 |
| magicaltrout | oh yeah lazyPower | 14:39 |
| magicaltrout | because the next level up said "ubuntu" i figured it was ubuntu and not juju | 14:40 |
| magicaltrout | iykwim | 14:40 |
| lazyPower | magicaltrout: https://kubernetes.io/docs/getting-started-guides/ubuntu/upgrades/ | 14:40 |
| magicaltrout | yeah i see it | 14:40 |
| dokerya | Hi | 14:41 |
| jwd | did i already mention that i like what i see so far ;-) | 14:45 |
| SimonKLB | jwd: it sure is awesome :) | 14:54 |
| jwd | yeah this will help me alot to speed up my building and testing processes | 14:55 |
| SimonKLB | jwd: are you deploying something from the charmstore are you planning on building charms for your own applications? | 14:56 |
| jwd | i will build my own soon | 14:56 |
| SimonKLB | jwd: cool! good luck :) | 14:57 |
| jwd | we run a multi tier stack build of alot components. and i been in need to remodel all of that. | 14:57 |
| SimonKLB | jwd: then juju will be perfect for you :) | 14:58 |
| jwd | i will start with some of the charms i seen and model around those | 14:58 |
| SimonKLB | sounds like a great approach | 14:58 |
| jwd | so many components to rethink .hehe | 14:58 |
| SimonKLB | it might be quite a bit of work to get it all charmed, but in the end im sure that it's going to be extremely rewarding :) | 15:00 |
| jwd | i did all we have so far by hand and ansible roles. we run around 80 vms in a openebula cloud atm :-) | 15:00 |
| jwd | so anything helping me to model that faster is a benefit | 15:01 |
| jwd | i just need to think about how to get that into a production ready setup asap | 15:01 |
| SimonKLB | jwd: i actually think it's possible to re-use a lot of the work youve done with ansible already | 15:02 |
| SimonKLB | someone else can step in and correct me if im wrong | 15:02 |
| jwd | i am sure i can reuse my work. | 15:02 |
| SimonKLB | jwd: https://jujucharms.com/docs/2.0/about-juju :) | 15:03 |
| jwd | i started with this yesterday, so i will need to learn a bit about the basics | 15:03 |
| jwd | as usual. alot new stuff to learn :-) | 15:05 |
| jwd | hehe got my development notebook to its limits :-) | 15:19 |
| jwd | guess its time to claim some funds for a bigger lab environment | 15:21 |
| Zic | lazyPower: hi, hey, a simple question today: what is precisely "GA"? I saw this many times in Ubuntu Insights concerning CDK | 15:35 |
| Zic | it seems to be a development acronym that I don't distinguish in English :> | 15:35 |
| lazyPower | Zic: General availability (GA) is the marketing stage at which all necessary commercialization activities have been completed and a software product is available for purchase, depending, however, on language, region, electronic vs. media availability. | 15:36 |
| Zic | oh, ok :) | 15:37 |
| Zic | thanks | 15:37 |
| lazyPower | np | 15:37 |
| SaMnCo | magicaltrout: have a look at my last post: https://goo.gl/22invt and jump to the conclusion ;) | 16:57 |
| lazyPower | SaMnCo: your post is already out of date :) that edge etcd charm is now in stable <3 | 16:58 |
| SaMnCo | aouch :D | 16:58 |
| SaMnCo | fixing... | 16:58 |
| SaMnCo | fixed | 16:59 |
| lazyPower | <3 like a boss sir. Thanks for keeping the world in the loop that we're one of if not the best solution to get moving with GPU's | 17:00 |
| SaMnCo | oh yeah :D | 17:00 |
| SaMnCo | I think the best actually. Really not easy with other stuff as you need to prep the drivers from cloud-init, which means adding logic to id if a node has GPUs or not... | 17:01 |
| lazyPower | :) i was allowing room for other opinions, no matter how iffy they might be :D | 17:01 |
| lazyPower | i'm clearly biased | 17:01 |
| SaMnCo | me too, but I've been testing a few things to understand how our UX differs from others, and I really really really like how the GPU stuff comes in | 17:02 |
| SaMnCo | this is from my most objective self, so be proud | 17:02 |
| SaMnCo | I wouldn't blog about it otherwise, opinions are my own on medium | 17:03 |
| lazyPower | <3 its taken a village but the effort has certainly been worth it | 17:04 |
| Budgie^Smore | o/ juju world | 17:04 |
| lazyPower | \o Budgie^Smore | 17:05 |
| Budgie^Smore | are we having fun yet lazyPower? | 17:09 |
| lazyPower | Budgie^Smore: yeah, i'm gutting TLS key authentication and replacing with basic-auth/token-auth w/ pass/token rotation. | 17:12 |
| lazyPower | how about yourself? | 17:12 |
| Budgie^Smore | lazyPower oh nice! so SSO should be easier to implement then? ... waiting for feedback from the interview yesterday, prepping for another tomorrow and a phone screen today | 17:14 |
| lazyPower | Budgie^Smore: well its more like, i skipped all the authentiation/authorization steps and tried to do sso without having any o those primitives in place and then got mad when it didn't work | 17:14 |
| Budgie^Smore | lazyPower ain't that always the way ;-) | 17:15 |
| lazyPower | i had some colleagues course correct me and check facts and we're now rebuilding that vector from teh ground up, because we made some pretty obtuse assumptions when we first landed our auth model | 17:15 |
| Budgie^Smore | lazyPower "oh this should be easy... damn it! why isn't this working?!?!" | 17:15 |
| Budgie^Smore | lazyPower well you know what they say about assuming anything ;-) | 17:17 |
| lazyPower | it makes us all grow in the end? | 17:17 |
| chatter29 | hey guys | 17:18 |
| chatter29 | allah is doing | 17:19 |
| chatter29 | sun is not doing allah is doing | 17:19 |
| chatter29 | to accept Islam say that i bear witness that there is no deity worthy of worship except Allah and Muhammad peace be upon him is his slave and messenger | 17:19 |
| Budgie^Smore | lazyPower "assume = to make an 'ass' out of 'u' and 'me'." - http://www.urbandictionary.com/define.php?term=Assume | 17:22 |
| lazyPower | Budgie^Smore: language! :P | 17:25 |
| Budgie^Smore | lazyPower sorry is 'arse' more acceptable ;-) | 17:26 |
| === frankban is now known as frankban|afk | ||
| magicaltrout | lazyPower: Waiting for kube-system pods to start | 17:40 |
| magicaltrout | actually might just be slow standby | 17:40 |
| magicaltrout | sweet | 17:40 |
| magicaltrout | forget it | 17:40 |
| magicaltrout | upgrade works amazing | 17:41 |
| lazyPower | magicaltrout: tweet that :) | 17:42 |
| magicaltrout | shipped | 17:44 |
| lazyPower | magicaltrout: <3 | 17:57 |
| jwd | is there a way to disable the automatic updates when a charm creates a node? | 18:27 |
| jrwren | jwd: a charm could have the side effect of disabling the underlying ubuntu unattended-upgrades, but there is no built in juju way, a charm would need that, preferably as a non-default config option. | 18:30 |
| jwd | oki | 18:30 |
| jwd | so much to learn :-) | 18:33 |
| Budgie^Smore | wow my kernel upgrade knowledge is way out of date! so how does juju handle livepatch? | 18:56 |
| rick_h | Budgie^Smore: it doesn't since you have to enable it and it has to be associated to your account? | 18:57 |
| Budgie^Smore | oh I get that part but then there is system level things that need to happen. I suppose that could be a custom os level image... still pretty interesting tech, wonder how it performs against a 4.x kernel no reboot upgrade | 18:59 |
| Budgie^Smore | just goes to show how long it has been since I researched that problem though lol ;-) | 19:01 |
| umbSublime | Is there a node limit, for a self-hosted MaaS with juju to deploy openstack ? | 19:37 |
| === dpb1_ is now known as dpb1 | ||
| rick_h | umbSublime: not really, it's up to the scaling of the maas/juju controller to handle the load. | 19:47 |
| rick_h | umbSublime: what are you looking for? | 19:47 |
| umbSublime | I think I got confused by the 10 Node free limit for autopilot | 19:47 |
| === tvansteenburgh1 is now known as tvansteenburgh | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
