babouhi, has anyone seen an issue where juju just entirely refuses to work?03:49
babouas in, I type "juju status" and it just hangs03:50
baboui am running ubuntu 16, I have restarted my computer, I have reinstalled juju03:50
baboui had been using it successfully to follow tutorials, but it has since stopped working03:51
blahdeblahbabou: There are a few bugs around which cause jujuds to become very busy; restarting jujud-machine-0 should sort them out03:51
blahdeblah^ That's for juju 1.x; for 2.x it might be called something slightly different03:51
baboui'm running 2.0.203:52
blahdeblahIt will be called something similar (maybe the same); ssh directly to your juju controller and systemctl restart 'juju*'03:53
baboui'm running locally with lxd03:53
babouin that case I would enter that command on my local machine, right?03:55
blahdeblahbabou: Within your lxd container03:56
blahdeblahbabou: This could be something different as well, like your networking not set up quite right or something.03:57
babouthat doesn't seem to have worked04:02
baboutried what you said and tried restarting the container externally04:02
baboui can use a cloud container though, so I will just use that for now04:02
blahdeblahbabou: It could easily be something else then04:03
babouany other ideas?04:03
blahdeblahhave a look in ~/.local/share/juju04:03
blahdeblahthere's a file called controllers.yaml04:04
blahdeblahfind your controller, and there should be a line in its section like this: api-endpoints: ['']04:04
blahdeblahcheck that you can "nc -vz 17070"04:04
baboui have no fille called controllers.yaml04:05
blahdeblahwhat does "juju list-controllers" say?04:06
babouwait yes i do04:06
baboui found the line and tried this command04:08
babounc -vz
baboubut it just printed the help message04:08
blahdeblahbabou: no colon between IP & port04:24
babounc: connect to port 17070 (tcp) failed: No route to host04:24
blahdeblahbabou: Sounds like network setup then04:25
babouok thank you for your help04:34
kjackalGood morning Juju world07:51
mattywmorning folks, reading through the docs on constraints (https://jujucharms.com/docs/stable/charms-constraints) was wondering if it was possible to specify a constraint in a file ahead of time, so for example, all controllers bootstrapped on the given cloud will have that constraint?09:25
=== frankban|afk is now known as frankban
=== deanman_ is now known as deanman
holocronmattyw: did you read this here? https://jujucharms.com/docs/stable/charms-constraints#using-constraints-when-creating-a-controller13:35
mattywholocron, I did thanks, I was hoping to have a file somewhere so I could not have to remember to do it on bootstrap13:43
holocronMatty I see.. you want to do it post bootstrap. That I'm not sure about13:47
holocronAh, you want to apply it to all controllers. Now I got it13:48
mattywholocron, the other way around, I want it to happen to all future bootstraps13:51
mattywholocron, all future bootstraps for a given cloud anyway13:51
mattywholocron, if for example I'm too cheap to want to use m3.medium for all my instances13:51
mattywholocron, but I'm also too lazy to remember how cheap I am13:52
holocronI see.. sorry I don't know of a way. All my juju use cases are local LXD or manual13:55
rick_hmattyw: alias cheapbootstrap=juju bootstrap --constraints... ? :P13:57
rick_hmattyw: no, there's no "constraints via file" currently.13:57
rts-sanderhelp I do "juju deploy mysql" but it says connection refused14:56
magicaltroutwhat says connection refused rts-sander ?14:58
Merlijn_SHi @kjackal15:05
kjackalMerlijn_S: Hello!15:05
kjackalMerlijn_S: How can I help?15:06
Merlijn_SI'm working on a setup for a spark course. ~20 teams of students, each get their own jupyter notebook with pyspark integration that's connected to a shared spark-on-yarn + hdfs cluster15:06
kjackalMerlijn_S: sounds nice!15:07
Merlijn_Snow the issue is that while a sparkcontext is active, it is a running job in yarn. This means that it blocks resources, even though nothing is running.15:07
Merlijn_S~20 teams means ~20 spark contexts15:07
kjackalyes, true. Is there any parameter we can play with, let me see if I find something.15:09
Merlijn_Seach sparkcontext has its own yarn job that marks a certain amount of resources as "used", so the cluster runs out of resources.15:09
rts-sandermagicaltrout, https://justpaste.it/1228015:09
rts-sanderI guess I could download the charm and then deploy it from local15:10
magicaltroutrts-sander: looks like the server you're trying to run it on is behind a firewall15:10
magicaltroutor missing a route to the interwebs15:10
rts-sanderjuju version 1.25.6-xenial-amd6415:10
Merlijn_SI'm also not sure how the yarn resources work exactly. Does each "application container" get a fixed amount of resources or does this depend on the job itself?15:11
magicaltroutwhat you doing running 1.25 anyway ?! :P15:11
rts-sanderbecause we were going to use it in production before 2.0 was out of beta15:11
rts-sanderbut now we switched to 2.0 but I'm too lazy to upgrade15:11
magicaltroutif I go to the url in your error in my browser15:12
magicaltroutit downloads the charm15:12
magicaltroutso i guess its connectivity of some sort15:12
rick_hrts-sander: magicaltrout exactly15:12
rts-sandersame here15:12
rick_hrts-sander: it looks like you can't reach the api.jujucharms.com to get the zip of the charm from the store15:12
rick_hrts-sander: so the only thing to do is to get the charms locally and deploy that way if you can't augment the connectivity15:12
rts-sanderI can wget it15:13
rick_hrts-sander: there's a download link here: https://jujucharms.com/mysql/ on the right side to get the zip15:13
rick_hrts-sander: right, but can where the controller lives get to it?15:13
* rick_h goes back to read where the juju controller is sitting (local or on an openstack or?)15:13
rts-sanderit's a local deployment15:13
rick_hrts-sander: hmm, so is lxc setup with a bridge where it can reach the internet through your computer then?15:14
* rick_h thought we set that up in 1.2515:14
rick_hrts-sander: maybe an intermittent issue? did you try more than once and get it?15:14
rts-sanderI tried multiple times15:14
rts-sanderwhen I first installed it I could get charms through it15:14
rts-sanderI'll deploy from local, it will take less time than trying to figure this out15:15
rick_hrts-sander: hmm, that sounds like something intermittent then perhaps. I guess can you try now? If you can wget the file and lxc is setup to work.15:15
kjackalMerlijn_S: this looks like a good description of YARN esource amangement: https://www.cloudera.com/documentation/enterprise/5-3-x/topics/cdh_ig_yarn_tuning.html15:23
kjackalMerlijn_S: And this is how you set the cores and mem from spark https://www.mapr.com/blog/resource-allocation-configuration-spark-yarn15:25
kjackalMerlijn_S: If I understand your request correctly you need to either overcommit resources so that the utilisation increases or do some kind dynamic allocation/resource preemption15:28
Merlijn_Skjackal: We're running the slaves inside LXD containers; running multiple slaves in multiple lxd containers solves the issue by overcommitment, but I worry that when they start to run real queries everything will bork...15:31
=== scuttle|afk is now known as scuttlemonkey
Merlijn_Skjackal: So if I understand correctly, It's indeed spark that decides what the size of the container is. The size of the container is the same no matter how much is actually used in the jobs.15:45
Merlijn_Skjackal: So ideally, I find a way to make the sizes of these containers dynamic depending on what actual jobs are running. If that doesn't work then I'll just have to find a way to make the spark containers smaller.15:46
kjackalMerlijn_S: give me some time. In a meeting15:48
Merlijn_Skjackal: Related: How do the bigtop charms calculate what the "total memory" of a machine is?15:48
Merlijn_Skjackal: aight :)15:48
petevgcory_fu: I updated https://github.com/juju-solutions/matrix/pull/63/files, with code that catches the TestFailure in a slightly bettter place. (Also set the exit code to 1 if we have a generic uncaught Exception.)16:11
kjackalMerlijn_S: I am back. Not sure how bigtop queries for the available memory. I will have to look at the code, If you want me to.16:11
cory_fupetevg: Excellent16:11
kjackalMerlijn_S: I would start with the configuration of spark. Like spark.executor.cores as described here: http://spark.apache.org/docs/1.6.1/configuration.html16:14
kjackalMerlijn_S: with --conf you are able to set the config variables: in spark-submit16:14
Merlijn_Skjackal: I'm using pyspark; any idea how those config values translate to that?16:16
kjackalMerlijn_S: I see there is a SPARK_CONFIGURATION variable in jupiter configuration (never used that)16:16
kjackalMerlijn_S: are you using this pyspark? http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.SparkConf16:18
Merlijn_Skjackal: I'm using findspark: https://github.com/minrk/findspark16:19
Merlijn_Skjackal: so that translates to: http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.SparkContext16:21
kjackalSeems so16:21
kjackalMerlijn_S: and you can set the configuration there16:21
Merlijn_Skjackal: k, thx, I'll take a look at that.16:27
Merlijn_Skjackal: I'm also interested in the bigtop available memory logic. Currently, it only uses 2/3 of the real memory on the machines.16:30
Merlijn_Skjackal: and it differs a lot between machines that have the same hardware.16:31
Merlijn_Skjackal: scrap that, it's the same; it's 2/3 of the total memory of each machine.16:32
Merlijn_Skjackal: ever heard of 'dynamic resource allocation' of spark contexts? That might be what I'm looking for: https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation16:42
kwmonroeMerlijn_S: dyn res allocation won't be easy.  it looks like it requries a jar + yarn-site.xml config changes to all nodemanagers, and we're currently not exposing those config options :/16:55
kwmonroeMerlijn_S: sorry i'm playing catch-up here.. but you also said "now the issue is that while a sparkcontext is active, it is a running job in yarn".  are you in yarn-client mode?  if so, the driver (which instantiates a sparkcontext) does not run on the yarn cluster.  it runs whereever you start the spark application.17:03
kwmonroeMerlijn_S: if you're in yarn-cluster mode, of course, the driver will indeed take one of your yarn nodemanagers to do its thing.17:04
petevgcory_fu: pushed one more update to that gating PR (I wrote some better .matrix tests, and found out that we couldn't turn off gating -- the update contains the fix + the tests.)17:22
petevg cory_fu: I kind of want to integrate those .matrix tests into tox, but your clever method of running main only works once -- the second time we try to do it, we've already set a bus and done other globally relevant stuff to matrix, so it doesn't work. I'm kind of tempted to write a script that sniffs out .matrix files in the test dir, and runs them via17:25
petevgPython subprocess, checking for a 0 or non zero exit code. Don't want that to delay the gating stuff, though. Will open a ticket ...17:25
cory_fupetevg: Couldn't you just add them as additional test plans in the existing functional test?17:28
petevgcory_fu: they have to be completely separate matrix runs.17:29
cory_fupetevg: Oh17:29
Merlijn_Skwmonroe: sorry, was out to get food. I'm talking about the sparkcontext itself, not the driver. The sparkcontext itself takes up space while it is alive. So every interactive spark session blocks unused resources.17:42
Merlijn_Skwmonroe: So basically, a multi-tenant interactive spark cluster is more or less a no-go without dyn res allocation..17:45
larrymi_afkI'm trying to remove a relation to re-add it and juju remove-relation seems to work but when I do juju add-relation .. it keep saying relationship exist. Is there a way to force break the relation?17:51
=== larrymi_afk is now known as larrymi
* larrymi tries juju destroy-relation17:54
larrymihmm.. destroy-relation doesn't work either17:57
lazyPowerlarrymi - is the unit halted in a hook context after removing the relation?18:00
lazyPoweras in, debug-hooks or some such18:00
larrymilazyPower: ah yes I see a config-changed hook failure18:03
larrymilazyPower: not on the unit itself but on the other endpoint18:04
lazyPoweryeah, that will halt the removal of the relationship in the controller18:06
larrymilazyPower: ah but that's not from the removal .. that was there before18:06
lazyPowerso  its in progress but not completed, and its right its been removed but "isnt really removed"18:06
lazyPowerits in queue18:06
lazyPowerif you resolve that failure and step that remote unit through, it should hit the -departed and -broken hooks and clear itself up18:06
larrymilazyPower: right18:06
larrymilazyPower: ah that makes sense. Thanks! :-) I'll give that a try.18:07
=== frankban is now known as frankban|afk
marcoceppirick_h: SHOWTIME?18:57
arosalesrick_h: got a link to the show ?18:57
marcoceppiaccidental caps18:58
rick_hmarcoceppi: arosales working on it18:58
rick_hhmm, won't let me check the "hangouts on air" button :/18:59
rick_hmarcoceppi: arosales https://hangouts.google.com/hangouts/_/ytl/VusL3EZZPRDpX4QUibgOwiPCpKQKzhydHyEehU55OXw=?eid=103184405956510785630&hl=en_US&authuser=019:02
rick_hto join the hangout19:02
rick_hhad to create a new one :(19:02
rick_hhttps://www.youtube.com/watch?v=n86tRu4xpYU to watch19:02
arosalesrick_h: thanks, omw19:04
rick_hanyone else coming? jcastro19:04
rick_hnatefinch: or redir or anyone want to join feel free19:04
CoderEuropehttps://www.youtube.com/watch?v=AfBnrLvDFy8     The Juju Show #3     |    Last streamed live on 5 Oct 201619:06
CoderEuropecfgmgmt session19:06
CoderEuropeWhy aren't we showing this today ?19:09
rick_harosales: coming or should we go on?19:09
rick_hCoderEurope: things and stuff, have to watch! :P19:09
CoderEuroperick_h: I have been waiting 3 days for this show = Is it happening, or not ?19:10
rick_hCoderEurope: yes, we were waiting for arosales19:10
rick_hbut he's gone awol so we're going on now19:10
marcoceppiCoderEurope: the link to watch is here: https://www.youtube.com/watch?v=n86tRu4xpYU19:11
CoderEuropeokay - I shall await ....19:11
natefinchrick_h: I have a fix for the lxd thing, so i'll be making a test and committing it.  have fun, though!19:11
arosalesrick_h: please start without me19:12
bdx_rick_h: I neglected to see the examples directory in libjuju19:17
lazyPoweris that a community member that just joined?! hype!19:17
bdx_there are a bunch of examples that I seemed to of missed, but it was a good learning experience none the less19:18
lazyPowerrick_h  is it unfiled if we just discovered it today? :)19:19
CoderEuropeQUESTION: First of all thankyou for the video \o/ - they're fabulous for the community ! 2ndly, @Discourse juju charm (Modern Forum Type) is really needed for the lug I am involved with - What are the chances ?19:20
lazyPoweroooo its mreed hype19:21
lazyPowerCoderEurope - we had a very old discourse charm, however we're interested if there's a community member thats interested in maintaining the discouse charm19:21
arosalesCoderEurope: there is a community developed Discourse charm @ https://jujucharms.com/u/marcoceppi/discourse/precise/2719:21
bdx_thejujushow: the libjuju work I've been cutting has already made it into my jenkins post build step .... I'm simply using it to run actions on my juju deployed applications following successful build step completion19:21
CoderEuropelazyPower: I would but it may be beyond me .. but I have spoken about this before :D19:23
lazyPowerCoderEurope - well, if your LUG is interested in learning, we can certainly host a charm school for anyone interested19:23
lazyPowerfree training to get you up and moving with juju with a focus / eye towards developing the discourse charm19:23
Merlijn_Sbdx_: are you James? Have you looked at the ci pipeline bundle? It's more or less what we wanted to build..19:25
arosalesbdx_: https://jujucharms.com/u/juju-solutions/cwr-ci/ for reference, is what I think Merlijn_S  is referencing19:26
bdx_Merlijn_s: yes, its me19:26
Merlijn_Sbdx_: Yes. + they're working on support for testing and publishing full bundles. preview here: https://github.com/juju/python-libjuju19:26
bdx_Merlijn_s: niceeeee19:26
lazyPowermarcoceppi - juju status-history19:26
sparkiegeekjuju show-status-log19:26
lazyPowersparkiegeek high5 on gmta19:27
sparkiegeeklazyPower: o/ :)19:27
Merlijn_Sbdx_ : sorry, preview ci for bundles here: https://github.com/juju-solutions/layer-cwr/tree/build-bundles19:27
magicaltrout32 days....19:27
magicaltroutbugger, better come up with some content19:27
bdx_alias wjst="watch -n 0.2 -c juju status --color"19:27
bdx_Merlijn_s: for me its more about integrating application ci with juju though19:29
lazyPowermarcoceppi - this may be a good time to bring up layer-debug19:29
bdx_Merlijn_s: not so much CI on the charms themselves, which I think is more your use case right?19:29
skayI am not familiar with layer-debug.19:29
skaydefinitely bring that up19:30
CoderEuropeQUESTION: Not really a question- But , but ... I would just like to say that Beards' win in the show, today :D19:30
lazyPowerskay - we (The eco team) wrote a layer to provide scaffolding for reporting on a unit to do debug. In our case we wanted entwork info dump, container / file descriptor info dumps, fs dumps, etc.19:30
lazyPowerso i hope they cover it, might b fun :D19:30
Merlijn_Sbdx_: I'd think that's more or less the same issue. Since charm tests can be whatever. I'd think you would just create a charm test that starts the tests of the actual application inside the charm..19:31
sparkiegeekhttps://bugs.launchpad.net/juju/+bug/1645422 - retry-provisioning bug19:31
mupBug #1645422: retry-provisioning doesn't retry failed deployments on MAAS <landscape> <maas-provider> <retry-privisioning> <juju:Triaged> <https://launchpad.net/bugs/1645422>19:31
bdx_Merlijn_S: application CI vs charm CI .... we have a team that writes selenium test suites for each of our apps19:34
bdx_rick_h: true that19:35
Merlijn_Sbdx_: I have no experience with application CI so I'll just shut up :)19:35
bdx_Merlijn_S: lol ... oh man .. lets catch up soon19:36
=== tris- is now known as tris
sparkiegeekrick_h: FYI: the Google logo is obscuring the first line of your terminal19:40
sparkiegeekQUESTION: Does interactive juju add-cloud pre-seed potential MAAS API endpoints from ~/.maasclidb ?19:42
sparkiegeekooh nice19:44
sparkiegeekno more tickling Juju in three different ways to bootstrap in MAAS19:44
petevgcory_fu: updated the crashdumping/gating PR to address your comments: https://github.com/juju-solutions/matrix/pull/63/files19:45
petevgcory_fu: if you have the branch checked out locally, you'll need to delete it and refetch -- someone's "trivial" indentation fixed created conflicts that I needed to clean up with a rebase :-p19:45
cory_fupetevg: :)  Sorry about that19:46
lazyPowerMarco: I really think it would be solid to talk about layer-debug even if only briefly19:47
petevgIs cool.19:47
lazyPowermarcoceppi ^19:47
lazyPowerif you have k8s deployed, you can just run it :)19:47
sparkiegeekQUESTION: are there any ACLs planned for actions?19:50
sparkiegeekI might not want everyone to be able to get debug information19:50
sparkiegeekcover it next time, all good :)19:50
CoderEuropebye guys !19:50
lazyPowerGreat show fellas19:50
CoderEuropelazyPower: Was it you that was to ping me about discourse Xenial charm ?19:51
lazyPowerCoderEurope - thats marcoceppi, but i'm happy to sit in and provide feedback/assistance19:51
CoderEuropeokay cheers.19:51
marcoceppiCoderEurope: o/19:51
marcoceppiso I wrote this a long /long/ time ago. But they've moved to docker distribution. So it should be pretty straight forward to re-charm again where we run their bits in their docker package and then bolt on the rest from the outside19:52
CoderEuroperight -oh19:52
marcoceppiCoderEurope: would you ahve time time later this week, or next week, to hash it out a bit?19:52
CoderEuropemarcoceppi:   - I have a new laptop coming in this/next week - Had to stretch to the chromebook just to tune in today :D19:54
rick_hmbruzek: do you have a link for the layer to stick in the show notes?19:54
marcoceppiCoderEurope: cool, give me a ping here when you're set up19:54
CoderEuropemarcoceppi: cool-beans !19:54
rick_hmbruzek: ty19:54
mbruzekrick_h: https://github.com/charms/layer-debug19:54
rick_hmarcoceppi: jcastro either of you know why the comments are disabled when I look at the video, but if I go to the edit/advanced settings, allow comments is checked?19:58
marcoceppirick_h: is it publshed?20:00
rick_hmarcoceppi: hmm, says "public" not seeing anything addressing "published" specifically20:02
sparkiegeekI still have problems accessing the links in the show notes in iOS - the long links get truncated by YouTube (fine) but it actually changes the href to be truncated too :(20:04
sparkiegeekrick_h: think you could use a link shortener to work around their bug?20:04
rick_hsparkiegeek: doh20:06
rick_hsparkiegeek: you're asking for an increased production level here. I feel like I need to go to film school for this :P20:06
sparkiegeekrick_h: well normally I wouldn't pay it any attention, but there's this guy I follow on Twitter who keeps harking on about them :)20:08
rick_hsparkiegeek: lol20:08
magicaltroutin python world can i do status_get and get the message?20:28
magicaltroutoh yeah it appears it already does20:30
justicefrieshey all, picking up after a few weeks. is juju on OSX compiled with Go 1.7 yet?20:31
lazyPowerjusticefries it is not20:33
lazyPoweri still get the random go stacktracing here and there20:33
lazyPowermagicaltrout yep :D20:36
lazyPowerrick_h - wasn't there some talk about the go 1.7 bit and how we handle that for brew?20:36
cory_fupetevg: I added a comment reply to your PR, but I really think that we need to keep the test with glitch in and figure out how to make it work with the gating.  That was the entire point of adding the gating logic in the first place.20:55
petevgcory_fu: I thought that we had decided not to run glitch by default.20:55
petevgcory_fu: ... because it conflicts with the goal of providing a reliable automated testing framework for now.20:56
cory_fupetevg: The reason we added the gating flag was because we wanted to still be able to run the glitch test but not have it influence the success or failure20:56
cory_fuSo it would be informational but not affect pass / fail20:56
petevgcory_fu: yes. But unless we add a generic Exception check, we can't do that, because we also fail on uncaught Exceptions.20:57
petevgcory_fu: we could also just not throw an error on uncaught exceptions if the task doesn't gate, but that feels more likely to mask genuine issues to me.20:57
cory_fuWe need to address that, then.  And I don't think a blanket catch is the right solution.  What exception is being thrown and why?  Is it a bug in glitch, or is it an expected failure case?  If the latter, then it should be converted to a TestFailure or added to the gating logic20:58
petevgcory_fu: I'm talking about things that you posted -- the NoneType surfacing because of a bug in core.20:58
petevgcory_fu: I've caught that by checking for TypeError, along with some other general Python errors, but I don't know what other errors libjuju or core will throw (especially as libjuju is under rapid development).20:59
petevgcory_fu: I want matrix to surface those errors, actually. But I don't want to do it in a way that messes the automated testing of charms bit.20:59
petevgcory_fu: I think that the best way to wind up with a glitch that gates in a controlled manner is to turn any Exception that comes up during a glitch run into a TestFailure.20:59
petevgcory_fu: that way, if we gate on glitch, we see the error. If not, it's in the logs, but it doesn't break stuff.21:00
cory_fupetevg: How is that NoneType error not a bug in glitch / matrix?21:00
petevgcory_fu: it's a bug in core/libjuju.21:00
petevgcory_fu: it's one of the things that can happen if you try to send a command to a machine that is rebooting.21:00
petevgcory_fu: you periodically complain about it in PRs, but then you also complain about it when I try to fix it in a generic way :-p21:00
cory_fuI don't think that NoneType error is a bug in libjuju.  I think it's a bug in matrix in that it continues to try to perform glitch operations after encountering a TestFailure21:01
petevgcory_fu: The NoneType error happens in libjuju.21:01
cory_fupetevg: Sure it does, because we're trying to call model methods after the connection was closed21:02
petevgcory_fu: and if I mask it in glitch by adding a check for it in the reboot action, I disable glitch's ability to tell us whether or not that error is fixed.21:02
petevgcory_fu: let's jump into the hangout.21:02
petevgcory_fu: wrote code, tested, and pushed the restored matrix.yaml, and Exception catching glitch.21:18
cory_fuI'll take a look in a minute21:19
petevgcory_fu: ack. Didn't check in the "gating: false" lines in the matrix.yaml. Fixed and pushed.21:34
cory_futvansteenburgh: Quick PR for you: https://github.com/juju/python-libjuju/pull/4021:57
jhobbsdo bundles need metadata.yaml's to be in the charm store?22:06
marcoceppijhobbs: nope, just a readme and bundle.yaml22:08
jhobbsahh it was the README i was missing22:10
jhobbsthanks marcoceppi22:10

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!