[01:07] <menn0> woot! the logging-to-mongodb feature branch has landed
[01:08] <anastasiamac> menn0: wow! congrats :D well done!
[01:08] <menn0> anastasiamac: cheers
[01:08] <menn0> I just checked the size of the diff to master: 3,500 additions and 852 deletions
[01:08] <anastasiamac> menn0: would love to know of ur experiences in terms of wirtting ci tests for it as well as user-facing documentation
[01:09] <anastasiamac> menn0: when did u create the branch? (for how long have u had it running?)
[01:09] <menn0> anastasiamac: there is no user facing documentation, b/c from the user's perspective nothing changes
[01:10] <menn0> anastasiamac: it's just a necessary change for JES ... and it's a little less fragile than all-machines.log
[01:11] <menn0> anastasiamac: not sure how old the branch is... around mid April
[01:11] <wallyworld> except we can't use a text editor when mongo dies :-(
[01:11] <anastasiamac> menn0: so there r no changes to configuration
[01:11] <anastasiamac> me?
[01:11] <menn0> wallyworld: you still have a text log file of all received logs per state server
[01:12] <wallyworld> which we need to accumulate by hand and which are not interleaved
[01:12] <menn0> wallyworld: that's the fallback
[01:12] <wallyworld> yeah, i know, i'm just grumpy
[01:12] <menn0> wallyworld: yes, if there are multiple state servers
[01:12] <wallyworld> no i mean all machines log also contained logs from nodes
[01:12] <wallyworld> all interleaved
[01:13] <wallyworld> so you could easily see sequencing
[01:13] <menn0> wallyworld: yeah I understand
[01:13] <wallyworld> i often ssh into a state server and grep that file
[01:13] <wallyworld> far easier than debug log
[01:13] <menn0> wallyworld: well the next thing is to expand debuglog's capabilities then
[01:13] <wallyworld> yeah
[01:14] <wallyworld> sorry for grumping, will just make my life harder now
[01:14] <wallyworld> i understand the reasons for it
[01:14] <menn0> wallyworld: for example, selecting time ranges is largely implemented on the server side but needs client side support
[01:14] <wallyworld> it's the lack of exploring which is the main thing
[01:14] <wallyworld> easier to just flitter around in a text file
[01:16] <anastasiamac> menn0: weren't u at one stage considering to make log output destination (file, db, etc) configurable?
[01:17] <menn0> anastasiamac: no I don't belive so. it's hard to effectively handle logs for multiple environments in a single file.
[01:18] <menn0> wallyworld, anastasiamac: with the way it's the feature is structured it wouldn't be too hard to generate a all-machines.log or even separate log files per env from the DB.
[01:19] <wallyworld> yay
[01:19] <menn0> wallyworld, anastasiamac: with the bonus that it doesn't involve rsyslogd
[01:19] <wallyworld> even better :-D
[01:19] <menn0> we'd just need a worker which tails the DB like the debuglog API does and generates the file
[01:20] <menn0> all the pieces are decomposed in such a way that this wouldn't be hard at all
[01:21] <wallyworld> friday labs :-)
[01:22] <anastasiamac> menn0: this is great! and the feature (as well as its design) would b an awesome demo for team meeting - again tyvm for the feature :)
[01:46] <menn0> anastasiamac: yep good idea
[01:46]  * menn0 is having lunch so sorry for the delay
[02:21] <wallyworld> waigani: hey, did you see the maas output on that bug? looks like maas is doing the right thing?
[02:22] <waigani> wallyworld: ah really? hmph. well there goes my theory. At least we can test with that as mock output - try to reproduce it in a test.
[02:23] <wallyworld> waigani: well, i think it is correct, only took a quick look
[02:23] <wallyworld> but we have the output
[02:23] <waigani> wallyworld: I'll try to take a look today, but I need to finish off this branch - I'm on leave for the next 4 days
[02:24] <waigani> great that we've got the output though, should be helpful
[02:32] <wallyworld> waigani: ok, see how you go, let me know where you get to
[02:32] <waigani> wallyworld: will do, I'm not far off...
[04:09] <waigani> wallyworld: looks like gomaasapi wan't reading the substatus. Here's a PR to fix that: https://code.launchpad.net/~waigani/gomaasapi/substatus/+merge/264371
[04:09] <wallyworld> waigani: ty, looking
[04:09] <waigani> wallyworld: I've also added that to the bug as a comment.
[04:10] <wallyworld> waigani: so that's a fix for the test service in gomaasapi, we still need to fix juju as well right?
[04:11] <waigani> wallyworld: once that gets signed off, it should just be a matter of bumping the dependancies.tsv
[04:11] <waigani> wallyworld: and probably adding a test
[04:11] <wallyworld> waigani: how? the pr above doesn't fix anything
[04:11] <wallyworld> it fixes a test server
[04:12] <waigani> wallyworld: ugh, facepalm
[04:12] <waigani> wallyworld: let me go back
[04:12] <wallyworld> ok
[04:13] <waigani> wallyworld: right, yeah sorry so the juju side is still to come
[04:13] <wallyworld> sure
[04:14] <waigani> but it looks like that's the problem, as you can see from that test
[04:14] <waigani> wallyworld: the one thing that makes me nervous is that we are flattening two dimensions into one
[04:14] <wallyworld> let me look at the maas output
[04:15] <waigani> wallyworld: as juju status doesn't have any concept of a 'substatus' so, we just need to be careful how we decide which status/substatus gets reported
[04:19] <wallyworld> waigani: so the maas "deployment_status" api call returns a string doesn't it?
[04:19] <wallyworld> "Failed Deployment" or "Deployed" etc
[04:19] <waigani> wallyworld: yeah
[04:19] <wallyworld> so we don't need to worry about interpretting status vs subststus
[04:20] <wallyworld> is it only older maas deployemnts without the new api call?
[04:20] <waigani> wallyworld: let me look and I'll get back to you
[04:21] <wallyworld> ok
[04:21] <waigani> wallyworld: just seeing how we dial up that status from the juju side
[04:21] <wallyworld> juju calls deployment_status
[04:22] <wallyworld> well, it does while it waits for the node to come up
[04:31] <waigani> wallyworld: add this to tip of 1.24: http://pastebin.ubuntu.com/11853823/
[04:32] <waigani> wallyworld: it passes
[04:32] <wallyworld> looking
[04:32] <waigani> wallyworld: and yeah we call deployment_status in maas/environ.go deploymentStatusCall
[04:33] <waigani> wallyworld: so it's reporting the correct error
[04:34] <wallyworld> waigani: i'm not sure that's all correct - that test could pass if the status stays "pending"
[04:34] <wallyworld> i don't see how we're testing the maas status interpretation
[04:35] <wallyworld> and adding a test against a modified test server in gomaas doesn't really prove anything
[04:36] <waigani> wallyworld: right, fair point.
[04:36] <waigani> wallyworld: how about I checkout 1.24.1 (same as in bug), revert changes to gomaasapi and work on improving this test?
[04:37] <wallyworld> try and reproduce a test failure (write a new failing test). then fix juju so the test passes
[04:37] <wallyworld> TDD and all that
[04:37] <wallyworld> target the test to the specific issue
[04:37] <waigani> wallyworld: actually I did run that test on the unmodifed server - and if you comment out the "substatus" line, it's fails - as expected
[04:37] <wallyworld> other more general bootstrap tests could also later need changing, but there should be a targetted, specific test done first
[04:38] <wallyworld> yes, but that's because the test is relying on the behaviour of hte test server
[04:38] <wallyworld> and the test server reports "Deployed" - status 6
[04:38] <wallyworld> in the absense of sub status 11
[06:04] <waigani> wallyworld: I can't get it to fail, I've written a summary on the bug. I've asked to look at the logs.
[06:05] <wallyworld> did you test with real maas?
[06:05] <wallyworld> or vmaas
[06:05] <waigani> wallyworld: I was just about to say, that's the next step
[06:06] <wallyworld> it won't fail if you just test with the gomaas test server
[06:06] <wallyworld> unless the test server mimics the maas output
[06:06] <wallyworld> which it doesn't appear to
[06:06] <waigani> wallyworld: yeah, that's what I did - as explained in the bug
[06:07] <wallyworld> ok, i'll read the bug :-)
[06:09] <wallyworld> waigani: thanks for looking into it, enjoy the time off. i'll see what we can make of it
[06:11] <waigani> wallyworld: thanks. I'm keen to know the cause.
[06:11] <wallyworld> i'll update the bug
[06:11] <wallyworld> when we find it
[06:11] <waigani> ah I was just about to
[06:12] <waigani> wallyworld: updated
[06:13] <wallyworld> waigani: it could be that an old api call to get status is being made, and hence sub status is ignored - that would be my guess
[06:13] <wallyworld> anyways, we'll see
[09:01] <TheMue> fwereade: interested in a quick chat ;) we're alone today
[09:31] <perrito666> fwereade: hi, please dont forget my branch if you have the chance
[11:53] <fwereade> perrito666, sorry, the diff looks mangled? would you update and try again please?
[12:33] <perrito666> fwereade: sure
[12:35] <perrito666> mm, says rbt it cannot reach reviewboard, odd
[13:04] <perrito666> fwereade: sorry rbt is playing dumb with me
[13:38] <TheMue> voidspace: heya Michael, everything alright with your car?
[14:02] <sinzui> katco: perrito666 : there is a new reression in master, bug 1473461 Maybe we want to backout a branch becaue the fix is to write a darwin-specific ./cmd/jujud/util/password/password_darwin.go
[14:02] <mup> Bug #1473461: OSX/darwin builds fail: undefined: password.EnsureJujudPassword <ci> <osx> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1473461>
[14:03] <mup> Bug #1473450 opened: upgrade-juju from 1.18 to 1.20.4 leaves some agents down <juju-core:New> <https://launchpad.net/bugs/1473450>
[14:03] <mup> Bug #1473461 opened: OSX/darwin builds fail: undefined: password.EnsureJujudPassword <ci> <osx> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1473461>
[14:10] <ericsnow> TheMue: was my review of http://reviews.vapour.ws/r/2129/ helpful?
[14:10] <TheMue> ericsnow: very, thanks. I'm working on it.
[14:10] <ericsnow> TheMue: cool :)
[14:11] <perrito666> bogdanteleaga: that password thing is yours :)
[14:11] <ericsnow> TheMue: I'll try and give you a quick turnaround as soon as you have any comments or updates to the patch
[14:12] <TheMue> ericsnow: there has been a longer discussion about the EntityWatcher before. the first Addresser approach used the IP address values. but then Dimiter said it would make sense to use common.LifeGetter and common.Remover.
[14:12] <mup> Bug #1473466 opened: When using openstack as provider lxc containers get a hardcoded ip <sts> <juju-core:New> <https://launchpad.net/bugs/1473466>
[14:12] <TheMue> ericsnow: so we needed tags for ip addresses, an earlier chnage.
[14:12] <TheMue> ericsnow: and this code is mostly based on http://paste.ubuntu.com/11803914/
[14:13] <ericsnow> TheMue: ah
[14:13] <TheMue> ericsnow: should have written more into the description *lol*
[14:13] <ericsnow> TheMue: it just struck me as odd that the EntityWatcher was part of this patch :)
[14:14] <ericsnow> TheMue: yeah, how terrible of you <wink>
[14:14] <TheMue> ericsnow: it rushed in as a side-effect. would have been worth an extra change, yes
[14:14] <TheMue> hehe
[14:16] <ericsnow> TheMue: not really a blocker, but putting multiple effectively distinct changes in a single patch can be a problem every long once in a while (in this case not a big deal)
[14:18] <TheMue> ericsnow: what? you need more than one commit to implement <insert favourite complex feature here>? :D
[14:19] <ericsnow> TheMue: well, not me personally ;)  just this guy I know (okay it's me)
[14:19] <TheMue> :,)
[14:42] <mup> Bug #1473470 opened: Windows cannot ensurePassword <ci> <regression> <windows> <juju-core:Triaged> <https://launchpad.net/bugs/1473470>
[14:54] <alexisb> hello akhavr welcome!
[14:54] <akhavr> alexisb: :)
[14:56] <perrito666> ashipika: around?
[14:57] <ashipika> perrito666: here
[14:58] <perrito666> just something I found and might bite you in the future
[14:58] <ashipika> perrito666: do tell, please
[14:58] <perrito666> ashipika: priv
[17:20] <perrito666> sinzui: around?
[17:22] <sinzui> perrito666: I am
[17:22] <perrito666> sinzui: I am trying to use boot_context in my CI test
[17:23] <perrito666> i feel a bit of a lack of clarity regarding what should the arguments be
[17:26] <sinzui> perrito666 in deploy_stack.py deploy_job() -> deploy_job_parse_args(), _deploy_job() -> boot_context()
[17:27] <sinzui> perrito666: the parse args will have descriptions
[17:27] <perrito666> ah, excelent, thank you
[17:27] <sinzui> perrito666: I think we want to add a ricj docstring to boot_context() to encourage it's use
[17:28] <sinzui> perrito666: I will take a stab at the docstring in a few minutes, and pass it for review for you and maybe abentley
[17:54] <mup> Bug #1473517 opened: juju environment not usable after the upgrade <juju-core:New> <https://launchpad.net/bugs/1473517>
[18:37] <perrito666> sinzui:  back, sorry, 2 things, 1) what is bootstrap_host supposed to be and 2) job name can be set to something arbitrary or do I need to add it as an arg?
[18:38] <sinzui> job_name is arbitrary yes, but it is alos unique so that the script and base env can be run in parallel
[18:39] <perrito666> and bootstrap_host?
[18:39] <sinzui> perrito666: bootstrap_host is not required. it is a placement directive for bootstraping manual or maas
[18:40] <perrito666> None will suffice or you are expecting it to be a given type?
[18:40] <sinzui> perrito666: yep
[18:44] <perrito666> ok this function is too long
[18:49] <perrito666> sinzui: ok, almost there:  agent_url and agent_stream?
[18:51] <sinzui> perrito666: agent_stream == agent-stream in config. None means don't override
[18:51] <sinzui> perrito666: agent_url == agent-metadata-url in config. None means don't override
[18:52] <sinzui> perrito666: We use these often and will be required to run revision tests in parallel when we need concurent streams
[18:52] <sinzui> well we do, which is why the configs force the testing streams
[18:53] <perrito666> sinzui: so you are saying I should make these args
[18:55] <sinzui> perrito666: all of those things need to be args becauee we may need to run the test with manual and we certainly will be running with streams created for the test
[18:56] <perrito666> sinzui: I find then odd that those are not part of add_basic_testing_arguments
[18:57] <mbruzek> sinzui: Do you know if anyone has an RPM to install juju client?
[18:57] <sinzui> perrito666: the only arg from that may be dropped from (job_name, client, bootstrap_host, machines, series, agent_url, agent_stream, log_dir, keep_env, upload_tools) is upload_tools because we discovered this week that we cannot fallback to upload-tools with the new os and arch requirements
[18:58] <sinzui> mbruzek: I have not seen one. mbruzek Since client relies on ubuntu-ness I doubt anyone succeeded
[18:58] <perrito666> sinzui: ack
[18:59] <sinzui> mbruzek: the centos client will work for centos https://launchpad.net/juju-core/+milestone/1.24.2
[18:59] <sinzui> mbruzek: only centos7 actually. I thnk an error is quickly raise of the host os/series is not recognised
[19:00] <mbruzek> sinzui: Marco and I just tried to alien the debian packages, it didn't work... yet
[19:01] <sinzui> mbruzek: I am sure the deps wont work becaue mongo is not packaged to meet Juju's needs
[19:01] <mbruzek> OK Thanks sinzui
[19:15] <abentley> sinzui or jog: Could you please review https://code.launchpad.net/~abentley/juju-ci-tools/s3-download/+merge/264450 ?
[19:15] <sinzui> abentley: looking
[19:16] <abentley> sinzui: thanks.
[19:17] <abentley> sinzui, perrito666: I think that --upload-tools is nice to have for devs who are testing their self-built jujus.
[19:17] <sinzui> abentley: I can and maybe should install s3cmd on all slaves
[19:17] <sinzui> abentley: My one mac has it
[19:17] <abentley> sinzui: Oh.  Well, if we do that, we can rip out s3 support for the workspace runner.
[19:18] <sinzui> abentley: then I wont because I like what you have written
[19:19] <abentley> sinzui: I do like the idea of isolating the resource retrival/publication from the testing step.
[19:20] <sinzui> yeah, this is awkwardly coupled. Isn't this because we haven't updated the build job to also upload to s3 yet?
[19:21] <abentley> sinzui: It's because we haven't done download support for workspace-runner.
[19:21] <abentley> sinzui: We can scp files, which is useful, but the remote client can't download files from s3.
[19:32] <sinzui> abentley: This command now works on the osx-slave
[19:32] <sinzui> s3cmd -c cloud-city/juju-qa.s3cfg ls s3://juju-qa-data/
[19:33] <abentley> sinzui: Okay, I can switch it over to use s3cmd.
[19:34] <sinzui> abentley: I can add s3cmd to windows, but I don't want to give it credentials. I think I put myself in a corner
[19:35] <abentley> sinzui: If the plan is to ultimately have our windows jobs uploading everything directly to s3, that would be a problem.
[19:36] <abentley> sinzui: Obviously, we can have separate win credentials.
[19:43] <sinzui> abentley: The win-client-deploy job need the installer built by build-win-client.The current rule is the ci job procurs the installer under test and delivers it to the win machine.
[19:46] <abentley> sinzui: Right.  Basically the same pattern I was following with run-osx-client.
[19:46] <sinzui> yes. In egnlish, test this <client> using with <script>
[19:47] <abentley> sinzui: But then at the end, ci-director uploads the console log to s3.
[19:48] <sinzui> yes, and that is also true for windows.
[19:48] <abentley> sinzui: I'm imagining that eventually, we'll want workspace-runner to be able to do that.
[19:51] <sinzui> r=me abentley
[19:52] <abentley> sinzui: I have updated the code so that s3cmd is run remotely.  Do you want that version instead?
[19:52] <sinzui> abentley: I got scared.
[19:54] <sinzui> abentley: I trust the osx machine but not the win machines. I think it is easier to maintain the idea that testing host cannot retrieve the things under test
[19:54] <abentley> sinzui: Okay.
[20:24] <mbruzek> sinzui: Those Juju binary files in the tar.gz worked on CentOS7.  Thank you!
[20:24] <mbruzek> marcoceppi: ^
[20:25] <sinzui> mbruzek: I should hope so, we officially distribute them
[20:25] <mbruzek> sinzui: Is there a test process for installing Juju on CentOS7 ?
[20:25] <mbruzek> to verify they work?
[20:26] <sinzui> mbruzek: no. We are still tinkering with  what we recommend
[20:26] <sinzui> mbruzek: I verified them using a centos machine I provisiioned in aws. I could boostrap an aws env
[20:27] <mbruzek> sinzui: thanks.  I did not think they would not work, I was just letting you know I was able to get them to work.  Thanks.
[20:27] <sinzui> mbruzek: this will be tested automatiically later this month.
[20:27]  * mbruzek regrets the double negative
[20:29] <mbruzek> sinzui: Do you know any work to get the tools build for centos7?  We could not deploy a centos7 charm because there were no tools
[20:31] <sinzui> mbruzek: you don't need to build them, they are officially published. The problem is that no one created a way to manually provision or bootstrap them. mgz and and I are still playing with a non-maas way to provision an aws machine with cloud-init
[20:31] <sinzui> mbruzek: https://streams.canonical.com/juju/tools/streams/v1/com.ubuntu.juju:released:tools.sjson lists the 1.24.0 and 1.24.2 agents
[20:32] <sinzui> mbruzek: This is the most recent centos7 agent https://streams.canonical.com/juju/tools/releases/juju-1.24.0-centos7-amd64.tgz
[20:34] <mbruzek> sinzui: marcoceppi downloaded a centos7 image from AWS that appeared to include cloud-init  https://aws.amazon.com/marketplace/pp/B00O7WM7QW/ref=srh_res_product_title?ie=UTF8&sr=0-2&qid=1436560475610
[20:35] <mbruzek> Starting with CentOS-7 we now include cloud-init support in all CentOS AMI's,
[20:35] <sinzui> mbruzek: yep that is what mgz and I plan to use
[20:35] <sinzui> that is also the machine I tested the client with
[20:36] <mbruzek> if I downloaded those tools could I use the --upload-tools flag in Juju?
[20:38] <sinzui> mbruzek: no you cannot. that simple and practical feature was not built first :(
[20:39] <sinzui> mbruzek: I think upload-tools or manual provisioning is the most sensible first level support, instread we got maas. The official way to write charm for centos is to get a maas with handmade centos images first :(
[20:40] <mbruzek> sinzui: OK well if you and mgz are working on this I would be interested in helping getting amazon working.
[20:40] <sinzui> mbruzek: we will give you an update next week.
[20:41] <mbruzek> sinzui: I spoke with a potential customer they like Ubuntu but have some CentOS that they can not work around and specifically asked if Juju would work with CentOS
[20:42] <mbruzek> sinzui: thanks for answering my questions
[20:43] <sinzui> mbruzek: maybe by 1.25. 1.24 is for cloud engineers who are willing cheat every step to make something work
[20:44] <mbruzek> sinzui: understood.  Consider me interested and I understand we actually want them using Ubuntu.  I would just like to demo CentOS without having to install maas
[20:47] <sinzui> mbruzek: me too. My last thought was to bootstrap an ubuntu env, bring up a centos machine, run a prep script to satisfy add-machine, then juju add-machine to let juju really add it. The prep script would provide fake commands or alternates to for the cloud-init to work. For example. Create the ubuntu user. create an lsb_release script that returns centos7
[21:08] <perrito666> can I mark comments as addressed someway in launchpad?
[21:37] <sinzui> perrito666: no, just comment that you are done
[21:40] <abentley> sinzui: I've successfully built osx client into (test) s3: https://console.aws.amazon.com/s3/home?region=us-east-1#&bucket=ws-runner-test&prefix=juju-ci/products/version-2873/build-osx-client%20%20/build-17/foo/
[21:40] <sinzui> abentley: \o/
[21:40] <xwwt> abently: good news
[21:40] <xwwt> abentley: good news
[22:58] <perrito666> the fact that save produces something called "unsaved comment" in launchpad is a tad confusing