| sinzui | wallyworld, Yes, I think that would be lovely and "sign-tools" is clear | 00:25 |
|---|---|---|
| wallyworld | ack | 00:25 |
| wallyworld | will do that today after some other work - i'm making so that sync-tools always uploads to old /tools location until 1.16 is released | 00:26 |
| wallyworld | should hopefully address maas issue where no public tools are available. i was really hoping that public repository would be done so that this would not have been necessary | 00:28 |
| davecheney | https://bugs.launchpad.net/juju-core/+bug/1236662 | 00:35 |
| _mup_ | Bug #1236662: environs/ec2: if the bootstrap node is restarted, all the agents in the environment cannot reconnect <juju-core:New> <https://launchpad.net/bugs/1236662> | 00:35 |
| davecheney | https://bugs.launchpad.net/juju-core/+bug/1237151 | 00:46 |
| _mup_ | Bug #1237151: set rpc logging to TRACE and agents back to DEBUG by default <papercut> <juju-core:Triaged by thumper> <https://launchpad.net/bugs/1237151> | 00:46 |
| davecheney | https://bugs.launchpad.net/juju-core/+bug/1237163 | 01:28 |
| _mup_ | Bug #1237163: juju deploys wrong machine when no machine matching constraints is available <juju-core:New> <https://launchpad.net/bugs/1237163> | 01:28 |
| davecheney | sidnei: ping | 03:26 |
| wallyworld | jam: i just kicked the landing bot in the bollocks. it seems that it had not run from about 08-10-13 15:20 UTC. i deleted the flock file in ~tarmac. what is a little confusing is that the lock file was dated jul 29. | 05:59 |
| jam | wallyworld: it isn't created, what is happening is a test run left a process running that has the file open | 06:00 |
| wallyworld | why 29th july? | 06:00 |
| jam | wallyworld: most of the time it is a rogue mongodb process | 06:00 |
| jam | wallyworld: because we don't ever delete the file | 06:01 |
| jam | it is just used with flock | 06:01 |
| wallyworld | sure. but i would have thought it would have been dated around 8th oct or whatever | 06:01 |
| jam | wallyworld: the "flock" model is that you open a file handle, and then flock it, and then all children that are spawned from that process have the file handle in their process | 06:01 |
| jam | so until they all die | 06:01 |
| jam | the lock is held | 06:01 |
| jam | wallyworld: and when the process dies | 06:02 |
| jam | it doesn't *delete* the file | 06:02 |
| jam | we just keep using the same file | 06:02 |
| jam | wallyworld: it isn't open(O_EXCLUSIVE) | 06:02 |
| jam | it is flock(EXISTING_FILE) | 06:02 |
| jam | though that file will be created if it doesn't exist | 06:02 |
| wallyworld | ok | 06:03 |
| jam | wallyworld: anyway, the *actual* fix is to kill 16262 whicth is a mongod process that has been running for over 6hours cpu time | 06:03 |
| wallyworld | ah ok | 06:03 |
| jam | wallyworld: so if the bot looks stalled, I generally bring up top, and look for a stale process | 06:03 |
| wallyworld | ok, will do that next time | 06:03 |
| dimitern | mornin' all | 06:44 |
| fwereade | dimitern, heyhey | 06:45 |
| fwereade | wallyworld, jam, axw: mornings | 06:45 |
| wallyworld | hello | 06:45 |
| jam | morning fwereade | 06:46 |
| rogpeppe | fwereade, axw, jam, wallyworld, dimitern: mornin' all | 06:48 |
| fwereade | rogpeppe, heyhey | 06:48 |
| wallyworld | what he said | 06:49 |
| axw | morning all :) | 06:54 |
| dimitern | it's very jolly this morning it seems :) | 06:55 |
| * fwereade was just looking at the constraints matching logic as referenced by davecheney, and What The Fuck | 06:57 | |
| fwereade | it looks like we ignore the heuristics to begin with | 06:57 |
| fwereade | so ofc they don't get used | 06:57 |
| fwereade | and if we fail we *then* just apply the heuristic and give them, eh, some shit, whatever, who cares | 06:57 |
| fwereade | certainly nothing that matches the constraints though | 06:58 |
| jam | fwereade: so I think the only heuristics we give are for mem, right? So you could argue we could check if there is a mem constraint, and if not set it. | 06:59 |
| fwereade | yeah | 07:00 |
| jam | fwereade: though what *should* we do if we can't find a match? | 07:00 |
| fwereade | that's what we did when I wrote it | 07:00 |
| jam | just fail, I guess | 07:00 |
| fwereade | we should apply the heuristics *first* | 07:00 |
| fwereade | try to get something matching user requests + our own preferences | 07:00 |
| fwereade | (if there's no conflict) | 07:00 |
| fwereade | if that doesn't work, just use the user requirements alone | 07:00 |
| fwereade | if that doesn't work, FAIL | 07:00 |
| axw | fwereade: so I just noticed today that the 1.16 release milestone is set for <24h from now | 07:10 |
| axw | if you didn't realise, null provider is pretty hobbled at the moment | 07:10 |
| axw | just proposing the last fix now, but I guess it's too late to get it in (as well as get the others approved) | 07:11 |
| fwereade | axw, ok, can you give me the really high-level version please? | 07:15 |
| axw | fwereade: sure, I'll just get the bug numbers to reference | 07:16 |
| axw | fwereade: when bootstrapping the null provider, the prompt for "ssh sudo ..." is hidden (https://bugs.launchpad.net/juju-core/+bug/1235716) | 07:18 |
| _mup_ | Bug #1235716: sshstorage does not display sudo prompt <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1235716> | 07:18 |
| axw | fwereade: when bootstrapping into a machine that already has tools, bootstrap fails completely (https://bugs.launchpad.net/juju-core/+bug/1235717) | 07:19 |
| _mup_ | Bug #1235717: null provider bootstrap fails with error about sftp scheme <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1235717> | 07:19 |
| axw | fwereade: if you try to bootstrap a machine whose series doesn't match default-series, it thinks it can't find tools (https://bugs.launchpad.net/juju-core/+bug/1236691) | 07:19 |
| _mup_ | Bug #1236691: null provider bootstrap fails if default-series does not match target <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1236691> | 07:19 |
| axw | the sudo one is easy to fix easily, but maybe not if ctx.Stdin/Stdout etc. have to be threaded through juju | 07:20 |
| axw | the others have fixes but are a bit more involved | 07:20 |
| axw | fwereade: they all have workarounds, so they're not crippling, but do lead to poor user experience | 07:21 |
| fwereade | axw, tbh I am not inclined to make those part of 1.16 -- we've still got the null provider documented as experimental, right? | 07:22 |
| fwereade | axw, so if they have workarounds I really don't want to take the instability risk today | 07:23 |
| axw | fwereade: we do, though that doc is still pending approval.. | 07:23 |
| axw | fwereade: fair call | 07:23 |
| fwereade | axw, is that into juju-core/docs? sorry, I haven't really been following reviews there | 07:24 |
| axw | fwereade: tho the sudo one is good risk/reward ratio I think | 07:24 |
| axw | fwereade: yeah | 07:24 |
| axw | fwereade: https://code.launchpad.net/~axwalk/juju-core/manual-provisioning-docs/+merge/188501 | 07:24 |
| fwereade | axw, for doc reviews that languish, it would probably be best to lean on evilnickveitch, marcoceppi_, or jcastro | 07:25 |
| axw | fwereade: okey dokey, ta | 07:25 |
| fwereade | wallyworld, I see you have signing ones up, I'm about to take a look at those (also need breakfast... :/) -- any joy re sync-tools to old location? or should I get someone else to take a look? must be late for you now | 07:26 |
| * fwereade hears that a wonderful person has made him toast, goes and eats it, bbs | 07:26 | |
| evilnickveitch | axw, fwereade it is in the queue! there is some other complicated stuff landing at the moment | 07:27 |
| axw | cool, thanks evilnickveitch | 07:27 |
| axw | sync-tools is reviewed, fwereade | 07:27 |
| fwereade | axw, sorry, which one? sync-tools-to-old-location-in-1.16? | 07:34 |
| fwereade | axw, wallyworld: that looks like it's not fixed | 07:34 |
| fwereade | axw, all I see is an MP for juju-core | 07:34 |
| fwereade | wallyworld, ^ | 07:34 |
| axw | fwereade: ah sorry, just in juju-core | 07:34 |
| fwereade | axw, wallyworld: nothing for juju-core/1.16 | 07:35 |
| fwereade | axw, wallyworld: that should surely *not* be in trunk at all, should it? | 07:35 |
| axw | good point | 07:36 |
| fwereade | axw, wallyworld: trunk is just for code >= 1.17, and 1.16 is the last time we need that code | 07:36 |
| fwereade | wallyworld, can we back that out and retarget it please? | 07:42 |
| wallyworld | fwereade: i thought trunk was still for 1.16 | 07:47 |
| wallyworld | have we branched | 07:47 |
| fwereade | wallyworld, we branched from 1.15.1 to the 1.16 series | 07:47 |
| wallyworld | oh ok. i didn't realise. was there an email | 07:47 |
| fwereade | wallyworld, probably not, I'd thought it was already "normal" :( | 07:48 |
| wallyworld | i would have thought we only branch when 1.16 is cut | 07:48 |
| wallyworld | i guess there's two opinions on when to branch - before or after release | 07:49 |
| fwereade | wallyworld, I'd prefer to keep putative releases isolated from the hurly-burly of trunk | 07:49 |
| axw | wallyworld: I think 1.15.1 is kinda like 1.16.0 rc1 | 07:49 |
| wallyworld | fair enough. can be argued either way :-) | 07:50 |
| fwereade | wallyworld, yeah, what axw said | 07:50 |
| wallyworld | branching early means more work. branching later means more care needed | 07:50 |
| fwereade | wallyworld, axw: (if we had a mechanism that allowed for things like "RC1" it would be great, wouldn't it) | 07:51 |
| axw | ;) | 07:51 |
| wallyworld | fwereade: so all my 4 or so branches i have up - i now need to land in two places :-( | 07:51 |
| wallyworld | well, all except the one above | 07:51 |
| axw | fwereade: I thought I had the 1.16.0 tests fixed, but apparently they fail on precise .. still looking into it | 07:51 |
| fwereade | axw, yeah, I was looking at that and I'm afraid I... just don't really understand wtf is going on there | 07:52 |
| axw | fwereade: something to do with the metadata, I don't know where it's coming from tho. anyway, just letting you know I'm still looking at it | 07:52 |
| fwereade | axw, I see no evidence that what we're doing makes any sense in the first place in either dev or releasemode | 07:53 |
| axw | fwereade: yeah... quite a burden for what we get out of it | 07:53 |
| fwereade | axw, wallyworld: any idea why we have two changes to upload, called at different times depending on the provider kind (which the command should not know anyway)? | 07:54 |
| fwereade | s/changes/chances/ | 07:54 |
| wallyworld | where is this in the code? | 07:54 |
| fwereade | wallyworld, cmd/juju/bootstrap | 07:54 |
| axw | fwereade: you mean how "local" always forces upload? | 07:54 |
| fwereade | .go | 07:54 |
| fwereade | axw, yeah, that is crack for a start | 07:54 |
| fwereade | axw, and then later there's stuff to auto-upload *anyway* | 07:55 |
| fwereade | axw, so it's not just a massive layering violation, it's also completely unnecessary | 07:55 |
| fwereade | I *think* | 07:55 |
| wallyworld | fwereade: the local provider hack was done by tim for a reason i can;t recall, but i think it had to do with making tools available for the lxc container | 07:56 |
| wallyworld | maybe because we "know" there are never tools available for the local provider? | 07:56 |
| wallyworld | so we need to force an upload? | 07:57 |
| fwereade | wallyworld, it was done by tim because oscon, and never tidied up because I wasn't able to convince him it was crack until iom, at which point more things were on our plate | 07:57 |
| fwereade | wallyworld, but it's not even upload-tools | 07:57 |
| fwereade | wallyworld, he completely changed the meaning of upload-tools | 07:57 |
| fwereade | wallyworld, so now upload-tools and sync-tools are just kinda different ways to do the same thing | 07:57 |
| wallyworld | upload tools calls sync tools | 07:57 |
| wallyworld | because sync tools knows how to deal with the metadata etc | 07:58 |
| fwereade | wallyworld, right, but it's not actually syncing -- and while they used to be separate they arenow so entangled that I have no idea wtf to do about them | 08:00 |
| wallyworld | i need to look at the code in more detail to re-grok it | 08:01 |
| fwereade | wallyworld, anyway, I don't mean to whine at you, you have been a powerful force for sanity | 08:04 |
| davecheney | what is the add-machine syntax to make a new lxc continaer ? | 08:04 |
| TheMue | morning | 08:06 |
| rogpeppe | davecheney: lxc:1 ? | 08:07 |
| davecheney | rogpeppe: muchas gracias | 08:08 |
| wallyworld | fwereade: no problem, feel free to vent :-) | 08:08 |
| * fwereade breathes deeply for a moment ;) | 08:09 | |
| fwereade | wallyworld, ok, so, I will be reviewing your branches for both trunk and 1.16 I think | 08:10 |
| wallyworld | fwereade: yeah, everything that's up now is intended for 1.16 | 08:10 |
| wallyworld | fwereade: i'm also doing the plugin juju_env one for davecheney | 08:11 |
| fwereade | wallyworld, can you back the sync-tools fix out of trunk and get it into 1.16 now while I focus on those for a bit? | 08:11 |
| wallyworld | sure. just about to have dinner though, so after that | 08:11 |
| wallyworld | fwereade: i assume the bot handles proposals against 1.16 ? | 08:11 |
| fwereade | wallyworld, yeah, it did for me yesterday anyhow ;p | 08:11 |
| wallyworld | cool :-) | 08:12 |
| davecheney | % juju add-machine 0 | 08:14 |
| davecheney | error: malformed container argument "0" | 08:14 |
| davecheney | WHUT ? | 08:14 |
| fwereade | wallyworld, https://codereview.appspot.com/14524043/ reviewed | 08:16 |
| fwereade | davecheney, add-machine lxc:0 | 08:16 |
| fwereade | davecheney, "0" already exists, can't be added again | 08:16 |
| wallyworld | fwereade: the empty key is moot because it won't be used cause there's no signed tools metadata | 08:16 |
| wallyworld | fwereade: and i plan to get the key off curtis or ben howard tonight (i hope) | 08:17 |
| fwereade | wallyworld, ha, ofc -- do we have one incoming from sinzui? | 08:17 |
| fwereade | wallyworld, ok, cool | 08:17 |
| fwereade | wallyworld, LGTM for both then | 08:18 |
| wallyworld | \o/ thanks :-) | 08:18 |
| wallyworld | i'll let you mark them as such and i'll land to trunk and 1.16 after dinner | 08:19 |
| fwereade | axw, https://code.launchpad.net/%7Eaxwalk/juju-core/lp1235717-refactor-simplestreams-writemetadata/+merge/189767 is for one of the bugs you linked above, right? I'll be ignoring it for a bit then | 08:19 |
| axw | fwereade: yes indeed | 08:19 |
| fwereade | wallyworld, likewise https://codereview.appspot.com/14530043/ -- very clean, ty | 08:21 |
| jam | davecheney: https://bugs.launchpad.net/bugs/1237151 thumper is away this week | 08:30 |
| _mup_ | Bug #1237151: set rpc logging to TRACE and agents back to DEBUG by default <papercut> <juju-core:Triaged by thumper> <https://launchpad.net/bugs/1237151> | 08:30 |
| axw | fwereade: I've been leaving this https://codereview.appspot.com/14218044/ until I rework according to your comment | 08:33 |
| axw | not necessary for 1.16.0 right? | 08:33 |
| fwereade | axw, I don't think it is, AFAIK everything works since we took the daily streams out of azure | 08:34 |
| axw | fwereade: yeah azure is fine. it's just a potential surprise for private cloud users | 08:35 |
| axw | no wait.. | 08:35 |
| axw | it's fine for htem at the moment | 08:35 |
| axw | never mind | 08:35 |
| fwereade | axw, I *think* it's fine, yeah | 08:35 |
| fwereade | axw, hmm, what do we need admin-secret and ca-private-key locally for? | 08:36 |
| davecheney | jam: i wanted to take a crack this arvo | 08:37 |
| davecheney | but got pulled into support | 08:37 |
| jam | fwereade: we need admin-secret in order to connect to mgo for the client, don't we ? | 08:37 |
| axw | fwereade: yeah, just for the cli | 08:38 |
| jam | davecheney: had to look it up: http://www.urbandictionary.com/define.php?term=arvo | 08:38 |
| jam | :) | 08:38 |
| axw | s/davecheney/davo/ | 08:39 |
| fwereade | axw, given that we've connected already, can we not get away without them? | 08:42 |
| axw | fwereade: heh, that would make sense :) I only put it in there because it was failing... I'll take a closer look at why | 08:42 |
| davecheney | jam: i'm sorry, it must be my accent | 08:48 |
| davecheney | oh crap | 08:48 |
| davecheney | add-machine lxc:1 created a new container, the juju didn't use it !! | 08:49 |
| jam | davecheney: is it a constraint issue? | 08:49 |
| jam | (not enough mem in the instance, etc) | 08:49 |
| axw | fwereade: 1.16.0 tests should be fixed now, do you want to review the CL before I land on 1.16 as well? https://codereview.appspot.com/14521054/ | 08:54 |
| fwereade | axw, I think I need you to explain how the change works, and what happens in dev vs release mode | 08:55 |
| axw | fwereade: documented here https://bugs.launchpad.net/juju-core/+bug/1237123/comments/1 | 08:56 |
| _mup_ | Bug #1237123: cannot set version to 1.16 <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1237123> | 08:56 |
| fwereade | axw, all I know is that I don't like or trust what we do today and so I'm suspcious of tweaks to it ;) | 08:56 |
| axw | if that doesn't explain it, lemme know | 08:56 |
| fwereade | axw, ah great, I will try to understand :) | 08:56 |
| TheMue | shit | 08:58 |
| TheMue | wife just detected that a wheel of the car is flat :( | 08:59 |
| fwereade | axw, // If Dev is already true, leave it alone. seems to be conditional on dev being *false*-- am I just misreading or something? | 08:59 |
| fwereade | axw, ohwait I see | 09:00 |
| axw | fwereade: sorry, did I add to the subtlety? :) | 09:00 |
| axw | fwereade: you're completely right about admin-secret not being needed there. my test broke because I took admin-secret out of my environments.yaml... but that branch doesn't have the change needed to Prepare admin-secret | 09:01 |
| fwereade | axw, I guess I still don't get what MinorVersion being set implies about devness | 09:01 |
| fwereade | axw, ok,but Prepare should be completely irrelevant here, right? nothing fails without admin-secret exxceptbootstrap AFAIK | 09:02 |
| axw | fwereade: yeah, my bootstrap command was failing because admin-secret was blank. | 09:03 |
| axw | hmm | 09:03 |
| axw | but why not when I copied it.. wtf | 09:03 |
| axw | fwereade: umm, yeah not really sure why re the MinorVersion either tbh | 09:03 |
| axw | that is the same as before | 09:04 |
| fwereade | axw, yeah, I can't figure it out either, and that's what scares me about the whole business | 09:07 |
| fwereade | axw, changing things without fully understanding them | 09:08 |
| fwereade | axw, SyncContext looks like it's got altogether too many magical meanings for the different fields | 09:09 |
| fwereade | TheMue, I don't suppose you can shed any light on this? (on your return, I guess...) | 09:10 |
| axw | fwereade: I'll have a look thru the log to see if I can figure anything out... | 09:11 |
| fwereade | axw, thanks | 09:11 |
| axw | fwereade: wallyworld changed this at r1807 | 09:13 |
| axw | wallyworld: ... any hints on the rationale for tying SyncTools.Dev to MinorVersion? | 09:14 |
| axw | fwereade: I think I must've had a stale .jenv file when I was testing the prechecker stuff | 09:16 |
| axw | one with a admin-secret in it | 09:16 |
| axw | so yeah... that code can/will go | 09:16 |
| TheMue | fwereade: eh, sorry, didn't follow. now have to wait for a technician, so I can continue here. | 09:19 |
| TheMue | fwereade: what's the problem I shall investigate? | 09:19 |
| fwereade | TheMue, we don't really know what's going on with SyncContext and wondered if you knew about it | 09:25 |
| fwereade | TheMue, https://codereview.appspot.com/14521054/patch/5001/6004 -- "syncContext.MinorVersion != -1" | 09:26 |
| jam | wallyworld: I just feel pretty strongly after some time that juju-metadata *shouldn't* be a plugin | 09:27 |
| rogpeppe | fwereade: i responded to your review here: https://codereview.appspot.com/14395043 | 09:27 |
| TheMue | fwereade: off the cuff not, but will take a look | 09:27 |
| wallyworld | jam: i agree. i never wanted it as a plugin in the first place | 09:28 |
| rogpeppe | jam: i agree too. | 09:28 |
| rogpeppe | jam: i never really understood the "too many commands" rationale for making it a plugin | 09:28 |
| wallyworld | axw: let me think - i can't recall off hand | 09:28 |
| jam | rogpeppe: especially given it is installed by default, so we still have those command | 09:28 |
| jam | s | 09:28 |
| rogpeppe | jam: ha, yes | 09:29 |
| wallyworld | jam: i think juju help commands was the reason | 09:29 |
| wallyworld | people didn;t want the list of commands to be tool large | 09:29 |
| jam | wallyworld: we could easily hide commands in help commands if we wanted to | 09:29 |
| jam | seems a better fix | 09:29 |
| rogpeppe | wallyworld: how do you mean? | 09:29 |
| jam | rogpeppe: didn't want to overload people looking for help with X with too many commands that might be useful perhaps | 09:29 |
| rogpeppe | jam: why would we want to hide it? it's a perfectly good command. | 09:29 |
| jam | I think that falls into "juju help" being opinionated | 09:29 |
| wallyworld | rogpeppe: well, juju help commands shows a list of all commands, and people didn't want that list to be too large | 09:29 |
| jam | and "juju help commands" giving everything | 09:30 |
| wallyworld | rogpeppe: i don't buy that argument. i was "forced" to make it a plugin | 09:30 |
| wallyworld | axw: it had to do with the legacy tools search functionality and the change to exact minor version matching | 09:31 |
| wallyworld | from memory | 09:31 |
| wallyworld | so, the old functionality searched for all tools with major version = 1 (say) | 09:31 |
| wallyworld | and this excluded dev versions i think | 09:32 |
| jam | axw: MinorVersion == odd means it is a dev version | 09:32 |
| jam | that has been the internal case for a long time | 09:32 |
| jam | long before wallyworld came along (IIRC) | 09:32 |
| wallyworld | and with the minor version matching, if we wanted a match for 1.13 (say), we needed to set dev = true | 09:32 |
| axw | jam: the code is just checking MinorVersion != -1 (i.e. minor specified) | 09:32 |
| jam | the build > 0 was a newer thing, but it was intended for "--upload-tools" to always generate a Dev version (vs oficial release) | 09:33 |
| axw | wallyworld: ok, I think I understand that | 09:33 |
| jam | axw: ah, see wallyworld's comment. | 09:33 |
| axw | wallyworld: perhaps it should be checking if it's -1 and odd then. | 09:33 |
| jam | if you want to match exact minor version, and you're running dev you have to search dev | 09:33 |
| axw | !=-1 | 09:33 |
| jam | axw: -1 is always odd, right? :) | 09:34 |
| axw | err yeah :) | 09:34 |
| wallyworld | -1 is the default | 09:34 |
| wallyworld | arg value | 09:34 |
| axw | right, so -1 is special | 09:34 |
| wallyworld | to distiguish from 0 | 09:34 |
| axw | fwereade: answer is ^^ | 09:35 |
| wallyworld | cause someone could search for major.minor = 2.0 (say) | 09:35 |
| fwereade | wallyworld, ah,I didn't see where -1was the default -- I saw a SyncContext somewhere that didn'tseemto set them,but then I see there's sometransformation thereof above | 09:36 |
| fwereade | wallyworld, axw: but, OK, I think I'm convinced by wallyworld's LGTM | 09:37 |
| wallyworld | fwereade: i did a LTGM on inspection only. it seems ok. but the proof is does it fix the bug and not cause regressions. | 09:38 |
| fwereade | wallyworld, axw: I think there's some call for a tech-debt bug around bootstrap at least though, it shouldn't be this hard to follow | 09:38 |
| wallyworld | +1000 | 09:38 |
| wallyworld | fwereade: it has all sort of "evolved" incrementally into a mess | 09:38 |
| fwereade | wallyworld, yeah, I know how that happens ;) | 09:38 |
| wallyworld | it needs to be properly refactored | 09:38 |
| fwereade | wallyworld, and, hey, we got value from the tech debt, but we should pay it down before it cripples us | 09:39 |
| * fwereade does a bit more deep breathing | 09:39 | |
| jam | axw: bug 1237295, are there prechecker tests that could actually become stale in the current system? Stuff like not supporting LXC won't change without a code change anyway | 09:39 |
| _mup_ | Bug #1237295: state.Prechecker/Environment in machine-agent may become stale <juju-core:Triaged> <https://launchpad.net/bugs/1237295> | 09:39 |
| wallyworld | yep. the short term focus was saucy and simplestreams working | 09:39 |
| axw | jam: no | 09:39 |
| wallyworld | axw: i'd love it if the change were tested manually to make sure it all hangs together | 09:39 |
| axw | not currently a problem | 09:39 |
| fwereade | axw, jam: Prechecker isn't merged yet is it? | 09:39 |
| axw | fwereade: no | 09:40 |
| axw | that's in preparation for my repropose | 09:40 |
| fwereade | axw, jam: it *would* be nice to get that into 1.16 actually, because someone already got confused creating a container on ec2 | 09:40 |
| axw | wallyworld: I have done a brief test, I'll do some more in a bit | 09:40 |
| fwereade | axw, jam: for some reason I am a little bit scared of it, but I can't figure out why | 09:40 |
| fwereade | axw, thanks, +100 | 09:41 |
| jam | fwereade: it does feel a bit like something that could screw up and not let you do something that you *should* be able to do | 09:41 |
| axw | fwereade: probably because it hunkers down into state ;) | 09:41 |
| jam | so I'm reasonably ok giving it some more ongoing testing in a 1.17 series | 09:41 |
| axw | it does seem a bit late to put it in imho | 09:42 |
| jam | fwereade: so here's a destroy-environment issue | 09:42 |
| fwereade | jam, oh yes... | 09:43 |
| jam | "juju bootsrap" tries to create a private bucket, fills in that detail into the .jenv. But you're on a private cloud so you have to upload your own image metadata | 09:43 |
| jam | so you shove it in your private bucket | 09:43 |
| jam | and then bootstrap again | 09:43 |
| jam | but bootstrap doesn't work because of the empty provider-state file left around the first time | 09:43 |
| jam | so you "juju destroy-environment" | 09:43 |
| jam | but now it lost what private bucket you were using | 09:43 |
| jam | so bootstrap can't work | 09:43 |
| fwereade | TheMue, ahh, here's dstroppa, would you have a word with him about the OS X build? and how practical it actually is to run non-statey provider tests on non-ubuntu? | 09:44 |
| jam | because it won't be able to find the metadata you just uploaded to your private bucket | 09:44 |
| davecheney | fwereade: juju deploy --to lxc:1 mysql just did this too me | 09:44 |
| davecheney | http://paste.ubuntu.com/6213065/ | 09:44 |
| davecheney | but I cannot reproduce the problem | 09:44 |
| davecheney | fwereade: can you think how this is possible | 09:44 |
| davecheney | and if I should | 09:44 |
| davecheney | a. raise an issue | 09:44 |
| davecheney | b. pretend it didn't happen | 09:45 |
| jam | davecheney: the syntax "--to lxc:1" means to create a new LXC on machine 1 and deploy to it | 09:45 |
| jam | davecheney: "--to 1/lxc/1" is the way to deploy to the first LXC container on machine 1 | 09:45 |
| jam | sorry | 09:45 |
| jam | 1/lxc/0 | 09:45 |
| jam | davecheney: so *as designed* | 09:45 |
| jam | though confusing, I understand | 09:46 |
| davecheney | jam: explain this | 09:46 |
| davecheney | http://paste.ubuntu.com/6213071/ | 09:46 |
| jam | "lxc:1" is always "a new LXC on 1" | 09:46 |
| davecheney | ^ destoryed env | 09:46 |
| davecheney | ran it again | 09:46 |
| davecheney | got this | 09:46 |
| davecheney | noce 1/lxc/0 | 09:46 |
| davecheney | note | 09:46 |
| jam | davecheney: local provider? | 09:46 |
| davecheney | jam: hells no | 09:46 |
| davecheney | ec2 | 09:46 |
| jam | bug #1237259 | 09:46 |
| _mup_ | Bug #1237259: juju status reports 'missing' instance-state when not run as root <juju-core:New> <https://launchpad.net/bugs/1237259> | 09:46 |
| davecheney | jam: false | 09:46 |
| jam | ah, should have been clear about the ec2* addresses | 09:46 |
| davecheney | jam: look at the machine id's | 09:47 |
| jam | davecheney: "juju-machine-1" ? | 09:47 |
| davecheney | mysql/0: | 09:47 |
| davecheney | agent-state: pending | 09:47 |
| davecheney | machine: 1/lxc/0 | 09:47 |
| jam | davecheney: that is what it is supposed to be | 09:47 |
| davecheney | yes, but the first time it wasn't | 09:47 |
| jam | 1/lxc/0 is the first LXC instance on macihne 1 | 09:47 |
| davecheney | http://paste.ubuntu.com/6213065/ | 09:47 |
| jam | davecheney: so the first time you did "juju add-machine lxc:1" which means create a new LXC on machine 1 and add it | 09:48 |
| jam | davecheney: then you did "juju deploy --to lxc:1" | 09:48 |
| jam | which means "create *another* lxc and deploy to it" | 09:48 |
| fwereade | davecheney, so if you ran exactly the same commands and got those different results there is surely a problem -- but what exactly did you run? | 09:48 |
| jam | if you did "juju deploy --to 1/lxc/0" it would have used the existing container | 09:48 |
| davecheney | fwereade: | 09:48 |
| davecheney | juju bootstrap | 09:49 |
| davecheney | juju add-machine | 09:49 |
| davecheney | juju deploy --to lxc:1 | 09:49 |
| davecheney | juju deploy --to lxc:1 mysql | 09:49 |
| davecheney | juju status | 09:49 |
| davecheney | it's like sometimes juju doesn't find the container it expected | 09:49 |
| davecheney | and creates another one | 09:49 |
| fwereade | davecheney, `--to lxc:1` always means "create a new container" | 09:50 |
| davecheney | fwereade: yes, so mysql/0 should have been created on 1/lxc/0 | 09:50 |
| davecheney | but sometimes machine 1 gets a second container, 1/lxc/1 | 09:50 |
| fwereade | davecheney, you were talking about doing explicit `add-machine lxc:1` at one stage,I thought? | 09:51 |
| davecheney | fwereade: i'm trying to reconfirm tat | 09:51 |
| davecheney | in which case | 09:51 |
| davecheney | add-machine lxc:1 and deploy --to lxc:1 mean _different_ things ? | 09:51 |
| jam | davecheney: they *both* mean create a new LXC and use it for this operation | 09:52 |
| jam | create a *new* LXC | 09:52 |
| fwereade | davecheney, they mean exactly the same thing -- create a new lxc container on machine 1 | 09:52 |
| jam | is key here | 09:52 |
| davecheney | oh | 09:52 |
| davecheney | riht | 09:52 |
| davecheney | i understand now | 09:52 |
| davecheney | lxc:machine-number | 09:52 |
| davecheney | not lxc:container-number | 09:52 |
| fwereade | davecheney, yeah | 09:52 |
| davecheney | right | 09:52 |
| davecheney | sorry for the noise | 09:52 |
| davecheney | am trying to trace a problem that jpds hit deploying mysql into a container | 09:53 |
| TheMue | sorry guys, i'm off due to the defect for some time, will continue later :( | 09:53 |
| davecheney | got sidetracked | 09:53 |
| fwereade | davecheney, it's the specific form of a more general <pseudo-provider>:<location> syntax | 09:53 |
| fwereade | davecheney, that should always imply new machine creation | 09:53 |
| davecheney | fwereade: right | 09:54 |
| davecheney | thanks | 09:54 |
| fwereade | davecheney, np, it's nice to find things that aren't bugs ;) | 09:54 |
| * davecheney goes back to tracing real bugs | 09:56 | |
| jam | wallyworld: fetchToolsHash assumes it can just http.Get any URL, it doesn't go via any of the DataSource objects | 09:56 |
| * wallyworld looks | 09:56 | |
| jam | so it (a) breaks with ssl-hostname-verification: false and (b) will fail if we don't have readable stuff, but I guess tools always need readable stuff | 09:56 |
| rogpeppe | jam: axw is on the road to fixing that | 09:57 |
| jam | rogpeppe: fixing what part ? | 09:57 |
| rogpeppe | jam: the fact that SyncTools uses http.Get at all | 09:57 |
| wallyworld | jam: it was assumed that tools as used for calculating the hash would always come from public urls | 09:57 |
| axw | rogpeppe: there's a CL up for that btw | 09:57 |
| wallyworld | that was previously the case also with sync tools i think | 09:57 |
| rogpeppe | axw: oh cool | 09:57 |
| jam | wallyworld: right, even in "private-bucket" we have to set it up as world-readable so that any system can read the tools | 09:57 |
| jam | *but* | 09:57 |
| jam | when you have to set ssl-hostname-verification: false | 09:58 |
| jam | that breaks your http.Get | 09:58 |
| wallyworld | oh | 09:58 |
| jam | wallyworld: if you used the DataSource associated with the URL I've configured it to have the right settings | 09:58 |
| rogpeppe | wallyworld: i don't think that can ever have been the case - the fetchToolsHash logic sometimes works on the target bucket which could always have been private | 09:58 |
| jam | rogpeppe: it can't | 09:58 |
| wallyworld | jam: with private bucket, we don't need world readable as storage URL() is used which gives a world readable url | 09:58 |
| rogpeppe | axw: link | 09:58 |
| jam | rogpeppe: because you always have to be able to download the tools for a machine that doesn't have any creds yet | 09:58 |
| axw | rogpeppe: un moment | 09:59 |
| jam | wallyworld: not on openstack | 09:59 |
| jam | because openstack | 09:59 |
| rogpeppe | wallyworld: it doesn't for the local provider | 09:59 |
| jam | doesn't have signed urls | 09:59 |
| axw | rogpeppe: https://codereview.appspot.com/14527043/ | 09:59 |
| jam | wallyworld: because TemporaryURL isn't by-default enabled on all the Openstack instances we know of | 09:59 |
| wallyworld | jam: yes, but private bucket is not intended to be used that way for openstack | 09:59 |
| rogpeppe | wallyworld: SyncTools should not use URL | 09:59 |
| jam | wallyworld: inspect the code, private-bucket is always made world readable for openstack | 09:59 |
| axw | rogpeppe: this doesn't change the SyncTools bits (much, yet) | 09:59 |
| wallyworld | jam: oh yeah | 09:59 |
| rogpeppe | axw: looking | 09:59 |
| jam | if you use "--upload-tools" it must be | 09:59 |
| jam | because that's where we write them, IIRC | 10:00 |
| wallyworld | rogpeppe: why? sync tools can get tools from a local directory if it wants | 10:00 |
| rogpeppe | wallyworld: because it's got the Storage with a Get method - URL is intended for sending across networks, which we're not doing here. | 10:01 |
| wallyworld | the data can come across a network | 10:01 |
| wallyworld | but if we have a storage abstraction, then we can use that | 10:02 |
| jam | rogpeppe: so we know that we will eventually need to get the data via URL | 10:03 |
| jam | because we do "wget" in cloud-init | 10:03 |
| rogpeppe | jam: that's not an issue for SyncTools though | 10:04 |
| rogpeppe | jam: and in particular for the target storage of the null provider, we don't have any url | 10:05 |
| davecheney | you know, if the cloud images came with lxc and the precise template already downloaded | 10:07 |
| davecheney | things would bootup a lot faster | 10:07 |
| rogpeppe | davecheney: +1 | 10:08 |
| wallyworld | fwereade: this one will make davecheney happy https://codereview.appspot.com/14499044 | 10:09 |
| jam | davecheney: well we don't actually support lxc on ec2 today anyway... :) | 10:10 |
| davecheney | jam: you cna make machines | 10:11 |
| davecheney | that bit works | 10:11 |
| axw | davecheney: not for long :) | 10:11 |
| davecheney | wallyworld: that *does* make me happy | 10:11 |
| davecheney | axw: you poo | 10:11 |
| jam | davecheney: axw is subitting a patch to disable it | 10:11 |
| davecheney | hang on | 10:11 |
| jam | since we can't make them routable yet | 10:11 |
| davecheney | starting lxc conatiners on precise is ultra fail | 10:12 |
| fwereade | wallyworld, LGTM | 10:12 |
| davecheney | i don't know why I'm even trying | 10:12 |
| jam | davecheney: 12.04.03 + cloud-tools should create nice LXC instances, I believe | 10:12 |
| wallyworld | fwereade: that was for 1.16 as well as trunk | 10:12 |
| davecheney | jam: creating hem works fine | 10:12 |
| jam | davecheney: and the ec2 images have been updated for 12.04.03 (which defaults to the raring HWE kernel) | 10:12 |
| davecheney | dealig with the kernel oopses is the tricky part | 10:12 |
| axw | fwereade, wallyworld: sync-tools and --upload-tools both work fine in my testing | 10:12 |
| fwereade | wallyworld, so I assumed :) | 10:12 |
| davecheney | jam: ORLEY ? | 10:12 |
| wallyworld | axw: awesome! | 10:13 |
| axw | good to merge into 1.16.0? | 10:13 |
| jam | davecheney: 12.04.03 defaults to the raring HWE kernel from reports of other people | 10:13 |
| fwereade | axw, go for it | 10:13 |
| davecheney | jam: testing | 10:13 |
| axw | jam: indeed it does | 10:13 |
| axw | I installed the other day | 10:13 |
| axw | fwereade: ta | 10:13 |
| davecheney | GOD DANMING | 10:13 |
| * wallyworld has about 4 branches to merge into 1.16 and trunk, and another to back out of trunk and put into 1.16. sigh | 10:13 | |
| davecheney | EC2 iS SO FUCKING SLOW | 10:13 |
| davecheney | WHY IS EVERYTHING SLW | 10:14 |
| fwereade | axw, jam: about Prechecker... how bad would it be to merge a 1.16-only hack that just blocks containers in non-maas environments only? in the addMachine gubbins somewhere? | 10:14 |
| davecheney | i feel my life draining away every day I use this product | 10:14 |
| fwereade | axw, jam: heavily flagged as crazy and dangerous, and *not* merged into trunk? | 10:14 |
| davecheney | jam: axw | 10:14 |
| davecheney | i call bullsht | 10:14 |
| davecheney | ubuntu@ip-10-248-78-68:~$ uname -a | 10:14 |
| davecheney | Linux ip-10-248-78-68 3.2.0-54-virtual #82-Ubuntu SMP Tue Sep 10 20:31:18 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux | 10:14 |
| wallyworld | davecheney: could be worse, azure sucks balls | 10:14 |
| * bigjools giggles | 10:14 | |
| davecheney | wallyworld: true, the universe can always invent a slower mousetrap | 10:14 |
| jam | wallyworld: is that a bad thing? | 10:14 |
| jam | I guess it depends what you are looking for from your provider | 10:15 |
| wallyworld | jam: depends if you like having your balls sucked | 10:15 |
| fwereade | davecheney, *we* can invent slower mousetraps any time you like! sometimes without even trying | 10:15 |
| wallyworld | bigjools does | 10:15 |
| bigjools | nom | 10:15 |
| jam | fwereade: it seems like the whole reason Prechecker was created was to handle the container issue, not landing it just to land a hackish version of it doesn't sound great. | 10:15 |
| fwereade | jam, it's bigger than that | 10:16 |
| davecheney | anywa, axw, you're full of crap | 10:16 |
| jam | wallyworld: which is why bigjools worked on the azure provider. It all fits together in the end. | 10:16 |
| davecheney | no HWE kernel | 10:16 |
| wallyworld | lol | 10:16 |
| axw | davecheney: eh.. I'm sure it did | 10:16 |
| jam | davecheney: it is possible that raw installs do, but EC2 doesn't | 10:16 |
| fwereade | jam, PrecheckInstance is at least as important, I think, even if we haven't got much in the way ofimplementations for it | 10:16 |
| jam | as in, they updated the image but only for tools and not for the kernel | 10:16 |
| davecheney | jam: as I understand it an AMI is the root disk | 10:17 |
| fwereade | jam, but if people read about containers and want to experiment with them, we currently accept anything and fail late | 10:17 |
| axw | sorry fwereade family just came home... reading back | 10:17 |
| davecheney | there is a separate file with the kernel | 10:17 |
| davecheney | not sure what it is called, AKI or something | 10:17 |
| davecheney | we don't get to control that | 10:17 |
| jam | fwereade: sure, but if we have code to do it in a reasonably nice way, it seems a bit silly to dump it for a hack because we aren't sure if the nice way is fully vetted. It feels like the hack will be *even less* vetted. Though perhaps focused enough to make the trade off worthwile. | 10:18 |
| davecheney | why does every container run 6 gettys ? | 10:18 |
| fwereade | davecheney, I though an AMI implied an AKI but you could swap it out if you knew what you were doing? could be wrong | 10:18 |
| davecheney | exactly which virtual console are people going to connect ot ? | 10:18 |
| davecheney | fwereade: I think you can | 10:18 |
| axw | fwereade: umm, if we were going to put in anything, I'd say what's already there | 10:18 |
| davecheney | but looks like it hasn't been done | 10:18 |
| axw | but... we've gone without it for this long...? | 10:19 |
| fwereade | axw, jam: well, the trouble is that what's already there is still kinda off -- the Prechecker looks like it's working better than it necessarily is, and when people come across that code they'll assume it works properly | 10:19 |
| fwereade | axw, jam: eg they'll use the environ's config to determine request sanity | 10:20 |
| fwereade | axw, jam: despite that config being arbitrarily out of date | 10:20 |
| jam | fwereade: if it is a *code in the future* problem, then we should just land it. If it is a "code is slightly wrong right now" then we shouldn't | 10:20 |
| axw | fwereade: oh sorry, are you saying something to include something independent of prechecker? | 10:21 |
| fwereade | jam, I dunno, I feel that past lack of attention to code in the future has put us where we are today | 10:21 |
| fwereade | jam, axw: I teeter still | 10:22 |
| jam | fwereade: is axw actively improving this situation ? The bug he reported seems to be so, and it still seems better to land something that is being actively developed than something that we want to get rid of as soon as possible. | 10:22 |
| fwereade | jam, axw: and in particular, historically, "ok land it and fix it" has come back to bite us because priorities shift and there's no time to fix | 10:22 |
| jam | so yes, be concerned about future code not getting enough attention, but do it by bringing it up and focusing on it in this case | 10:23 |
| axw | so, bzr/lp noob question: is this the right workflow for backporting a fix? branch 1.16, merge in specific revnos, create MP and approve? | 10:23 |
| axw | jam: I have started on fixing that bug, but only just | 10:23 |
| axw | I'm somewhere between 0 and -1 on putting a precheck change into 1.16 at this stage | 10:24 |
| fwereade | axw, jam: I'm just concerned that the "containers work on maas" messaging is perilously close to "containers work" | 10:27 |
| fwereade | axw, jam: and a straight rejection is a much better UX | 10:27 |
| fwereade | axw, jam: disappointing, but not enraging | 10:27 |
| axw | fwereade: totally understand and agree with that | 10:27 |
| axw | just slightly risky I think | 10:28 |
| jam | fwereade: so I agree, the disagreement is about "we have this stuff to do that, but we're going to punt it for the stable release and do a quick hack before we land the stuff we care about in the unstable release". | 10:28 |
| jam | It may be that scope means we need to | 10:28 |
| jam | but it doesn't seem less risky then landing what we've been developing | 10:28 |
| wallyworld | jam: so i have a branch which diverged from trunk after 1.16 was branched. i naiively put up a mp of that same branch but targetted to 1.16. but the diff had extra stuff. so i think my only choice is to cherry pick my changes and do a new branch stacked on 1.16 and then a mp from there? | 10:29 |
| jam | wallyworld: yes | 10:30 |
| wallyworld | sigh | 10:30 |
| jam | wallyworld: "bzr branch lp:juju-core/1.16 .../my/local/path" bzr merge $EXISTING_BRANCH -r X..Y | 10:30 |
| * wallyworld wishes we didn't branch 1.16 so soon :-( | 10:30 | |
| * wallyworld prefers to branch later | 10:31 | |
| wallyworld | jam: yeah, i know how to cherry pick :-P | 10:31 |
| axw | jam: I'm doing a backport now too; do I just do that, then push to LP, create an MP and approve? | 10:31 |
| fwereade | wallyworld, it was only like 5 days ago ;p | 10:31 |
| jam | axw: you need to target to the lp:juju-core/1.16 branch, you can do so with "lbox propose -for lp:juju-core/1.16" | 10:31 |
| wallyworld | fwereade: but there was still much to do for 1.16 | 10:32 |
| axw | jam: ta | 10:32 |
| fwereade | wallyworld, most of that only became apparent in those 5 days | 10:32 |
| fwereade | wallyworld, and I'd rather have specific fixes against a known quantity than deal with unrelated changes in trunk | 10:32 |
| wallyworld | perhaps. it's a religious argument :-) | 10:33 |
| wallyworld | my view is that you slow down trunk development and do testing in the week before | 10:33 |
| wallyworld | so you only commit to one branch and have goo qa and it's all still controlled | 10:34 |
| fwereade | that's reasonable too, indeed | 10:34 |
| fwereade | let's argue over beers in sfo :) | 10:34 |
| wallyworld | yep. i wouldnt be complaining if i didn;t have so many fucking branches to double land :-) | 10:34 |
| wallyworld | and i guess i sorta knew there was still much to do | 10:35 |
| wallyworld | eg signing metadata yada yada | 10:35 |
| * fwereade ciggie before meeting | 10:38 | |
| axw | wallyworld, fwereade: can you please review this for juju-core/1.16 -- https://codereview.appspot.com/14579044 | 10:41 |
| axw | I'll bbl, going to have dinner | 10:41 |
| wallyworld | ok | 10:43 |
| fwereade | TheMue, standup? | 10:47 |
| rogpeppe | fwereade, jam: g+ is misbehaving on me | 10:48 |
| jam | mgz: standup ? | 10:50 |
| jam | https://plus.google.com/hangouts/_/867e754a0a06dd4a7c52dc4e27457cd018da3960 | 10:51 |
| jam | mgz: TheMue ^^ | 10:51 |
| jam | wallyworld: was https://code.launchpad.net/~wallyworld/juju-core/plugin-env intended for 1.16 ? | 10:55 |
| wallyworld | yes, i have to do it in both | 10:55 |
| wallyworld | doing trunk first | 10:56 |
| jam | k, I targetted it | 10:56 |
| jam | your trunk is marked as merged | 10:56 |
| wallyworld | yes, i just merged it | 10:56 |
| === dstroppa_ is now known as dstroppa | ||
| === teknico1 is now known as teknico | ||
| * dimitern => lunch | 12:09 | |
| natefinch-afk | how come babies only sleep *after* meetings are over? :/ | 12:10 |
| === natefinch-afk is now known as natefinch | ||
| sidnei | davecheney: pong | 12:11 |
| * TheMue is back, and very happy *grmpflx* lost time and 120 bucks :( | 12:19 | |
| jam | can anyone do a quick review for landing in 1.16 branch: https://codereview.appspot.com/14494048/ | 12:20 |
| jam | hi TheMue, welcome back | 12:20 |
| TheMue | jam: yeah, thanks | 12:20 |
| TheMue | at least now trying to solfe the os x troubles together with dstroppa | 12:21 |
| TheMue | solve | 12:21 |
| jam | mgz: are you looking back at the bot charm now, or are you finishing up stuff for 1.16 ? | 12:21 |
| natefinch | TheMue - I'd like to hear more about that, since the code cross compiles for OSX just fine (on trunk at least) | 12:22 |
| TheMue | natefinch: yeah, here too, but not for dstroppa. that's wondering me | 12:23 |
| TheMue | natefinch: where I have massive problems is to run the tests. 1st my mongo has no ssl built in, but the second one is regarding simplestreams | 12:23 |
| mgz | jam: planning on charm poking, unless something needs looking at for 1.16 | 12:24 |
| natefinch | TheMue: I kinda doubt running the tests would work with cross compilation. | 12:25 |
| jam | mgz: k, then if you can investigate http://dave.cheney.net/2013/07/09/an-introduction-to-cross-compilation-with-go-1-1 while you're at it | 12:25 |
| jam | mgz: but if you want' you can review: https://codereview.appspot.com/14494048/ | 12:25 |
| natefinch | jam: reviewed | 12:25 |
| jam | thanks natefinch | 12:26 |
| mgz | that was fast :) | 12:26 |
| jam | mgz: it is a <1 line change | 12:27 |
| natefinch | mgz: cross compilation is basically just "install go from source, run a script to build the tools for all GOOS and GOARCH, then use the correct tool for the target system" | 12:28 |
| TheMue | natefinch: I'm compiling native here | 12:28 |
| natefinch | TheMue: ahh, ok. Well, it's nice to know it compiles on native OSX, at least on someone's machine | 12:29 |
| * natefinch is tempted to set up a hackintosh VM for testing | 12:29 | |
| TheMue | natefinch: already tried it a few weeks ago, w/o running the tests, but then installing our typical wordpress/mysql on ec2 | 12:30 |
| TheMue | natefinch: e'thing fine | 12:30 |
| natefinch | TheMue: nice nice | 12:30 |
| natefinch | TheMue: I broke it when I made changes for Windows for 1.14. unfortunately, no one noticed until 1.15 was out | 12:30 |
| natefinch | jam: ^^ we released 1.14 to OSX.... are we just releasing straight code for OSX, or binaries? (I don't know how homebrew works) Curious how it got released completely broken like that | 12:32 |
| TheMue | natefinch: I don't know how homebrew works, never used it | 12:33 |
| TheMue | natefinch: I've simply branched, gogetted ;) all 3rd party stuff and then go install | 12:34 |
| natefinch | TheMue: looks like homebrew is basically git to download zipped code that then runs ruby files to install stuff. Which sounds handy, but incredibly dangerous. | 12:37 |
| TheMue | natefinch: ic | 12:38 |
| TheMue | natefinch: for missing unix software i'm using macports which also loads sources and builds them directly | 12:39 |
| TheMue | natefinch: but imho the downloaded sources are from a special source code repository | 12:39 |
| natefinch | TheMue: yeah, homebrew seems to be touting itself as a macports alternative. | 12:39 |
| natefinch | TheMue: http://brew.sh/ | 12:40 |
| jam | natefinch: I'm pretty sure homebrew builds from a source file | 12:41 |
| jam | but I don't know it either | 12:41 |
| * rogpeppe goes to lunch and a dentist's appointment | 12:44 | |
| TheMue | rogpeppe: oh, had dentist early this morning too | 12:46 |
| TheMue | rogpeppe: thankfully only 5 min checkup | 12:46 |
| TheMue | rogpeppe: good luck | 12:46 |
| axw | fwereade: can I please get a review on https://codereview.appspot.com/14579044/ ? exactly same as the other one, but for 1.16 | 12:53 |
| fwereade | axw, ah, sorry | 12:55 |
| fwereade | axw, ha, sorry, I even had a tab open already with an unpublished LGTM | 12:55 |
| fwereade | axw, done now | 12:55 |
| axw | fwereade: thanks :) | 12:55 |
| wallyworld | fwereade: rightio, my 5 branches have landed in trunk and 1.16 and i've reverted the legacy tools code in trunk. i just need to get the public key for tools metadata and commit that. let me know if anything else comes up | 13:07 |
| sinzui | you have been very busy wallyworld | 13:10 |
| wallyworld | yes | 13:10 |
| wallyworld | be good when release is done | 13:10 |
| wallyworld | sinzui: if you get tip of 1.16, then you will have metadata signing support | 13:12 |
| sinzui | faboo | 13:12 |
| wallyworld | i just need the public key now :-) | 13:12 |
| sinzui | me too | 13:12 |
| wallyworld | i hope we get it in time to make the release | 13:13 |
| axw | wallyworld: has the bot been merging your MPs for 1.16? | 13:34 |
| axw | mine's been sitting there approved for 37 minutes | 13:34 |
| wallyworld | axw: yes :-) | 13:34 |
| wallyworld | axw: did you propose against 1.16? | 13:35 |
| axw | wallyworld: yup | 13:35 |
| axw | wallyworld: https://code.launchpad.net/~axwalk/juju-core/1.16/+merge/190085 | 13:35 |
| wallyworld | link to mp? | 13:35 |
| wallyworld | i'll take a look, it may just be queued behinds others | 13:36 |
| axw | thanks | 13:36 |
| wallyworld | axw: running now | 13:37 |
| wallyworld | almost done | 13:37 |
| axw | wallyworld: cool, ta | 13:37 |
| wallyworld | weird branch name though :-) | 13:37 |
| axw | wallyworld: yeah I forgot to give it a useful name | 13:38 |
| wallyworld | figured as much :-P | 13:38 |
| axw | luckily I only hae one to do ;) | 13:38 |
| wallyworld | so you think :-) | 13:39 |
| * fwereade is seriously tempted to delete environs/instances and start afresh :( | 13:39 | |
| * axw runs | 13:39 | |
| axw | third system syndrome? :) | 13:39 |
| fwereade | axw, heh | 13:39 |
| fwereade | axw, just premature abstraction | 13:39 |
| fwereade | axw, it's got ec2-specific bits mixed all through | 13:40 |
| fwereade | axw, and it's completely broken | 13:40 |
| axw | fair enough, I was just being facetious :) | 13:40 |
| fwereade | axw, I know :) | 13:41 |
| axw | fwereade: I'd be keen to hear what things you don't like about it, maybe at SFO | 13:42 |
| fwereade | axw, mainly that if it can't find an instance type it just gives you whatever | 13:42 |
| fwereade | axw, which is just... broken | 13:42 |
| axw | ? :( | 13:42 |
| axw | is that the constraints thing you were talking about before? | 13:43 |
| wallyworld | fwereade: i remember talking about that with you and as i recall we originally returned an error but i think we were told we had to return *something* | 13:43 |
| wallyworld | unless i'm mistaken | 13:43 |
| fwereade | wallyworld, I think you promised to fix it but we all got distracted, it was a stressful time, just before april ;p | 13:44 |
| fwereade | wallyworld, and I rationalized that since we had to have it on openstack animplementation that nostly worked for both beat one that only worked for ec2 | 13:45 |
| wallyworld | i can't properly recall now. i do recall that all the code was originally for ec2 | 13:45 |
| wallyworld | we deal with provider specific attributes by allowing them to be nil | 13:47 |
| fwereade | wallyworld, and you (quite defensibly) really wanted to avoid copying logic -- so I'm not even sure it turned out *worse* than it would have done if you'd heeded my urging to keep them separate, we know well how that ends up usually | 13:47 |
| fwereade | wallyworld, I'm just grumpy because it's just come fully to my attention that it's been broken for 6 months | 13:47 |
| hazmat | fwereade, re bugs, if you have a moment, i'd like a moment to discuss https://bugs.launchpad.net/juju-core/+bug/1174610 which originally got marked opinion. | 13:47 |
| _mup_ | Bug #1174610: unit ids should be unique <juju-core:New> <https://launchpad.net/bugs/1174610> | 13:47 |
| hazmat | it recently got expired automatically by lp, just wanted to get it back into the queue/hopper | 13:47 |
| wallyworld | i think the logic works ok - deal with a bunch of common attributes, and handle possibly nil ones | 13:48 |
| fwereade | wallyworld, and I'm having some difficulty seeing how to tease it apart | 13:48 |
| wallyworld | it get complicated because we have instances and instance types and they have to be smashed together | 13:48 |
| wallyworld | instances come from simplestreams, instance types from flavours | 13:48 |
| wallyworld | i think we read the possible instances first, then look up the instances types, and then see what is compatible and also satifying constraints | 13:49 |
| fwereade | hazmat, ok, I see -- does it matter for services as well, or just for units? | 13:50 |
| wallyworld | if no constraints satisfied, it uses heuristics to just pick something | 13:50 |
| fwereade | wallyworld, no | 13:50 |
| fwereade | wallyworld, it skips the heuristics it's meant to use | 13:50 |
| fwereade | wallyworld, and then if it can't find a matching instance type it *then* uses the heuristics to pick something that *doesn't* match | 13:51 |
| hazmat | fwereade, just for units really | 13:51 |
| fwereade | wallyworld, and all that stuff is tucked away inside a helper function | 13:51 |
| fwereade | wallyworld, when the falling backought to be top-level | 13:51 |
| wallyworld | the heuristics are only used if a match cannot be found | 13:51 |
| hazmat | fwereade, the service case was explicitly stated as a desired overlap | 13:51 |
| wallyworld | a constraints based match shoudl always be used first surely? | 13:52 |
| hazmat | fwereade, ie.. if i delete wordpress svc, i can get the same db back on reconnect to mysql | 13:52 |
| fwereade | wallyworld, because we may get a list of matches from constraints-with-heuristics, but find that none has a matching image, and need to fall back to try again without | 13:52 |
| hazmat | fwereade, where as log aggregation, metrics against a unit is bad overlap. | 13:52 |
| fwereade | wallyworld, a constraints based match ignores the heuristics | 13:52 |
| wallyworld | yes, as it should? | 13:52 |
| fwereade | hazmat, ok, yes, I see | 13:52 |
| fwereade | wallyworld, no! | 13:52 |
| wallyworld | if i say i want xyz, then that's what i should get | 13:53 |
| fwereade | wallyworld, that's why we're getting 512M instances on canonistack | 13:53 |
| fwereade | wallyworld, and if you don't say anything, we shouldn't just give you useless heaps of rust because you didn't specify you wanted something useful ;p | 13:53 |
| wallyworld | that's the argument i had at the time - i didn't want to do that | 13:53 |
| fwereade | hazmat, ok, I see your point there -- and I can see why it's unreasonable to expect people to use plain int ids (which don't themselves exist anyway) | 13:54 |
| wallyworld | but i'm not getting why the system should not use my constraints if i provide them. maybe i'm just being thivk | 13:54 |
| fwereade | wallyworld, it should | 13:54 |
| fwereade | wallyworld, but it should also try to give you something that's likely to work *as well as* matching your constraints | 13:55 |
| wallyworld | well, the user should know best | 13:55 |
| fwereade | wallyworld, if the only things that satisfy your constraints are piles of junk, well, fair enough | 13:55 |
| wallyworld | yes , that's my point | 13:55 |
| fwereade | wallyworld, and we fall back to the least worst of those | 13:55 |
| wallyworld | if constraints are satisfied, you get what you asked for | 13:56 |
| wallyworld | maybe there was a requirements misunderstanding - none of this was documented as far as i know | 13:56 |
| wallyworld | it was all sorta verbal | 13:56 |
| fwereade | wallyworld, it's a pretty close match for python, which was documented actually -- but, yeah, the precise mechanismchanged | 13:57 |
| wallyworld | but i do recall a disagreement on whether to return an error if a match could not be found. i sorta wish we did return an error instead of crap | 13:57 |
| fwereade | wallyworld, the main point is that your approach gives the user instances that do *not* match the constraints they asked for | 13:57 |
| fwereade | wallyworld, instead of returning an error | 13:58 |
| wallyworld | i did want to return an error | 13:58 |
| wallyworld | i could have sworn that i was not allowed to | 13:58 |
| fwereade | wallyworld, I must have just completely failed to communicate then | 13:58 |
| fwereade | wallyworld, I'm sorry | 13:58 |
| wallyworld | me too | 13:58 |
| wallyworld | i think the wires got crossed | 13:58 |
| wallyworld | i do recall not being sure at the time that what was done was ok or not | 13:59 |
| fwereade | wallyworld, yeah, I was not in a very good place at the time, and probably not being helpful | 13:59 |
| wallyworld | well, we both are to blame :-) | 13:59 |
| wallyworld | i knew a lot less back then | 13:59 |
| wallyworld | some would say not much has changed | 14:00 |
| fwereade | wallyworld, ehh, no worries, I'm a lot happier knowing you're not wedded to the approach | 14:00 |
| wallyworld | oh not at all | 14:00 |
| wallyworld | i just want it to work | 14:00 |
| fwereade | wallyworld, and, fwiw, I think you're great | 14:01 |
| wallyworld | but i do think that having provider specific things as nillable values can work ok | 14:01 |
| wallyworld | it;s just the matching logic that needs tweaking | 14:01 |
| wallyworld | constraints were implemented using the same approach | 14:02 |
| fwereade | wallyworld, yeah, I'm not really bothered by that side of it, except inasmuch as it's another thing to keep track of while unpacking it | 14:02 |
| wallyworld | yeah | 14:02 |
| wallyworld | my view is the common logic and all the benefits there outweight the unpacking cost | 14:02 |
| wallyworld | so, we need to get this fixed for 1.16 i assume | 14:03 |
| fwereade | wallyworld, I kinda feellike we do, now I've looked at it | 14:03 |
| wallyworld | i'm quite tired now, but can pick it up after a few hours sleep | 14:03 |
| fwereade | wallyworld, but you're not allowed to do any more work today, get some sleep | 14:04 |
| fwereade | wallyworld, if I don't get something useful done myself I can hand over when you come back | 14:04 |
| wallyworld | ok.i think the only thing i have to do next is commit the signing key | 14:04 |
| wallyworld | so hopefully this can be sorted out before the deadline. | 14:05 |
| wallyworld | fwereade: i think you would add the best value here by writing the correct tests and we then make those work. ie true TDD | 14:05 |
| wallyworld | clearly the tests cases are wrong | 14:06 |
| wallyworld | if stuff is broken | 14:06 |
| wallyworld | even if the implmenation has to be patched together a bit to make 1.16, at least the tests will tell us things will be ok | 14:07 |
| wallyworld | and we can fix properly for 1.18 | 14:07 |
| wallyworld | and the tests will document how it should work | 14:07 |
| fwereade | wallyworld, yeah, that's the plan :) | 14:08 |
| wallyworld | \o/ | 14:08 |
| === gary_poster is now known as gary_poster|away | ||
| sinzui | wallyworld, check your email for the public key | 14:12 |
| wallyworld | sinzui: great, thanks | 14:13 |
| * wallyworld will do a little branch to get that in | 14:13 | |
| === gary_poster|away is now known as gary_poster | ||
| sinzui | fwereade, jam (if you are about and care). I have a choice to make with the 1.16 series branch. I need to do a lot of merged, and if I want things to look perfect, do them in a different order to get the version fix, then inc the version, then merge everything else. I could merge everything else as a single branch. I could just re-fork 1.16 at the revision I want. At the moment, I don't see anything that has landed after 1954 | 14:44 |
| sinzui | that I don't want in the 1.16 branch | 14:44 |
| fwereade | sinzui, I couldn't swear to that... and I think we've also been merging to 1.16, haven't we? | 14:45 |
| fwereade | sinzui, what is there in trunk that's not in 1.16? | 14:45 |
| wallyworld | sinzui: both 1.16 and trunk now have public key. so if you have private key you should be able to sign tools metadata and juju should be able to use it | 14:46 |
| sinzui | fwereade, damn, I was supposed to be getting merge notification on this branch. My fault I am sure | 14:47 |
| sinzui | fwereade, wallyworld I will retry the inc 1.16 merge then and review the changes | 14:48 |
| fwereade | sinzui, so, just today, a constraints bug came up, that I *really* want to fix... but I think actually the right thing is to keep it out of 1.16, people have been putting up with it since forever | 14:53 |
| sinzui | fwereade, I have used the same thinking with some of the other issues saw yesterday. Regressions need fixing. Blockers to get to the new version need fixing. I don't think we need to delay a release for a fix that will come available in a few weeks with the next release | 14:56 |
| fwereade | sinzui, yeah | 15:00 |
| fwereade | sinzui, my desire to fix it is just based on personal upset/embarrassment | 15:00 |
| sinzui | fwereade, If the issue is fixed before we name the release revision, I am always happy to include | 15:01 |
| sinzui | it | 15:01 |
| sinzui | fwereade, I do check for fix committed bugs and make sure they have a milestone, and I propose them in the stable release if I think they are safe | 15:03 |
| * sinzui does just that for natefinch's lp:~natefinch/juju-core/019-osx-fix | 15:03 | |
| fwereade | sinzui, I probably won;t get it done without rushing, I'm still feeling my way a bit | 15:05 |
| sinzui | okay | 15:05 |
| fwereade | jam, rogpeppe: heh, did we actually assign that log-levels bug to someone? | 15:05 |
| fwereade | sinzui, when are you aiming to pull the trigger? | 15:06 |
| rogpeppe | fwereade: the one we were talking about in the meeting? | 15:06 |
| fwereade | rogpeppe, yeah | 15:06 |
| rogpeppe | fwereade: i don't think so | 15:06 |
| rogpeppe | fwereade: i had a brief glance at fixing the RPC logging to avoid logging pings; it's not that easy, sadly. | 15:06 |
| fwereade | rogpeppe, bugger... and it sort of trailed off without consensus | 15:06 |
| sinzui | fwereade, I see 7 bugs for 1.16 and 1 doesn't appear to be started. I don't think I can start the release for 8 hours. I really think I will be in 24 hours | 15:07 |
| rogpeppe | fwereade: because at that level we're writing reply messages but don't necessarily know what the call is that we're replying to | 15:07 |
| fwereade | rogpeppe, would you restate the objections to logging RPC at TRACE level? | 15:07 |
| fwereade | rogpeppe, ISTM that, today, we don't get them anyway | 15:07 |
| fwereade | rogpeppe, unless you change config | 15:07 |
| fwereade | rogpeppe, so that's pretty much moot | 15:08 |
| rogpeppe | fwereade: for me, seeing the RPC messages in the machine log has been indispensible in diagnosing issues | 15:08 |
| fwereade | rogpeppe, understood -- but have you needed them when you haven't known you would? how often has there been a case you can't repro? | 15:09 |
| rogpeppe | fwereade: almost always | 15:09 |
| rogpeppe | fwereade: it's usually been when someone's had a problem and has pasted me the machine log | 15:09 |
| fwereade | rogpeppe, ok, but, that doesn't work today, right? | 15:10 |
| rogpeppe | fwereade: i guess it's changed | 15:10 |
| rogpeppe | fwereade: if you bootstrap --debug, do you see it now? | 15:10 |
| fwereade | rogpeppe, offhand, not sure | 15:11 |
| fwereade | rogpeppe, but I would expect so | 15:11 |
| rogpeppe | fwereade: so when are people seeing the RPC messages now? | 15:12 |
| fwereade | rogpeppe, when they set DEBUG to look at the hook output, I think | 15:12 |
| rogpeppe | fwereade: so can't they just set debug output for the hooks only? | 15:12 |
| fwereade | rogpeppe, heh, probably -- but the other side of it is that they generally *do* want hook output it seems | 15:13 |
| rogpeppe | fwereade: we could log hook output at Info level | 15:13 |
| fwereade | rogpeppe, maybe the answer is just a slightly more sophisticated default config that includes DEBUG for ... whatever the uniter logger is called | 15:14 |
| rogpeppe | fwereade: juju.worker.uniter | 15:14 |
| rogpeppe | fwereade: i wonder whether we should have some logging modules that aren't named after juju packages | 15:15 |
| rogpeppe | fwereade: oriented towards user logging requirements | 15:15 |
| fwereade | rogpeppe, I have no objection to such things in principle, for sure | 15:16 |
| fwereade | rogpeppe, at the moment we're only logging WARNING and above by default anyway | 15:17 |
| rogpeppe | fwereade: then we could make sure that a given stream sees a set of coherent log messages, even if the messages happen to be generated in several different modules. | 15:18 |
| rogpeppe | s/modules/packages/ | 15:18 |
| fwereade | rogpeppe, indeed, as pointed out by hazmat in http://www.mail-archive.com/juju-dev@lists.ubuntu.com/msg00214.html | 15:18 |
| rogpeppe | fwereade: we could actually *design* the log messages that we want the user to see. | 15:18 |
| hazmat | rogpeppe, there's a bug to adjust the root=DEBUG pinger=INFO | 15:19 |
| hazmat | which will address both issues, the default of INFO is too quiet | 15:19 |
| hazmat | well it addresses some of the issues, the issues in that email are still relevant | 15:19 |
| rogpeppe | hazmat: unfortunately there's no "pinger" log module | 15:19 |
| rogpeppe | hazmat: all the rpc messages are logged at transport level | 15:20 |
| hazmat | rogpeppe, can that be changed? | 15:20 |
| hazmat | rogpeppe, the pinger messages cause the logs to fill up, and have almost zero value outside of an rpc dev.. and even then its questionable. | 15:20 |
| rogpeppe | hazmat: it's the easiest place to do that (and reliable because we get to see the underlying representation), but it may be possible to do something more flexible | 15:21 |
| rogpeppe | hazmat: yeah, i agree | 15:21 |
| hazmat | rogpeppe, the rpc messages are fine i think for debug transport, its just the pings, that obscfucate things. | 15:21 |
| fwereade | hazmat, looks like the root default is WARNING anyway which is *definitely* too quiet | 15:21 |
| hazmat | fwereade, rogpeppe for actual debug-log usage by end users the critical piece missing is both tagging log messages with their semantic location (unit id and machine id) instead of ip and then filtering capabilities at view time instead.. really adjusting the generation/collection to adjust the view is broken. | 15:24 |
| rogpeppe | hazmat: what about the problem of filling up the log file? | 15:25 |
| hazmat | cause needs change, but if an env don't have the data then the user is hosed. | 15:25 |
| rogpeppe | hazmat: or is writing to that file considered "viewing"? | 15:25 |
| hazmat | rogpeppe, writing to the log file is not viewings, its collection / aggregation. The pinger is the number one by far issue with that, and we should default to that off.. Moreoever its pretty damm common to use log rotation and compression here. | 15:25 |
| rogpeppe | hazmat: if we're using compression then the pinger noise won't be an issue, will it? | 15:26 |
| hazmat | which in the absence of pinger, means we only have messages that matter for archival, ie related to changes. | 15:26 |
| rogpeppe | hazmat: (not that i'm saying we shouldn't switch it off if we can do so reasonably) | 15:27 |
| hazmat | rogpeppe, yes.. its still an issue.. because its A) not useful B) total obscures real info in viewing without filtering C) generates additional writes/exceess logs.. | 15:27 |
| hazmat | rogpeppe, ie. just because its compressing, doesn't mean we should loop garbage into it. | 15:27 |
| rogpeppe | hazmat: for semantic location, i'm thinking about a particular log module which relays messages related to particular units or machines, tagged in a consistent way | 15:28 |
| fwereade | rogpeppe, hazmat: OK, I am pretty convinced that (1) we don't want rpc traffic in there by default (2) we do want hook output in there by default (3) hook output as it stands is not so helpful because it's not usefully badged | 15:29 |
| === natefinch is now known as natefinch-afk | ||
| fwereade | rogpeppe, hazmat: with an option on (4) FFS filter debug-log at server side | 15:31 |
| rogpeppe | fwereade: as i've said, i think at this relatively unstable stage of juju development, having rpc traffic (not pings) in there by default adds considerable value. | 15:33 |
| rogpeppe | fwereade: particular as we move towards doing more and more stuff via the API | 15:33 |
| rogpeppe | fwereade: i've just realised that filtering out ping logs might not be as awkward as i thought | 15:34 |
| hazmat | rogpeppe, json encode the log records with multiple tags is the simplest way.. and what state of the art log aggregation typically does (heka/sensu/logstash) | 15:36 |
| rogpeppe | hazmat: that's one option indeed | 15:37 |
| hazmat | fwereade, re 4 .. what's FFS filter? | 15:38 |
| rogpeppe | fwereade: "for fuck's sake" :-) | 15:38 |
| fwereade | hazmat, For Something's Sake ;p | 15:38 |
| rogpeppe | ahem, pardon my bad french translation | 15:39 |
| fwereade | sorry brb | 15:39 |
| * hazmat learns something new | 15:39 | |
| fwereade | ehh, I've been cussing and blinding all week,not sure why I got bashful about it there | 15:39 |
| hazmat | fwereade, re .. 4.. i don't think it really matters if its server or client.. in terms of delivering the feature. yes it would be more efficient server side, but that's a bit more infrastructure. the feature itself in conjunction with 3, makes 1 and 2 effectively go away (assuming we switch default to root=debug). | 15:42 |
| hazmat | because a user can then find what they care about.. ie. roughly similiar to what i stated later in that same thread. http://www.mail-archive.com/juju-dev@lists.ubuntu.com/msg00216.html | 15:42 |
| hazmat | its still worthwhile to kill 1 though just due to its lack of usefulness (effectively looped garbage) | 15:43 |
| rogpeppe | hazmat: it's only the pings which are garbage | 15:44 |
| hazmat | rogpeppe, ah.. yes.. agreed.. i was specifically referencing the pings.. rpc traffic is useful. | 15:45 |
| fwereade | rogpeppe, hazmat: ok, true that it doesn't matter where that much, and I'm not *really* against rpc traffic | 15:46 |
| fwereade | rogpeppe, so do you have a clever plan for the pings? :) | 15:47 |
| hazmat | isn't it just a separate log channel in that module for transport pings or removing the log messages from them entirely? | 15:47 |
| rogpeppe | fwereade: i'm still diving in there. it would be easier to log everything in the rpc package rather than in the codec, but unfortunately we've lost information at that stage (for instance we won't see extra unrecognised fields in request bodies) | 15:48 |
| rogpeppe | actually, i think i might have a cunning plan | 15:48 |
| jcastro | I get " ERROR storage-auth-key: expected string, got nothing" when trying to bootstrap with the null provider | 15:50 |
| jcastro | but that key isn't mentioned in the pending docs | 15:50 |
| fwereade | jcastro, grar, I'll take a look quickly | 15:53 |
| fwereade | jcastro, yeah, looks like it should be, it's there in the boilerplate config as{{rand}} | 15:55 |
| jcastro | ack, I can fix it in the docs | 15:56 |
| fwereade | jcastro, thanks | 15:56 |
| * fwereade bbs again | 15:56 | |
| marcoceppi_ | jcastro: http://pad.ubuntu.com/7mf2jvKXNa | 15:58 |
| fwereade | rogpeppe, hazmat: ok, in terms of sheer usefulness (3) + (4) seem like the best way to spend my evening | 16:18 |
| fwereade | rogpeppe, if you have something clever for (1) go for it | 16:18 |
| rogpeppe | fwereade: i'd go for 3) | 16:18 |
| rogpeppe | fwereade: i'm still on it, trying not to muck up the whole interface for this case | 16:19 |
| rogpeppe | fwereade: still on (1), i mean | 16:21 |
| fwereade | rogpeppe, understood :) | 16:22 |
| jcastro | fwereade, null provider is running but it doesn't appear to be working, where does it log? | 16:34 |
| jcastro | when trying to do a bootstrap | 16:34 |
| jcastro | it appears like it didn't create anything in /var/lib/ like the command said it should have | 16:35 |
| jcastro | also, do I need password-less sudo access on the remote machine? | 16:35 |
| fwereade | jcastro, I think you do at the moment... that may be what you're hitting | 16:35 |
| jcastro | ok | 16:35 |
| jcastro | yeah I saw the command flash up in top but it didn't execute anything, and I was wondering if it was sitting there remotely waiting for a sudo password | 16:36 |
| TheMue | yeah, one step further why os x tests fail | 16:36 |
| fwereade | jcastro, there are several bugs open re null provider bootstrap, it's worth taking a look at those | 16:36 |
| TheMue | step by step, di damm, di damm | 16:36 |
| jcastro | fwereade, oh excellent, looking now | 16:36 |
| smoser | $ juju init -w | 16:37 |
| smoser | error: flag provided but not defined: -w | 16:37 |
| smoser | i'm following output of 'juju help local' | 16:37 |
| smoser | and it tells me that. | 16:37 |
| smoser | shall i open a bug (juju 1.14.1-precise-amd64) | 16:37 |
| jcastro | https://bugs.launchpad.net/juju-core/+bug/1235716 is what I was running into | 16:38 |
| _mup_ | Bug #1235716: sshstorage does not display sudo prompt <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1235716> | 16:38 |
| jcastro | fwereade, do you know if this is something that we'll shoot for 1.16 or is it point release material? | 16:40 |
| smoser | seems like its fixed in trunk | 16:40 |
| smoser | jcastro, did you say yesterday that 'juju bootstrap' on local provider does not need to be root ? | 16:41 |
| jcastro | it does need to be root | 16:41 |
| fwereade | jcastro, I would *love* to... I had thought sinzui was planning to start a bit earlier than he suggested, so I will try to get some of those reviewed in the hope that axw can land them tonight | 16:41 |
| jcastro | the idea is that it won't need to be someday | 16:41 |
| fwereade | jcastro, I twitch a little re any changes now, but it does seem low-risk/high-value | 16:42 |
| jcastro | fwereade, yeah it's a silly bug, once I got passed it it totally is doing the awesomeness though | 16:42 |
| fwereade | jcastro, sweet | 16:42 |
| jcastro | I'll link to docs on passwordless sudo and keyauth in the docs though, that way people can have a cleaner experience | 16:44 |
| TheMue | fwereade: quick question. what do you think how simple it could be to use mongo w/o ssl in case of a test (depending on the os)? | 17:06 |
| fwereade | TheMue, I doubt it'd be that hard, we set up mongo connections in relatively few places iirc | 17:07 |
| TheMue | fwereade: ok, will investigate in that direction | 17:07 |
| TheMue | fwereade: the two other changes that we would need to run the test on os x seem to be simple too | 17:08 |
| TheMue | fwereade: one is that strip when building jujud needs the additional arg -x and the second is that the series on os x is "unknown" | 17:09 |
| fwereade | TheMue, that first one sounds trivial, the second one might take some thinking | 17:10 |
| TheMue | fwereade: btw, dstroppa can continue to work now, solved his troubles | 17:10 |
| fwereade | TheMue, although a case could be made that it *shouldn't*, and anywhere it's causing trouble is evidence of poorly isolated testing | 17:10 |
| fwereade | TheMue, you rock, tyvm | 17:10 |
| TheMue | fwereade: *blush* | 17:11 |
| TheMue | fwereade: your last sentence: it should not fail (the assert should be against the series returned on os x) or the returned series should not be "unknown"? | 17:19 |
| TheMue | gna, network left me | 17:47 |
| TheMue | ok, good night | 17:56 |
| rogpeppe | fwereade: i'm not going to manage the ping-muting tonight, i'm afraid | 17:57 |
| fwereade | rogpeppe, don't worry | 17:57 |
| rogpeppe | fwereade: i think i know what to do, but it involves a bit of intricacy | 17:57 |
| fwereade | rogpeppe, I'm off and on but should manage something sane for the other bits, and hidden nastiness is better than obvious nastiness ;p | 17:58 |
| rogpeppe | fwereade: so that the codec can know what request is being replied to - easy in principal, but a couple of types need reworking to keep things cleanish | 17:59 |
| rogpeppe | principle, dammit, i always get that wrong! | 17:59 |
| rogpeppe | fwereade: g'night | 17:59 |
| rogpeppe | and g'night to all | 17:59 |
| sinzui | I see https://bugs.launchpad.net/juju-core/+bug/1237219 only has a merge into 1.16. Doesn't it also need to merge into trunk? | 18:07 |
| _mup_ | Bug #1237219: ssl-hostname-verification: false doesn't trigger if auth-URL is http:// <juju-core:In Progress by jameinel> <https://launchpad.net/bugs/1237219> | 18:07 |
| === natefinch-afk is now known as natefinch | ||
| fwereade | sinzui, it surely does, well spotted | 18:52 |
| sinzui | fwereade, https://launchpad.net/juju-core/+milestone/1.16.0 is one bug away from being ready. I will talk to thumper about committing or deferring it | 18:53 |
| fwereade | sinzui, thumper's on holiday, I'm looking into it myself at the moment, it's a little bit more involved than it looks | 18:54 |
| sinzui | fwereade, thank you for taking the time | 18:55 |
| fwereade | sinzui, no worries, thank you :) | 18:55 |
| natefinch | fwereade: how do I get the bot to update goamz? Trying to merge in the ec2 rootdisk stuff, but the bot seems to have an outdated goamz | 19:16 |
| fwereade | natefinch, is dependencies.tsv up to date? | 19:17 |
| natefinch | fwereade: yeah, lemme double check that the revision I specified is the right onw | 19:17 |
| fwereade | natefinch, if it's not paying attention to that, I'm not sure, I'm afraid | 19:17 |
| natefinch | fwereade: yeah, it's correct. So I guess the bot isn't paying attention, or has some other problem. | 19:18 |
| natefinch | well boo | 19:18 |
| sinzui | maybe we can update gobot to do the right thing. I think I have issue sorted out in my branch to create a tarball with the correct versions: https://codereview.appspot.com/14354043/ | 19:21 |
| natefinch | sinzui: that looks pretty good. Reminds me how much I hate bash scripts, though :) | 19:28 |
| sinzui | I despise bash | 19:28 |
| sinzui | I vowed to never write it again after working on the charmworld charm. | 19:28 |
| hazmat | so..maas tags aren't actually provider specific... | 19:31 |
| natefinch | hazmat: tags as a constraint are currently only supported on MaaS, but I know at least amazon has support for tags, though that's about all I know about amazon's tags | 19:34 |
| hazmat | natefinch, tags mean something totally different there | 19:34 |
| hazmat | natefinch, tags in aws are for resources not for classes | 19:34 |
| hazmat | in maas they reference an unallocated resource class.. in aws they reference an allocated group of resources | 19:36 |
| natefinch | hazmat: yeah, they're quite different. In theory we might still be able to use them with juju, but it's certainly a different beast | 19:37 |
| hazmat | and it would be nice to validate user vocabulary.. | 19:38 |
| hazmat | er.. user supplied values | 19:38 |
| natefinch | hazmat: what do you mean by validating their values? | 19:38 |
| hazmat | natefinch, what happens with an typo'd tag value? | 19:39 |
| hazmat | natefinch, pending forever? | 19:39 |
| natefinch | hazmat: yep. That's pretty much how we handle all constraints that can't be fulfilled right now. Not the best experience, for sure. | 19:39 |
| hazmat | natefinch, except its not a constraint that can't be fulfilled... its a user input error | 19:41 |
| natefinch | hazmat: there's no real way for us to tell that. If the user asks for an instance with 10T of disk space, is that a typo, or just the user asking us for something we can't fulfill? Maybe the tag was typoed and the user spelled it right in juju... the reason for the mismatch between the constraint and what's available doesn't matter. | 19:43 |
| natefinch | hazmat: regardless of why we can't fulfill the constraint... our current handling of it stinks. | 19:44 |
| hazmat | natefinch, ignoring the generic constraints, these provider constraints are introspectable or static, and yes.. this was validation was previously done. | 19:44 |
| natefinch | hazmat: it'll get fixed eventually, I'm sure. At least we *have* constraints by maas tags now | 19:49 |
| hazmat | agreed the reason doesn't matter, we just need to be better about reporting the mismatch as an unsolvable error upfront, rather then pending forever behavior | 19:49 |
| natefinch | hazmat: absolutely | 19:59 |
| hazmat | filed a bug 1237626 | 20:00 |
| _mup_ | Bug #1237626: constraints should be validated <juju:New> <https://launchpad.net/bugs/1237626> | 20:00 |
| fwereade | hazmat, fwiw constraint-prechecking is actually in progress but didn't make it in time | 20:15 |
| fwereade | hazmat, and it's not pending forever... it's a error, communicated back in status, and the machine is then amenable to destruction | 20:16 |
| hazmat | natefinch fwiw core's support of maas tags is at a bare minimum of the features available from tags or previously available, filed bug https://bugs.launchpad.net/juju-core/+bug/1237634 i don't think it matters atm to the users. | 20:16 |
| _mup_ | Bug #1237634: maas tags support should support full tag syntax <juju-core:New> <https://launchpad.net/bugs/1237634> | 20:16 |
| hazmat | fwereade, cool, async provider status makes those errors much better | 20:17 |
| fwereade | hazmat, python didn't do that, and last I heard nor did maas | 20:17 |
| fwereade | hazmat, python *accepted* &|etc | 20:17 |
| hazmat | fwereade, which part? | 20:17 |
| hazmat | fwereade, right those are valid maas tags syntax | 20:17 |
| hazmat | fwereade, core doesn't support that, it parses for a comma separated list only | 20:18 |
| hazmat | hmm.. | 20:18 |
| fwereade | hazmat, last time I looked at python it accepted but ignored them | 20:18 |
| hazmat | fwereade, it passed them through and validated the tag names | 20:19 |
| hazmat | fwereade, if maas doesn't support it.. that's quite surprising, else i can't think of why support and tests for that case would have been added. | 20:19 |
| * hazmat pokes through maas src | 20:19 | |
| fwereade | hazmat, """Parser and validator for tags constraint expressions | 20:21 |
| fwereade | Tag names are extracted and checked against the list of known tags | 20:21 |
| fwereade | reported by the api. | 20:21 |
| fwereade | Currently tag constraints consist of just comma and/or whitespace tag | 20:21 |
| fwereade | names, all of are required for a match. Extending this to support full | 20:21 |
| fwereade | boolean expressions would be possible, and some forward compatibility | 20:21 |
| hazmat | but perhaps in which case its an invalid bug | 20:21 |
| fwereade | is attempted. | 20:21 |
| fwereade | """ | 20:21 |
| hazmat | fwereade, hmm.. the logic in the method below explicitly strips of operators as it matches names | 20:22 |
| natefinch | hazmat: right, I don't think it was ever implement | 20:22 |
| natefinch | ed | 20:22 |
| hazmat | fwereade, natefinch yeah.. i don't see any support for boolean operators in the maas src unit tests. | 20:24 |
| hazmat | i'll mark the bug invalid | 20:24 |
| hazmat | fwereade, but that brings up my original comment, if the maas tags support isn't a form of provider specific constraint, does that mean that supporting provider specific constraints is still blocked? | 20:25 |
| natefinch | hazmat: there was some talk about using tags as a way to hack provider specific support | 20:27 |
| fwereade | hazmat, not *exactly*, in that there is no technical obstacle to getting instance-type | 20:28 |
| natefinch | tags=instancetype:m1.small | 20:28 |
| fwereade | natefinch, no, certainly not | 20:29 |
| hazmat | natefinch, as opposed to instance-type=m1.small? | 20:29 |
| natefinch | fwereade: wouldn't be my first choice, no | 20:29 |
| hazmat | fwereade, but that's also in the spirit of just globally adding them as a constraint of all providers ? | 20:29 |
| natefinch | certainly adding an instance-type constraint is pretty trivial code | 20:29 |
| hazmat | ie do we also do ec2-zone that way? | 20:29 |
| fwereade | natefinch, hazmat: instance-type and zone are not yet there -- but zone when implemented will not I think be a constraint | 20:30 |
| fwereade | hazmat, but rather a placement request | 20:30 |
| hazmat | ie. this isn't really adding provider specific constraints, its just extending the global constraint vocabulary | 20:30 |
| hazmat | fwereade, really it wants both | 20:31 |
| hazmat | fwereade, the zone constraint because it does matter for various reasons (reserved instances, volume attach etc) | 20:31 |
| fwereade | hazmat, well, I think zone is the wrong level for constraints actually -- constraints make more sense at a higher level, eg distribute units for safety | 20:31 |
| hazmat | fwereade, the balance constraint/placement is good for ha as opposed to the current hot swap on the constraint, but both are good | 20:31 |
| hazmat | fwereade, there is more than ha as a reason for the zone constraint | 20:32 |
| fwereade | hazmat, sure -- it's also good to put things near one another, but that is also IMO not best expressed as provider-specific zones | 20:33 |
| hazmat | fwereade, part of the issue with juju is not being flexible enough for the real world, while automated features are great, alot of whats been the focus for the last cycle is on the flexibility for the real world. | 20:33 |
| hazmat | fwereade, the zone constraint because it does matter for various reasons (reserved instances, volume location for attachment, global inbalance across zones (including non juju managed resources), etc) | 20:34 |
| fwereade | hazmat, are you saying zones are important (I would not disagree) or that they should be expressed as constraints (I'm not sure I would) | 20:34 |
| fwereade | agree that is | 20:35 |
| fwereade | sorry english :/ | 20:35 |
| natefinch | fwereade: I'm guessing he doesn't care about the terminology as long as he gets the end result he wants :) | 20:35 |
| hazmat | my name is tron.. i speak for the users ;-) | 20:36 |
| fwereade | hazmat, the focus has been explicitly away from clever automation and towards the ability to explicitly specify placement | 20:36 |
| fwereade | hazmat, lxc:3 and ssh:user@host are the first steps there | 20:37 |
| hazmat | fwereade, constraint has been the only mechanism / vocabulary for that to date, if there's something else for placement that's great.. my concern is that such placement is policy not mechanism, and what's needed is mechanism. | 20:37 |
| hazmat | fwereade, exactly.. the way to use those for work is via --to which is mechanism not policy.. constraints are an even better mechanism as their state captured. | 20:38 |
| hazmat | fwereade, i don't nesc. see why we should invent a new end user syntax for specifying service placement, when effectively constraints does the same | 20:39 |
| hazmat | ie.. why invent --show-log and deprecated -v , when the latter is an established convention | 20:39 |
| fwereade | hazmat, the distinction there is between service-level properties and the details of individual machines (units) | 20:41 |
| fwereade | hazmat, and re show-log -- we would love to be able to draw a distinction between user output and logging, which are not at all the same thing, and part of that is trying to tease apart the way they've been glommed together | 20:43 |
| hazmat | fwereade, re service level properties, that's valid if placement would actualize differently to each instance, but i think that to an end user the distinction is rather academic and the exposed interface is roughly the same, ie your referencing an implementation detail, else we're just inventing a new set of commands for the same service level operations. | 20:44 |
| hazmat | fwereade, re logging.. again that's an implementation detail... to the end user we deprecated a rather standard unix cli option -v | 20:45 |
| hazmat | and replaced it with some idiosyncratic syntax | 20:45 |
| fwereade | hazmat, that *has* been the focus -- to have placement specified explicitly, without being clever or automatic | 20:47 |
| fwereade | hazmat, it's in the service of getting -v/-q that are actually meaningful | 20:49 |
| hazmat | fwereade, you mean not specified on the service per unit? | 20:49 |
| hazmat | er.. not specified on the service but per unit for placement? | 20:49 |
| fwereade | hazmat, yes | 20:49 |
| hazmat | ah add-unit as the vertex shader.. | 20:50 |
| hazmat | fwereade, that still doesn't really invalidate the use of constraints here | 20:51 |
| hazmat | constraints get captured at a unit level | 20:51 |
| hazmat | hmm | 20:51 |
| fwereade | constraints get captured at a unit level but have never been specifed/able at a unit level | 20:52 |
| hazmat | fwereade, right, but is this really any different than that.. and how does this go back to the higher level automated policy of something like balanced? | 20:52 |
| fwereade | hazmat, the idea is to make the tools available first, and to allow for smart automation thereof later | 20:55 |
| hazmat | fwereade, the question i have is this really any different than specifying constraints with add-unit | 20:57 |
| hazmat | ie. just an additional lookup level to the constraint amalgation | 20:57 |
| fwereade | hazmat, perspective question, I guess: there is no *technical* reason these things *couldn't* be expressed as constraints... but I think that constraints become less and less valuable, and more confusing, as they delve deeper into provider-specificity | 20:59 |
| fwereade | hazmat, as far as possible they should be specified in a juju-level language | 20:59 |
| fwereade | hazmat, this is sadly leaky ofc | 20:59 |
| hazmat | fwereade, well the start of this discussion was how to expose provider specific capabilties | 21:00 |
| hazmat | ie. its not leaky.. its kinda of the point | 21:01 |
| fwereade | hazmat, no argument that we do need to expose provider-specific capabilities | 21:02 |
| fwereade | hazmat, just that those related directly to placement are so provider-specific that they don't fit well as constraints | 21:04 |
| fwereade | hazmat, tags as global constraints (rather than maas-specific ones) *are* rather designed to allow an alternative backdoor into provider-specificity | 21:05 |
| hazmat | fwereade, except i might want to specify them at a service level | 21:05 |
| fwereade | hazmat, I think the line between *what* you run and *where* you run it is pretty clean | 21:07 |
| hazmat | true.. but that's potentially subtle | 21:08 |
| hazmat | fwereade, is specifying a spot instance a where or what on | 21:09 |
| hazmat | as an example | 21:09 |
| fwereade | hazmat, spot instances are so far off they're over the horizon :/ | 21:10 |
| hazmat | its a what on.. but the distinction i think is subtle.. and having a new syntax to express is a implementation purity leaking to the end user... ie is there harm in simplifying the expression to the end user by combining the two. | 21:10 |
| fwereade | hazmat, the potential harm is in defining a constraints vocabulary before we understand the set of use cases that people have and that we want to support | 21:11 |
| hazmat | fwereade, that harm exists regardless of the xpression form | 21:11 |
| fwereade | hazmat, a vocabulary for tactics doesn't restrict a future vocabulary for strategy, but defining a vocab for strategy today *does* | 21:13 |
| hazmat | fwereade, putting them in the same book, doesn't restrict the words to describe either | 21:13 |
| hazmat | that's a valid point though | 21:14 |
| hazmat | its just not clear that complex config here isn't something i wouldn't want specified at a higher level than add-unit, in which case its just as easily captured in an expanded notion of the constraint language | 21:15 |
| fwereade | hazmat, it opens up a bunch of opportunities for conflict between expressions of related ideas at different levels | 21:15 |
| hazmat | fwereade, not expressing as constraints opens up conflict opportunity even more | 21:15 |
| hazmat | as you have orthogonal lanes of expression | 21:16 |
| fwereade | hazmat, it's easier to explain that explicit placement directives override constraints -- just as --to does today -- than to enumerate how variousconstraints allplay together, especially when inherited constraints comeinto play | 21:17 |
| natefinch | hazmat: if you have instance-type=m1.small cpu-cores=4 ... what do you do? | 21:17 |
| fwereade | natefinch, hazmat: heh, I think we do what we did in python and complain that those just don't play well together | 21:18 |
| hazmat | natefinch, error to the user on conflicted constraint | 21:18 |
| hazmat | natefinch, thats a case of always conflict | 21:19 |
| hazmat | fwereade, the different levels for constraint always get realized in a concrete and introspectable form that's self-consistent. placements aren't really overrides though, there additions to. | 21:19 |
| hazmat | fwereade, and really i'd want to be able to specify them at the service level as well | 21:19 |
| fwereade | hazmat, I think placements are end-runs around constraints | 21:20 |
| fwereade | hazmat, did jitsu deploy-to honour constraints? ;) | 21:20 |
| hazmat | fwereade, so as an end user.. it seems quite strange to specify effectively the same thing twice.. ie. what on and where.. are really just where do i want to run this workload | 21:21 |
| hazmat | fwereade, fair enough. although that tempts me to resurrect all the discussion on why that's broken ;-) | 21:22 |
| fwereade | haha | 21:22 |
| hazmat | thats an interesting perspective though | 21:22 |
| hazmat | fwereade, do you have any other examples of placement as an end run around constraint | 21:25 |
| hazmat | i guess --to is enough by itself, but its quite specialized. | 21:25 |
| hazmat | interestingly enough --to and manual placement are the basis for juju behaving like chef or puppet. | 21:25 |
| hazmat | er.. manual provisioning | 21:25 |
| hazmat | fwereade, thanks for the discussion, happy to continue another day | 21:27 |
| fwereade | yeah, it does definitely end up somewhere quite similar if you focus on those in particular, but I still see them as building blocks for more interesting stuff... we're just not doing it yet | 21:27 |
| fwereade | hazmat, cheers :) | 21:27 |
| wallyworld | fwereade: hey, where did you get to with the instance type selection? | 21:53 |
| fwereade | wallyworld, nowhere I'm afraid -- I got caught up in talk of logging, and decided that that's upsetting more people right now | 21:53 |
| wallyworld | fair enough | 21:54 |
| wallyworld | we can target 1.18 then | 21:54 |
| wallyworld | i assume we are still on track for 1.16 today? | 21:55 |
| natefinch | fwereade: I have some code towards an instance-type constraint for ec2 btw | 21:57 |
| fwereade | wallyworld, yeah, I think so, I still hope to have something saner for the logging shortly | 21:58 |
| fwereade | natefinch, cool -- we'll want it to work for openstack too, I think :) | 21:58 |
| natefinch | fwereade: btw, I'd really like to get the ec2 root disk stuff into 1.16, but with the bot not able to merge it, I'm kinda stuck. Not sure what to do about that | 21:59 |
| bac | hi sinzui, charmworld has a dependency on ppa:ce-orange-squad/experimental which does not yet have saucy packages. would it be possible to update that ppa? | 21:59 |
| wallyworld | natefinch: what's the issue with the bot? | 21:59 |
| fwereade | natefinch, crap, sorry, I was not paying proper attention -- but wallyworld I think *does*know the bot | 22:00 |
| * wallyworld knows a little | 22:00 | |
| natefinch | wallyworld: it seems to not be updating a dependency in goamz that my branch needs | 22:00 |
| sinzui | bac: I think so | 22:00 |
| wallyworld | natefinch: i can try and manually update the goamz tree | 22:00 |
| fwereade | I at least ought to know where to find out about the bot though... wallyworld, so we have useful instructions anywhere? is it a matter of trawling though old emails? | 22:00 |
| bac | sinzui: that would be swell. would allow 'make check' from a dev box that is saucified. | 22:00 |
| wallyworld | fwereade: i think so. there's an ip address that i ssh in to | 22:01 |
| sinzui | ah, I always build in precise so I haven't even used the raring | 22:01 |
| wallyworld | fwereade: 10.55.32.52 | 22:01 |
| wallyworld | and from there you can su to the tarmac user etc | 22:01 |
| wallyworld | i find the ip address via nova list | 22:02 |
| wallyworld | using the correct credentials for canonistack | 22:02 |
| wallyworld | natefinch: so you just need goamz updated to tip? | 22:02 |
| natefinch | wallyworld: yeah | 22:02 |
| sinzui | bac: I am attempting a quick copy, but expect Lp to smack me | 22:02 |
| wallyworld | natefinch: ok, give me a sec | 22:02 |
| natefinch | wallyworld: thanks for the help | 22:03 |
| sinzui | bac, do you need elasticsearch as well as charm-tools? | 22:03 |
| wallyworld | natefinch: now on rev 43 | 22:04 |
| davecheney | sidnei: ping | 22:04 |
| wallyworld | see if that works | 22:04 |
| natefinch | wallyworld: awesome, thanks | 22:04 |
| wallyworld | np :-) | 22:04 |
| bac | sinzui: i think just es-java | 22:04 |
| sinzui | looks like this will work. I see a wait for publication | 22:05 |
| bac | cool, thanks sinzui | 22:05 |
| wallyworld | natefinch: looks like the dependencies.csv is not updated | 22:06 |
| wallyworld | it only says rev 42 | 22:06 |
| wallyworld | so that is the issue i think, assuming folks have done the tarmac set up to look there for twhat revs to use | 22:06 |
| wallyworld | ah, dependencies.tsv | 22:07 |
| wallyworld | natefinch: so you should update that file in your current juju-core branch you are trying to land | 22:07 |
| wallyworld | be sure to use the correct rev id also | 22:08 |
| wallyworld | not just the rev number | 22:08 |
| wallyworld | you can find the rev id using the --show-ids arg to bzr log | 22:09 |
| sinzui | bac: still waiting. I see the charm-tools has already arrived | 22:09 |
| sinzui | bac, I think everything is in saucy now | 22:17 |
| natefinch | wallyworld: dependencies.tsv is correct in my branch | 22:21 |
| wallyworld | ok | 22:21 |
| wallyworld | must not be used yet then i guess | 22:21 |
| sinzui | wallyworld, natefinch I don't think it is used but gobot yet. I use it for release tarballs and packaging building | 22:27 |
| wallyworld | ah ok | 22:28 |
| sinzui | I believe jam and mgz tinker when they see deps change | 22:28 |
| wallyworld | i updated goamz manually just before | 22:28 |
| wallyworld | i miss python :-( | 22:28 |
| wallyworld | i just don't get why Go folks bury their heads in the sand with dep management | 22:29 |
| * sinzui is watching his new script build juju-core debs for 1.16.0 testing | 22:29 | |
| wallyworld | yay | 22:29 |
| sinzui | And I have | 22:31 |
| sinzui | 8 minutes from the moment we think we have a rev, I can have a deb that I can install to build tools and test | 22:31 |
| davecheney | \o/! | 22:32 |
| sinzui | wallyworld, in fact. I think fwereade believes this open bug is quite a bit of work: https://launchpad.net/juju-core/+milestone/1.16.0 | 22:32 |
| wallyworld | sinzui: the logging changes? | 22:33 |
| davecheney | if it's to big | 22:34 |
| davecheney | there is a simpler solution | 22:34 |
| sinzui | wallyworld, davecheney: I can start testing tip of lp:juju-core/1.16 after dinner. I can know if tip is a viable rc candidate. Then we can decide if we want to include the logger fix. | 22:34 |
| davecheney | fix the default logger so it uses this config | 22:34 |
| wallyworld | ok | 22:34 |
| davecheney | export JUJU_LOGGING_CONFIG='juju=DEBUG;juju.rpc=INFO' | 22:34 |
| davecheney | job done | 22:34 |
| sinzui | wallyworld, that bug is half-regression. We did mean to change it, but we need some behaviour put back | 22:34 |
| wallyworld | ok, so davecheney's idea above may work? | 22:35 |
| davecheney | it works | 22:35 |
| davecheney | i use that line when deploying | 22:35 |
| wallyworld | i'm not sure where to set the default logging config, but i can look | 22:35 |
| davecheney | so where the current default logging config just says 'juju=WARNING' | 22:36 |
| davecheney | just change it to | 22:36 |
| davecheney | export JUJU_LOGGING_CONFIG='juju=DEBUG;juju.rpc=INFO' | 22:36 |
| davecheney | to get the old behavior | 22:36 |
| wallyworld | ok | 22:36 |
| bigjools | not sure why someone thought this was a good idea, but constraining by host name in maas requires a tag with the hostname. WTF | 22:36 |
| davecheney | .... | 22:36 |
| wallyworld | sinzui: i can do a branch for davecheney's change ready for after your dinner if you want | 22:37 |
| sinzui | wallyworld, I appreciate your effort. | 22:37 |
| wallyworld | np :-) | 22:37 |
| * wallyworld creates a branch and sets to work | 22:38 | |
| natefinch | bigjools: we added support for maas tags, that's all that's different from 1.14. | 22:38 |
| sinzui | I am off to feed children, but I leave you with my rediscovery of an old commandL | 22:39 |
| sinzui | bzr cat -d lp:juju-core/1.16 dependencies.tsv -r -1 | 22:39 |
| fwereade | davecheney, wallyworld: there is an even simpler solution if it comes to it -- just log at INFO level by default, that'll get you hook output | 22:40 |
| bigjools | natefinch: ok, I was told that juju core expects tags matching the hostname for hostname constraining, which is a bit crackful. | 22:40 |
| wallyworld | fwereade: i'm not across it fully. it seems though that davecheney needs debug level in places? | 22:41 |
| fwereade | davecheney, wallyworld: but the trouble is that we don't really want to drop the rpc logging | 22:41 |
| fwereade | davecheney, wallyworld: and what we *really* need I think is the ability to filter debug-log a bit more sanely | 22:41 |
| natefinch | bigjools: I think that was my mistake... someone asked about making tags for hostnames in maas | 22:41 |
| fwereade | davecheney, but I may be misunderstanding your particular use case | 22:41 |
| wallyworld | fwereade: won't using JUJU_LOGGING_CONFIG='juju=DEBUG;juju.rpc=INFO' as per dave's suggestion work? as a short term 1.16 fix? until get get it right for 1.18? | 22:42 |
| bigjools | natefinch: yeah we're missing maas-name, or equivalent | 22:42 |
| fwereade | wallyworld, davecheney: jam and rogpeppe are in agreement that there are several bugs they would have had serious trouble with if they hadn't had the rpc spam to look at after the fact | 22:42 |
| wallyworld | ah developer vs user friction :-) | 22:43 |
| natefinch | bigjools: yeah, we don't have that. Someone asked me to do maas tags, so I did that. | 22:43 |
| fwereade | bigjools, natefinch: ehh, I thought we'd agreed to drop maas-name some time back | 22:43 |
| wallyworld | fwereade: tim has put in a way to dynamically change logging | 22:43 |
| fwereade | bigjools, natefinch: if we need it, I think it's a placement request | 22:43 |
| wallyworld | fwereade: why can't we default to sane values for user | 22:43 |
| * natefinch ducks and lets fwereade and bigjools duke it out | 22:43 | |
| wallyworld | and allow devs to change as necessary | 22:43 |
| davecheney | fwereade: right, so at the moment you have no RPC logging data at all | 22:43 |
| jpds | fwereade: Filing one now. | 22:44 |
| davecheney | how does your solution to #1237151 resolve jam and rogers concerms ? | 22:44 |
| fwereade | davecheney, this is true | 22:44 |
| bigjools | fwereade: people are depending on placement using hostnames in the python app | 22:44 |
| bigjools | so this is a juju-core adoption blocker for them | 22:44 |
| fwereade | davecheney, we log at DEBUG by default; badge unit log output with names; and make debug-log a little bit closer to python by implementing (at minimum) --include | 22:45 |
| jpds | Yeah, pyjuju did maas-name, and I'm filing a bug for maas-name in juju-core now. | 22:45 |
| davecheney | fwereade: please clarify | 22:46 |
| davecheney | you would LIKE to log at DEBUG | 22:46 |
| davecheney | or you think we currently log at DEBUG ? | 22:46 |
| * wallyworld gets the popcorn | 22:46 | |
| fwereade | davecheney, there is some degree of upset that we don't, on the basis that debug logging of rpc has been repeatedly helpful in diagnosing problems via user-submitted logs | 22:47 |
| davecheney | fwereade: please read https://bugs.launchpad.net/juju-core/+bug/1237151 | 22:47 |
| davecheney | the deafult logging level was raised to WARNING for the agents | 22:47 |
| davecheney | this caused me to table flip | 22:48 |
| davecheney | i want it back where it was | 22:48 |
| davecheney | i think you do as well | 22:48 |
| fwereade | davecheney, I do, the problem is setting rpc logging to TRACE | 22:48 |
| fwereade | davecheney, what debug output were you after in particular though? | 22:49 |
| davecheney | fwereade: i don't want to workship solutions | 22:49 |
| davecheney | I just want the 1.14.0 behavior back | 22:49 |
| davecheney | it wasn't perfect | 22:49 |
| davecheney | but everyone knows how to read the logs in that format | 22:49 |
| davecheney | as of now | 22:49 |
| davecheney | when my environment fucks out | 22:49 |
| davecheney | I have zero logging | 22:50 |
| davecheney | tim said he would fix it last week | 22:50 |
| davecheney | he forgot | 22:50 |
| fwereade | davecheney, ok then, I agree, now is probably not the time to aim for perfect, let's just treat it as a regression and go back to DEBUG across the board for now? | 22:50 |
| davecheney | fwereade: agreed | 22:50 |
| wallyworld | davecheney: fwereade: so JUJU_LOGGING_CONFIG='juju=DEBUG;juju.rpc=INFO' then? | 22:51 |
| wallyworld | or debug everywhere? | 22:51 |
| fwereade | wallyworld, 'juju=DEBUG' please | 22:51 |
| wallyworld | ok | 22:51 |
| davecheney | fwereade: wallyworld thanks | 22:52 |
| wallyworld | np | 22:53 |
| fwereade | bigjools, jpds: ok, so, maas-name is a bad constraint because it makes no sense at a service level | 22:55 |
| bigjools | I agree | 22:56 |
| bigjools | there are almost certainly better ways | 22:56 |
| fwereade | bigjools, jpds: would you object to placement syntax? eg `juju deploy whatsit --to envname:hostname`? | 22:56 |
| jpds | fwereade: Right, but I can do that to bootstrap? | 22:57 |
| bigjools | I think hostname constraints generally are weird in a cloud | 22:57 |
| jpds | It's not a cloud. | 22:57 |
| jpds | It's deploying a cloud on top of MAAS. | 22:57 |
| fwereade | jpds, if you could, would that suffice? | 22:57 |
| bigjools | MAAS gives you a cloud, so yeah, it is | 22:57 |
| jpds | I want to be able to say, install mongodb on this little VM I prepared and don't touch my precious metal. | 22:58 |
| jpds | fwereade: Yes. | 22:58 |
| jpds | s/mongodb/bootstrap node/ | 22:59 |
| fwereade | jpds, ok, great, thanks; and if that were internally icky, how would you feel about putting it in environment config? say, `bootstrap-placement: hostname` | 23:00 |
| jpds | fwereade: Well, it's not just the bootstrap I care about. | 23:00 |
| davecheney | fwereade: it's more comlex than that | 23:00 |
| davecheney | all the ceph nodes have to land on the machines with two luns | 23:01 |
| wallyworld | fwereade: i've not had much to do with logging set up - i can't find where to change the default logging level for juju from warning to debug | 23:01 |
| davecheney | all the compute node have to NOT land on any of the machines with two luns | 23:01 |
| jpds | fwereade: I know that I have servers with two disks, to I need to be able to deploy to those specifically. | 23:01 |
| fwereade | jpds, ok,sorry I misled, I was only fretting about the interaction between bootstrap and --to | 23:01 |
| davecheney | wallyworld: http://paste.ubuntu.com/6215921/ | 23:01 |
| davecheney | this is what I have so far | 23:01 |
| jpds | For things like cinder, swift, ceph like davecheney said. | 23:01 |
| fwereade | wallyworld, so at the moment it comes from loggo itself | 23:02 |
| wallyworld | oh really???? | 23:02 |
| wallyworld | the library???? | 23:02 |
| wallyworld | :-( | 23:02 |
| fwereade | jpds, assuming `--to envname:hostname` works everywhere except bootstrap, would using env config for bootstrap hurt too badly? | 23:03 |
| jpds | fwereade: I opened https://bugs.launchpad.net/juju-core/+bug/1237709 if you want to add your thoughts to it. | 23:03 |
| fwereade | jpds, thanks | 23:03 |
| davecheney | wallyworld: nah, that can't be it | 23:03 |
| davecheney | we control the root logger in cmd/logging.go | 23:03 |
| wallyworld | davecheney: so you are saying the by changing the cmd logging, when the agents have workers started, the correct logging will be used? | 23:03 |
| jpds | fwereade: Shouldn't. | 23:03 |
| fwereade | wallyworld, davecheney: it so is. if it's not set there it falls all the way back to loggo's default, which is WARNING | 23:03 |
| davecheney | wallyworld: i don't understand what you said | 23:03 |
| fwereade | wallyworld, set-env logging-config="blahblahblah" | 23:04 |
| fwereade | wallyworld, will work after the fact | 23:04 |
| davecheney | if you don't want to touch the code | 23:04 |
| davecheney | add an env var to the upstart scripts | 23:04 |
| davecheney | ok, that needs touching code and tests as well | 23:04 |
| fwereade | jpds, cheers | 23:04 |
| fwereade | davecheney, no, please do *not* touch the upstart scripts | 23:05 |
| fwereade | davecheney, wallyworld: there's an environment config setting | 23:05 |
| wallyworld | davecheney: you said you wanted agents to use debug, so i was reasoning that your pastebin did that by changing cmd default logging level cause that's how agent workers are started | 23:05 |
| fwereade | davecheney, wallyworld: that will work just fine | 23:05 |
| wallyworld | but changning cmd logging also affects user cli commands | 23:06 |
| wallyworld | doesn't it? | 23:06 |
| wallyworld | which we don't want | 23:06 |
| wallyworld | why can't we just have agent workers started with debug flag | 23:06 |
| davecheney | wallyworld: yes, it will probably revert some behavior | 23:06 |
| wallyworld | or, make the loggo api call to set the agent logger to use debug | 23:07 |
| wallyworld | ie in agent.go we have var logger = loggo.GetLogger("juju.agent") | 23:08 |
| davecheney | sgtm | 23:08 |
| wallyworld | if i make that logger default to debug, that should fix it? | 23:08 |
| davecheney | wallyworld: my patch proposed above fixes the bug | 23:10 |
| davecheney | the cli is not affected | 23:10 |
| wallyworld | davecheney: so it's effectively these 2 lines | 23:11 |
| wallyworld | +// quick fix for LP #1237151 | 23:11 |
| wallyworld | +level := loggo.INFO | 23:11 |
| wallyworld | but that's info, not debug? | 23:12 |
| wallyworld | and it will affect cli commands as well | 23:12 |
| davecheney | it does affect them | 23:13 |
| davecheney | but is invisible without -v | 23:13 |
| davecheney | which sets the logging level to debug anyway | 23:13 |
| wallyworld | maybe we need to change jujud | 23:13 |
| wallyworld | -v is deprecated | 23:13 |
| wallyworld | i think jujud is the correct approach isn't it, since that is used to launch the agent workers | 23:14 |
| wallyworld | so cli not affected | 23:14 |
| fwereade | wallyworld, agents run a logger worker that takes it from environment config | 23:14 |
| fwereade | wallyworld, that's where it needs to be set | 23:14 |
| wallyworld | so jujud registers these commands | 23:15 |
| wallyworld | jujud.Register(&BootstrapCommand{}) | 23:15 |
| wallyworld | jujud.Register(&MachineAgent{}) | 23:15 |
| wallyworld | jujud.Register(&UnitAgent{}) | 23:15 |
| wallyworld | jujud.Register(&cmd.VersionCommand{}) | 23:15 |
| wallyworld | if we can find a way to set up logging for those as we want, i think that seems best? and isolated from other things? | 23:16 |
| fwereade | wallyworld, environs/config/config.go:107 | 23:16 |
| fwereade | wallyworld, that's where we take it from loggo if not otherwise set | 23:17 |
| wallyworld | fwereade: so if i change jujud to set the env variable? | 23:17 |
| fwereade | wallyworld, it won;t work | 23:17 |
| fwereade | wallyworld, the logger worker will overwrite it with what's in env config | 23:17 |
| wallyworld | or poke it into logging-config | 23:17 |
| wallyworld | i am -1 on doing something that also affects the cli | 23:18 |
| davecheney | wallyworld: it doesn't affect the cli | 23:18 |
| davecheney | it is masked/reset/something but other forces | 23:18 |
| davecheney | https://codereview.appspot.com/14595043 | 23:19 |
| davecheney | use it or throw it out the window | 23:19 |
| wallyworld | davecheney: it looks like your change will have the cli use debug instead of warning by default | 23:21 |
| fwereade | davecheney, wallyworld: except, fuck *that* looks like it's affected by what's been set up in cmd/logging.go, by means of spooky action at a distance | 23:21 |
| wallyworld | the cli was changed so that it only showed warning, unless showlog was epcified, then it showed info | 23:22 |
| davecheney | wallyworld: lucky(~) % juju status | 23:22 |
| davecheney | environment: ap-southeast-2 | 23:22 |
| davecheney | machines: | 23:22 |
| davecheney | "0": | 23:22 |
| davecheney | agent-state: started | 23:22 |
| davecheney | agent-version: 1.17.0.1 | 23:22 |
| davecheney | dns-name: ec2-54-253-151-61.ap-southeast-2.compute.amazonaws.com | 23:22 |
| davecheney | instance-id: i-8646ffba | 23:22 |
| davecheney | instance-state: running | 23:22 |
| davecheney | series: precise | 23:22 |
| davecheney | hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M | 23:22 |
| davecheney | services: {} | 23:22 |
| davecheney | yet surprisingly, it didn't | 23:22 |
| wallyworld | the above is with your change? | 23:23 |
| davecheney | yup | 23:23 |
| davecheney | lucky(~) % juju deploy mysql | 23:23 |
| davecheney | lucky(~) % | 23:23 |
| davecheney | silent, as the grave | 23:23 |
| wallyworld | that's because show log was not specified | 23:24 |
| wallyworld | if you specify show log, then you get debug | 23:24 |
| wallyworld | but you don't want that, you want info | 23:24 |
| wallyworld | unless debug is asked for | 23:24 |
| wallyworld | so, the behaviour is: | 23:25 |
| wallyworld | anything >= warning, print to stderr | 23:25 |
| wallyworld | info only show if showlog = true | 23:25 |
| wallyworld | or, if debug is specified, show that if showlog = true | 23:26 |
| wallyworld | your change makes it so that debug is the default rather than info for the case when show log is asked for | 23:26 |
| davecheney | it sure does | 23:27 |
| wallyworld | so we need for confine the changes to only stuff running on the nodes ie launcged by jujud | 23:27 |
| davecheney | ok, lets to fwereade 's ida | 23:27 |
| davecheney | ida | 23:27 |
| davecheney | idea | 23:27 |
| wallyworld | we don't want debug for cli | 23:27 |
| wallyworld | by default | 23:27 |
| fwereade | davecheney, wallyworld: if we could reliably tell when the logging config was the loggo default, I think we could just handle it in config.go where I pointed you | 23:28 |
| wallyworld | sounds ok i think. i was just trying to justify why i was -1 on the changes being done in cmd/logging | 23:29 |
| wallyworld | fwereade: we could add a default schema value for "logging-config" ie <root>=WARNING;juju.agent=DEBUG;juju.rpc=DEBUG | 23:35 |
| davecheney | wallyworld: lets try that | 23:36 |
| fwereade | wallyworld, I'd prefer <root>=DEBUG, I don;t think that affects the CLI, doesit? | 23:36 |
| wallyworld | cause line 107 for config.go - by default loggo ignore loggers with unspecified levels | 23:36 |
| wallyworld | fwereade: not sure. probably not actually | 23:37 |
| fwereade | wallyworld, so the problem with a default is that we want to take it from the env if it's set | 23:37 |
| wallyworld | fwereade: we will do | 23:37 |
| wallyworld | i think | 23:37 |
| wallyworld | ah no | 23:37 |
| fwereade | wallyworld, it can be done but will take work and testing, I think | 23:37 |
| fwereade | wallyworld, well everything will, but it's not entirely trivial, I mean | 23:37 |
| wallyworld | so order of precedence is: 1. ENV, 2. yaml ? | 23:38 |
| wallyworld | yaml = logging-config | 23:38 |
| wallyworld | in which case i can flip the check in config | 23:38 |
| fwereade | wallyworld, I *think* that we expect bootstrap's --logging-config > yaml > env | 23:38 |
| fwereade | wallyworld, and the problem is that the path from bootstrap into envconfig is via loggo rather than passed directly | 23:39 |
| wallyworld | fwereade: so - in jujud, why can't i just set the env when the commands are launcged | 23:39 |
| wallyworld | so logging-config still takes precedence | 23:40 |
| fwereade | wallyworld, because the logger worker will go and get the real value out of env config and set it | 23:40 |
| fwereade | wallyworld, ha | 23:40 |
| fwereade | wallyworld, you know what we could do, I think | 23:40 |
| wallyworld | but logging-config will be "" by default no? | 23:40 |
| wallyworld | so won't my method work because of that? | 23:41 |
| fwereade | wallyworld, the config will be whatever cameout of loggo in the absence of other instructions | 23:41 |
| fwereade | wallyworld, ie <root>=WARNING | 23:42 |
| wallyworld | ah. i thought it would be empty because loggers by default have level=unspecified | 23:42 |
| wallyworld | but maybe it sets root regardless | 23:42 |
| fwereade | wallyworld, the root logger starts out as WARNING | 23:42 |
| wallyworld | would it be too dirty to check for that? | 23:43 |
| fwereade | wallyworld, I'm starting to feel tempted :( | 23:43 |
| wallyworld | yeah :-( | 23:43 |
| * natefinch contemplates raid 0 w/ 3 SSDs just to speed up his tests | 23:45 | |
| fwereade | wallyworld, ok, I think it's *slightly* less bad than the alternative, which would be to add *more* spooky action at a distance so config.go could tell whether any logging had been set explicitly | 23:46 |
| fwereade | wallyworld, ok, do it, and assign thumper a bug to unfuck it when he comes back | 23:47 |
| wallyworld | fwereade: ok. so i'll dick with config.go to leave logging-config env setting "" if loggo info comes back as "root=<WARNING>" (or whatever) | 23:48 |
| wallyworld | and then i need to change jujud to set logging config in env to juju=DEBUG | 23:48 |
| natefinch | When someone gets a chance - constraint to specify ec2 instance types by name: https://codereview.appspot.com/14523052/ | 23:55 |
| * natefinch is well past EOD | 23:56 | |
| natefinch | g'night all | 23:56 |
| fwereade | wallyworld, do not touch jujud | 23:57 |
| wallyworld | yeah, just figured that out | 23:57 |
| wallyworld | i'm changing config.go | 23:57 |
| fwereade | wallyworld, if loggo info comes back as default set something less crazy in logging-cnfog | 23:57 |
| wallyworld | if c.asString("logging-config") == "" { | 23:57 |
| wallyworld | if environmentValue := os.Getenv(osenv.JujuLoggingConfig); environmentValue != "" { | 23:57 |
| wallyworld | c.m["logging-config"] = environmentValue | 23:57 |
| wallyworld | } else { | 23:57 |
| wallyworld | loggoConfig := loggo.LoggerInfo() | 23:57 |
| wallyworld | if loggoConfig != "<root>=WARNING" { | 23:57 |
| fwereade | er, you know what I mean | 23:57 |
| wallyworld | c.m["logging-config"] = loggoConfig | 23:57 |
| wallyworld | } else { | 23:57 |
| wallyworld | c.m["logging-config"] = "<root>=DEBUG" | 23:57 |
| wallyworld | } | 23:57 |
| wallyworld | } | 23:57 |
| wallyworld | } | 23:57 |
| wallyworld | does the above look ok? | 23:57 |
| fwereade | yeah, sounds good | 23:57 |
| wallyworld | i'll add todo and bug # etc | 23:58 |
| fwereade | wallyworld, perfect, you anticipate my clumsy typing :) | 23:58 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!