[00:02] SpamapS: all 5 issues your reported for limesurvey should now be fixed [00:03] SpamapS: thanks again for your help [00:20] Can Juju manage deployments between different environments? [00:21] Like, I've got three bare metal servers, so to juju that's three environments - but I would like to relate things between environments [02:06] <_mup_> Bug #900104 was filed: watch_related_units callback is icky < https://launchpad.net/bugs/900104 > [06:10] marcoceppi: 3 bare metal servers would just be one env if you used orchestra [06:26] * marcoceppi needs orchestra [06:26] Thanks SpamapS [06:44] marcoceppi: to the bigger question.. right now envs are decent ways to group availability zones.. so crossing envs usually involves something special to handle the latency between two envs. :) [06:58] hrm.. limesurvey's sessioning doesn't seem to work right [06:58] loses my session whenever it load balances to another node === highvolt1ge is now known as highvoltage [08:37] <_mup_> Bug #900186 was filed: no spec for machine placement story < https://launchpad.net/bugs/900186 > [10:45] <_mup_> Bug #900227 was filed: guarantee start hook execution on machine boot < https://launchpad.net/bugs/900227 > [11:33] !Rejoin [11:33] :P [12:10] TheMue: to check out the sources [12:11] it's best, i think, to create a directory to hold all the branches [12:11] in bazaar, each branch has its own directory [12:11] i use $HOME/src/juju [12:11] reminds me of svn [12:12] Houston, we have a problem! My ThinkPad stopped charging its batteries! [12:13] i usually keep one pristine directory for each branch, representing the trunk [12:13] mainerror: ouch [12:13] mainerror: so your name matches, that's bad (in this case) [12:13] -_- [12:13] TheMue: then to check out the python sources, for example, i'd do: cd $HOME/src/juju; bzr branch lp:juju juju-trunk [12:13] Sad mainerror is sad. [12:14] TheMue: to check out the go juju sources, bzr branch lp:juju/go go-trunk [12:14] oops, forgotten one crucial step [12:15] TheMue: before you start checking things out, should make a shared repo so all the branches can share data [12:16] TheMue: in my case, cd $HOME/src/juju; bzr init-repository [12:16] ok [12:16] TheMue: then you can branch to your heart's content. [12:16] needs a locations, so in your case just . ? [12:17] yeah [12:17] fine [12:17] TheMue: before you start work on some source code, create a branch for that item of work - usually the branch directory name reflects the work that you're doing [12:18] TheMue: i usually start when a generic name then change it to something more specific when i know what the merge request does [12:18] rog: ok, very important hints [12:19] rog: I'm currently fetching the sources, so it works fine [12:19] cool [12:19] TheMue: when you actually want to submit some code for review, you should use Gustavo's lbox tool [12:19] TheMue: (you can get it with apt-get install lbox) [12:19] Hello jujuers [12:19] niemeyer: yo! [12:20] moo gustavo [12:20] heya niemeyer [12:21] rog: which repository for lbox? [12:21] TheMue: good point, i can't remember. one mo. [12:25] TheMue: sudo add-apt-repository ppa:gophers/go [12:27] lbox installed O> [12:29] TheMue: You can get the full workflow here: http://blog.labix.org/2011/11/17/launchpad-rietveld-happycodereviews [12:31] niemeyer: thx, I've already read it after you've published it. but now I have to dig deeper. [12:35] hey guys. Great to see that my charm has been put into lp:charm/limesurvey. Small question for you: what's the process for me to upload changes to it from now on? Make the changes in my branch and request a merge? [12:37] ok, afk for lunch [13:27] niemeyer: greetings. nothing to report for goyaml, it builds out of the box with weekly. [13:27] mpl: Fantastic [13:27] mpl: I think we're good, as far as weekly goes [13:28] mpl: I have a suggestion that may enable you to collaborate more closely with rog [13:28] mpl: with something more involved [13:28] cool [13:28] * rog is listening [13:28] mpl: We need to introduce ssh support [13:28] mpl: We use ssh in a few different ways [13:29] mpl: I suggest you spend some time looking at the Python implementation to understand some of the background [13:29] mpl: Before diving onto the implementation [13:29] mpl, rog: I'd like to try using exp/ssh for the forwarding [13:30] niemeyer: the zookeeper connection? [13:32] niemeyer: when you say the python implementation, you mean the parts of juju that use ssh? [13:33] rog: Yeha [13:33] mpl: That's right [13:33] niemeyer: seems like a good idea. [13:34] mpl: Check out juju/state/ssh*.py, and juju/providers/ec2/* for hints of how that [13:34] 's used [13:34] rog, niemeyer: so how is the zookeeper connection and that forwarding you're talking about done at the moment in the go port? [13:35] mpl: There's nothing [13:35] not at all, or with some other way than ssh? [13:35] oh ok. [13:35] mpl: haven't even got as far as starting zk yet... [13:37] rog: how do you suggest we go about that? shall I read those bits of code and report to you when I'm done or when I have questions about it? [13:38] mpl: Yeah, the first task is to understand how it works today, so you have an idea of what has to be done [13:38] mpl: i suggest you look at the way the python code does its ssh connection, and see if you can replicate that with a local zookeeper instance (started by gozk) and an independent go program, all running on the same machine [13:39] niemeyer: that sound reasonable? [13:39] mpl: Basically, the zk connection is insecure at the moment, and we use ssh to fix that [13:39] rog: +1 [13:40] got it, thanks. === hazmat` is now known as hazmat [14:50] Lunch time [14:50] biab [15:39] marcoceppi: howdy [15:39] marcoceppi: I was off on Friday other than the charm school, how's it going? === jcastro_ is now known as jcastro [15:48] jcastro: good [15:49] marcoceppi: hey so, just so I have the spreadsheet clear, which one are you working on now? [15:49] phpmyadmin? [15:49] or did that one get accepted? [15:50] it's just about ready, I need to finish the package based installation: found a few bugs [15:50] ah awesome [15:50] any idea what george is working on next? [15:50] Steam is in a holding pattern while I try to get all the game's server configs sorted out. They're not included by default and poorly documented game to game, I've also got Redmine going [15:50] jcastro: no idea [15:51] * jcastro nods [15:51] I see nijaba rocked out limesurvey [15:51] yeah, I gave it a spin over the weekend, it rule [15:51] s [15:51] jcastro: just keeping my word to you :) [15:52] nijaba: that's going to be a great charm, it's actually very useful to people and very cloudy ... up when you need it, and then goes away when you don't [16:11] seriously.. limesurvey is fun... [16:11] I created a survey.. "can you crash this site?" Y/N [16:11] was thinking we could give it a shot with a massive scale-out [16:12] * SpamapS moves from cold drafty downstairs to overly warm ovenlike upstairs [16:14] SpamapS, you're in LA. how can it be cold? in contrast it's already at a toasty 3 deg F outside [16:17] I noticed something that might be a bug, but I'm not sure. If you do a deploy of mysql charm + charm that requires mysql, add-relation, upgrade non mysql charm, destroy charm that required mysql, deploy again and add-relation MySQL charm doesn't send over the relation information. [16:20] Should MySQL delete the database after a service has broken the relation? [16:22] jimbaker: 45F is "cold" here. ;) [16:22] marcoceppi: no [16:22] marcoceppi: but it should re-set the info [16:23] print "database exists, assuming configuration has happened already" [16:23] sys.exit(0) [16:23] marcoceppi: definitely broken [16:23] Okay, so if it detects that the database is already there, it should send over the information again. Seems like that would put a lot of work on the charm to check if database already exists to avoid re-installing it [16:24] marcoceppi: honestly, the "db" relation is silly the way it uses the service name for the database. [16:24] marcoceppi: puts a lot of restrictions on how you can use the relation [16:24] I think, and I could be wrong - because I understand the use case of keeping the database around, a db-relation-broken should remove the database [16:25] marcoceppi: mysql-shared is a more realistic use of mysql [16:25] marcoceppi: no, we want to make data cleanup manual [16:25] marcoceppi: thats also why machines aren't terminated automatically [16:25] cool, well I'm having issues with db-admin-relation-changed, and I just realized it shouldn't be creating a database in the first place! [16:25] marcoceppi: *exactly* [16:25] So my point is moot [16:25] but it's a bug of another color [16:26] there's also a bug where the username/password is *always* randomly generated.. [16:26] so when the relation is broken.. we can't cleanup the users [16:27] I thought MySQL charm stored that data locally for tracking? [16:28] * marcoceppi was using an older version of the MySQL charm [16:29] user = subprocess.check_output(['pwgen', '-N 1', '15']).strip().split("\n")[0] [16:29] marcoceppi: thats in common.py [16:29] Should the charm track user/pass per private IP? [16:29] IP? [16:29] no [16:29] private host* [16:30] 1 service, 1 user, but it can grant that user access from all the addresses of the service [16:31] this is something that confused me early on ... you get *one* set of relation data that you can set.. [16:31] so the mysql unit's relation-set must work for *all* the related units on the other side. [16:31] you can't have a user per unit without doing relation-set wordpress-0-user=foo [16:31] then thats pointless because they all get access to it anyway [16:38] So, it's really per service then [16:38] Should the charm store the user/pass info on a per service basis? or does it do that already? [16:39] it must actually [16:39] but it doesn't right now [16:40] Hum, I suppose plain text storage would be too much to ask for, in regards to security? [16:40] Because that would probably be the easiest [16:40] Simplest thing to do is just to put it in files. [16:40] we don't have to store the password [16:40] just the usernames to clean up [16:41] but frankly.. the root PW is stored (and must be).. so.. we don't have much to protect in that manner. [16:41] Oh, I see now. Each unit of a service gets it's own username and password for the services db [16:42] no [16:42] Just when I thought I had it [16:42] each service gets a username and password [16:42] Where is that tracked then? Bootstrap? [16:42] where is what tracked? [16:42] The username and password. [16:43] in the relation settings [16:44] If it's tracked in the relation settings, why can't the charm just pull the username from that when relation is broken and do a clean up then? [16:44] * marcoceppi is poking in areas he doesn't quite understand [16:49] marcoceppi: because the way broken is triggered is the relation settings node is deleted from zookeeper ;) [16:49] marcoceppi: which is, I think, a bug. ;) [16:49] ah :) [16:49] bug #791042 to be exact [16:50] <_mup_> Bug #791042: *-relation-broken has no way to identify which remote service is being broken < https://launchpad.net/bugs/791042 > [16:50] marcoceppi: you're going through the same questions I did .. so.. if nothing else you're helping me feel less like an A-hole for reporting so many bugs ;) [16:50] \o/ [16:55] niemeyer: just fixed a little error on there: https://wiki.ubuntu.com/gozk :) [16:56] SpamapS: what you don't get an "your attention to detail is appreciated" message everytime you open a new bug? :) [17:00] Since most of my bugs are "This $@!% is $%#!ing broken %!@ing fix it!" ... no. ;) [17:02] mpl: Cheers! :) [17:02] np [17:23] SpamapS, its not actually deleted, but i'm not sure that its reachable via the cli in that case [17:26] hazmat: good to know :) [17:48] gn all :) [17:48] :P [17:48] SpamapS: sir [17:49] koolhead17: aye? [17:58] niemeyer: I get a weird behaviour with that gozk example on the wiki. the first error check passes, but then I never get passed event := <-session and I continuously get messages such as this one: [17:58] 2011-12-05 18:54:51,868:28689(0x7f3c313d0700):ZOO_ERROR@handle_socket_error_msg@1528: Socket [127.0.0.1:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 0ms) [17:58] SpamapS: am finally close to running custom cloud image + juju + openstack [17:58] and yes, I have zookeeper running, I can connect to it with zkCli.sh [17:59] mpl: Looks like it's trying to connect to the wrong server? [18:01] niemeyer: well, the server is indeed listening on 2181. besides, if that was the case, shouldn't it just fail at the first error check after gozk.Init, instead of filling my term with those messages? [18:05] hmm, wtf zkCli.sh doesn't report any error when there's no server running. [18:05] koolhead17: custom cloud image ? [18:06] rog: Have we ever done that refactoring so that all errors are sent to event channels, including temporary ones? [18:06] mpl: Well, we have tests [18:06] mpl: Can you get tests to pass? [18:07] niemeyer: off for day. will have a look tomorrow. [18:07] fwereade, take care! [18:07] SpamapS: i have my openstack inside proxy and in order to get internet on instance i have to modify the cloud image [18:11] rog, fwereade: Cheers! [18:24] koolhead17: sounds interesting! [18:30] * TheMue has to admit that his Python time has been some years ago. Will have to dig a bit into twisted. [18:33] niemeyer: after a bzr pull in gozk, there still is a problem with weekly: retry_test.go:198 it seems err.Error() should be changed to err.String() here. or String() to be changed to Error() in gozk.go [18:35] mpl: Crap, I may have missed that [18:36] niemeyer: that said, it seems my default zookeeper env is not really well setup. I'll try and dig further. [18:40] SpamapS: did you know about this? https://launchpad.net/juju-gui [18:55] mpl: There's a lost branch from rog [18:55] mpl: Not sure if it'll fix it, but I'll work on getting it merged right now, and make sure it works with weekly [18:59] niemeyer: well, I went the quick route and just changed that one to String() on my end so I could go further. [18:59] mpl: Good move [19:00] niemeyer: then the tests fail because default configuration here expects things to be in root only locations, like the log file. [19:00] and I don't run gotest as root. [19:00] mpl: Cool, I'm with you.. I'm just finishing merging that branch and solving conflicts, and will then see if there's anything else to address [19:00] mpl: Definitely, we never do that [19:01] I'll have to leave in a couple mins to get my food as the local indian though :) [19:04] bbl [19:14] mpl: "OK: 39 passed" [19:15] <_mup_> juju/zookeeper r24 committed by gustavo@niemeyer.net [19:15] <_mup_> Merged update-server-interface branch from Roger. [r=niemeyer] [19:15] <_mup_> This improves the interface used to manage the zookeeper processes. [19:15] <_mup_> This changeset also includes fixes to get the branch integrated [19:15] <_mup_> into the current state of trunk. [19:45] niemeyer: thx. I'll retry as soon as I can, and when I have a more suitable zookeeper env setup. [19:45] mpl: FWIW, I don't have a special zk env setup [19:46] mpl: I'm using packaged zk [19:46] niemeyer: I've aptitude installed it [19:46] and done nothing else more. [19:46] mpl: Me too [19:47] hmm [19:47] mpl: Are you on Oneiric? [19:47] nope, lynx. [19:47] but I'm on lynx on my other laptop, and I remember the tests passed fine there. [19:48] *laptop as well, ... [19:48] Hrm, is it possible that resolved --retry does not work? [19:49] http://paste.ubuntu.com/760813/ [19:49] anyway, I have a stupid flat tire to fix so I can go to work tomorrow, so ttyl. [19:49] Thats what I see in the logs after retrying the db relation [19:49] mpl: lynx? [19:49] niemeyer: lucid lynx [19:49] mpl: Ah [19:49] http://paste.ubuntu.com/760814/ [19:49] after that [19:49] so it didn't really retry the hook that failed [19:49] mpl: Yeah, that might be a real issue [19:51] SpamapS: Is that trunk or main? [19:51] s/main/universe [19:51] 427 [19:51] so.. basically trunk [19:51] 426 actually [19:51] ii juju 0.5+bzr426-1juju2~preci next generation service orchestration system [19:52] SpamapS: Ok, so it's worth checking with hazmat/fwereade [19:53] SpamapS: There has been work taking place in that area [19:53] I'll try to narrow it to a reproducible test case [19:53] But thus far, looks like any hook that errors is not retried [19:53] * hazmat checks traceback [19:54] bug 878462 [19:54] <_mup_> Bug #878462: resolved --retry does not retry the hook < https://launchpad.net/bugs/878462 > [20:10] hazmat: yeah I thought I had run into this before [20:27] SpamapS, before it was with wrt to non rel hooks [20:27] SpamapS, it looks like there is fix for this already in restart-transitions [20:33] hazmat: so something in flight right now? [20:33] SpamapS, yup [20:33] cool! [20:33] * SpamapS changes topic.. [20:33] so.. philosophy question here.. [20:34] mysql has about 450 variables that can be set at various times [20:34] some only at startup [20:34] some without restart.. [20:34] I'm of a mind to just expose two things in config.yaml.. "Alternative my.cnf" which would just be your hand-crafted my.cnf .. and "dynamic-global-variables" which would be a set of vars that you want to set. [20:35] SpamapS, use postgres ;-) [20:35] * hazmat wonders if that's a valid answer [20:35] Yes, its valid as long as you'd like to spend 2x as much on scaling. ;) [20:35] to be fair, you'd have 0.1% more correct data. :) [20:36] for these and other numbers, please see "my rear end" which is where I pulled them from. :) [20:36] SpamapS, those startup values be changed between starts? [20:36] Same thing applies to postgresql actually. [20:36] SpamapS, postgres has a lot less hacks masquerading as config [20:36] hazmat: some of them have really strang effects when you change them between restarts... but all of them *can* be changed. [20:37] hazmat: http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html .. which one is more of a hack than slony? [20:37] ;) [20:37] SpamapS, slony is so pre 9.0 ;-) [20:38] ok well now that we've gotten the religious argument out of it.. lets *PRETEND* that I said postgres and I was talking about wal log sizes and buffer sizes. [20:38] (lets also pretend that postgres's multi-process model can scale beyond 1000 concurrent connections too! ;) [20:39] postgres has tons of tunables .. and some things that can't be changed like charsets [20:39] so its the same philosophical situation [20:40] you have a database, which is tunable. Should we try to model all of them into a giant config.yaml .. or just give admins the tools to set them. [20:40] It would be *cooler* if they were in a config.yaml [20:40] but more work now, and later when new versions of mysql come out. [20:42] how is this not configuration management? [20:45] elmo: it is config management.. we've said all along that config management tools are useful for applying the things that one has told juju about. [20:46] SpamapS: Yes, they should be go into config.yaml over time.. is there any disagreement there? [20:46] If puppet already has a module for building my.cnf with all the settings specified or not.. I'll gladly steal that for the charm. [20:47] niemeyer: I'm asking philosophically, do we want to just let people specify their own my.cnf or build the my.cnf for them from the individual settings. [20:47] hazmat: It'd be nice to get the latest agreement in terms of the subordinate spec committed onto the documentation as a draft [20:48] SpamapS: I'm personally fine with whatever people are happiest with, but my gut feeling is that independent options are nicer to work with [20:50] niemeyer: Ok well there's another way to look at it which is to have some higher level options, like 'dataset_size' or 'transaction_safety' which adjust several options based on best practices. [20:51] SpamapS: Maybe.. having conflicting options has implications, though [20:51] niemeyer: specific overrides general is typically how I'd handle that. [20:52] SpamapS: Sure, but it's not clear what's specific and what's general [20:53] so "dataset_size=99G" is general.. "innodb_buffer_pool_size=3915852021" is general [20:53] err [20:53] specific [20:53] :-P [20:53] SpamapS: If one sets two conflicting options, one of them will win, and it's not clear what's what [20:54] SpamapS: Not advocating against it, but just recognizing that there are cognitive challenges there [20:54] sure it is.. if I set both of those, I'd fully expect innodb_buffer_pool_size to win even if normally dataset_size wins.. I'd also log a warning.. "dataset_size ignored for innodb_buffer_pool_size" [20:55] The moment where I start doing individual variables is where I stop trusting the general settings to do anything for me. [20:55] SpamapS: Me, on the other hand, being a postgres newbie, would be completely puzzled [20:55] well why are you setting a specific variable if you're a noob? [20:55] SpamapS: That's what I mean.. you're taking the stance of having full understanding of what's going on and expecting others to be equally informed. Doesn't work in practice. [20:56] Ok not sure where I took a stance. My suggestion is that we make config.yaml comprised of big easy knobs for newbies, and tiny intricate knobs for the experts [20:57] SpamapS: and my suggestion is to use caution while doing that, because setting conflicting options blindly confuses people [20:57] Yeah, I think we'd have to have it document in the description which variables it was setting. [20:58] "Changes query_cache_size, innodb_buffer_pool_size, etc. etc." [20:58] SpamapS: There are other things that may be done, such as using appropriate naming that gives hints [20:58] override_* for the variables might help [20:59] like... set dataset-size=99G override-innodb-buffer-pool-size=500G seems clear to me that I'm overriding that one variable. [20:59] SpamapS: Yeah, or the other side, wizard_*.. [20:59] IMO the charm should set those depending on the size of the instance it;s on [20:59] s/_/-/ [20:59] It's more hardware dependent then anything else [20:59] marcoceppi: you don't necessarily have permission to take all the resources on the instance you're on. [20:59] for LXC, that would be disaster. :) [21:00] they all say they have 4G, because my laptop has 4G [21:00] blurg [21:00] i wonder if they would say that if we where using the mem cgroup [21:00] probably [21:00] probably just a bug in the /proc wrapping [21:01] but thats beside the point [21:01] Even on a bare instance.. with subordination.. you don't know how much RAM to leave for instrumentation/extra stuff [21:02] I think we should strive for charms to have defaults that work best on m1.small .. and people can tune-up from there. Eventually maybe we will be able to have config settings that default to '2*constraint(mem)' [21:02] or, I suppose better would be contraint(mem)/2 ;-) [21:03] Ok for mysql I'll just expose general-dataset-size and general-query-cache-mode .. if people want to set more, they can add them as needed I suppose. [21:04] (though that gets back to the whole question of how do I fork and switch my deployed service to my local charm) [21:06] SpamapS, yeah.. we'll need to be able to upgrade across repos to support that [21:07] we've effectively said specifying a --repository doesn't affect the lookup order.. [21:07] which is problematic when your trying to say which repository to use [21:07] I think it has to be an explicit decision [21:07] "I was using that charm identifier, now please start using this one" [21:07] i guess you do so via a fully qualified reference [21:07] upgrade-charm doesn't take a charm name, it takes a service name [21:08] juju upgrade --using=cs:~fewbar/mysql database [21:08] seems like thats not an "upgrade" .. more of a switch or change [21:09] More of a psychological problem :) [21:18] SpamapS: Agreed.. it should be its own internal command, even if we implement it internally as mostly the same [21:19] s/internal command/command [21:19] juju replace database cs:~fewbar/mysql [21:20] Or some such [22:50] <_mup_> Bug #900517 was filed: config-get on an int set to 0 does not return '0' but an empty string < https://launchpad.net/bugs/900517 > [22:56] * SpamapS commits config.yaml for memcached [23:15] m_3_: ping [23:43] jimbaker, i think forget how tricky/messy this sshclient/forward code was [23:44] hazmat, i tried rewriting that code twice, so i agree with that! [23:46] i figured a rewrite might help understand why i was getting that strange behavior seen in bug 892254, but unfortunately it was too big of a hole, so i punted [23:46] <_mup_> Bug #892254: SSHClient does not properly handle txzookeeper connection errors < https://launchpad.net/bugs/892254 > [23:47] hazmat, it would be worth fixing test_share_connection if you end up fixing sshclient === mwhudson_ is now known as mwhudson [23:53] m_3_: last week we upgraded from cactus to diablo, if you guys would like to try things out again :) [23:54] I had to deal with that stupid flat tire, so I couldn't pursue that further. I'll retry tomorrow. especially since I'll have my other machine, on which I believe the tests pass. See you tomorrow.