davecheneyfwereade: thumper https://github.com/juju/juju/pull/36200:19
perrito666fwereade: ping 200:30
davecheneythumper: fwereade http://en.wikipedia.org/wiki/.local01:53
davecheneyNetworking device hostnames ending with .local are often employed in private networks, where they are resolved either via the multicast domain name service (mDNS) and/or local Domain Name System (DNS) servers.01:53
wallyworldaxw: if you have a moment, could you give https://github.com/juju/juju/pull/361 the once over as well? i've tested live on hp cloud with juju-gui deployed to ensure gui is updated as services are added/removed01:55
axwwallyworld: sure, looking01:55
menn0fwereade: https://github.com/juju/juju/pull/36401:57
fwereadeaxw, commented quickly on your doc, thoughts?01:59
axwfwereade: I was trying to avoid timers, because it seems unnecessary. we already have timer based pinging on the API, so we should be able to use connection liveness to monitor presence02:01
axwfwereade: since the state server is a machine agent, it will have an entry in the presence map too02:01
axwI just meant that we'll need to propagate that into state somehow02:01
fwereadeaxw, ok, cool -- I'm mainly just saying that I don't think we need to add to the peergrouper02:02
axwbecause EnsureAvailability needs something to decide whether a state server is healthy02:02
axwfwereade: something needs to monitor mongod's health though right? doesn't need to be the peergrouper, but some worker?02:02
fwereadeaxw, the api servers will be keeping track of their own connected clients, and those forwarded from other agents, and we may as well pipe that same info into state in the same sort of way we do for environs02:02
fwereadeaxw, do we specifically need to monitor mongo's health if we can monitor state server health?02:03
axwfwereade: no, it was just something rogpeppe and I wanted to do originally, because mongod and jujud could independently die02:03
axwcould be deferred02:03
fwereadeaxw, that's true, but I worry it's an overcomplication in this specific context02:04
axwlet's strike that out for now, it's strictly an improvement over what we've got now anyway02:04
fwereadeaxw, fwiw, if we were doing it, I feel we could probably keep it local to the individual state servers -- if they detect problems with their local mongo, they can mark themselves messed-up02:05
axwfwereade: I suppose they could just sever their connection to the other state servers02:05
cmarstasdomas, thumper: http://paste.ubuntu.com/7839596/02:09
axwfwereade: I feel like I must've missed something, because the existing implementation is quite complicated. were there previously other use cases?02:17
axwwallyworld: phew. lgtm02:34
wallyworldaxw: yeah, sorry :-)02:34
wallyworldaxw: i just found another collection in the watcher that was missed02:34
wallyworldso i'm retesting02:34
wallyworldaxw: also, we can change oplog max size to 5GB02:35
axwokey dokey02:35
wallyworldbut need to test a real deployment with a deploy -n X02:35
axwsure, I'll do some testing02:35
wallyworldi'm trying to use aws but keep running out of instances02:35
axwyou're on it?02:35
wallyworldyeah, but feel free to test on another platform02:36
wallyworldbut i think we'll be ok02:36
axwI'll try with Azure, but I can't see it being a problem02:36
wallyworldme either02:36
axwwallyworld: did you just change maxOplogSizeMB in mongo/prealloc.go?02:38
wallyworldaxw: actually, i'll land the io timeout one and then will need to backport to 1.20 (will be messy), so if you could do the branch to reduce the max oplog size that would be great02:38
wallyworldthere's been a lot of change in trunk in the same areas as touched by the collection stuff02:39
axwfun :(02:40
axwwallyworld: has anyone managed to look at a production juju installation's oplog?02:41
wallyworldaxw: i think kapil did in the bug report02:41
wallyworldand nate was going to ask in #mongo for advise02:42
wallyworldbut i need to ping him cause i haven't hard back02:42
wallyworldapil wanted the size reduced also02:42
axwhmm it shows the cap size, but not usage02:43
wallyworldyeah, i'm not sure we have that data02:44
axwerr, azure simplestreams is busted02:46
axwwallyworld: do I poke IS or what?02:47
axwthere's only trusty images for China02:47
wallyworldoh :-(02:47
axwin the index02:47
wallyworldsigh, yeah poke #is02:48
wallyworldthey were meant to put tests in place to catc this02:48
wallyworldi gave them the tools02:48
wallyworldthumper: yo02:49
wallyworldor menn002:49
menn0wallyworld: yep?02:50
wallyworldfresh deploy of a 1.21alpha1 environment, dpeloyed gui, all looks ok, but02:50
wallyworldmachine-0: 2014-07-23 02:49:28 ERROR juju.worker runner.go:218 exited "upgrade-steps": unexpected quit02:50
wallyworldmachine-0: 2014-07-23 02:49:28 INFO juju.worker runner.go:252 restarting "upgrade-steps" in 3s02:50
wallyworldlog is full of the above02:50
wallyworldi can ssh in and poke around. have you seen that?02:51
wallyworldthis is on aws02:51
menn0can I get the full log?02:51
menn0also, where can I get the code for that release? I'd like to see what made it in.02:51
wallyworldmenn0: this is trunk02:52
wallyworldi'll pastebin the log02:52
menn0built when? this code has changed quite a bit over the past 3 days (one change is testing for merge right now)02:53
axwwallyworld: hloeung says the server team manages the index, so I guess I'll just email Ben Howard?02:53
wallyworldaxw: yep, and cc scott moser02:53
menn0wallyworld: I will try to repro02:54
wallyworldmenn0: i built it just before. i have my own changes in there concerning copying sessions when talking to mongo, so it's possible that make be involved, but everything else works ok02:54
wallyworldby in there, i mean in state02:54
wallyworldno the upgrade code itself02:55
wallyworldthe changes are below the waterline02:55
menn0wallyworld: ok. let me have a dig.02:56
axwbest merge directive evar02:58
davecheneywaigani: http://www.shag.com/02:59
davecheneysomething tells me that he wouldn't have his art reprodued on a 3 buck sticker sold on ebay02:59
wallyworldaxw: say wot03:02
axwwallyworld: https://github.com/juju/juju/pull/364#issuecomment-4982704103:02
wallyworldtrust menn0 to be a smartarse03:03
wallyworldwell played, sir03:03
axwwallyworld: how many instances did you get up to on ec2?03:04
wallyworldaxw: about 10 or so03:04
wallyworldand then the account ran out03:04
wallyworldvery slow03:05
axwah, there may be trusty images in the daily stream for azure... will try that03:05
davecheneywho wants to be mean to me ? https://github.com/juju/juju/pull/36503:20
davecheneygentle ping, https://github.com/juju/juju/pull/36203:22
axwwallyworld: actually, bootstrapping azure won't tell us much. it's root disks aren't that big (25G), so the oplog isn't large anyway03:22
axwwallyworld: did you create a large root-disk on ec2?03:22
wallyworldaxw: no. but the idea was to see if a smaller oplog could handle a stress test with large numbers of units etc deployed03:23
axwwallyworld: ok, I see03:23
wallyworldif the smaller one could handle it, then 5GB will be plenty03:23
axwfair enough03:23
davecheneyi think maas is unique in the fact it gives you a large /03:24
davecheney'cos you get 100% of the underlying machine03:24
axwI thought you could request whatever you want with ec2?03:24
axwone of the cloud providers does that03:24
wallyworlddavecheney: even if te disk is large, i don't get why it's not partitioned to have a smaller /03:25
davecheneywallyworld: maas is web scale03:25
menn0wallyworld: fwereade and I are going to change Runner slightly so that a worker doesn't have to wait for the stop channel to close if it wants to exit without error03:25
menn0wallyworld: this is the source of that problem03:25
wallyworldok, sounds good03:26
menn0wallyworld: we thought that had already been done but it wasn't (we should have checked more closely...)03:26
wallyworldnp, easy enough to fix :-)03:26
menn0wallyworld: at any rate, it's only log spam and shouldn't have any real adverse affects otherwise. the upgrade-steps worker will be getting restarted but then exiting immediately over and over.03:27
wallyworldyeah, that's what i saw03:27
wallyworldaxw: yay, just merged trunk version of the copy session stuff back to 1.20, soooo many conflicts \o/03:29
axwfun times ahead03:30
wallyworldyeah, i so want to get this comitted today so we can build a 1.20.2 rc03:30
thumperwallyworld: oh hai03:54
thumperwallyworld: I have a question for you too03:54
wallyworldwasn't me03:54
thumperwallyworld: I'm looking at the disk configstore, and wanting to remove the "create an empty file" bit03:55
thumperwallyworld: fwereade said to ping you because you have have experience with this03:55
wallyworldi do?03:55
thumperand he vaguely remembered talking to you about it03:55
thumperbut if you didn't remember, then to ignore it03:55
wallyworldlet me look at the code03:55
axwwallyworld: I just found a bug in azure, load balancing is broken03:57
axwpossibly just on trunk, will need to verify with 1.2003:57
wallyworldthumper: yeah, i must have been on crack cause i don't recall the conversation03:57
thumperwallyworld: ok, cool03:57
wallyworldaxw: well at least you found it03:57
davecheney^ anyone? anyone? Beuller ?04:15
axwwallyworld: 3 state servers, 10 ubuntu units deployed = 2.34MB oplog over 1 hour04:16
davecheneyaxw: nice04:16
axwmostly quiescent tho04:16
wallyworldaxw: will 5GB should be *plenty*04:16
axwwallyworld: I think that change I made to API host ports doesn't actually stop the oplog from getting spammed04:19
axwseems the assertion still gets inserted into the transaction log04:20
wallyworldi wondered about that04:20
axwso it's either racy or spammy04:20
wallyworldwhether assert noops got logged04:20
wallyworldgiven the frequency of change, it's ok to be "spammy"04:21
wallyworldsince it's not really that spammy anyway04:21
wallyworldluckily mongo is web scale :-)04:21
wallyworldaxw: did you forward port the api host ports race fix to master?04:24
axwwallyworld: yep04:25
wallyworldok, that bit of code is coming up as a conflict doing my backport04:25
wallyworldand it appeared to be showing the old code in master, but like my mistake04:25
davecheneythumper: while you're slcking off,  https://github.com/juju/juju/pull/362,  https://github.com/juju/juju/pull/36504:26
axwwallyworld: where did the 5GB number come from?04:28
wallyworldaxw: of of thin air. greater than 1GB but an order of magnitude less than 50GB04:29
wallyworldwe use max 1GB for local provider04:29
axwwe use 1MB for local04:29
wallyworldah, ooops04:29
wallyworldi thought it was G04:29
wallyworldso maybe 1GB is sufficient04:29
axwI think so04:29
wallyworldnow that there's some numbers04:30
waiganiaxw: hello :)04:30
axwwaigani: ahoy04:30
menn0wallyworld: btw, fix for that upgrade-steps problem on the way: https://github.com/juju/juju/pull/36604:30
waiganiaxw: I've got a question - let's see how good your memory is04:30
wallyworldmenn0: awesome, tnaks04:31
jamaxw: if you wanted a worst case, you could use the xplod charm04:31
waiganiaxw: https://codereview.appspot.com/70190050/diff2/20001:120001/state/state.go04:31
jaminstead of ubuntu04:31
waiganiaxw: I'm trying to remember while I added additionalValidation04:31
jamaxw: xplod04:32
jamaxw: xplod04:32
axw? :)04:32
jamcopy & paste fale04:32
axwI'll give that a shot04:32
jamaxw: when you add units, they start a peer chatter amongst them04:32
jameach one tries to increment a number and report it back to the rest of the peers.04:32
jamaxw: also, as it is a simple charm, you can "juju deploy --to 1" to get more of them04:33
jam(well add-unit --to 1)04:33
jamaxw: though the fslock means they won't really scale super huge on one machine04:33
axwjam: cool, thanks04:33
axwwaigani: umm04:34
waiganiaxw: yeah hehe04:34
* axw greps for uses of UpdateEnvironConfig04:34
waiganiaxw: none04:34
jamdavecheney: why no uppercase letters (A-F) ?04:34
waiganiaxw: fwereade has suggested removing it04:35
waiganiaxw: but just wanted to double check that there wasn't a good reason to keep it04:35
axwwaigani: not none04:35
fwereadewaigani, there's one test that uses it to fuck things up creatively04:35
fwereadewaigani, axw: and, yeah, we do need that functionality04:35
axwthere's a bit of code in there that checks agent-version isn't set04:35
fwereadewaigani, axw: the fact that the test *can* fuck it up is evidence the method itself is a bit broken04:36
davecheneyjam: 'cos that is what we say is a uuid04:36
fwereadewaigani, axw: but it's gradually been becoming less os over time which is nice04:36
davecheneyjam: is that my old xplod charm ?04:36
jamdavecheney: I adapted yours and put a peer relation in it to make it easier to use04:49
jamyou still get N^2 by just adding units04:50
axwjam: stupid question, is there a charm URL to deploy that directly?04:50
axwor do I need to fetch it and local:04:50
jamaxw: not that I know of. branch it locally and then --repository local:04:50
axwokey dokey04:50
davecheneyjam: be careful04:52
davecheneythat charm for my ec2 credentials locked out for abuse :)04:52
jamdavecheney: we fixed that bug :), and I've used it on EC2 and scaled up to about 200*10 units04:53
jam(more than 100 doesn't help because of CPU bound on the individual machines)04:53
axwwhy did it get you locked out?04:53
jamaxw: we used to have a bug where every hook04:53
jamwould call a Provider04:53
jamwould make a Provider call (for the API server IP addresses)04:54
jamso hundreds of calls per second?04:54
axwI see :)04:54
davecheneymade it hard to kill that environment ...04:54
jamdavecheney: ai04:54
jamaiui, you ran into rate limiting first, which meant you couldn't kill it04:54
jamwhich let it run away until they shut it down for you :)04:54
davecheneycouldn't even use the aws console04:55
davecheneyas your quote counts towards that api as well04:55
davecheney(probably calls the same endpoint under the hood)04:55
jamdavecheney: I wonder if you used IAM credentials you could get just the one IAM account locked out, and still access it as your super users.05:00
axwjam, wallyworld: thoughts on https://bugs.launchpad.net/juju-core/+bug/1344940/comments/17 ?05:37
_mup_Bug #1344940: Juju state server database is overly large <canonical-is> <cloud-installer> <landscape> <mongodb> <juju-core:Triaged by axwalk> <juju-core 1.20:Triaged by axwalk> <https://launchpad.net/bugs/1344940>05:37
wallyworldaxw: i think 1GB to be safe05:38
wallyworldthat's imo05:38
axwwallyworld: I'll decrease min to 512MB, and max to 1024MB05:41
wallyworldok, sounds good05:41
axwand if mark doesn't like it, we'll take it from there05:41
jamaxw: sounds good to me, there is a lot of "how out of date can a replica get and come back without a full sync". If we want to support 1 day? 5 hours? all that stuff is pretty arbitrary05:44
axwI think 24 hours would be reasonable, but... I'm not an ops guy05:44
jamaxw: again, it depends a lot on how big the actual DB becomes, to determine how long a full sync is going to cost05:45
jamagain, our estimates are that it isn't that bad05:45
wallyworldaxw: do you know why "git diff X...Y" works but "git merge X...Y" doesn't?05:45
jamaxw: it would matter more if you had a 1TB database, and churn of less that 1GB/day, doing a full sync would be painful05:45
axwwallyworld: I don't05:45
jamwallyworld: git cherrypick, IIRC05:46
jamwallyworld: git cherry-pick -h05:46
wallyworldjam: i tried that but it brings in other stuff that doesn't show up in the diff05:46
jamwallyworld: so you can always do "git dif X..Y | patch -p1"05:47
wallyworldtries that but it says the patch can't apply05:47
wallyworldalthough i didn't do the -p105:47
jamwallyworld: p1 matters, because they put "a/" and "b/" prefixes05:47
jamyou need to be in the root as well05:48
jamit may be that conflicts/moved code/etc means a patch won't do a good job05:48
jam*I* would try cherry-pick as that is the intended git method (AFAICT)05:48
wallyworldjam: yeah, but for some reason, although the diff shows as correct, cherry-pick was bringin in revs i didn't see in the diff, and aborts partr way through05:49
wallyworldi just want it to apply whats in the diff and let me resolve merge conflicts after05:49
axwwallyworld: I've just been listing the commits explicitly in "git cherry-pick" and that worked fine, FWIW05:49
axwI don't tend to have a lot of commits in a PR though05:50
wallyworldi just want to take the difference between master and a feature branch and backport05:50
axwoplog moar smaller05:50
wallyworldnfi why this is so easy in bzr and so hard in git05:50
jamwallyworld: http://stackoverflow.com/questions/449541/how-do-you-merge-selective-files-with-git-merge seems to recommend rebasing until you get clean commits, then cherrpicking them...05:53
wallyworldjam: thanks will look. i've not had much luck with rebasing in the past., will try again05:54
jamwallyworld: the other approach is specifically rebasing the desired revisions into the target05:54
wallyworldso many revisions i'd rather just apply trhe whole diff and resolve conflicts05:54
jamand then remove the revisions you don't want merged05:54
wallyworldi can't see why diff works and I can't just use "merge" in place of "diff" in the command line05:55
jamwallyworld: patch can't handle merge conflicts, because it only has 2 inputs (you need 3 with a diff3 approach). I can think of how you could do it manually (checkout the common ancestor, checkout both merge tips, etc.)05:55
jamwallyworld: I don't understand git merge internals05:55
jamit may refuse to do merges that don't include the whole DAG which is what you are trying to do.05:55
jamthey have cherrypick for that, but it only seems to support single revs05:55
wallyworldtl;dr; it sucks05:55
axwwallyworld: gotta go pick up my daughter, but I just confirmed that the azure bug is present in 1.20.1 too06:15
axwI've added it to the milestone06:15
wallyworldaxw: np, thanks, i have to head out for a bit as well06:15
jamcmars: are you still around?07:08
jamI'd like to discuss your plans with https://github.com/juju/juju/pull/36707:08
Egoist_How to set in environments.yaml for openstack to use only one security group?07:54
=== uru_ is now known as urulama
axwjam: I'm guessing wallyworld's branch has broken the tests, but not really sure08:28
axwit touched a lot of statey things08:28
axwmy branch is failing the tests too08:28
niedbalskiaxw, thanks for the reply on the thread.08:30
axwniedbalski: hey nps. not sure if that helped at all08:30
axwhopefully gives some context tho08:31
niedbalskiaxw, yep, i needed some context08:31
rogpeppe1axw: hiya08:40
axwrogpeppe1: howdy08:40
rogpeppe1axw: how's tricks?08:40
axwnot too shabby08:40
axwand with you?08:40
rogpeppe1axw: pretty good. currently in london for the gui sprint08:41
rogpeppe1axw: doing charm store stuff08:41
rogpeppe1axw: i was just wondering if you had any opinions about charm.Reference08:41
axwrogpeppe1: I don't know what it is, so nope ;)08:41
rogpeppe1axw: ah, ok. i think it has to die :-)08:42
rogpeppe1axw: but i'd like to speak to someone who might've been originally involved in creating it08:42
axwrogpeppe1: looks vaguely related to the changes made not too long ago to support cs:<charm> without specifying series08:43
axwand having the charm store tell us which one to use08:43
rogpeppe1axw: yeah08:43
rogpeppe1axw: the problem is that it makes it impossible to have a bunch of charm urls, some of which specify the series and some of which do not08:44
axwrogpeppe1: looks like cmars is the man to talk to08:45
rogpeppe1axw: unfortunately he's also sprinting... in NZ08:45
axwsorry, out of my domain08:45
axwah :(08:45
rogpeppe1axw: np08:45
bodie_https://github.com/juju/juju/pull/370 fwiw --09:21
bodie_something's wacky and the sun is coming up, so I'm putting this one down for now, but if anyone knows about how the cmd.out.Write method handles maps and --format=json, that insight would be valued09:22
=== Guest30538 is now known as wallyworld
TheMuejam: I’m in da house10:46
jamTheMue: for some reason pidgin isn't beeping at me10:52
axwwallyworld: there's a "Revert" button on merged PRs if you didn't realse11:26
axwno need to do a reverse PR or anything11:26
wallyworldaxw: ah, rightio, thanks, didn't see that11:32
mgzbut reverse prs are fun...11:35
mgz(also, I'm not actually sure how much I trust github's merge algo stuf...)11:36
wallyworld_mgz: katco: axw: sorry, network problems, be there soon12:01
mgzwallyworld_: no probs12:02
perrito666good morning12:26
mgzhey perrito66612:26
natefinchI'm going to be out for a bit, have to take my 1 year old to the doctor.  Poor thing has had a 102.5° (39°C) fever for 36 hours.12:32
perrito666ouch natefinch best of lucks with that12:33
* natefinch need a bot to just automatically convert F/miles to C/km for the rest of the non-US (aka non-backwards) world12:34
katconatefinch: gosh hope he starts feeling better nate12:38
perrito666I am deleting and re-creating a folder in a given function, how can I test that the new folder is not the same, I tried FileInfo's ModTime, but the deletion/creation happens too fast for ModTime to change and I really do not think this justifies adding a sleep of 1 sec in the middle, ideas are welcome13:35
katcoperrito666: i really hate to rely on sleeps; they so non-deterministic. i like mgz's idea13:39
perrito666katco: exactly I am trying to find an alternative13:39
mgzyou'll have to skip on windows but that's not too bad13:40
perrito666mgz: not an issue this is restore13:40
perrito666wow os.FileInfo could really have more information13:40
hackedbellininatefinch: hi! Is there an eta for when I will be able to update to that new version that will correct the issue I'm having?13:56
katcohackedbellini: i believe natefinch is out at the moment, taking care of a sick child.13:57
hackedbellinikatco: ahhh, ok. np, will talk to him later so :)13:58
=== ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Bugs: 4 Critical, 152 High without fixes - https://bugs.launchpad.net/juju-core/
perrito666natefinch: how was it?14:30
natefinchperrito666: possibly an ear infection.  Not conclusive, but possible, so she's on antibiotics.14:31
ericsnownatefinch: yuck14:31
ericsnownatefinch: hope she feels better soon!14:32
* TheMue always felt bad when the kids have been ill. today the kids feel bad when I’m ill. :D14:33
* perrito666 has no kids but knows a lot about ear pain14:34
TheMueI thankfully seldom had, but the kids from time to time.14:36
TheMue…oooOOO( Or at least I cannot remember if I had it as kid too sometimes. )14:37
katcohave we ever discussed using the irc bot to do CI notifications in the room? maybe just failures or other important things?14:44
* perrito666 upgrades ISP to double speed for U$D3/month14:45
perrito666I love promotional loopholes14:45
natefinchkatco: no, but it's a good idefa14:45
katcowhat bot are we using?14:46
natefinchperrito666: I called my ISP about upgrading, and they said "Oh, you have 50/25 right?"  And I was like, no I have 70/35.  And they said "you can't have that, we don't offer that".  Well, actually you do/did due to a promotion.  But thanks for being a jerk.14:47
natefinchkatco: no idea  bug #134771514:48
_mup_Bug #1347715: Manual provider does not respond after bootstrap <bootstrap> <ci> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1347715>14:48
* natefinch wasn't sure if the bot said what it was... 14:48
natefinchI've been using IRC for approximately 360 days, so I'm not really the one to ask :)14:49
perrito666natefinch: what was the question?14:49
katconatefinch: it did. were you just posting that bug as an example, or are you wanting me to look at that particular one?14:49
* perrito666 got upgraded but also disconnected14:49
natefinchkatco: just an example. First one that I found14:50
* natefinch wasn't sure if mup was the bot's name or just the name we gave it to post as14:51
katconatefinch: probably a name someone gave14:51
katcoi'll post something to juju-dev14:51
natefinchmgz probably knows14:51
mgzmu p~muo mu?14:52
perrito666ah, iirc, jenkins has a bot builtin or can have one14:52
perrito666I had that working for a project once14:52
katcoperrito666: yeah it does14:52
katcoperrito666: it's a plugin i think14:52
perrito666very annoying thing14:52
perrito666a lot of finger pointing14:52
* katco currently waiting for jenkins to fail so she can resubmit14:53
* katco would rather not keep flipping to the jenkins console14:53
hackedbellininatefinch: hi! You weren't here before, so I'll resend the question I've made earlier =P14:53
hackedbellininatefinch: hi! Is there an eta for when I will be able to update to that new version that will correct the issue I'm having?14:53
natefinchhackedbellini: I'll get the fix in today, and then I think we're making a cut at end of week anyway, so.... early next week?14:55
mgzmup is supybot, but there's a launchpad project somewhere with stuff in14:55
katcomgz: while you're here... is there any way i can kill a jenkins job i know will fail?14:56
mgzkatco: only with admin access... so you can poke me14:56
katcomgz: #10314:57
mgzthe current one?14:57
katcothe command i used to run my tests locally skipped over one that's failing14:57
katcoty sir14:57
mgzyou need a little magic to requeue it14:57
katcomgz: oh, i just did $$merge$$14:58
mgzeither craft your won github comment with "merge failed: " in it, or get me to requeue via jenkins14:58
katcothe second $$merge$$ seemed to work... huh14:58
perrito666natefinch: standing thinguie15:01
mgzkatco: oh, you got a tests failed message15:01
katcomgz: is that surprising?15:01
hackedbellininatefinch: ok, no problem! I'm just anxious for that fix because without it my juju is dead =P15:02
hackedbellinilet me know if you need me to test anything.15:02
natefinchhackedbellini: understandable15:02
natefinchperrito666: ok15:02
mgzkatco: if I got the abort in, that wouldn't happen - might have finished anyway or something15:03
katcoahhh ok15:03
katcoi follow now; that's why i would have had to craft the failure manually15:03
TheMueaaaaah *jump *jump* *jump* lxc-ls now shows an ipv6 address for a container15:05
TheMuesadly had to set it internally by hand, it is not set during deployment15:05
perrito666ericsnow: what is the whole cleanup thing https://github.com/juju/juju/pull/334/files#diff-076396fa7fd3f93945528111df2d8319R48 ?15:24
ericsnowperrito666: it's so that we don't leave any empty file behind in case of an error15:25
perrito666I meant the cleanup variable15:25
ericsnowperrito666: that could definitely use a comment15:26
perrito666it seems to me that you only delete when there is an error?15:26
ericsnowperrito666: that's right15:26
perrito666ericsnow: you can name the error return and check it15:26
perrito666since deferred call is after the function exit15:27
ericsnowperrito666: ah, good point15:27
ericsnowperrito666: note that in the review and I'll take care of it15:28
perrito666will do, just wanted to make sure I understood your intentions correctly15:29
ericsnowperrito666: you do15:29
ericsnowperrito666: and thanks :)15:29
perrito666does anyone know what is the possible error output of filepath.ABS ?15:32
perrito666oh I see15:33
natefinchanyone familiar with the upgrade logic?15:34
natefinchmgz ^^ ?15:36
mgzhmm. not very15:37
perrito666sinzui: Ill take https://bugs.launchpad.net/juju-core/+bug/134293715:37
_mup_Bug #1342937: Juju restore  fails Could not get lock /var/lib/dpkg/lock <backup-restore> <ci> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1342937>15:37
sinzuithank you perrito66615:37
natefinchsinzui: I'm working on #134272515:37
_mup_Bug #1342725: C:/Juju/lib/juju/nonce.txt does not exist, bootstrap failed in win <ci> <regression> <windows> <juju-core:Triaged> <https://launchpad.net/bugs/1342725>15:37
natefinchI had most of a fix, but it conflicts with the windows changes gsamfira is doing, so I'm going to grab the fix from his code.15:38
jcw4perrito666: https://github.com/juju/juju/pull/351 is updated... if you get a chance I'd appreciate a look15:50
perrito666jcw4: will do15:59
jcw4tx perrito66616:00
perrito666jcw4: interesting test https://github.com/juju/juju/pull/351/files#diff-7195d3d7d4a41d504d4c75799ca3e540R34216:01
jcw4perrito666: I knew I couldn't get anything past your eagle eye16:04
jcw4I remembered I hadn't done that test after merging in master and didn't want to make more changes... I suppose I'll have to now16:04
* natefinch tries desperately to get his logs above 1 meg16:04
natefinchyay, log rotation16:06
jcw4natefinch: yay16:07
* jcw4 assumes thats a good thing16:07
natefinchhaha yes16:07
perrito666natefinch: if you have anything using mysql turn on debug mode16:07
perrito666that should do16:07
natefinchmy log rotation package lets you specify a max size, but it's in megabytes, which is a surprisingly large amount of plaintext16:08
natefinch-rw-r--r-- 1 root   root   1.0M Jul 23 12:06 machine-0-2014-07-23T16-06-26.278.log16:09
natefinch-rw-r--r-- 1 root   root   288K Jul 23 12:07 machine-0.log16:09
=== jcw4 is now known as jcw4|on-the-road
wwitzel3natefinch: I thought we were just using logrotate?16:16
natefinchwwitzel3: logrotate won't work on windows16:17
wwitzel3hah, dumb, didn't think of that16:18
natefinchwwitzel3: and we weren't using anything.  I had investigated logrotate, and it probably would have been ok, but honestly probably more work than using a pure-go solution (which also happens to be portable to other OSes)16:18
wwitzel3natefinch: makes sense16:25
natefinchreview for whoever https://github.com/juju/juju/pull/37516:53
=== Ursinha is now known as Ursinha-afk
katcobackport for v1.20: https://github.com/juju/juju/pull/37617:08
katconatefinch: would we ever consider qualifying our variable names to include size units? e.g.: MaxSizeInMB?17:15
natefinchkatco: I wouldn't.... I might consider making a custom type like type Megabyte int   so that you can't accidentally convert from one to the other.17:18
katcoi still like axw's idea to duplicate the time package's implementation for size units17:19
katcobut i wouldn't mind more discussion on the naming; what do you dislike about that technique?17:19
natefinchkatco: I thought of that, and actually did that for v1 of lumberjack (had Megabyte = 1024*1024  and Gigabyte as 1024*Megabyte)17:19
natefinchkatco: the problem that came up was deserializing from a config file.... the config file can't use those constants, and can't just use 1024*1024, so it ended up needing to be like MaxSize: 10000000017:20
katcoi wonder how serialization works with the time package17:21
natefinchkatco: plus it just wasn't necessary to let people specify log size down to the byte.  no one's going to want a log that rolls over at 54 bytes or something17:21
katconatefinch: no, agreed. it's all about clarity.17:21
natefinchkatco: there are custom serialization formats17:21
katcoi.e.: if i see MaxSize, i need to go look at the declaration to know what the unit is17:22
katcowhich is kind of annoying17:22
natefinchkatco: yeah, I know17:22
katcohence, MaxSizeInMB17:22
natefinchit seems wrong.... like hungarian notation... but I can't explain why17:23
katcohungarian notation was wrong b/c it caused tons of churn all over the codebase if you ever changed the type. i suppose you could have the same issue here17:24
katcoi guess the difference is that hungarian notation was trying to represent the syntax in the name, and this would be representing the value in the name; something appropriate for a variable name, and probably OK to cause refactoring since you would be evaluating all the places it's used anyhow17:25
natefinchIt's also the kind of thing you only ever use once in your application and then never touch again17:26
katcowhat the variable?17:27
natefinchin this specific case17:29
natefinchyou set the logging output in one spot one time and you're done.17:29
perrito666katco: you dont need to go to the  declaration, just MaxSize = "n" and try to build and the error will tell you :p also I am surprised emacs does not tell you that17:29
katcoperrito666: rofl17:30
katcoperrito666: emacs had already simulated, and predicted the outcome of this conversation, and decided it was not worth its time.17:30
katcoperrito666: also, any stupid things i say are emacs trying to undermine my credibility. (nods)17:31
katconatefinch: yeah it's definitely not something i would dig my heels in on, but interesting conversation.17:31
natefinchkatco: I'm not sure you're wrong.  I have a vague leaning against it, but like I said, I can't really explain why.17:32
katconatefinch: sometimes those gut checks are correct.17:32
perrito666well it reminds me of old php code where they did typing trough variable naming17:33
perrito666people that is17:33
katcoperrito666: see aforementioned comment about name trying to express syntax vs. type of value.17:33
katcoperrito666: that strikes me as hungarian notation, slightly different from this.17:33
perrito666we will end up with a size struct and MB() GB() B() methods :p17:34
natefinchI rather like go's named types, like type Megabytes int, however it doesn't help in this case because you can still just do MaxSize = 100, since constants are untyped and mold to fit the type they're assigned to.17:34
perrito666anyway, I was about to say, for me file size magnitudes should always be in bytes17:35
perrito666which is default for most unix tools17:36
=== Ursinha-afk is now known as Ursinha
natefinchperrito666: I know, but see above about deserializing from a config file.  I did that in v1, and ended up with a config file that had maxsize = 10000000017:47
natefinchand that's just ugly17:47
natefinchI don't ever want to have to count zeroes17:47
perrito666natefinch: most unix config files hold that kind of values17:48
perrito666or modern ones accept the unit as part of the value17:48
natefinchyeah, the unit deserialization is a possibility17:48
natefinch200MB  or 1.5GB etc17:48
natefinchmegabytes is fine17:49
perrito666most unix commands evolved from times where bytes where something meaningful so they added units when 1T was something that you could have on your laptop I guess17:49
katco200 * size.MB17:49
natefinchkatco: that's fine for code, but doesn't work in deserialization, which is a big use case in logging configuration17:50
katcooh you're talking about the use-case for fiddling with these settings in the config file where such niceties aren't present17:50
katcoi missed that, sorry17:50
natefinchright, np17:51
natefinchmakes me want to write a size package so I can have those niceties17:51
natefinchplus, "package size" is um.... amusing17:51
katcooh lord lol17:52
natefinchwhelp, I know what my next project is :)17:53
perrito666ohh man, I arrived to late for the size and package jokes17:54
katcobut you have to give it a name that's a double entendre17:54
natefinchthe key is figuring out how much double entendre you can fit in and still make it unclear if it's on purpose or not17:54
katco+1 nate. +1.17:54
perrito666natefinch: the thing is to actually name it size, make it very useful pakcage, and then get proper english speakers to compliment you on it17:55
katco"One of the driving principles of the X project is that size shouldn't matter. It's how you use it. With this, we thrust our package into the go community and await feedback."17:56
katcooh look, an emacs plugin to watch jenkins status. :)18:18
perrito666ericsnow: ping18:22
ericsnowperrito666: hey18:23
perrito666ericsnow: hey, I cannot find what did you patch to make this work https://github.com/juju/juju/pull/334/files#diff-baa2cc9d463ab23cb9521ade2d84a5e9R9418:23
perrito666a little help?18:23
ericsnowperrito666: it's in setData()18:24
perrito666ericsnow: I guessed so much, I was not sure how that goes all the way down to backup18:25
ericsnowperrito666: the actual patching happens in SetUpTest()18:26
perrito666ohh I see18:26
perrito666I thought yo where cheating the whole thing so it would return the same hash18:26
ericsnowperrito666: tempting but no :)18:26
ericsnowperrito666: by cheating I would have been "done" a lot earlier :)18:27
perrito666ericsnow: I am done R up to api_test.go later Ill continue with the rest18:32
perrito666but you have a handful of my comments,18:32
ericsnowperrito666: awesome18:32
perrito666:) I am sure more savvy people can give you even better comments18:32
ericsnowperrito666: thanks so much18:32
perrito666actually today you could force our own natefinch or cmars to review your code :p or thats what the ocr schedule says18:35
perrito666bbl, bike time18:35
natefinchdamn, am I OCR today?  Where's that OCR list?18:40
perrito666natefinch: sent you the link in priv18:41
perrito666natefinch: and since you are ocr, that link I just posted is a large part of restore18:45
perrito666ericsnow: do you mind if I take over https://github.com/juju/juju/pull/113 ? or are you going to work on it? I am really looking forward to have a functional restore (after I fix this week's chapter bug of old restore)18:47
ericsnowperrito666: go ahead :)18:47
natefinchquick, everyone hide18:55
arosalesfyi, we have a juju core panic on Power reported in bug https://bugs.launchpad.net/ubuntu/+source/juju-core/+bug/134732218:55
_mup_Bug #1347322: juju ssh results in a panic: runtime error <ppc64el> <juju-core:Triaged> <juju-core (Ubuntu):Confirmed> <https://launchpad.net/bugs/1347322>18:55
arosalesnatefinch: lol :-)18:56
arosalesprevious bug was https://bugs.launchpad.net/ubuntu/+source/gccgo-4.9/+bug/130475418:56
_mup_Bug #1304754: gccgo has issues when page size is not 4kB <ppc64el> <trusty> <gcc:Fix Released> <gcc-4.9 (Ubuntu):Fix Released> <gccgo-4.9 (Ubuntu):Invalid> <gcc-4.9 (Ubuntu Trusty):Invalid> <gccgo-4.9 (Ubuntu Trusty):In Progress by doko> <gcc-4.9 (Ubuntu Utopic):Fix Released> <gccgo-4.9 (Ubuntu Utopic):Invalid> <https://launchpad.net/bugs/1304754>18:56
arosalesmbruzek: can reproduce this bug pretty easily so please feel free to ping him if any further data is needed.18:56
arosalesIf anyone has any insights into that bug it would be much appreciated as it is blocking juju deployments on power.18:58
natefinchthe ssh one?18:58
arosalesnatefinch: correct, I think mbruzek also say it on regular deploys18:59
arosalesssh reliable reproduces it though18:59
mbruzeknatefinch, if you need access to power system I can hook up18:59
arosalesnatefinch: initially we thought it to be the same as the compiler bug 130475419:00
_mup_Bug #1304754: gccgo has issues when page size is not 4kB <ppc64el> <trusty> <gcc:Fix Released> <gcc-4.9 (Ubuntu):Fix Released> <gccgo-4.9 (Ubuntu):Invalid> <gcc-4.9 (Ubuntu Trusty):Invalid> <gccgo-4.9 (Ubuntu Trusty):In Progress by doko> <gcc-4.9 (Ubuntu Utopic):Fix Released> <gccgo-4.9 (Ubuntu Utopic):Invalid> <https://launchpad.net/bugs/1304754>19:00
natefinchmgz: the machine agent and unit agent - they're separate processes running at the same time on the same machine, right?19:18
natefinchwallyworld: are you really there?19:31
natefinchmgz: you can ignore my previous question when and if you see it :)19:42
mbruzeknatefinch, I saw your update to 134732219:55
mbruzekWhich log do you need more of?  The text in the bug body is from the console.19:55
mbruzekI included all of the dmesg output from that system.19:56
natefinchmbruzek: the juju machine log should19:56
natefinchmbruzek: have more info19:56
natefinchmbruzek: this looks like a plain old code problem, not a compiler problem, though it could be the latter that just happens to show up as the former19:59
mbruzeknatefinch, I don't see an error in machine-1.log, do you want to see all-machines.log?19:59
=== Ursinha is now known as Ursinha-afk
natefinchmbruzek: oh, I think I was misunderstanding what I saw.  That's a panic in the CLI code.20:01
natefinchI think20:02
natefinchit's so hard to read when it's all wrapped wackily like that20:02
mbruzeknatefinch, yeah it is.20:02
natefinchmbruzek: if you can repro easily, can you get me cleaner output of that text?  or is that the best you can get?20:03
mbruzeknatefinch, when I get the panic the screen is garbled like that always20:03
mbruzeknatefinch, I can work on cleaning it up for you20:03
mbruzeknatefinch, I also got the machine logs from the system if they would be helpful20:03
natefinchmbruzek: never hurts to attach more logs to a bug :)20:04
_mup_Bug #1347322: juju ssh results in a panic: runtime error <ppc64el> <juju-core:Triaged> <juju-core (Ubuntu):Confirmed> <https://launchpad.net/bugs/1347322>20:05
mbruzeknatefinch, updated20:05
natefinchmbruzek: thanks20:05
natefinchmbruzek: i can clean up the log as easily as you, it's no problem20:05
mbruzeknatefinch, the panic only happens *after* I juju ssh to the ubuntu unit.  I just use the terminal a bit and it goes sideways.  I don't believe it is related to what I am running on the terminal.20:08
mbruzeknatefinch, to expediate the bug I juju sshed to the same unit 3 times and got the same grabled text in all three windows.20:08
natefinchoh weird.  so you're connected for a bit before it actually blows up?20:08
mbruzeknatefinch, yes I am not sure what sets it off.  I was doing several different things20:08
natefinchmbruzek: does it happen if you ssh into the machine the old fashioned way?20:09
natefinchin theory, juju ssh just gets the ssh info from state and then runs ssh like a normal person would20:10
=== Ursinha-afk is now known as Ursinha
davecheneythumper: https://bugs.launchpad.net/juju-core/+bug/134793921:51
_mup_Bug #1347939: build is unstable since 7524c62 <juju-core:Confirmed> <https://launchpad.net/bugs/1347939>21:51
davecheneywallyworld__: sad to say your session copy fixed broke the build, https://bugs.launchpad.net/juju-core/+bug/134793921:52
_mup_Bug #1347939: build is unstable since 7524c62 <juju-core:Confirmed> <https://launchpad.net/bugs/1347939>21:52
wallyworld__davecheney: i reverted it last night21:52
davecheneyok thanks21:52
wallyworld__it passed for me locally and on the bot21:52
wallyworld__but clearly there's a race in our tests21:53
davecheneywallyworld__: it's not a race21:53
davecheneyit's livelock21:53
davecheneywhen I run the test my cpu usage eventually goes to 0 and the test will timeout21:53
wallyworld__i haven't fully looked into it yet, just making an assumption as to why it passes sometimes and not others21:53
* perrito666 reviewed a 30 file pr22:57
davecheneywallyworld_: right-o, thanks22:57
perrito666davecheney: your post about conditional compilation is very cool, thank you, I did not know about _$GOARCH.go22:59
davecheneyperrito666: yup, the pattern is extended to23:00
davecheneyand even23:00
davecheneymattyw: tasdomas http://blog.nuclearsecrecy.com/2014/05/23/oppenheimer-gita/23:02
davecheney^ that quote23:02
perrito666waigani: funny network?23:02
mattywdavecheney, tasdomas the reflections of feynman: http://www.youtube.com/watch?v=6no328q_VGQ23:03
fwereadewallyworld_, ping23:03
perrito666sorry I meant wallyworld_23:03
fwereadewallyworld_, I'm trying to figure out tools selection for container proviisioners23:04
fwereadewallyworld_, we have a bug that needs to be fixed -- that env provisioners start machines with the agentVersion in env config, not the current running version23:04
wallyworld_was there a specific question?23:04
fwereadewallyworld_, and it's looking like the container ones do that as well, but very indirectly23:05
fwereadewallyworld_, ie they grab the tools with the envconfig agent-version, with arch/series taken from the current machine23:05
wallyworld_fwereade: should the env config version not match the running version?23:06
fwereadewallyworld_, but ISTM that they're all going through the same path, ultimately, in which they hit simplestreams23:06
fwereadewallyworld_, not necessarily23:06
fwereadewallyworld_, after the agent version is set by upgrade23:06
fwereadewallyworld_, but before the upgrade actually happens23:06
fwereadewallyworld_, we could provision a machine with tools not matching those running in the provisioner23:06
fwereadewallyworld_, and *that* would come up (assuming it *did*) already running new tools23:07
fwereadewallyworld_, and would never run upgrade steps23:07
perrito666fwereade: hey, you are here, apparently the cloudsigma links I pointed in my mails are  the new prs, none has more than 6 files and jam\d? said that those are the younes you guys are going to review23:07
fwereadewallyworld_, leading to all manner of potential unhappiness23:07
wallyworld_i'll have to look up the moving parts in the container provisioners to remember how the tools selection works in there23:07
fwereadeperrito666, are they? ok I am out of date there, I last spoke to jam about a week ago, he is undoubtedly more current on that23:07
fwereadewallyworld_, in particular the brokers get initialised with one set of tools23:08
perrito666fwereade: yup, he answered to the thread I think23:08
fwereadewallyworld_, looked up over the api with arch/series initialised from the current machine agent23:08
fwereadewallyworld_, and then when we want to run a container with a different series we just hack up the tools struct so the wrong tools look like they have the right series23:09
fwereadewallyworld_, and as if by magic everything somehow currently works23:09
fwereadewallyworld_, more or less23:09
wallyworldfwereade: the container provisioner gets the tools to run via a call to the Tools client api method23:11
fwereadewallyworld_, yeah -- and that I think is the problem, because *that too* is using envconfig's agent-version23:11
fwereadewallyworld, not that actual version that's really running on the machine23:12
wallyworldi *think* the assumption was that the env version should match the running version23:12
wallyworldthe upgrade procedure is designed to make that happen23:12
wallyworldie set env version, trigger upgrade worker, doenload new tools, restart agents23:13
wallyworldso by design, env version should match runnning version (perhaps implementaton doesn't match that design)23:14
wallyworldi'm hand waving a bit because i didn't do the initial implementation of all this23:14
wallyworldwhat bug are you seeing?23:14
wallyworld_fwereade: sorry, dropped off again, my irc is so flakey for some reason23:18
wallyworld_not sure if you responded23:18
fwereadewallyworld_, sorry, i've decided that it is a rabbit warren of death and I'm not going to fix it today23:19
fwereadewallyworld_, I found that the upgrader is *also* interestingly fucked wrt what tools it picks23:19
wallyworld_ok. is there a bug number so i can see the issue?23:19
fwereadewallyworld_, there might be, but it's really just a race we realised was important for HA upgrades yesterday23:19
fwereadewallyworld_, I *think* there was a bug a while ago23:19
wallyworld_ok. perhaps the original design is sound for a single state server23:20
wallyworld_but not ha23:20
fwereadewallyworld_, i'm just trying to understand the situation and flailing around and grabbing onto you23:20
perrito666ok fine people, my brain just SIGQUITted on me, see you all tomorrow, cheers23:20
fwereadewallyworld_, it's still a bug, fwiw, but probably less impactful23:20
fwereadewallyworld_, we can't assume that the provisioner running on version X is capable of correctly setting up an instance running !X23:21
perrito666sinzui: I have a solution for the apt lock issue but Ill code it tomorrow cheeers23:21
fwereadewallyworld_, worst case non-HA we just fail with one machine23:21
fwereadewallyworld_, worst case with HA is a bit worse, we end up with a state server never participating in the upgrade synchronisation23:22
fwereadewallyworld_, anyway23:22
fwereadewallyworld_, it's not your problem and you don't need to worry about it23:22
fwereadewallyworld_, sorry noise23:22
fwereadeperrito666, apt lock?23:24
fwereadeperrito666, context please?23:25
fwereadeperrito666, that solution *should* be the hook execution lock23:25
fwereadeperrito666, if it's anything else we may have a problem23:25
fwereadeperrito666, ehh, you're EOD23:25
fwereadeperrito666, please find someone who knows about the hook lock tomorrow before writing anything else that tries to fix apt contention23:26
fwereadeperrito666, sleep well :)23:26
davecheneyFAIL    github.com/juju/juju/replicaset 173.495s23:32
davecheneyalways fails on my machine23:32
davecheneycan anyone confirm they see the same thing ?23:32
_mup_Bug #1347969: FAIL: replicaset_test.go:155: MongoSuite.TestAddRemoveSetIPv6 <juju-core:New> <https://launchpad.net/bugs/1347969>23:39
davecheneythumper: fwereade https://github.com/juju/juju/pull/37823:47

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!