[00:02] * arosales doesn't see wallyworld around . . . [06:41] hmm... CI is unhappy [06:41] It looks like AMZ is just out of instances [06:41] but "hp upgrade" is failing [06:41] and has been since r2644 [06:42] I wonder if axw tested upgrade with the bootstrap patch version change [06:46] I see this in the logs, which looks worrying: machine-2: 2014-04-17 12:56:26 INFO juju.worker.apiaddressupdater apiaddressupdater.go:58 API addresses updated to [] [06:47] also weird, all-machines.log only shows machine-2 getting the updated tools. Nothing about machine-0 even noticing that it wanted them. [06:47] I do wish we could run in --debug mode... [06:48] I wonder if we could log what API's are being called in Info mode, even if we don't log all of the details we would in Debug mode. [06:49] anyway, upgrade is borked... :( === vladk|offline is now known as vladk [07:16] jam1: good morning [07:58] vladk, morning [07:58] vladk, are you working today? [07:59] vladk, jam is usually off on fridays (swapping them with sundays) [07:59] mgz, you around today? [08:00] * dimitern will desperately need reviewers today :/ [08:00] dimitern: morning, I'm working, and you? [08:01] vladk, yes - there was some misunderstanding on my part - i thought i had public holiday on monday and decided to take it, but it turned out it's today.. meh no big deal [08:20] dimitern, could you take a look https://codereview.appspot.com/88380044/ [08:20] vladk, looking [08:26] vladk, reviewed [08:37] dimitern: thanks [08:37] why do you setupNetworks only if config.HasNetworks? [08:37] I think they should setup always, so they setup also on bootstrap and add-machine commands. [08:39] vladk, eventually we'll do that, but for now the requirement is to set them up only when specified explicitly when deploying [08:46] dimitern: I told about this with jam, his opinion: [08:46] I'm probably happier if we set up everything rather than only the ones the user supplied [08:46] as then if you want to deploy another service, in say a container, then we know that we do have that network [08:46] now, when we have the NetworkWorker that can do dynamic setup of networks [08:46] it matters less [08:46] because then we can just set up the minimum, and then add ones that we need later. [08:47] I thought we were starting all by default. [08:48] vladk, exactly, the worker will give us that [08:48] vladk, but remember we're doing MVP now, so we're taking some shortcuts [09:12] morning all [09:12] rogpeppe: ping [09:13] * rogpeppe is not really here [09:13] It's a UK bank holiday today and Monday [09:13] rogpeppe: that's what I was checking [09:13] voidspace: indeed it is [09:13] * voidspace would like to not really be here as well [09:14] rogpeppe: so I was just checking in [09:14] voidspace: i'm just sorting out insurance and packing before going away tomorrow... [09:14] rogpeppe: happy good friday [09:14] voidspace: you too [09:14] rogpeppe: have a good weekend [09:14] voidspace: you'll be happy to head HA has now landed... [09:14] s/head/hear/ [09:15] rogpeppe: I just saw some emails [09:15] rogpeppe: awesome [09:15] rogpeppe: ah, looks like you're going on a proper holiday [09:15] voidspace: have a go - see if you can make it work... [09:15] rogpeppe: enjoy [09:15] voidspace: i am! [09:15] rogpeppe: will do, I'll try and break it :-) [09:15] vladk: dimitern: I'm "off" today, but if you need something you can ping me. [09:15] voidspace: taking advantage of colorado mountain stuff [09:16] rogpeppe: ah, of course [09:16] gophercon [09:16] Upgrade is broken, so I might give it a poke, as we can't do any sort of release with that [09:16] rogpeppe: see you in vegas then [09:16] voidspace: up [09:16] yup [09:16] voidspace: aye [09:16] hi voidspace, welcome back [09:16] (well, welcome back to IRC at least :) [09:16] jam1: hi, and thanks [09:17] voidspace: are you back in the UK? [09:17] jam1: yep [09:17] jam1: for a week at least [09:17] voidspace: lucky you to get to fly trans atlantic every other week [09:17] jam1: I'm waiting to see how bad the jetlag is [09:17] it usually lasts me a week [09:17] voidspace: just don't change your TZ for this week [09:17] so I should recover just in time [09:17] wake up 6 hours late [09:17] jam1: hah, I did consider it [09:17] jam1: my daughter has other ideas [09:17] voidspace: I thought you liked to sleep in and start late anyway [09:18] hah, normally I do [09:19] Brett Cannon (Python core dev) will be looking for work soon, and has Go experience (by the way) [09:19] * voidspace subtly changing topic away from my sleeping habits [09:20] he's an excellent dev, hopefully we have a slot for him when he becomes available [09:20] voidspace: no such luck mr sleepy. I think I've actually met Brett at a pycon a few years ago. Is he the one who was doing importlib stuff? [09:21] jam1: yep, currently a googler - great guy [09:21] voidspace: if he's looking, you should get his name in to Alexis, I think our slots are filling up pretty quickly. [09:21] jam1: he's not looking just yet - but planning a move in the next few months [09:22] he has to wait a bit longer for his options to vest, so I don't think we can tempt him into an early leave [09:22] he'll get in touch with me though, so we'll see [09:26] voidspace: so "a few months" is certainly long enough for things to change. But at least atm the head count should all be filled by then (I think) [09:31] jam1, ah, alright then [09:32] dimitern: since you and fwereade are hanging out, can you poke him about Manifest-charm-deployer ? I'm pretty sure I LGTM'd it, and it would be good to have in the next release [09:32] jam1, I'm catching up on email at the moment, and I'll want to run a fresh live test against reality with the latest code, but I'll land that today [09:32] fwereade: sounds good. [09:33] fwereade: as for "user" all the other files were explicitly checked with ft.File("name") preserveUser is checking the same thing but doesn't *look* the same as the previous N checks. [09:33] so ignore me [09:34] but I missed it because it wasn't matching the pattern [09:34] jam1, yeah, I worried vaguely that it was less obvious, but thought I'd prefer to stick with the var than dupe the definition [09:34] jam1, maybe I should be putting them allin vars, but that felt inconvenient [09:35] fwereade: at this point, we've spent too long discussing it vs just landing it :) [09:38] jam1, quite so :) [09:38] fwereade: do you have a take on the "juju bootstrap" should always be exactly pinned discussion? [09:39] I feel like the discussion has gotten into bickering, and I'm trying to keep it productive. [09:39] I feel like we haven't really come to a consensus [09:39] so I'm want to actually change our behavior without having that. [09:40] But I don't want to come across as just being petulant or defensive. [09:41] I think abentley does have some points we should consider, but I also want us to come up with a strong consensus as I'd rather have consistency in this area, rather than doing it X for 2 releases and then changing our minds again. [09:47] jam1, yeah, just catching up and pondering [09:49] fwereade: anyway, I'd appreciate more input in the thread, as I feel like more comments from me isn't productive anymore. [09:49] rogpeppe: if you're still here: https://bugs.launchpad.net/juju-core/+bug/1309444 [09:49] <_mup_> Bug #1309444: peergrouper spins in local/upgraded environment

[09:49] local provider doesn't support --replicaset (yet?) so the peergrouper just bounces endlessly [09:50] jam1, do you remember who's been working on the precise/trusty lxc issues? [09:50] and I *think* upgraded environments will do the same (today) [09:50] fwereade: do you have an issue in particular? [09:50] jam1: oops, the peergrouper worker should be disabled for local environments [09:50] jam1: upgraded environments might be ok if axw's branch has landed [09:50] rogpeppe: is it sufficient for it to see "not in replicaset mode" and just exit gracefully? [09:51] jam1: it could check the replica set status and see that there are no members [09:51] jam1: that would be somewhat more graceful [09:51] rogpeppe: well this ends up in the log 2x: [09:51] 2014-04-18 09:45:41 ERROR juju.worker.peergrouper worker.go:137 peergrouper loop terminated: cannot get replica set status: cannot get replica set status: not running with --replSet [09:51] 2014-04-18 09:45:41 ERROR juju.worker runner.go:218 exited "peergrouper": cannot get replica set status: cannot get replica set status: not running with --repl [09:51] thats a lot of not-getting the replica set status :) [09:52] jam1: it can't just exit though - otherwise it'll be restarted (we should perhaps fix that so it's possible for a worker to exit without being restarted) [09:52] rogpeppe: I thought we had a way for workers to exit with "I'm finished now" [09:52] jam1: i don't think so, but we may do [09:52] jam1, see #juju-gui just now [09:52] jam1: i always thought that just exiting with a nil error should be enough [09:52] rogpeppe, jam1, they were meant to not be restarted if they return nil [09:53] rogpeppe, jam1, not sure what happened if that never landed, I thought we rediscussed that exact issue a few weeks ago [09:53] fwereade: yeah, we should do that [09:53] fwereade: (if we don't already) [09:53] * rogpeppe is really gone now [09:54] rogpeppe: fwereade: "if workerInfo.start == nil { // The worker has been deliberately stopped" [09:55] jam1: ah, that's cool then [09:55] excellent [10:03] hello [10:50] mgz, perrito666, dimitern, fwereade: staup? [10:50] natefinch, coming [10:50] standup that is === psivaa is now known as psivaa-afk === vladk is now known as vladk|offline [14:23] fwereade, mgz, vladk|offline, natefinch, i'd appreciate a review on this critical bug fix https://codereview.appspot.com/89260044 [15:31] sinzui, when you're about to release 1.19.1, please add this to the release notes https://bugs.launchpad.net/juju-core/+bug/1307513/comments/1 [15:31] <_mup_> Bug #1307513: Support multiple (physical & virtual) network interfaces with the same MAC address on the same machine

[15:33] dimitern, Fab! [15:33] Thank you very much dimitern [15:34] sinzui, :) np === vladk|offline is now known as vladk === vladk is now known as vladk|offline === vladk|offline is now known as vladk === vladk is now known as vladk|offline === vladk|offline is now known as vladk === vladk is now known as vladk|offline [18:43] sinzui: I have a fix for this bug, but I don't think I actually know the area of the code well enough to be confident that it's the right fix. It sort of looks like it should never have worked before: https://bugs.launchpad.net/juju-core/+bug/1304407 [18:43] <_mup_> Bug #1304407: juju bootstrap defaults to i386

[18:44] natefinch, I think other rules that forced the local arch were in play [18:45] sinzui: could be. It looks like the code that picks the image gets a list of matching ones back (one for amd64 and one for 386) and then just takes whichever is first [18:46] ouch [18:48] natefinch, Isn't the real issue with that bug is that we think amd64 is preferred either because AWS prefers it or because we see our local arch as the preference? [18:50] natefinch, would setting a large mem constraint also force selection of amd64? (all the i386 instances have small memory) [18:51] sinzui: what I was seeing was that we were passing in the constraints the user had defined (in this case, no constraints), and then filtering the list of images down to the cheapest ones, which leaves m1.small, and there's two versions, 386 and amd64. Since there was more than one that matched what the user wanted we just picked the first one. I don't know how it was being restricted to local arch before. [18:52] sinzui: what my change does is that if there's more than one image that matches what the user requested, it prefers to choose the one with the same arch as the local machine [18:52] sinzui: but if such a thing doesn't exist, it just picks whatever is first in the list [18:53] natefinch, I agree with your solution. I suppose for many people, the arch is not important so ling as the service works [18:54] sinzui: right. If it were up to me, I'd probably just default to always choosing amd64... it's generally the default these days anyway, and matching the dinky old laptop someone is using to run the client on is not very intuitive to me.... but I'm not sure if other people had a specific reason for matching the local machine [18:55] natefinch, I agree. I suspect the surprise was we expect the more powerful /better arch to be selected [18:56] sinzui: I can send a quick email to the list about it. either way is trivial to code. I'd think most people would presume 64 bit is better, all things being equal. [18:58] yep [19:51] sinzui: any idea on how to reproduce the upgrading issue? I just went 1.18->trunk without a hitch [19:52] natefinch, Your units upgraded? [19:53] sinzui: yep just standard wordpress/mysql [19:53] sinzui: but 1.19 now [19:53] (1.19.1.1) [19:54] The tests all set tool-metadata-url to the testing streams [19:55] CI is republishing tools now. in 15 minutes there will be tools that are trunk [19:56] sinzui: I did use --upload-tools, that probably skews things [19:56] Yes, users are not supposed to use that [19:57] I cannot set tools on joyent because several the libs used by precise are tool old [19:57] and the machines are not allowed to get deps from anywhere other than Lp [19:57] * sinzui ponders giving up for the day [19:59] sinzui: I'll take another look without upload-tools [20:20] sinzui: how do I get it to upgrade without if 1.19 hasn't been released [20:21] set tools-metadata-url to one of the testing streams [20:21] natefinch, which cloud are you using [20:21] sinzui: aws [20:22] natefinch, juju-dist.s3.amazonaws.com/testing/tools [20:22] hmm, publication of the latest rev is stalled though [20:22] * sinzui looks [20:30] juju status [20:30] hehh [20:32] man I hate that we have environments.yaml and the jenvs [20:33] I always go edit the environments.yaml first and wonder why it doesn't do anything [20:37] sinzui: I can't make tools-metadata-url work. I put it in the correct jenv, but I still get no upgrades available [20:38] natefinch, This is what I have for aws: http://pastebin.ubuntu.com/7278665/ [20:40] sinzui: maybe the problem is that I changed it after I bootstrapped [20:40] I already reported that bug :) [20:40] natefinch, I think I cannot be changed if it was ever set [20:40] But when not set, you can set it once [20:41] sinzui: is there more to setting it than just editing the jenv? [20:41] natefinch, I would prefer to run the tests by bootstrapping with the released stream, then change tools-metadata-url to use the testing stream [20:42] natefinch, I used juju set-env tools-metadata-url=https://juju-dist.s3.amazonaws.com/testing/tools [20:42] It works for my joyent env which didn't have that key set [20:43] sinzui: ahh, that worked [20:44] natefinch, oh was that key set in the env before? [20:44] I want to update the bug with your experience [20:45] sinzui: no, it wasn't set before [20:45] sinzui: I just thought I could edit the jenv directly, but that doesn't seem to work [20:46] the jenv is just the pre-state used to bootstrap the env. [20:47] sinzui: I thought that was the environments.yaml :/ [20:47] I think there is a bug reported asking that juju warn when the jenv doesn't match the env [20:47] sinzui: I guess that's the pre-pre-state [20:47] :) [20:50] sinzui: anyway, my upgrade worked fine [20:51] natefinch, looky http://ec2-54-84-137-170.compute-1.amazonaws.com:8080/job/hp-upgrade/1090/console [20:51] That just happened in CI [20:51] What is the lastest revs? [20:52] joyent is testing upgrade now [20:53] sorry, not sure what you're asking about latest revs [20:53] joyent just passed [20:55] natefinch, r2655 works [20:56] dimerrs branch doesn't look related, but it has a positive effect [20:57] natefinch, local just passed [20:57] azure and aws are testing now [20:57] and you effectively did the aws test this hour [20:58] sinzui: yeah, that's cool [20:58] man..... I really don't get how launchpad is organized. How do I just get a list of commits to trunk? It shouldn't be that hard to find [20:59] ahh, I see.. I can't click on trunk, because that's a "Series" [20:59] I like the qbzr extension locally [20:59] Lp lists the last 10 commits to the branck [20:59] sinzui: huh, never occurred to me [21:00] the branch is owned by gobot [21:00] https://code.launchpad.net/~go-bot/juju-core/trunk [21:00] I know that since I need to explicitly be that bot to tag the branch. [21:00] sinzui: right [21:01] sinzui: Andrew made a commit this morning that looks like it might have been more likely to fix things. At least it mentioned upgrade changes. [21:01] sinzui: 2654 [21:02] I think so to reading the log, but the hp, joyent, and local upgrade tests failed with that specific rev. [21:03] sinzui: weird [21:04] sinzui: well, EOD for me regardless. Glad it seems to be upgrading now, whatever the reason [21:04] Have a nice weekend natefinch [21:05] sinzui: you too