[00:42] thumper: thats most of the fix [00:42] there is still another place they need to put a mutex, in _loopWorker or whatever it's called [01:17] I think this bootstack bug #1463480 might be a dup of bug #1416928. Can anyone take a look and sanity check for me? [01:17] Bug #1463480: Failed upgrade, mixed up HA addresses [01:17] Bug #1416928: juju agent using lxcbr0 address as apiaddress instead of juju-br0 breaks agents [01:37] wallyworld: PR for #1447899 https://bugs.launchpad.net/juju-core/+bug/1447899 I assume we'll also want to target 1.22? [01:37] Bug #1447899: upgrade fails if no explicit version is specified [01:37] waigani: um, we do have a window for 1.22.6 fixes. i don't know that this bug is worth backporting though [01:38] wallyworld: okay, that PR targets 1.25 - as specified by the bug report [01:38] sounds good ty === kadams54-away is now known as kadams54_ === anthonyf is now known as Guest23073 [02:53] waigani: have you tested that tools upgrade fix live? [02:54] axw: yes, sorry forgot to put that in the PR [02:54] axw: tested on aws [02:54] axw: ec2 provider [02:54] waigani: nps, thanks. LGTM [02:54] axw: used same versions as in original bug [02:54] cool [02:54] axw: awesome, thanks [02:56] thumper: panic: rescanned document misses transaction in queue [02:56] goroutine 189 [running]: [02:56] runtime.panic(0xe6fa60, 0xc210b9e990) /usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6 [02:56] your build just failed with this error [02:56] menn0: this looks very serious [02:56] ?! [02:57] waigani: with forward/back ports, I don't normally wait for LGTM unless there were significant merge differences/conflicts [02:57] davecheney: I think this is what William has been looking at fixing recently [02:58] axw: yep, same. It just automatically pops up in RB. I usually close them straight away. [02:58] davecheney: https://bugs.launchpad.net/juju-core/+bug/1449054 [02:58] Bug #1449054: Intermittent panic: rescanned document [02:59] waigani: okey dokey, just thought I'd mention in case you were waiting around [02:59] ship-it'd anyway [02:59] davecheney: looks like it's been fixed in 1.22, 1.23 and 1.24 but not master [02:59] actually, not 1.23 [02:59] it's probably just a matter of bumping the mgo version but it hasn't been done for some reason [03:00] axw: sweet, cheers [03:04] waigani: are there any details for this bug ? https://canonical.leankit.com/Boards/View/115065967/115396849 [03:05] davecheney: only happens on my machine. Let me write you an email with details. === natefinch-afk is now known as natefinch [03:06] ok [03:13] davecheney, wallyworld: this was the talk I was thinking of. The whole thing is very good, but the link brings you to a 2 minute section where he talks about not using assert libraries for testing. https://www.youtube.com/watch?v=yi5A3cK1LNA&t=9m40s [03:23] wallyworld, davecheney: here's a test file for a package I wrote recently with tests just using the stdlib's testing package. https://github.com/natefinch/pie/blob/master/pie_test.go I don't think it's terribly more verbose than using gocheck. [03:23] natefinch: it's terrible IMO :-) all those it statements when a sinlge line asset will do [03:24] 3 lines of syntax vs 1 [03:24] and no tooling for things like deep equals, err nil, length checks etc [03:25] If statements... yeah, those are rough [03:26] expecially simple ones like if len(a) != expected { [03:26] boilerplate [03:27] and you've only done simple checks so far [03:27] wait till you need to do deep equas, or same contents etc etc [03:28] wallyworld: sure, some helper functions can help... but you don't need thousands of lines of code just to avoid writing an if statement.... line returns aren't harder to type than other characters [03:28] you're missing the point - frameworks do more than just eliminate lots of boilerplate [03:28] wallyworld: and honestly, most of my gocheck code looks just like these tests. I really try hard to make tests simple [03:28] they ensure consistency and common patterns across the code base [03:32] natefinch: wallyworld leave me out of this [03:33] davecheney: heh will do [03:33] i don't want to start a land war in asia [03:33] all I want is gocheck to be maintained [03:33] +1 [04:46] wallyworld,axw: ping [04:46] wot [04:46] looking at fixing the help command [04:46] \o/ [04:46] so we can go: [04:46] juju help system environmenst [04:46] and not have to put the help in the middle [04:46] -e as well? [04:46] however, storage is doing weird shit [04:47] -e is harder and out of scope from this change [04:47] storage is 3 levels [04:47] why do you have your own Command struct in storage for the super command [04:47] that doesn't matter [04:47] by wrapping it in another struct [04:47] it means I can't go: [04:47] i can't give you a good answer off hand [04:47] juju help storage pool [04:48] I may just throw that piece away and remove a level of indirection [04:48] I think it'll still work as expected [04:48] let me go look at the code real quick [04:49] thumper: yeah, i have NFI, that doesn't make sense why it was done [04:49] remove it i say [04:49] ok, I'll do that in my branch [04:49] ty [04:52] yep, still works [05:44] axw: would it make sense for resource-get to send its streamed resource data to its stdout and the charm which invoked the get can then suck on stdout to consume? or redirect to file or pipe elsewhere or whatever [05:44] wallyworld: yes, that's what I was thinking [05:45] wallyworld: so you can "resource-get thingy | my-fancy-stream-processing-program" or whatever [05:45] axw: the reason i initially went with an option for consuming the data in the hook was convenience [05:45] yep [05:45] wallyworld: consuming the data? [05:45] wallyworld: you mean unpacking onto disk? [05:45] save the charm author from having then to invoke the hook tool [05:45] or installing deb or whatever [05:45] yeah, consume the data [05:46] wallyworld: yes, that's probably the common scenario, so we should have that as an option/probably default [05:46] ok, will add it back [05:46] i still think we'll need to introduce a usage-type [05:46] wallyworld: sorry, I didn't realise you took it out in favour of this [05:46] np :-) [05:47] wallyworld: I was thinking it'd be a property of the resource type. is that conflating? [05:48] axw: yeah, because some charms will want to just get the deb or zip as a file, others would want to unit agent to install or unpack say [05:48] IMO [05:49] so having a separate usage-type attribute allows the charm to say it wants not just the file, but some type specific post processing [05:49] wallyworld: the reason that feels awkward to me, is that what you do with the resource depends on the type of the resource. and the type of the resource may change depending on how you deploy the service, right? [05:50] wallyworld: e.g. I might deploy with a deb to install mysql, or with a github repo that's my fork of mysql [05:50] wallyworld: so what does the usage type mean in that case? [05:50] they would be different types though , right? [05:51] this is hard to discuss here, let's defer to tomorrow in 1:1 [05:51] wallyworld: sure :) === anthonyf is now known as Guest46780 === kadams54_ is now known as kadams54-away [07:19] // TODO(rog) Fix this so it doesn't wait for so long. [07:19] // https://bugs.launchpad.net/juju-core/+bug/1163983 [07:19] c.Fatalf("timed out waiting for agent to terminate") [07:19] ... Error: timed out waiting for agent to terminate [07:19] Bug #1163983: some tests take an unnecessarily long time waiting for the poll interval [07:19] .... seriously === anthonyf is now known as Guest47313 [08:31] dimitern: ping [08:36] TheMue, pong [08:37] dimitern: one question regarding IPAddress() and the address type [08:37] dimitern: in state [08:37] TheMue, yeah? [08:38] dimitern: the signature is IPAddress(value string) ... and here value is taken as the docID. but the returned address type itself has also a field called Value containing the IP-Address. how they all are related? [08:39] dimitern: does the docID contain the same value as Value and only exists for database reasons? [08:40] dimitern: oh, and yes, params.Address contains Value as field (a string) but no ID. ?!? [08:40] TheMue, the docID should contain the environment UUID as prefix to the address value [08:43] dimitern: hmm, ok, so when retrieving the IP addresses via the API I only have the value, which doesn't allows me to retrieve the IP addresses again (as state instances) for later removal. [08:44] dimitern: any troubles with extending the params.Address with the ID? [08:44] *aaaargh* [08:44] * TheMue doesn't cry due to addresses [08:45] thought I could stay on the veranda, but left hand a lawn mower and right hand craftsmen building on a house [08:45] too much noise *grmblx* [08:57] TheMue, the env uuid is automatically prepended to the id you pass [08:57] TheMue, so you don't need anything else than the value [08:58] if you use full disk encryption then ubuntu creates an unecrypted /boot partition where the linux image lives [08:59] it makes it 237Mb - so when an update includes a new image (leaving the current one there too, plus any old ones which aren't auto un-installed) [08:59] it fills up! [08:59] every other time an update includes an updated image I have to do surgery on /boot [09:00] and because the other partition is encrypted resizing partitions is non-trivial (can be done by booting a live-cd which I'll do at some point) [09:00] *sigh* [09:00] dimitern: great [09:00] and due to the bug with plymouth after an upgrade (when you have an encrypted filesystem) I'm still doing a recovery boot every time [09:01] jam, voidspace, dooferlad, standup? [09:01] dimitern: omw [09:01] dimitern: currently with mark, I'll be late if I'm there [09:04] jam, np, sure [09:20] dimitern: we could then filter juju-private from status if necessary [09:21] * dimitern steps out for a short while [09:21] voidspace, if we need to, yes [09:22] voidspace, TheMue, dooferlad, I'd appreciate a review on http://reviews.vapour.ws/r/1918/ [09:22] dimitern: just seen it [09:23] dimitern: although this is putting workarounds in place for code that only has imaginery future use cases... *sigh* [09:25] fwereade: I have a relation-get question for you when you have time [09:25] wallyworld: when you are back, we did spend about 10min on resources, and there was a pretty big thing to consider [09:29] dimitern: hmmm... a network *also* requires a ProviderId [09:30] dimitern: ok to allow that to be empty? They're supposed to be unique per provider I guess. [09:32] dimitern: yeah, there's an index on provider id [09:32] unique index [09:36] dimitern: what we *want* here is an interface that isn't associated with a provider network (except maybe the default one) [09:36] dimitern: we need to be clear about our model though [09:46] dimitern: so we either need to model the provider network that the nic is associated with *or* we need interfaces not associated with a network [09:46] dimitern: the latter seems easier at this stage [09:57] dimitern: or we could make provider id a sparse index I guess... [09:58] although we'd still have problems with duplicates [10:03] * dimitern is back [10:03] voidspace, ProviderId is supposed to be coming from the maas network name [10:04] voidspace, but I guess setting it to the same as the network name *for now* should be fine [10:06] dimitern: this is not just for maas [10:06] dimitern: this is for ec2 (and then openstack too) [10:10] voidspace, I know [10:10] voidspace, but the model (as implemented in state) will change anyway (e.g. subnets instead of networks, while networks are a different concept) [10:11] dimitern: right, but at the time we're creating the network configuration for the container I'm not sure that we *have* the network name [10:11] as it's not relevant [10:12] I'll look through the code a bit more carefully [10:12] voidspace, we *should* have it, as the container's host is already on it [10:12] voidspace, but it's not populated I guess [10:13] I don't think we have it in configureContainerNetwork I mean [10:13] I'll check [10:14] the interface info we're working with comes from the PrepareContainerInterfaceInfo call and NetworkName is not populated [10:14] I can see if it's possible to populate it there [10:14] voidspace, right [10:15] voidspace, if the host itself had it populated at provisioning, it should be possible to get it from there and use the same [10:15] voidspace, but it won't be populated on the host unless you deploy it e.g. with --networks maas-net-name [10:16] right - we do copy the NetworkName into the result, so we're being given an empty name and ProviderId [10:16] voidspace, so just hardcoding it to juju-private (with a TODO comment please that we'll need to fix that as the model gets changed) in PCII() should do the trick I guess? [10:16] dimitern: for provider Id too? that has to be unique [10:16] it's actually ProviderId that's the problem now [10:17] voidspace, unique how? [10:17] dimitern: there's a unique index on provider id in mongo [10:17] voidspace, it's no [10:17] dimitern: go look in state/open.go [10:17] voidspace, hmm [10:18] voidspace, that's correct - providerid + envuuid has to be unique within the networks collection [10:18] right, so we need a unique provider id and at the moment it's empty [10:19] voidspace, but state.AddNetwork works with existing providerId+envuuid docs and just ignores it [10:19] dimitern: yes, just found that [10:19] SetInstanceInfo ignores it [10:19] not AddNetwork [10:19] but same effect [10:19] voidspace, so it *should* be fine [10:19] dimitern: so set ProviderId to juju-private too? that's actually incorrect - network name is internal to us, provider id is meant to be, well, a provider id :-) [10:20] dimitern: I can call it "invalid" to make it explicit that we've faked it - or "juju-private-default" [10:20] which is a bit of a less scary name [10:20] if it ever leaks [10:20] voidspace, how about "not-available" ? [10:20] ok [10:21] so long as that can never clash with a real one [10:21] cool, but please make it a const somewhere (network/ I guess) [10:21] which I guess is probably safe... [10:22] I don't believe it matters, as it's not used anywhere user-visible [10:22] dimitern: the danger with ProviderId is that we actually attempt to fetch it from the provider at some point [10:22] and if it accidentally matches a real one then weird things will happen [10:23] that's why I suggested a juju prefixed name [10:23] voidspace, "juju-not-available" ? [10:23] :) [10:23] heh [10:25] voidspace, let's go with "juju-unknown" ? [10:25] ok [10:26] bootstrapping with this code... [10:26] and going for coffee whilst it's churning [10:47] hrmm coffee, good idea [10:55] wwitzel3: o/ [10:56] dimitern: and that seems to work, but the network *is* displayed in "juju status" [10:58] voidspace, unless tests start failing for status I can live with that for now :) [10:58] dooferlad, I've discovered another issue with GetContainerInterfaceInfo (or around it) [10:59] dooferlad, after host reboot, no routes are added for KVM instances' IPs (for LXC works) [11:21] dooferlad, additionally, ip route add can fail with exit 2 (RTNETLINK: File exists) and that should be handled better (it currently fails the process of the network config on the host altogether) [11:23] dooferlad, it should ignore exit 2 and only fail for exit 1 (e.g. Error: an inet prefix is expected rather than "10.15.0.". ; when doing ip route add 10.15.0. dev eth0) [11:34] dooferlad, I've added a card for you with details, please ping me if unclear [11:38] dimitern: just back from lunch. Looking. [11:39] dimitern: any idea why it has HTML tags in the description text? Rather noisy to read! [11:40] dimitern: I've seen the constants, and I liked them. I meant those expectScripts in the tests [11:44] dooferlad, where are these HTML tags? [11:45] dimitern: don't worry, I got rid of them. They were in the card description. Perhaps some copy/paste thing? [11:45] TheMue, ah, I see [11:45] dimitern: The leankit UI is horrible anyway - I don't spend much time looking at it, so it isn't a big deal. [11:45] dooferlad, ah ok [11:52] Can someone with familiarity of the OpenStack provider respond to the email in the Juju mailing list about Rackspace? [12:00] marcoceppi: we probably just need a bug filed for firewall-group: none still creating groups [12:02] marcoceppi: that setting was only added for scale testing, not for clouds that don't actually have the firewalling calls [12:02] but provider could not do the work [12:03] mgz_: could you let them know that? seems like it's the only blocker for rackspace which would be a huge win fwiu [12:03] I doubt it's the *only* blocker... [12:03] certainly the first though, and probably wouldn't be hard to make it work [12:04] I never bothered because it was a pain to get an account and I wanted the instance-level firewalling support [12:05] marcoceppi: I'll reply to the mail [12:06] mgz_: thank you! [12:32] voidspace, hey [12:33] voidspace, just a heads up - I've filed a separate bug 1464237 for the devices api work, leaving bug 1348663 only for the latest fix I did [12:33] Bug #1464237: juju should use devices API on MAAS 1.8+ for addressable containers [12:33] Bug #1348663: DHCP addresses for containers should be released on teardown [12:36] * dimitern steps out [12:39] is ther any way to get the environment details from the api? I'm trying to gather the region a deployment is in from API only [12:39] (if region is set) [12:40] Environment.info doesn't seem to have those details [12:42] ah, found it, get_env_config in the python-jujuclient will retrieve this information [12:44] Bug #1464235 opened: failed to create a security group with firewall-mode: none [12:45] Bug #1464237 opened: juju should use devices API on MAAS 1.8+ for addressable containers === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 [13:39] reviews appreciated - http://reviews.vapour.ws/r/1919/ - port of the fix for bug 1348663 to master [13:39] Bug #1348663: DHCP addresses for containers should be released on teardown [13:45] Bug #1464254 opened: charmsSuite teardown fails [13:45] Bug #1464255 opened: statesuite teardown fails === kadams54 is now known as kadams54-away [14:02] wwitzel3: standup === kadams54-away is now known as kadams54 [14:16] * TheMue 's yesterday rant about a pure functional persistency layer and a dumb model would make mocking simpler. model could be used, in-memory persistency mock would be maintained with real persistency layer [14:29] TheMue: agreed. Moonstone is making strides towards such a dream. [14:29] katco: aaaaaaaaaaaah, great enlighting news [14:29] katco: makes my day [14:29] TheMue: mind you, it will take awhile to get to the end-state, but we are chipping away :) [14:30] katco: sure, nothing to be done in a week. *lol* but it's worth it. [14:32] * TheMue with this news definitely feels better while hacking his current mocks [14:37] dimitern: so, cmd/juju tests time out [14:37] dimitern: I'm guessing there's something that blocks waiting for status to match... [14:38] dimitern: I'll find it [14:38] dimitern: fixed a handful of other tests [14:39] voidspace, right, I've suspected this much, but hopefully it shouldn't be too hard to fix [14:41] dimitern: yeah, just a case of finding the right test :-) [14:48] Bug #1464280 opened: storeManagerStateSuite teardown fails [14:48] dimitern: the panic traceback is 30211 lines long [14:48] dimitern: that's quite a few goroutines... [14:49] voidspace, wow :) not unusual [14:50] voidspace, 90% of the cases the first few tracebacks are the most useful [14:51] voidspace, I'd appreciate a review on http://reviews.vapour.ws/r/1919/ btw - live tests just passed ok on maas and ec2 [14:53] dimitern: ok [14:53] dimitern: I think I found the test in that panic [14:53] dimitern: not obvious from reading the test why it should block [14:53] but about to try it [14:53] dimitern: anyway, looking [14:54] dimitern: so you went with reboot doing an "ifdown" [14:54] voidspace, try also running suspected tests in isolation [14:54] dimitern: that's what I'm about to do... [14:54] voidspace, yeah, works well [14:54] dimitern: ok [14:57] dimitern: I hate tests that just copy large blocks of text from the code into the test [14:57] dimitern: just testing that the config is added would be enough, duplicating the script in the test proves nothing [14:58] voidspace, it proves the shutdown job for systemd is as expected for vivid [14:59] dimitern: but you got expected by copying and pasting [14:59] dimitern: so if there's an error in it the error will be in the test too [14:59] dimitern: an equally good test would just be checking a match for some unique part of it, and it wouldn't bloat the tests so much [14:59] dimitern: and they wouldn't be so fragile [15:00] I had tests fail because my editor trimmed whitespace from a script [15:00] and the whitespace was duplicated in the test :-/ [15:00] voidspace, heh :) good point, but I've also live tested that it works [15:00] I'm talking about test cleanliness [15:00] I'm sure it works, it's just horrible tests [15:00] they're a menace [15:01] voidspace, since that PR is for master I'm fine with trying to make the tests a bit better if you think it's worth it - I'll check your comments [15:01] dimitern: ok, I'll put a comment on the test [15:01] voidspace, cheers [15:01] dimitern: hmmm, that test passes on its own [15:01] dimitern: but it takes >5minutes [15:02] dimitern: I suspect they're timing out because they're awfully slow tests [15:02] (cmd/juju I mean) [15:02] I wonder why these changes would make them slower [15:02] there's some extra logging I can pull out [15:02] voidspace, try running them with -race - might uncover the issue [15:03] dimitern: ok, thanks [15:03] wwitzel3: starting to think it might be on my end. [15:03] wwitzel3: flash doesn't seem to want to work [15:03] katco: I'm back in [15:06] * dimitern will be back in 1h [15:12] dimitern: a ShipIt with an open issue for the worst offender of the tests... [15:12] test run with "-race" still going [15:13] and killed [15:14] trying just the one test [15:17] dooferlad, dimitern, could you guys take a quick peek at bug 1463480? I suspect that it's a dup of bug 1416928, but I'm not 100% [15:17] Bug #1463480: Failed upgrade, mixed up HA addresses [15:17] Bug #1416928: juju agent using lxcbr0 address as apiaddress instead of juju-br0 breaks agents [15:25] cherylj: looking [15:48] Bug #1464304 opened: Sending a SIGABRT to jujud process causes jujud to uninstall (wiping /var/lib/juju) [15:53] ^^ what, thats a feature [15:54] Bug #1464304 changed: Sending a SIGABRT to jujud process causes jujud to uninstall (wiping /var/lib/juju) [15:55] mup: infer RandomChoice[{'thats a feature', 'thats a bug'}] [15:55] natefinch: Cannot infer much out of this. :-( [15:55] aww [16:04] ericsnow: just finishing up a quick snack so we can work up until you have to take off [16:05] wwitzel3: wrapping up a review so we should be good [16:06] Bug #1464304 opened: Sending a SIGABRT to jujud process causes jujud to uninstall (wiping /var/lib/juju) === kadams54 is now known as kadams54-away [16:11] what's the flag to gocheck to not swallow logging output as it's running? [16:13] -gocheck.vv [16:20] ericsnow: heading to moonstone === kadams54-away is now known as kadams54 [17:36] Bug # changed: 1403165, 1441478, 1442257, 1453785, 1454697, 1455158, 1456989, 1463117, 1463439 [17:36] Bug #1464335 opened: debug-log does not work with local provider === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 [18:36] Bug #1464356 opened: TestCloudInit fails === kadams54 is now known as kadams54-away [18:53] evidently gc.ErrorMatches can't handle error messages with line returns in them [18:53] because even this assert is failing: c.Assert(err, gc.ErrorMatches, `.*`) [18:56] well if it uses the regex, it is correct [18:56] .* does not include newlines [18:56] iirc [18:56] and this is why I hate regexes... well, one of a thousand such reasons [19:36] Bug #1464369 opened: Sufficiently sized int config values are converted to floats [19:48] Bug #1464369 changed: Sufficiently sized int config values are converted to floats === kadams54-away is now known as kadams54 [20:40] Bug #1464392 opened: mgo iterators not being closed in state/actions.go [20:58] dinner time! [20:58] mup: infer RandomChoice[{'bye!', 'adios!'}] [20:58] natefinch: Cannot infer much out of this. :-( [20:58] mup: infer RandomChoice[{'bye', 'adios'}] [20:58] natefinch: bye. [20:58] man... picky === natefinch is now known as natefinch-afk [21:27] alexisb: oh hai [21:27] heya thumper [21:27] I need to chat go with you today before I go [21:27] :) [21:28] but I am in meetings for a but [21:28] bit [21:28] still [21:30] thumper, I will ping you when I am done with calls [21:30] kk [21:30] * thumper is about to have a meeting too [21:36] cherylj, what is the next step for https://bugs.launchpad.net/juju-core/+bug/1463480 [21:36] Bug #1463480: Failed upgrade, mixed up HA addresses [21:38] alexisb: I chatted with thumper about it a few minutes ago, and I think there's not much we can do. It is a known problem that was fixed in 1.22 [21:39] we can give them a workaround to fix the problem if they see it again [21:43] cherylj, ack, can you please follow-up with pete and make sure they go through the work-around [21:44] alexisb: will do === kadams54 is now known as kadams54-away [23:05] ok thumper [23:05] our 1x1 hangout