[00:13] great, now my ppc64 machines are offline [01:09] thumper: http://paste.ubuntu.com/11087681/ [01:09] what is this test trying to test ? [01:09] what should be listening on port 19034 [01:20] katco: you still around? [01:27] davecheney: I'd like your insight on a review comment. I have a struct that implements an interface from the same package. The struct has no exported methods other than the ones on the interface, and no exported data fields. I have a constructor function for the struct... I had made the struct non-exported and had the constructor just return the interface. I was given a review comment that returning an interface is an an [01:27] tipattern, and to just have it return the concrete value. I had made it return the interface to keep the package's API smaller. What are your thoughts? [01:28] natefinch: link to review ? [01:28] davecheney: http://reviews.vapour.ws/r/1619/ [01:29] * davecheney looks [01:29] I converted cmd/jujud/agent/AgentConf from a struct to an interface to make it testable... I don't necessarily like making an interface just to make something testable, but it seemed like the most reasonable solution. [01:33] davecheney: the relevant changes are from revision 1 to revision 2 [01:34] kk [01:41] davecheney: looking now [01:42] wallyworld, thumper: update on the CI blocker. I have a working fix that I've manually tested in a variety of bootstrap, upgrade and HA scenarios. [01:42] wallyworld, thumper: tests need updating though and that'll take a while [01:42] menn0: ETA? [01:45] thumper: no idea. anywhere from 30 to 90 mins? [01:45] menn0: ack [01:46] * natefinch thinks he has a different definition than menn0 has of what "no idea" means. [01:46] natefinch: :) [01:47] :) [01:55] somewhere between a femtosecond and a millenium [01:55] now that's more like it. [01:59] thumper: any idea what is going on there ? [01:59] davecheney: sorry, there wasn't quite enough context [02:00] davecheney: which test? [02:00] davecheney: the proxy detection was probably part of the auto-proxy detection [02:00] but the error I'm guessing would be the test trying to get to cdimages [02:00] no? [02:02] davecheney: any thoughts on that review? No big deal if you don't have time, just wanted anything you had off the top of your head. [02:31] thumper: http://paste.ubuntu.com/11087681/ [02:31] wallyworld, thumper: here's the blocker fix for 1.23. http://reviews.vapour.ws/r/1654/ [02:32] what is this test looking for on port 19034 ? [02:32] looking [02:34] davecheney: likely to be the random port number assigned to either mongo or the api server [02:37] not random [02:37] happens over and over again [02:37] hmm.. [02:37] all the way through the test [02:37] all the way through the test suite [02:37] /home/tim/go/src/github.com/juju/juju/testing/environ.go: [02:37] 52: "state-port": 19034, [02:38] so the state server didn't start up ? [02:38] * thumper shrugs... [02:38] maybe [02:38] righto [02:38] thanks [02:38] davecheney: or... [02:38] hang on [02:38] davecheney: the test isn't a mgo suite so no mongo started, but the code is expecting it to be started? [02:39] if that port isn't random [02:39] and two tests using that suite run at the same time ... [02:39] they'll fight over the port [02:39] ack [02:39] how the f did this ever work ? [02:39] I didn't think we ran things in parallel [02:39] go test github.com/juju/juju/... [02:40] don't we compile in parallel, but run in serial? [02:40] ^ will run n cores of tests concurrently [02:40] not sure that is how gocheck works [02:40] gocheck is a test [02:40] menn0: lgtm, a question about var name [02:40] but each packge is it's own test runner [02:40] and up to n cores of them will be running at any time [02:41] davecheney: I thought that for most of the tests, we used random port assignment [02:41] davecheney: and the fixed numbers were only used for static checks [02:41] indeed [02:41] but perhaps I'm wrong [02:42] maybe this suite is only used by one provider [02:42] the local provider tests [02:42] I have certainly run tests in parallel before [02:42] in multiple terminals [02:42] wallyworld: yeah, I might call it needReplicasetInit and change the sense. good call. [02:42] if you pass multiple packages to go test [02:42] they will be run in parallel [02:46] ah fark [02:46] adding lxd ppa brought in golang 1.3.3 [02:47] my build .a files are now all fubared [02:47] * thumper goes to delete stuff [02:48] ... [02:48] why is rm -r pkg not working? [02:50] * thumper is going crazy [02:51] davecheney: sanity check plz [02:54] damn it [02:54] * thumper reboots [03:12] anyone about? I'm seeing weird errors from a juju bootstrap [03:12] WARNING juju.replicaset replicaset.go:87 Initiate: fetching replication status failed: cannot get replica set status: can't get local.system.replset config from self or any seed (EMPTYCONFIG) [03:13] natefinch: i read that review twice [03:13] are you testing AgentConf ? or the thing that consumes AgentConf ? [03:14] davecheney: the thing that consumes it [03:14] davecheney: which is why I wanted to make it an interface so that I could mock out its behavior [03:14] in that case, I agree [03:14] that returning the interface is ok? [03:14] although dogmatic, it seems like a good rule of thumb that functions should return concrete types and consume interfaces [03:14] oh, ok [03:15] I'm cool with that [03:15] and consuming an interface gives you heaps of flexibilty to control the _outputs_ of that interface [03:15] so i think this change is good [03:16] but i am reminded of this quote [03:16] Don’t be trapped by dogma – which is living with the results of other people’s thinking. [03:28] ah, LP#1412621 and LP#1441913 seems similar-ish. [03:38] <_thumper_> davecheney: got a minute? I have an x509 cert question, and I found your name in the source :) === _thumper_ is now known as thumper [03:43] * thumper stabs x509 and asn.1 in the face [03:44] * thumper goes to make coffee [03:44] * thumper guesses davecheney is having lunch [03:53] wallyworld: trying to merge this CI blocker but so far i've had good ol' "bad record mac" and EC2 failures [03:53] ffs [03:53] :( [03:53] bad record MAC happens so often [03:54] fixed in 2.6, we need to get off 2.4 [03:55] * thumper nods [03:57] thumper: menn0: we need to get onto 2.8+ to get rid of the noisy logs, but W only has 2.6 so there's a packaging process that has to be followed [03:57] * thumper grunts [03:57] thumper: i'm just off for lunch [03:58] what line ? [03:58] this was probabl somethig about the ASN.1 decoding [03:58] wallyworld: we need some packaging monkey at our beck and call [03:58] yes we do [03:58] davecheney: I have a cert pool, and I get the subjects out [03:58] davecheney: I was wanting some meaningful decoding for the test [03:59] davecheney: but asn1.Unmarshall with a pkix.Name fails [03:59] when I assumed it would work [03:59] given that it is what is passed in to the cert [03:59] what's the error [03:59] asn1.StructuralError{Msg:"tags don't match (16 vs {class:0 tag:17 length:13 isCompound:true}) {optional:false explicit:false application:false defaultValue: tag: stringType:0 set:false omitEmpty:false} @2"} [04:00] urgh, it's been 4 years since I left that comment turd [04:00] asn1 marshalling looks a bit ick [04:00] yueah, there are lots of non obivious differences between a stirng [04:00] a mature string (or seomthing) [04:00] looking at the raw bytes, it looks like it definitely has the Org and CN [04:00] an empty string [04:00] a raw string [04:00] a set of raw bytes [04:00] etc [04:01] none of them map cleanly on to the string "tim" [04:01] 16 SEQUENCE, SEQUENCE OF [04:01] 17 SET, SET OF [04:01] map vs [] i'd say [04:01] whatever you're doing now [04:01] do the opposite [04:01] davecheney: ugh... I'm using pxik.Name which uses []string for members [04:02] nfi sorry [04:02] it's been 4 years [04:02] and I think I only fixed it enogh last time to decode some SNMP records [04:03] hmm... for the test, the CN would be fine [04:03] perhaps I should just define a struct that uses just that... [04:05] davecheney: don't stress, I'm just kinda gunna ignore it [04:05] and assume if the loading of the pool worked, a count of certs should be fine [04:17] wallyworld: while testing upgrades of HA envs (1.20 to 1.23 on EC2) I ran into a failure of the "migrate tools into environment storage" upgrade step [04:17] wallyworld: something about an invalid URL [04:17] oh joy [04:17] got an error message? [04:18] pgrade step "migrate tools into environment storage" failed: cannot find tools in provider storage: cannot read product data, invalid URL "https://s3-us-west-2.amazonaws.com/46494ee31f7046468790ca9b1ac37795/tools/streams/v1/com.ubuntu.juju:released:tools.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI3NTF6NNI5343APQ%2F20150512%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20150512T040240Z&X-Amz-Expires=604800&X-Amz-Signature=3 [04:18] 131f68ef4919a36267d56a20490b4746208423247196fbac578dc637d370144&X-Amz-SignedHeaders=host" not found [04:22] menn0: i *think* we have now switched to V4 auth for S3, perhaps there's an issue in that region or something [04:23] wallyworld: file a bug? [04:23] hmmm, i think i truncated to url, fixing that gave a sig mismatch which i think might be expected if i just cut and paste [04:24] but it could also be a real issue [04:24] did it happen all the time? [04:24] or just the once? [04:25] i've only tried this particular upgrade (1.20 to 1.23) once [04:25] aws does not always work all the time [04:25] CI needs to often retry [04:26] wallyworld: but there were a bunch of attempts to hit that URL and they all failed [04:26] sometimes though there are cloud outages [04:26] i can't see offhand where any juju bug would be [04:26] wallyworld: ok it could be that [04:26] the url looks correct [04:27] wallyworld: the upgrade retried 5 times and has now given up [04:28] not sure what to do - juju is asking S3 for a file and S3 says f*ck off [04:30] wallyworld: so we hit that URL something like 50 times over 10 minutes and it didn't work [04:32] hmmmm, NFi offhand [04:35] menn0: if i try the url without the auth stuff i get a weird redirect from aws. not sure if that's relevant but i've not seen that before [04:36] wallyworld: here's the full correct URL [04:36] https://s3-us-west-2.amazonaws.com/46494ee31f7046468790ca9b1ac37795/tools/streams/v1/com.ubuntu.juju:released:tools.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI3NTF6NNI5343APQ%2F20150512%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20150512T043517Z&X-Amz-Expires=604800&X-Amz-Signature=3754e068be3d7588022c5ed7d485fff4f3c2df1fd917ed12c29fa7bd9a430a12&X-Amz-SignedHeaders=host [04:36] wallyworld: the error is about signing: The request signature we calculated does not match the signature you provided. Check your key and signing method. [04:36] yes, but i've seen that will url pastes before [04:37] wallyworld: ok right [04:37] it may be relevant here but i can't be sure off hand [04:37] wallyworld: i've just searched the logs and I see the same error there [04:37] i wonder if you specified an explicit region would it work [04:37] hmmm, maybe that region is having issues with V4 [04:41] menn0: it merged \o/ [04:43] thumper: thank $deity for that [04:43] menn0: I think $deity should be Odin [04:43] was always partial to Norse [04:43] thumper: that fits you so well [04:46] getting the 1.24 and master merges ready now [04:49] menn0: although perhaps it should be Loki as he was god of mischief, and that fits the intermittent nature better I think [04:50] thumper: ha :) [04:50] Bug #1454043 was opened: Machine.SetAddresses should not generate transactions for noop changes

[04:57] thumper: if we r voting, i'd say Odin [04:58] thumper: Odin's "special" warriors were berserkers === urulama__ is now known as urulama [05:23] davecheney: remember this 10 minute go test timeout thing... [05:23] davecheney: are you sure that `go test -test.timeout=1200s ` doesn't increase that to 20 minutes? [05:42] wallyworld, thumper: the fix for the CI blocker is now merged into 1.23, 1.24 and master [05:42] huzah [05:44] wallyworld: CI hasn't run the functional-ha-backup-restore job with the new code yet [05:44] yeah, have to wait to unblock [05:45] wallyworld: i'll add a note to the ticket to make it clear when it can be unblocked [05:46] sounds good, ty [05:48] menn0: hoo ray [05:52] axw: ping [05:54] Bug #1454059 was opened: SetTools fills up transaction log

[06:30] TheMue: ping [07:53] morning [07:53] jam: pong [07:53] hi TheMue just looking for someone from the team. We found a bug in the instance poller and were wondering who did the Addresses consolidation. [07:54] dimiter might be more knowledgeable if he's around today [07:54] jam: yeah, but he's out until thursday [07:55] jam: Address poller? hmm, could you tell me more about it? [07:58] TheMue: basic bug is https://bugs.launchpad.net/juju-core/+bug/1454043 [07:58] Bug #1454043: Machine.SetAddresses should not generate transactions for noop changes

[07:58] TheMue: essentially the InstancePoller is doing a check [07:58] If machine.Addresses() != cloud.Addresses() { machine.SetAddresses(cloud.Addresses)) [07:58] TheMue: however, machine.Addresses() merges "machine" addresses with "cloud" addresses [07:59] thus once you have any machine addresses, the comparison *always* fails. [07:59] iiirks [07:59] TheMue: essentially SetAddress() && Addresses() gives you a different value [07:59] which isn't a great API internally :) [08:00] hmmm, will be hard to disagree ;) [08:01] morning all [08:02] voidspace: heya [08:02] * TheMue is taking a deeper look [08:04] voidspace, morning [08:05] voidspace, you did work on the networking stuff right? [08:08] TheMue: o/ [08:08] mattyw: uhm... yes... [08:08] voidspace, I've not really started looking at it - so be prepared for a totally daft question.... [08:09] mattyw: back in a minute, ask the question and I'll answer [08:09] voidspace, is it possible to move a unit from one space to another? [08:09] mattyw: oh [08:09] mattyw: units aren't "in spaces" [08:09] mattyw: units are on machines, machines are on networks / have interfaces - and those networks / interfaces have subnets that are in spaces [08:10] mattyw: you specify the space constraints at *deploy time* (so we can pick a machine0 [08:10] mattyw: and then a unit endpoint can get an address on any subnet available to the machine [08:10] mattyw: that is how it *will* work I believe [08:10] mattyw: not implemented yet [08:11] voidspace, ok understood - so I can't use networking to cause partitions in a running service (for testing purposes) [08:11] mattyw: but it's machines that are "in spaces", and we pick a machine for the unit based on the spaces you specify (for endpoint bindings) at deploy time [08:11] mattyw: not dynamically no [08:12] voidspace, understood [08:15] mattyw: as I understand it you can install a charm with a different name in a different space, but not different units of one service [08:15] TheMue, that makes a lot of sense [08:32] jam: I like your solution, always had my troubles with the mixing of both address sets. [08:45] Bug #1454143 was opened: Deployer failure upgrading to proposed 1.23.3 from 1.23.2 client

[09:02] voidspace: hangout [09:02] dooferlad: omw [11:12] TheMue: dooferlad: oh, forgot to mention in standup - but last night I spent a bit of time on my HP Microserver [11:12] voidspace: and now, up and running? [11:12] TheMue: dooferlad: got the ram and HD fitted. Unfortunately failed to get an OS installed as the flash drive I made from 15.04 was corrupt [11:12] TheMue: dooferlad: I did find a python interface to the OneView HP RESTful API [11:12] * ashipika has quit (Quit: Leaving.) [11:12] oops... [11:13] hehe [11:13] https://github.com/HewlettPackard/python-hpOneView [11:13] Not clear if it supports the functionality we need - but a good place to start. [11:13] voidspace: could you point me with an URL to the HW? [11:14] TheMue: the doc by dooferlad has a link to the device on ebuyer (UK site of course) [11:14] https://docs.google.com/document/d/1jjw80acrwVMITCiAKI3UJ5FoZFuN9kuBcG0mtHRmMIA/edit [11:14] it's the G1610T HP Proliant Microserver [11:14] voidspace: thx [11:14] ? [11:16] ashipika: heh [11:16] ashipika: I managed to get the wrong thing in my copy-paste buffer... [11:16] ashipika: meant to post a link instead... sorry [11:17] voidspace: np :) [11:20] voidspace, TheMue: The API spec is in that document. We only need a single endpoint: {"Action": "Reset", "ResetType": } where can be "On", "ForceOff". That is it. [11:21] As long as you can find the IP address that the management interface is listening on we are fine. [11:22] voidspace: you don't need to install an OS, you just need to get in the BIOS to set up network booting. [11:35] dooferlad: sure, but I want to make sure it works and have something to play with :-) [11:35] dooferlad: pretty sure it network boots by default [11:35] dooferlad: it's definitely looking for pxe [11:36] TheMue: hey, did you see that 1.24 bug i assigned you? it appeared as if you were the last one to touch that code :-) looks like a simple fix. are you able to fix it? [11:37] wallyworld: not yet seen, one moment [11:37] morning [11:37] wallyworld: #1437266? [11:38] perrito666: heya [11:38] perrito666: morning, i haven't landed the 1.24 merge into master which will bring in the charm.v5 stuff you need because 1.24 was blocked; should be unblocked soon [11:38] np [11:43] TheMue: yeah, that's the one [11:44] wallyworld: ok, will do it [11:44] ... now [11:44] \o/ ty [11:52] wallyworld: as you're here, quick question [11:53] wallyworld: when setting the instance ID on a newly provisioned machine I need to set the instance id on a bunch (potentially) of IP addresses as well [11:53] wallyworld: state already has a method to fetch all the IP addresses associated with the machine [11:53] wallyworld: after the successful transaction, ok to just call addr.SetInstanceId on them all [11:54] wallyworld: or is it significantly preferred that I build this in as part of the transaction (by extending []txn.Op for setting the instance id on the instanceData record) [11:54] ? [11:54] voidspace: give me a few minutes to check, i can't recall the code 100% [11:55] wallyworld: well, it's mostly new code [11:55] wallyworld: specifically I'm looking at state.Machine.SetProvisioned [11:56] setting the instance id is a method call on machine, so it seems having a method to set the instance id on the addresses fits [11:56] but it means a few extra DB calls instead of building it into a single transaction [11:57] voidspace: if the method is on state and hence talking to mongo directly, a few extra calls isn't so bad in general [11:57] we do that elsewhere [11:57] wallyworld: well, it's on state.IPAddress which has a direct connection to state [11:57] wallyworld: cool, thanks [11:57] it's directly working with state entities, no extra layers [11:58] if depends if we expect all the data to be written at once for integrity [11:58] hmmm... although these are cross collection anyway [11:58] so the transaction doesn't guarantee that I don't think [11:59] instanceDataC and ipaddressesC [11:59] IIRC [11:59] yeah, cross collection txns i think can be an issue [12:00] is it an issue if the instance ids are not all set at once? [12:01] wallyworld: nope. They're only used on machine destruction. [12:02] wallyworld: so unless you can set the instance id on the machine and then destroy the machine (EnsureDead *and* Remove) before they're set [12:02] which I'm pretty sure you can't... [12:02] voidspace: i think then it sounds ok to set individually via repeated direct calls to mgo from state [12:02] seing the final code would help [12:04] wallyworld: cool, thanks [12:05] wallyworld: I'll link you to the wip code to get a clearer idea. [12:05] sorry i was only vague [12:06] hmmm... problem as always is what to do with the error [12:06] if setting the id on an address fails *after* the transaction succeeds returning an error will make it look like setting the instance id failed [12:07] maybe better to do it in the transaction so we get a single failure [12:07] slightly more fiddly code which I was hoping to avoid [12:12] davecheney: are you still around? [12:13] voidspace: another option is to rollback using a defer [12:21] wallyworld: right, thanks [12:39] rogpeppe1: hiya, i have a question for you if you are free at some point [12:39] wallyworld: go ahead. [12:40] wallyworld: i am fairly busy but will time-slice :) [12:41] rogpeppe1: so, there's a bug. deployer calls AllWatcher.Next(). this errors with an error that appears to come from an update() call at some point on one of the backing objects. 1. does this sound plausible that an error generated during an update() callback on megawatcher would appear when Next() is called? [12:41] 2. the error seems "impossible" [12:42] the error is due to a missing status record for a unit [12:42] but when units are created, they get a status record as part of the txn ops [12:42] wallyworld: perhaps Next is returning the error from an underlying tomb? [12:43] could be i guess. i didn't realise the update() on the mulitwatcher set the tomb error [12:43] wallyworld: if that tomb is killed with an error, all watchers will die with that error. [12:43] wallyworld: i don't know either [12:43] wallyworld: it's a random guess without looking at the code [12:43] wallyworld: (which i haven't looked at in ages and ages) [12:44] ok, , i'll keep digging, ty [12:44] wallyworld: np, good luck! [12:44] i'll need it :-) [13:07] sinzui: can we unblock 1.24? bug 1453805 seem sot have fixed the HA functional tests in CI [13:09] wallyworld, Not until i understand all the failures. [13:10] sinzui: you talking about the maas 1.7 failure? [13:11] wallyworld, I am talking about all maas failures and kvm failures [13:11] no maas passed. [13:11] oh ok, i just saw the onw maas 1.7 one [13:12] wallyworld, CI will unblock when the branch was blessed, so CI thinks wont unblock until QA proves Juju wasn't at fault for any of the failures [13:13] ok, i was hoping that as soon as the recorded blocking bug was seen to be fixed we could unblock [13:14] wallyworld, yes, and CI recognises that as a blessed branch. CI doesn't care about a specific test. It cares about having a releasable branch [13:16] ok, so that means that even though the bug causing the original blockage is fixed, we still need to wait in case there are new bugs oto be raised [13:22] katco, We need someone to look into bug 1454143. We may not be able to the finale release of 1.23.3 [14:04] katco: sorry, omw, was chatting with cory_fu and lost track of time [14:04] dpb1: hey can you provide any guidance on bug 1454143 [14:05] #1454143 [14:05] Don't go blaming me. ;) [14:05] * dpb1 kicks mup [14:05] : hi! here's bug https://bugs.launchpad.net/juju-core/+bug/1454143 [14:05] * ericsnow worries that mup might not be well [14:05] LOL [14:06] * perrito666 kicks his computer in anger because nothing works well today [14:06] niemeyer has been pinged, he's probably looking at it [14:15] mgz, still no luck on dreamcompute, I'm stuck, any help someone on your team could provide would be most welcome! [14:18] jcastro: okay, I'll try to have a go myself today [14:18] I saw you hadn't got a response to your mail [14:18] heh [14:19] Will we be testing on dreamcompute in the future for releases? [14:19] we haven't got a plan to at the moment, it seems reasonable though [15:29] ericsnow, wallyworld 1.23 and 1.24. bug 1392407 was the involved in a system failure caused by a held lock [15:30] ericsnow, wallyworld CI blessed and opened 1.24. I opened 1.23 because I think I accounted for every failure as a cloud or substrate issue [15:30] sinzui: cool, thanks! [15:40] o/ - Juju's amazon provider does not understand how to leverage the VPC correct? Thats still something we do manually, and then use the manual provider to consume? [15:47] lazyPower, that is correct VPC support is still under development [15:47] alexisb: thanks, thast what I thought :) [15:47] just making sure i hadn't missed an update === ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs: None [16:24] ericsnow: you around? [16:24] natefinch: otp [16:24] natefinch: what's up? [16:27] ocr: can I get a stamp on backport http://reviews.vapour.ws/r/1660/ === natefinch is now known as natefinch-afk [17:26] I'm running into a weird issue bootstrapping on AWS in a VPC w/ 1.24 and 1.23 - i've confirmed the issue affects both - https://bugs.launchpad.net/juju-core/+bug/1412621 [17:26] Bug #1412621: replica set EMPTYCONFIG MAAS bootstrap

[17:26] i attached here as it was the same issue - but the bug title seems to scope it to maas. [17:29] and I think i see whats happening, its inserted a host that is the NAT'd address as the replset configuration, and not the local address of the unit [17:40] updated the bug w/ the workaround/steps to fix the bootstrap issue - now the system appears stuck when issuing "juju status" [17:41] Bug #1454359 was opened: Unexpected api shutdown followed by panic [17:42] is there anyone around to do reviews atm? [18:36] cherylj: perrito666: wwitzel3: have time for a hangout? [18:37] katco: sure [18:37] cherylj: perrito666: wwitzel3: https://plus.google.com/hangouts/_/canonical.com/moonstone?authuser=1 [18:37] getting some tea brt [18:38] Bug #1453805 changed: Juju takes more than 20 minutes to enable voting