/srv/irclogs.ubuntu.com/2015/05/13/#juju-dev.txt

menn0wallyworld: this doesn't actually need a review right? http://reviews.vapour.ws/r/1653/diff/00:32
mupBug #1454468 was opened: Hyper-V Server 2012 R2 node deployed by maas but juju status remains pending <oil> <juju-core:New> <https://launchpad.net/bugs/1454468>00:48
wallyworldmenn0: hey, sorry was out at accountant. anastasia has looked at it, i was just waiting for 1.24 to be unblocked01:15
wallyworldthanks for asking01:15
wallyworldericsnow: it was master01:17
davecheneythumper: i have a horrible feeling I know what is going on with arm64 and juju01:17
davecheneylong story short, the text of most of the net error messages has changed01:17
davecheneyi'm sure we're doing string sniffing01:18
davecheneythat's why it's not figuring out when ports are not available and stuff01:18
menn0wallyworld: ok cool01:22
menn0wallyworld: what's blocking 1.24? I don't see anything01:23
wallyworldmenn0: was blocked yesterday, and i landed another one first this morning before i had to pop out. will land this one now01:23
menn0wallyworld: cool cool01:25
davecheneythumper: yup, i was right01:26
davecheneyjuju local provider tests fail on amd6401:26
davecheneyunder go 1.501:26
davecheney(which is what we have to use for arm64)01:26
davecheneyit's the changes to the net output text01:26
davecheney:-<01:26
davecheneygrr01:26
natefinch-afk...and what have we learned, kids?  Don't do string matching on error messages.01:32
mupBug #1454359 changed: Unexpected api shutdown followed by panic <juju-core:New> <https://launchpad.net/bugs/1454359>01:36
davecheneynatefinch-afk: sad trombone01:39
=== natefinch-afk is now known as natefinch
natefinchyeah....   it's unfortunate that gocheck makes it so easy to do string parsing on errors01:40
thumperdavecheney: net output text?01:41
davecheneymy guess is somewhere in the local provider we're looking a err.Error() to figure out what kind of failure it was01:41
thumperugh01:42
davecheneyyeah, this is going to suck for a while01:43
davecheneystraddling a bunch of different go versions01:43
natefinchgood for us... make us stop doing stupid things01:44
davecheneyhttp://paste.ubuntu.com/11105279/01:44
davecheneylets hear it for juju/errors01:45
menn0davecheney: nice01:46
natefinchI need a smart diff tool that can diff two git patch files to prove a forward port really is just a copy of another PR.02:21
natefinchit has to be smart, because for some reason, the order of files in the patches is not constant, so you can't easily just do a text diff02:21
natefinchrelated... can someone review this forward port to 1.23 of a fix already merged to 1.22?  http://reviews.vapour.ws/r/1659/02:23
natefinchmenn0: if you're not working on that customer issue? ^02:23
menn0natefinch: if it's a forward port of an already reviewed PR with minimal additional changes then it doesn't need another review02:27
natefinchmenn0: even better02:27
menn0natefinch: I did see it :)02:27
natefinch:)02:28
davecheneynatefinch: menn0 thumper https://github.com/juju/juju/pull/230402:38
natefinchdavecheney: wow, that seems kind of broken02:39
davecheneythe contract for the net pacakge says the errors will be of type *net.OpError02:39
davecheneynatefinch: https://groups.google.com/d/topic/golang-dev/nHMOfuSBYLs/discussion02:40
mwhudsondavecheney: is this why everything was timing out?02:41
davecheneyyup02:43
davecheneywell, sort of the opposite02:43
davecheneythe test was expecting that ECONNREFUSED was ok02:43
davecheneyerr02:43
davecheneyit was expecting to get ECONREFUSED02:43
davecheneyit got another error and freaked02:43
natefinchwish there had just been a IsConnectionRefused(error) bool02:44
davecheneyi considered breaking that big block of logic out into a function02:45
davecheneyif you review my PR02:45
davecheneyyou can ask me to do that02:45
natefinchdavecheney: I meant in the stdlib :)02:45
davecheneyhmm02:46
davecheneymaybe02:46
davecheneybut where does it end02:46
natefinchdang, I always forget how to do "fix it then ship it" :/02:49
natefinchthere, close enough02:49
menn0davecheney: looking02:49
natefinchdavecheney: lol, just saw your post to that golang-dev thread on this change.02:50
davecheneynatefinch: pride comes before the fall02:51
davecheneynatefinch: menn0 thanks for your review02:58
davecheneyi've pulled the logic out into a function02:58
davecheneylet's see if it'll fit through the CI needle02:59
mupBug #1454481 was opened: juju log spams ERROR juju.worker.diskmanager lsblk.go:111 error checking if "sr0" is in use: open /dev/sr0: no medium found <juju-core:New> <https://launchpad.net/bugs/1454481>03:21
* thumper snorts03:22
thumperdavecheney: I also looked but you already had two shipits03:22
thumperso I didn't bother adding another03:22
thumperwas more snorting at the 'fit through the CI needle" comment :-)03:23
anastasiamacthumper: interestinghow ur virtual reactions are so different to ur real life ones..03:24
thumperanastasiamac: or not so different...03:27
thumperanastasiamac: although to be honest, I have never stabbed anyone in the face in person03:27
mupBug #1454481 changed: juju log spams ERROR juju.worker.diskmanager lsblk.go:111 error checking if "sr0" is in use: open /dev/sr0: no medium found <juju-core:New> <https://launchpad.net/bugs/1454481>03:27
anastasiamaccan't imagine u  "snort"ing :D03:28
* natefinch can.03:28
* natefinch ducks03:28
anastasiamacu r brave, Nate :D03:28
natefinchanastasiamac: nah, I'm just very far away ;)03:29
anastasiamac:D03:30
mupBug #1454481 was opened: juju log spams ERROR juju.worker.diskmanager lsblk.go:111 error checking if "sr0" is in use: open /dev/sr0: no medium found <juju-core:New> <https://launchpad.net/bugs/1454481>03:36
thumperWT actual F03:44
thumperha03:44
thumperhmm03:45
thumperI should set an alias for 'got est' to be 'go test'03:45
thumperI hate timing based intermittent failures in our test suite03:45
thumperjust sayin03:45
thumperActionSuite.TestActionsWatcherEmitsInitialChanges03:46
* thumper looks at jw403:46
anastasiamac-*- thumper watches jw4 sleep...03:48
thumperfark03:49
* thumper hates git at times03:49
* thumper branches then does a git dance03:50
davecheneyhttp://juju-ci.vapour.ws:8080/job/github-merge-juju/3219/console03:55
davecheneywhat gives03:55
davecheneyis this a flake ?03:55
davecheneythumper: well, i'm testing with go trunk03:55
thumperprobably one of the more boring fixes for a critical issue http://reviews.vapour.ws/r/1669/03:55
davecheneyi need to check it'll fit through the 1.2 sieve03:56
davecheneygit it03:56
davecheneyhulk smash03:56
thumperdavecheney: looks like an ec2 failure03:56
thumpermenn0: you were reviewer right?03:57
menn0thumper: I am03:57
thumpermenn0: care to look at the above review?03:57
menn0thumper: looking03:58
thumpermenn0: cheers03:58
menn0thumper: wow so that was a bit of screwup04:04
thumperyeah04:04
thumpernaming things is hard, right?04:04
menn0thumper: that explains the huge number of SetAddresses calls04:05
menn0and the resulting txns04:05
thumperright04:05
menn0thumper: ship it... no comments04:05
* thumper looks at something04:05
thumperthis would add one transaction per machine per 15 minutes04:06
thumperso in an environment with 100 machines, 400 per hour04:06
thumperhmm...04:09
thumperI wonder if I got that __fixes__ thing right, or even if this fix needed it04:09
natefinchthumper: AFAIK nothing is blocked04:10
thumpermerge accepted04:10
* natefinch has landed stuff on 1.22, 1.23, 1.24, and master today04:10
thumpermenn0: I'll look to forward port the fix once it has landed04:11
menn0thumper: sweet04:11
jw4thumper: Yeah, that WatcherEmitsIntialChanges test sux0rs04:14
jw4thumper: mea culpa04:14
jw4hopefully I'll get a chance to fix it if no-one else does this cycle04:15
axwmenn0: would you please take a look at http://reviews.vapour.ws/r/1670/? trivial change, fixes a critical bug04:15
jw4I'll squeeze it in with Actions 2.004:15
menn0axw: looking04:16
wallyworldaxw: what about existing systems - do we need an upgrade step?04:16
menn0axw: looks good. what manual testing have you done to confirm it works?04:18
jw4anastasiamac: it's only 9:20 pm... but I feel like I *should* be sleeping04:19
anastasiamacjw4: well, if ur were working from an office, would u have been available?..04:20
jw4anastasiamac: nope04:20
jw4anastasiamac: at least not normally :)04:20
davecheneychecking: go vet ...04:21
davecheneyprovider/vsphere/ova_import_manager.go:269: missing verb at end of format string in Debugf call04:21
davecheneywhat's the point of doing this check if it doesn't fail ?04:21
anastasiamacjw4: :D ur dedication and quick reponse is appreaciated04:21
jw4davecheney: you ranted about that last week didn't you?04:21
jw4anastasiamac: ;)04:21
davecheneyi rant about a lot of stuff04:21
jw4:)04:21
davecheneyit's hard for me to keep track04:21
wallyworlddavecheney: i already ranted about that earlier today too04:25
wallyworlddavecheney: i think go vet doesn't always exit with an error (at least for the Go version used by the bot)04:26
wallyworldbut ffs, why don't developers have it turned on locally04:26
jw4wallyworld, davecheney the scripts/verify.bash file does return non-zero... the error must be swallowed elsewhere in CI?04:26
wallyworldjw4: it depends on the version og go vet i believe04:26
jw4wallyworld: ah04:27
wallyworldearlier versions were broken04:27
wallyworldand CI uses Go 1.2.104:27
axwwallyworld: worker/rsyslog will rewrite the config if changes, so no upgrade step required04:27
wallyworldaxw: awesome thanks, just wanted to double check due to the sensitity of the site04:27
* thumper goes to walk the dog while the landing bot does its thing04:28
axwmenn0: I confirmed that the config is updated, and that logging still works/is distributed. I couldn't repro; I left a note to that effect in the bug, will test manually if someone is able to give me the steps04:28
=== thumper is now known as thumper-bbs
menn0axw: I suspect you need to have an HA setup where rsyslog isn't up on one or more of the state servers so that the outbound queue fills up. with your change, the queue should fill up to 512MB instead of to all free space.04:30
axwmenn0: yeah, I tried that04:30
menn0axw: (the outbound queue should fill up on the hosts where rsyslog is working obviously)04:31
axwmenn0: from the bug: "I tried creating an HA env, creating a giant machine-0.log and all-machines.log on machine 0, and then disabling rsyslog on the 2nd and 3rd state servers; didn't cause the spool to grow by more than a couple MB."04:31
menn0axw: then I don't know. jam might have better context (he's around)04:32
axwjam: ^^  any ideas on how to repro https://bugs.launchpad.net/juju-core/+bug/145380104:33
mupBug #1453801: /var/spool/rsyslog grows without bound <stakeholder> <juju-core:Triaged> <juju-core 1.22:In Progress by axwalk> <juju-core 1.23:New> <juju-core 1.24:New> <https://launchpad.net/bugs/1453801>04:33
jamaxw: looking04:35
jamaxw: put 4GB of data into machine-0.log04:36
axwjam: from the bug: "I tried creating an HA env, creating a giant machine-0.log and all-machines.log on machine 0, and then disabling rsyslog on the 2nd and 3rd state servers; didn't cause the spool to grow by more than a couple MB."04:36
axw(giant was ~512MB, not 4GB, though)04:37
jamaxw: so to start with what version is your env?04:37
axwjam: 1.22.304:37
jamaxw: so to repro, I'd recommend using 1.20 if you can04:38
jamat the very least because I believe rsyslog might be configured slightly differently (forward messages from the log file vs connect to rsyslog and send messages directly)04:38
jamyou're Right about ActionQueueMaxDiskSpace04:38
axwI'll give 1.20 a shot04:39
jamaxw: do you see if 1.22 is reading machine-X.log and forwarding the content?04:44
jamaxw: ah I have another way04:45
jampython!04:45
axwjam: it was when rsyslog was enabled, yes04:45
jampython -m "import syslog; syslog.openlog('juju-test'); syslog.syslog(syslog.LOG_WARNING, 'test this out'))"04:45
jamaxw should end up in the all-machines.log04:45
jamaxw: and you can trivially loop over that to create as much logspam as you want04:45
axwjam: thanks, I'll try with that04:46
thumper-bbsgrr... bad record mac05:02
axwjam: I think my previous attempts failed because I was creating log lines that were too long, not fitting into syslog messages05:03
axwthis seems to be creating more spool files05:03
jamaxw: so you need lots of short messages rather than few long ones ?05:03
axwjam: at least log lines smaller than 1MB :)05:03
jamaxw: we probably have some sort of "max message length"05:04
axwyes probably. I'm testing with 1K now, seems to be doing the trick. now to see what happens when it gets to 512MB05:04
davecheneywhat the heck http://paste.ubuntu.com/11107599/05:08
davecheney... error stack: github.com/juju/juju/environs/bootstrap/bootstrap.go:103: Juju cannot bootstrap because no tools are available for your environment.05:10
davecheneyYou may want to use the 'agent-metadata-url' configuration setting to specify the tools location.05:10
davecheneydid juju just tell me to go fuck muself ?05:10
thumper-bbsdavecheney: but nicely...05:11
davecheneyok, i think these are all symptoms of missing tools05:12
=== TheRealMue is now known as TheMue
jamaxw: how goes?05:39
axwjam: trying to figure out where my rsyslog config files went...they've disappeared out of /etc/rsyslog.d05:40
jamaxw: that doesn't sound very good. I thought juju writes them on startup/etc.05:41
axwjam: yeah, worker/rsyslog is meant to write it out05:41
thumper-bbswallyworld: we aren't reviewing forward ports of fixes are we?06:00
* thumper-bbs forgot to change nick06:00
=== thumper-bbs is now known as thumper
* thumper guesses not06:04
axwjam: finally, verified. I think the worker/rsyslog code isn't very robust to rsyslog being restarted externally06:09
axwi.e. if it's not running, and tries to stop/restart it, it'll be unhappy06:09
jamaxw: ah, we're probably just issuing a "stop" which will fail if it isn't running, and not handling the "stop a stopped service"06:10
jamaxw: but you found you could overload the queue ?06:10
axwjam: yes, and it stopped growing when I added the patch06:10
jamaxw: can you confirm if the patch would obviously apply to a 1.20 branch?06:15
jamaxw: and any thoughts on how we might test in "as realistic as we can for CI" ?06:16
axwjam: yes, the 1.20 code is pretty similar, so would work there06:17
jamaxw: you're doing this on 1.22, right? Have you checked 1.24/master as well?06:17
jamI haven't heard the official statement from alexis and wes about what version we're targetting06:17
axwjam: ensure-availability, juju ssh 1 "sudo stop rsyslog", logspam on machine 0 until all-machines.log grows to 512MB, and then a bit more06:17
axwjam: I thought 1.22 was the important one, that's all I've looked into so far. AFAIK the others haven't changed dramatically06:18
axwI'll work on forward porting them now06:18
jamaxw: I agree with that basic sentiment, wes and alexis were supposed to meet last night to discuss official upgrade plans for them06:19
axwwallyworld: do you know any more about ^^ ?06:19
* thumper head desks06:31
* thumper grunts06:32
thumper*another* damn intermittent failure06:33
thumperfuck fuck fuckity fuck06:33
thumperprovider/vsphere tests on 1.2406:34
thumperand for extra hillarity06:35
thumperenviron_broker_test.go:93:06:35
thumper    c.Assert(err, gc.ErrorMatches, "no mathicng images found for given constraints: .*")06:35
thumper... error string = "invalid URL \"http://cloud-images.ubuntu.com/releases/streams/v1/index.json\" not found"06:35
thumper... regex string = "no mathicng images found for given constraints: .*"06:35
thumpernote the spelling mistake in the regex06:35
* thumper fixes that first06:36
jamaxw: I'm happy with your patch, but I'm wondering why we ended up with lines added in 2 places06:42
wallyworldaxw: we are going to patch 1.22 afaik06:42
jamah non API vs API06:42
wallyworldjam: that's your understanding too right? we talked this morning at the release meeting that the imminent arrival of 1.22.x into trusy is now being delayed till these issues are fixed in 1.2206:44
jamwallyworld: Last I had heard it was going to be discussed at the release meeting, which I was not at06:44
jamand it was a "do we go for 1.22 and push back 1.24"06:44
wallyworldso, if i understood correctly, it is 1.2206:45
wallyworldas we want these fixes in trusty also06:45
thumperwallyworld: care to cast your eye over this commit? https://github.com/howbazaar/juju/commit/9757821b5070ff26510cedc58e7919450ebfa9a606:47
thumperwallyworld: not sure why it was intermittently failing06:47
wallyworldlooking06:47
thumperwallyworld: but with this patch, it passes all the time06:47
thumperwallyworld: the log showed that the file source was read, and didn't find an image06:48
thumperbut sometimes it would try to get to cloud-images.ubuntu.com...06:48
thumperno idea why it was only sometimes06:48
wallyworldthumper: NFI about intermittent nature either06:48
wallyworldbut good that you fixed the other stuff06:49
thumperthis does seem to make it go away though06:49
wallyworldhmmm06:49
wallyworldif it works, but would like to understand the root cause06:49
wallyworldi'm looking at another bug related to this06:49
wallyworldi'll poke around a bit06:49
thumperyeah... me too06:49
thumperfix it taking a while to land06:50
thumperlanded in 1.2206:50
thumpertrying 1.2306:50
wallyworldthumper: i'm looking at bug 145242206:50
thumperbefore I try 1.2406:50
mupBug #1452422: Cannot boostrap from custom image-metadata-url or by specifying metadata-source <sts> <juju-core:Triaged> <https://launchpad.net/bugs/1452422>06:50
wallyworldwe're not overlapping are we?06:50
thumperdon't think so06:50
thumpernot at all06:50
thumperall my changes have been around machine Addresses and SetAddresses06:50
wallyworldok, just when you said me too i wasn't sure06:50
thumperjust a slight diversion to fix the intermittent vsphere test failure06:51
wallyworldok06:51
thumperme too was relating to wanting to know the root cause06:51
wallyworldah06:51
wallyworldanyway, +1 on that fix06:51
thumperI hate weird shit like that06:51
wallyworldyup06:51
thumpercheers06:51
thumperjust checking06:51
wallyworldthe regexp typo made me laugh06:52
wallyworldtalk about tweaking the test to match bad code :-)06:52
wallyworldyou can tell it wasn't TDD :-)06:52
jamwallyworld: prob just copy and paste06:53
wallyworldyup06:53
jamthumper: do we know what menn0's plan is for working on https://bugs.launchpad.net/juju-core/+bug/1453785 ?07:04
mupBug #1453785: transaction collection (txns) grows without bound <stakeholder> <juju-core:Triaged> <juju-core 1.22:In Progress by menno.smits> <juju-core 1.23:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1453785>07:04
jamI feel like we should have some coordination to find out when we can purge items from the txn collection07:04
jamit is possible that everything can be purged once they have been in APPLIED07:04
thumperjam, wallyworld: quick hangout to hand off?07:12
wallyworldsure07:13
thumperwallyworld, jam: https://plus.google.com/hangouts/_/canonical.com/handoff?authuser=007:13
jamrogpeppe2: ^^07:13
axwwallyworld: when you're free, can you please pastebin the output of "sudo lsblk"? I don't have an optical drive :)07:17
wallyworldaxw: me either07:17
axwwallyworld: oh, I thought sr0 was optical. well, anyway, I don't have one of them07:18
wallyworldaxw: neither does my output07:18
axwwallyworld: same machine you had juju running on? juju just runs "lsblk"...07:19
axwwallyworld: juju just runs lsblk. you're on the same machine you had juju running, where the log was spammed?07:19
axwsorry, thought I was disconnected07:19
axwwallyworld: sorry, I'm an idiot07:20
axwwallyworld: you didn't send the bug report... :)07:20
wallyworld:-)07:23
mupBug #1454599 was opened: firewaller gets an exception if a machine is not provisioned <cpec> <stakeholder> <juju-core:Triaged> <https://launchpad.net/bugs/1454599>09:40
wallyworldaxw: very small review? http://reviews.vapour.ws/r/1677/09:47
axwlooking09:48
wallyworldty09:48
axwwallyworld: done09:51
wallyworldaxw: tyvm09:52
voidspacewallyworld: axw: ping - know anything about lifecycle watchers?09:53
wallyworlda little09:53
axwvoidspace: what about them?09:53
voidspacewallyworld: axw: I have a new watcher / worker combo watching for when IPAddresses become Dead and releasing them with the provider09:54
wallyworldohh, nice09:54
voidspacemachine removal marks the addresses as dead, which should trigger the worker to release and remove them09:54
voidspacethe watcher / worker is tested - setting an IPAddress to Dead triggers its removal09:54
voidspacemachine removal is tested09:54
voidspaceremoving a machine marks associated IP addresses as dead09:55
voidspacebut an end-to-end test fails09:55
voidspaceallocating an ip address to a machine and then removing the machine *does* mark the address as Dead09:55
voidspacebut the watcher doesn't seem to notice it - it's not released09:55
voidspaceI wonder if I'm missing anything obvious09:55
wallyworldit is most likely a resource catsing issue09:56
wallyworldcasting09:56
wallyworldi've seen before09:56
axwvoidspace: where's the worker?09:56
voidspaceI figure it maybe something about our test infrastructure - two states or something09:56
voidspaceworker/addresser/worker.go09:56
voidspacelet me link you to the current WIP09:56
wallyworldwhere if  struct was not castable to an interface i can't remember, events were rejected09:56
voidspace<voidspace> I figure it maybe something about our test infrastructure - two states or09:57
voidspacedammit09:57
voidspaceaxw: https://github.com/juju/juju/compare/1.23...voidspace:addresser-machine-destruction09:57
voidspacewallyworld: interesting09:57
voidspacethe only difference is that when a machine is removed the ipaddress is set to Dead as part of a bigger transaction09:57
voidspaceaxw: TestMachineRemovalTriggersWorker is the failing test09:58
axwmk09:58
voidspaceaxw: it fails in waitForReleaseOp09:58
voidspace(and without waiting for the release op it fails because the address really isn't removed)09:58
wallyworldvoidspace: i'll see if i can find the code09:58
voidspaceif I change state.Machine.Remove to call address.EnsureDead (set the address to dead in its own transaction) the test *still fails*09:59
voidspaceyet tests that do *exactly that* pass09:59
voidspaceso I suspect test infrastructure problems09:59
voidspacewallyworld: ok, cool09:59
voidspacethe debug output shows that the watcher never sees the event10:00
axwvoidspace: what first came to mind was that you might need to do s.State.StartSync() just before waitForReleaseOp... but it looks like you're using just the one State, and not BackingState+State10:08
axwvoidspace: nothing jumps out, sorry10:09
wallyworldvoidspace: do you have the watcher code?10:09
axwwallyworld: it's a plain old lifecycle watcher: https://github.com/juju/juju/blob/master/state/watcher.go#L17410:10
wallyworldoh, right10:11
voidspaceStartSync doesn't appear to help10:13
voidspaceI see the ip address life set to Dead - watcher isn't triggered10:13
voidspacewallyworld: axw: ok, time to start digging into the event code10:14
voidspacehah10:15
voidspaceaxw: wallyworld: moving the StartSync to later in the test worked10:15
voidspacemagically...10:15
voidspaceaxw: wallyworld: thanks...10:15
voidspaceyay for mysterious magic10:15
wallyworlddidn't do anything, glad you got it working10:15
axwvoidspace: where in the test did you add it?10:15
wallyworldbut i hate magic :-)10:15
axwI'm curious to know why that works10:15
wallyworldme too10:16
voidspaceaxw: just before the "machine.EnsureDead()"10:16
axwhuh, that doesn't make any sense10:16
voidspacetest now passes10:17
axwvoidspace: I can't repro success with that change, is it passing reliably for you?10:17
voidspacecelebration coffee10:17
voidspaceaxw: I also needed to tweak the instance ID I allocate to10:18
voidspaceaxw: just pushed a passing branch10:18
voidspaceah no10:18
voidspaceaxw: just failed10:18
voidspaceaxw: maybe it's a timing issue10:19
voidspacegoddammit10:19
axwvoidspace: if the sync were to make sense anywhere, it'd be after the machine.Remove()10:19
axwbut that fails for me too10:20
voidspaceI just had two passes10:20
voidspacenow two fails10:20
voidspaceaxw: with *three* calls to StartSync it reliably passes, remove any one and it seems to fail10:23
voidspaceafter machine provisioning, after address creation and allocation and after machine removal10:23
axwvoidspace: still fails with three for me; I suspect it's just adding enough time for it to see the event in your case10:24
voidspacethat's awful10:24
voidspacehmmm... no, it seems like only the first two are needed10:24
axwsorry, didn't try in all those spots10:24
voidspaceit's still reliably passing10:24
voidspacethat kind of makes sense - it syncs the two new entities - the machine and the address10:25
axwvoidspace: sorry not sure, gotta go help get kids ready for bed.. I'd be interested to know if you get to the bottom of it10:26
voidspaceaxw: well, that works...10:27
voidspaceI can dig through the connsuite and try and see where we end up using the different states10:27
voidspaceaxw: laters o/10:28
natefinchis wallyworld on?  I want to bitch about launchpad some more ;)10:50
wallyworld\o/10:50
natefinchwallyworld:  I added a tag to a bug,  but when I click on the link for that tag that was created, it brings me to a list of bugs that doesn't include the bug I just tagged10:51
natefinchbug: https://bugs.launchpad.net/juju-core/+bug/145228510:51
mupBug #1452285: logs don't rotate <cpec> <logging> <stakeholder> <juju-core:Fix Released by natefinch> <juju-core 1.22:Fix Committed by natefinch> <juju-core 1.23:Fix Committed by natefinch> <juju-core 1.24:Fix Committed by natefinch> <https://launchpad.net/bugs/1452285>10:51
natefinchlink from tag: https://bugs.launchpad.net/juju-core/+bugs?field.tag=cpec10:51
wallyworldit the bug assigned to the 1.24 series?10:52
wallyworldyes, i can see it is10:52
mupBug #1452285 changed: logs don't rotate <cpec> <logging> <stakeholder> <juju-core:Fix Released by natefinch> <juju-core 1.22:Fix Committed by natefinch> <juju-core 1.23:Fix Committed by natefinch> <juju-core 1.24:Fix Committed by natefinch> <https://launchpad.net/bugs/1452285>10:52
mupBug #1454627 was opened: presence shouldn't try to hold all possible Sequences at once <performance> <tech-debt> <juju-core:Triaged> <https://launchpad.net/bugs/1454627>10:52
wallyworldnatefinch: the default search criteria only includes open bugs i think10:53
wallyworldso if the bug is fix committed it won't show up (a guess)10:54
wallyworldyou may need to go to advanced deatch so you can explicitly select the bug sates you want10:54
wallyworldstates10:54
wallyworldeg fix committed, triaged, in progress etc10:54
natefinchahh, that actually almost makes sense :)10:55
wallyworldnatefinch: and i just tested it, advanced search works10:57
wallyworldif you clock on advanced search, you'll see the default tags10:57
natefinchwallyworld: yeah, I did too.10:57
wallyworlds/tags/states10:57
wallyworldnot obvious i agree10:57
natefinchit didn't help that I'd *just* set it to fix released...  plus the whole "only searching one series" thing.10:58
wallyworldyeah11:00
dooferladTheMue: http://reviews.vapour.ws/r/1679/ - if you could take a look. This should be getting familiar now!11:25
jamwallyworld: or axw: question about "juju status" and agent up/down11:25
wallyworldyo11:26
jamthis is older code, but it might be stuff that you guys have thought about recently11:26
TheMuedooferlad: sure, will do11:27
TheMuedooferlad: done11:30
TheMuedooferlad: a diff between two PRs would be nice, so that it more easily can be compared11:30
* TheMue is at lunch now11:30
mupBug #1454661 was opened: presence collection grows without bound <performance> <tech-debt> <juju-core:Triaged> <https://launchpad.net/bugs/1454661>11:47
anastasiamacjam: ping12:06
jamhey anastasiamac, sorry I got my head deep in this sky stuff. Give me a sec and I'll be right there.12:07
anastasiamacjam: can reschedule if it's easier :D12:07
jamanastasiamac: joining now12:14
voidspaceknown problem in ubuntu 15.04: can't enter decryption password for encrypted hard drive on boot12:29
voidspacehttps://bugs.launchpad.net/ubuntu/+source/linux/+bug/135968912:29
mupBug #1359689: cryptsetup password prompt not shown <apport-collected> <iso-testing> <kernel-da-key> <kernel-fixed-upstream> <kernel-graphics> <rls-v-incoming> <utopic>12:29
mup<vivid> <linux (Ubuntu):Triaged by mathieu-tl> <linux (Ubuntu Utopic):Triaged> <linux (Ubuntu Vivid):Triaged by mathieu-tl> <https://launchpad.net/bugs/1359689>12:29
voidspace*sigh*12:29
voidspaceand there's no supported way to remove full disk encryption either12:32
mupBug #1454676 was opened: failed to retrieve the template to clone - 500 Internal Server error - error creating container juju-trusty-lxc-template - <oil> <juju-core:New> <https://launchpad.net/bugs/1454676>12:32
mupBug #1454676 changed: failed to retrieve the template to clone - 500 Internal Server error - error creating container juju-trusty-lxc-template - <oil> <juju-core:New> <https://launchpad.net/bugs/1454676>12:41
mupBug #1454676 was opened: failed to retrieve the template to clone - 500 Internal Server error - error creating container juju-trusty-lxc-template - <oil> <juju-core:New> <https://launchpad.net/bugs/1454676>12:47
mupBug #1454678 was opened: "relation-set --file -" doesn't seem to work <landscape> <juju-core:New> <https://launchpad.net/bugs/1454678>12:47
mupBug #1454678 changed: "relation-set --file -" doesn't seem to work <landscape> <juju-core:New> <https://launchpad.net/bugs/1454678>12:53
mupBug #1454678 was opened: "relation-set --file -" doesn't seem to work <landscape> <juju-core:New> <https://launchpad.net/bugs/1454678>12:59
wwitzel3ericsnow: ping13:07
mupBug #1454697 was opened: jujud leaking file handles <cpec> <stakeholder> <juju-core:Triaged> <https://launchpad.net/bugs/1454697>13:38
jcastromgz, any luck with dreamhost?13:47
mgzjcastro: expect to have some news shortly, I need to poke a bit more13:51
jcastroack13:52
katcoericsnow: standup14:04
katcovoidspace: hey are you in #juju on canonical's irc?15:20
natefinchkatco: he hides there as mfoord15:24
lazyPowerx-post from #juju -- "Man, actions + the new status  stuff in 1.24 is really nice. hattip @ jujucore for this"15:42
=== anthonyf is now known as Guest56189
rogpeppe2anyone know how to change the debug level of a running juju environment?16:24
mgzrogpeppe2: `juju set-environment logging-config=<>`16:28
rogpeppe2mgz: ah, ok, i wondered if it was set-environment16:28
rogpeppe2mgz: the help could be more helpful there, i think :)16:28
mgz`juju help logging` isn't bad... just not super concise16:28
=== anthonyf is now known as Guest84788
natefinchcherylj: how's the file handle bug going?17:20
=== kadams54 is now known as kadams54-away
katcocherylj: perrito666: btw if you don't think you'll get a patch up for your bugs before your EOD, please be sure to update the bug with any information so we can hand them off17:48
perrito666katco: sure17:48
* perrito666 tries to reproduce a bug by having crappy db17:56
perrito666s/db/bw17:56
cheryljnatefinch, katco:  sorry, was at an appointment.  I'm still digging into that bug18:00
=== kadams54-away is now known as kadams54
katcowwitzel3: hey do you have some code i can look at for the container management stuff?18:11
wwitzel3katco: I do18:13
wwitzel3katco: https://github.com/wwitzel3/juju/tree/ww3-container-mgmt18:14
perrito666oh great, I am almost falling sleep on the kb and spotify decides to play total eclipse of the heart18:14
* perrito666 makes coffee18:14
katcolol18:14
wwitzel3katco: that is fairly recent, I haven't pushed anything from today yet, will after it is actually compiling :)18:14
katcowwitzel3: can i suggest this (https://github.com/juju/juju/blob/master/apiserver/leadership/leadership.go#L41-L49) as a way of doing IoC for the apiserver stuff?18:21
katcowwitzel3: also wondering if we really need a state object in api/procmanager/procmanager.go? what functionality are we using from state?18:23
katcowwitzel3: and last observation, ericsnow and i have been talking about organizing code into modules, so would it make sense to put all of this code in a central spot, and then utilize the interesting bits in the various areas of Juju?18:25
wwitzel3katco: yeah, I like the idea of all the code being in the same package, we are going to be registering / unregistering the process with state, those methods aren't there yet18:30
katcowwitzel3: ah gotcha. so could we just pass in a few closures or an interface that handle the registration?18:30
wwitzel3katco: that is the idea, yeah18:31
katcowwitzel3: awesome... looking forward to seeing the next iteration of code18:31
wwitzel3katco: thanks for looking at the WIP, appreicate the reivew, it should all be a little more concerete tomorrow and we can shop it to every one via a review18:33
katcowwitzel3: sweet18:33
natefinchcherylj: want some help?  I could help try to repro or something.18:33
perrito666natefinch: have a maas?18:34
* perrito666 grins18:34
natefinchperrito666: nope.  I got a maas half set up at the sprint but then hit some problems and never got further with it18:35
perrito666meh, I thought you had a hardware maas18:35
natefinchperrito666: I have hardware that I could probably make into maas given time. ... but time is not something I have much of.18:36
cheryljnatefinch: yeah, that would be helpful if you could do that...18:36
natefinchhaha juju 1.20 does *not* appreciate all the extra garbage in my environments.yaml19:03
natefinchhuge list of 'WARNING unknown config field "blah"'19:04
natefinchsinzui: what do we expect for backwards compatibility between juju versions?  I had a 1.22 local environment and tried to  juju status using a 1.20 client, and it just failed19:11
natefinch(hung forever)19:11
sinzuinatefinch, that is a very bad. I don't think CI has seen that though. The compatibility tests for 1.20 clients to 1.22 servers could get status and do other ops19:12
natefinchsinzui: I wonder if juju local is just special19:13
sinzuinatefinch, shouldn't be, but since everyones local id a little different it can be19:13
natefinchcherylj: for what it's worth, I can't seem to reproduce the file descriptor leak.  At least from the proposed hypothesis of it just being because the API server is down.19:35
=== kadams54 is now known as kadams54-away
natefinchcherylj: granted, I'm using 1.20.14, not 1.20.11 like at the customer site... I can try 1.20.11 and see if it changes anything though (also trying on juju local, but I can't imagine that matters).19:35
cheryljnatefinch: yeah, I haven't had much luck with that either.19:36
perrito666and you dont have a lot of descriptors open?19:40
perrito666if it is a  leak it most likely showing even when not arriving to a critical point19:40
sinzuinatefinch, I think you have found a regressions. I may need to block 1.22.1 going into trusty.19:43
sinzuinatefinch, 1.20.x client sees this error talking to a 1.22.x env: x509: certificate is valid for localhost, juju-apiserver, juju-mongodb, not anything19:44
natefinchsinzui: oops19:45
sinzuinatefinch, I wonder if this error is about closing a security issue, in which case, it is intentional19:46
natefinchsinzui:  I don't know19:54
sinzuinatefinch, I think this is just for new envs. I am retesting upgraded envs19:54
natefinchsinzui: yes, this was not an upgraded environment19:57
TheMueanyone free for reviewing http://reviews.vapour.ws/r/1681/ ?19:58
sinzuinatefinch, this is just old clients cannot be guaranteed to talk to envs bootstrapped by newer/securer clients. upgraded envs continue to to work. I just took 1.20.11 to 1.21, 1.1.22, then 1.23 and all is good19:59
natefinchsinzui: ok... I find that odd, but since I don't really care about backwards compatibility personally, I'm ok with it if you're ok with it ;)20:00
perrito666brb, new firmware for my wifi card20:03
perrito666well, no improvements, new hardware an linux is a nightmare20:04
mupBug #1454829 was opened: 1.20.x client cannot communicate with 1.22.x env <compatibility> <status> <juju-core:Triaged> <https://launchpad.net/bugs/1454829>20:36
natefinchwell, the good news is, we're going to have a brand new A/C unit.  That's also the bad news.20:41
=== kadams54-away is now known as kadams54
perrito666natefinch: well, that fix gave you an extra year20:43
natefinchperrito666: yes, but it cost me $100020:43
natefinchperrito666: I don't really want to pay $1000 for a year of A/C20:44
perrito666no warranty on the fix?20:45
perrito666the bad part is, if you had known this a couple of months ago you could have saved the time you spent trying to protect it from the falling ice20:46
natefinchperrito666: not sure about warranty, probably not (or not more than like 30-60 days)20:49
natefinchgotta run, Lily has an art show at her school20:50
natefinchcherylj: FWIW, I have some agents that are very very slowly gaining file handles (like one per half hour), not sure where though, so I'll leave them to run and see what happens.20:50
perrito666there is a joke around there about running, heat and AC but I cannot quite make it20:51
natefinchhaha20:51
perrito666bbl20:55
=== _thumper_ is now known as thumper
=== Spads_ is now known as Spads
mupBug #1454658 was opened: TestUseLumberjack fails on windows <blocker> <ci> <regression> <test-failure> <windows> <juju-core:Triaged> <juju-core 1.23:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1454658>21:27
thumperFFS21:48
thumperwaigani: I don't suppose you have a windows box handy to run tests on?21:49
thumperwaigani: can I get you to look at bug 1454658 ?21:49
mupBug #1454658: TestUseLumberjack fails on windows <blocker> <ci> <regression> <test-failure> <windows> <juju-core:Triaged> <juju-core 1.23:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1454658>21:49
waiganithumper: windows? what's that?21:49
thumperwaigani: CI blocker, and very simple fix21:49
thumperthis is the merge that brought in the failure: https://github.com/juju/juju/pull/2303/files21:49
waiganithumper: okay, I'm giving this upgrade bug one final pock (I may have Stockholm syndrome)21:49
thumperwaigani: please leave it for a bit, and look at this critical bug21:50
waigani*poke21:50
waiganithumper: yep, will do21:50
thumpercmd/jujud/agent/machine_test.go needs two tweaks21:50
thumperfunc (FakeConfig) LogDir() string should wrap the file path in filepath.FromSlash21:51
thumperunit_test needs it too in the same palce21:52
thumperand the test that checks the filename also needs filepath.FromSlash21:52
thumperI *think* that should be sufficient21:52
thumperto fix the windows issue21:53
* thumper has other ports to fix21:53
=== kadams54 is now known as kadams54-away
waiganithumper: sure. I'm not going to be able to test in on windows though.21:53
thumperwaigani: it is purely an assumption on slash file path separators21:53
thumperwaigani: that's fine, as long as it passes locally, submit it as a fix for that bug and CI will tell us21:53
thumperI'm 98% sure this will fix it21:54
waiganiok, on it21:54
thumpercheers21:54
thumperwaigani: also not that you should start on the 1.23 branch21:55
thumperand forward port through 1.24 and master21:55
waiganiI was going to ask, okay21:55
thumpers/not/note/21:55
wallyworldalexisb: you joining sky handoff?22:02
waiganithumper:  http://reviews.vapour.ws/r/1683/22:06
waiganithumper: local unit tests pass22:07
thumperwaigani: first one merged, now to forward port :)22:35
waiganithumper: okay. ports don't need reviews right?22:35
thumperwaigani: as long as they apply cleanly (and it should)22:36
waiganithumper: https://github.com/juju/juju/pull/2323 - 1.24 port22:38
thumperwaigani: LGTM22:39
thumperwaigani: the one thing I'd add for the next one is to mention in the pull request that it is a forward port of a previously reviewed and landed fix22:39
thumperwaigani: please also keep the bug tasks up to date :-) ta22:42
waiganithumper: https://github.com/juju/juju/pull/2324 - 125 port, added comment22:44
thumperwaigani: nice, thanks - generally I wait for the previous target to merge before pushing the next in22:45
waiganithumper: yep. I haven't tried to merge 1.25, waiting for 1.2422:45
thumpercool22:46
mupBug #1454870 was opened: Client last login time writes should not use mgo.txn <juju-core:In Progress by thumper> <juju-core 1.22:In Progress by thumper> <juju-core 1.23:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1454870>22:57
thumpermenn0: do you know of examples in state where we write to the db without a transaction?23:00
menn0thumper: state/sequence.go does23:03
thumperta23:03
waiganithumper: 1.24 landed, 1.25 landing... bug status updated23:03
menn0thumper: but that uses mgo.Change and Apply which is a little esoteric23:04
menn0thumper: State.AddCharm doesn't use a txn... and probably should. that looks like a bug to me.23:06
thumpermenn0: if not mgo.Change and Apply then what?23:15
menn0thumper: someCollection.Insert/Update/RemoveId/etc etc23:16
thumperhmm..23:17
menn0there's lots of methods on collections which let you add, modify and delete docs23:17
thumperdo we do an Update on a collection anywhere?23:17
* thumper looks for the mgo docs23:17
menn0thumper: not in state proper23:17
menn0thumper: you probably want UpdateId23:19
thumperyeah, that is what i'm doing :)23:20
menn0thumper: i've just noticed that the collection type you get back due to the auto multi-env stuff doesn't support any of the Update* methods :(23:21
menn0thumper: easily added though23:21
thumperagh23:22
menn0thumper: i probably didn't implement them b/c we weren't using them anywhere23:22
thumperI don't think that is in 1.2223:22
thumperbut I'll talk to you about adding it as I go23:22
menn0thumper: might not be, in which case you're ok ther23:22
menn0there23:22
menn0thumper: and the compiler will tell you when you get to the version that does have the multi-env collections stuff23:23
menn0:)23:23
* thumper nods23:23
thumpermenn0: oh shit23:25
thumpermenn0: it is in 1.2223:25
menn0thumper: ha23:25
menn0thumper: ok, it's easy enough to add the required method(s)23:25
* thumper nods23:25
menn0thumper: do you just need UpdateId/23:25
thumperI'm going to add update and updateid for consistency23:30
=== bradm_ is now known as bradm
thumpermenn0: could I get you to look at http://reviews.vapour.ws/r/1687/ for me plz?23:49
thumpermuch appreciated23:49
* thumper goes to the gym23:49
menn0thumper: looking23:55

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!