/srv/irclogs.ubuntu.com/2016/05/18/#juju-dev.txt

mupBug #1582948 opened: juju status: base statuses of "unknown" are not instilling confidence <cdo-qa> <juju-core:New> <https://launchpad.net/bugs/1582948>00:40
davechen1yyay, launchpad is having a fit and CI jobs are failing03:10
natefinchwallyworld: the charmstore should just set origin to store on the resource data it gives back. If it's possible some other jazzy values could be set in the future, let's just set the right thing now and not have to worry about it later.  Having a field in the return struct that is explicitly not set is galling03:17
wallyworldnatefinch: that can be done in charmrepo right?03:38
wallyworldnatefinch: just saw the admin->controller rename - did you check with QA to see what impacts it would have on their scripts?03:39
natefinchwallyworld: got a ping from cheryl to hold up, so I'm not landing it for now. It honestly didn't occur to me that it would break their scripts.... I kinda wish we didn't have all these implicit dependencies in the CI code :/03:40
natefinchwallyworld: about the origin - it's the charmstore code itself that needs to return "store"03:40
wallyworldnatefinch: well, it's to be expected isn't it? the model name is a part of the output we need to query03:40
wallyworldnatefinch: the charmstore won't do that I don't think so we'll just need to do it in charmrepo for now03:41
natefinchwallyworld: I'll fix it for now, but I think it's wrong.  Either the charmstore should always populate it, or never populate it.  Not populating it some of the time and expecting everyone to just understand that means origin=store is just plain bad.03:43
wallyworldnatefinch: not disagreeing :-) need to follow up this end. this will get us out of trouble though for now03:43
natefinchwallyworld: trouble for deleting a value that isn't set that only we read....03:48
natefinchhttp://data.whicdn.com/images/104484357/original.gif03:49
wallyworldnatefinch: it will be set in the future, but just not now03:51
natefinchhttps://imgflip.com/readImage?iid=931020003:52
=== JoseeAntonioR is now known as jose
davechen1y        if tools.SHA256 == "" {05:13
davechen1y                logger.Warningf("no SHA-256 hash for %v", tools.SHA256)05:13
davechen1yspot the bug05:13
=== hazmat_ is now known as hazmat
=== bodie__ is now known as bodie_
=== meetingology` is now known as meetingology
=== cargonza_ is now known as cargonza
=== xnox_ is now known as xnox
=== Ursinha_ is now known as Ursinha
=== gsamfira_ is now known as gsamfira
voidspacedimitern: ping10:13
dimiternvoidspace: pong10:14
voidspacedimitern: I have a test for linklayerdevices that no longer fails, and I think the new behaviour is *correct*, but want to run it by you10:15
dimiternvoidspace: sure10:15
voidspacedimitern: there is a test that calling SetLinkLayerDevices with a matching name and provider id to an existing linklayer device fails validation10:15
voidspacedimitern: this used to fail validation because the provider id matched an existing one which was checked before everything eslse10:15
voidspacedimitern: *however*, if the name matches (i.e. the docID is the same) it's treated as an update not an insert10:16
voidspacedimitern: and as the providerid is already set in the original the insert doesn't attempt to change the provider id10:16
voidspacedimitern: so the update works and the test *fails*10:16
voidspacedimitern: however, because this is actually an update not an insert I think the new behaviour is correct10:17
dimiternvoidspace: well, that test won't be relevant anymore10:17
voidspacedimitern: yep, just checking you agreed I could just remove the test10:18
voidspacedimitern: I'm adding a new test that an update that attempts to add a duplicate provider id fails10:18
dimiternvoidspace: because it tests 'failing validation', whereas now we fail as a side-effect of the txn.Ops triggering ErrAborted10:18
dimiternvoidspace: I'd like to have a look at the code still, but so far your plan sounds good10:19
voidspacedimitern: cool10:19
dimiternvoidspace: (when you're ready ofc)10:19
voidspacedimitern: I think all tests now pass10:19
voidspacedimitern: doing a run to check10:19
voidspaceif they pass I'll propose the branch10:19
dimiternvoidspace: sweet!10:19
voidspacedimitern: by the way - current behaviour for an update that attempts to change ProviderID is that the change is silently ignored10:20
voidspacedimitern: shall I change that to an error?10:20
voidspace(this is the current *pre-existing* behaviour - not a change I've made)10:21
dimiternvoidspace: it should only be an error if the original prID != from the new one we're trying to set (and the org is also != "")10:21
voidspacedimitern: yep, exactly10:23
voidspacedimitern: currently if original != "" then the new one is just ignored10:24
voidspacedimitern: ok, all state tests pass - I'll make this change and a test and then propose10:25
dimiternvoidspace: please make sure you also run worker/provisioner/ and apiserver/common/networkingcommon/ tests as well to double check10:27
voidspacedimitern: well, the merge attempt runs everything...10:28
dimiternvoidspace: true :)10:32
davechen1yhttps://github.com/juju/testing/pull/9910:33
voidspacedimitern: http://reviews.vapour.ws/r/4859/10:43
mupBug #1583109 opened: error: private-address not set <juju-core:New> <https://launchpad.net/bugs/1583109>10:44
voidspacedimitern: state, apiserver and provisioner tests pass10:47
dimiternvoidspace: otp, will look in a bit10:53
mupBug #1583109 changed: error: private-address not set <juju-core:New> <https://launchpad.net/bugs/1583109>10:59
mupBug #1583109 opened: error: private-address not set <juju-core:New> <https://launchpad.net/bugs/1583109>11:08
dimiternvoidspace: reviewed11:12
jamespagedimitern, not sure whether you have cycles but that bug ^^ is killing our CI atm...11:27
dimiternjamespage: oh, ok I'll have a look in a bit11:28
jamespagedimitern, thankyou11:28
dimiternjamespage: it looks like the machine hosting keystone/0 did not start ok so that's why the unit had no private address11:37
jamespagedimitern, nope - the machine did start ok11:37
jamespageits running the install hook when that happens11:37
jamespagedimitern, I have the console output for the machine - one sec11:38
dimiternjamespage: I can see machine "7" stuck in "pending" from the linked paste of juju status11:38
jamespagedimitern, I think that's the effect the error has...11:38
jamespagedimitern, https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-cinder/317913/1/2016-05-18_09-29-55/test_charm_amulet_full/logs/cinder-0-nova-console-1.bz211:39
jamespagethats the console log from the machine11:39
dimiternjamespage: that looks like the log for machine-1 do you have the log of machine-7 from that same run?11:40
jamespagedimitern, sorry crossed tests - that was from a different failed run11:41
jamespagedimitern, https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline/openstack/charm-ceph/317910/1/2016-05-18_08-48-05/test_charm_amulet_smoke/logs/keystone-0-nova-console-7.bz211:42
dimiternjamespage: I'm looking at https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline/openstack/charm-ceph/317910/1/2016-05-18_08-48-05/test_charm_amulet_smoke/juju-stat-yaml-collect.txt11:42
jamespagedimitern, https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline/openstack/charm-ceph/317910/1/2016-05-18_08-48-05/index.html11:42
dimiternjamespage: ah. ok11:42
jamespagefor the full index of data11:42
jamespagedimitern,11:43
jamespagemachine-7[3059]: 2016-05-18 09:13:45 INFO juju.worker runner.go:275 stopped "machiner", err: setting machine addresses: cannot set machine addresses of machine 7: state changing too quickly; try again soon11:43
jamespagemachine-7[3059]: 2016-05-18 09:13:45 DEBUG juju.worker runner.go:203 "machiner" done: setting machine addresses: cannot set machine addresses of machine 7: state changing too quickly; try again soon11:43
jamespagemachine-7[3059]: 2016-05-18 09:13:45 ERROR juju.worker runner.go:223 exited "machiner": setting machine addresses: cannot set machine addresses of machine 7: state changing too quickly; try again soon11:43
jamespagemachine-7[3059]: 2016-05-18 09:13:45 INFO juju.worker runner.go:261 restarting "machiner" in 3s11:43
jamespageseems symptomatic of when this happens11:43
dimiternjamespage: right! that I know about11:44
jamespagedimitern, one might cause the other?11:44
dimiternjamespage: unfortunately I wasn't able to reproduce it at will, seems quite intermittent11:44
jamespagedimitern, intermittent - yes11:44
jamespagedimitern, but I hit this three times today - bear in mind we're spinning 1000's of units of charms a day...11:44
dimiternjamespage: indeed - that error causes the MA to bounce repeatedly, before setting status to "started", and because setting the addresses failed both the machine and unit will lack a private address11:45
dimiternjamespage: well, it will help to collect the machine-0 logs with logging-config='<root>=TRACE' when it happens11:46
jamespagebeisner, can we turn that on?11:46
jamespage^^11:46
dimiternjamespage: beisner: juju set-env logging-config='<root>=TRACE'11:47
dimitern(assuming 1.25)11:47
beisnerdimitern, will enable that for all future jobs11:48
dimiternjamespage: that will produce tons of extra logs, beware, but will also include details down to the actual mgo transactions11:48
beisnerwe'll need to turn it back off as soon as we figure out what's going on11:48
dimiternbeisner: cheers! with your setup it should be easy to catch why it happens, however intermittent it might be11:49
beisnerdimitern, we have enough activity that we're seeing it quite often actually11:49
beisnerdimitern, with each deploy being 10-25 units, chances are, 1 of those units will have this happen on about half the jobs.11:49
beisnerso we should be able to repro with logs in short order11:49
dimiternbeisner, jamespage: when it happens again, machine-0.log (where the apiserver is) and machine-X.log (the one "pending") will both be very useful11:50
beisnerdimitern, our bot can't automatically grab the log from machine-x because the agent is awol, it has no address, and we can't ssh into it.11:50
beisnerso that will take a manual reproduction11:50
dimiternbeisner: well, can you get to the juju's machine-0.log ?11:51
beisnerdimitern, definitely11:51
dimiternbeisner: sweet! that's the important one actually (where the txns are logged)11:51
=== \b is now known as benonsoftware
beisnerjamespage, dimitern - hrm.  this will be extremely tricky as we can't set-env logging until after bootstrap, but bootstrap is done by amulet, after our script has left the building.11:58
beisnerdimitern, is there an environments.yaml way to do this?11:58
dimiternbeisner: sure, add `logging-config: '<root>=TRACE'` to the env.yaml12:00
beisnerdimitern, ok cool  juju run sed fu for the win.  we already had that =DEBUG :-)    all set here jamespage12:05
dimiternbeisner: thanks!12:10
dimiternbeisner, jamespage: please, let me know when you repro the issue with the extra logging12:10
* dimitern steps out for ~1h12:46
mupBug #1583170 opened: `juju help placement` does not exist <juju-core:New> <https://launchpad.net/bugs/1583170>13:45
OerHeks#1583170 seems like #158094613:47
mupBug #1583170: `juju help placement` does not exist <juju-core:New> <https://launchpad.net/bugs/1583170>13:47
mupBug #1580946: Juju 2 help commands for constraints or  placement return ERROR unknown command <helpdocs> <juju-core:Fix Committed by cherylj> <https://launchpad.net/bugs/1580946>13:47
tvansteenburghOerHeks: definitely. i didn't see that13:54
OerHeksi got distracted by juju/juju213:55
mupBug #1583170 changed: `juju help placement` does not exist <juju-core:Invalid> <https://launchpad.net/bugs/1583170>14:15
alexisbdimitern, ping14:49
dimiternalexisb: hey14:51
* perrito666 returns from the dead15:04
lazyPowerwb zombie perrito66615:06
perrito666lazyPower: tx, hey are those brains you have there?15:06
lazyPowerindeed, have some seasoning salt15:07
lazyPoweryou're going to inherit all my charms once you eat my brains though... i setup a deadman switch. #yolo15:07
* perrito666 eats a cracker15:08
lazyPower;) i thought that might dissuade you15:10
=== tasdomas` is now known as tasdomas
natefinchkatco, ericsnow: oh yeah, I also did the rename admin model to controller, but it was put on hold.16:11
katconatefinch: mark doesn't want it anymore?16:12
natefinchkatco: not sure. Cheryl posted to the PR "Please don't land this just yet. We're still getting feedback on this requested change."16:12
katconatefinch: k, thanks for doing that anyway16:12
natefinchkatco: I should probably land all but the actually constant name, and then all we'd need to do is change the constant in another PR :)16:13
natefinchkatco: but I'll wait for now, see how it shakes out16:14
katconatefinch: hey that's a good idea16:14
dimiternbabbageclunk, voidspace, dooferlad: anyone still around? I'd appreciate a review on this (mostly removals of legacy/obsolete/unused code): http://reviews.vapour.ws/r/4865/16:18
dimiternthat's a prerequisite to fixing bug 157484416:18
mupBug #1574844: juju2 gives ipv6 address for one lxd, rabbit doesn't appreciate it. <conjure> <juju-release-support> <landscape> <lxd-provider> <juju-core:In Progress by dimitern> <rabbitmq-server (Juju Charms Collection):New> <https://launchpad.net/bugs/1574844>16:18
babbageclunkSure - looking now16:18
natefinchkatco: here you go: branched, then put controller model name back to what it was before: https://github.com/juju/juju/pull/542816:19
dimiternbabbageclunk: thanks!16:19
natefinchericsnow: this is your weekly notification that reviewboard is broken16:19
voidspacedimitern: I'm still around16:23
voidspacedimitern: ah, babbageclunk beat me to it16:23
ericsnownatefinch: looks like it's just you <wink>16:24
dimiternvoidspace: I'm sure you'd like it though :)16:24
natefinchericsnow: weird: https://github.com/juju/juju/pull/542816:24
ericsnownatefinch: yeah, looking at it16:24
babbageclunkdimitern: reviewed, looks great!16:28
dimiternbabbageclunk: awesome, tyvm!16:28
babbageclunkdimitern: (Well, there was one comment.)16:30
babbageclunkdimitern: In exchange, can you explain to me how watchers work? :)16:31
dimiternbabbageclunk: sure, I can try :)16:31
babbageclunkdimitern: So the remaining failures I have of my changed code against mongo2.4 are watchers.16:32
dimiternbabbageclunk: basically any change to a collection as a result of runing []txn.Op is reported by watchers16:32
dimiternbabbageclunk: and the changes (IIRC) are detect by looking into the txnLog16:32
dimiterndetected*16:33
babbageclunkdimitern: ok, so they're goroutines that watch txns.log with specific filter criteria?16:33
dimiternbabbageclunk: yeah, more or less - the nitty gritty low level details are in state/watcher/ (rather than state/watcher.go)16:34
babbageclunkdimitern: Hmm. I clear out txns.log (actually, since it's a capped collection I drop and recreate it).16:35
babbageclunkdimitern: ...between tests I mean.16:35
voidspacedimitern: hah, yeah - looks good16:36
dimiternbabbageclunk: there's also the "presence" thingy - which is related to watchers, but I'm less familiar with it (can't remember whether it was tracking active agent connections or life cycle changes alive->dying->dead in a collection)16:36
voidspacedimitern: can you really remove stuff from the agent/format-1.18.go though?16:37
dimiternbabbageclunk: there's more to txns than the log, there are a few other internal collections used to track txn state, what was applied, whether it's being applied or aborted, etc (something called "stash" IIRC)16:37
dimiternvoidspace: yeah, I think so - don't let the name mislead you :) that's the current format we're using16:38
voidspacedimitern: ah....16:38
babbageclunkdimitern: Thanks, that gives me some stuff to go on with.16:40
voidspacedimitern: my only worry would be that this might cause IPv6 addresses to leak out unwanted16:42
voidspacedimitern: but as far as I can tell that was a risk anyway16:42
voidspacedimitern: so I don't think this makes it worse16:42
dimiternvoidspace: there are no changes to behavior - the removed code paths deal with preferIPv6=true, but as I pointed out in the PR description, it was hard-coded to false some time ago now16:43
voidspacedimitern: heh, ok16:44
katconatefinch: sorry was otp. lgtm16:44
natefinchkatco: thanks :)16:45
dimiternbabbageclunk: I've fwd you a couple of ML discussions around txns and pruning logs16:46
babbageclunkdimitern: Changing tack a bit - I've got a test that's failing here: https://github.com/juju/juju/blob/master/state/volume_test.go#L35416:53
babbageclunkdimitern: Saying got []string{"0/0", "0/1"}, expected []string{"0/1", "0/2"}16:54
babbageclunkdimitern: What are the values in the change list? Are they units?16:55
dimiternbabbageclunk: ah, that sounds like a sequence is reset to 016:55
dimiternbabbageclunk: so some entities, e.g. machines and units, but also others rely on auto-incremented id coming from the sequences collection16:56
babbageclunkdimitern: Sequences sound like some state that I might be clobbering now that wasn't before?16:56
babbageclunkdimitern: Aha, that sounds like a possibility. I'm going to dump what's in that collection in successful and failing runs.16:57
dimiternbabbageclunk: yeah, it's either that or the test itself relies on volume entities starting from 1 rather than 0?17:01
babbageclunkdimitern: Yeah, those are the values in the assertion.17:03
babbageclunkdimitern: So maybe there's something that makes sure sequence is populated appropriately that I'm blowing away?17:04
dimiternbabbageclunk: I doubt that - but have a look at StorageStateSuiteBase and/or ConnSuite it uses for possible clues17:05
* dimitern hits eod17:12
mupBug #1583274 opened: Openstack base bundle 42 fails to start rabbitmq when deployed with Juju/MAAS 2.0 <cdo-qa> <deploy> <openstack> <juju-core:New> <https://launchpad.net/bugs/1583274>17:31
redirI think I need another coffee before looking at your PRs ericsnow18:07
ericsnowredir: k :)18:08
redirfwereade__: yt?18:09
marcoceppiwallyworld: problem18:17
marcoceppikatco: I've tried to attach a resource18:18
marcoceppiand it's not working18:18
katcomarcoceppi: ok. can you pastebin the steps or something?18:19
marcoceppikatco: http://paste.ubuntu.com/16498534/18:20
katcomarcoceppi: ty18:20
katcomarcoceppi: i think you have to specify the --resource flag for each resource when publishing18:24
katcomarcoceppi: attaching just make the blob available to be referenced18:25
katcomarcoceppi: publishing is the act of coupling the charm in the channel with a revision of a resource that has been attached18:25
katcomarcoceppi: does that make sense?18:28
marcoceppikatco: what's the param for --resource in publish?18:28
katcomarcoceppi: <resource name>-<revision>18:29
katcomarcoceppi: `charm list-resources ~marcoceppi/charm-svg` lists the revisions as -1. that's interesting.18:30
wallyworldkatco: marcoceppi: i noticed the -1 thing too and assumed it was because no resources had been published yet18:33
katcowallyworld: marcoceppi: ah that's exactly what it is. but: charm list-resources ~marcoceppi/charm-svg -c development18:35
katcowallyworld: marcoceppi: also lists -1. because you don't really publish to the development channel, that's just a dumping ground18:35
wallyworldkatco: i get a macaroon error18:36
katcomarcoceppi: can you try: charm publish ~marcoceppi/charm-svg-3 --resource python-jujusvg-1 --resource webapp-118:36
katcowallyworld: what version of the charm tool are you on?18:37
ericsnownatefinch: FYI, looks like you already have a review request up (unpublished?  discarded?) for that commit hash18:37
wallyworldkatco: recent version, casey is checking18:39
mupBug #1255799 changed: juju  installing cloud-tools archive on non-bootstrap nodes <cloud-archive> <manual-provider> <juju-core:Won't Fix> <https://launchpad.net/bugs/1255799>18:40
mupBug #1258132 changed: [manual] bootstrap fails in livecd due to juju-db not starting <manual-provider> <juju-core:Won't Fix> <https://launchpad.net/bugs/1258132>18:40
mupBug #1583304 opened: upload-tools appends instead of increment version numbers <ci> <packaging> <juju-core:Triaged> <https://launchpad.net/bugs/1583304>18:52
marcoceppikatco: thanks! wallyworld it's pushed19:05
wallyworldmarcoceppi: ty, will look. i can't log in to charm store atm though. sigh19:05
marcoceppiwallyworld: sounds like a personal problem?19:05
wallyworldmarcoceppi: maybe, but casey also can't log out19:05
wallyworldso there's something weird happening19:06
redirstory of my life wallyworld19:12
natefinchericsnow: thanks... I really wish reviewboard/the bot would handle that differently.  it's a different PR, it should just get a new review19:18
ericsnownatefinch: agreed19:18
* perrito666 postpones all his meetings because he lost the ability to speak in english for long periods19:20
natefinchlol19:22
natefinchreview anyone? http://reviews.vapour.ws/r/4801/  katco, this is the choose-a-series logic one that you already reviewed once.19:43
katconatefinch: tal19:44
katconatefinch: did you just move the code to different files? hard to see the diff19:47
natefinchkatco: yes, and added comments to the fields in series selector.  Mostly I punted on the supportedSeries changes, because it would complicate this code due to things that are really outside its purview19:48
=== redir is now known as redir_lunch
katconatefinch: understand. ship it19:50
natefinchkatco: thanks19:50
natefinchkatco: huzzah, now the change admin model name to "controller" is a single line change: https://github.com/juju/juju/pull/5419/files19:55
katconatefinch: :) sometiems DRY is a good thing19:56
katcoericsnow: hey, why are we registering the components in our tests? e.g. cmd/juju/service/bundle_resource_test.go ?20:25
ericsnowkatco: because of state and the full-stack testing20:26
katcoericsnow: can you elaborate a bit? fwereade__ got me curious20:27
ericsnowkatco: if you try to do resource-related stuff in state and resources haven't been registered in state then it blows up20:28
ericsnowkatco: some of our full-stack tests do resource-related stuff (e.g. bundle_resource_test.go)20:28
katcoericsnow: ah, so some of the existing tests, which are full stack, deal with state which expects resources to be there. therefore it must be registered.20:30
ericsnowkatco: yep20:30
katcofwereade__: there is your answer. we are not in a final state ^^20:30
katcofwereade__: this is a mid-step solution20:31
=== redir_lunch is now known as redir
perrito666anastasiamac_: redir hey, I can talk for more than a few mins, ill miss standup22:47
perrito666I got restore working, now working on the tests, cheers22:47
redirhope you feel better RSN perrito66622:47
anastasiamac_perrito666: well done with restore \o/ get better :D22:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!