/srv/irclogs.ubuntu.com/2016/06/02/#juju-dev.txt

menn0alexisb: no worries. i'm in the hangout when you're ready00:00
wallyworldredir: can you pop back into the standup hangout?00:10
thumperaxw: just looking at migrating the storage collections00:50
thumperaxw: which order should I do them?00:51
thumperwhat's the dependency tree like?00:51
thumperaxw: storageC first?00:52
* thumper thinks block devices first00:56
axwthumper: I'd do block devices, volumes, volume attachments, filesystems, filesystem attachments, storage, storage attachments00:58
thumperaxw: ta00:58
natefinchahhhhhhhhh01:05
natefinchno wonder my logging isn't coming through: reconfiguring logging from "<root>=DEBUG" to "<root>=WARNING;unit=DEBUG"01:05
perrito666oghh, we suck at naming things01:15
perrito666c.client.Do, really? the one thing that is not a verb and the method is called Do01:15
natefinchperrito666: https://golang.org/pkg/net/http/#Client.Do01:19
natefinchperrito666: should be called send01:20
natefinchthumper: where do we store the logging config on the server?  I'm debugging a system where the juju client doesn't work, but the log level is too high for what I need.  I didn't see it in the agent.conf01:23
perrito666natefinch: lol, seems like a widespread bad practice01:37
menn0natefinch: you want to change the log level of the client or the server?01:40
menn0natefinch: to change the logging config for a model:01:42
menn0juju set-model-config logging-config='<root>=DEBUG'01:43
menn0natefinch: for the client:01:43
menn0JUJU_LOGGING_CONFIG='<root>=DEBUG' juju --debug ...01:44
menn0the specific logging config is up to you of course01:44
natefinchwallyworld: https://github.com/crewjam/rfc542401:49
thumpernatefinch: it is stored in the config01:56
thumperfor a model01:56
natefinchthumper: where is the config stored?02:01
natefinchthumper: in the DB?02:01
thumperenvirons/config/config.go02:02
davecheneythumper: sorry i missed your ping02:46
davecheneyi was on another call02:46
davecheneyand then it scrolled off the screen02:47
davecheneyTHE RACE BUILD IS GREEN! http://data.vapour.ws/juju-ci/products/version-4018/run-unit-tests-race/build-1518/consoleText02:56
davecheneyoh02:57
davecheneyit's not02:57
thumperheh02:57
davecheney+ echo 'Race detection compliancy is not supported by 1.25.6'02:57
davecheneyRace detection compliancy is not supported by 1.25.602:57
davecheneyit's jus been disabled ...02:57
davecheneycompliancy is not a word people02:57
thumperdavecheney: I've sorted my issue any way02:59
davecheneycool02:59
davecheneyglad I could help03:00
mupBug #1588135 opened: worker/terminationworker: test timeout during race build <juju-core:New> <https://launchpad.net/bugs/1588135>03:07
mupBug #1588137 opened: cmd/jujud/agent: incorrect use a sync.Waitgroup <juju-core:New> <https://launchpad.net/bugs/1588137>03:10
menn0thumper, wallyworld, axw, anastasiamac : two of you please: http://reviews.vapour.ws/r/4963/03:11
wallyworldok03:11
axwmenn0: looking03:11
menn0thanks03:12
* thumper rages03:12
thumperdoer?03:13
thumperreally?03:13
thumperI'm going to doer the next person that adds 'er' on to things it shouldn't be on03:13
davecheneythump-er03:16
davecheneythumper: this bug is bad, https://launchpad.net/bugs/158813703:16
mupBug #1588137: cmd/jujud/agent: incorrect use a sync.Waitgroup <juju-core:New> <https://launchpad.net/bugs/1588137>03:16
axwis she a doer, eh, eh03:18
axwmenn0: LGTM03:18
natefinchlol doer, yeah03:19
natefinchthumper: so.... where is the config stored?  the DB, I presume, since it's per-model?03:28
mupBug #1588143 opened: cmd/juju/controller: send on a closed channel panic <juju-core:New> <https://launchpad.net/bugs/1588143>03:28
thumpernatefinch: in the environ config03:35
thumperyes, the db03:35
thumperpretty sure it is in the modelC03:35
* thumper thinks03:35
thumperno03:35
thumpermaybe03:35
thumpercan't remember03:35
thumperdavecheney: hmm... interesting03:36
thumperdavecheney: we obviously need some way to reject the connection before trying to do wg.Add03:36
davecheneythumper: i have a fix03:37
thumperdavecheney: coolio03:37
davecheneythis is important to fix because if the wg count goes wrong, the apisever doesn't stop and -- tada, tset timeout03:38
davecheneyit's probably a rare race03:38
davecheneyi'm trying ti reproduce it locally with a stress test03:38
davecheneythumper: short version is we need to reverse the order of operations inside the srv.traceRequest method, plus a bit of other cleanup03:38
thumperk03:39
davecheneywhat is o_O is the way the apiserver shuts down when the mongo pinger returns an error03:39
davecheneythere is no other way to do it03:39
davecheneythat I can see03:39
davecheneywhich is kind of crap03:39
davecheneythe mongo pinger should be external tot he apisever and call it's Close method externally03:39
davecheneyrather than being tightly coupled03:40
mupBug #1588147 opened: when i bootstrap whith the option --metadata-source=/root/juju  option,there is not any debug or error info about the  specified the metadata source <juju-core:New> <https://launchpad.net/bugs/1588147>03:40
davecheneythumper: good news, I can repro the failure locally04:01
davecheneytesting fix now04:01
thumperawesome04:01
thumperthat's often the hard bit04:01
davecheneythumper: http://reviews.vapour.ws/r/4964/04:21
davecheney^ careful review appreciated04:21
davecheneyi need to run some more stress tests before I'm confident with this one04:21
natefinchI cannot, for the life of me, figure out why my log statements aren't getting hit.04:28
mupBug #1498081 changed: loopback storage provider not completing successfully <storage> <juju-core:Expired> <https://launchpad.net/bugs/1498081>04:28
thumperdavecheney: looks good to me.04:34
davecheneythumper: thanks, my laptop is currently screeeaming running a stress test04:37
davecheneyi'll submit this later if it looks ok04:37
natefinchwallyworld or anyone else want to tell me what stupid thing I'm doing such that I'm not seeing logging output where I think I should?04:50
wallyworldbe careful what you wish for04:50
wallyworldnatefinch: do you have info?04:52
natefinchwallyworld: I added some log lines, at Error level (although david fixed the log level on the machines, so debug+ is shown).  I'm definitintely running my function since the error message is correct, but I'm not getting the log statements.  I checked the md5 of the jujud I built and the one that's running and they're the same.04:54
wallyworldnatefinch: where do you see the error message if not in the logs?04:55
natefinchwallyworld: what I see is the error message returned from the function that my log messages are in... but the log messages aren't getting triggered, even thouhg one of them is at the very top of that function04:56
wallyworldand you are using debug log?04:57
natefinchwallyworld: I never use debug log.  it's too damn slow.  I'm just tailing the machine log04:57
wallyworldhmmm, have you tried looking in the logsink file also?04:58
natefinchI don't even know what the logsink file is04:59
wallyworldit's what the controllers write out as a backup to the logs in the db04:59
natefinchahh hmm04:59
natefinchwhere is it?05:00
wallyworld/var/log/juju maybe05:00
natefinchwallyworld: oh, I'm a dip... this is 1.25, btw05:01
wallyworldoh, then allmachines.log is relevant right?05:01
natefinchwallyworld: yes05:01
wallyworldand there's nothing in there?05:01
natefinchnot for my log lines, no05:01
wallyworldthis is a machine agent on a worker machine?05:02
natefinchyeah, it's a container on a non-controller machine05:02
wallyworldand you're looking at /vr/log/juju in the container rootfs?05:03
natefinchwallyworld: yep05:03
wallyworldand it's not something that the agent on the host machine f the container logs?05:03
natefinchI replaced the jujud binary for the container, because the container hit the bug we're looking for05:04
wallyworldi think you should have done upgrade-juju to ensure all the binaries are the same05:05
natefinchbe that as it may.... the only thing I really care about is getting a tiny bit more logging on this one machine, and not trying to upgrade an entire openstack deploy before doing so05:06
wallyworldthe only thing i can think of is that the log is not coming from where you think it is05:06
wallyworldand an older binary is not producing it05:07
natefinchI think what I'll do is change the error message that is coming out of it, to make sure it's actually running my code05:08
wallyworldthat's a good step05:08
natefinchalso adding some old fashioned printf, which I think we still redirect to the logs05:10
bradmis there any way to force what IP juju agent uses?  I'm getting it using the wrong ip, and I want to force it to the right one05:13
bradmbasically it seems to be using the numerically lowest one05:13
bradmlooks like bug 146919305:14
mupBug #1469193: juju selects wrong address for API <kvm> <local-provider> <lxc> <network> <sts> <juju-core:Expired> <https://launchpad.net/bugs/1469193>05:14
bradmthis isn't with lxc, but with maas05:14
bradmwell, no local provider, we do have lxcs being spun up05:14
wallyworldbradm: i don't think there's a way to force it,05:19
bradmwallyworld: .. really.05:20
bradmthat's going to suck, its picking an IP from the cloud floating IP range as the juju api server05:20
wallyworldit's not  exactly straighforward - we can't guarntee to know the available ips ahead of time05:21
wallyworldso any solution requires a bit of thought to cover all corner cases05:22
bradmright..05:22
wallyworldbradm: I think network spaces is intended to contribute to a solution, for 2.0 at least05:24
wallyworldyou can bind a relation endpoint to a given nic, or something to that effect i think05:25
bradmthis is getting the wrong IP from the bootstrap node, so the agents don't even start05:25
bradmwill have to hack up something with routing, I guess05:26
natefinchhonestly, using the local provider for anything more than a demo on your laptop is asking for trouble05:27
bradmthis isn't local provider05:28
bradmthis is maas05:28
bradmhopefully using maas isn't asking for trouble :)05:28
natefinchno no :)05:28
natefinchmy mistake :)05:28
bradmthe bug is local provider I think, its the closest thing05:28
bradmwant me to reopen that one?  or a new one?05:29
bradmgiven its a different provider I guess..05:29
natefinchthere's been a bug in the past where we chose the wrong IP address... I think it's been addressed, but possibly only in 2.0?  I'm not sure.  I assume you're on 1.25?05:29
bradmyeah, 1.25.5 for this customer05:29
bradmits picking the IP on br1 for the bootstrap node, not on br005:31
wallyworldbradm: you are best to ask the folks in europe (dimiter, andy) in an hour or so05:31
wallyworldthyey are across this much better than me05:32
bradmcool, cool.05:32
bradmI suspect there's something not quite right with the routing for this network range too, but still..05:32
natefinchwallyworld: somehow my binary is not being run... even though I can confirm the md5 matches the one I built on my laptop.  bizarre.05:33
wallyworldnatefinch: you just copied it across the old one i assume and restarted the jujud service05:34
natefinchwallyworld: yep05:35
wallyworldobviously something went wrong, maybe upgrade-juju --upload-tools is the right thing after all05:35
natefinchgah, I can try05:36
wallyworldnot sure why you're against it05:36
wallyworldi use it a bit in testing, not on openstck though05:36
natefinchgoing through the whole upgrade process... just a lot to go wrong05:37
wallyworldas opposed to doing it by hand and having it fail? :-P05:37
natefinchthe binary is in the right spot and seems to be the one being run :/05:38
natefinchso... how does --upload-tools pick the jujud to upload?05:38
bradmis there anyway to tell a container to retry provisioning?  juju retry-provisioning doesn't seem to support containers05:38
wallyworldbradm: yeah, i don't think you can, i think thatwas just for cloud instances05:39
wallyworldnatefinch: the same way as bootstrap --upload-tools does05:39
bradmbummer.05:39
wallyworldi could be wrong, but that's my recollection05:40
natefinchwallyworld: whoever thought that magically picking some random binary to upload was a good idea?  Why not let the user just specify a path? :/05:40
wallyworldnatefinch: yeah, can't argue there, that was before my time05:41
natefinchwallyworld: now I get to wait to see if this works.05:45
natefinchwallyworld: so far, not looking good.  juju status is not connecting05:45
wallyworldshouldn't take too long05:45
natefinchhey, it finished05:47
natefinchhey, it finished05:47
natefinchhey, it finished05:47
natefinchoops, wrong window, sorry05:47
natefinchwell crap, it used the wrong jujud05:51
natefinchI thought it was supposed to use one that exists in the same directory as the juju that is running, but evidently not.05:51
natefinchI gotta go to bed... I have to be up in ~4.5 hours05:52
natefinchwallyworld: ^05:53
wallyworldnatefinch: it uses the one in the path, not sure05:53
wallyworldtomorrow is anothe day05:54
natefinchwallyworld: ahh, yeah, probably05:54
axwwallyworld: tests are still running, but optimistically send PR: https://github.com/juju/juju/pull/551806:40
wallyworldok06:40
wallyworldaxw: reading the cover description - maybe i misunderstood before. we still need custom image metadata in state, but structured in the image metadata collection, not a simplestreams json blob06:46
axwwallyworld: "Custom image metadata no longer needs to be written to the blob store in Mongo" -- it's no longer in the blob store, but structured metadata is still there06:47
wallyworldaxw: ah, right, sorry, misread it06:47
axwwallyworld: do you have a doc for the rest of what's coming for bootstrap?07:04
* axw bbs07:05
wallyworldaxw: posted to #juju07:05
=== frankban|afk is now known as frankban
axwwallyworld: I'm not there (desktop is off for diagnosis, don't have the auth set up on this laptop), can you email me?07:31
wallyworldsure07:31
mupBug #1588186 opened: reboot-executor does not run in jujud tests <juju-core:Triaged> <https://launchpad.net/bugs/1588186>07:43
axwwallyworld: is it known that HA is broken atm?08:27
axwI see no critical bug blocking master, so I guess not08:28
frobwarebabbageclunk: ping - sync?08:34
babbageclunkfrobware: oops! Jumping into juju-sapphire now08:34
babbageclunkfrobware: oh no - saw the right one08:35
=== rogpeppe1 is now known as rogpeppe
babbageclunkfrobware: in sync now.08:36
babbageclunkfrobware: is it just named that for the pun?08:36
dimiternFYI fwereade just texted me his internet connection is down atm09:01
voidspacedimitern: hangouts playing up but omw09:02
voidspacegah09:04
voidspacenot working - freezing firefox09:04
voidspacedimitern: frobware: dooferlad: babbageclunk: firefox crashed!09:19
dooferladvoidspace: mine died too09:19
dooferladvoidspace: and it is chrome09:19
voidspacenice09:19
babbageclunkvoidspace, dooferlad: google hangouts experimenting with "kill packets"09:34
voidspacebabbageclunk: you jest...09:34
dooferladbabbageclunk: :-)09:34
babbageclunkvoidspace, dooferlad: future iterations expected to have lethal options09:34
dimiternLOL09:34
dimiternthen they'll just send killbots next time09:35
mwhudsonconnection interrupted by ordnance09:39
anastasiamacdooferlad: standup today ?11:31
perrito666morning all11:34
perrito666mongodb is an endless source of joy12:57
natefinchis there a JFDI for upgrade-juju?  --reset-previous-upgrade does not seem to actually do anything14:30
natefinchwallyworld, fwereade, perrito666? ^14:33
natefinchwallyworld: (this is why I don't use upgrade-juju btw - ERROR some agents have not upgraded to the current environment version 1.25.5.2: machine-0-lxc-0, machine-0-lxc-1, machine-0-lxc-2, machine-0-lxc-3, unit-ceph-mon-0, unit-glance-0, unit-landscape-client-3, unit-landscape-client-6, unit-landscape-client-7, unit-ntpmaster-0)14:36
mupBug #1588390 opened: 2.0 beta7: can't bootstrap with vsphere cloud provider - ERROR invalid config: host: expected string, got nothing <oil> <juju-core:New> <https://launchpad.net/bugs/1588390>14:36
perrito666I know very little about upgrade-juju14:39
mupBug #1588390 changed: 2.0 beta7: can't bootstrap with vsphere cloud provider - ERROR invalid config: host: expected string, got nothing <oil> <juju-core:New> <https://launchpad.net/bugs/1588390>14:39
perrito666natefinch: look in the archive for mails from menno on that aspect14:40
perrito666there was a way to kick the chair from under juju to trigger an upgrade14:40
perrito666iirc, it requierd touching mongo14:40
perrito666whoever shortened lxc names, thank you14:43
natefinchheh14:44
mupBug #1588390 opened: 2.0 beta7: can't bootstrap with vsphere cloud provider - ERROR invalid config: host: expected string, got nothing <oil> <juju-core:New> <https://launchpad.net/bugs/1588390>14:46
alexisbperrito666, that was thumper15:09
alexisbmorning/evening all15:09
perrito666alexisb: Ill make sure to thank him15:09
perrito666morning alexisb15:09
mupBug #1588403 opened: Tab completion missing in Juju 2.0 betas <juju-core:New> <https://launchpad.net/bugs/1588403>15:13
natefinchfwereade: is there a way to force upgrade-juju to work even with units in error states?15:33
fwereadenatefinch, hmm, not without surgery that I can think of15:46
fwereadenatefinch, if you can create the upgrade doc I don't think the errors will impede the process15:46
natefinchfwereade: so... all I really want to do is swap out the binary for one particular machine... but somehow when I do so, it is not running the new binary... I think I must be doing something very dumb.15:48
fwereadenatefinch, ah, right: ok, exactly what did you swap out? tools/<version>?15:51
natefinchfwereade: yes15:51
fwereadenatefinch, what's in the tools/<agent> dir?15:52
natefinchfwereade: hang on, sorry... a previous failed juju-upgrade is putting me in a bad state15:54
fwereadenatefinch, (it certainly *used* to just be a symlink to the one in version, but I have a feeling there was some churn there at some point)15:54
fwereadenatefinch, you can make the tools think they're a different version by putting some file next to them, if that helps? you can use that to get everything reporting a consistent version if you have to15:56
natefinchfwereade: ahh.. it is not a symlink15:56
fwereadenatefinch, yay, vague instinct saves the day ;p15:56
natefinchlol, another bug I fixed elsewhere getting hit on this machine.  it's gonna be one of those days16:22
alexisbnatefinch, is this dpb machine?16:28
natefinchalexisb: yes, it's not  a big deal... there were multiple machines experiencing the bug so I just switched to a different one16:32
natefincher experiencing the bug I'm debugging, not the one I already fixed16:33
=== frankban is now known as frankban|afk
alexisbdimitern, ping16:45
mgzcan I get a stamp on reviews.vapour.ws/r/4968 for service-to-application16:51
mupBug #1588446 opened: MAAS bridge script emits "RTNETLINK: file exists" during bootstrap many times <bootstrap> <maas> <network> <juju-core:New for frobware> <https://launchpad.net/bugs/1588446>16:55
mgzocr, or anyone, poke ^17:03
perrito666mgz: shipit17:17
mgzthank you horachan17:17
redirwho knows about user management?18:30
=== redir is now known as redir_lunch
=== redir_lunch is now known as redi
=== redi is now known as redir
natefinchcmars: FYI I am aware of the PRs against npipe... will get to them when I have some time.  Let me know if they become critical for me to look at.19:53
cmarsnatefinch, thanks. i've been hacking away at this with gsamfira. going to try to confirm my latest fix with jujud tests, will let you know19:54
natefinchcmars: glad you're getting help from Gabriel, that windows code is nasty business, and he knows his stuff.19:54
natefinchcmars: feel free to ping me when you think it's all stable19:55
cmarssinzui, what kind of instance would run the windows CI test, run-unit-tests-win2012-amd64 ?20:03
cmarshow many cores would it have?20:03
sinzuicmars: it is a c4.xlarge20:05
alexisbcmars, you are a rockstar thank you!20:07
cmarssinzui, thanks! gsamfira, looks like 4 cores running windows tests20:08
gsamfiratest will probably pass if you set GOMAXPROCS=1...but that is not a fix.20:10
mupBug #1274755 changed: simplestreams test metadata only lists tools for arm and amd64 <arm64> <hs-arm64> <simplestreams> <tech-debt> <juju-core:Won't Fix> <https://launchpad.net/bugs/1274755>20:42
mupBug #1517092 changed: [xenial] libgo panic doing a bootstrap on ARM64 <2.0-count> <arm64> <bootstrap> <xenial> <juju-core:Fix Released> <https://launchpad.net/bugs/1517092>20:42
=== natefinch is now known as natefinch-afk
wallyworldcmars: sorry, that may have been me who removed the model uuid param :-(21:27
cmarswallyworld, no worries21:28
cmarsI really ought to have CI tests to catch this stuff21:30
mupBug #1588542 opened: Juju2 is slow with MAAS2, log shows errors, works anyway <juju-core:New> <https://launchpad.net/bugs/1588542>21:51
alexisbthumper, ping22:00
alexisbaxw, wallyworld ping22:32
wallyworldalexisb: otp with rick22:32
alexisbwallyworld, ack22:32
axwalexisb: pong22:51
alexisbheya axw, I need someone to hash through manual provider planning with me22:51
alexisbdo you have time to chat?22:51
axwalexisb: sure22:52
alexisbhttps://hangouts.google.com/hangouts/_/canonical.com/core-leads-call22:52
alexisbwallyworld, feel free to join us when you are free22:52
perrito666wallyworld: wanna call off the 1:1? ill be there for standup anyway22:53
wallyworldperrito666: sorry :-( just finished with rick, now this one, i'll catch you after standup22:53
perrito666wallyworld: dont worry I forgive you (did you see that restore landed fixing 2 bugs in its way?)22:54
wallyworldperrito666: yay, ty22:54
menn0wallyworld: the charms upload HTTP endpoint seems to require a series argument... this doesn't work for multi-series charms23:08
menn0wallyworld: should this be relaxed?23:08
wallyworldmenn0: yeah23:08
menn0wallyworld: I'm trying to use the endpoint to upload charms during migration23:08
wallyworldgood catch23:09
menn0but because I'm testing with the ubuntu charm it's failing because the charm URL series is blank23:09
wallyworldseries in url should now be optional23:09
menn0ok cool23:10
wallyworldanastasiamac: perrito666: be right there23:16
mupBug #1588559 opened: juju agree should tell the user what to do next <juju-core:New for cmars> <https://launchpad.net/bugs/1588559>23:28
wallyworldcmars:23:52
wallyworld# github.com/juju/juju/cmd/juju/application23:52
wallyworldcmd/juju/application/deploy_test.go:931: unknown metricRegistrationPost field 'ApplicationName' in struct literal23:52
wallyworldcmd/juju/application/deploy_test.go:978: unknown metricRegistrationPost field 'ApplicationName' in struct literal23:52

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!