[00:00] alexisb: no worries. i'm in the hangout when you're ready [00:10] redir: can you pop back into the standup hangout? [00:50] axw: just looking at migrating the storage collections [00:51] axw: which order should I do them? [00:51] what's the dependency tree like? [00:52] axw: storageC first? [00:56] * thumper thinks block devices first [00:58] thumper: I'd do block devices, volumes, volume attachments, filesystems, filesystem attachments, storage, storage attachments [00:58] axw: ta [01:05] ahhhhhhhhh [01:05] no wonder my logging isn't coming through: reconfiguring logging from "=DEBUG" to "=WARNING;unit=DEBUG" [01:15] oghh, we suck at naming things [01:15] c.client.Do, really? the one thing that is not a verb and the method is called Do [01:19] perrito666: https://golang.org/pkg/net/http/#Client.Do [01:20] perrito666: should be called send [01:23] thumper: where do we store the logging config on the server? I'm debugging a system where the juju client doesn't work, but the log level is too high for what I need. I didn't see it in the agent.conf [01:37] natefinch: lol, seems like a widespread bad practice [01:40] natefinch: you want to change the log level of the client or the server? [01:42] natefinch: to change the logging config for a model: [01:43] juju set-model-config logging-config='=DEBUG' [01:43] natefinch: for the client: [01:44] JUJU_LOGGING_CONFIG='=DEBUG' juju --debug ... [01:44] the specific logging config is up to you of course [01:49] wallyworld: https://github.com/crewjam/rfc5424 [01:56] natefinch: it is stored in the config [01:56] for a model [02:01] thumper: where is the config stored? [02:01] thumper: in the DB? [02:02] environs/config/config.go [02:46] thumper: sorry i missed your ping [02:46] i was on another call [02:47] and then it scrolled off the screen [02:56] THE RACE BUILD IS GREEN! http://data.vapour.ws/juju-ci/products/version-4018/run-unit-tests-race/build-1518/consoleText [02:57] oh [02:57] it's not [02:57] heh [02:57] + echo 'Race detection compliancy is not supported by 1.25.6' [02:57] Race detection compliancy is not supported by 1.25.6 [02:57] it's jus been disabled ... [02:57] compliancy is not a word people [02:59] davecheney: I've sorted my issue any way [02:59] cool [03:00] glad I could help [03:07] Bug #1588135 opened: worker/terminationworker: test timeout during race build [03:10] Bug #1588137 opened: cmd/jujud/agent: incorrect use a sync.Waitgroup [03:11] thumper, wallyworld, axw, anastasiamac : two of you please: http://reviews.vapour.ws/r/4963/ [03:11] ok [03:11] menn0: looking [03:12] thanks [03:12] * thumper rages [03:13] doer? [03:13] really? [03:13] I'm going to doer the next person that adds 'er' on to things it shouldn't be on [03:16] thump-er [03:16] thumper: this bug is bad, https://launchpad.net/bugs/1588137 [03:16] Bug #1588137: cmd/jujud/agent: incorrect use a sync.Waitgroup [03:18] is she a doer, eh, eh [03:18] menn0: LGTM [03:19] lol doer, yeah [03:28] thumper: so.... where is the config stored? the DB, I presume, since it's per-model? [03:28] Bug #1588143 opened: cmd/juju/controller: send on a closed channel panic [03:35] natefinch: in the environ config [03:35] yes, the db [03:35] pretty sure it is in the modelC [03:35] * thumper thinks [03:35] no [03:35] maybe [03:35] can't remember [03:36] davecheney: hmm... interesting [03:36] davecheney: we obviously need some way to reject the connection before trying to do wg.Add [03:37] thumper: i have a fix [03:37] davecheney: coolio [03:38] this is important to fix because if the wg count goes wrong, the apisever doesn't stop and -- tada, tset timeout [03:38] it's probably a rare race [03:38] i'm trying ti reproduce it locally with a stress test [03:38] thumper: short version is we need to reverse the order of operations inside the srv.traceRequest method, plus a bit of other cleanup [03:39] k [03:39] what is o_O is the way the apiserver shuts down when the mongo pinger returns an error [03:39] there is no other way to do it [03:39] that I can see [03:39] which is kind of crap [03:39] the mongo pinger should be external tot he apisever and call it's Close method externally [03:40] rather than being tightly coupled [03:40] Bug #1588147 opened: when i bootstrap whith the option --metadata-source=/root/juju option,there is not any debug or error info about the specified the metadata source [04:01] thumper: good news, I can repro the failure locally [04:01] testing fix now [04:01] awesome [04:01] that's often the hard bit [04:21] thumper: http://reviews.vapour.ws/r/4964/ [04:21] ^ careful review appreciated [04:21] i need to run some more stress tests before I'm confident with this one [04:28] I cannot, for the life of me, figure out why my log statements aren't getting hit. [04:28] Bug #1498081 changed: loopback storage provider not completing successfully

[04:34] davecheney: looks good to me. [04:37] thumper: thanks, my laptop is currently screeeaming running a stress test [04:37] i'll submit this later if it looks ok [04:50] wallyworld or anyone else want to tell me what stupid thing I'm doing such that I'm not seeing logging output where I think I should? [04:50] be careful what you wish for [04:52] natefinch: do you have info? [04:54] wallyworld: I added some log lines, at Error level (although david fixed the log level on the machines, so debug+ is shown). I'm definitintely running my function since the error message is correct, but I'm not getting the log statements. I checked the md5 of the jujud I built and the one that's running and they're the same. [04:55] natefinch: where do you see the error message if not in the logs? [04:56] wallyworld: what I see is the error message returned from the function that my log messages are in... but the log messages aren't getting triggered, even thouhg one of them is at the very top of that function [04:57] and you are using debug log? [04:57] wallyworld: I never use debug log. it's too damn slow. I'm just tailing the machine log [04:58] hmmm, have you tried looking in the logsink file also? [04:59] I don't even know what the logsink file is [04:59] it's what the controllers write out as a backup to the logs in the db [04:59] ahh hmm [05:00] where is it? [05:00] /var/log/juju maybe [05:01] wallyworld: oh, I'm a dip... this is 1.25, btw [05:01] oh, then allmachines.log is relevant right? [05:01] wallyworld: yes [05:01] and there's nothing in there? [05:01] not for my log lines, no [05:02] this is a machine agent on a worker machine? [05:02] yeah, it's a container on a non-controller machine [05:03] and you're looking at /vr/log/juju in the container rootfs? [05:03] wallyworld: yep [05:03] and it's not something that the agent on the host machine f the container logs? [05:04] I replaced the jujud binary for the container, because the container hit the bug we're looking for [05:05] i think you should have done upgrade-juju to ensure all the binaries are the same [05:06] be that as it may.... the only thing I really care about is getting a tiny bit more logging on this one machine, and not trying to upgrade an entire openstack deploy before doing so [05:06] the only thing i can think of is that the log is not coming from where you think it is [05:07] and an older binary is not producing it [05:08] I think what I'll do is change the error message that is coming out of it, to make sure it's actually running my code [05:08] that's a good step [05:10] also adding some old fashioned printf, which I think we still redirect to the logs [05:13] is there any way to force what IP juju agent uses? I'm getting it using the wrong ip, and I want to force it to the right one [05:13] basically it seems to be using the numerically lowest one [05:14] looks like bug 1469193 [05:14] Bug #1469193: juju selects wrong address for API

[05:14] this isn't with lxc, but with maas [05:14] well, no local provider, we do have lxcs being spun up [05:19] bradm: i don't think there's a way to force it, [05:20] wallyworld: .. really. [05:20] that's going to suck, its picking an IP from the cloud floating IP range as the juju api server [05:21] it's not exactly straighforward - we can't guarntee to know the available ips ahead of time [05:22] so any solution requires a bit of thought to cover all corner cases [05:22] right.. [05:24] bradm: I think network spaces is intended to contribute to a solution, for 2.0 at least [05:25] you can bind a relation endpoint to a given nic, or something to that effect i think [05:25] this is getting the wrong IP from the bootstrap node, so the agents don't even start [05:26] will have to hack up something with routing, I guess [05:27] honestly, using the local provider for anything more than a demo on your laptop is asking for trouble [05:28] this isn't local provider [05:28] this is maas [05:28] hopefully using maas isn't asking for trouble :) [05:28] no no :) [05:28] my mistake :) [05:28] the bug is local provider I think, its the closest thing [05:29] want me to reopen that one? or a new one? [05:29] given its a different provider I guess.. [05:29] there's been a bug in the past where we chose the wrong IP address... I think it's been addressed, but possibly only in 2.0? I'm not sure. I assume you're on 1.25? [05:29] yeah, 1.25.5 for this customer [05:31] its picking the IP on br1 for the bootstrap node, not on br0 [05:31] bradm: you are best to ask the folks in europe (dimiter, andy) in an hour or so [05:32] thyey are across this much better than me [05:32] cool, cool. [05:32] I suspect there's something not quite right with the routing for this network range too, but still.. [05:33] wallyworld: somehow my binary is not being run... even though I can confirm the md5 matches the one I built on my laptop. bizarre. [05:34] natefinch: you just copied it across the old one i assume and restarted the jujud service [05:35] wallyworld: yep [05:35] obviously something went wrong, maybe upgrade-juju --upload-tools is the right thing after all [05:36] gah, I can try [05:36] not sure why you're against it [05:36] i use it a bit in testing, not on openstck though [05:37] going through the whole upgrade process... just a lot to go wrong [05:37] as opposed to doing it by hand and having it fail? :-P [05:38] the binary is in the right spot and seems to be the one being run :/ [05:38] so... how does --upload-tools pick the jujud to upload? [05:38] is there anyway to tell a container to retry provisioning? juju retry-provisioning doesn't seem to support containers [05:39] bradm: yeah, i don't think you can, i think thatwas just for cloud instances [05:39] natefinch: the same way as bootstrap --upload-tools does [05:39] bummer. [05:40] i could be wrong, but that's my recollection [05:40] wallyworld: whoever thought that magically picking some random binary to upload was a good idea? Why not let the user just specify a path? :/ [05:41] natefinch: yeah, can't argue there, that was before my time [05:45] wallyworld: now I get to wait to see if this works. [05:45] wallyworld: so far, not looking good. juju status is not connecting [05:45] shouldn't take too long [05:47] hey, it finished [05:47] hey, it finished [05:47] hey, it finished [05:47] oops, wrong window, sorry [05:51] well crap, it used the wrong jujud [05:51] I thought it was supposed to use one that exists in the same directory as the juju that is running, but evidently not. [05:52] I gotta go to bed... I have to be up in ~4.5 hours [05:53] wallyworld: ^ [05:53] natefinch: it uses the one in the path, not sure [05:54] tomorrow is anothe day [05:54] wallyworld: ahh, yeah, probably [06:40] wallyworld: tests are still running, but optimistically send PR: https://github.com/juju/juju/pull/5518 [06:40] ok [06:46] axw: reading the cover description - maybe i misunderstood before. we still need custom image metadata in state, but structured in the image metadata collection, not a simplestreams json blob [06:47] wallyworld: "Custom image metadata no longer needs to be written to the blob store in Mongo" -- it's no longer in the blob store, but structured metadata is still there [06:47] axw: ah, right, sorry, misread it [07:04] wallyworld: do you have a doc for the rest of what's coming for bootstrap? [07:05] * axw bbs [07:05] axw: posted to #juju === frankban|afk is now known as frankban [07:31] wallyworld: I'm not there (desktop is off for diagnosis, don't have the auth set up on this laptop), can you email me? [07:31] sure [07:43] Bug #1588186 opened: reboot-executor does not run in jujud tests [08:27] wallyworld: is it known that HA is broken atm? [08:28] I see no critical bug blocking master, so I guess not [08:34] babbageclunk: ping - sync? [08:34] frobware: oops! Jumping into juju-sapphire now [08:35] frobware: oh no - saw the right one === rogpeppe1 is now known as rogpeppe [08:36] frobware: in sync now. [08:36] frobware: is it just named that for the pun? [09:01] FYI fwereade just texted me his internet connection is down atm [09:02] dimitern: hangouts playing up but omw [09:04] gah [09:04] not working - freezing firefox [09:19] dimitern: frobware: dooferlad: babbageclunk: firefox crashed! [09:19] voidspace: mine died too [09:19] voidspace: and it is chrome [09:19] nice [09:34] voidspace, dooferlad: google hangouts experimenting with "kill packets" [09:34] babbageclunk: you jest... [09:34] babbageclunk: :-) [09:34] voidspace, dooferlad: future iterations expected to have lethal options [09:34] LOL [09:35] then they'll just send killbots next time [09:39] connection interrupted by ordnance [11:31] dooferlad: standup today ? [11:34] morning all [12:57] mongodb is an endless source of joy [14:30] is there a JFDI for upgrade-juju? --reset-previous-upgrade does not seem to actually do anything [14:33] wallyworld, fwereade, perrito666? ^ [14:36] wallyworld: (this is why I don't use upgrade-juju btw - ERROR some agents have not upgraded to the current environment version 1.25.5.2: machine-0-lxc-0, machine-0-lxc-1, machine-0-lxc-2, machine-0-lxc-3, unit-ceph-mon-0, unit-glance-0, unit-landscape-client-3, unit-landscape-client-6, unit-landscape-client-7, unit-ntpmaster-0) [14:36] Bug #1588390 opened: 2.0 beta7: can't bootstrap with vsphere cloud provider - ERROR invalid config: host: expected string, got nothing

[14:39] I know very little about upgrade-juju [14:39] Bug #1588390 changed: 2.0 beta7: can't bootstrap with vsphere cloud provider - ERROR invalid config: host: expected string, got nothing

[14:40] natefinch: look in the archive for mails from menno on that aspect [14:40] there was a way to kick the chair from under juju to trigger an upgrade [14:40] iirc, it requierd touching mongo [14:43] whoever shortened lxc names, thank you [14:44] heh [14:46] Bug #1588390 opened: 2.0 beta7: can't bootstrap with vsphere cloud provider - ERROR invalid config: host: expected string, got nothing

[15:09] perrito666, that was thumper [15:09] morning/evening all [15:09] alexisb: Ill make sure to thank him [15:09] morning alexisb [15:13] Bug #1588403 opened: Tab completion missing in Juju 2.0 betas [15:33] fwereade: is there a way to force upgrade-juju to work even with units in error states? [15:46] natefinch, hmm, not without surgery that I can think of [15:46] natefinch, if you can create the upgrade doc I don't think the errors will impede the process [15:48] fwereade: so... all I really want to do is swap out the binary for one particular machine... but somehow when I do so, it is not running the new binary... I think I must be doing something very dumb. [15:51] natefinch, ah, right: ok, exactly what did you swap out? tools/? [15:51] fwereade: yes [15:52] natefinch, what's in the tools/ dir? [15:54] fwereade: hang on, sorry... a previous failed juju-upgrade is putting me in a bad state [15:54] natefinch, (it certainly *used* to just be a symlink to the one in version, but I have a feeling there was some churn there at some point) [15:56] natefinch, you can make the tools think they're a different version by putting some file next to them, if that helps? you can use that to get everything reporting a consistent version if you have to [15:56] fwereade: ahh.. it is not a symlink [15:56] natefinch, yay, vague instinct saves the day ;p [16:22] lol, another bug I fixed elsewhere getting hit on this machine. it's gonna be one of those days [16:28] natefinch, is this dpb machine? [16:32] alexisb: yes, it's not a big deal... there were multiple machines experiencing the bug so I just switched to a different one [16:33] er experiencing the bug I'm debugging, not the one I already fixed === frankban is now known as frankban|afk [16:45] dimitern, ping [16:51] can I get a stamp on reviews.vapour.ws/r/4968 for service-to-application [16:55] Bug #1588446 opened: MAAS bridge script emits "RTNETLINK: file exists" during bootstrap many times

[17:03] ocr, or anyone, poke ^ [17:17] mgz: shipit [17:17] thank you horachan [18:30] who knows about user management? === redir is now known as redir_lunch === redir_lunch is now known as redi === redi is now known as redir [19:53] cmars: FYI I am aware of the PRs against npipe... will get to them when I have some time. Let me know if they become critical for me to look at. [19:54] natefinch, thanks. i've been hacking away at this with gsamfira. going to try to confirm my latest fix with jujud tests, will let you know [19:54] cmars: glad you're getting help from Gabriel, that windows code is nasty business, and he knows his stuff. [19:55] cmars: feel free to ping me when you think it's all stable [20:03] sinzui, what kind of instance would run the windows CI test, run-unit-tests-win2012-amd64 ? [20:03] how many cores would it have? [20:05] cmars: it is a c4.xlarge [20:07] cmars, you are a rockstar thank you! [20:08] sinzui, thanks! gsamfira, looks like 4 cores running windows tests [20:10] test will probably pass if you set GOMAXPROCS=1...but that is not a fix. [20:42] Bug #1274755 changed: simplestreams test metadata only lists tools for arm and amd64

[20:42] Bug #1517092 changed: [xenial] libgo panic doing a bootstrap on ARM64 <2.0-count>

=== natefinch is now known as natefinch-afk [21:27] cmars: sorry, that may have been me who removed the model uuid param :-( [21:28] wallyworld, no worries [21:30] I really ought to have CI tests to catch this stuff [21:51] Bug #1588542 opened: Juju2 is slow with MAAS2, log shows errors, works anyway [22:00] thumper, ping [22:32] axw, wallyworld ping [22:32] alexisb: otp with rick [22:32] wallyworld, ack [22:51] alexisb: pong [22:51] heya axw, I need someone to hash through manual provider planning with me [22:51] do you have time to chat? [22:52] alexisb: sure [22:52] https://hangouts.google.com/hangouts/_/canonical.com/core-leads-call [22:52] wallyworld, feel free to join us when you are free [22:53] wallyworld: wanna call off the 1:1? ill be there for standup anyway [22:53] perrito666: sorry :-( just finished with rick, now this one, i'll catch you after standup [22:54] wallyworld: dont worry I forgive you (did you see that restore landed fixing 2 bugs in its way?) [22:54] perrito666: yay, ty [23:08] wallyworld: the charms upload HTTP endpoint seems to require a series argument... this doesn't work for multi-series charms [23:08] wallyworld: should this be relaxed? [23:08] menn0: yeah [23:08] wallyworld: I'm trying to use the endpoint to upload charms during migration [23:09] good catch [23:09] but because I'm testing with the ubuntu charm it's failing because the charm URL series is blank [23:09] series in url should now be optional [23:10] ok cool [23:16] anastasiamac: perrito666: be right there [23:28] Bug #1588559 opened: juju agree should tell the user what to do next [23:52] cmars: [23:52] # github.com/juju/juju/cmd/juju/application [23:52] cmd/juju/application/deploy_test.go:931: unknown metricRegistrationPost field 'ApplicationName' in struct literal [23:52] cmd/juju/application/deploy_test.go:978: unknown metricRegistrationPost field 'ApplicationName' in struct literal