| wallyworld | cherylj: so, we'll need to get that fix for bug 1567170 fixed | 00:00 |
|---|---|---|
| mup | Bug #1567170: Disallow upgrading with --upload-tools for hosted models <upgrade-juju> <juju-core:In Progress by cox-katherine-e> <https://launchpad.net/bugs/1567170> | 00:00 |
| cherylj | yes, it landed in next, so we have some time | 00:00 |
| wallyworld | ah phew ok | 00:00 |
| mwhudson | wallyworld: so juju is looking for mongodump in /usr/lib/juju/bin/mongodump? | 00:02 |
| wallyworld | mwhudson: from memory, yreah | 00:02 |
| mwhudson | ERROR while creating backup archive: while dumping juju state database: error dumping databases: error executing "/usr/lib/juju/bin/mongodump": 2016-04-13T00:02:21.072+0000error parsing command line options: --dbpath and related flags are not supported in 3.0 tools.; See http://dochub.mongodb.org/core/tools-dbpath-deprecated for more information; 2016-04-13T00:02:21.072+0000try 'mongodump --help' for more information; | 00:02 |
| mwhudson | sounds like juju needs moar fixing | 00:02 |
| mwhudson | but well no reason not to upload my package | 00:03 |
| wallyworld | mwhudson: yeah, juju is currently hard coded to look in the mongo 2.4 binary path for mongodump | 00:03 |
| perrito666 | wallyworld: I fixed my patch, can you look at it so I can merge it after dinner? | 00:03 |
| wallyworld | perrito666: will do | 00:03 |
| wallyworld | mwhudson: it should be a quick fix to sort out that mongodump path, i'll do that today | 00:04 |
| mwhudson | wallyworld: that was after i copied the 3.2 tools to /usr/lib/juju/bin/ | 00:04 |
| mwhudson | wallyworld: looks like the command line needs to change too | 00:04 |
| wallyworld | oh right, yes, i didn't read the error | 00:04 |
| wallyworld | damn it | 00:04 |
| mwhudson | BTW MIR for this is going to be a doozy | 00:04 |
| wallyworld | yes | 00:05 |
| mwhudson | but that's not today's problem either | 00:05 |
| wallyworld | mwhudson: so we have today to get a blessed beta4 for includsion in xenial main | 00:05 |
| wallyworld | is my understanding | 00:05 |
| mwhudson | wallyworld: well i guess Someone (tm) should get to working filing MIRs for the deps | 00:06 |
| wallyworld | mwhudson: yeah, i must admit i have nfi about the process with all that | 00:06 |
| wallyworld | it is being driven by others | 00:06 |
| wallyworld | mwhudson: but i thought we had permission to put mongo packages needed for juju into main | 00:07 |
| wallyworld | along with juju itself | 00:07 |
| wallyworld | isn;t that already approved? | 00:07 |
| mwhudson | wallyworld: oh maybe | 00:08 |
| wallyworld | i thought so but am out of the loop a little on all that | 00:08 |
| menn0 | wallyworld, axw: I'm thinking about picking up bug 1456916. this also relates to these old tickets: bug 892552 and 802117 | 00:29 |
| mup | Bug #892552: juju does not extract system ssh fingerprints <docs> <feature> <ssh> <pyjuju:Triaged> <juju-core:Triaged> <https://launchpad.net/bugs/892552> | 00:29 |
| wallyworld | menn0: it will be a fair bit of work | 00:30 |
| menn0 | wallyworld, axw: to me, the easiest win seems to be to use per-model known hosts files for juju ssh and juju scp (instead of /dev/null) | 00:30 |
| perrito666 | wallyworld: why would we embed base suite? | 00:31 |
| wallyworld | perrito666: because that supresses logging output to console and other things | 00:31 |
| perrito666 | k | 00:31 |
| menn0 | wallyworld, axw: that doesn't help with the "first connect" problem but does prevent MITM after the first connect while avoiding the scary warnings when addresses are reused between models | 00:32 |
| wallyworld | menn0: seems ok to me - so long as we can extract the fingerprints from cloud init, but i am not an expert | 00:32 |
| menn0 | I guess there's still the problem of killing a machine and recreating it in the same model which could lead to the same address being reused | 00:32 |
| wallyworld | that is the issue | 00:32 |
| wallyworld | one of | 00:33 |
| menn0 | wallyworld: yeah, one of | 00:33 |
| menn0 | wallyworld: ok so the ultimate solution would be for juju to extract the ssh key for each new machine from cloud-init and store it... and then have juju ssh and juju scp get told the key before connecting so it can be written into the client's known_hosts | 00:36 |
| menn0 | wallyworld: something like that? | 00:36 |
| axw | menn0: yep | 00:37 |
| wallyworld | menn0: +1 | 00:37 |
| wallyworld | menn0: i assume we pass pass a bespoke known hosts file to the client each time | 00:37 |
| wallyworld | mwhudson: are we putting juju-mongo3.2 into trusty? | 00:39 |
| wallyworld | cherylj: i think you did a fix so that if mongo 3.2 were not found, it would not retry? ill need to retest, but it appeared not to work when i bootstrapped a trusty controller. took ages to bootstrap and cloud init was filled with mongo3.2 retries | 00:41 |
| wallyworld | i may be able to get a fix done today if i can sort out some other issues | 00:42 |
| axw | wallyworld: we don't pass any known_hosts, we pass /dev/null. that's part of the problem | 00:46 |
| axw | I railed against that at the time it was done, but the change went in anyway | 00:47 |
| wallyworld | sigh | 00:47 |
| wallyworld | axw: awesome, i removed juju-mongodb (only have mongo 3.2 installed) and tests fail \o/ | 00:47 |
| axw | wallyworld: :/ | 00:48 |
| axw | wallyworld: I'm guessing testing/mgo.go needs updating to look in the right spot | 00:48 |
| wallyworld | yep | 00:48 |
| wallyworld | i might worry about that later | 00:48 |
| wallyworld | perrito666: btw, removing ensureAdminUser is fine; need to double check restore, but bootstrap etc all works as expected | 00:53 |
| perrito666 | sweeeet, this is a great week | 00:53 |
| redir | see you tomorrow juju-dev | 01:08 |
| === redir is now known as redir_eod | ||
| cherylj | wallyworld: yeah, I did. The fix was in juju/utils, so you'll need to make sure your deps are up to date to pull it in | 01:16 |
| wallyworld | ok, i'll double check | 01:18 |
| wallyworld | cherylj: ah, i changed my scripts (long story) and my godeps one had an issue, utils just got updated, so i'll retry | 01:19 |
| cherylj | ok, phew | 01:20 |
| cherylj | :) | 01:20 |
| thumper | menn0: if you get a minute https://github.com/juju/gomaasapi/pull/33 | 01:20 |
| wallyworld | mwhudson: when you upload the mongo 3.2 tools, the package will get installed automatically with mongo 3.2 db right? | 01:22 |
| * thumper needs food | 01:23 | |
| * thumper goes to hunt in the kitchen | 01:23 | |
| wallyworld | mwhudson: i'm testing a fix for that mongodump issue (the path and the args) and have no tools available - can you provide me wit the deb i can install manually? | 01:24 |
| mup | Bug #1569632 opened: status: decide on statuses for migration <juju-core:Triaged> <https://launchpad.net/bugs/1569632> | 01:24 |
| mup | Bug #1569632 changed: status: decide on statuses for migration <juju-core:Triaged> <https://launchpad.net/bugs/1569632> | 01:30 |
| mwhudson | wallyworld: it's in ppa:juju/experimental | 01:33 |
| wallyworld | ta | 01:33 |
| mwhudson | wallyworld: juju-mongo-tools32 | 01:33 |
| mwhudson | wallyworld: juju-mongo-tools3.2 | 01:33 |
| mwhudson | rather | 01:33 |
| wallyworld | mwhudson: yay, my fix worked, backups good now | 01:38 |
| mwhudson | wallyworld: nice | 01:38 |
| wallyworld | mwhudson: but we'll need that tools deb in the repos etc before beta4 goes out | 01:39 |
| mwhudson | wallyworld: slangasek said he'd look at it today | 01:39 |
| wallyworld | awesome | 01:39 |
| mup | Bug #1569632 opened: status: decide on statuses for migration <juju-core:Triaged> <https://launchpad.net/bugs/1569632> | 01:45 |
| === alexisb-afk is now known as alexisb | ||
| alexisb | wallyworld, menn0, axw is one of you available to look at: https://bugs.launchpad.net/juju-core/+bug/1569467 | 01:48 |
| mup | Bug #1569467: backup-restore loses the hosted model <backup-restore> <blocker> <ci> <regression> <juju-core:Triaged by cherylj> <https://launchpad.net/bugs/1569467> | 01:48 |
| cherylj | I suspect that it has come from the made-model-workers branch | 01:49 |
| axw | alexisb: this is a known deficiency of backup/restore, meant to be fixedin 2.1 or later IIRC | 01:49 |
| cherylj | but it's just a theory at this point | 01:49 |
| cherylj | I don't think it is | 01:49 |
| cherylj | I can do this test manually and it works | 01:49 |
| cherylj | It's just failing in CI | 01:49 |
| cherylj | for some reason | 01:49 |
| axw | cherylj: oh? I was under the impression that backup/restore didn't work with hosted models | 01:49 |
| cherylj | axw: the backup is being created from the admin model | 01:50 |
| alexisb | axw, BR should work on the admin model | 01:50 |
| axw | ok | 01:50 |
| cherylj | the problem that's happening in CI is that the dummy hosted model that is created to rebootstrap with is not removed | 01:50 |
| cherylj | and the existing models don't seem to get "hooked up" | 01:50 |
| === natefinch-afk is now known as natefinch | ||
| cherylj | I'm wondering if the restore is actually dying in the controller | 01:51 |
| wallyworld | i'm working on some other backup fixes for mongo 3.2 (with tools), can look a bit later | 01:51 |
| cherylj | and it doesn't complete | 01:51 |
| cherylj | I think we need another run with the environment kept after the failure | 01:51 |
| alexisb | thank you wallyworld we leave it in your capable hands | 01:51 |
| cherylj | I'll update the bug with where I am with things | 01:51 |
| cmars | anyone seen messages like this when a unit gets a hook error? https://paste.ubuntu.com/15803425/ | 02:07 |
| cmars | juju resolved --retry doesn't seem to get picked up unless i reboot the machine that it's happening on | 02:08 |
| cmars | should i open a bug? | 02:08 |
| natefinch | cmars: probably | 02:09 |
| cmars | ok | 02:09 |
| cmars | d'oh, i already did. ok, attached a full log to LP:#1566130 | 02:13 |
| mup | Bug #1566130: awaiting error resolution for "install" hook <juju-core:Triaged> <https://launchpad.net/bugs/1566130> | 02:13 |
| natefinch | wallyworld, anastasiamac: I have a question about the imagemetadata api endpoint. It has code to default the arch to amd64: https://github.com/juju/juju/blob/master/apiserver/imagemetadata/metadata.go#L184 ...when would that be triggered? Why would we ever get image metadata that doesn't specify an arch? Isn't the arch like one of the most important pieces of information? | 02:28 |
| anastasiamac | natefinch: hopefully never... | 02:29 |
| anastasiamac | (as in hopefully nver triggered) | 02:29 |
| natefinch | anastasiamac: the reason I ask is because of bug #1564791 | 02:29 |
| mup | Bug #1564791: 2.0-beta3: LXD provider, jujud architecture mismatch <blocker> <lxd> <juju-core:Triaged by natefinch> <https://launchpad.net/bugs/1564791> | 02:29 |
| anastasiamac | natefinch: so when u reproduce the bug, u fall into this code? | 02:30 |
| natefinch | anastasiamac: when using LXD, machines after the bootstrap machine on non-amd64 hosts are downloading amd64 juju binaries for some reason, which obviously don't work | 02:30 |
| natefinch | anastasiamac: it's easy to reproduce the bug, it's very hard to add debugging changes, because it only happens if you *don't* use upload-tools. I'm actually not sure how to test this with my own binary | 02:31 |
| anastasiamac | natefinch: I *think* u will only have no arch by this stage if ur image metadat does not have arch... which would b weird | 02:32 |
| wallyworld | natefinch: i think some early simplestream metadata may have omitted the arch, not sure now. but now a days, it should never be "" | 02:32 |
| natefinch | wallyworld: ok, this may be a red herring then. | 02:32 |
| wallyworld | natefinch: well, depends on the simplestreams metadata | 02:33 |
| wallyworld | if they generate metadata without an explicit arch, then boom | 02:33 |
| natefinch | actually, looks like they're explicitly requesting the wrong tools, now that I re-read the logs ni the bug | 02:34 |
| natefinch | Attempt 1 to download tools from https://10.0.3.164:17070/tools/2.0-beta3-xenial-amd64 | 02:35 |
| natefinch | so, not really the fault of the streams that it gave them what they were asking for | 02:36 |
| natefinch | I'm having a heck of a time trying to figure out how we determine the arch that eventually goes into that download URL, though | 02:36 |
| wallyworld | natefinch: it should be the arch of the host on which the tools are to be run | 02:41 |
| wallyworld | menn0: can i bother you for a 99% red one? http://reviews.vapour.ws/r/4562/ | 02:50 |
| menn0 | wallyworld: looking | 02:52 |
| menn0 | wallyworld: best. change. ever. | 02:54 |
| wallyworld | yay :-) | 02:55 |
| natefinch | wallyworld: I know the arch should be the arch of the host on which it is run, but in this case, the bug is that it's not :) | 02:58 |
| wallyworld | i need to read the bug | 02:59 |
| natefinch | wallyworld: basically just that non-amd64 LXD environments are downloading amd64 tools for some reason | 03:00 |
| mup | Bug #1554677 changed: Provider help topics need to be updated for 2.0 <2.0-count> <docteam> <juju-release-support> <juju-core:Invalid> <https://launchpad.net/bugs/1554677> | 03:00 |
| mup | Bug #1569652 opened: help text for juju grant needs improving <helpdocs> <juju-core:New> <https://launchpad.net/bugs/1569652> | 03:00 |
| mup | Bug #1569654 opened: help text for juju revoke needs improving <helpdocs> <juju-core:New> <https://launchpad.net/bugs/1569654> | 03:00 |
| wallyworld | were they bootstrapped on amd64 | 03:01 |
| wallyworld | with --upload-tools | 03:01 |
| natefinch | wallyworld: no. it's been seen on arm64 and s390x | 03:02 |
| natefinch | wallyworld: upload-tools actually fixes the problem | 03:02 |
| wallyworld | hmmm, not sure then, i'd have to debug | 03:05 |
| wallyworld | you'd need to trace the simplestreams request and response | 03:05 |
| natefinch | wallyworld: do you have tips on how to debug? Since upload-tools avoids the bug, I don't really know how to get my own jujud into the system, but still have it look to simplestreams to get tools for further machinoes | 03:06 |
| === menn0 is now known as menn0-afk | ||
| wallyworld | i'd just hack the code somewhere, eg don't store the uploaded tools in state to force a simplestreams check or something | 03:07 |
| wallyworld | not sure off hand, i'd need to read the code | 03:08 |
| natefinch | wallyworld: ok, np | 03:08 |
| mup | Bug #1567104 changed: Unable to connect to API <ci> <intermittent-failure> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1567104> | 03:12 |
| terje | hi, I'm trying to understand when config.yaml is parsed when deploying a charm. | 03:19 |
| terje | I'd like to use some values stored in config.yaml when my install, start and stop hooks are executed. | 03:19 |
| terje | Do I have the right idea/ | 03:20 |
| terje | ? | 03:20 |
| natefinch | terje: yep.... anything the user sets in the charm's config will be available to the charm code by having the charm call the config-get command, which will any configuration set on the charm (default in yaml) | 03:23 |
| natefinch | terje: https://jujucharms.com/docs/stable/authors-hook-environment#config-get | 03:23 |
| terje | ok, great. | 03:25 |
| cherylj | wallyworld: looks like perrito666's fix for the mongo path has some genuine test failures: http://juju-ci.vapour.ws:8080/job/github-merge-juju/7344/console | 03:25 |
| wallyworld | cherylj: already on it :-) | 03:25 |
| cherylj | thanks :) | 03:26 |
| wallyworld | a couple of missing "." | 03:26 |
| terje | natefinch: is config-get a shell function, sourced prior to executing my hooks? | 03:26 |
| terje | (I don't see a shell command on my system) | 03:26 |
| natefinch | terje: yep, there's a bunch of shell commands that get installed with a charm. They're called "hook tools" in juju terminology: https://jujucharms.com/docs/stable/authors-hook-environment#hook-tools | 03:27 |
| natefinch | terje: they are in the path when a hook is executing | 03:27 |
| terje | awesome, thank you. | 03:27 |
| terje | ok | 03:27 |
| wallyworld | axw: https://bugs.launchpad.net/bugs/1551779 is fixed right? | 03:30 |
| mup | Bug #1551779: New azure provider ignores agent mirrors <2.0-count> <azure-provider> <jujuqa> <regression> <streams> <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1551779> | 03:30 |
| axw | wallyworld: no, didn't make the cut: https://github.com/juju/juju/pull/5068 | 03:31 |
| wallyworld | ok, np | 03:31 |
| axw | it's ready to go whenever | 03:31 |
| wallyworld | jus saw the bug get changed, i thought it was fix committed | 03:31 |
| wallyworld | axw: i have to go afk for 45 minutes or so, if you see any landing failures, could you please retry? | 03:40 |
| axw | wallyworld: okey dokey | 03:40 |
| wallyworld | ta | 03:41 |
| mup | Bug #1568177 changed: configFunctionalSuite.TestUsingTCPRemote lxdclient no addresses match <ci> <lxd> <regression> <test-failure> <unit-tests> <juju-core:Invalid by sinzui> <https://launchpad.net/bugs/1568177> | 03:43 |
| thumper | sweet | 04:05 |
| thumper | well that was out of context... | 04:06 |
| thumper | it was for this: <wallyworld> mwhudson: yay, my fix worked, backups good now | 04:06 |
| mwhudson | :)\ | 04:06 |
| thumper | irc window was scrolled | 04:06 |
| natefinch | man, juju debug-log is sloooooooow | 04:21 |
| menn0 | wallyworld: http://reviews.vapour.ws/r/4563/ reviewed | 04:45 |
| wallyworld | menn0: ty | 04:45 |
| wallyworld | menn0: the previous PR was actually lgtm, i just fixed the tests, i'll see if i can pull the job | 04:45 |
| wallyworld | or i can tweak those method names in a driveby next time | 04:46 |
| menn0 | wallyworld: it's really not that important | 04:48 |
| menn0 | wallyworld: don't pull the current job | 04:48 |
| wallyworld | menn0: ok, i'll fix real soon in another branch | 04:48 |
| menn0 | wallyworld, axw: regarding SSH host key management... looks like there's 2 ways to tackle it | 04:59 |
| menn0 | you can have cloud-init post them to a URL | 04:59 |
| menn0 | or your can pre-generated them and have cloud-init install them for you | 04:59 |
| menn0 | I prefer the latter as it's simpler, but it does mean a little extra work for the server to do for every new machine | 05:00 |
| wallyworld | menn0: we did explorer having a listener on the cloud init url at one point (for provisioning status and errors), but it is more moving parts | 05:01 |
| wallyworld | what's the cpu cost of generating a key? | 05:01 |
| menn0 | wallyworld: pre-generating them server side does seem better | 05:01 |
| menn0 | wallyworld: I will check | 05:01 |
| axw | menn0: I don't think we can do option 1 without a lot of added complexity | 05:04 |
| axw | as in, where do we post to | 05:05 |
| axw | menn0: my intention was to generate client-side for bootstrap, then have the server generate a new key and publish it into state | 05:05 |
| axw | menn0: then we'd query state for public keys when we want to SSH | 05:05 |
| axw | no prompting needed for verifying fingerprints | 05:06 |
| menn0 | axw: I don't like the complexity of #1 either. we'd need a endpoint on the API server. | 05:06 |
| menn0 | axw: I think we're thinking the same thing. | 05:06 |
| menn0 | axw: what do you mean generate the key client side and then have the server generate a new key though? | 05:07 |
| menn0 | axw: what i'm thinking is that when a new machine is to be created, a the host keys are generated on the server and inserted into state, and also passed to cloud-init | 05:08 |
| menn0 | when you pass the keys to cloud-init, ssh keys aren't generated on the machine and the one that were passed are used instead | 05:08 |
| menn0 | axw: http://cloudinit.readthedocs.org/en/latest/topics/examples.html#configure-instances-ssh-keys | 05:08 |
| menn0 | axw: "ssh_keys" | 05:08 |
| axw | menn0: sec, trying to find something. there's something about cloud-config not being completely secure. I think it might just be because anything on the machine can query the metadata | 05:11 |
| menn0 | axw: ok, that's worth following up | 05:11 |
| axw | menn0: eh can't find it, I'm sure there's a bug aobut it tho. but basically on AWS and others, you can get the cloud-config metadata by GETting a statically defined URL | 05:12 |
| axw | menn0: no ACLs other than being on the machine | 05:13 |
| menn0 | axw: ok that would be bad | 05:13 |
| axw | so if you wanted to run something non-privileged on the machine, it could easily become privileged | 05:13 |
| axw | menn0: so, what I was thinking is that we'd pass through ssh_keys, and then have the machine agent generate a new one on startup | 05:13 |
| axw | menn0: the initial ssh_keys one is really only necessary for the bootstrap machine tho | 05:14 |
| menn0 | axw, wallyworld: FWIW generating both the RSA and DSA keys at the default (recommended) bit sizes takes less that 0.2s (combined) on my machine | 05:14 |
| wallyworld | menn0: and for dense openstack deployments with containers, there may be several spun up at once | 05:15 |
| axw | (and containers can access the metadata URL) | 05:15 |
| wallyworld | i guess the host controller needing to run that many containers would cope | 05:16 |
| axw | menn0: not sure if what I said about bootstrap & ssh_keys made any sense, let me know if a hangout would help | 05:16 |
| menn0 | axw: it mostly makes sense but a hangout would be helpful | 05:17 |
| axw | menn0: ok just eating lunch, give me 15 mins or so | 05:17 |
| menn0 | axw: np | 05:17 |
| axw | wallyworld: sorry, got distracted and forgot to check your merge again | 05:22 |
| wallyworld | axw: tis ok. there was a spurious failure, i resubmitted | 05:23 |
| axw | would be good to have the CI bot sending messages in here | 05:23 |
| menn0 | axw: +1 | 05:24 |
| === mup_ is now known as mup | ||
| axw | menn0: https://plus.google.com/hangouts/_/canonical.com/juju-ssh-keys?authuser=1 | 05:38 |
| axw | if you're free now | 05:38 |
| menn0 | axw: coming! | 05:40 |
| axw | menn0: wrong button | 05:50 |
| axw | brb | 05:50 |
| menn0 | LOL | 05:50 |
| axw | what have I done | 05:50 |
| dimitern | yay! master can bootstrap on xenial again :) | 06:01 |
| dimitern | there's another issue today though | 06:13 |
| dimitern | 2016-04-13 06:12:08 ERROR juju.worker.dependency engine.go:526 "proxy-config-updater" manifold worker returned unexpected error: unknown watcher | 06:13 |
| dimitern | id (not found) | 06:13 |
| wallyworld | axw: this shit just keeps popping up everywhere :-( http://reviews.vapour.ws/r/4564/ | 06:48 |
| axw | wallyworld: :/ looking | 06:49 |
| wallyworld | ta | 06:49 |
| axw | wallyworld: LGTM | 06:54 |
| wallyworld | ty | 06:54 |
| dimitern | wallyworld: hey, thanks for fixing that juju-mongodb3.2 issue! | 06:59 |
| wallyworld | dimitern: np, one more to go, landing now, affects restore | 06:59 |
| wallyworld | wasa joint effort with horatio | 07:00 |
| dimitern | wallyworld: I've found a new proxyupdater issue on master, which I'm testing a fix for - such a fix should qualify for landing on master, but first I'll file a bug report for it | 07:00 |
| wallyworld | ok. we need to get a bless within 6 hours or so, maybe a bit more | 07:01 |
| mup | Bug #1569725 opened: proxyupdater api facade does not set NotifyWatcherId in the result <juju-core:In Progress by dimitern> <https://launchpad.net/bugs/1569725> | 07:13 |
| dimitern | frobware, voidspace, dooferlad: http://reviews.vapour.ws/r/4565/ fixes the bug above, please take a look | 07:23 |
| frobware | dimitern: looking | 07:40 |
| frobware | dimitern: apart from the test changes is the fix just in proxyupdater.go? | 07:44 |
| frobware | dimitern: ah, I see it. | 07:44 |
| dooferlad | dimitern: +1 from me. | 07:50 |
| * dooferlad will be back a bit later | 07:51 | |
| dimitern | frobware: the fix is mostly in the api/proxyupdater changes | 07:51 |
| frobware | dooferlad: are we doing 1:1 in 10? | 07:52 |
| frobware | dooferlad: oh, it's 9:30... | 07:53 |
| dooferlad | frobware: indeed | 07:53 |
| * dooferlad really is going now | 07:53 | |
| dimitern | frobware: I've observed at last what you're describing - lxd containers coming up with 1 NIC (as reported by lxc info), but inside having multiple NICs; applying the machine profile we create directly on the container manually seems to fix this (lxc info shows 8 NICs as expected) | 07:53 |
| frobware | dimitern: it depends... the output from lxc list (with netinfo) can dribble in | 07:55 |
| frobware | dimitern: if you can get to the same stage then don't apply the profile manually, just sit on watch lxc list for a bit | 07:55 |
| dimitern | frobware: so not seeing all nics there is ok as long as they actually work - that's what you're saying? | 07:55 |
| dimitern | :) | 07:55 |
| frobware | dimitern: there's the two case: 1) we don't get anything. 2) as you've descibed, but I'm not 100% sure this is because the info from lxc list can dribble in over time. | 07:56 |
| frobware | dimitern: I want to change the way we apply the profile anyway. it can be applied to the container directly (need to expose some LXD api) | 07:57 |
| dimitern | frobware: I've seen the dribbling you're talking about | 07:57 |
| frobware | dimitern: want to sync for 30 mins? | 07:58 |
| dimitern | frobware: don't you have another call? | 07:58 |
| frobware | dimitern: would be interested in any results from LXD bashing | 07:58 |
| frobware | dimitern: yep, but in 30mins | 07:58 |
| dimitern | frobware: I'm not there yet (heavily testing lxd deployments in w/ w/o charms) | 07:58 |
| frobware | dimitern: sure, but (gentle nudge) let's sync anyway. :) | 07:59 |
| dimitern | frobware: so if you don't mind I'd like to get the AC removal well tested first and finish it, so I can focus on LXD testing | 07:59 |
| dimitern | i.e after standup will be better | 07:59 |
| frobware | dimitern: sure | 07:59 |
| dimitern | frobware: cheers | 08:00 |
| frobware | jam: ping | 08:02 |
| jam | frobware: hi | 08:02 |
| frobware | jam: I'm guessing my PR for detectSubnet() does the wrong thing... | 08:02 |
| jam | frobware: so its entirely possible that we're using it incorrectly, but the intent of detectSubnet is find a subnet that is not in use | 08:03 |
| jam | and the highest one we found +1 is not in use, but clearly the highest one we find *is* in use. | 08:04 |
| frobware | jam: so then my fix is not a fix... | 08:04 |
| frobware | jam: it would read better as detectSubnetNotInUse() :- | 08:04 |
| jam | frobware: renaming "detectSubnet" to findUnusedSubnet would probably be a step in figuring out what is wrong | 08:04 |
| jam | frobware: jinx? | 08:04 |
| jam | paraphrase jinx | 08:05 |
| dimitern | 2016-04-13 08:04:42 DEBUG juju.cmd.juju.commands ssh.go:180 proxy-ssh is false | 08:05 |
| frobware | jam: OK, I'll revert and look at that this morning. | 08:05 |
| dimitern | wtf?! since when | 08:05 |
| dimitern | can't juju ssh 1/lxd/0 because of that | 08:05 |
| jam | dimitern: there was a big discussion about it recently | 08:05 |
| dimitern | only juju ssh 0 works | 08:05 |
| dimitern | jam: and it was decided proxy-ssh is bad? | 08:06 |
| frobware | dimitern, jam: interesting. pretty sure I was ssh'ing to 0/lxd/0 yesterday | 08:06 |
| jam | dimitern: the main motivation was that in multi-model controllers, proxy-ssh means you have to have ssh access to the API server. | 08:06 |
| jam | which is very bad for security | 08:06 |
| voidspace | dimitern: frobware: is "next" unblocked yet? (as in - do lxd tests pass again, can we land branches) | 08:06 |
| jam | If I let you create a model on my controller, that doesn't mean I want you to have SSH access to root my API server. | 08:06 |
| voidspace | ah, I see updates on master and next | 08:07 |
| dimitern | jam: it depends whether 0/lxd/0 has a dns-name coming from a vlan (e.g. 10.100.19.0/24), which might only be accessible via the controller | 08:07 |
| voidspace | so probably | 08:07 |
| jam | frobware: dimitern: well aren't X/lxd/Y supposed to be on the routable subnets anyway? | 08:07 |
| jam | dimitern: so for people that have complex networking setups, they can do the jumps themselves, which is slightly unfortunate, or they can pass "--proxy" to juju ssh | 08:08 |
| frobware | jam: yes, though it's nominally easier (for me) to think of them as logical machines (i.e., 0/lxd/3) then to dig (ho) out its IP addr | 08:08 |
| jam | but the default is that you can only use it if you are an admin on the controller. | 08:08 |
| dimitern | jam: they are, but what we deem the machine's private address for ssh still uses the legacy scoped selection + sorting | 08:08 |
| dimitern | nice! --proxy it is then | 08:08 |
| dimitern | yay! works | 08:08 |
| jam | dimitern: so for people that aren't admins "--proxy" will fail (which is why we set it to default false) | 08:09 |
| jam | but if you have access to the API server its available for you | 08:09 |
| dimitern | jam: yeah, this makes sense, but unfortunately also means we have to sort out the private address picking now | 08:10 |
| jam | dimitern: well, we should generally fix that anyway, right ? :) | 08:10 |
| dimitern | jam: indeed :) | 08:10 |
| frobware | dimitern, jam: http://reviews.vapour.ws/r/4566/ | 08:13 |
| jam | frobware: shipit | 08:14 |
| frobware | ty | 08:14 |
| jam | 'this change is not incorrect' is confusing | 08:14 |
| jam | but the change itself is good | 08:14 |
| dimitern | frobware: LGTM | 08:14 |
| frobware | jam: :) the change is incorrect (was in my head, but clearly not my fingers). | 08:16 |
| dimitern | wallyworld: hey, is there a way to disable the model switching to "default" from "admin" after bootstrap? | 08:16 |
| dimitern | it's a bit frustrating to do parallel bootstrap tests on different controllers | 08:17 |
| dimitern | axw: ^^ | 08:17 |
| wallyworld | dimitern: no, that was the intended behaviour | 08:17 |
| axw | dimitern: not at the moment. that shouldn't stop parallel bootstrap though | 08:18 |
| wallyworld | we don't wanted users in general doing things with the admin model | 08:18 |
| frobware | wallyworld: it's an odd experience first time round though | 08:18 |
| dimitern | wallyworld, axw: makes sense, but it's a bit inconvenient to have to pass -m local.xxx:yyy each time.. I guess needs to be scripted | 08:18 |
| wallyworld | frobware: why is it odd? you deploy workloads to hosted models | 08:19 |
| wallyworld | dimitern: juju switch is your friend | 08:19 |
| dimitern | the only reason I do it it's because I don't want to wait yet another 5m after bootstrap to be able to add a container to a machine in the default model | 08:19 |
| frobware | wallyworld: as a new user I bootstrap and then try and add-machine lxd:0 and it doesn't. I guess most people deploy workloads. | 08:20 |
| dimitern | wallyworld: switch doesn't work well with parallel bootstraps - each does a switch at the end | 08:20 |
| frobware | wallyworld: heh, dimitern's comment is what I try to avoid too | 08:20 |
| wallyworld | frobware: the default hosted model is empty that is true | 08:21 |
| wallyworld | machine 0, the old so called bootstrap machine, is not something we want users to think about | 08:21 |
| frobware | wallyworld: and I still don't understand how I can get upgrade-juju to work in the default model | 08:21 |
| wallyworld | the controller is an inrernal detail | 08:21 |
| dimitern | I almost wish to have a --use-admin-model flag to bootstrap :) i.e. I want you to not switch to "default" as I know what I'm doing | 08:22 |
| axw | perhaps we could special case --default-model=admin? (pretty sure that is broken atm, now that I think of it) | 08:23 |
| axw | so if you did "juju bootstrap -d admin", then you would have no secondary model | 08:23 |
| dimitern | axw: that sounds exactly what I'd use | 08:23 |
| wallyworld | frobware: i think you need to upgrade the controller tools first | 08:23 |
| wallyworld | frobware: hosted models need to use tools compatible with their host | 08:24 |
| frobware | wallyworld: ok, let me spin up a machine in the default model and report on my upgrade step. my experience my actually be a symptom of bug #1569361 as I'm generally only trying to use LXD containers | 08:25 |
| mup | Bug #1569361: LXD containers fail to upgrade because the bridge config changes to a different IP address <network> <juju-core:In Progress by frobware> <https://launchpad.net/bugs/1569361> | 08:25 |
| voidspace | babbageclunk: did your storage stuff land? | 08:32 |
| voidspace | babbageclunk: I see that it didn't yet | 08:37 |
| voidspace | babbageclunk: I have a branch with deployment status implemented for maas 2 - trying it with your storage stuff merged in | 08:37 |
| voidspace | ok, it fails because a default zone is specifed | 08:37 |
| babbageclunk | voidspace: No, I was out by the time they got Jenkins unblocked. Do you know, is the next branch still going? | 08:38 |
| voidspace | babbageclunk: stuff has landed on next - so I assume we're still using it | 08:38 |
| babbageclunk | voidspace: How do I retry a merge - is it just $$retry$$ | 08:38 |
| voidspace | thumper: should we be targetting next or master | 08:38 |
| babbageclunk | voidspace: cool, thanks. | 08:38 |
| voidspace | babbageclunk: $$anything$$ | 08:38 |
| babbageclunk | voidspace: really? I thought there were different commands? | 08:39 |
| voidspace | babbageclunk: nope | 08:39 |
| voidspace | it's a regex match | 08:39 |
| babbageclunk | voidspace: ok, fixing merge conflicts and kicking it off again. | 08:42 |
| babbageclunk | voidspace: I've heard mention of $$jfdi$$ though - is that a special case? | 08:44 |
| voidspace | babbageclunk: that is special | 08:46 |
| voidspace | babbageclunk: so, *something* is adding a zone of "default" to my StartInstance call - which is causing the AllocateMachine call to fail | 08:46 |
| voidspace | this didn't happen before I don't think | 08:46 |
| voidspace | but I haven't found where it's added yet | 08:46 |
| voidspace | ah no - the machine was deployed, so there actually were no machines available | 08:47 |
| voidspace | the error message was just confusing! | 08:47 |
| babbageclunk | voidspace: ok, that was an easy merge. Just running the tests and pushing now, then I'll queue the merge again. | 08:48 |
| voidspace | ok, now I get: "ERROR could not record instance in provider-state: cannot record state instance-id: Not Found | 08:48 |
| voidspace | - 4y3h7p" | 08:48 |
| voidspace | babbageclunk: so somewhere we're being inconsistent with what we use as an id | 08:48 |
| voidspace | however, it looks like bootstrap continues | 08:49 |
| voidspace | babbageclunk: this is with your storage branch and my branches with storage/constraints/deploymentStatus | 08:50 |
| voidspace | cloud-init is running | 08:52 |
| voidspace | dimitern: frobware: so with the latest of my branches (not yet tested) and babbageclunk's not-yet-merged storage branch | 08:53 |
| voidspace | frobware: dimitern: bootstrap gets as far as deploying the node (maas status changes to Deployed) | 08:53 |
| dimitern | voidspace, babbageclunk: great! | 08:53 |
| voidspace | waiting to see if it gets any further - bootstrap has not yet returned. It has already reported one error, which will probably be fatal later but it hasn't stopped bootstrap. | 08:54 |
| dimitern | I'll give maas2 bootstrap a try today then | 08:55 |
| voidspace | ERROR failed to bootstrap model: bootstrap instance started but did not change to Deployed state: instance "4y3h7p" is started but not deployed | 08:58 |
| voidspace | not true - it is Deployed | 08:58 |
| voidspace | probably caused by: | 08:58 |
| voidspace | ERROR could not record instance in provider-state: cannot record state instance-id: Not Found | 08:58 |
| voidspace | - 4y3h7p | 08:58 |
| wallyworld | axw: off to soccer, i had to require that restore fix, pprpof test error, if it fails again and you notice, could you requeue for me? | 09:00 |
| axw | wallyworld: ok, gotta go make dinner shortly though | 09:00 |
| wallyworld | np, only if you notice | 09:01 |
| wallyworld | i'll check later | 09:01 |
| wallyworld | looks like restore is f&cked | 09:01 |
| wallyworld | errors setting restore status | 09:01 |
| axw | :/ | 09:01 |
| wallyworld | txn assert error | 09:01 |
| wallyworld | but Assert is empty | 09:01 |
| wallyworld | need to diagnose | 09:01 |
| axw | fwereade_: I have pulled Status and Life back out put params.Model, and made worker/undertaker responsible for setting model status. if you have time, PTAL | 09:19 |
| fwereade_ | axw, will do, tyvm | 09:20 |
| fwereade_ | axw, while they're not *immediately* relevant, did you have any thoughts re the philosophy-of-status essay in my reply? | 09:21 |
| voidspace | babbageclunk: if you merge voidspace/maas2-deployment-status into your storage branch | 09:33 |
| voidspace | babbageclunk: and then attempt to bootstrap | 09:33 |
| voidspace | babbageclunk: you should see the "cannot record state instance-id" error | 09:33 |
| voidspace | babbageclunk: it would be great to work on that | 09:34 |
| voidspace | babbageclunk: isn't there also your interfaces work to complete? (a MAAS 2 version of maasObjectNetworkInterfaces | 09:35 |
| babbageclunk | voidspace: yup - I need to get stop-instances and interfaces up to date with next and do PRs for them. | 09:36 |
| voidspace | babbageclunk: maas2Instance.volumes also needs implementing - and *may* be the cause of this failure (unlikely though) | 09:36 |
| babbageclunk | voidspace: then I'll grab your branch and see if I can find the instance-id problem | 09:36 |
| voidspace | babbageclunk: cool | 09:43 |
| === cargonza_ is now known as cargonza | ||
| voidspace | dimitern: that system id is correct | 09:55 |
| voidspace | dimitern: the deployed machine is 4y3h7p, the rack controller is 4y3h7n | 09:55 |
| dimitern | voidspace: hmm, but does the provider-state file get created ok? | 09:57 |
| voidspace | dimitern: so it's possible a bug in the way we fetch the instance (machine) | 09:57 |
| voidspace | dimitern: no idea | 09:57 |
| voidspace | dimitern: it needs investigating | 09:57 |
| dimitern | voidspace: and then we get 404 trying to get instance 4y3h7p? | 09:57 |
| voidspace | dimitern: but the id is correct | 09:57 |
| voidspace | dimitern: well, that's what it looks like on the basis of no investigation beyond checking the system id of the machine | 09:58 |
| axw | fwereade_: sorry, went afk. I do like the philosophy of separating collection, and summarisation/representation/visualisation. part of the reason why I removed the migration statuses from my PR | 09:58 |
| dimitern | voidspace: maybe we're not passing agent-name and it doesn't find it.. | 09:58 |
| voidspace | dimitern: babbageclunk is going to look into it | 09:58 |
| voidspace | dimitern: we are passing agent name | 09:58 |
| dimitern | voidspace: ok | 09:58 |
| voidspace | dimitern: well, at the juju level | 09:58 |
| voidspace | it's possible gomaasapi screws things up :-) | 09:58 |
| voidspace | but we can check that too | 09:58 |
| axw | fwereade_: i.e. because migration and lifecycle are quite different, and so their statuses should probably be recorded separately | 09:58 |
| axw | fwereade_: what I don't have a good idea about is how to represent all of that to the user. but at least we can make that call at the UI level | 10:01 |
| babbageclunk | frobware, dimitern, frobware: StopInstances branch - http://reviews.vapour.ws/r/4567/ | 10:12 |
| babbageclunk | (menn0 told me the trick of removing changes from the branch it was chained off - sorry about the other times it wasn't done!) | 10:13 |
| dimitern | babbageclunk: LGTM | 10:27 |
| dimitern | babbageclunk: urm.. I meant the other one of yours, looking at the one above now | 10:28 |
| mup | Bug #1569802 opened: add support for "decrement-container-interfaces-mtu" config option <juju-core:Triaged> <https://launchpad.net/bugs/1569802> | 10:28 |
| babbageclunk | dimitern: awesome, thanks | 10:33 |
| dimitern | babbageclunk: reviewed | 10:36 |
| dimitern | mgz: aren't we using go 1.6 for merge gating now? just noticed "go version go1.2.1" still used by github-merge-juju | 10:58 |
| babbageclunk | :( My maas2-storage merge run got a failure in worker/resumer tests. | 10:59 |
| babbageclunk | They pass when I run them locally - anyone else seeing that? | 11:01 |
| dimitern | babbageclunk: yeah, that's one of the flaky ones - seen it before, just $$retry$$ | 11:02 |
| dimitern | frobware, babbageclunk: fyi - https://github.com/juju/juju/pull/5130 if you have no objections, I suggest merging this | 11:46 |
| babbageclunk | dimitern: makes sense to me. | 11:49 |
| frobware | dimitern: I got a failed CI unit test run this morning for my LXD revert - do you know if this was already present in next? | 11:50 |
| dimitern | frobware: which one? | 11:52 |
| frobware | dimitern: http://juju-ci.vapour.ws:8080/job/github-merge-juju/7359 | 11:52 |
| dimitern | frobware: that looks like it's on next and also a mongo-related flakiness | 11:56 |
| frobware | dimitern: yeah, submitted again | 11:57 |
| mup | Bug #1555211 changed: Model name that "destroy-model" accepts doesn't match "list-models" output <2.0-count> <juju-release-support> <juju-core:Fix Released> <https://launchpad.net/bugs/1555211> | 12:11 |
| TheMue | (late) morning | 12:18 |
| voidspace | babbageclunk: did you make any progress tracking that issue? | 12:18 |
| babbageclunk | voidspace: no, sorry - was making review changes and wrangling branches. | 12:21 |
| babbageclunk | voidspace: I just got up to trying out your branch now, but when I try bootstrapping I get "ERRIR Requested map, got <nil>." | 12:22 |
| wallyworld | fwereade_: i am stupid, can i ask you to look at something? | 12:23 |
| wallyworld | i sort of hope for ann answer without having to dig too deeply and debug | 12:24 |
| babbageclunk | voidspace: Ed Hope-Morley was just here. | 12:24 |
| voidspace | babbageclunk: you need to set the MAAS2 feature flag | 12:26 |
| voidspace | babbageclunk: ah, cool | 12:26 |
| voidspace | babbageclunk: export JUJU_DEV_FEATURE_FLAGS=maas2 | 12:26 |
| babbageclunk | voidspace: thanks, was just asking that. | 12:27 |
| voidspace | babbageclunk: otherwise it assumes maas 1 (we put the maas 2 work behind a feature flag) | 12:27 |
| babbageclunk | voidspace: now I get a panic! | 12:28 |
| babbageclunk | voidspace: nil-pointer dereference. Do I need to be on a later version of MAAS? | 12:29 |
| babbageclunk | voidspace: http://pastebin.ubuntu.com/15809705/ | 12:31 |
| babbageclunk | voidspace: Oh, is that because the storage is nil? Does this branch have the storage changes in it? | 12:33 |
| voidspace | babbageclunk: no - you need my branch plus storage | 12:33 |
| voidspace | babbageclunk: is storage merged? | 12:34 |
| babbageclunk | voidspace: ok | 12:34 |
| babbageclunk | voidspace: No, grr - and the build for it just failed, I think because interfaces just got merged. | 12:34 |
| babbageclunk | voidspace: 5th or 6th time's the charm! | 12:35 |
| voidspace | :-) | 12:36 |
| babbageclunk | voidspace: the fun bit is, I've still got a merge for stop-instances in the queue. That's based off storage, and I've updated it now so storage might get merged with that. | 12:43 |
| babbageclunk | voidspace: What happens to the storage PR in that case? Will it just become merged automatically when all of its changes are already in the destination branch? | 12:45 |
| babbageclunk | voidspace: Right, my head's feeling a bit explody, I'm going for a run. | 12:45 |
| voidspace | babbageclunk: if you set stop-instances to merge and that is based off storage then yes - merging stop-instances will merge storage I think | 12:47 |
| voidspace | babbageclunk: but as you want to merge both that doesn't matter | 12:47 |
| voidspace | babbageclunk: not sure what will happen to the PR, it will *probably* get marked as merged | 12:48 |
| babbageclunk | voidspace: yup, that's all good. | 12:48 |
| * babbageclunk is actually going for a run now | 13:18 | |
| === babbageclunk is now known as babbageclunk-afk | ||
| fwereade_ | wallyworld, sorry, missed you, in meeting | 13:19 |
| fwereade_ | wallyworld, how can I help, if you haven't already solved it? | 13:19 |
| wallyworld | fwereade_: there's a txn that returns Aborted - it appears to have just started to act up | 13:20 |
| wallyworld | func (info *RestoreInfo) SetStatus(status RestoreStatus) | 13:20 |
| wallyworld | in state/backups/restore.go | 13:20 |
| wallyworld | the initial state being passedin is Pending | 13:21 |
| wallyworld | and the txn fails, so restore aborts | 13:21 |
| wallyworld | i can't see why off hand | 13:21 |
| wallyworld | if i comment out the assert, it works | 13:21 |
| wallyworld | initially, it's an empty assert for pending | 13:21 |
| wallyworld | ah, not state/backups, just state | 13:22 |
| fwereade_ | wallyworld, hold on, ringing a bell... | 13:22 |
| fwereade_ | wallyworld, heh, there might be a few things going on | 13:23 |
| wallyworld | it's such a simple txn | 13:24 |
| fwereade_ | wallyworld, the failure would seem to indicate that something else has already set the status away from pending | 13:24 |
| wallyworld | fwereade_: not really | 13:24 |
| wallyworld | for pending, the assert is empty | 13:25 |
| wallyworld | ah, but initially there won't be a doc | 13:25 |
| fwereade_ | wallyworld, d'oh, yes | 13:26 |
| wallyworld | so it should be Insert | 13:26 |
| wallyworld | so how the fuck did this ever work | 13:26 |
| wallyworld | the code is from 2014 | 13:26 |
| fwereade_ | wallyworld, heh, I actually think I must have broken it | 13:27 |
| wallyworld | maybe something else inserts a doc initially | 13:27 |
| wallyworld | oh, when? | 13:27 |
| fwereade_ | wallyworld, creating a RestoreInfoSetter would implicitly set a status, which would trigger txn failures when two things created them at once | 13:28 |
| fwereade_ | wallyworld, especially infurating when one of them was only creating the "setter" in order to get, grrmbl | 13:28 |
| wallyworld | see https://bugs.launchpad.net/juju-core/+bug/1569467/comments/2 | 13:28 |
| mup | Bug #1569467: backup-restore loses the hosted model <backup-restore> <blocker> <ci> <regression> <juju-core:Triaged by cherylj> <https://launchpad.net/bugs/1569467> | 13:28 |
| wallyworld | that comment has a timeline of sorts | 13:29 |
| fwereade_ | wallyworld, but, yeah, if there wasn't *something* else creating the restore status, I don't see how that could have ever worked | 13:29 |
| wallyworld | fwereade_: a quick code search shows nothing now is creating an initial entry | 13:29 |
| wallyworld | cherylj: see scrollback - that restore issue has been identified as due to a refactoring earlier - i don't think it can be fixed in time for beta4 | 13:30 |
| fwereade_ | wallyworld, concur | 13:31 |
| wallyworld | cherylj: we will just have to release note it and fix for the next release | 13:31 |
| fwereade_ | wallyworld, but, well, I made that change a long time ago and I'm 99% sure it was on MADE-model-workers pre-bless... maybe something else got mangled in a merge? | 13:32 |
| wallyworld | fwereade_: or CI didn't pick it up | 13:32 |
| wallyworld | because the reason it has started failing is due to newer multi-model tests | 13:32 |
| fwereade_ | wallyworld, right, but I'm with you in that I can't see how it could have worked at all | 13:33 |
| wallyworld | where the hosted model is created with a name named after the test | 13:33 |
| fwereade_ | wallyworld, (by the way, fun feature of that file: const currentRestoreId = "current") | 13:33 |
| wallyworld | fwereade_: one option maybe just to get it working is to comment out the restore status setting | 13:34 |
| fwereade_ | wallyworld, ha | 13:34 |
| wallyworld | as in restore will work but status will be unknown; i'd have to check to see where status is used | 13:34 |
| fwereade_ | wallyworld, yeah... you know, I'm actually not sure it is even used | 13:35 |
| wallyworld | fwereade_: looks like a watcher is used to set a restoring flag on the machine agent | 13:36 |
| wallyworld | so we can block api calls | 13:37 |
| wallyworld | so it sort of is needed i think | 13:37 |
| fwereade_ | wallyworld, yeah, sort of, even though that flag is not goroutine safe, and doesn't appear to have any way to dc existing connections | 13:38 |
| fwereade_ | wallyworld, wait a mo | 13:39 |
| fwereade_ | wallyworld, if the assert is empty | 13:39 |
| fwereade_ | wallyworld, I don't think that'd ErrAborted, would it? | 13:40 |
| wallyworld | fwereade_: tl;dr; no quick fix for beta4 for inclusion in xenial | 13:40 |
| wallyworld | given we need to get a CI bless asap now | 13:40 |
| fwereade_ | wallyworld, surely it would run and try to modify a document that wasn't there and fail silently? | 13:40 |
| fwereade_ | wallyworld, yeah, agreed | 13:41 |
| wallyworld | fwereade_: that's what i thought, but it seems it does because the doc doesn't exist and Insert is not used | 13:41 |
| fwereade_ | wallyworld, oh, that scenario explicitly gives us ErrAborted? | 13:41 |
| wallyworld | i'm going by what the logs report | 13:41 |
| wallyworld | appears so | 13:41 |
| wallyworld | according to the logs | 13:41 |
| wallyworld | either way, it's broken | 13:42 |
| wallyworld | fwereade_: i hacked up a version where for pending, the first status set call, i made the Assert nil, and that seemed to allow it to get futher | 13:42 |
| wallyworld | so in that case it may have silently failed, because the next status set call to restoring failed | 13:43 |
| fwereade_ | wallyworld, it'll still fail later though, won't it? FinishRestore will never succeed | 13:43 |
| wallyworld | since that call did have an assert | 13:43 |
| wallyworld | yep | 13:43 |
| mup | Bug #1569898 opened: cmd/pprof: sporadic test failure <juju-core:New> <https://launchpad.net/bugs/1569898> | 13:44 |
| wallyworld | so the restore nazi says "no restore for you in beta4" (with apologies to Seinfeld and the Soup Nazi) | 13:44 |
| wallyworld | cherylj: when is the drop dead cutoff for beta4? | 13:45 |
| cherylj | ha, yesterday | 13:45 |
| cherylj | heh | 13:45 |
| cherylj | we need to get a release out asap | 13:45 |
| cherylj | this whole restore situation is confusing to be because it works for me on joyent | 13:46 |
| cherylj | to me, not to be | 13:46 |
| cherylj | heh | 13:46 |
| cherylj | wallyworld: in your recreate, did you see anything weird for mongo in syslog? | 13:48 |
| wallyworld | cherylj: didn't look in syslog - the error is quite apparent from the juju logs | 13:49 |
| wallyworld | cherylj: are you sure it worked? | 13:49 |
| wallyworld | it may have appeared to work | 13:49 |
| wallyworld | it would look like it did from juju status | 13:49 |
| wallyworld | but it would not have | 13:49 |
| cherylj | wallyworld: I could run juju status on the hosted model and it showed me the right info (correct machine / service) and list-models showed the right info | 13:50 |
| cherylj | I bootstrapped with a different model name for the default model | 13:50 |
| cherylj | and that was right too | 13:50 |
| wallyworld | hah ok, i did not see that in my testing, it failed deterministically, and the cause is now apparent from the code | 13:51 |
| wallyworld | cherylj: did you use upload-tools | 13:51 |
| wallyworld | your restore may have grabbed beta3 tools | 13:51 |
| wallyworld | which may not have the problem | 13:51 |
| cherylj | I did upload-tools | 13:52 |
| cherylj | er wait | 13:52 |
| cherylj | not for restore, you're right | 13:52 |
| wallyworld | that may well be the reason | 13:52 |
| cherylj | thank goodness there's some sense to the situation then | 13:52 |
| cherylj | heh | 13:52 |
| wallyworld | yeah, agreed, we needed to be able to explain it | 13:52 |
| wallyworld | but sadly we will go to beta4 without restore working | 13:53 |
| wallyworld | cherylj: i landed a few branches today to fix various issues with restore and mongo3.2; so all that should be ok now, but of course now that that is fixed, we see this other issue, although i did much of my testing on trusty containers for the lastest failures | 13:54 |
| mup | Bug #1569898 changed: cmd/pprof: sporadic test failure <juju-core:New> <https://launchpad.net/bugs/1569898> | 13:56 |
| natefinch | wallyworld: do you know where I should be looking for the code I need to hack for upload-tools? | 14:02 |
| mup | Bug #1569898 opened: cmd/pprof: sporadic test failure <juju-core:New> <https://launchpad.net/bugs/1569898> | 14:02 |
| mup | Bug #1569914 opened: help text for juju show-controller needs improving <helpdocs> <juju-core:New> <https://launchpad.net/bugs/1569914> | 14:02 |
| wallyworld | natefinch: to do which bit? | 14:02 |
| wallyworld | natefinch: did you see john's explanation? | 14:03 |
| wallyworld | you may not need to hack anything | 14:03 |
| natefinch | wallyworld: I saw the text, but it doesn't really help me. | 14:04 |
| wallyworld | doesn't it explain why amd64 is being chosen? | 14:04 |
| wallyworld | the provider is not asking for the right arch | 14:05 |
| === dimitern` is now known as dimitern | ||
| wallyworld | i only skimmed it | 14:05 |
| natefinch | wallyworld: it doesn't say where the code is that it's talking about. The provider is a ton of twisty code | 14:05 |
| wallyworld | i have not looked at it | 14:05 |
| natefinch | wallyworld: also, when he says images, does he mean the OS image or the tools image? | 14:06 |
| wallyworld | os image | 14:06 |
| natefinch | wallyworld: well, ok, I know that's incorrect | 14:06 |
| natefinch | wallyworld: the alias of the OS image may be ubuntu-trusty, but the arch is correct... we can see that in lxd... and the fact that the OS image runs at all means it's the right arch | 14:07 |
| natefinch | wallyworld: the problem is that cloud-init is requesting the wrong tools version | 14:07 |
| natefinch | wallyworld: e.g. Attempt 1 to download tools from https://10.0.3.164:17070/tools/2.0-beta3-xenial-amd64... | 14:08 |
| mup | Bug #1569914 changed: help text for juju show-controller needs improving <helpdocs> <juju-core:New> <https://launchpad.net/bugs/1569914> | 14:08 |
| wallyworld | right ok, so that last bit is the binary version string | 14:09 |
| natefinch | wallyworld: what I have been trying to figure out is how we generate the URL to request.... it's just really hard to trace it back just by looking at the code, and I haven't been able to figure out how to use upload-tools and still let the code request stuff from streams | 14:09 |
| wallyworld | i gave you instructions :-) | 14:09 |
| wallyworld | which i have not tried, i think they will work | 14:10 |
| natefinch | wallyworld: I don't know where any of that code lives | 14:10 |
| natefinch | wallyworld: I spent a few hours trying to find that code last night | 14:10 |
| wallyworld | i'd have to search for it to be exact. there's a ToolsStorage struct in state which is used to store the tools | 14:11 |
| wallyworld | then i'd sesrch for what increments the version Build field | 14:11 |
| wallyworld | to see how to stop that ebing incremented | 14:11 |
| wallyworld | you could also trace what sets the cloud init config | 14:12 |
| wallyworld | see see where it is telling it to download tools of a certain arch | 14:12 |
| natefinch | I have tried that, too. | 14:13 |
| katco | wallyworld: cherylj: doh... sorry i inverted the fix to that bug | 14:13 |
| wallyworld | katco: you had your glasses on back to front | 14:13 |
| cherylj | heh | 14:13 |
| katco | wallyworld: lol | 14:13 |
| mup | Bug #1569914 opened: help text for juju show-controller needs improving <helpdocs> <juju-core:New> <https://launchpad.net/bugs/1569914> | 14:14 |
| wallyworld | natefinch: i just did a quick code search - did you look at func InstanceConfig() in instanceconfig.go ? | 14:17 |
| wallyworld | that's where the cloud init tools are set up i think | 14:17 |
| cherylj | natefinch: some (possibly) interesting factoids | 14:17 |
| cherylj | 1 - This test had been passing on arm64 until the host was updated to xenial | 14:17 |
| tych0 | frobware: hi, what's the current state of lxd/juju? | 14:18 |
| tych0 | frobware: i'm finally done with some customer stuff and can get back to looking at it now | 14:18 |
| tych0 | (well, i have to fly back to denver, but i can look this afternoon/late evening) | 14:18 |
| cherylj | 2 - the problem with s390 could be attributed to s390x not being an valid arch in beta3 (see bug 1554675) | 14:19 |
| mup | Bug #1554675: Unable to bootstrap lxd provider on s390x <lxd> <s390x> <juju-core:Fix Committed by cherylj> <juju-core 1.25:Fix Released by cherylj> <https://launchpad.net/bugs/1554675> | 14:19 |
| wallyworld | natefinch: so looking at that code, it appears the tools arch comes from the machine's recorded HardwareCharacteristics, which may be nil | 14:21 |
| wallyworld | this is just a guess - i've not seen this code before, it's all been refactored | 14:22 |
| wallyworld | so when the container is created, if the hardware characteristics do not include the arch, that could explain it | 14:22 |
| wallyworld | but i'd need to trace it through | 14:23 |
| wallyworld | you don't need to hack around upload tools to do that | 14:23 |
| wallyworld | just add extra debugging | 14:23 |
| wallyworld | cherylj: natefinch: using upload tools would get around that invalid arch issue in beta3 | 14:25 |
| cherylj | wallyworld: for the s390 case at least. Don't know if it's the same for arm | 14:25 |
| wallyworld | cherylj: arm should have been properly defined in juju since forever | 14:26 |
| === babbageclunk-afk is now known as babbageclunk | ||
| cherylj | yeah, I know :( | 14:27 |
| cherylj | but, it may not make a difference | 14:27 |
| cherylj | how could they have bootstrapped a s390x lxd provider anyway? | 14:28 |
| cherylj | I don't know... just spewing data points | 14:28 |
| sinzui | wallyworld: cherylj: since agents use deb archs "dpkg --print-architecture" on the client host will show probbly arch for the container | 14:28 |
| babbageclunk | voidspace, dimitern: gah! Is allWatcherStateSuite.TestStateWatcherTwoModels another flaky test? | 14:28 |
| cherylj | yes | 14:29 |
| babbageclunk | :( | 14:29 |
| babbageclunk | oh, cherylj, that probably wasn't to me, was it. | 14:29 |
| cherylj | babbageclunk: it was, sorry :) | 14:30 |
| babbageclunk | cherylj: d'oh | 14:31 |
| babbageclunk | cherylj: ok, requeuing. Thanks! | 14:31 |
| natefinch | wallyworld: where's the code that gets the arch from the machine? | 14:31 |
| wallyworld | can't recall, i 'll look | 14:32 |
| tych0 | frobware: i just tried a deploy with master on GCE, looks like it's still broken with no network. i know you said you had a patch for this in progress, what's the state of it? | 14:32 |
| === tasdomas` is now known as tasdomas | ||
| katco | cherylj: really quick review for inverting that bug fix? http://reviews.vapour.ws/r/4571/diff/# | 14:34 |
| cherylj | katco: we should probably cancel the $$merge$$ request on my revert, then | 14:34 |
| cherylj | sinzui: can we do that? ^^ | 14:35 |
| wallyworld | natefinch: i've traced it back from the state docs - there's a SetInstanceInfo() API on the machine facade called by the provisioner | 14:35 |
| cherylj | it's still in the queue | 14:35 |
| sinzui | cherylj: me trues | 14:35 |
| wallyworld | natefinch: and that gets it's infro from StartInstance() | 14:35 |
| katco | cherylj: whatever is easier | 14:36 |
| sinzui | cherylj: your PR? | 14:36 |
| wallyworld | natefinch: this is maas right? | 14:36 |
| natefinch | wallyworld: lxd | 14:36 |
| cherylj | sinzui: https://github.com/juju/juju/pull/5131 | 14:36 |
| natefinch | wallyworld: sounds like the problem may be that startinstance isn't setting the arch when it should be | 14:36 |
| wallyworld | yes, which is what john said i think | 14:37 |
| sinzui | cherylj: I don't see it in the queue to cance | 14:37 |
| sinzui | wow 2 hours ago | 14:37 |
| cherylj | yeah, there's quite a backlog for merging | 14:37 |
| sinzui | cherylj: revert-upload-tools is canceled | 14:38 |
| cherylj | thanks, sinzui :) | 14:38 |
| katco | sinzui: ty! | 14:38 |
| wallyworld | natefinch: looks like lxd does the right thing at a quick look | 14:38 |
| wallyworld | but all this can be debugged | 14:38 |
| natefinch | wallyworld: yeah, doing that now | 14:39 |
| wallyworld | ok, good luck, i'm out of here for a few hours to zzzzzzzzz | 14:39 |
| natefinch | wallyworld: good, talk to you tomorrow | 14:39 |
| wallyworld | ttyl | 14:40 |
| bogdanteleaga | sinzui, cherylj, can this get bumped to a higher priority? https://bugs.launchpad.net/juju-core/+bug/1567676 | 14:42 |
| mup | Bug #1567676: windows: networker tries to update invalid device and blocks machiner from working <juju-core:Triaged> <https://launchpad.net/bugs/1567676> | 14:42 |
| bogdanteleaga | the machiner is broken | 14:42 |
| cherylj | bogdanteleaga: sure I can target for rc1 | 14:44 |
| babbageclunk | voidspace: ok, so I see the power-settings clearing bug Tim was talking about, I think. | 14:45 |
| voidspace | babbageclunk: heh, much better not to set the power :-) | 14:46 |
| katco | cherylj: any suggestions on what to pick up next? | 14:47 |
| cherylj | you can take bogdanteleaga's bug he just mentioned :) | 14:47 |
| katco | cherylj: let me tal | 14:47 |
| cherylj | I just added a card for it | 14:47 |
| katco | cherylj: can i reproduce this on linux, or do i need a windows installation? | 14:48 |
| cherylj | katco: I suspect it's windows only | 14:50 |
| katco | cherylj: do we have any windows installations to test against? | 14:50 |
| cherylj | katco: there is in CI. sinzui, abentley - can katco access a MAAS that has windows images to deploy? | 14:51 |
| sinzui | cherylj: katco sure. I can add katco to munna | 14:52 |
| katco | sinzui: cherylj: ty... any documentation on how to utilize this? | 14:52 |
| sinzui | katco: I have no documentation. These masses are all new to me | 14:53 |
| katco | sinzui: ok | 14:54 |
| katco | sinzui: do i need to be on a vpn or anything? | 14:54 |
| sinzui | katco: cloud-city and the CI runs have configureations in them | 14:54 |
| sinzui | katco: you will not, you well enter as the CI bot with all its privs | 14:55 |
| katco | natefinch: standup time | 15:02 |
| frobware | tych0: it's currently in master | 15:03 |
| frobware | alexisb, tych0: is is possible to get GCE creds to poke around? | 15:03 |
| dimitern | alexisb: ping | 15:07 |
| alexisb | dimitern, omw | 15:09 |
| dimitern | ok | 15:10 |
| mup | Bug #1569948 opened: help text for juju list-machines needs improving <helpdocs> <juju-core:New> <https://launchpad.net/bugs/1569948> | 15:14 |
| mup | Bug #1569949 opened: log spam: "skipping observed IPv6 address ..." <juju-core:New> <https://launchpad.net/bugs/1569949> | 15:14 |
| natefinch | katco, ericsnow: and once I figure out how to do this dance with upload-tools, I'll write it down so others can benefit from it. | 15:25 |
| katco | natefinch: sounds good | 15:27 |
| ericsnow | natefinch: nice | 15:28 |
| katco | ericsnow: natefinch: btw they're doing something to the sidewalk again -.- i'm hopeful they don't cut anything this time | 15:28 |
| ericsnow | katco: buena suerte | 15:28 |
| terje | I'm deploying a charm I'm working on to private openstack clouds. In doing so, I need a few things (endpoints, username/pw, token's), etc. | 15:34 |
| terje | Currently I put them in config.yaml and use config-get to access them when the charm deploys but since they differ, per install this isn't going to work | 15:35 |
| terje | is there a better way to go about this? | 15:35 |
| katco | terje: you might have more luck in #juju | 15:36 |
| katco | terje: that's where the charming folks hang out, and i'm sure they've run into that problem before | 15:37 |
| cherylj | perrito666: ping? | 15:37 |
| perrito666 | cherylj: pong | 15:37 |
| natefinch | yeah, folks on this channel just aren't as charming | 15:37 |
| cherylj | perrito666: looks like the mongo path changes broke windows unit tests: http://paste.ubuntu.com/15813652/ | 15:37 |
| perrito666 | cherylj: whaaa? I am pretty sure I +build !windows to that test | 15:38 |
| mup_ | Bug #1569963 opened: log spam: image metadata, apiworkers manifold worker, KVM <juju-core:New> <https://launchpad.net/bugs/1569963> | 15:38 |
| mup_ | Bug #1569969 opened: No way to set credential for current model. <juju-core:New> <https://launchpad.net/bugs/1569969> | 15:38 |
| perrito666 | cherylj: there it is https://github.com/juju/juju/blob/master/mongo/internal_test.go#L4 I wonder what is going on | 15:39 |
| perrito666 | natefinch: do you know if build flags are valid for tests? | 15:39 |
| terje | lulz, ok. thanks. There's never anyone active in #juju but I'll give it a shot. | 15:39 |
| cherylj | perrito666: the failing test is in state/backups/internal_test.go | 15:42 |
| natefinch | perrito666: absolutely | 15:42 |
| cherylj | which does not have the windows build flag | 15:42 |
| perrito666 | cherylj: oh, I see, I did not touch that why would it be failing | 15:42 |
| cherylj | (sorry, I cut that part off from the paste) | 15:42 |
| * perrito666 goes check | 15:43 | |
| cherylj | here's the full paste: http://paste.ubuntu.com/15813799/ | 15:44 |
| === mup_ is now known as mup | ||
| perrito666 | cherylj: I see no state/backups/internal_test.go | 15:44 |
| perrito666 | cherylj: actually, grep says that test is not here | 15:45 |
| cherylj | https://github.com/juju/juju/blob/master/state/backups/internal_test.go#L28 | 15:45 |
| perrito666 | cherylj: ohhh, I see, Ian most likely merged this while my patch was on the queue | 15:46 |
| perrito666 | there it is | 15:47 |
| terje | crickets in #juju .. | 15:49 |
| katco | marcoceppi: anyone that can help terje out? | 15:49 |
| marcoceppi | terje: katco hi, I'll answer in #juju | 15:50 |
| katco | marcoceppi: thx dude | 15:50 |
| terje | yea, thanks! | 15:50 |
| katco` | gah... stupid construction! power keeps cutting out | 15:59 |
| katco` | thank god the internet seems unaffected | 15:59 |
| fwereade_ | praise be to the internet | 16:01 |
| katco` | may we all bask in its warm glow and loving embrace | 16:02 |
| katco` | so sayeth we all | 16:02 |
| === katco` is now known as katco | ||
| natefinch | sinzui: we have 2.0beta3 in streams somewhere, right? | 16:03 |
| natefinch | or mgz_ ^ | 16:03 |
| sinzui | natefinch: devel streals | 16:03 |
| sinzui | devel streams | 16:03 |
| natefinch | sinzui: what's the url for that? I'm trying to write some documentation and I can't actually find those tools in streams. I'm sure I'm just missing them somewhere | 16:04 |
| sinzui | natefinch: agent-stream: devel | 16:04 |
| sinzui | no need to set agent-metadata-url | 16:05 |
| natefinch | sinzui: no no... what do I type into chrome? | 16:05 |
| natefinch | sinzui: I just want to see what's available | 16:05 |
| sinzui | natefinch: This is the official streams location https://streams.canonical.com/juju/tools/streams/v1/index2.json | 16:06 |
| natefinch | sinzui: where's the actual list of available images? I don't know how to parse that json | 16:08 |
| sinzui | natefinch: images? oh, that would be cloud-images.ubuntu.com | 16:09 |
| natefinch | sinzui: s/images/tools | 16:09 |
| natefinch | sinzui: for example: https://streams.canonical.com/juju/tools/proposed/ | 16:10 |
| natefinch | sinzui: except, with 2.0beta3 in the list | 16:10 |
| natefinch | sinzui: this is for human consumption, not machine | 16:10 |
| sinzui | natefinch: that is obsolete from a long time ago | 16:10 |
| natefinch | sinzui: somewhere there is a link to juju-2.0beta3-trusty-amd64.tgz .... can you pleaase just find that url for me? | 16:11 |
| sinzui | natefinch: https://streams.canonical.com/juju/tools/streams/v1/index2.json is what juju uses | 16:12 |
| sinzui | natefinch: in that file is a path the the stream you want to use, "devel" | 16:12 |
| sinzui | natefinch: a devel juju will then read https://streams.canonical.com/juju/tools/streams/v1/com.ubuntu.juju-devel-tools.json | 16:13 |
| natefinch | sinzui: what is the base of the relative path that they give? e.g. "agent/2.0-beta3/juju-2.0-beta3-trusty-amd64.tgz" ? | 16:16 |
| sinzui | natefinch: the agent-metadata-url used by Juju. Juju's default is https://streams.canonical.com/juju/tools | 16:17 |
| mup | Bug #1569982 opened: pathsSuite.TestPathDefaultMongoExists failing on windows <blocker> <ci> <juju-core:Triaged by cherylj> <https://launchpad.net/bugs/1569982> | 16:18 |
| natefinch | sinzui: ahh, thank you | 16:18 |
| sinzui | natefinch: The directories are browsable on that server :) https://streams.canonical.com/juju/tools/agent/ | 16:19 |
| natefinch | sinzui: yes, that's the URL I was looking for | 16:19 |
| sinzui | cherylj: bug 1559715 is about the model instances being left behind | 16:23 |
| mup | Bug #1559715: restore-backup is unreliable <backup-restore> <ci> <destroy-controller> <destroy-environment> <regression> <juju-ci-tools:Triaged> <juju-core:Triaged> <https://launchpad.net/bugs/1559715> | 16:23 |
| sinzui | cherylj: I will re-title it | 16:23 |
| cherylj | perrito666: https://github.com/juju/juju/pull/5135 | 16:29 |
| cherylj | sinzui: yeah, good idea | 16:29 |
| tych0 | jam: frobware: https://github.com/juju/juju/pull/5136 | 16:46 |
| frobware | tych0: your testing for this problem is from the tip of master? | 16:47 |
| tych0 | frobware: yes | 16:47 |
| frobware | tych0: ok, makes perfect sense | 16:48 |
| tych0 | frobware: although i haven't tested against "next", but it looked to me like they were the same | 16:48 |
| frobware | tych0: the reason I was asking is because there is (or has been a race) in that part of the code | 16:48 |
| tych0 | right, i assume that is what you were working on? | 16:49 |
| frobware | tych0: the "if len(networkConfig.Interfaces) > 0 " can happen very occasionally on MAAS | 16:49 |
| tych0 | you mean it is == 0 even on maas? | 16:49 |
| frobware | tych0: or, put another way, it's possible for len == 0 even on MAAS... | 16:49 |
| frobware | tych0: let me dig out the bug | 16:49 |
| tych0 | frobware: right, i'm not testing on mass, but gce | 16:49 |
| tych0 | and it is always broken for me | 16:50 |
| frobware | tych0: bug #1564395 | 16:50 |
| mup | Bug #1564395: newly created LXD container has zero network devices <bootstrap> <conjure> <network> <juju-core:In Progress by dimitern> <https://launchpad.net/bugs/1564395> | 16:50 |
| tych0 | that's what i was complaining about a few weeks ago | 16:50 |
| frobware | tych0: can you send me the machine-0.log without your fix? | 16:51 |
| mup | Bug #1566531 changed: Instances are left behind testing Juju 2 <ci> <destroy-environment> <ec2-provider> <jujuqa> <juju-ci-tools:Invalid> <juju-core:Fix Released> <https://launchpad.net/bugs/1566531> | 16:51 |
| mup | Bug #1570009 opened: pathsSuite.TestPathDefaultMongoExists fails because of windows path <ci> <regression> <test-failure> <unit-tests> <windows> <juju-core:Triaged> <https://launchpad.net/bugs/1570009> | 16:51 |
| tych0 | frobware: sure | 16:51 |
| tych0 | (i'm on a plane right now so it might be slow :) | 16:51 |
| frobware | tych0: planes are fast. :-D | 16:51 |
| tych0 | frobware: but i think without the above juju/lxd container type is basically broken everywhere, no? | 16:51 |
| cherylj | can I get a quick review? http://reviews.vapour.ws/r/4574/ | 16:52 |
| frobware | tych0: yes-ish. what's not clear is whether that's due to the bug ^^ -- ie., by the time we get to "your patch" it was going to be empty come what may | 16:52 |
| tych0 | frobware: http://paste.ubuntu.com/15815783/ | 16:52 |
| tych0 | frobware: oh. does juju provide network config in all cases? | 16:53 |
| tych0 | frobware: i read somewhere in the code that it was opitonal | 16:53 |
| tych0 | maybe in the struct InstanceSpec comments or something | 16:53 |
| frobware | tych0: checking... | 16:57 |
| === redir is now known as redir_brb | ||
| tych0 | frobware: ah, it's on StartInstanceParams | 16:59 |
| * babbageclunk is BACK! | 17:00 | |
| mup | Bug #1570009 changed: pathsSuite.TestPathDefaultMongoExists fails because of windows path <ci> <regression> <test-failure> <unit-tests> <windows> <juju-core:Triaged> <https://launchpad.net/bugs/1570009> | 17:21 |
| === redir_eod is now known as redir | ||
| frobware | tych0: followed up in RB | 17:31 |
| cherylj | perrito666: review, pretty please? http://reviews.vapour.ws/r/4574/ | 17:31 |
| frobware | cherylj: updated the release notes but will follow-up again tomorrow in the cold light of day... | 17:31 |
| perrito666 | cherylj: ship it | 17:32 |
| cherylj | tyvm, frobware! | 17:32 |
| tych0 | frobware: i'm not sure i understand your comment | 17:41 |
| tych0 | frobware: since we don't create those devices at the moment, don't we want to use the default profile whether they're nil or not? | 17:42 |
| mup | Bug #1305509 changed: state/watcher: possible data race in commonWatcher <race-condition> <juju-core:Fix Released> <https://launchpad.net/bugs/1305509> | 17:51 |
| mup | Bug #1517747 changed: provider/joyent/gomanta: data race <2.0-count> <race-condition> <juju-core:Triaged> <https://launchpad.net/bugs/1517747> | 17:51 |
| mup | Bug #1570031 opened: Cannot bootstrap MAAS2 since Juju2 does not use MAAS API V2 <juju-core:New> <https://launchpad.net/bugs/1570031> | 17:51 |
| frobware | tych0: it's not clear whether we should immediately fall back to "default" | 17:53 |
| frobware | tych0: I need to able to test this. | 17:54 |
| tych0 | frobware: what other options are there? | 17:54 |
| natefinch | I love the way in juju status that we try to save horizontal space by labelling the Instance ID column "INS-ID" ... but then the actual ID is juju-c9d1b54c-95ba-4046-8b16-06e16b02ada8-machine-0 | 17:55 |
| frobware | tych0: NetworkConfig.Device and NetworkConfig.NetworkType may have valid values | 17:55 |
| tych0 | frobware: right, but nothing in the code uses those right now right? | 17:55 |
| frobware | tych0: in container/lxd.go no, but that could be an oversight having only tested this with MAAS | 17:56 |
| frobware | tych0: I'm trying to understand from the POV of the other providers, particularly AWS with and without juju's address-allocation feature flag | 17:57 |
| tych0 | right. i'm not proposing a fix for that, i'm proposing a fix for something else: that there is no networks at all in containers on GCE or AWS | 17:57 |
| frobware | tych0: so it wasn't clear to me this was broken on AWS. | 17:57 |
| tych0 | i think it's broken everywhere that doesn't provide network configuration, which includes AWS right? | 17:58 |
| tych0 | i haven't actually tested it there, i just heard someone (maybe you?) mentioned that GCE and AWS were similar in that respect | 17:58 |
| frobware | tych0: just trying on AWS | 18:05 |
| mup | Bug #1570031 changed: Cannot bootstrap MAAS2 since Juju2 does not use MAAS API V2 <juju-core:New> <https://launchpad.net/bugs/1570031> | 18:12 |
| mup | Bug #1570035 opened: Race in api/watcher/watcher.go <ci> <race-condition> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1570035> | 18:12 |
| cherylj | katco: ericsnow just opened up a bug with the same error of that windows bug you're working on | 18:17 |
| cherylj | katco: and I'm assuming he's not on windows :) | 18:17 |
| katco | cherylj: k | 18:17 |
| cherylj | katco: so maybe there's a way to recreate without windows | 18:17 |
| katco | cherylj: dunno... ericsnow, are you? ;p | 18:17 |
| cherylj | bug 1569963 | 18:17 |
| mup | Bug #1569963: log spam: image metadata, apiworkers manifold worker, KVM <juju-core:New> <https://launchpad.net/bugs/1569963> | 18:17 |
| ericsnow | cherylj: not on windows :) | 18:17 |
| ericsnow | katco: ^^^ | 18:17 |
| katco | cherylj: this looks like a completely different error? | 18:18 |
| cherylj | machine-0: 2016-04-13 14:54:23 ERROR juju.worker.dependency engine.go:526 "apiworkers" manifold worker returned unexpected error: setting controller network config: cannot set controller provider network config: cannot set link-layer devices to machine "0": invalid device "unsupported0": Type "" not valid | 18:18 |
| katco | cherylj: oh nm i see the message now | 18:18 |
| cherylj | ok, thought I was going crazy | 18:18 |
| cherylj | which is entirely possible | 18:18 |
| katco | aren't we all? heh heh heh | 18:19 |
| frobware | tych0: I'm using your tree and AWS seems busted (for add-machine lxd:0) | 18:25 |
| tych0 | frobware: oh? what's the symptomp? | 18:26 |
| frobware | tych0: bleh, scrap that. Warning != Error. | 18:27 |
| tych0 | frobware: ok, my abttery is going to die. let me know what you figure out about aws | 18:33 |
| frobware | tych0: just testing one more path... | 18:34 |
| * frobware mulls over the fact that he is doing basic sanity bootstrap... that a machine and/or CI could do... | 18:39 | |
| frobware | tych0: not sure if you're still about but ... http://pastebin.ubuntu.com/15818135/ | 18:41 |
| frobware | tych0: that's your tree, bootstrapped as: JUJU_DEV_FEATURE_FLAGS=address-allocation juju bootstrap a1 aws --upload-tools | 18:41 |
| frobware | tych0: I'll need to dig into this more tomorrow | 18:41 |
| frobware | tych0: so, the problem above is because: | 18:48 |
| frobware | ubuntu@ip-10-229-48-181:/var/log/lxd/juju-machine-0-lxd-0$ sudo lxc profile list | 18:48 |
| frobware | docker | 18:48 |
| frobware | tych0: there is no default profile | 18:48 |
| mup | Bug #1568668 changed: landing bot uses go 1.2 for pre build checkout and go 1.6 for tets <juju-ci-tools:New> <https://launchpad.net/bugs/1568668> | 19:03 |
| === alexisb is now known as alexisb-afk | ||
| mup | Bug #1568668 opened: landing bot uses go 1.2 for pre build checkout and go 1.6 for tets <juju-ci-tools:New> <https://launchpad.net/bugs/1568668> | 19:18 |
| katco | natefinch: how's your stuff going? | 19:19 |
| natefinch | katco: I may have just found a key piece.... I still haven't figured out how to disable all the upload-tools stuff, but I just found a spot where we haven't set the arch when getting user data config (i.e. the thing that sets up cloudinit and which determines the tools we download) | 19:21 |
| natefinch | katco: maybe... still not sure | 19:21 |
| cherylj | katco: do you know if there was ever a plan to support nested lxd containers for the lxd provider (such that deploy --to lxd:# would work)? | 19:22 |
| katco | cherylj: that's certainly a use-case | 19:23 |
| cherylj | katco: but would we expect it to work now? (because we don't allow it because of the "can host containers" check) | 19:24 |
| katco | cherylj: i don't think we specifically added support for that in the provider | 19:24 |
| cherylj | ok | 19:24 |
| katco | cherylj: so right now it doesn't work because of a security setting? | 19:25 |
| rick_h_ | cherylj: katco that's the profile issue | 19:25 |
| cherylj | katco: I think it's because of the check that we do to determine if a machine can host a container | 19:25 |
| rick_h_ | cherylj: katco where you have to use the docker profile to get that to work | 19:25 |
| cherylj | the check is "are we a container" I think | 19:25 |
| katco | rick_h_: cherylj: right, that's where i was going... can we tweak the profile to disable that security check | 19:25 |
| katco | cherylj: oh, it's a juju thing, not lxd? | 19:26 |
| cherylj | wait, I'm thinking of a different bug | 19:26 |
| cherylj | maybe it's not that check that's failing | 19:26 |
| cherylj | (sorry, I'm slowly losing my mind) | 19:26 |
| cherylj | ah, yes it is the same error output: "cannot add a new machine: machine 1 cannot host lxd containers" but it doesn't fail at deploy time? weird. | 19:28 |
| cherylj | this is bug 1569106 for reference | 19:28 |
| mup | Bug #1569106: juju deploy <service> --to lxd:0 does not work with lxd provider <conjure> <lxd> <placement> <juju-core:New> <https://launchpad.net/bugs/1569106> | 19:28 |
| mup | Bug #1568668 changed: landing bot uses go 1.2 for pre build checkout and go 1.6 for tets <juju-ci-tools:New> <https://launchpad.net/bugs/1568668> | 19:30 |
| natefinch | katco or ericsnow: care to pair on this? A fresh pair of eyes may be a big help | 19:36 |
| ericsnow | natefinch: sure | 19:37 |
| katco | natefinch: trying to get this windows test going | 19:37 |
| ericsnow | natefinch: I need a break :) | 19:37 |
| natefinch | ericsnow: cool | 19:37 |
| natefinch | katco: np | 19:37 |
| natefinch | katco: good luck | 19:38 |
| === JoseeAntonioR is now known as jose | ||
| tych0 | frobware: wat | 20:37 |
| tych0 | frobware: how is there no default profile? that is odd. | 20:37 |
| thumper | morning | 20:45 |
| perrito666 | hi thumper | 20:46 |
| thumper | hey perrito666 | 20:47 |
| * perrito666 wonders what exactly did he change in his linkedin lately that provoked a wave of recruiters | 20:48 | |
| === terje is now known as Guest13601 | ||
| mup | Bug #1569072 changed: juju2 bundle deploy help text out of date <landscape> <juju-core:New> <https://launchpad.net/bugs/1569072> | 20:52 |
| === \b is now known as benonsoftware | ||
| natefinch | katc0, cherylj: pretty sure ericsnow and I figured out the problem with lxd on non-amd64. | 21:00 |
| cherylj | yay! | 21:00 |
| cherylj | natefinch: what was it? | 21:00 |
| natefinch | cherylj: we were getting a list of valid tools, and not filtering it for valid architectures before picking one off the list. It just happens to always be amd64 first... and just happens to be ignored during bootstrap | 21:01 |
| cherylj | nice | 21:02 |
| === _stowa_ is now known as _stowa | ||
| mup | Bug #1570096 opened: No way to remove user; remove-user command is missing <juju-core:New> <https://launchpad.net/bugs/1570096> | 21:28 |
| redir | that seems like a pretty solid bug | 21:34 |
| === axw___ is now known as axw | ||
| thumper | wallyworld: meeting? | 22:03 |
| wallyworld | thumper: on way, last meeting ran over | 22:03 |
| === aluria` is now known as aluria | ||
| perrito666 | wallyworld: I am leaving for like an hour or so, if you change your mind regarding the standup please have anastasiamac_ or axw contact me via some communication protocol from this century, preferably one that rings my phone such as twitter | 22:38 |
| wallyworld | perrito666: ok, will do | 22:39 |
| anastasiamac_ | wallyworld: axw: I proposed a quick fix for cloud lookup to include built-in providers consistently http://reviews.vapour.ws/r/4579/ | 23:05 |
| anastasiamac_ | cherylj: if u don't want it in beta4 ^^^, I can re-propose against next with more testing around the area... | 23:06 |
| thumper | trivial review for someone: https://github.com/juju/gomaasapi/pull/37 | 23:19 |
| thumper | axw: you on call? | 23:20 |
| axw | thumper: I am | 23:20 |
| axw | (currently reviewing in fact) | 23:20 |
| * axw enqueues | 23:20 | |
| axw | thumper: can we not change gomaasapi to drop the empty-value ?op= | 23:23 |
| thumper | axw: we *could*, but I'm more hesitant to change parts that are used already... | 23:23 |
| thumper | it is almost certainly likely to be fine | 23:24 |
| thumper | but still has me a bit squeemish | 23:24 |
| axw | thumper: ok | 23:24 |
| menn0 | axw: quick state API question: http://paste.ubuntu.com/15822102/ | 23:52 |
| menn0 | I think I prefer the latter | 23:52 |
| wallyworld | natefinch: how goes the non amd64 lxd provider fix? | 23:52 |
| ericsnow | wallyworld: we paired right before he had to go for dinner and it looks like we came up with a fix | 23:57 |
| ericsnow | wallyworld: natefinch should be back in a little while to wrap it up | 23:58 |
| wallyworld | ericsnow: oh, that is awesome | 23:58 |
| wallyworld | what was the root cause? | 23:58 |
| axw | menn0: sorry going to meeting, will look in a little while | 23:59 |
| ericsnow | wallyworld: the list of tools that gets passed in is basically "all available" and amd64 is the first one | 23:59 |
| menn0 | axw: no rush | 23:59 |
| ericsnow | wallyworld: the provider was simply using the first one in the list | 23:59 |
| ericsnow | wallyworld: needed to filter for arch first | 23:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!