| thumper | waigani: no real idea what is wrong with your clone | 00:40 |
|---|---|---|
| * thumper goes to make fud | 00:40 | |
| waigani | thumper: do you have the link to the prototype code for the btrfs stuff? | 02:05 |
| === mwhudson is now known as zz_mwhudson | ||
| === tasdomas` is now known as tasdomas | ||
| * fwereade out for a bit | 09:07 | |
| jam | dimitern: I'm going to miss the standup today, I have a management training course, can you keep everyone on track? | 09:17 |
| dimitern | jam, sure, no worries | 09:21 |
| hazmat | g'morning | 10:16 |
| natefinch | dimitern: standup | 10:48 |
| natefinch | mgz: ^ | 10:48 |
| ahasenack | guys, can someone look at these two bugs (dupes): lxc doesn't work with the maas provider | 12:22 |
| ahasenack | https://bugs.launchpad.net/juju-core/+bug/1274210 | 12:22 |
| _mup_ | Bug #1274210: Creation of LXC bridge in 1.17.0 fails <micro-cluster> <juju-core:Fix Released by rogpeppe> <juju-core (Ubuntu):Fix Released> <https://launchpad.net/bugs/1274210> | 12:22 |
| ahasenack | https://bugs.launchpad.net/juju-core/+bug/1271144 | 12:22 |
| _mup_ | Bug #1271144: br0 not brought up by cloud-init script with MAAS provider <landscape> <local-provider> <lxc> <juju-core:Triaged> <https://launchpad.net/bugs/1271144> | 12:22 |
| ahasenack | it's because bridge-utils is installed *after* the bridge is configured and networking is restarted | 12:22 |
| ahasenack | so br0 never comes up | 12:22 |
| ahasenack | a) configure bridge | 12:22 |
| ahasenack | b) restart networking (does nothing useful) | 12:22 |
| ahasenack | c) install bridge-utils | 12:23 |
| ahasenack | either brigde-utils should be installed before (b), or (b) should happen after (c) | 12:23 |
| ahasenack | oh, only happens with the 1.17 series | 12:23 |
| ahasenack | 1.16.6 works fine | 12:23 |
| hazmat | ahasenack, bridge utils gets installed early in the 1.17.3 | 12:47 |
| ahasenack | hazmat: not in the bootstrap node | 12:48 |
| hazmat | hmm | 12:48 |
| ahasenack | hazmat: the above happens in the bootstrap node, for some reason it's provisioned differently | 12:48 |
| hazmat | ahasenack, quite possible.. could you pastebin /var/lib/cloud/instance/user-data.txt | 12:49 |
| hazmat | on the bootstrap node | 12:49 |
| ahasenack | I don't have it anymore | 12:49 |
| ahasenack | but it's easy to reproduce | 12:49 |
| hazmat | yup | 12:49 |
| hazmat | no worries. | 12:49 |
| * hazmat connects to a maas env | 12:50 | |
| ahasenack | hazmat: I have a bootstrap node on maas almost up, if you're interested | 12:58 |
| hazmat | i've got one | 13:03 |
| hazmat | ahasenack, sure.. re pastebin of its that user-data on the bootstrap node | 13:03 |
| ahasenack | hazmat: http://pastebin.ubuntu.com/6994263/ | 13:04 |
| hazmat | ahasenack, oh.. i forgot.. synchronous bootstrap | 13:04 |
| ahasenack | hazmat: here is a bootstrap --debug from sparkiegeek: http://paste.ubuntu.com/6994260/ | 13:05 |
| hazmat | cool | 13:05 |
| hazmat | debugged in the other channel.. bridge only setup on nodes using userdata | 13:11 |
| hazmat | which doesn't include the bootstrap node.. for those following along | 13:11 |
| === edamato is now known as edu-afk | ||
| mattyw | dimitern, thanks for the reviews - I'll take a look at them and see what I can sort out | 13:35 |
| hazmat | well more specifically bridge-utils not installled on bootstrap node till after network create/restart. | 13:42 |
| hazmat | which is basically what you diagnosed ahasenack .. just a circle to find out how. | 13:42 |
| dimitern | mattyw, cheers | 13:47 |
| hazmat | fwereade, ping | 15:15 |
| fwereade | hazmat, pong | 15:15 |
| === allenap_ is now known as allenap | ||
| * fwereade bbiab again | 15:53 | |
| wrtp | fwereade, dimitern, natefinch: a review of this would be very much appreciated: https://codereview.appspot.com/68650044 | 18:47 |
| * wrtp is done for the day | 18:47 | |
| dimitern | wrtp, looking | 18:47 |
| fwereade | wrtp, enjoy your evening | 18:47 |
| wrtp | dimitern: thanks | 18:47 |
| wrtp | fwereade: will do! | 18:47 |
| wrtp | dimitern: there are a couple of drive-by fixes in there that could perhaps be factored out (the testing/mgo.go one prevents a race detector complaint) | 18:49 |
| dimitern | wrtp, ha! i was just wondering just about that | 18:49 |
| wrtp | dimitern: the change in voyeur just makes it more convenient to use | 18:50 |
| dimitern | wrtp, ok istm those are small enough to have them in the same cl | 18:50 |
| wrtp | dimitern: thanks; i thought so, but opinions differ... | 18:51 |
| natefinch | wrtp: looking, too | 18:54 |
| === zz_mwhudson is now known as mwhudson | ||
| stokachu | in juju 1.16 the bootstrap command didnt block but in 1.17 bootstrap runs synchronously and we are seeing it block for 10m | 19:34 |
| stokachu | b/c of the way are scripts work during the cloud installation | 19:35 |
| stokachu | should this be considered a bug? | 19:35 |
| stokachu | our* | 19:35 |
| wrtp | stokachu: most people consider it a feature | 19:41 |
| stokachu | wrtp: hah | 19:41 |
| stokachu | wrtp: did that behavior change between those versions? | 19:43 |
| natefinch | stokachu: yes, behavior changed recently. I believe it was between those two releases that it went from async to sync. It's not a bug, it's intended, because otherwise it's hard to know when bootstrap succeeds, without spamming juju status. | 19:59 |
| natefinch | stokachu: you can get previous behavior by simply running bootstrap asynchronously yourself | 20:00 |
| stokachu | just push to the bg? | 20:00 |
| stokachu | i thought there were talks on a switch | 20:01 |
| stokachu | even though its intended is it not a bug since that is a breaking change from 1.16 to 1.17 | 20:01 |
| natefinch | stokachu: it really shouldn't break anything unless you were counting on being able to get stuff done before the bootstrap node comes up, which is essentially purposely creating a race condition | 20:02 |
| natefinch | stokachu: at worst it should make your script take longer, but it shouldn't *fail* | 20:03 |
| natefinch | stokachu: there's a bootstrap-timeout value you can set in your provider to customize how long it'll wait for bootstrap to finish. It defaults to 10 minutes, which sounds like is not long enough for you. | 20:04 |
| natefinch | stokachu: there's no switch because you can just do juju bootstrap& ... so no real need for a switch | 20:05 |
| stokachu | ok | 20:05 |
| stokachu | bbcmicrocomputer: ^ | 20:05 |
| natefinch | stokachu: juju help bootstrap gives more details about customizing the timeout etc | 20:06 |
| stokachu | natefinch: yea we'll probably incrase the timeout and background it | 20:07 |
| stokachu | i think that'll work, thanks | 20:07 |
| natefinch | stokachu: welcome | 20:10 |
| thumper | fwereade: ping | 20:15 |
| fwereade | thumper, pong, I'm *finally* looking properly at that doc | 20:15 |
| thumper | fwereade: ok, could talk through a few ideas, may be helpful | 20:15 |
| fwereade | thumper, sure, give me 5 mins to grab a drink | 20:16 |
| thumper | kk | 20:16 |
| wallyworld | thumper: do you have time for a hangout? | 22:58 |
| wallyworld | sinzui: hi, you still around? | 23:23 |
| sinzui | I am | 23:23 |
| wallyworld | i found a problem with the metadata on https://juju-dist.s3.amazonaws.com/testing/tools/ | 23:24 |
| sinzui | really! | 23:24 |
| sinzui | well I shouldn | 23:24 |
| wallyworld | can you pull up https://juju-dist.s3.amazonaws.com/testing/tools/streams/v1/com.ubuntu.juju:released:tools.json | 23:24 |
| sinzui | t question you. | 23:24 |
| wallyworld | see line 4 | 23:24 |
| wallyworld | "version": "1.17.2 | 23:24 |
| wallyworld | the version attr at that top level overrides the version at the item level | 23:25 |
| wallyworld | so it masks the various versions | 23:25 |
| sinzui | waigani, so why did juju metadata do that? | 23:25 |
| wallyworld | and the 1.17.4 tools metadata records are hidden | 23:25 |
| sinzui | wallyworld, , so why did juju metadata do that? | 23:25 |
| wallyworld | that i don't know - so you get juju metadata generate-tools ? | 23:26 |
| wallyworld | used | 23:26 |
| sinzui | wallyworld, this looks the same, but it works. https://juju-ci.s3.amazonaws.com/tools/streams/v1/com.ubuntu.juju:released:tools.json | 23:26 |
| sinzui | wallyworld, remember the logs show that the tools url was not used! | 23:26 |
| sinzui | wallyworld, while we entered /testing/tools, juju used /tools | 23:27 |
| wallyworld | sinzui: what i did just now - i hacked juju so that it only uses tools-url, and then i did a juju metadata validate-tools | 23:27 |
| wallyworld | the debug shows tools-url is used | 23:27 |
| wallyworld | and shows that only 1.17.2 tools can be found | 23:27 |
| wallyworld | since that's the version at that top level | 23:27 |
| wallyworld | ah | 23:28 |
| wallyworld | 2014-02-25 23:21:03 DEBUG juju.environs.simplestreams simplestreams.go:535 using mirrored products path: https:/juju-dist.s3.amazonaws.com/tools/streams/v1/com.ubuntu.juju:released:tools.json | 23:28 |
| wallyworld | seems like it is getting the /tools from the mirrors file | 23:29 |
| wallyworld | let me check | 23:29 |
| sinzui | wallyworld, slow down you are trying to solve a hack, not the actual problem | 23:29 |
| sinzui | wallyworld, I am about to update juju-ci to use the proper url, you can watch the out and see that the tool-url is wrong, it is not what we entered | 23:30 |
| wallyworld | ok, but my debug shows tools-url is being used | 23:30 |
| wallyworld | and that it seems like mirrors processing then causes /tools to be used | 23:31 |
| sinzui | There are no mirror files on aws | 23:31 |
| sinzui | I set this tools-url: http://juju-dist.s3.amazonaws.com/testing/tools | 23:31 |
| sinzui | wallyworld, the log has started http://162.213.35.54:8080/job/aws-deploy/860/console | 23:31 |
| sinzui | and I see this | 23:32 |
| sinzui | tools-metadata-url: http://juju-dist.s3.amazonaws.com/testing/tools. | 23:32 |
| sinzui | and maybe to your point, the wrong juju was found | 23:32 |
| wallyworld | sinzui: while we wait, the cpc-mirros.json file on testing/tools has this | 23:32 |
| wallyworld | "mirror": "https://juju-dist.s3.amazonaws.com/tools", | 23:32 |
| wallyworld | "path": "streams/v1/com.ubuntu.juju:released:tools.json", | 23:32 |
| wallyworld | so my debug shows testing/tools is being used | 23:33 |
| sinzui | well not you your point, to my point, the wrong url was used, so it looked at what I released last week and used it | 23:33 |
| wallyworld | but that then the mirrors file causes it to redirect to /tools | 23:33 |
| sinzui | oh! | 23:33 |
| sinzui | we don't permit mirrors files on the CPCs | 23:34 |
| wallyworld | huh? | 23:34 |
| sinzui | only streams gets the file | 23:34 |
| wallyworld | i thought we did, so that the CPCs loaded tools locally | 23:34 |
| sinzui | streams.canonical.com get it, and any site I choose to use as a proxy | 23:34 |
| wallyworld | so you saying that testing/tools should not have a mirror file? | 23:35 |
| sinzui | waigani, but you are correct, that file got put there. | 23:35 |
| waigani | wallyworld: ^ | 23:35 |
| sinzui | wallyworld, I am say that, nor should any CPC. why put a mirror file to the site you are on | 23:35 |
| sinzui | sorry waigani | 23:36 |
| waigani | sinzui: no problem ;) | 23:36 |
| * sinzui starts purging | 23:36 | |
| wallyworld | sinzui: if that console output had --debug that would show more info to help diagnose the issue and the search path used by simplestreams | 23:37 |
| sinzui | wallyworld, I know, abentley doesn't care for that level of detail in the log | 23:39 |
| wallyworld | sure, normally i would agree. but it would have helped here :-) | 23:39 |
| sinzui | wallyworld, new run http://162.213.35.54:8080/job/aws-deploy/861/console | 23:39 |
| wallyworld | maybe we can turn it on when there's an issue | 23:39 |
| sinzui | the version is correct | 23:39 |
| wallyworld | sinzui: my validate-tools now passes | 23:40 |
| wallyworld | now that the mirros file is removed | 23:40 |
| wallyworld | $ juju metadata validate-tools | 23:41 |
| wallyworld | matching tools versions: | 23:41 |
| wallyworld | 1.17.4-precise-amd64 | 23:41 |
| wallyworld | it failed just before but works now | 23:41 |
| abentley | sinzui: It's not the level of detail that's the problem, it's the fact that it contain credentials. | 23:41 |
| sinzui | wallyworld, in testing_to_aws in http://bazaar.launchpad.net/~juju-qa/juju-core/ci-cd-scripts2/view/head:/publish-public-tools.bash we do filter *mirror*. it has always been that way, but If someone put one there, it will not be removed by the sync op | 23:41 |
| sinzui | does --debug contain credentials? Oh is the config output? | 23:42 |
| wallyworld | i would hope --debug doesn't contain credentials but if it does that sucks and should be fixed | 23:43 |
| wallyworld | sinzui: i've been thinking for a while i should add extra output to validate-tools to not only print the tools found but also the path from which they were got incl whether from tools-url vs cloud storage etc and whether a mirror was used | 23:44 |
| abentley | wallyworld: That's what I remember. | 23:44 |
| wallyworld | abentley: it may have been that way previously. i seem to have a vague, perhaps incorrect, recollection "someone" fixed it | 23:45 |
| wallyworld | i can verify that | 23:45 |
| wallyworld | s/can/will | 23:45 |
| sinzui | wallyworld, that would be a lovely diagnostic. When I report bugs like this, I do test manually with --debug, I saw the urls switch, but did not notice the mirror use | 23:45 |
| abentley | wallyworld: I believe it was printing out the config, as sinzui suggested. | 23:45 |
| wallyworld | abentley: i think config is now sanitised to exclude credentials, but am not 100% | 23:46 |
| wallyworld | sinzui: if i add the extra output, the ci scripts could do a validate-tools at the state and check the output is as expected before bootstrapping and failing | 23:46 |
| wallyworld | s/state/start | 23:46 |
| sinzui | wallyworld, they certainly can | 23:46 |
| wallyworld | same for image metadata | 23:47 |
| sinzui | success | 23:47 |
| wallyworld | \o/ | 23:47 |
| * sinzui starts the upgrade job | 23:47 | |
| * wallyworld crosses fingers | 23:47 | |
| sinzui | Finland has police raindeer. I bet the have fairy lights and bells instead of sirens | 23:49 |
| wallyworld | lol | 23:49 |
| * sinzui restores the url for the trusty tests | 23:51 | |
| sinzui | I am the likely person who put the mirror file there. I can't think how, but I will take the blame | 23:53 |
| wallyworld | sinzui: i do also want to get to the bottom of how we got those top level version attrs in the json. i'll check the ci scripts and metadata generation cmd in juju to see what might be going on | 23:54 |
| wallyworld | i'm just glad we found what was happening | 23:54 |
| wallyworld | and it gives me an excuse to spend the time to do better diagnostics | 23:54 |
| sinzui | fab, all works, now that all the mirror files are deleted. Thank you very much wallyworld | 23:59 |
| wallyworld | no problem. pleased to help. i'm so grateful for the CI stuff you guys have done | 23:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!