/srv/irclogs.ubuntu.com/2018/04/03/#cloud-init.txt

=== simpoir|afk is now known as simpoir
=== nacc_ is now known as nacc
Beretblackboxsw, seen this - https://www.dropbox.com/s/tqao6tbumfb0vbh/Screenshot%202018-04-03%2010.47.30.png?dl=0 before?15:54
dpb1Beret: if this is NUCs, typically I see that when disks start going bad?15:56
Beretit is15:57
BeretI figured15:57
Beretok15:57
blackboxswI haven't yet. was looking over other bugs saw andreas hit a bug related to that failure path, but it was zfs15:57
hexorgHello all17:46
hexorgI can't seem to find a particular answer for cloud init - can I ask it here?17:46
hexorgIs there a way to tell write_files module to wait until the users module is finished?17:47
blackboxswhexorg: ask away questions/discussion is always welcome in this channel17:47
blackboxswif someone doesn't know now, maybe others will be able later17:47
hexorgthanks :)17:48
hexorgI'm trying to write_files into a newly created user folder, but cloud-init seems to run write_files before users17:48
hexorgas a result, write files fails with no such directory17:48
blackboxswhexorg, generally,  ordering of module sequence  is not user- (#cloud-config) configurable, but the order in which modules are run is defined in /etc/cloud/cloud.cfg for each stage cloud_init_modules(run at init stage), cloud_config_modules: run in modules-config stage, cloud_final_modules: run last17:50
blackboxswlet me see if I can answer the specific question or if I have to pass.17:50
blkadderYeah I have run into that... Hacky way of dealing with it was to to write files to a temp directory then move them into place after user is created.17:53
hexorgYeah ok. Just making sure I'm not missing some more direct way17:54
blackboxswhexorg: right, write-files is run in init stage which happens in the init-network stage of cloud-init boot per http://cloudinit.readthedocs.io/en/latest/topics/boot.html. And  - users-groups17:54
blackboxsw lives by default at the end of that list because it might depend on files written by write_files.17:54
blackboxswyou might also be able to run your user creation logic in runcmd which runs in the cloud-init final stage (after both write_files and user creation modules)17:55
blackboxswwhich I think is what blkadder is referring to17:55
hexorgUnderstandable. Thanks!17:56
blkadderblackboxsw, Yep.17:56
rharperblackboxsw: almost done with your fixes17:57
blackboxsw+1 rharper, I've got to get my branch in shape for dropping ifconfig17:57
* blackboxsw tests the last branch for SRU now18:40
blackboxswhttps://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/34200718:40
=== Raboo_ is now known as Raboo
rharperplatform: lxd encountered error: 'Operation' object has no attribute 'description'19:21
rharperblackboxsw: powersj thoughts?19:21
rharperoh, I think that's the parsing of the cloud-init result/status json files19:23
rharperthat likely happens if it's not yet booted19:24
blackboxswmeh rharper I'm going to reject exception_cb  branch, the raising of exceptions in principle makes sense, but the logic checking for exc.code doesn't seem to behave as expected on 404s (the httpcode isn't attached to the UrlError raised)19:45
rharperok19:45
blackboxswso, I think more rework is needed there, and we'll need discussion on it19:46
rharperI read that code multiple times, but really needed either more unittests or integration tests to validate exactly what behaviors we want19:46
rharperI think we should have a series of unittests that cover the various expected behavior paths we need, and then run this against that19:46
blackboxswyep, and the existing code I believe actually doens't work right19:50
blackboxsweven before the rewrite. or the previous rewrite19:50
blackboxswnot a critical issue (as it'll ultimately retry more than it is supposed to) which costs time, not functionality19:50
blackboxswbut yeah something smells a bit there19:51
blackboxswyep confirmed, that exception_cb refactor needs work, I confirmed that even the implementation smoser took was inconsistent. after we SRU I'll put up a branch which adds unit test coverage to examine proper exception raising behavior from readurl.20:15
blackboxswwe can pay this "risk" cost on next SRU.20:15
blackboxswin terms of having to retest on the affected clouds20:16
blackboxswto make sure there isn't a regression20:16
blackboxswok I'm putting up bionic merge proposal now20:16
blackboxswrharper: here's the proposal for syncing tip to bionic https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/34260520:25
blackboxswrharper: I'm putting together the SRU for xenial and artful now (should have the same content bump)20:26
blackboxswxenial SRU: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/34260620:35
blackboxswrharper: artful SRU too https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/34260820:39
rharperblackboxsw: ok, reviewing20:39
blackboxswthere are the three release candidates20:39
blackboxswthanks20:39
* blackboxsw ran the new-upstream-snapshot from qa-scripts https://github.com/cloud-init/qa-scripts/blob/master/scripts/new-upstream-snapshot20:39
blackboxswmeh my comments on smoser's branch are as follows rharper  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/34200720:43
bjonnhHi, I'm trying to disable ipv6 in an lxd container (ubuntu), I got cloud-init to write the sysctl for it20:44
blackboxswsorry for the thrashing. only one minor diff is required in his implementation, but I'd feel better if we got some good unit test coverage on the function20:44
bjonnhbut I can't get it to restart systemd20:44
dpb1bjonnh: does lxd have a setting to do that?20:45
bjonnhdpb1: NO…20:45
bjonnhand they say "just set the sysctl"20:45
bjonnhipv6 is disabled on host20:45
nacc_bjonnh: restart systemd?20:46
bjonnhbut containers still get an ipv6 link-local20:46
bjonnhnacc_: sorry restart a systemd service20:46
nacc_bjonnh: ah ok :)20:46
blackboxswhttps://github.com/lxc/lxd/issues/333320:46
bjonnhoh20:46
bjonnh- [systemctl, restart, systemd-sysctl]20:46
bjonnhI think I had to put "" around systemd-sysctl20:46
blackboxswsee alberto's comment about disabling ipv6 in containers20:46
blackboxswif that helps20:46
blackboxswlxc network set lxdbr0 ipv6.address none20:46
bjonnhis it possible to start something really early in cloud-init with my user conf?20:47
bjonnhblackboxsw: I can't do that because I'm using my own bridge20:47
dpb1yes, what blackboxsw said20:47
bjonnh(that doesn't have ipv6…)20:47
bjonnhso lxc complains that it cannot manage my device20:47
dpb1bjonnh: you want this globally or per container?20:47
bjonnhglobally20:48
bjonnhI don't have anything ipv6 here20:48
rharperblackboxsw: approved push to ubuntu/devel, I got the same you did20:49
dpb1bjonnh: so, can't you just reconfigure your bridge to not have anything ipv6?20:50
bjonnhdpb1: that's my point… It doesn'…20:50
blackboxswrharper: will push to ubuntu/devel and see if we can get an upload there.20:50
dpb1bjonnh: sorry, don't follow that one20:50
rharperI don't think you can disable the kernel ipv6 setting from within an unpriv container20:51
bjonnhthe host has ipv6 disabled20:52
rharperwith what setting? are you ignore RA s ?20:52
bjonnhnet.ipv6.conf.vlanbr2.disable_ipv6 = 120:52
bjonnhshould I do20:52
bjonnhnet.ipv6.conf.vlanbr2.accept_ra = 020:52
bjonnhtoo?20:52
rharperyes20:52
bjonnhoh20:52
rharperthat will prevent any RAs from showing up on your interfaces20:53
* dpb1 hesitates to ask why the need to disable ipv6 on this host :)20:54
rharperWhen this value is changed from 0 to 1 (IPv6 is being disabled),20:54
rharperit will dynamically delete all address on the given interface.20:54
rharperI suspect that at the time it's set, it drops addrs, but if you accept RAs then new ones can come in20:55
bjonnhdpb1: because I have nothing ipv6 and the update of packages throws me:  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::16). - connect (101: Network is unreachable) [IP: 2001:67c:1562::16 80]20:55
bjonnhand waits for a second then switch to the ipv620:55
bjonnhipv4 sorry20:55
dpb1bjonnh: yet the router is advertising it?20:55
bjonnhmaybe dnsmasq is doing something on its side20:56
bjonnh20:56
dpb1that wouldn't totally shock me :/20:56
blackboxswpowersj: pylxd issue we've seen before? https://pastebin.ubuntu.com/p/VZHHpHD4WN/20:58
blackboxswper ci build https://jenkins.ubuntu.com/server/job/cloud-init-ci/968/console20:59
powersjblackboxsw: yes usually that's when pylxd is out of sync with lxd20:59
powersj:\ so hopefully that isn't the case21:00
blackboxswpossible given the release push on friday I suppose21:05
rharperblackboxsw: what's the xenial and artful sru bug numbers ?21:05
rharpernew-upstream-snapshot said something to me about that21:06
rharperhrm, your xenial branch didn't have the changelog update ?21:07
rharpernor the artful one21:07
blackboxswrharper: the SRU bug is https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/175940621:08
ubot5`Ubuntu bug 1759406 in cloud-init (Ubuntu) "sru cloud-init (17.2-35-gf576b2a2-0ubuntu1~16.04.1 update to 18.2-0ubuntu1~16.04.1)" [Medium,Confirmed]21:08
blackboxswhrm, checking xenial21:08
rharperblackboxsw: shouldn't we see a changelog diff between your branch and origin/ubuntu/xenial ?21:08
rharperlike we did for ubuntu/devel ?21:08
blackboxswrharper: line 100 of the visual diff at https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/34260821:08
* rharper is blind21:09
rharperyes21:09
rharperconfirmed blind21:09
* rharper continues21:09
bjonnh|  eth0  | True | fe80::216:3eff:fed9:8c65/64 |  still gets a link-local21:09
bjonnhrouter has ipv6 fully disabled21:09
blackboxswthe changelog diff between xenial and devel debian/changelogs should only have minor diffs in package version numbers & maybe the new-upstream-snapshot which hasn't been removed from bionic21:10
* blackboxsw is still fumbling around with why pylxd is complaining (as I thought we pinned it in tox integration-requirements.txt)21:12
blackboxswbjonnh: hrm, I'm not quite sure on the ipv6 container at the moment, maybe somebody else has a clue there.21:13
powersjblackboxsw: pylxd probably didn't change, but lxd could have21:13
bjonnhso I'm able to disable it but this happens after the package upgrade21:13
bjonnhthat cloud-init does21:13
powersjblackboxsw: ah yes... lxd 3.0 is now installed21:13
powersjthat happened yesterday21:13
powersjblackboxsw: https://github.com/lxc/pylxd/issues/28421:15
dpb1bjonnh: honestly, I'd ask about this in #lxcontainers.  it's weird to me that you are having to try to workaround this in cloud-init on each instance21:16
dpb1bjonnh: you should do it on the host21:16
rharperblackboxsw: xenial is the same, though you put xenial-proposed in the release ? is that what we normally do /21:18
bjonnhwell I'm using a bridge21:18
bjonnhso it is an instance by instance problem21:18
rharperblackboxsw: same for artful21:18
bjonnh(it is not the lxd bridge, it is a bridge over a vlan on its specific subnet)21:19
rharperbjonnh: and your link-local ipv6 interrupts apt ?21:19
bjonnhinside the instances yes21:19
bjonnhit slows them down21:19
bjonnhit does it only during startup21:19
bjonnhafter that I'm able to set the required sysctls21:19
bjonnhso it stop allowing ipv621:20
rharperI'm surprised, I've no ipv6 available here but I wouldn' think the ipv6 addr for the archive is reachable via the link-local, wouldn't think it would try21:20
bjonnhme neither…21:20
bjonnhI've never seen that…21:20
rharpernor I21:21
rharperso something is special about this setup I think21:21
blackboxswrharper: for each release xenial and artful we should run dch --release -D artful-proposed or xenial-proposed for the debian/changelog to match the former released stream in debian/changelog21:25
blackboxswat least as the final step prior to the upload21:25
rharperblackboxsw: ok, I wasn't sure21:25
blackboxswso I have always dch --release -D artful-proposed     or xenial-proposed instead of UNRELEASED21:25
rharperI do the dch release21:25
rharperit was whether it should have -proposed or not21:26
rharperit seems (to me) strange to put a pocket value into the changelog when it's going to get copied over into the archive21:26
blackboxswyeah we decided in  changelog we want to leave it all as -proposed to indicate when we started performing SRUs for a given stream21:26
rharperbut maybe there's some backend magic that fiddles that value in the change log21:26
rharperok21:26
blackboxswbecause any changelog entries before the first SRU would have the base 'xenial|artful'21:26
rharperyeah, makes sense21:27
blackboxswyeah nothing seems to fiddle with it post-release: https://pastebin.ubuntu.com/p/b2639pyZtx/  that's from apt-get changelog on xenial21:28
blackboxswxenial-proposed still listed in there21:28
blackboxswwas wondering whether it'd get scrubbed21:28
rharperhehe21:29
* rharper relocates back home21:30
blackboxswyeah that pylxd traceback started happening between Apr 2, 2018 8:27 PM and  Apr 3, 2018 7:11 PM21:37
blackboxswand looks like it affects rharper's ntp branch too21:37
blackboxswok so I feel good this isn't related to the branches I put up against ubuntu/devel|artful|xenial21:38
blackboxswbut need to fix ci21:38
blackboxswtrying to reproduce the problem locally21:38
powersjblackboxsw: you can also hop on the CI box21:44
blackboxswit'll be faster.... locally on my xenial box, no error. trying on my other box now not seeing it either21:45
blackboxswwill do21:45
* blackboxsw digs up the doc21:45
powersjblackboxsw: lxd version?21:45
blackboxsw2.0.1121:45
powersjneed 3.0 ;)21:45
blackboxswyep need the snap looks like21:46
powersjor bionic ;)21:46
blackboxswsnap == faster path to the failure I'm expecting21:47
blackboxsw:O021:47
powersjyeah21:47
powersjmuch faster21:47
rharperpowersj: blackboxsw: is ci back up now? we did some backend storage work for lxd21:52
powersjrharper: it is, but it appears we did get a lxd 3.0 upgrade last night21:53
blackboxswpowersj: rharper I can't ssh as ubuntu to ci21:53
powersjblackboxsw: jenkins@21:53
rharperblackboxsw: Ill import you to ubuntu as well21:53
rharperblackboxsw: your lp name ?21:53
rharperI don't see you in either key files21:54
blackboxswssh-import-id chad.smith21:54
rharperok, in as ubuntu21:54
blackboxswthx21:54
* blackboxsw can take over the world now21:54
blackboxswthx21:54
rharperand jenkins21:54
rharperwhat's the pylxd trace back ?21:55
blackboxswrharper: https://pastebin.ubuntu.com/p/VZHHpHD4WN/21:55
rharperso, lxd pushed 3.0 into the stable branch ?21:55
rharperthat doesn't seem right21:55
powersjsnap info lxd21:55
rharperso I thought that was related to the cloud-init result.json but that's really pylxd ?21:56
powersjyep21:56
rharperwe can switch to 2.0 track21:57
rharperFor the LXD snap, 3 tracks are provided:21:57
rharperlatest (latest LXD feature release, currently 3.0)21:57
rharper2.0 (previous LTS release)21:57
rharper3.0 (current LTS release)21:57
powersjI'm fine with moving to 2.0 temporarily, especially if you are trying to get a release out21:58
rharperif they're not going to release pylxd in step with the base, then we need to run  behind tip21:58
rharperand dpb1 I'd like to raise this as an issue with the lxd team21:58
rharperwe continually get broken every single time they change21:58
powersjwell... pylxd did get updated to fix things21:58
powersjwe just hard code the version21:58
rharperbut not *before*21:58
rharperthe release21:58
rharperit should block a release21:58
powersjit was updated before21:58
powersjmonth ago or so21:58
rharperso, then I'm confused21:58
rharperoh, it's not packaged with lxd ?21:58
powersjcorrect21:58
rharperbut there is a dependency ther e21:59
rharperthat's still crappy21:59
powersjit is21:59
powersjfrustrating even21:59
rharperI suspect this is one of those snap not-yet-solved thingys21:59
powersjsmoser and I chatted about getting rid of pylxd at sprint21:59
rharperit's supposed to be stand alone ?21:59
rharperyeah21:59
dpb1yes, that ^22:00
rharperdpb1: that said, openstack has to have this problem as well22:00
rharperthey're not going to switch to a cli anytime soon22:00
rharperone should be able to express dependencies between snaps, or the snap (lxd) would need to provide the pylxd bindings in the snap22:00
rharperpowersj: so we switch back to 2.0 channel or can we bump the pylxd or do we have to change the ci call ?22:01
powersjeither a) switch back to 2.0 to fix things quickly and move on or b) bump pylxd (which we will have to do eventually anyway)22:02
dpb1yes please, let's focus on practicle.  we can corner stgraber at the sprint22:02
* blackboxsw just reproduced the issue on jenkins workspace22:02
blackboxswok22:02
powersjI'd prefer to update tox.ini to use a newer pylxd22:02
powersjthat way we keep using lxd 3.0 and move on22:03
blackboxsw+122:03
powersjand can talk about this at later date22:03
blackboxswI'm updating now to test22:03
blackboxswhrm just updating to tip of github/lxc/lxd isn't cutting  it . lemme do a tox -r -e to make sure it actually pulled in latest22:04
powersjyeah good idea to blow away .tox22:05
powersjor do that22:05
rharperurg22:05
rharperone more commit to master =P22:06
blackboxswyeah will have to respin on that22:06
blackboxswso tomorrow for SRU22:06
blackboxswI'll have the branch queued and landed in tip tonight with powersj blessing, then we can do the dance on bionic artful xenial tomorrow22:07
blackboxswnote grabbing tip of pylxd hits another traceback that'll need a tiny tweak to integration tests :)22:08
powersjpaste?22:08
blackboxswenroute22:08
blackboxswheh version 3 :)22:08
blackboxswhttps://pastebin.ubuntu.com/p/8WtSTRKCTh/22:09
powersjooooo yes22:09
powersjthe logging22:09
blackboxswthere'll never be a v. 3 :)22:09
powersjwell the issue we were having with v1 and v2 should be fixed in v322:09
powersjwhich is why that is there22:09
powersjit has to do with console logging with the lxd snap22:10
powersjwow smoser and I messed up there :)22:10
rharperoi22:10
rharperare we sure we don't want to just revert to 2.022:10
blackboxswstr has no attribute startwith22:10
rharperand sort this lxd/pylxd/ci mess out later ?22:10
rharperthat's typo22:11
blackboxsw:)22:11
blackboxswa tiny little s22:11
rharperunless blackboxsw typo'd irc22:11
blackboxswnope official committed typo22:11
rharper=/22:11
rharperhow'd flake8 not get that ?22:11
blackboxswgood pt22:11
rharperor pylint22:11
blackboxswflake8 look at cloud_tests?22:11
blackboxswor ignore that dir22:11
powersjit should look at tests22:12
blackboxswnope both flake and pylint look at tests, right22:12
rharper cloudinit/ tests/ tools/22:12
powersjwhich has unit and cloud tests22:12
rharperwhat file ?22:12
blackboxswtox.ini22:13
powersjfailure was in tests/cloud_tests/platforms/lxd/instance.py", line 213, in _has_proper_console_support22:13
blackboxswnext failure: powersj: rharper: https://pastebin.ubuntu.com/p/nykVTpBQrh/22:13
rharperI found it, I  meant the startwith22:13
rharperit's in instance.py22:13
rharpermy flake8 says something about local variable e22:13
rharperso maybe it's a lint issue22:14
blackboxswto speed up iterations, I'm running tox -r -e citest -- run --verbose --os-name xenial --test modules/apt_configure_sources_list.yaml --platform lxd22:14
powersjlxc is not operational on torkoal22:14
powersj$ lxc list22:14
powersjError: Get http://unix.socket/1.0: dial unix /var/snap/lxd/common/lxd/unix.socket: connect: no such file or directory22:14
rharperok, bbiab22:14
blackboxswwell that could cause problems ;)22:14
rharpershocking that flake8 and pylint don't care22:15
dpb1powersj: group/permissions errors?22:15
powersjhmm that socket file doesn't exist22:16
powersjblackboxsw: try now22:21
powersjfwiw looked at https://github.com/lxc/lxd/issues/424522:22
blackboxswpowersj: yep good find22:24
blackboxsw... runs fine now with tox and cloud_test patch22:25
blackboxswgetting patch together22:25
blackboxswhttp://paste.ubuntu.com/p/sVNf2nKCnS/22:26
blackboxswsetting pin now22:26
blackboxswok pin works http://paste.ubuntu.com/p/DQ799wxc4K/22:29
powersj+122:29
blackboxswpowersj: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/34261722:33
dpb1what did you do?22:33
dpb1to fix it?22:33
blackboxswpowersj: fixed the world. I just changed pinned version and fixed cloud_tests22:35
blackboxswpowersj: did you sudo snap refresh lxd ?22:35
blackboxswper that issue?22:35
powersjblackboxsw: I did a sudo snap refresh lxd and a sudo snap restart lxd22:35
powersjlxc list sat there for 2mins and then the world worked22:36
powersjblackboxsw: you left in a debug statement22:36
powersjblackboxsw: in cloudinit/url_helper.py22:36
blackboxswbah powersj I had uncommitted changes in that branch that I pulled in unknowingly... repushing in 2 mins22:37
blackboxswpowersj: force pushed. https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/34261722:39
powersj+1'ed22:40
rharperso no idea why lint or flake didn't find it ?22:46
blackboxswthanks powersj, yeah just awaiting completion of https://jenkins.ubuntu.com/server/job/cloud-init-ci/970/22:50
rharperso strange22:50
blackboxswand looks good22:50
powersjblackboxsw: yea ship it22:50
=== nacc_ is now known as nacc
blackboxswok landed22:52
blackboxswwill repropose branches xenial|artful|devel tonight22:53
blackboxswbut need to make some dinner at the moment22:53
rharpersomething about our .pylintrc in cloud-init blocks it22:55
rharperif I put a simple test into a different dir, then I get22:55
rharperModule test22:55
rharperE:  2,31: Instance of 'str' has no 'startwith' member (no-member)22:55
powersjinteresting sine we specifically allow errors22:56
rharperyeah, havent' tracked down the line yet22:56
* blackboxsw repushed https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/34260522:58
blackboxswfor bionic releaes22:58
rharperno, that's not it22:58
rharpersomething else inthe structure, I removed .pylintrd and didn't find it either22:58
rharperhrm, so, info is a dict (load_yaml), then we have two gets, which return a value from the dict, which it cannot know23:02
rharperso, if you str(dver) in there23:02
rharperthen pylint finds it23:02
blackboxswjust force pushed xenial and artful branches23:02
blackboxswneed to await CI on them23:02
blackboxsw-> dinner23:02
rharperbut, our .pylintrc still isn't happy with that23:02
rharperoh man23:10
rharperpylint just does a regex on the source file23:10
rharperthe http.client and m_.* have pylint ignore that file23:11
blackboxswmeh we should improve/limit that ignore if we can23:16
rharperI don't know what to do about that23:17

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!