smoserpowersj, blackboxsw yeah, that kind of stinks. basically i think we *have* to detach package build dependencies form tox requirements.txt01:38
smoserwhich i think might in-advertantly have been why it was the way it was when blackboxsw found it and thought WTH01:38
smoserthe package build on xenial didn't magically start depending on python3-pycodestyle just because trunk moved some versions in its tox.01:39
smoseras an example, see that https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial did not break because of this change.02:09
askbcan anyone assist with this https://bugs.launchpad.net/cloud-init/+bug/169242402:33
ubot5Ubuntu bug 1692424 in cloud-init "util.py[WARNING]: Failed to disable password for user centos" [Undecided,New]02:33
askbIs there are doc for building the latest verion on centos7 ?02:34
=== sambetts|afk is now known as sambetts
=== rangerpbzzzz is now known as rangerpb
smoseraskb, can you get a cloud-init.log from the system ?13:34
askbsmoser, yup13:35
askbdo you want me to upload the logs ?13:35
askbsmoser, http://paste.openstack.org/show/610552/13:38
askbsmoser, forgive my ignorance, but are there any docs for building the rpm/srpm for centos713:44
askbI noticed that there is a spec.in13:44
askb"make rpm" does not work13:45
smoserpowersj, blackboxsw https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/32454113:55
smoseraskb, hm...13:55
smoserlet me find13:55
smoserI had https://gist.github.com/smoser/19e65095b342e98fd4d6466e4d4fa1ac13:55
smoserwe're workign on getting rpm build into daily c-i, and i'm not sure what is being used for that.13:56
smoserhmm.. i see ./packages/brpm is busted. odd13:57
smoseraskb, i thik that gist will work. brpm is failing for me here, but because i dont have cheetah installed and the spec file needs it.14:01
powersjsmoser: yeah the build test that I pointed out is what uses packages/brpm14:01
powersjI expect that to work :)14:01
smoserpowersj, we need to move the spec.in to be jinja2 rather than cheetah14:02
powersjand we had a centos 6 test case fail last night14:02
powersjI'm going to go eat/wake-up and then I'll file bugs and we can chat about it after standup as far as what you want to see for fixes14:02
askbare these the same errors you are getting http://paste.openstack.org/show/61055714:03
askbsmoser, ^^14:03
smoseraskb, i suspect that whatever you're seeing is fixed in a newer release. 0.7.5 is quite old.14:03
smoseraskb, yes.14:03
smoseryou'll need python-cheetah installed14:03
smoserwhich that gist will install for you.14:03
smoserie, if you grab that 'centos-setup' and run ./centos-setup build14:04
askbI am not building it with lxc ... instead straight on centos7 virt env ?14:05
askbshould I be using the lxc instead14:05
smoserwell, the script will install the deps for you14:05
smoseri jsut showed how to do it all with lxc14:05
smoserbut all it amounts to is runing that script inside the container14:05
smoserwhich should be not really any different than inside a centos714:06
askbyou mean from the gist above correct?14:06
smoser"virt env" == "kvm guest" or python virtual env14:06
askbpython virtual environment14:07
smoserwell, package build is going to use system ddependencies...14:07
smoserpowersj, i'im not sure what is changed there.14:13
smoserbut tests/cloud_tests is python3 only. and that syntax is not python2.6 compatible14:13
askbsmoser, Is the script to be run the - centos-setup - on debian/ubuntu ?14:16
smoserto build an rpm you really need to be in a centos (or rhel) environment14:16
smoser./centos-setup build14:17
smoserwill install the necessary packages to build14:17
smoserthen you can run14:17
askboh this says "lxc: containers not found" on centos714:17
smoserand build a package14:17
smosersure. skip the lxc portion, just use centos-setup14:17
smoserwget that and chmod and execute it.14:17
askbsmoser, I was able to build an rpm - thanks!14:38
askbsmoser, will update a bit later how it goes with the testing since its getting 1am here14:38
smoseraskb, good luck14:39
smoserand sleep tight14:39
askbsmoser, quick question: how do I build the latest stable version/release ?14:39
smoseryou can git checkout 0.7.814:39
smoserand do the same ./packages/bdrpm14:39
askb0.7.9 is the head ... of the branch now ?14:40
smoser0.7.9 is newest release14:41
smoser'master' branch is trunk14:41
askbawesome! :)14:42
smoseraskb, i updated that gist README hoepfully to be more clear14:44
smoserand to down-play the lxd thing so as to not confuse people that.14:44
smoserblackboxsw, updated https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/32445016:30
blackboxswthanks+1 let's land I've added https://trello.com/c/iJdLSnh4/91-add-root-dir-lookup-for-unittests16:33
blackboxswheh, I'll steer you later or tomorrow to the real meat: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/324499 :)16:34
=== sambetts is now known as sambetts|afk
blackboxswit's up for review. I'm working on the old git review queue for cloud-init at the moment16:35
blackboxswminor comment and approve of https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/32427416:42
smoseryeah, got to add that.16:42
smoserpowersj, are you looking for fix centos6 ?17:51
powersjsmoser: I wasn't planning on looking at it today, but if you have something let me know and I can ack or test or merge or whatever :P17:52
smoserblackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/32455717:55
smosercan you ack that quick ? its just part of17:55
smoser https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/32427417:56
smoserbut the part that *has* unit test17:56
harlowjasmoser and others http://lists.openstack.org/pipermail/openstack-dev/2017-May/117401.html17:57
harlowjai responded a little bit, but feel free to :-P17:57
smoserharlowja, thanks. i dont have mch to add there.18:02
blackboxswapproved the branch smoser18:38
blackboxswsmoser: I'm reviewing https://code.launchpad.net/~akaris/cloud-init/+git/cloud-init/+merge/322992 and forgot what your thoughts were on us using python's netaddr instead of trying to carry our own burden of mask2cidr cidr2mask calculations19:15
blackboxswI was debating about cobbling up a quick suggestion branch w/ unit tests for the review if you netaddr is a  likely candidate for replacing this somewhat fragile code19:15
blackboxswif you *think* netaddr is a likely candidate....19:16
blackboxswbut again, maybe cloud init wants to reduce requirements on external packages for the sake of speed, dunno.19:17
blackboxswit's nice to have simple validation functions like valid_ipv4, valid_ipv6 IPNetwork.netmask builtins19:19
smoserblackboxsw, python2.6 ENO_NET_ADDR19:21
smoserand even the rhel 6 version of python2.719:22
smoserblackboxsw, i was looking at that too and trying to clean it up abit19:25
blackboxswwon't touch it since you are playing in that space, but ping me for a review on it when you are done. I don't want to miss what approach you are looking for.19:26
blackboxsws/review/review please/ :)19:26
smoserblackboxsw, http://paste.ubuntu.com/24646853/19:33
smoserthat is what is in my buffer right now19:33
smoseri was looking at xnox's cidr branch19:33
smoser(which had the same change as akaris)19:33
blackboxswsmoser: why not mask = str(mask) in line 162 of your paste, that was you are properly converting to the expected type for both ipv4 & ipv6 handling.19:36
smoseryou just ean19:37
smoser mask = str(mask)19:37
smoserif : in mask:19:37
smoserwell, with proper quotes19:37
smoserright ?19:37
blackboxswthen you don't have to perform str(mask) twice19:37
blackboxswalso that function doesn't have a return or error if a string w/out . or : is presented19:38
blackboxswdoes   subnet['prefix'] = mask_to_net_prefix(subnet.get('netmask'))   # Where None is returned  cause any problems? (line 52 of your paste)19:40
blackboxswheh and I have no idea what's happening on line 8519:41
blackboxsw87 rather19:41
blackboxswWIP paste I bet19:42
blackboxsw+1 I like the general approach in either case. given netaddr DNE on some of our support matrix19:47
jbrowneHello, I'm running into a race condition between cloud-init and apt-daily.service on Xenial.  This was mentioned in passing in https://bugs.launchpad.net/cloud-init/+bug/1594576, but I don't see a cloud-init issue open specifically about it.  Warning: small wall of text to follow.19:56
ubot5Ubuntu bug 1594576 in cloud-init "cc_salt_minion behaves badly if apt-get update fails" [Medium,Triaged]19:56
jbrownecloud-init when installing packages is encountering a lock on APT due to the apt-daily service running simultaneously.19:57
jbrowneRunning a dead simple user data, I see that the stock AWS Ubuntu Xenial AMI does not encounter the race condition as the timer state is:19:57
jbrowneThu 2017-05-25 15:34:01 UTC  20h left      n/a  n/a    apt-daily.timer              apt-daily.service19:57
jbrowneWhile the timer state of my packer-built AMI (based on the stock AMI) is:19:57
jbrowneFri 2017-05-19 08:40:01 UTC  5 days ago    Wed 2017-05-24 19:19:21 UTC  4s ago apt-daily.timer              apt-daily.service19:57
jbrowneSearching extensively (including IRC logs) I haven't found any discussion in the cloud-init project about this issue and am happy to open an LP bug.19:57
jbrowneThere's a lot of discussion about when unattended upgrades should run, etc. and if it is a systemd issue, but regardless it seems like cloud-init should be defensive about this.19:57
jbrowneI would naively argue that cloud-init should retry on apt issues, but I see that was deemed unneded in another case (https://bugs.launchpad.net/cloud-init/+bug/1594967)19:58
ubot5Ubuntu bug 1594967 in cloud-init "cloud-init lack of retry causes apt-get failures to be fatal" [High,Fix released]19:58
jbrowneOne are in which I am ignorant is how the actual AMIs are built; I'm trying to determine if something specifically clears systemd's timer states when making an image, etc.19:58
jbrowneWas digging into cloud-image-utils and then decided to just post here since that area also appears to be s-mosers.19:58
blackboxswjbrowne: peeking for context.  Without context of cloud-init's intended behavior, I would prefer apt to retry upon failure, or checksum mismatch which has hit me multiple times in other projects. Let me see if I can get more context on this specific issue. Thanks for "the deets".19:59
jbrowneBasically if one uses cloud-init for:20:00
jbrowne  - awscli20:00
jbrowneand systemd has launched some other apt related process concurrently there can be a lock contention20:00
jbrowneI have a slew of kinda related bugs I've found in other projects that build their own AMIs (chef, vagrant, etc.) and have hit this lock contention with cloud-init, but I didn't want to dump an entire thesis into the channel. :)20:01
blackboxswI'd naively like to see at least a simple retry fallback mechanism (maybe 3 retries on a backoff algo) before failing. Yes dpkg lock contention has hit a myriad of other projects I've been on. I'd think we could build in a simple retry/log effort before failing.20:02
blackboxswbut, I'm just reading through the cloud-init code now (and your references) to see what the existing state of affairs is.20:03
blackboxswwe've recently talked side-channel about the potential of adding a decorator to retry certain functions that are known to get a benefit from retries on expected failure. This might be a case where something like that could come in handy.20:08
blackboxswit *looks* like we could potentially build in a retry mechanism just in Runners.run of cloudinit/helpers.py.  I'll peek at whether that's viable.20:16
smoserblackboxsw, http://paste.ubuntu.com/24647424/20:42
smoserthats where i am now.20:42
smoserjbrowne, why would systemd launch some other apt related process ?20:42
blackboxswyowsa smoser20:42
jbrowneapt-daily is no longer in /etc/cron.daily.  Now on a timer20:43
jbrowneIf your machine is down during the trigger time (or a custom built AMI) systemd will trigger that timer during boot -> apt-get update in another process20:43
smoser http://paste.ubuntu.com/24647437/20:43
smoserjbrowne, ack on that. that is a mess. cloud-init should probably disable that thing and then re-enable it.20:44
jbrowneThere's a lot of chatter about how this has changed when unattended updates run and conflict between not running during boot and machines that are up for short periods of time (laptops), etc.20:44
smoserthat is generally garbage really.20:44
smoserno user can sanely run 'apt-get update' or 'apt-get install' because some system thing might be doing something.20:44
smosershoudl really be fixed at the apt level20:44
jbrowneI can see the argument for using systemd timer instead of cron.daily due to it blocking other cron jobs, but changing fundamental behavior like this bugs me.20:45
smoserblackboxsw, second paste. i pasted the file not diff.20:45
jbrowneAlso unattended upgrades is on by default now in Xenial, but that's a whole 'nother story. :)20:45
blackboxswyeah I'm on 2nd paste now. it looks a lot simpler20:45
blackboxswsmoser: not sure, might have to default to string for the get on line 25 +                if (subnet.get('type', '').endswith('6') or20:46
jbrowneThe reason I brought this up in channel rather than just opening an LP bug was to get a sense of how the cloud-init team feels about it.  My current work around is to have packer disable the apt-daily.service in systemd and I re-enable it on launch.20:46
blackboxswif type isn't present, subnet.get would return None and you can't call endswith on that20:46
smoserjbrowne, yeah. you want to file a bug ? you can file against cloud-init.20:47
smoserbut it seems honestly silly / broken that 'sudo apt-get update' fails 30 seconds of every day by default.20:48
smoseror whatever that ends up being.20:48
smoserblackboxsw, well, i think its busted if the subnet doesnt have a type20:48
smoserblackboxsw, so thats my attempt at noramlizing the fields in subnets. then have to adjust callers to expect more sanity.20:49
blackboxswsmoser: "no user can sanely run 'apt-get update' or 'apt-get install' because some system thing might be doing something."  yep. it is insane, but it's reality at the moment, the other issue is checksum mismatches in apt repositories to all apt users who happen to call apt update while the distro is being updated. Something about the packages.list files not20:50
blackboxswbeing fully written during that update causes spurious failures on any remote caller to apt update.20:50
smoserwell hash sum mismatches are fixed now.20:50
smoserin xenial+20:50
smosereverything is by hash20:50
blackboxswahh good. I hadn't followed that fix. (because I had a workaround for it ;) )20:50
blackboxswbut I agree that generally cloud-init shouldn't put workarounds in for other things that really should be fixed.20:51
blackboxsws/other things/external dependencies/20:51
smoserhe doesn't point to the launchpad bugs20:51
smoser https://bugs.launchpad.net/ubuntu/+source/apt/+bug/97207720:51
ubot5Ubuntu bug 972077 in apt (Ubuntu) "apt repository disk format has race conditions" [Medium,Fix released]20:51
smoserbut then andreas hit that today on xenial... as he was using a br.archive.ubuntu.com mirror20:52
smoserthat i guess doesn't have the server side fixes needed20:52
jbrowneI'll file an LP bug, thanks.21:01
smoserplease paste bug url here jbrowne21:02
jbrownewill do21:02
ubot5Ubuntu bug 1693361 in cloud-init "cloud-init sometimes fails on dpkg lock due to concurrent apt-daily.service execution" [Undecided,New]21:10
smoserblackboxsw, i'm interested in your thoguths on that pastebin .. even though its not in any real nice stae. tox fails at the moment.21:18
=== rangerpb is now known as rangerpbzzzz
blackboxswsmoser: yes definitely. will test it out now. could add a couple of unit tests and post you a diff if you want21:25
blackboxswsmoser did you want to delete all empty/None keys from the dict?21:42
blackboxsw+    for k in ('address', 'gateway', 'netmask'):21:42
blackboxsw+        if k in subnet and not subnet[k]:21:42
blackboxsw+            del subnet[k]21:42
blackboxswcould be done w/  subnet = dict((k, v) for k, v in subnet.iteritems() if v)21:42
blackboxswalso s/if 'netmask' in subnet: del subnet['netmask'] can be consolidated to    subnet.pop('netmask', None)21:46
blackboxswsmoser: some minor changes to normalize_subnet http://paste.ubuntu.com/24648411/22:21
blackboxswsmoser: also it seems like logic is duplicated a lot between _normalize_subnet and normalize_route with the exception that the key we are normalizing in routes is 'network|destination'  and in subnet is 'address' it could be small private helper function that takes the address_keys  and could be called inside normalize_subnet():22:29
blackboxsw_normalize_net(addr_keys=['address'])     and in normalize_route as _normalize_net(addr_keys=['network', 'destination'])22:29
blackboxswthat wasn't proper english at all, but hopefully you got the gist22:29
blackboxswok last pass on this for today http://paste.ubuntu.com/24648847/23:28
blackboxswcommon _normalize_net_keys() called from both normalize_subnet and normalize_route23:29

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!