| smoser | powersj, blackboxsw yeah, that kind of stinks. basically i think we *have* to detach package build dependencies form tox requirements.txt | 01:38 |
|---|---|---|
| smoser | which i think might in-advertantly have been why it was the way it was when blackboxsw found it and thought WTH | 01:38 |
| smoser | the package build on xenial didn't magically start depending on python3-pycodestyle just because trunk moved some versions in its tox. | 01:39 |
| smoser | as an example, see that https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial did not break because of this change. | 02:09 |
| askb | can anyone assist with this https://bugs.launchpad.net/cloud-init/+bug/1692424 | 02:33 |
| ubot5 | Ubuntu bug 1692424 in cloud-init "util.py[WARNING]: Failed to disable password for user centos" [Undecided,New] | 02:33 |
| askb | Is there are doc for building the latest verion on centos7 ? | 02:34 |
| === sambetts|afk is now known as sambetts | ||
| === rangerpbzzzz is now known as rangerpb | ||
| smoser | askb, can you get a cloud-init.log from the system ? | 13:34 |
| smoser | /var/log/cloud-init.log | 13:34 |
| askb | smoser, yup | 13:35 |
| askb | do you want me to upload the logs ? | 13:35 |
| askb | smoser, http://paste.openstack.org/show/610552/ | 13:38 |
| askb | smoser, forgive my ignorance, but are there any docs for building the rpm/srpm for centos7 | 13:44 |
| askb | I noticed that there is a spec.in | 13:44 |
| askb | "make rpm" does not work | 13:45 |
| smoser | powersj, blackboxsw https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324541 | 13:55 |
| smoser | askb, hm... | 13:55 |
| smoser | let me find | 13:55 |
| smoser | I had https://gist.github.com/smoser/19e65095b342e98fd4d6466e4d4fa1ac | 13:55 |
| smoser | we're workign on getting rpm build into daily c-i, and i'm not sure what is being used for that. | 13:56 |
| smoser | hmm.. i see ./packages/brpm is busted. odd | 13:57 |
| smoser | askb, i thik that gist will work. brpm is failing for me here, but because i dont have cheetah installed and the spec file needs it. | 14:01 |
| powersj | smoser: yeah the build test that I pointed out is what uses packages/brpm | 14:01 |
| powersj | I expect that to work :) | 14:01 |
| smoser | powersj, we need to move the spec.in to be jinja2 rather than cheetah | 14:02 |
| powersj | and we had a centos 6 test case fail last night | 14:02 |
| smoser | :( | 14:02 |
| powersj | https://jenkins.ubuntu.com/server/job/cloud-init-centos-6/10/consoleText | 14:02 |
| powersj | I'm going to go eat/wake-up and then I'll file bugs and we can chat about it after standup as far as what you want to see for fixes | 14:02 |
| askb | are these the same errors you are getting http://paste.openstack.org/show/610557 | 14:03 |
| askb | smoser, ^^ | 14:03 |
| smoser | askb, i suspect that whatever you're seeing is fixed in a newer release. 0.7.5 is quite old. | 14:03 |
| smoser | askb, yes. | 14:03 |
| smoser | you'll need python-cheetah installed | 14:03 |
| smoser | which that gist will install for you. | 14:03 |
| smoser | ie, if you grab that 'centos-setup' and run ./centos-setup build | 14:04 |
| askb | I am not building it with lxc ... instead straight on centos7 virt env ? | 14:05 |
| askb | should I be using the lxc instead | 14:05 |
| smoser | well, the script will install the deps for you | 14:05 |
| smoser | i jsut showed how to do it all with lxc | 14:05 |
| smoser | but all it amounts to is runing that script inside the container | 14:05 |
| smoser | which should be not really any different than inside a centos7 | 14:06 |
| askb | you mean from the gist above correct? | 14:06 |
| smoser | "virt env" == "kvm guest" or python virtual env | 14:06 |
| smoser | https://gist.github.com/smoser/19e65095b342e98fd4d6466e4d4fa1ac | 14:06 |
| askb | python virtual environment | 14:07 |
| smoser | well, package build is going to use system ddependencies... | 14:07 |
| smoser | powersj, i'im not sure what is changed there. | 14:13 |
| smoser | but tests/cloud_tests is python3 only. and that syntax is not python2.6 compatible | 14:13 |
| askb | smoser, Is the script to be run the - centos-setup - on debian/ubuntu ? | 14:16 |
| smoser | to build an rpm you really need to be in a centos (or rhel) environment | 14:16 |
| smoser | ./centos-setup build | 14:17 |
| smoser | will install the necessary packages to build | 14:17 |
| smoser | then you can run | 14:17 |
| smoser | ./packages/brpm | 14:17 |
| askb | oh this says "lxc: containers not found" on centos7 | 14:17 |
| smoser | and build a package | 14:17 |
| smoser | sure. skip the lxc portion, just use centos-setup | 14:17 |
| smoser | wget that and chmod and execute it. | 14:17 |
| askb | smoser, I was able to build an rpm - thanks! | 14:38 |
| askb | smoser, will update a bit later how it goes with the testing since its getting 1am here | 14:38 |
| smoser | askb, good luck | 14:39 |
| smoser | and sleep tight | 14:39 |
| askb | smoser, quick question: how do I build the latest stable version/release ? | 14:39 |
| smoser | you can git checkout 0.7.8 | 14:39 |
| smoser | and do the same ./packages/bdrpm | 14:39 |
| askb | 0.7.9 is the head ... of the branch now ? | 14:40 |
| smoser | sorry | 14:41 |
| smoser | 0.7.9 is newest release | 14:41 |
| smoser | 'master' branch is trunk | 14:41 |
| askb | awesome! :) | 14:42 |
| smoser | askb, i updated that gist README hoepfully to be more clear | 14:44 |
| smoser | and to down-play the lxd thing so as to not confuse people that. | 14:44 |
| smoser | blackboxsw, updated https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/324450 | 16:30 |
| blackboxsw | thanks+1 let's land I've added https://trello.com/c/iJdLSnh4/91-add-root-dir-lookup-for-unittests | 16:33 |
| smoser | k. | 16:33 |
| blackboxsw | heh, I'll steer you later or tomorrow to the real meat: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/324499 :) | 16:34 |
| === sambetts is now known as sambetts|afk | ||
| blackboxsw | it's up for review. I'm working on the old git review queue for cloud-init at the moment | 16:35 |
| blackboxsw | minor comment and approve of https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324274 | 16:42 |
| smoser | looking | 16:42 |
| smoser | yeah, got to add that. | 16:42 |
| smoser | powersj, are you looking for fix centos6 ? | 17:51 |
| powersj | smoser: I wasn't planning on looking at it today, but if you have something let me know and I can ack or test or merge or whatever :P | 17:52 |
| smoser | ok | 17:55 |
| smoser | blackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324557 | 17:55 |
| smoser | can you ack that quick ? its just part of | 17:55 |
| smoser | https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324274 | 17:56 |
| smoser | but the part that *has* unit test | 17:56 |
| harlowja | smoser and others http://lists.openstack.org/pipermail/openstack-dev/2017-May/117401.html | 17:57 |
| harlowja | i responded a little bit, but feel free to :-P | 17:57 |
| smoser | harlowja, thanks. i dont have mch to add there. | 18:02 |
| harlowja | np | 18:02 |
| blackboxsw | approved the branch smoser | 18:38 |
| smoser | thanks. | 18:57 |
| blackboxsw | smoser: I'm reviewing https://code.launchpad.net/~akaris/cloud-init/+git/cloud-init/+merge/322992 and forgot what your thoughts were on us using python's netaddr instead of trying to carry our own burden of mask2cidr cidr2mask calculations | 19:15 |
| blackboxsw | I was debating about cobbling up a quick suggestion branch w/ unit tests for the review if you netaddr is a likely candidate for replacing this somewhat fragile code | 19:15 |
| blackboxsw | if you *think* netaddr is a likely candidate.... | 19:16 |
| blackboxsw | but again, maybe cloud init wants to reduce requirements on external packages for the sake of speed, dunno. | 19:17 |
| blackboxsw | it's nice to have simple validation functions like valid_ipv4, valid_ipv6 IPNetwork.netmask builtins | 19:19 |
| smoser | blackboxsw, python2.6 ENO_NET_ADDR | 19:21 |
| smoser | and even the rhel 6 version of python2.7 | 19:22 |
| smoser | blackboxsw, i was looking at that too and trying to clean it up abit | 19:25 |
| blackboxsw | won't touch it since you are playing in that space, but ping me for a review on it when you are done. I don't want to miss what approach you are looking for. | 19:26 |
| blackboxsw | s/review/review please/ :) | 19:26 |
| smoser | blackboxsw, http://paste.ubuntu.com/24646853/ | 19:33 |
| smoser | that is what is in my buffer right now | 19:33 |
| smoser | i was looking at xnox's cidr branch | 19:33 |
| smoser | (which had the same change as akaris) | 19:33 |
| blackboxsw | smoser: why not mask = str(mask) in line 162 of your paste, that was you are properly converting to the expected type for both ipv4 & ipv6 handling. | 19:36 |
| smoser | you just ean | 19:37 |
| smoser | mask = str(mask) | 19:37 |
| smoser | if : in mask: | 19:37 |
| smoser | well, with proper quotes | 19:37 |
| smoser | right ? | 19:37 |
| blackboxsw | yeah. | 19:37 |
| blackboxsw | then you don't have to perform str(mask) twice | 19:37 |
| blackboxsw | also that function doesn't have a return or error if a string w/out . or : is presented | 19:38 |
| blackboxsw | does subnet['prefix'] = mask_to_net_prefix(subnet.get('netmask')) # Where None is returned cause any problems? (line 52 of your paste) | 19:40 |
| blackboxsw | heh and I have no idea what's happening on line 85 | 19:41 |
| blackboxsw | 87 rather | 19:41 |
| blackboxsw | WIP paste I bet | 19:42 |
| blackboxsw | +1 I like the general approach in either case. given netaddr DNE on some of our support matrix | 19:47 |
| jbrowne | Hello, I'm running into a race condition between cloud-init and apt-daily.service on Xenial. This was mentioned in passing in https://bugs.launchpad.net/cloud-init/+bug/1594576, but I don't see a cloud-init issue open specifically about it. Warning: small wall of text to follow. | 19:56 |
| ubot5 | Ubuntu bug 1594576 in cloud-init "cc_salt_minion behaves badly if apt-get update fails" [Medium,Triaged] | 19:56 |
| jbrowne | cloud-init when installing packages is encountering a lock on APT due to the apt-daily service running simultaneously. | 19:57 |
| jbrowne | Running a dead simple user data, I see that the stock AWS Ubuntu Xenial AMI does not encounter the race condition as the timer state is: | 19:57 |
| jbrowne | Thu 2017-05-25 15:34:01 UTC 20h left n/a n/a apt-daily.timer apt-daily.service | 19:57 |
| jbrowne | While the timer state of my packer-built AMI (based on the stock AMI) is: | 19:57 |
| jbrowne | Fri 2017-05-19 08:40:01 UTC 5 days ago Wed 2017-05-24 19:19:21 UTC 4s ago apt-daily.timer apt-daily.service | 19:57 |
| jbrowne | Searching extensively (including IRC logs) I haven't found any discussion in the cloud-init project about this issue and am happy to open an LP bug. | 19:57 |
| jbrowne | There's a lot of discussion about when unattended upgrades should run, etc. and if it is a systemd issue, but regardless it seems like cloud-init should be defensive about this. | 19:57 |
| jbrowne | I would naively argue that cloud-init should retry on apt issues, but I see that was deemed unneded in another case (https://bugs.launchpad.net/cloud-init/+bug/1594967) | 19:58 |
| ubot5 | Ubuntu bug 1594967 in cloud-init "cloud-init lack of retry causes apt-get failures to be fatal" [High,Fix released] | 19:58 |
| jbrowne | One are in which I am ignorant is how the actual AMIs are built; I'm trying to determine if something specifically clears systemd's timer states when making an image, etc. | 19:58 |
| jbrowne | Was digging into cloud-image-utils and then decided to just post here since that area also appears to be s-mosers. | 19:58 |
| blackboxsw | jbrowne: peeking for context. Without context of cloud-init's intended behavior, I would prefer apt to retry upon failure, or checksum mismatch which has hit me multiple times in other projects. Let me see if I can get more context on this specific issue. Thanks for "the deets". | 19:59 |
| jbrowne | Basically if one uses cloud-init for: | 20:00 |
| jbrowne | packages: | 20:00 |
| jbrowne | - awscli | 20:00 |
| jbrowne | and systemd has launched some other apt related process concurrently there can be a lock contention | 20:00 |
| jbrowne | I have a slew of kinda related bugs I've found in other projects that build their own AMIs (chef, vagrant, etc.) and have hit this lock contention with cloud-init, but I didn't want to dump an entire thesis into the channel. :) | 20:01 |
| blackboxsw | I'd naively like to see at least a simple retry fallback mechanism (maybe 3 retries on a backoff algo) before failing. Yes dpkg lock contention has hit a myriad of other projects I've been on. I'd think we could build in a simple retry/log effort before failing. | 20:02 |
| blackboxsw | but, I'm just reading through the cloud-init code now (and your references) to see what the existing state of affairs is. | 20:03 |
| blackboxsw | we've recently talked side-channel about the potential of adding a decorator to retry certain functions that are known to get a benefit from retries on expected failure. This might be a case where something like that could come in handy. | 20:08 |
| blackboxsw | it *looks* like we could potentially build in a retry mechanism just in Runners.run of cloudinit/helpers.py. I'll peek at whether that's viable. | 20:16 |
| smoser | blackboxsw, http://paste.ubuntu.com/24647424/ | 20:42 |
| smoser | thats where i am now. | 20:42 |
| smoser | jbrowne, why would systemd launch some other apt related process ? | 20:42 |
| blackboxsw | yowsa smoser | 20:42 |
| smoser | fudge. | 20:43 |
| jbrowne | apt-daily is no longer in /etc/cron.daily. Now on a timer | 20:43 |
| jbrowne | If your machine is down during the trigger time (or a custom built AMI) systemd will trigger that timer during boot -> apt-get update in another process | 20:43 |
| smoser | http://paste.ubuntu.com/24647437/ | 20:43 |
| smoser | jbrowne, ack on that. that is a mess. cloud-init should probably disable that thing and then re-enable it. | 20:44 |
| jbrowne | There's a lot of chatter about how this has changed when unattended updates run and conflict between not running during boot and machines that are up for short periods of time (laptops), etc. | 20:44 |
| smoser | that is generally garbage really. | 20:44 |
| smoser | no user can sanely run 'apt-get update' or 'apt-get install' because some system thing might be doing something. | 20:44 |
| smoser | shoudl really be fixed at the apt level | 20:44 |
| jbrowne | I can see the argument for using systemd timer instead of cron.daily due to it blocking other cron jobs, but changing fundamental behavior like this bugs me. | 20:45 |
| smoser | blackboxsw, second paste. i pasted the file not diff. | 20:45 |
| jbrowne | Also unattended upgrades is on by default now in Xenial, but that's a whole 'nother story. :) | 20:45 |
| blackboxsw | yeah I'm on 2nd paste now. it looks a lot simpler | 20:45 |
| blackboxsw | smoser: not sure, might have to default to string for the get on line 25 + if (subnet.get('type', '').endswith('6') or | 20:46 |
| jbrowne | The reason I brought this up in channel rather than just opening an LP bug was to get a sense of how the cloud-init team feels about it. My current work around is to have packer disable the apt-daily.service in systemd and I re-enable it on launch. | 20:46 |
| blackboxsw | if type isn't present, subnet.get would return None and you can't call endswith on that | 20:46 |
| smoser | jbrowne, yeah. you want to file a bug ? you can file against cloud-init. | 20:47 |
| smoser | but it seems honestly silly / broken that 'sudo apt-get update' fails 30 seconds of every day by default. | 20:48 |
| smoser | or whatever that ends up being. | 20:48 |
| smoser | blackboxsw, well, i think its busted if the subnet doesnt have a type | 20:48 |
| smoser | blackboxsw, so thats my attempt at noramlizing the fields in subnets. then have to adjust callers to expect more sanity. | 20:49 |
| blackboxsw | smoser: "no user can sanely run 'apt-get update' or 'apt-get install' because some system thing might be doing something." yep. it is insane, but it's reality at the moment, the other issue is checksum mismatches in apt repositories to all apt users who happen to call apt update while the distro is being updated. Something about the packages.list files not | 20:50 |
| blackboxsw | being fully written during that update causes spurious failures on any remote caller to apt update. | 20:50 |
| smoser | well hash sum mismatches are fixed now. | 20:50 |
| smoser | in xenial+ | 20:50 |
| smoser | everything is by hash | 20:50 |
| blackboxsw | ahh good. I hadn't followed that fix. (because I had a workaround for it ;) ) | 20:50 |
| blackboxsw | but I agree that generally cloud-init shouldn't put workarounds in for other things that really should be fixed. | 20:51 |
| blackboxsw | s/other things/external dependencies/ | 20:51 |
| smoser | http://www.chiark.greenend.org.uk/~cjwatson/blog/no-more-hash-sum-mismatch-errors.html | 20:51 |
| smoser | he doesn't point to the launchpad bugs | 20:51 |
| smoser | https://bugs.launchpad.net/ubuntu/+source/apt/+bug/972077 | 20:51 |
| ubot5 | Ubuntu bug 972077 in apt (Ubuntu) "apt repository disk format has race conditions" [Medium,Fix released] | 20:51 |
| smoser | but then andreas hit that today on xenial... as he was using a br.archive.ubuntu.com mirror | 20:52 |
| smoser | that i guess doesn't have the server side fixes needed | 20:52 |
| jbrowne | I'll file an LP bug, thanks. | 21:01 |
| smoser | please paste bug url here jbrowne | 21:02 |
| jbrowne | will do | 21:02 |
| jbrowne | https://bugs.launchpad.net/cloud-init/+bug/1693361 | 21:10 |
| ubot5 | Ubuntu bug 1693361 in cloud-init "cloud-init sometimes fails on dpkg lock due to concurrent apt-daily.service execution" [Undecided,New] | 21:10 |
| smoser | gracias | 21:17 |
| smoser | blackboxsw, i'm interested in your thoguths on that pastebin .. even though its not in any real nice stae. tox fails at the moment. | 21:18 |
| === rangerpb is now known as rangerpbzzzz | ||
| blackboxsw | smoser: yes definitely. will test it out now. could add a couple of unit tests and post you a diff if you want | 21:25 |
| blackboxsw | smoser did you want to delete all empty/None keys from the dict? | 21:42 |
| blackboxsw | + for k in ('address', 'gateway', 'netmask'): | 21:42 |
| blackboxsw | + if k in subnet and not subnet[k]: | 21:42 |
| blackboxsw | + del subnet[k] | 21:42 |
| blackboxsw | could be done w/ subnet = dict((k, v) for k, v in subnet.iteritems() if v) | 21:42 |
| blackboxsw | also s/if 'netmask' in subnet: del subnet['netmask'] can be consolidated to subnet.pop('netmask', None) | 21:46 |
| blackboxsw | smoser: some minor changes to normalize_subnet http://paste.ubuntu.com/24648411/ | 22:21 |
| blackboxsw | smoser: also it seems like logic is duplicated a lot between _normalize_subnet and normalize_route with the exception that the key we are normalizing in routes is 'network|destination' and in subnet is 'address' it could be small private helper function that takes the address_keys and could be called inside normalize_subnet(): | 22:29 |
| blackboxsw | _normalize_net(addr_keys=['address']) and in normalize_route as _normalize_net(addr_keys=['network', 'destination']) | 22:29 |
| blackboxsw | that wasn't proper english at all, but hopefully you got the gist | 22:29 |
| blackboxsw | ok last pass on this for today http://paste.ubuntu.com/24648847/ | 23:28 |
| blackboxsw | common _normalize_net_keys() called from both normalize_subnet and normalize_route | 23:29 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!