[11:20] <cpaelzer> smoser: already around?
[12:18] <cpaelzer> smoser: while the former two quesitons are solved by time, I would still ahve another if you are around
[13:22] <smoser> cpaelzer, here now.
[13:22] <smoser> what sup?
[13:23] <cpaelzer> hiho
[13:23] <cpaelzer> I'm testing more nd more regarding the cloud-init issue you reported due to my merge
[13:23] <cpaelzer> most things are done, I found a few on top thou
[13:23] <cpaelzer> working on these, but that was not the question
[13:23] <cpaelzer> at some point I'll have to throw it into a adt or ppa to test if it works there as well
[13:24] <cpaelzer> since packaging isn't in the repo's itself I wondered what the recommended way is to do so
[13:24] <cpaelzer> I mean there are many ways
[13:24] <cpaelzer> pull-lp-source - merge a bzr tree in
[13:24] <cpaelzer> bzr tree and merge a debian dir into that
[13:24] <cpaelzer> or something totally different
[13:24] <cpaelzer> I assume you don't do that manually each timne, so I wanted to know which way I should go to test ppa/adt builds
[13:25] <cpaelzer> smoser: ^^
[13:25] <smoser> cpaelzer, right. need to test building in ppa.
[13:26] <smoser>  ./packages/bddeb
[13:26] <smoser> it hsa --help, but largely i'd run:
[13:26] <smoser>  ./packages/bddeb -S
[13:26] <smoser> then
[13:26] <smoser>  sbuild ...
[13:27] <cpaelzer>   /packages/bddeb -S gievs a .dsc and a orig tarball then?
[13:27]  * cpaelzer is reading bddeb ...
[13:27] <smoser> yeah, it uses bzr export to get a .orig and then goes from there.
[13:28] <cpaelzer> ok nice, then I can hit these stages once I fixed a corner case of the keyserver issues
[13:28] <cpaelzer> smoser: btw I also made sure I port fixes I did this way between curtin/cloud-init as I work on the same code for both
[13:28] <cpaelzer> so I hope to end the day with an update to the curtin MP and a new cloud-init MP
[13:29] <smoser> ok. great.
[13:50] <GivenToCode> hi i have a custom cfg in /etc/cloud/cloud.cfg.d that specifies an fs_setup to mkfs ext4 on ephemeral0 (ec2), however on c3 instances types we get an error in the log that /dev/xvdb is already mounted when trying to mkfs...
[13:50] <GivenToCode> this is 0.7.5-0ubuntu1.18
[13:51] <GivenToCode> I would expect fs_setup to run before any mounting, right?
[13:52] <GivenToCode> there is an entry for xvdb in fstab, and I've tried to override that by specifying a mounts [ ephemeral0, null ]
[14:11] <cpaelzer> GivenToCode: yes it should set up fs_setup things before the mount step of cloud-init
[14:11] <cpaelzer> GivenToCode: but I'd wonder if something else could mount it even before
[14:11] <cpaelzer> GivenToCode: depending on you influence to this environemnt you could add in your .cfg commant to check the status early on
[14:12] <cpaelzer> GivenToCode: like early_commands: checkmounts: 'mount'
[14:13] <cpaelzer> and then check that output if it really is mounted at the beginning
[14:14] <cpaelzer> cpaelzer: sorry I'm mixing up things here - let me rethink
[14:14] <cpaelzer> writing too muhc curtin / cloud-init code at the same time - that would have been the curtin answer
[14:15] <cpaelzer> GivenToCode: doc/examples/cloud-config-boot-cmds.txt should be better
[14:17] <cpaelzer> GivenToCode: if you prefer an url http://cloudinit.readthedocs.io/en/latest/topics/examples.html#run-commands-on-first-boot
[14:49] <cpaelzer> smoser: did the mirrorlist issues (the last one you added the skipif for) happen to you in sbuild as well, or only in the buildd's?
[14:50] <smoser> cpaelzer, no. only in a buildd
[14:50] <smoser> cpaelzer, but think about it working on centos
[14:50] <cpaelzer> smoser: k, thx - setting up a victim ppa then
[14:50] <smoser> we want unit tests to work there.
[14:50] <cpaelzer> smoser: I already have
[14:50] <smoser> ok.
[14:50] <smoser> i maeant to tell you this...
[14:50] <smoser> there is a 'FilesystemMocking' class
[14:50] <smoser> or somethignt
[14:50] <cpaelzer> smoser: I already covered that - at least I think so :-)
[14:51] <smoser> aht basically mocks all fsreads.
[14:51] <smoser> to another directory that you can popoulate
[14:51] <cpaelzer> smoser: for this in particular it doesn't need a fs mocking class
[14:51] <smoser> ok. thats fine.
[14:51] <cpaelzer> the feature of influencing apt environment is very very ubuntu/debian specific
[14:51] <cpaelzer> so are the tests
[14:51] <cpaelzer> but that can be nicely done with a skipIF to the apt binary
[14:52] <cpaelzer> I covered this and more and you can take a look once it is mature enough for a merge proposal
[14:52] <cpaelzer> for now I seem to uncover further issues faster than I like and EOD is coming closer ...
[14:52] <cpaelzer> :-/
[14:53] <smoser> cpaelzer, i dont know..
[14:53] <smoser> it is in fact debian/ubuntu specific
[14:53] <smoser> but that does not mean the unit tests should not run elsewhere.
[14:54] <cpaelzer> for the start I'd like to disable things so they work properly
[14:55] <cpaelzer> over time one can (r not) commit the time to enable this on more environments
[14:55] <cpaelzer> that is what I'd prefer
[14:55] <cpaelzer> I feel to postpone too much as I try to get this done - so I wouldn't add a centos env test to the task list right now
[14:56] <cpaelzer> but you are right overall - you want a centos dev on cloudinit to see when he breaks things
[14:56] <smoser> well, thats fine. but generally speaking the failure i saw was from either a file existing or not existing (or a directory or something)
[14:56] <smoser> i'm not really sure
[14:56] <smoser> and the fact that that exists or does not exist is somethign that would ideally be covered by the unit test
[14:56] <smoser> which should not require the system its running on to be in a given state
[14:57] <cpaelzer> ah that one - it is a file delivered by cloud-init itself, and we don't check if the file exists
[14:57] <cpaelzer> we check if cloud-init "checks" if the file exists
[14:57] <cpaelzer> because that is the code path that it should take and that is what the unit test does
[14:57] <cpaelzer> the file doesn't exist on my system either
[14:57] <cpaelzer> but the test works
[14:58] <cpaelzer> what happens there is the issue in find_apt_mirror_info that I need to debug in the ppa environment, because I can't see locally what is breaking
[14:58] <cpaelzer> an issue in that function causes the one you have seen as follow on error
[14:58] <smoser> right. there is a build logo from an existing one if you wantd.
[14:58] <cpaelzer> I have checked the two logs you linked
[14:59] <smoser> yah. ok.
[14:59] <cpaelzer> but I need more debug data on it
[14:59] <smoser> you're doing well,. sorry to sound like i'm giving you flack
[14:59] <cpaelzer> for now I'm unhappy wit hthe gpg key things - I wanted to fix them right
[14:59] <smoser> you wrote a bunch of test cases for stuff that wasnt' tested and fixed a bunch of stuff and there was fallout
[14:59] <smoser> the first part is +100
[14:59] <smoser> )
[14:59] <cpaelzer> I'm fine and we will make it good :-)
[14:59] <cpaelzer> thanks
[14:59] <smoser> the second part is admittedly *my* fault and cloud-init fault for accepting your MP before it built in a ppa
[14:59] <smoser> :)
[14:59] <cpaelzer> so I added a nice fallback in case network is unavailable - but it will use real network if it is available
[15:00] <cpaelzer> unfortunately now that is "can" work it complains in sbuild envs for things that are not writable
[15:00] <cpaelzer> will get it done ...
[15:00] <smoser> please dont depend on presense of a keyserver
[15:00] <cpaelzer> just gets a longer and longer task :-)
[15:00] <smoser> those things really suck
[15:00] <cpaelzer> smoser: thats what I ment - I don't DEPEND anymore, but IF I can reach it properly I do
[15:00] <smoser> that will cause arbitrary failures
[15:00] <cpaelzer> and if I can't reach it, it has a fallback that works
[15:01] <cpaelzer> so the test is happy with and without keyserver
[15:01] <cpaelzer> it even works with broken keyservers or random dns names
[15:01] <cpaelzer> but IF it works it just is closer to the real thing
[15:01] <cpaelzer> which isn't bad for a unit test
[15:01] <smoser> i was honestly fairly OK with not unit testing that shell blob. and prefer that to cloud-init's unit tests mucking with my .gpg dir
[15:02] <cpaelzer> hehe
[15:02] <cpaelzer> it isn't a shell blob anymore
[15:02] <cpaelzer> :-)
[15:02] <smoser> ok. but i still would really rather it not ever modify my .gpg dir
[15:02] <cpaelzer> thatr with the .gpg dir is what I have to tackle next
[15:02] <cpaelzer> for sbuild env anyway
[15:02] <cpaelzer> smoser: so you tihnk it is ok to throw parts away and mock that side of things?
[15:03] <cpaelzer> ok, I can do that - that will throw away some code that cost me some sweat today, but it was good for the learning still
[15:06] <cpaelzer> at least we got the shell blob to become python as you wanted as well, thanks to a nice suggestion of rharper and some polishing I did
[15:07] <rharper> cpaelzer: nice
[15:07] <cpaelzer> I think with the plan to not mess around with .gpg at all in the tests I won't complete it today anyway
[15:07] <cpaelzer> smoser: do you mind if it is tomorrow and not today that you get the next MP revision?
[15:07] <rharper> yeah, I'd be very dubious of relying on a .gpg dir
[15:08] <rharper> rather a tmpdir and export GPG var to point to it
[15:08] <rharper> certainly GPG will check ENV for that location
[15:08] <cpaelzer> rharper: it has an argument
[15:08] <cpaelzer> rharper: just need to find a way to insert that into the subp's
[15:08] <cpaelzer> I still want to test the original code and not mock it all
[15:09] <rharper> sure, lemme see if I've example of setting env for subp
[15:09] <smoser> you can just mock the output of gpg
[15:09] <smoser> capture that call really is i think the right way to do it.
[15:10] <cpaelzer> I'd mock the gpg calls but then assert on them to make sure they were poked the right way
[15:10] <smoser> honestly, mocking the call to that shell blob as i did, we have 2 lines of untested code out of that.
[15:10] <smoser> and no mucking with your .gpg
[15:10] <cpaelzer> mucking vs mocking :-P
[15:10] <smoser> for code that hasnt changed and worked fine for 4 years (except for arguably invalid user input)
[15:11] <smoser> basically, i dont want you to spend a bunch of time making sure we get test coverage on that.
[15:11] <cpaelzer> ok
[15:11] <cpaelzer> I'll try to mock it then and get it working tomorrow
[15:12] <rharper> cpaelzer: Popen takes a env= param; that's not exposed in subp IIRC, but one could,  then myenv = os.environ_copy(), and myenv[VAR]=VAL , pass env=myenv;
[15:12] <cpaelzer> rharper: thanks for checking, but given the requirements - for now - I'd go with smoser and "just" mock some things
[15:12] <rharper> yep
[15:14] <smoser> subp takes env also.
[15:14] <smoser> yeah.
[15:24] <rharper> nice
[15:24] <GivenToCode> cpaelzer, yes something is mounting everything in fstab before fs_setup, we don't have anything custom running before cloud-init
[15:38] <GivenToCode> ok, i think the fix is to remove it from fstab when we bake the image
[15:59] <ajorg> smoser: Can I submit GitHub pull requests instead of bzr branches for contributions?
[16:02] <smoser> ajorg, wont be github pull requests. but launchpad git pull requests should be shortly.
[16:02] <smoser> ajorg, basically upstream code path will still be launchpad but revision control in git
[16:02] <ajorg> okay, I'll try to learn bzr for today and maybe leave some of our more interesting patches for when git is available.
[16:08] <ajorg> If I did a terrible job of preparing this, it would be good to correct me now as I have several more patches I'll be trying to submit this week: https://code.launchpad.net/~ajorgens/cloud-init/python26/+merge/296575
[16:09] <smoser> ajorg, that looks fine. i'm really sorry about regressing 2.6 function
[16:09] <ajorg> Thanks.
[16:09] <ajorg> By way of introduction, I'm the cloud-init maintainer at AWS.
[16:09] <ajorg> (for the Amazon Linux AMI)
[16:10] <ajorg> We have a feature or two, several bugfixes, and a bunch of backward compatibility stuff for our 0.5 fork from before Yum / RPM were supported.
[16:11] <ajorg> I'd like to upstream whatever parts are palatable.
[16:11] <smoser> we need to get some c-i in place that can test on a 2.6.  the thing that makes it hard is that there is actally not even a python2.6 *available* in a supported ubuntu release (12.04 shipped 2.7)
[16:11] <ajorg> Anything I can do to get that in place?
[16:12] <ajorg> CentOS 6 testing would be prudent, and would cover 2.6
[16:12] <smoser> harlowja_at_home, has mentioned some options with his remote_tox
[16:12] <ajorg> cool
[16:12] <smoser> but yeah, we need to get this stuff in place.
[16:12] <harlowja_at_home> https://github.com/harlowja/remote_tox ifu care
[16:13] <smoser> dict comprehension is just so nice
[16:13] <smoser> i think thats the right word
[16:13] <ajorg> I know!
[16:13] <smoser> {a:b for a, b in d.items()}
[16:13] <harlowja_at_home> dict((a,b) for a, b in d.items()) ?
[16:13] <harlowja_at_home> just use the word 'dict' lol
[16:13] <smoser> harlowja_at_home, deal.
[16:14] <ajorg> Amazon Linux AMI has been on 2.7 for a few releases, so we won't be a fruitful source of 2.6 patches going forward.
[16:15] <ajorg> hmm, I get to learn how to do author attribution in bzr for some of these
[16:16] <smoser> ?
[16:16] <smoser> author attribution ?
[16:16] <ajorg> assuming it's possible, submitting a patch in someone else's name (other authors @amazon.com)
[16:17] <ajorg> I know how to do it in git.
[16:25] <smoser> ah. well you can commit with '--author='
[16:25] <smoser> that sprobably the thing you want.
[16:25] <smoser> then you become 'committer' and author is author
[16:28] <ajorg> perfect, very much like git
[16:28] <ajorg> thank you
[17:06] <smoser> larsks, fyi, i just uploaded a 0.29 tarball with your growpart fix https://launchpad.net/cloud-utils/trunk/0.29
[17:10] <larsks> smoser: thanks for the heads up!
[17:49] <harlowja> ok backs
[19:08] <larsks> This is a dumb question...under ubuntu, "python3 setup.py install ..." wants to put cloud-init into site-packages, but that dir doesn't exist.  What's the right way to install it?
[19:11] <larsks> Ah, --prefix=/usr/local seems to be the thing.
[19:12] <smoser> yeah, need some --prefix.
[19:12] <smoser> ./packages/bddeb though would probably be preferable in most cases
[19:12] <larsks> This is just for testing, not packaging...I was just confused by the "site-packages" does not exist error.
[19:13] <larsks> Apparently I haven't run setup.py on an ubuntu system before.