[12:30] <niluje> hello, I'm trying to write a connector for Scaleway (https://www.scaleway.com/)
[12:30] <niluje> By running `cloud-init --file ~/cloudinit.yaml init` with "datasource_list: [Scaleway]" in ~/cloudinit.yaml, I have an exception I don't understand:
[12:30] <niluje> https://pastebin.com/98PwbVBR
[12:31] <niluje> any chance someone can help me understand what is going on?
[12:52] <smoser> rharper,  you do not need python-argparse on centos7
[12:52] <smoser> python-argparse is really just a 2.6 thing. as its builtin to 2.7+
[12:52] <smoser> $ python3 -c 'import argparse; print(argparse)'
[12:52] <smoser> <module 'argparse' from '/usr/lib/python3.5/argparse.py'>
[12:53] <smoser> $ python2 -c 'import argparse; print(argparse)'
[12:53] <smoser> <module 'argparse' from '/usr/lib/python2.7/argparse.pyc'>
[12:53] <smoser> i'm not saying that the rpm isn't broken, but the answer is to probably not depend on it. not sure how really to do that in rpm an dsuch.
[14:16] <smoser> niluje, hm.. not sure what woudl have ahppened there.
[14:17] <smoser> i suspect you must have a network_config thing that is returning invalid data
[14:17] <rharper> smoser: I know we don't need it; but somehow it's in the rquires.txt file which python checks when loading
[14:18] <smoser> rharper, that gets used for the rpm i guess ?
[14:18] <rharper> smoser: harlowja: running with this which works but not sure that's the right thing to do: http://paste.ubuntu.com/24466885/
[14:19] <rharper> smoser: it's used by python setuptools or whatever, entry point thingy that reads egginfo
[14:19] <niluje> about my issue, here is the code of the connector: https://github.com/brmzkw/cloud-init-scaleway/commit/110bb3e02608cb29ec67fe180cac4a48e3bc548b  and my configuration + the errors I have: https://pastebin.com/qjBW8TGg
[14:19] <rharper> https://github.com/certbot/certbot/commit/8f101034960ffc1e47879314585898efda234e60
[14:20] <rharper> this is sorta the issue as well;
[14:20] <smoser> niluje, can you get a cloud-init.log ?
[14:20] <rharper> IIUC, special case python2.6 to add that dep (in packaging) and from > 2.6 it's standard library, so shouldn't be in requirements (or at least not show up in egginfo) ?
[14:20] <niluje> smoser: where is it stored?
[14:20] <niluje>  /var/lib/cloud?
[14:20] <smoser> i suspect that you've gone down the path of reading command line parameters that cloud-init has... and that it went wrong.
[14:20] <smoser> /var/log/cloud-init.log
[14:21] <smoser> if it sees ip= on the command line, it takes *that* as the fallback networking, and then tries to read files that ubuntu's initramfs tools writes
[14:21] <rharper> so, on ubuntu, we package the requires.txt file, but it's empty
[14:22] <niluje> smoser: cloud-init.log is empty
[14:22] <smoser> ubuntu ?
[14:24] <niluje> yes, xenial
[14:24] <niluje> I git-clone'd the cloud-init repository and pip install -e .
[14:26] <smoser> hm..
[14:27] <smoser> i suspect that there is no /etc/cloud/cloud.cfg.d/loggin* ?
[14:27] <smoser> (i'd install from package and just make your updates that way)
[14:27] <smoser> or build a package from trunk and install it.
[14:27] <smoser>  ./tools/bddeb
[14:27] <smoser> installing with pip is not something i've really ever tested.
[14:27] <niluje> ewww
[14:27] <niluje> ok
[14:27] <niluje> let me try installing the package with apt then
[14:28] <smoser> you'll still fail for sure
[14:28] <smoser> but hopefully get a log :)
[14:32] <niluje> how to you work on cloud-init then?
[14:32] <niluje> apt-get install cloud-init, then git clone, pip install -e ., $> cloud-init init?
[14:35] <niluje> I just apt-get install'ed the package, updated /etc/cloud/cloud.cfg to add "datasource_list: [Scaleway]" and call "cloud-init init" but cloud-init is trying to contact the EC2 APIs :/
[14:40] <smoser> niluje, generally i install the package. taht is what a distro is going to do anyway.
[14:40] <smoser> hm..
[14:40] <niluje> and then?
[14:41] <niluje> how do you edit the code?
[14:41] <smoser> when hacking i just change files in /usr/share///
[14:41] <smoser> and re-try and such
[14:41] <smoser> then i take those chagnes and re-build / dpkg -i
[14:42] <smoser> yeah, i know its not the best. but dpkg -i isn't actually *all* that different from pip install.
[14:42] <niluje> hm
[14:42] <niluje> how do you build the package?
[14:42] <smoser> ./tools/bddeb
[14:43] <smoser> soryr
[14:43] <smoser> ./packages/bddeb
[15:37] <smoser> https://bugzilla.redhat.com/show_bug.cgi?id=1194451
[16:28] <smoser> niluje, so... what was happenging for you was that
[16:28] <smoser> a.) you boot with 'ip=' on the kernel cmdline (which makes good sense)
[16:28] <smoser> b.) cloud-init recognizes that as saying taht the initramfs configured networking and that it should read its networkign from that.
[16:29] <smoser> rather than generating "fallback" config, which is "dhcp on the first nic you think looks good"
[16:29] <smoser> because obviously bouncing the nic would be bad on a network moutned root device
[16:30] <smoser> the problem is that your initramfs does not generate /run/net-<NAME>.cfg files that cloud-init expected to read some information from.
[16:30] <smoser> and thus it ends up generating a completely empty config that fails
[16:30] <niluje> ok
[16:30] <niluje> but should our initramfs generate the /run/net-<NAME>.cfg file?
[16:31] <smoser> well, mkinitramfs-tools does (ubuntu/debian)
[16:31] <smoser> but there is no official format of that, its just kidn of happensatnce
[16:31] <smoser> the other option would be to have cloud-init realize there are no net-* and do some other sort of searching to figure out what the network config is.
[16:32] <niluje> ok so what you did (I haven't checked yet) is remove the read from /run/net-xxx.cfg and it works, right?
[16:32] <smoser> no. those files do *not* exist for you
[16:32] <smoser> and cloud-init is expecting them
[16:32] <niluje> I know
[16:32] <niluje> I understand that
[16:32] <smoser> oh. right. yeah.
[16:32] <smoser> http://paste.ubuntu.com/24467477/
[16:32] <smoser> i just made it not pay any attention to the kernel cmdline
[16:33] <smoser> but that probably fails eslwehere
[16:33] <niluje> ok
[16:33] <smoser> as cloud-int will write a /etc/network/interfaces that says it should dhcp on eth0
[16:33] <niluje> ok
[16:33] <smoser> which best case fails on 'ifup' (because that is already up)
[16:33] <smoser> and worst case brings down the link that you have to the root device
[16:34] <niluje> so what I need to do is 1/ first, create manually the file un /run, 2/ make cloud-init work 3/ in //, update our initrd to create the files in /run/net-xxx.cfg
[16:34] <niluje> and everything should work
[16:34] <niluje> correct?
[16:34] <smoser> if your initramfs writes those files, it should work yeah.
[16:34] <niluje> ok
[16:34] <smoser> theres no standard way to do this.
[16:35] <niluje> I don't even know what is the format of those files :p will check
[16:36] <niluje> thanks a lot for your help smoser
[16:43] <smoser> niluje, see cloudinit/net/__init__.py _klibc_to_config_entry
[16:45] <smoser> blackboxsw, http://paste.ubuntu.com/24467573/
[16:45] <smoser> thats what i'll just upload with
[16:46] <blackboxsw> smoser, ok so that's our bug list for the SRU?
[16:47] <smoser> yeah
[16:47] <smoser> hm.. and maybe i'd rip out the  yum one
[16:47] <smoser> (not reference it)
[16:48] <smoser> yeah, i think i iwill
[16:53] <niluje> there's no reference to _klibc_to_config_entry in cloudinit/net/__init__.py
[16:53] <niluje> k in cloudinit/net/cmdline.py
[16:55] <smoser> sorry
[16:55] <smoser> yeah
[16:55] <smoser> niluje, and then in the tests therea re examples
[16:58] <niluje> smoser: this is the function reading /run/net-xx.cfg and convert it to a cloud-init config object, right?
[16:59] <niluje> sorry for the stupid questions I'm not familiar with cloud-init so I struggle a bit to understand what needs to be done
[17:00] <niluje> ok yes, that's it
[17:00] <niluje> I will look into it tomorrow
[17:00] <niluje> thanks a lot again for your help smoser, I'll post a message here when the connector will be working
[17:02] <smoser>  _klibc_to_config_entry is what converts
[17:02] <smoser> and the test functions show examples of the net-*
[17:08] <blackboxsw> smoser, so SRU info, last time you created https://public.etherpad-mozilla.org/p/cloud-init-sru-info   will we do the same type of thing for this sru?
[17:09] <smoser> blackboxsw, i think its useful, yeah.
[17:09] <rharper> https://docs.fedoraproject.org/en-US/Fedora/8/html/SELinux_FAQ/index.html#id503239 ; that's troublesome;  ifconfig output is thrown away due to selinux, you can get it by reading it from the pipe (instead of the terminal) and writing to file (ifconfig -a | cat >out)
[17:09] <smoser> blackboxsw, https://public.etherpad-mozilla.org/p/cloud-init-sru-info
[17:11] <blackboxsw> ... clunky  rharper
[17:11] <rharper> yeah, I'm more worried about cloudinit.util.subp
[17:11] <rharper> not sure that's working as expected
[17:11]  * rharper is testing that now
[17:13] <smoser> rharper, see, i told you enforcing=off
[17:13] <smoser> :)
[17:13] <rharper> lol
[17:13] <rharper> not the *right* answer
[17:13] <rharper> but yes
[17:13] <rharper> you were right
[17:14] <smoser> if everyone in the world says that 2+2 = 5, then that is 'right' for some definition of right ;)
[17:14] <smoser> i wonder
[17:15] <smoser> rharper, if we close the standard input
[17:15] <rharper> playing with subp now
[17:15] <smoser> hm..
[17:15] <rharper> I think it's already closed though
[17:15] <smoser> well, standard output is probably indirectly attached to a temrinal
[17:16] <smoser> i don tknow
[17:16] <rharper> serial console
[17:16] <smoser> well, /dev/console
[17:16] <rharper> but yeah; this gets into the magic of tty
[17:16] <rharper> and other things
[17:17] <smoser> we should be ablet to just close enough file handles and such though so that , before that subp (or all with 'capture') no open file handles are a termianl
[17:17] <smoser> id' think that would do it
[17:18] <rharper> so, subp defauts to data=None (stdin is fd of /dev/null) and capture=True, sets up a pipe
[17:18] <rharper> I just need to confirm we get output; there's a separate netstat -rn returning 1; I don't yet know why
[17:19] <rharper> https://bugs.launchpad.net/ubuntu/+source/net-tools/+bug/1251563
[17:19] <rharper> well, tha;s just strange
[17:19] <rharper> bbiab
[19:19] <powersj> https://paste.ubuntu.com/24468378/
[19:19] <powersj> ^ unit test failures on centos 6 + 7
[19:19] <powersj> I'll go play with https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/323265 shortly
[19:24] <smoser> powersj, great.
[20:22] <rharper> https://bugzilla.redhat.com/show_bug.cgi?id=1406520
[20:22] <rharper> it appears that's hitting in centos7 image
[20:22]  * rharper is confirming 
[20:26] <rharper> bummer, I don't see 2.5-7 released yet
[20:37] <powersj> https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/323351
[20:37] <powersj> All I got rid of was the use of the array for the patches and got rid of one that was useless
[20:38] <powersj> oh and cleaned up pylint errors
[20:38] <powersj> rharper: if you could help me better understand what else you would want changed to smoser original proposal that would be good. You mentioned moving the asserts
[20:38] <rharper> powersj: yeah, lemme get a pointer
[20:42] <rharper> powersj: so, each of the patch_XXX should be @mock.patch.object(util, 'write_file') on the helper function; then pass that in as a mock_xxx to the _apt_source_list method;  the decorator does the start/stop automatically for you;  then in them ethod, the mocks that have return values or side-effects, you just need to apply those based on the input,
[20:43] <powersj> oh you wanted decorators
[20:43] <rharper> powersj: I think all of them are constant values, except the mock_shouldcfg, in which case you just mockshouldcfg.return_value = (cfg_on_empty, "this is for test")  in the method
[20:43] <rharper> first
[20:44] <rharper> and then, I really want the caller who knows if they sent "debian", "ubuntu" to run the asserts;
[20:45] <powersj> so end that function after running cc_apt_configure.handle
[20:45] <rharper> so, test_apt_v3_source_list_debian knows what config it sent and which distro is being used
[20:45] <rharper> knowing those things, it can test that it expects for should_config to return true
[20:46] <rharper> the other variable is "is_system_snappy"  which I think I'd like to assume its no , and then have a single test_apt_v3_source_list_snappy; where we return True on that and then can confirm that apt_configure wasn't run
[20:47] <rharper> that'll break up much of the if block of asserts based on the config_on_empty list
[20:48] <rharper> the test_ method calling the helper should construct the path and test it explicitly;  if there is something that we know *always* gets called then I think the helper could assert it; but I'd rather the caller do the asserts since it's setting up the test scenario
[20:48] <rharper> if that makes sense
[20:48] <powersj> rharper: ok I think I follow all that. I'll give it a shot and come back.
[20:48] <powersj> it does
[20:49] <rharper> cool
[20:49] <rharper> would you like me to paste this log into the MR  comment ?
[20:49] <powersj> sure :)
[20:49] <rharper> k
[21:02] <rharper> smoser: http://paste.ubuntu.com/24468995/  ; looks like we get to have "fun" with centos7 network/cloud-init.local ordering  in systemd
[21:04] <rharper> or I need to figure out how in brpm to tell it we require systemd;  noticed that difference in the 0.7.5 vs. a local build
[21:10]  * rharper learns of -b systemd in brpm 
[21:10] <rharper> now for more fun
[21:21] <rharper> ha, systemd units worked!