[00:00] <blackboxsw> hrm... ok seeing we get through cloud-init modules:config stages which means the datasource succeeded
[00:01] <blackboxsw> hrm have to step away a bit. sorry for the moment dojordan_ will check it out
[00:01] <blackboxsw> I'll have something on this later
[00:01] <blackboxsw> sorry I should have tested this again this morning
[00:01] <dojordan_> no worries
[00:02] <dojordan_> one thing would be great if you could do, would be to change logging level so we can see more info on the serial port
[00:07]  * blackboxsw clicks enable boot diagnostics logging in the UI and clicks reboot on this instance
[00:19] <blackboxsw> and bailing for dinner
[00:37] <dojordan_> same problem on artful
[01:20] <blackboxsw> smoser: just pushed a merge proposal into bionic for today. I need a bit more time to triage what gives on Azure :/
[01:20] <blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/336513
[01:20] <blackboxsw> ok I'm out for the night. gotta do the bedtime routine with the kiddos. more on azure first thing in my morning
[01:21] <dojordan_> thanks for all the help, sounds good
[01:47] <smoser> blackboxsw: fudge
[01:47] <smoser> https://jenkins.ubuntu.com/server/job/cloud-init-ci/725/console
[01:47] <smoser> :-(
[01:47]  * smoser fail
[01:58] <smoser> i'm fixing and pushing. http://paste.ubuntu.com/26448094/
[01:58] <smoser> rharper, powersj blackboxsw if you're still around, to disagree or +1 that.
[01:58] <rharper> looking
[01:58] <smoser> running tox + centos build && git push upstream HEAD
[01:59] <rharper> looks sane
[01:59] <smoser> thakns
[02:17] <rharper> btw, it's a royal pain to launch multi-subnet/ip instances via the console;  also it would be of great help if the ec2 docs would tell you what the format of the instance data is, for example local-ipv4s is a list of some sort of ipv4 addresses;  but is it comma separated, newline, space?  I can't find any examples with my google-fu so trying to get an instance up to check
[02:46] <rharper> *finally*
[02:46] <rharper> vpc is timeconsuming
[03:00] <blackboxsw> hab
[03:00] <blackboxsw> bah
[03:04] <blackboxsw> resubmitting  the merge proposal
[03:07] <blackboxsw> with a new snapshot from master
[03:09] <rharper> ok, have crude ec2 network metadata to v1 config
[03:09]  * rharper calls it a night
[03:14] <blackboxsw> nice
[03:18] <blackboxsw> ok new MP against bionic up. thanks for the fix smoser
[03:19] <blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/336514
[15:33] <smoser> blackboxsw: on azure...
[15:33] <smoser> you there?
[15:34] <smoser> dojordan_: west coasters.
[15:36] <smoser> (they stay up  late.. https://finance.yahoo.com/news/exclusive-fitbits-6-billion-nights-sleep-data-reveals-us-110058417.html )
[16:03] <blackboxsw> here
[16:03] <blackboxsw> :)
[16:19] <blackboxsw> ok azure triage time
[17:05] <dojordan_> here now @blackboxsw
[17:06] <blackboxsw> good to know you come to work at a reasonable time like the rest of us :) I'm walking through failure path again, as smoser surmised it's likely the cdrom disappearing on us before I cloud-init clean --reboot.... so I'm adding logs etc now and going through that to confirm
[17:07]  * blackboxsw wasn't sure yet why this seemed to work with tip  of master though too. 
[17:07] <dojordan_> hmm, interesting idea. doesn't waagent copy the ovf_env.xml off of the cd ?
[17:09] <dojordan_> FWIW we remove the CD as soon as we get a provisioning message
[17:44] <dojordan_> question, shouldn't we be keeping around the ovf-file before rebooting?
[17:44] <dojordan_> i think clean --reboot deletes it
[17:45] <dojordan_> err, nvm, it lives in /var/lib/waagent/ovf-env.xml
[17:58] <dojordan_> @blackboxsw, same problem on xenial
[18:00] <smoser> dojordan_: i had asked blackboxsw to edit /etc/cloud/cloud.cfg.d/05_logging.cfg and change the console logging from WARN to DEBUG
[18:01] <smoser> and then collect console log
[18:01] <smoser> are you able to easily do that too ?
[18:01] <dojordan_> yeah
[18:02] <smoser> dojordan_: for boot diagnostics
[18:02] <smoser> which storage account type do you need ?
[18:03]  * blackboxsw is already mid reboot/test on  my bionic instance with debug console logs enabled, an azure storage account created and
[18:03] <blackboxsw> boot logs enabled
[18:04] <blackboxsw> ubuntu@40.70.46.88
[18:04] <smoser> ssh-import-id smoser ?
[18:04] <blackboxsw> checking boot logs now to make sure cloud-init reported correctly on this last clean boot
[18:04] <blackboxsw> already done for dojordan and smoser
[18:04] <blackboxsw> i'm in byobu term
[18:05] <smoser> permission denied
[18:05] <smoser> in
[18:05] <blackboxsw> added agin
[18:05] <blackboxsw> must've typod
[18:06] <dojordan_> denied
[18:06] <smoser> try again
[18:07] <dojordan_> cool
[18:08] <blackboxsw> ok let's see here..... cehcking azure cli now to make sure I could see boot logs
[18:08] <blackboxsw> before rebooting
[18:08] <dojordan_> worst case i can always get them :)
[18:09] <blackboxsw>  az vm boot-diagnostics  get-boot-log --ids /subscriptions/12aad61c-6de4-4e53-a6c6-5aff52a83777/resourceGroups/SRUGRP10/providers/Microsoft.Compute/virtualMachines/my-b1
[18:09] <blackboxsw> 'ascii' codec can't decode byte 0xe2 in position 40610: ordinal not in range(128)
[18:09] <blackboxsw> hrm oops az cli
[18:09] <blackboxsw> checking UI
[18:09] <blackboxsw> serial log in UI is working for me
[18:09] <blackboxsw> ok
[18:10] <blackboxsw> old log
[18:10] <blackboxsw> http:pastebin.ubuntu.com/26453160
[18:10] <blackboxsw> http://pastebin.ubuntu.com/26453160
[18:11] <smoser> blackboxsw: 'ordinal not in range'
[18:11] <smoser> ?
[18:11] <blackboxsw> yeah az cli cloudn't decode the boot logs on the machinie
[18:11] <smoser> is that because az is trying to .decode() the console log ?
[18:11] <blackboxsw> yeah
[18:11] <smoser> :-(
[18:11] <blackboxsw> so something to file against azure cli when I dig into it :/
[18:11] <blackboxsw> but UI works
[18:11] <dojordan_> ugh, ill make a bug report
[18:12] <blackboxsw> thanks dojordan_
[18:12] <blackboxsw> lemme get az cli version
[18:12] <dojordan_> can you pastebin the ui logs?
[18:12] <blackboxsw> http://paste.ubuntu.com/26453179/
[18:12] <smoser> dojordan_: you're in good company. this week, we've hit.
[18:12] <smoser>  https://github.com/lxc/pylxd/issues/268
[18:12] <smoser> and
[18:12] <smoser>  https://github.com/boto/botocore/issues/1351
[18:12] <blackboxsw> dojordan_: ui logs is http://pastebin.ubuntu.com/26453160
[18:13] <dojordan_> second boot?
[18:13] <smoser> blackboxsw: yeah, go for it.
[18:13] <blackboxsw> dojordan_: smoser 2nd rebooting now
[18:13] <blackboxsw> ok
[18:16] <blackboxsw> hrm any way to show in cli what power state is on node
[18:16] <dojordan_> let me see
[18:18] <blackboxsw> dojordan_: ok it's looping
[18:18] <blackboxsw> just got logs smoser dojordan_
[18:18] <blackboxsw> looping on reprovidsiondata
[18:18] <blackboxsw> copying now
[18:19] <blackboxsw> new boot log http://pastebin.ubuntu.com/26453225
[18:20] <blackboxsw> I'm looking now
[18:20] <blackboxsw> yeah it's looping on 404 from reprovisioning
[18:21] <blackboxsw> so, something triggered that poll which shouldn't have
[18:21] <dojordan_> DataSourceAzure.py[INFO]: Creating a marker file to poll imds
[18:21] <dojordan_> yup
[18:21] <smoser> well, that part seems like it is functioning as designed.
[18:21] <blackboxsw> hahah
[18:21] <blackboxsw> :)
[18:21] <smoser> dojordan_: logging seems extremely verbose if you're expecting this to sit up for 24 hours before use
[18:22] <dojordan_> but we won't log debug by default right?
[18:22] <smoser> debug does go to log file, but not to console
[18:22] <smoser> looks like < 1k/second. but that'd add up.
[18:23] <smoser> but thats not the issue.
[18:23] <smoser> why did we get into the imds
[18:23] <blackboxsw> so cfg.PreprovisionedVm == True
[18:24] <blackboxsw> something in _extract_preprovisioned_vm_setting returns True
[18:24] <blackboxsw> we need to look over that ovf file again
[18:24] <blackboxsw> I think
[18:24] <dojordan_> my guess is the refactoring broke something. the weird thing is it should have been covered by ut
[18:24] <blackboxsw> yeah I thought so too
[18:25] <dojordan_> im re reading my code now and no idea...
[18:26] <blackboxsw> I'm starting up a 2nd vm now and will run _extract_... on the doc
[18:27] <dojordan_> smart
[18:28] <dojordan_> we didnt see any of those debugs though...
[18:29] <smoser> blackboxsw: smoser@52.151.23.91
[18:29] <smoser> if you want
[18:29] <smoser> take it
[18:30] <blackboxsw> 40.79.65.171
[18:30] <blackboxsw> as well
[18:30] <blackboxsw> 40.79.65.161 rather

[18:31] <blackboxsw> ok,,,, so that should've been interpreted as false
[18:32] <dojordan_> oh no
[18:32] <dojordan_> bool("false") is true
[18:32] <blackboxsw> ahhahhha
[18:32] <blackboxsw> ohhh right
[18:32] <blackboxsw> didn't translate from string type
[18:33] <dojordan_> ill push a fix
[18:33] <dojordan_> thanks for all the help
[18:33] <blackboxsw> cheers. gotta go pickup a kiddo from school
[18:33] <blackboxsw> see ya in a bit
[19:37] <dojordan_> @smoser, @blackboxsw, I pushed a fix, and added another UT that would have caught it. Testing now in azure.
[19:47] <dojordan_> my thoughts on removing the verbose logging: maybe just log a byte every request of something. Also, do we have a log level that goes to the console by default?
[19:58] <blackboxsw> dojordan_: warning level is configured to the console by default.
[20:04]  * blackboxsw wonders about us adding a param in a subsequent branch to url_helper.readurl(quiet=(False|True) then callers handling retries outside of that could turn down the volume of logs
[20:10] <blackboxsw>  testing your latest branch now too
[20:18] <dojordan_> same here, *fingers crossed*
[20:21] <dojordan_> i got permission denied using password auth but at least the ECSDA host key changed on me
[20:23] <smoser> ssh auth shouldnt be affected. if you get there, it really should let you in
[20:23] <smoser> nothing woudl have deleted your keys
[20:23] <dojordan_> password would have been redacted in the ovf-env.xml, not sure what that changes
[20:24] <blackboxsw> I know it's a nit, but changing the log message Start polling IMDS from debug -> warning feels like it really shouldn't be a warning level log
[20:25] <blackboxsw> maybe I'm wrong (I know you are probably just trying to get it to show up in default console log configuration)
[20:26] <dojordan_> right, im open to other options, but it would be nice to get to the console
[20:27] <smoser> i can  understand wanting to see somethign on the console (for azure platofrm perspectivee)
[20:27] <smoser> but itkind of stinks from the users' perspective.
[20:27] <smoser> they have a right to expect WARN in the logs to mean "something went wrong"
[20:28] <smoser> but here nothing in their control actually went wrong.
[20:28] <dojordan_> true...
[20:28] <dojordan_> im fine reverting it now that we found this bug
[20:28] <smoser> yeah. i think that is best for now.
[20:28] <blackboxsw> dojordan_: one more thing while you are in there.
[20:28] <smoser> i have said many times i think python logging lacks level granularity
[20:28] <smoser> and cloudd-init usage of what *is* there is bad.
[20:29] <blackboxsw> there's a util.translate_bool that might be of use in checking that truthy value from ovf file
[20:29] <smoser> it seems to me that this should qualify as INFO level
[20:29] <smoser> and at some point maybe a concerted effort couldg et INFO to the console
[20:33] <blackboxsw> btw smoser and dojordan_ success ubuntu@40.79.65.161
[20:33] <dojordan_> sweet!
[20:33] <dojordan_> what distro?
[20:34] <blackboxsw> dojordan_: bionic, running through xenial now
[20:34] <dojordan_> Y
[20:37] <dojordan_> cool, I got back in on xenial too
[20:38] <dojordan_> just pushed those two changes (correct log level, and util.translate_bool)
[20:47] <blackboxsw> great, xenial looks good for me too.
[20:48] <smoser> \o/
[20:50] <blackboxsw> ok, I'll land this when ci completes it's vote dojordan_
[20:51] <dojordan_> thanks!
[20:57] <blackboxsw> thanks for "dotting the i's and crossing the t's"
[20:59] <robjo> smoser: As touched on in previous discussion platform.linux_distribution() is deprecated in upstream Python and as of version 3.7 is expected to go away, in 3.6 on SUSE it returns an empty tuple, thus useless
[20:59] <robjo> I take it in Ubuntu you guys patched Python
[21:00] <robjo> anyway, I think we shoud make a decisison if we expand the dependencies to python-distro or if cloud-init gets it's own function to determine the distribution
[21:00] <robjo> thoughts?
[21:01] <robjo> https://github.com/nir0s/distro#distro---a-linux-os-platform-information-api
[21:01] <smoser> hm.
[21:02] <smoser> i think i'd just want to build my own.
[21:02] <smoser> s/my/own/
[21:02] <smoser> s/my/our/
[21:03] <smoser> i dont want an external dependency for something as seemingly simple as "figure out if you are on ubuntu, suse, redhat, ...".
[21:03] <smoser> :-(
[21:03] <robjo> fair enough, something like this?
[21:04] <robjo> if os.path.exists('/etc/os-release'):
[21:04] <robjo>   use it and determine the distro
[21:04] <robjo> else:
[21:04] <robjo>   try:
[21:05] <robjo>     platform.linux_distribution()
[21:05] <robjo> except:
[21:05] <robjo> ......
[21:05] <robjo> Sound reasonable?
[21:05] <smoser> yeah i guess. id 'also like to olet the packager easily just set it
[21:08] <robjo> well that's the other option, just punt and make the person running setup set the distro then we save the code all together
[21:12] <smoser> well, i thin i'd l ike it to do the right thing, but if the logic that is there doesnt "do the right thing", then let the packager set it.
[21:13] <smoser> i want trunk to "just work" though
[21:13] <robjo> OK, I'll see what I can come up with
[21:24] <robjo> https://bugs.launchpad.net/cloud-init/+bug/1745235
[23:15] <dojordan_> @smoser and @blackboxsw, thank you for all your help landing this PR. When will the nightly azure images contain these changes?
[23:15] <blackboxsw> heh, was going to ping you that I just landed it :)
[23:15] <blackboxsw> should be in bionic tomorrow
[23:16] <blackboxsw> I'm thinking we will probably SRU in February.... so xenial, artful would have it our next SRU
[23:21] <dojordan_> bionic will work for me :)
[23:22] <blackboxsw> dojordan_: oopsie, sorry I need to propose for merging into bionic
[23:22] <blackboxsw> I'll put up another merge proposal tonight. we can probably land that tomorrow and it'll be published friday
[23:22] <blackboxsw> just landing robjo's btrfs branch too
[23:22] <dojordan_> gotcha. Is the bionic branch just a delayed mirror of master?
[23:24] <blackboxsw> dojordan_: yeah the way we structure bionic publishing is just to mirror all content from master tip
[23:26] <blackboxsw> for SRUs into xenial, zesty artful releases we take  a snapshot of tip as well and if some significant behavior change requires attention to retain backward compatibility we carry a small patch to retain behavior in xenial.
[23:28] <blackboxsw> since bionic is not officially in feature freeze until March 2018, any change in behavior of cloud-init is given the go-ahead, so snapshots are easy https://wiki.ubuntu.com/BionicBeaver/ReleaseSchedule
[23:29] <blackboxsw> SRUs into ubuntu series that are 'stable/released' require a bit more work on our end with testing/verification
[23:29] <blackboxsw> https://wiki.ubuntu.com/CloudinitUpdates for our SRU process (TMI I know)
[23:30] <dojordan_> got it, this explains a lot. (not TMI :) )