[16:02] <blackboxsw> oops well I had the date wrong in the topic for this... but here goes
[16:04] <blackboxsw> #startmeeting Cloud-init bi-weekly status meeting
[16:04] <meetingology> Meeting started Mon Apr 30 16:04:15 2018 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
[16:04] <meetingology> Available commands: action commands idea info link nick
[16:04] <blackboxsw> hi folks, sorry for the mis-representation of when our cloud-init status meeting date. It's time for another episode/update of the happenings in cloud-init.
[16:05] <blackboxsw> Next meeting will be in two weeks: May 7th
[16:05] <blackboxsw> at 16:00 UTC
[16:06] <blackboxsw> The last couple weeks on the upstream side of the house has been a big push to get testing and stability into master for the Ubuntu Bionic release freeze
[16:06] <blackboxsw> ... I'd better start with the topic
[16:07] <blackboxsw> #topic Recent Changes
[16:07] <blackboxsw> The last couple weeks on the upstream side of the house has been a big push to get testing and stability into master for the Ubuntu Bionic release freeze.
[16:07] <robjo> May 7th would be 1 week from today that should be May 14th
[16:07] <blackboxsw> robjo: gah, I did it again. Thank you... glad someone's listening.    Next cloud-init status meeting Monday May 14th 16:00 UTC
[16:08] <blackboxsw> #topic #cloud-init Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting: Monday 5/14 16:00 UTC | cloud-init 18.2 released (03/28/2018)
[16:08] <blackboxsw> ok topic agrees in channel now, so I don't botch it at the end of meeting
[16:09] <blackboxsw> Along with a blitz for stability in Bionic the following changes have been shepherded into tip of master
[16:09] <blackboxsw>     - Add reporting events and log_time around early source of blocking time
[16:09] <blackboxsw>       [Ryan Harper]
[16:09] <blackboxsw>     - IBMCloud: recognize provisioning environment during debug boots.
[16:09] <blackboxsw>       (LP: #1767166)
[16:09] <blackboxsw>     - net: detect unstable network names and trigger a settle if needed
[16:09] <blackboxsw>       [Ryan Harper] (LP: #1766287)
[16:09] <blackboxsw>     - IBMCloud: improve documentation in datasource.
[16:09] <blackboxsw>     - sysconfig: dhcp6 subnet type should not imply dhcpv4 [Vitaly Kuznetsov]
[16:09] <blackboxsw>     - packages/debian/control.in: add missing dependency on iproute2.
[16:09] <blackboxsw>       (LP: #1766711)
[16:09] <blackboxsw>     - DataSourceSmartOS: add locking of serial device.
[16:09] <blackboxsw>       [Mike Gerdts] (LP: #1746605)
[16:09] <blackboxsw>     - DataSourceSmartOS: sdc:hostname is ignored [Mike Gerdts] (LP: #1765085)
[16:09] <blackboxsw>     - DataSourceSmartOS: list() should always return a list
[16:09] <blackboxsw>       [Mike Gerdts] (LP: #1763480)
[16:09] <blackboxsw>     - schema: in validation, raise ImportError if strict but no jsonschema.
[16:09] <blackboxsw>     - set_passwords: Add newline to end of sshd config, only restart if
[16:09] <blackboxsw>       updated. (LP: #1677205)
[16:09] <blackboxsw>     - pylint: pay attention to unused variable warnings.
[16:09] <blackboxsw>     - doc: Add documentation for AliYun datasource. [Junjie Wang]
[16:09] <blackboxsw>     - Schema: do not warn on duplicate items in commands. (LP: #1764264)
[16:10] <blackboxsw> the general theme has been: new IBMCloud datasource support for cloud-init, SmartOS datasource work by mgerdts, and some json schema improvements
[16:12] <blackboxsw> so background on IBM, is that their support used to be ConfigDrive based datasource only, but there is now some additional support for different IBM boot/provisioning  stages, hence a new datasource that can support different boot modew
[16:12] <blackboxsw> *boot modes
[16:14] <blackboxsw> over the last two weeks we've landed an SRU into xenial and artful: 18.2-4-g05926e48-0ubuntu1~16.04.1  and   bionic sits at 18.2-14-g6d48d265-0ubuntu1
[16:15] <mgerdts> On the SmartOS side, my changes are driven by our adoption of bhyve (moving away from kvm/qemu).  qemu provides a dhcp server VMs could fall back to if could-init was missing or misbehaving.  bhyve doesn't have that, so I've been working on getting cloud-init to be more stable with the bhyve serial metadata service.
[16:15] <blackboxsw> Also, to our continuous integration on jenkins we now have an additional test for proposed packages in ubuntu for the bionic release to make sure ubuntu doesn't break across pending upgrades
[16:15] <blackboxsw> #link https://jenkins.ubuntu.com/server/job/cloud-init-integration-proposed-b/
[16:16] <blackboxsw> that integration tests hits the suite of platforms lxd, kvm and ec2
[16:16] <blackboxsw> excellent mgerdts, and thanks for the blitz on these branches
[16:17] <blackboxsw> looks like there are a few still in our review queue that we'll be able to get through once the dust settles on the bionic release  (which should be this week)
[16:17] <blackboxsw> #link https://code.launchpad.net/~cloud-init-dev/cloud-init/+git/cloud-init/+ref/master/+activereviews
[16:19] <mgerdts> Is now the right time to discuss bug 1765801, or is that later?
[16:20] <blackboxsw> I think over the last 2 weeks there have been a couple of requests in channel for how someone goes about getting newer cloud init into RHEL7, if anyone on the line today knows the contact point or process for that it'd be helpful. larsks doesn't seem to be around
[16:20] <blackboxsw> mgerdts: probably in about 10 mins. thanks for brining it up
[16:20] <blackboxsw> hopefully less.
[16:20] <blackboxsw> ok I think that's it for recent changes, next topic (in-progress dev, then office hours (and bug discussion))
[16:21] <blackboxsw> #topic In-progresss Development
[16:21] <blackboxsw> We'll make this one short:
[16:22] <blackboxsw> for ubuntu : bionic just went feature freeze last week, our team has a couple of IBM-related cheanges that we are pulling together for a quick SRU into xenial/artful to handle upgrade path from configdrive -> IBMCloud that we are working on the beginning of this week
[16:22] <blackboxsw> we are also trying to wrap up validation of a Bionic SRU per the following bug
[16:22] <blackboxsw> #link https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1767412
[16:23] <blackboxsw> which grabs a number of the updates I listed in the last topic
[16:23] <blackboxsw> since Ubuntu tends to sync all changes from tip into each release stream
[16:23] <mgerdts> Is there any chance the SmartOS changes can piggy back on that IBM SRU
[16:24] <mgerdts> asked too soon - I see they are mentioned in that bug.
[16:25] <blackboxsw> mgerdts: no worries. good ask. probably not for this IBM SRU into xenial/artful which is going to be an exception to our update rule and only be a single cherry pick, but planning a folllowup SRU in about 2 weeks which will pull all changes from tip into artful/xenial/bionic/chunky releases
[16:25] <mgerdts> ok
[16:25] <blackboxsw> the cherry pick is to fasttrack it for IBM into xenial with minimal risk.
[16:25] <blackboxsw> and we want to pull in all your changes if we can (and perform additional validation)
[16:25] <blackboxsw> so the next SRU is our target
[16:26] <blackboxsw> Also inprogress is some more Azure work on pre-provisioning that should land shortly:
[16:26] <blackboxsw> #link https://code.launchpad.net/~jocha/cloud-init/+git/cloud-init/+merge/344192
[16:27] <blackboxsw> as well as some builddeb fixes and network configuration printout fixes from smoser
[16:29] <blackboxsw> smoser and rharper also worked out  some issues on specific google regions where cloud-init was getting hit by a race condition. Cloud-init started up before the kernel/udev was able to rename network devices to stable names like ens4 etc, so cloud-init's network configuration written ended up breaking because it represented devices like eth0 etc.
[16:29] <blackboxsw> there are a couple of branches in flight to fix this issue:
[16:29] <blackboxsw> #link https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344181
[16:30] <blackboxsw> #link https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/344198
[16:30] <blackboxsw> ok I think that's it for in-progress work. So we'll head to office hours so we can chat bugs, branches reviews etc
[16:31] <blackboxsw> #topic Office Hours (next ~30 mins)
[16:31] <blackboxsw>  We'll be hanging out here for anyone who wants more eyes on a review, feature discussions or bug triage....
[16:32] <blackboxsw> well, some of us will be :) a couple of us are at a feature planning conference for the week.
[16:33] <mgerdts> In https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712 smoser said that he was concerned about how this would interact with eventual network hotplug
[16:34] <mgerdts> There doesn't seem to be a timeline for network hotplug and the lack of network autoreconfig on reboot is has popped up a couple times in the past week.  This is just with a couple early adopters and internal users.
[16:34] <blackboxsw> #link https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712
[16:35] <blackboxsw> just to track it in the meeting
[16:35] <mgerdts> So coming up with some mechanism to make this work soon is pretty important to us.
[16:35] <mgerdts> gotcha, will be sure to do that in the future.
[16:35]  * blackboxsw reads up on that link 
[16:35] <blackboxsw> no worries, I'm pedantic :)
[16:36] <mgerdts> That's how you got chosen to run the meeting, I suppose.  :)
[16:36] <blackboxsw> yeah network hotplug will have a long tail as far as feature develpment (agreed). I believe it's on our charter for this next quarter. but that's what is being discussed this week
[16:37] <blackboxsw> heh on meeting comment ;) too true
[16:39] <blackboxsw> so mgerdts your branch allows metadata to set maintain_network to allow cloud-init to control network configuration each reboot with a True value
[16:39] <blackboxsw> ?
[16:39] <mgerdts> yes
[16:39] <mgerdts> if it's not set to true in our metadata, the traditional behavior stays.
[16:40] <mgerdts> That is, in the default path, any customization that someone does in the guest will not get whacked.
[16:41] <robjo> cloud-netconfig handles hotplug https://github.com/SUSE/Enceladus/tree/master/cloud-netconfig contributions for other distros welcome
[16:42] <blackboxsw> nice reference robjo
[16:42] <blackboxsw> #link https://github.com/SUSE/Enceladus/tree/master/cloud-netconfig
[16:43] <robjo> We currently have no GCE specific information but that is easy enough to add. The GCE guest environment handles this and we use the GCE guest environment code in our images in GCE
[16:43] <blackboxsw> mgerdts: so can a user turn off that feature on an instance once they've already deployed, or is it create-time only
[16:43] <mgerdts> It can be flipped at any time, in the current implementation.
[16:44] <mgerdts> current implementation is only in a development branch
[16:47] <blackboxsw> mgerdts: the only things I can see being an issue with the maintain network in cloud-init is that we are adding the cost of another function call  && metdata dict parse to look for a signal about maintaining the network. I agree that cloud-init having granularity between is_new_instance vs just re-do network, is something that cloud-init should have.
[16:48] <blackboxsw> we probably need to discuss this too with rharper about what short-term vision we can get to while we await our network hotplug support in cloud-init proper
[16:49] <blackboxsw> I'd tend to agree that waiting on fully baked hotplug solution is probably too long in this case
[16:49] <blackboxsw> as that runway will be at least 2 months I'd think
[16:50] <blackboxsw> ok, I'll take an action item to resolve this if we can by next meeting
[16:50] <mgerdts> Not only that, but support for it will likely require changes in the host as well.  We tend not to do host updates very often, so it could be a year or more after the feature is available in images before it will be useful.
[16:51] <blackboxsw> #action blackboxsw to have discussions w/ team on datasource maintaining network on each reboot per  https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712
[16:51] <meetingology> ACTION: blackboxsw to have discussions w/ team on datasource maintaining network on each reboot per  https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712
[16:51] <mgerdts> thanks
[16:51] <blackboxsw> good topic.
[16:52] <mgerdts> Is there another place that is good to catch up with larsks or other people that can offer guidance on for redhat/centos?
[16:52] <blackboxsw> let's see, anything else folks want to chat about? stagnant reviews, bugs of interest etc?
[16:53]  * blackboxsw looks at the last cloud-init community summit attendees list to see if rhel folks have another contacts that was supposed to replace larsks
[16:53] <stanguturi> Chad, Is it possible that someone from cloud-init team can take a look at https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1766538
[16:55] <blackboxsw> mgerts, ryan mccabe is a potential contact too, looks like he's not here either today.
[16:56] <mgerdts> ok, thanks
[16:56] <blackboxsw> hrm, yeah not certain what mechanism is used to get cloud-init updated into RedHat mgerdts. Maybe filing a redhat bug about the request
[16:57] <blackboxsw> mgerdts: https://bugzilla.redhat.com/ maybe
[16:58] <blackboxsw> stanguturi: yes we can, we are trying to sort and understand any bugs against Bionic that we can
[16:58] <mgerdts> ok, I can try that.
[16:58] <blackboxsw> #link https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1766538
[16:58] <stanguturi> blackboxsw: Thanks
[17:00] <blackboxsw> stanguturi: ok, so this is netplan + cloud-init related right?
[17:00] <stanguturi> blackboxsw: Yes.
[17:01] <akik> what does network hotplug mean in cloud-init context?
[17:02]  * blackboxsw tries to remember what vmware datasource does, (like writing files direct to network /etc/network/interfaces.d)
[17:02] <blackboxsw> akik: https://hackmd.io/M1Tae41PQBC7a9qMsurTJw?both  is a shared document for comment on hotplug in cloud-init
[17:02] <blackboxsw> #link https://hackmd.io/M1Tae41PQBC7a9qMsurTJw?both
[17:03]  * blackboxsw looks to see if there was a better doc hrm
[17:03] <stanguturi> blackboxsw: Oh. But in the case of netplan, why does cloud-init remembers?
[17:04] <akik> blackboxsw: does it mean that cloud-init stays running, waiting for new network interfaces to appear?
[17:04] <blackboxsw> akik: right, it would mean that you wouldn't have to reboot cloud-init if devices get added at a later time (post-boot)
[17:05] <blackboxsw> cloud-init would listen to some sort of event channel and react, re-write, and apply network config to add new devices
[17:06] <akik> would it do the same thing as you could do with ansible or puppet? sorry i'm trying to understand why you would do it with cloud-init
[17:08] <blackboxsw> akik: you would try to do it with cloud-init if you didn't want to rely on additional configuration management solutions if the only thing you needed was network config to reflect reality (not full system configuration and system automation)
[17:09]  * blackboxsw has more puppet/chef background than ansible.
[17:09] <blackboxsw> cloud-init does currently detect and write network configuration based on what the user/cloud-metadata tell us is the proper config for the instance
[17:10] <akik> i only thought of cloud-init to do the initial configuration
[17:10] <blackboxsw> so it would follow that if the metadata could dynamically tell the instance that network config has changed, cloud-init should probably try to react to that to fix the config to match the updated network configuration
[17:11] <blackboxsw> akik: correct. cloud-init current only handle initial boot config and leaves the rest up whatever mechanism someone uses to update detailed config after that boot
[17:11] <akik> ok thanks
[17:12] <blackboxsw> akik: and we'd make that feature configurable (handle hotplug:True/False) so if users have other services handling hotplug cloud-init wouldn't collide
[17:13] <blackboxsw> ok I think we're hitting the end of office hours. please feel free to continue discussion, we all poke around here throughout the day as our primary means of communication
[17:13] <blackboxsw> thanks robjo akik stanguturi and mgerdts for the lively discussion
[17:13] <blackboxsw> stanguturi: I'll dig up more info on that bug today
[17:13] <mgerdts> thank you
[17:13] <blackboxsw> as always notes will be here
[17:14] <blackboxsw> #link https://cloud-init.github.io/
[17:14] <blackboxsw> #endmeeting
[17:14] <meetingology> Meeting ended Mon Apr 30 17:14:19 2018 UTC.
[17:14] <meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2018/cloud-init.2018-04-30-16.04.moin.txt
[17:33] <stanguturi> Thanks blackboxsw
[17:41] <smoser> blackboxsw: after quite too long... just updated
[17:41] <smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344784
[17:43] <smoser> there is one behavioral change there.
[18:03] <blackboxsw> smoser: checking now
[18:04] <smoser> just rebaed and fixed the pylint herror
[18:09] <smoser> blackboxsw: could you  mark https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/342334 as approved ? or just make sure you'lre ok with that ? and then i'll queue it into ubuntu/devel
[18:13] <blackboxsw> smoser: ok so you drop the is_ibm_cloud == not_found  detection from nocloud in ds-identify now right?
[18:20] <blackboxsw> minor unittest change and +1 on it. I'm testing the updated deb in ibm w/ a nocloud setup  to be sure.
[18:21] <smoser> correct. i dropped that (previously)
[18:21] <smoser> the new change is to drop the ibm_cloud detection for the config-drive seed case.
[18:21] <blackboxsw>  ok I must've missed that earlier, but I wanted a specifically exercise tha path (as I didn't look at it formerly)
[18:24] <blackboxsw> do we want to queue your merge 344784 too?
[18:30] <blackboxsw>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/342334 is approved, but  doesn't it need a rebase now to fix versioning in debian/chanelog
[18:32] <smoser> blackboxsw: yeah... so thats kind of an interesting topic i talked iwth rharper some last week.
[18:33] <smoser> last time i proposed two MPs for merge int ubuntu/* branches (the addition of the isc-dhcp-client and iproute2)
[18:33] <smoser> i did not upate the debian/changelog
[18:33] <smoser> largely because of that.
[18:33] <smoser> but then... debian/changelog didn't get updated...
[18:33] <smoser> i caught those later before the SRU upload.
[18:33] <smoser> i'm not really sure how to handle this on the commits to ubuntu/ branches
[18:42] <blackboxsw> so a commit to debian/changelog with an out of order version won't matter will if we are following up with something >= during our next 'C' package release right?
[19:03] <smoser> blackboxsw: you are corect. we do not want to merge that out of order like that.
[19:03] <blackboxsw> Just validated nocloud and configdrive behavior on IBM: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344784 can land with minor unittest tweak
[19:08] <smoser> blackboxsw: the line go shorter then original
[19:08] <smoser> ie.. at first i neded the paren
[19:08] <smoser> i'll drop . thanks.
[19:08] <blackboxsw> guessed as much
[19:09] <smoser> pushed update
[19:22] <blackboxsw> smoser: landed in tip
[19:27] <blackboxsw> as far as the dropping of new-upstream-snapshot. it has current merge conflicts with ubuntu/devel, so I'm not sure how to resolve it other than rebasing and rebuilding debian/changelog with a new entry, or manually resolving the conflict and injecting this debian/changelog item where you think best in the most recent changelog section.
[19:28] <blackboxsw> I guess it's easiest in the future if we don't let changes against ubuntu/devel age, and try to resolve them before we pull in any other new-upstream-snapshots against that branch.
[19:32] <smoser> blackboxsw: cat a minute ?
[19:32] <smoser> chat even
[19:36] <blackboxsw> chat with a cat on my way
[20:07] <blackboxsw> https://hackmd.io/jlq3C4qbSgurZ_DZ5GTiuw
[20:36] <ahasenack> did you guys see #1768118? fwiw, i tried such an upgrade and it worked just fine
[20:39] <blackboxsw> bug #1768118
[20:40] <ahasenack> not sure why it was filed against cloud-init, it's lacking many details
[20:40] <blackboxsw> thanks ahasenack looking now.
[20:40] <blackboxsw> smoser's on it ahasenack thanks
[20:44] <blackboxsw> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1766538 here's another network related concern to peek at
[20:51] <rharper> blackboxsw: I don't suspect that's a cloud-init issue, 18.04 desktop nor server are going to have ifupdown installed;  I don't think we're getting the whoe picture there
[20:52] <blackboxsw> rharper: yeah we are talking about this now. we're gonna ask for cloud-init collect-logs
[20:52] <smoser> rharper: yeah.
[20:52] <rharper> well, logs won't matter
[20:53] <blackboxsw> and go to bed ;)
[20:53] <rharper> 18.04 has never had ifupdown in it
[20:53] <rharper> so someone's not telling the whole story
[20:53] <rharper> it's also vmware
[20:53] <blackboxsw> yeah agreed
[20:53] <smoser> it absolutely rendered witih netplan
[20:53] <rharper> which has quite a few odd bugs filed w.r.t cloud-init and no vmware datasource, etc
[20:53] <smoser> the logs are clear on that.
[20:53] <rharper> so I suspect testing issue
[20:53] <rharper> hehe
[20:53] <blackboxsw> and multiple boots in that log listed too
[20:54] <blackboxsw> might have started as xenial or something who knows
[20:54] <rharper> yes
[20:54] <rharper> so, a dist-upgrade to 18.04 would make sense
[20:54] <rharper> but the scenario is far from clear in the bug
[20:57] <smoser> mgerdts: https://pythonhosted.org/smartdc/
[20:57] <smoser> is that something you would recommend ? have experience with ?
[20:59] <mgerdts> I don't have experience with it, but I can ask around internally to see what people say.  There's also #smartos that is pretty active.
[20:59] <mgerdts> version history that ends about 5 years ago makes me worry.
[21:00] <blackboxsw> rharper: marked incomplete
[21:00] <smoser> mgerdts: fair. i'm basically looking for something to launch an instance with python. none of our tools are node , so i dont want that at the moment.
[21:06] <blackboxsw> smoser: for xenial https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/344866
[21:09] <smoser> blackboxsw: i think you cherry-ipcked the wrong thing
[21:09] <blackboxsw> wow typo, pulling again sorry
[21:09] <smoser> 11172924a48a47a7231d19d9cefe628dfddda8bf
[21:11] <mgerdts> smoser: one engineer reported having success a couple years back, but others remarked that the examples seemed to use obsolete arguments.  It doesn't seem as though there's a better maintained alternative.  :(
[21:12] <smoser> mgerdts: thanks.
[21:12] <mgerdts> In particular: "the createmachine example there uses "dataset" for the image argument... using URNs instead of UUIDs."
[21:13] <smoser> blackboxsw: i'll go ahead and grab that
[21:13] <smoser> but it loks like there are some differences between yours and mine
[21:14] <smoser> http://paste.ubuntu.com/p/5NBPqmpBCS/
[21:14] <smoser> i think that is due to
[21:14] <smoser> a.) ~/.quiltrc http://paste.ubuntu.com/p/M3v6cJDfHg/
[21:14] <smoser> b.) i probably have newer git and it is picking 8 hash by default
[21:14] <blackboxsw> I repushed, but go4it smoser
[21:15] <blackboxsw> checking
[21:15] <smoser> probably some changes needed to debian/cherry-pick to specify both of those things
[21:16] <blackboxsw> probably right, I did this from a xenial env
[21:16] <blackboxsw> if I did it from my bionic env it probably would have been equal
[21:17] <smoser> well, the 'ab' style comes from ~/.quiltrc
[21:17] <smoser> maybe i can look at making it do the rigiht thing all the time tomorrow
[21:22] <smoser> blackboxsw: uploading. i did take my version, but with your name.
[21:22] <smoser> as soon as build completes it will upload. oh. fudge. failed build.
[21:22] <smoser> fudge.
[21:22] <smoser> namedtuple usage
[21:23] <smoser> http://paste.ubuntu.com/p/99RW7NrSTQ/
[21:24] <smoser> that commit isit stand alone.
[21:24] <smoser> it needs 6ef92c98c3d2b127b05d6708337efc8a81e00071
[21:25] <smoser> blackboxsw: i'll be b ack in in 3 hours or so. sorry
[21:26] <blackboxsw> gah, ok
[21:27] <blackboxsw> sounds good
[21:32] <blackboxsw> strange as tox completed for me locally
[21:33] <blackboxsw> ahh can reproduce w/ make deb
[21:46] <smoser> blackboxsw: i think maybe just cherry pick the one you had and the other
[21:46] <smoser> they're both ibm related
[21:46] <blackboxsw> wow 3 hours was fast
[21:46] <smoser> and for ibm internal test, they might want the debug thing.
[21:46] <blackboxsw> sorry pushing that
[21:46] <smoser> yeah.
[21:46] <smoser> i just thought that
[21:46] <smoser> and i have to go afk now.
[21:46] <smoser> that make sense thouigh? 6ef92c98c3d2b127b05d6708337efc8a81e00071
[21:46] <smoser> is what you need to make the test ru
[21:46] <smoser> run
[21:46] <smoser> i think