blackboxsw_ | same falcojr I'll close out on that review now and put up a 20.3 branch for review | 02:47 |
---|---|---|
paride | smoser, TIL https://github.com/chrlutz/apt-repos, not packaged yet | 09:48 |
paride | `apt-repos dsc` + dget could finally be a good way to download a src package with its .orig from a repo without setting it up in sources.list | 09:48 |
meena | oh, nice | 09:49 |
Carbrich_R2 | HI all, I am having a weird issue with cloud-init, for some kind of strange reason it gets terminated. And I cannot figure out what is causing the issue. I only have this issue with CentOS8, You can find information on | 09:51 |
Carbrich_R2 | https://community.theforeman.org/t/cloud-init-does-not-install-puppet-but-runs-the-final-module-and-succeeds-according-cloud-init/20155/23 of my issue. The only part that is missing that if I remove /var/lib/cloud/ and reboot the client all scripts and providers work. Sometimes 1 of 100 cloud-init works without issues. | 09:51 |
Carbrich_R2 | on should be at but ok. As long as you guys understand it :) | 09:52 |
meena | Carbrich_R2: what's makecache, and why is that part of the puppet installation? | 09:54 |
Carbrich_R2 | HI, Initially I tried a cloud config with installing puppet, later on the road I also removed that part to investigate what is going wrong. At this point I am only trying to install package "SL" with the native package provider. And that also fails. | 09:55 |
meena | https://community.theforeman.org/t/cloud-init-does-not-install-puppet-but-runs-the-final-module-and-succeeds-according-cloud-init/20155/26 ⬅️ this is interesting | 09:56 |
meena | the easiest way to figure out if cloud-init got updated is to check the version before and after the run. but dnf's logs should have some clues, as well | 09:56 |
Carbrich_R2 | Even more strange sometimes let`s say 1 of 100 it works. and after removing '/var/lib/cloud/ and restart the machine again wil result in OK. | 09:57 |
Carbrich_R2 | to anwser your qeustion meena there is nothing logged about updating cloud-init | 09:58 |
Carbrich_R2 | in the dnf lgos | 09:58 |
Carbrich_R2 | logs. If I do a casual ping with count 20 it stops after 8-10 tries. It executes runcmd part for example and it get a sigterm. Nothing in dnf logs etc. | 09:59 |
Carbrich_R2 | All end up with the conclusion that something is killing cloud-init. And I cannot figure out what is happening under the hood. Any idea`s? | 10:00 |
meena | I'm wondering if there's some kind of time limit on execution, but that would be new to me | 10:02 |
Carbrich_R2 | It almost looks like some race condition with system services, that causes this issue. I have a similar setup on our CentOS7 deployment street and that works fine. | 10:04 |
meena | oh, hrm, i thought we fixed those | 10:04 |
meena | in newer versions | 10:05 |
meena | Carbrich_R2: can you upgrade and run your test? | 10:05 |
Carbrich_R2 | which versions? I updated to 19.4 yesterday | 10:05 |
Carbrich_R2 | and tried with that version also | 10:05 |
Carbrich_R2 | :) | 10:05 |
Carbrich_R2 | meena I have no problems with upgrading, but which version do you advice? | 10:07 |
meena | https://github.com/canonical/cloud-init/pull/227 hrm, i thought we merged this "fix" | 10:07 |
meena | Carbrich_R2: the current release is 20.2 | 10:08 |
meena | https://copr.fedorainfracloud.org/coprs/g/cloud-init/el-stable/ ⬅️ im confused about the last release being 9 months ago | 10:11 |
meena | maybe these are just the SRU releases? | 10:11 |
meena | aaaaand I'm very confused about the versions listed there | 10:13 |
Carbrich_R2 | meena No problem, That confused me also. But in what version is that issue fixed you are referring to? | 10:53 |
Carbrich_R2 | meena https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/ I could try a Daily build, but the repo file is missing.. | 10:56 |
Carbrich_R2 | They are referring to " # Save the above yum config to /etc/yum.repos.d/CentOS-cloudinit.repo" But there is nothing above that text.. Confusing | 10:57 |
meena | https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/repo/epel-8/group_cloud-init-cloud-init-dev-epel-8.repo | 10:58 |
Carbrich_R2 | meena THX found it already indeed :) clicked on one of the icons down on the page. | 10:59 |
meena | same | 11:00 |
Carbrich_R2 | Cool, I1`ll have to rebuild my base image with this repo and import in vcenter before i can test. I`ll come back to you if I know the outcome. It will take some time :) | 11:00 |
beantaxi | 8 days ago (https://github.com/canonical/cloud-init/pull/518), make-mime.py was moved from tools to devel, and going forward will be part of the CLI. Seems like a nice idea, and the docs have been able to reflect it. | 11:47 |
beantaxi | However, while the code is in master, it's not in a distro, and it looks like it's not tagged by 2.2-94 either. And so obviously it's not on my ubuntu EC2. Any idea when that will all happen? And, is it considered a docs bug, when the docs on the public web refer to changes that have not happened yet? | 11:49 |
smoser | paride: thanks for the apt-repos reference. | 12:59 |
smoser | beantaxi: what is 2.2-94 ? | 13:01 |
smoser | readthedocs has docs for older versions also. and source code for your distro should contain docs that match exactly with your packaged version. | 13:02 |
smoser | with regard to "when will make-mime" become part of ubuntu, probably in the next SRU (stable release update) | 13:05 |
AnhVoMSFT | @rharper @smoser if the customer has security requirements on /var/tmp being noexec, is there any workaround to allow cloud-init to still do DHCP to talk to metadata service? | 13:42 |
beantaxi | smoser: Thanks. And sorry - I meant 20.2-94, or more precisely 20.2-94-g3d06abc2-0ubuntu1, the github tag. My fault for trying to be terse instead of copy/pasting. (Really sorry -- I | 13:52 |
beantaxi | 'I'd be mad if someone did that and I wasted time figuring out what they were talking about) | 13:53 |
beantaxi | Also thanks for pointing out the docs version option, which I now found, and it does indeed refer to tools/ rather than devel. All good. | 13:55 |
beantaxi | As for in which release will the move to devel happen ... I suppose that question was meta, and I'm curious about if there is an easy way to tell, or a reliably way to make assumptions, about what release a PR will be a part of, other than asking here. | 13:56 |
smoser | beantaxi: well, looking at the debian/changelog it appears every ~ 2m | 14:18 |
smoser | i think thie goal was 1m. | 14:18 |
smoser | AnhVoMSFT: not that i'm aware of. | 14:23 |
beantaxi | smoser: Thanks ... I'm now looking here for Ubuntu 18.04 -> https://launchpad.net/ubuntu/bionic/+source/cloud-init/+changelog, where the release schedule has more variance, and release dates are included at the end of the changelog, when they are included. But I'm new at this; maybe this is the wrong place to look. | 14:26 |
beantaxi | Either way, a heursitic of 'on the order of months' is valuable. | 14:26 |
smoser | beantaxi: maybe Odd_Bloke or blackboxsw_ could point on what the expected cadence is. | 14:46 |
smoser | some other info at https://github.com/cloud-init/ubuntu-sru and https://wiki.ubuntu.com/CloudinitUpdates | 14:47 |
beantaxi | That looks great. I don't think I'll need anything precise but if I do I know who to ask. Thanks! | 14:58 |
blackboxsw_ | @ahosmanMSFT I thought there was a recent bug/branch to try to address the noexec on /tmp/noexec for dhcp https://github.com/canonical/cloud-init/commit/db86753f81af73826158c9522f2521f210300e2b | 15:04 |
Carbrich_R | meena It only worked two times from the 10 and now i am only getting to see the sigterm again. | 15:06 |
Carbrich_R | 2020-08-25 15:03:53,142 - subp.py[DEBUG]: Running command ['/var/lib/cloud/instance/scripts/runcmd'] with allowed return codes [0] (shell=False, capture=False)2020-08-25 15:03:54,145 - util.py[DEBUG]: Cloud-init 20.2+137.gc0450c02-1.el8 received SIGTERM, exiting... Filename: /usr/lib64/python3.6/subprocess.py Function: _try_wait Line number: | 15:06 |
Carbrich_R | 1424 Filename: /usr/lib64/python3.6/subprocess.py Function: wait Line number: 1477 Filename: /usr/lib64/python3.6/subprocess.py Function: communicate Line number: 8552020-08-25 15:03:54,145 - handlers.py[DEBUG]: finish: modules-final/config-scripts-user: FAIL: running config-scripts-user with frequency | 15:06 |
Carbrich_R | once-per-instance2020-08-25 15:03:54,146 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False) | 15:06 |
blackboxsw_ | so I think that pr was to use the known dhclient path if var/tmp is noexec. If that doesn't work, then cloud-init falls back to use the dhclient in /usr/sbin/dhclient | 15:10 |
AnhVoMSFT | Thanks @blackboxsw, I'll take a look at the PR | 15:50 |
blackboxsw_ | @AnhVoMSFT that said, I think ubuntu will still run into an issue with AppArmor profiles on ubuntu in such scenarios | 15:53 |
blackboxsw_ | but it'll 'work' for other distros which don't use apparmor | 15:53 |
blackboxsw_ | beantaxi: the plan for cloud-init releases is to try to create an upstream release (20.2, 20.3, 20.4 etc)any time we have a new significant feature to make SRU generally this will likely mean at least 4 times per year. We document the estimated upstream release date (20.3) in the topic of this channel, and Canonical cloud-generally. | 16:10 |
blackboxsw_ | specifically for your commit, we are trying to cut that 20.3 release today (we were waitong on an azure-related fix for cloud-init related boot timeouts which just landed last night) | 16:10 |
blackboxsw_ | I expect we'll have a release of 20.3 (containing the commitish you are interested in) into Ubuntu 20.10 (groovy) and will start an SRU process to release into Ubuntu Xenial (16.04), bionic and focal | 16:11 |
blackboxsw_ | agreed though. we can/should add some semblance of docs with that intent | 16:12 |
blackboxsw_ | well looks like it's that time again | 16:19 |
blackboxsw_ | #startmeeting cloud-init status meeting | 16:24 |
meetingology | Meeting started Tue Aug 25 16:24:27 2020 UTC. The chair is blackboxsw_. Information about MeetBot at http://wiki.ubuntu.com/meetingology. | 16:24 |
meetingology | Available commands: action commands idea info link nick | 16:24 |
blackboxsw_ | #chair smoser rharper falcojr lucasmoura | 16:24 |
meetingology | Current chairs: blackboxsw_ falcojr lucasmoura rharper smoser | 16:24 |
blackboxsw_ | hey folks, welcome to another cloud-init bi-weekly (or bi-monthly) community status meeting | 16:25 |
blackboxsw_ | or semi-monthly | 16:25 |
blackboxsw_ | ... anyhow. We use this platform/channel to discuss latest and greatest cloud-init, as well as ensuring that there are a couple of upstream developers present to field questions or discussion as needed. | 16:26 |
blackboxsw_ | We gather here in this IRC channel every 2 weeks to discuss current development tasks and progress on cloud-init. All questions and side-conversations welcome | 16:26 |
blackboxsw_ | we keep our meeting minutes from previous meetings here: | 16:26 |
blackboxsw_ | #link https://cloud-init.github.io | 16:27 |
blackboxsw_ | The topics we'll cover today: Previous Actions, Recent Changes, In-progress Development, Community Charter, Office Hours (~30 mins). | 16:27 |
blackboxsw_ | #topic Previous Actions | 16:27 |
blackboxsw_ | Looks like no carryover actions from last meeting | 16:27 |
blackboxsw_ | #topic Recent Changes | 16:27 |
blackboxsw_ | The following changes have landed in tip of master since last meeting (08/14/20). | 16:30 |
blackboxsw_ | found via git log --since 2020-08-014 | 16:30 |
blackboxsw_ | #link https://paste.ubuntu.com/p/h2qxwKwvFw/ | 16:30 |
blackboxsw_ | things to note. we *JUST* cut 20.3 upstream release as of 20 seconds ago. Thanks a bunch James(falcojr) for getting the Azure timeout pr up to close out this upstream release. | 16:31 |
blackboxsw_ | and thanks smoser, rharper and meena for all the reviews and work here. | 16:31 |
blackboxsw_ | in the last couple weeks we got fixes to reduce boot timeouts for certain azure accelerated network instances, fix oracle datasource retries, handle compressed user-data on juju deployed machines in 'cloud-init query', and early boot dhclient will not attempt to run outside of the /var/tmp sandbox directory if that directory is marked no-exec | 16:34 |
blackboxsw_ | thanks otubo for that noexec branch | 16:34 |
blackboxsw_ | #topic In-progress Development | 16:35 |
blackboxsw_ | So thanks all for the 20.3 upstream release. We will be tagging that release and pushing that tag to master just after this meeting | 16:36 |
blackboxsw_ | falcojr: is our release lead for this upstream release and SRU so he'll be working through the release process work items. Thanks falcojr. | 16:36 |
blackboxsw_ | -next step on the release process is to publish to Ubuntu Groovy (20.10) and then queue up a -proposed cloud-init SRU upload into xenial, bionic and focal. | 16:37 |
blackboxsw_ | we will then begin the SRU testing (which we hope to keep at around ~7days) | 16:37 |
blackboxsw_ | as smoser mentioned earlier. cloud-init has to follow this process to update cloud-init in stable Ubuntu releases | 16:38 |
blackboxsw_ | #link https://wiki.ubuntu.com/CloudinitUpdates | 16:38 |
blackboxsw_ | An email will be sent to the cloud-init mailinglist notifying the community about the SRU under test in the event that folks have spare cycles to pitch in on some of the verification effort | 16:39 |
blackboxsw_ | also "in-progress" paride is cleaning up a bunch of automated CI-related failures and lint issues in cloud-init, so expect some branches against cloud-init on that front as well | 16:40 |
blackboxsw_ | #topic Community Charter | 16:40 |
blackboxsw_ | The following topics are still topics for ongoing community development anyone new to cloud-init, or with a bit of time could easily grab one of these bitesized tasks: | 16:41 |
blackboxsw_ | JSON schema extensions to validate user-data before instance launch: https://bugs.launchpad.net/cloud-init/?field.tag=bitesize | 16:41 |
blackboxsw_ | - Datasource documentation and updates | 16:41 |
blackboxsw_ | - cloudinit.net refactor into distro-specific networking subclasses cloudinit.distros.networking: https://bugs.launchpad.net/cloud-init/+bugs?field.tag=net-refactor | 16:41 |
blackboxsw_ | As always: thank you all for bug contributions, PR submissions, triage and discussion participation. | 16:41 |
blackboxsw_ | If anyone would like to be involved more than they currently are, please feel free to contact us here in IRC #cloud-init on Freenode or on the mailing list cloud-init@lists.launchpad.net and we can see how best we can get you "set up" | 16:41 |
blackboxsw_ | #topic Office Hours (next ~30 mins) | 16:42 |
blackboxsw_ | his time of the meeting is really just an open door for any discussions, concerns, bugs, questions or general prodding of upstream devs to make sure existing development work is unblocked where possible. | 16:42 |
blackboxsw_ | *This time | 16:42 |
blackboxsw_ | while we're at it with meeting time.... I'll set the next cloud-init status meeting date in the topic of this IRC channel | 16:42 |
AnhVoMSFT | When do you anticipate the next SRU would be? | 16:42 |
AnhVoMSFT | (after 20.3) | 16:43 |
blackboxsw_ | AnhVoMSFT: thanks for the question. I hope today for 20.3, I think 20.4 will likely align with the next Ubuntu release cycle (20.10) so, October, 2020 | 16:44 |
blackboxsw_ | as mentioned in passing earlier, we are looking at trying to create an upstream release of cloud-init when any significant feature set has landed in tip to make release verification easier (and hopefull include less sprawling sets of broad commit streams). | 16:46 |
blackboxsw_ | this ultimately may translated to 4-5 releases per year | 16:46 |
blackboxsw_ | I expect a frequency of < 3 months | 16:46 |
blackboxsw_ | #topic #cloud-init pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Sep 8 16:15 UTC | 20.2 (Apr 26) | 20.3 (estimated Aug 19th) https://bugs.launchpad.net/cloud-init/+filebug | 16:48 |
blackboxsw_ | next status meeting sept 8th, same time ^ | 16:49 |
AnhVoMSFT | Thanks. There's some planned work for Azure's pre-provisioning v2 and will need some change in cloud-init. We're trying to get a sense of when cloud-init SRUs land so that we can communicate the timeline | 16:49 |
blackboxsw_ | @AnhVoMSFT generally cadence has been ~3 months between SRUs, I think we'd like to see a slightly higher frequency than that because our last SRU was so costly. but I think expectation for this pass is likely 10/15/2020 | 16:52 |
blackboxsw_ | also, if we are looking to test features in Ubuntu Groovy 20.10 images, they don't require an SRU, so upload to that development release are possible any time (we could do that as frequently as weekly if need be) | 16:53 |
AnhVoMSFT | thanks for the clarification @blackboxsw_ | 16:56 |
blackboxsw_ | so as of today, I expect we'll have an SRU in October and likely something beginning of Jan | 16:57 |
blackboxsw_ | I'll take an action for us to communicate via mailinglist the next estimated SRU once this SRU for 20.3 closes out | 16:57 |
blackboxsw_ | #action bbsw add workitem to SRU release process to announce to mailinglist estimated next SRU timeframe | 16:59 |
meetingology | ACTION: bbsw add workitem to SRU release process to announce to mailinglist estimated next SRU timeframe | 16:59 |
blackboxsw_ | and again we hope to have 20.3 SRU complete by next week | 16:59 |
blackboxsw_ | complete, as in publish the 20.3 SRU for next week into Ubuntu xenial and later | 16:59 |
blackboxsw_ | I expect today we publish tip of master (20.3) into Ubuntu Groovy (20.10) so expect to see it in your friendly neighborhood ubuntu cloud-images in the next day or two | 17:00 |
blackboxsw_ | falcojr: I just annotated tag 20.3 and pushed to upstream | 17:04 |
blackboxsw_ | so we have a signed tag | 17:04 |
blackboxsw_ | falcojr: so next task is https://trello.com/c/KxShylli/14-upload-source-tarball-to-launchpad | 17:04 |
falcojr | Cool, I can jump on that | 17:05 |
blackboxsw_ | ok I think we'll wrap up the status meeting. Thanks all for tuning in! | 17:12 |
blackboxsw_ | #endmeeting | 17:13 |
meetingology | Meeting ended Tue Aug 25 17:13:02 2020 UTC. | 17:13 |
meetingology | Minutes: http://ubottu.com/meetingology/logs/cloud-init/2020/cloud-init.2020-08-25-16.24.moin.txt | 17:13 |
Carbrich_r | meena I Was logged out, so I have no idea if you wrote back. | 17:21 |
blackboxsw_ | meeting minutes for the cloud-init status meeting just posted to https://cloud-init.github.io/status-2020-08-25.html#status-2020-08-25 | 17:48 |
rharper | blackboxsw_: thanks! | 17:49 |
blackboxsw_ | np; and ditto | 17:49 |
blackboxsw_ | it'll be good to get this release out and verified | 17:49 |
meena | Carbrich_r: nope, sorry. i had left to pick up my daughter | 17:56 |
meena | who's still not asleep | 17:56 |
Carbrich_r | meena No problem I also went a away to pick-up my two boys :) | 17:56 |
Carbrich_r | I have create a bug report on: https://bugs.launchpad.net/cloud-init/+bug/1892902 With some more log details. | 17:57 |
ubot5 | Ubuntu bug 1892902 in cloud-init "Cloud-init received SIGTERM and is thereby unable to install packages or running runcmd completely" [Undecided,New] | 17:57 |
Carbrich_r | meena Now seeing ubot5 I am in a doubt if that was the right place :S | 17:57 |
Carbrich_r | I`ll have to arrange a few things I`ll check later on. | 18:00 |
meena | I've been bringing my daughter to daycare by bike, and picking her up, by bike, since Friday. and today, i can't feel my body where it doesn't hurt. | 18:00 |
meena | Carbrich_r: now that North America is awake, and the meeting is over, it would be good to repeat your question | 18:01 |
Carbrich_r | I`ll try to do it later on this evening :_ | 18:02 |
Carbrich_r | doorbell is going | 18:02 |
meena | all the people with an @ in front of their name know vastly more than i… aaaaand gone | 18:14 |
Carbrich_r | HI All I have the strange issue that cloud-init sigterms during a runcmd or package install with the native package provider. I have create a bug on https://bugs.launchpad.net/cloud-init/+bug/1892902. That explains my issue I hope. | 18:15 |
ubot5 | Ubuntu bug 1892902 in cloud-init "Cloud-init received SIGTERM and is thereby unable to install packages or running runcmd completely" [Undecided,New] | 18:15 |
Carbrich_r | meena Thx for the tip! about U.S. part ot the world :) That is getting awake. | 18:16 |
=== Carbrich_R_ is now known as Carbrich_R | ||
Carbrich_R | . | 19:58 |
ananke | I'm having a hell of a time trying to figure out why cloud-init throws an error with the following apt config: https://dpaste.com/GZMDAVVEM | 20:52 |
ananke | the error is TypeError: 'str' object does not support item assignment | 20:52 |
ananke | and here's a bigger section of the log: https://dpaste.com/F526QRPAW | 20:53 |
ananke | I'd appreciate any hints on what I may be doing wrong | 20:53 |
ananke | seems the issue persists if I remove the key part, so it's just the source | 21:02 |
beantaxi | Until someone more knowledgable comes along ... which wouldn't be hard ... | 21:03 |
beantaxi | It does not appear that your config matches the structure in the example on https://cloudinit.readthedocs.io/en/20.2/topics/modules.html#apt-configure | 21:04 |
=== meena7 is now known as meena | ||
=== Gaffel_ is now known as Gaffel | ||
beantaxi | They have sources.source1.source and .key. You have sources.source1 and .key | 21:05 |
beantaxi | I have zero idea if that is relevant or not. Just something I noticed. | 21:05 |
ananke | I was following the example from https://cloudinit.readthedocs.io/en/latest/topics/examples.html#install-and-run-chef-recipes, not sure if that differs in that aspect | 21:07 |
beantaxi | Yep. looks like you do match that example. Sorry - that's all I got. | 21:08 |
ananke | though I have nothing to lose to try the other way, thank you :) | 21:08 |
beantaxi | Good luck! | 21:10 |
ananke | ha! it worked! thank you so much | 21:11 |
beantaxi | No way | 21:12 |
beantaxi | Haha -- good on you for trying it! | 21:12 |
powersj | ananke, if you could, can you file a doc bug @ https://bugs.launchpad.net/cloud-init/+filebug | 21:12 |
ananke | powersj: will do! later tonight if you don't mind, parent-teacher conference time | 21:13 |
powersj | no rush, but want to remember to fix this | 21:13 |
blackboxsw_ | falcojr: I'm reviewing your upload pr https://github.com/canonical/cloud-init/pull/548 by performing the same basic new-upstream-snapshot steps now to confirm we see nearly identical diffs | 21:39 |
blackboxsw_ | confirmed no reall diff (other than pkging author) between your branch and mine for upload | 21:42 |
blackboxsw_ | falcojr: https://paste.ubuntu.com/p/Jhvp3YQwZF/ | 21:42 |
blackboxsw_ | I'm confirm sbuild-it works and then I'll hit build-and-push to perform the upload | 21:44 |
blackboxsw_ | assuming that succeeds, we are good to go on groovy release of 20.3 | 21:44 |
blackboxsw_ | bah james I missed the following diff on the ubuntu/devel branch | 22:05 |
blackboxsw_ | https://paste.ubuntu.com/p/RhN4fpbkX5/ | 22:06 |
blackboxsw_ | putting up a minor pr now | 22:06 |
blackboxsw_ | n/m I just changed UNRELEASED to groovy in debian/changelog. which we should have done via a sed command on your branch, but that was my fail to catch it (why I'd like to do more frequent releases, so my muscle memory can take care of things so my brain can forget them) | 22:12 |
blackboxsw_ | falcojr: upload to groovy complete. we can send out the release email now | 22:12 |
blackboxsw_ | thanks @! | 22:12 |
blackboxsw_ | community notice: [ubuntu/groovy-proposed] cloud-init 20.3-0ubuntu1 (Accepted) | 22:15 |
blackboxsw_ | ok 20.3 release is "in" for Ubuntu Groovy (20.04) should show up in cloud images in the next day or two | 22:15 |
blackboxsw_ | falcojr: so we can now start the SRU for 20.3 into Xenial, Bionic and Focal. (once your 20.3 release email is out to the mailing list) | 22:16 |
blackboxsw_ | thanks again | 22:16 |
falcojr | Sounds good. Thank you | 22:26 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!