| === matsubara is now known as matsubara-afk | ||
| ScheerI | Hello Dear Members! I would like to use xen under ubuntu maas cloude. Can someone show me some directions about it? | 11:09 |
|---|---|---|
| ScheerI | I was searching howtos about, but I did'nt find anything. | 11:10 |
| roaksoax | rvba: howdy! | 13:12 |
| roaksoax | allenap: howdy | 13:12 |
| rvba | \o roaksoax. | 13:12 |
| allenap | roaksoax: Hello! | 13:12 |
| roaksoax | rvba: so I'fe found a couple errors that seems to be related to JS script | 13:13 |
| rvba | roaksoax: I'm listening :) | 13:13 |
| roaksoax | allenap: so I so you added python-tx-tftp to trunk.. where's the source that you wanted me to package | 13:13 |
| allenap | roaksoax: otp, be with you in 20 minutes. | 13:14 |
| roaksoax | rvba: i found them while testing the power stuff for ipmi, which also is buggy and doesn't work. Trying to fix it | 13:14 |
| roaksoax | allenap: sure | 13:14 |
| allenap | roaksoax: https://github.com/shylent/python-tx-tftp | 13:14 |
| roaksoax | rvba: let me get you access to the instance first | 13:14 |
| rvba | k | 13:14 |
| roaksoax | rvba: oh and the SRU for maas 1.0... you are killing me man.. it requires overwriting the config on the SRYU and I'm not sure whether we wanna do that | 13:15 |
| roaksoax | :) | 13:16 |
| roaksoax | robbiew: you are welcome! here's the ipmi patch http://paste.ubuntu.com/1084518/ you should patch /usr/share/pyshared/provisioningserver/ | 14:08 |
| roaksoax | robbiew: then sudo service maas-pserv restart && sudo service apache2 restart | 14:08 |
| roaksoax | robbiew: i haven't yet uploaded a new version as there are a few JavaScript issues I wanna run first with rvba | 14:08 |
| rvba | roaksoax: I'm back, I'll have a look at the JS issue right now. | 14:09 |
| robbiew | RoAkSoAx: nice, thanks! | 14:09 |
| roaksoax | rvba: awesome. so 1. Add button doesn't respond (sometimes) 2. When editing node, sometimes power parameters boxes are not automatically shown. 3. when editing a node, some images seem to not show. That's up to bzr720 | 14:10 |
| rvba | roaksoax: the "sometimes" is frightening. | 14:10 |
| roaksoax | rvba: is enum.js correlated to any of these? | 14:11 |
| rvba | roaksoax: I don't think so. | 14:12 |
| roaksoax | robbiew: you're welcome. just let me know if you run into any more issues | 14:13 |
| roaksoax | rvba: ok! | 14:13 |
| robbiew | RoAkSoAx: for sure | 14:14 |
| rvba | roaksoax: that's what my gut feeling tells me but I'll investigate and get back to you. | 14:14 |
| roaksoax | rvba: alright, cool, thanks | 14:14 |
| rvba | roaksoax: the problem is indeed related to the enum.py change. | 14:19 |
| rvba | roaksoax: we're looking for code snippets based on the names of the available power methods. | 14:20 |
| roaksoax | rvba: hold on, I applied this patch http://paste.ubuntu.com/1084518/ for it to work | 14:21 |
| roaksoax | rvba: however, even with the patch applied there's JS issues | 14:21 |
| roaksoax | rvba: i.e. I select a different power method, and the boxes stay and JS does not update | 14:21 |
| rvba | roaksoax: all the problems I see are related to the wrong enums.py. | 14:26 |
| roaksoax | rvba: this is what I see: http://people.canonical.com/~andreserl/power.png | 14:27 |
| roaksoax | allenap: the tft is still not operational right? | 14:32 |
| rvba | roaksoax: also, the problem with the images is a bug, plain and simple, a wrong path. I'll fix that right away. | 14:35 |
| roaksoax | rvba: cool | 14:36 |
| roaksoax | rvba: btw.. who disabled http://paste.ubuntu.com/1083378/ | 14:38 |
| roaksoax | rvba: its breaking enlistment | 14:38 |
| roaksoax | ) | 14:38 |
| roaksoax | rvba: because maas-enlist in precise sends after commmissioning action = 2 | 14:39 |
| rvba | roaksoax: hum, bzr blame says Jeroen did that, not sure why... | 14:40 |
| rvba | jtv: are you around by any chance? | 14:40 |
| roaksoax | rvba: heh, ok, I'll reenable it | 14:40 |
| rvba | roaksoax: let me track down in which branch it was disabled first. | 14:41 |
| roaksoax | k ;) | 14:41 |
| rvba | I don't think Jeroen did that without a good reason. | 14:41 |
| rvba | roaksoax: that change looks wrong. | 14:47 |
| roaksoax | rvba: ok, I'll get that fixed then | 14:47 |
| rvba | roaksoax: cool | 14:47 |
| roaksoax | allenap: so let me know when we can discuss the tftp stuff please | 14:51 |
| roaksoax | rvba: would you be so kind of reviewing https://code.launchpad.net/~andreserl/maas/reenable-after-commission/+merge/114206 | 14:59 |
| roaksoax | abd https://code.launchpad.net/~andreserl/maas/correct_power_methods/+merge/114198 | 14:59 |
| roaksoax | i need to roll up an update | 14:59 |
| roaksoax | a packaging update | 14:59 |
| rvba | roaksoax: I'm getting confused now, why did you s/ipmi/ipmitool/ exactly, that seems to be what caused the problem...? | 15:00 |
| roaksoax | rvba: because ipmi looks for a file like: /etc/cobbler/power/power_ipmi.template which doesn't exist and consequently fails | 15:01 |
| rvba | I mean these values are completely internal. But the problem arises where there is a difference between enum.py and enums.js. | 15:01 |
| rvba | roaksoax: oh, I see, maybe we should simply rename the template then. | 15:01 |
| roaksoax | rvba: right, but renaiming the template causes more problems because we have to SRU to precise, and roll testing updates of maas-provision and etc etc | 15:02 |
| rvba | Well, I guess it's six of one and half-a-dozen of the other. | 15:02 |
| rvba | roaksoax: ok | 15:03 |
| roaksoax | rvba: i think once cobbler is ditched, we can change them back | 15:03 |
| rvba | Like I said, these values are internal so it's really not a problem. | 15:03 |
| roaksoax | ok cool :) | 15:03 |
| rvba | roaksoax: but that template should be shipped in MAAS itself rather than fetching those from cobbler. | 15:05 |
| roaksoax | rvba: right, but we are still using cobbler for power management | 15:05 |
| rvba | roaksoax: MAAS supports that now, the templates are in: ./src/provisioningserver/power/templates/ | 15:06 |
| rvba | but we only have ether_wake.template virsh.template right now. | 15:07 |
| roaksoax | rvba: and are these templates being used to do the power management? | 15:07 |
| roaksoax | rvba: because, as I can see, these are still pushed into cobbler | 15:08 |
| rvba | roaksoax: they should be. At least everything is in place to do so. I'm not sure if the very last bit of wiring has been done though. | 15:08 |
| roaksoax | rvba: ok, as far as robbiew tests from yesterday, it is cobbler the one who stll does the power management | 15:09 |
| roaksoax | rvba: http://pastebin.ubuntu.com/1083356/ -> that's the error | 15:09 |
| roaksoax | in cobbler | 15:09 |
| roaksoax | rvba: that's why I';m saying, once we ditch cobbler we can simply merge rename then and add the templates | 15:10 |
| robbiew | RoAkSoAx: I haven't tested the patch you sent yet, but I was able to set IPMI-over-LAN without the previous error...so a good sign | 15:11 |
| robbiew | oddly, I had to select IPMI-over-LAN...save...and then go back to add the IPMI-over-LAN info | 15:11 |
| rvba | robbiew: that's weird. | 15:12 |
| rvba | robbiew: you mean the power info was not persisted the first time you saved the page? | 15:12 |
| roaksoax | rvba: it didn't persist because it failed to save | 15:14 |
| rvba | robbiew: about the branch "reenable-after-commission", I don't think we want to enable "NODE_AFTER_COMMISSIONING_ACTION.CHECK". | 15:14 |
| robbiew | RoAkSoAx: right...I selected it, and I wasn't able to fill in the info | 15:15 |
| robbiew | until I saved and went back to edit again | 15:15 |
| rvba | robbiew: err, roaksoax ^ | 15:15 |
| roaksoax | robbiew: yeah that's a JS issue that will be fixeddnoce role up the nn package | 15:15 |
| roaksoax | robbiew: i'll update the branch | 15:15 |
| roaksoax | err | 15:15 |
| roaksoax | rvba: :) | 15:15 |
| rvba | haha :) | 15:15 |
| roaksoax | haha | 15:15 |
| roaksoax | rvba: so anyways, the problem was thii. 1. edit node. 2. select ipmi. 3. save failure. Why? because when the information was sent to cobbler there was no power template named power_ipmi.template | 15:16 |
| roaksoax | rvba: cobbler failed, telling maas "I failed", and consequently maas didn;t save | 15:17 |
| rvba | I get it now. Note that this whole power management UI was never meant to be used with Cobbler. | 15:17 |
| roaksoax | rvba: so changing s/ipmi/ipmitool sets the power type ot ipmitool in cobbler, which works, and maas can save the node | 15:17 |
| roaksoax | rvba: oh indeed, but it is still doing so | 15:17 |
| roaksoax | rvba: still sending the updates to it | 15:18 |
| rvba | roaksoax: Right, we need to change this. | 15:18 |
| robbiew | RoAkSoAx: just did a juju deploy and the machine was allocated...but not powered on :/ | 15:18 |
| roaksoax | robbiew: uhmm ok I think what it is wrong | 15:21 |
| roaksoax | robbiew: i think we need the ipmi templates then | 15:21 |
| robbiew | RoAkSoAx: I have templates in /etc/cobbler/power | 15:22 |
| robbiew | do they need to be somewhere else? | 15:22 |
| roaksoax | robbiew: i think maas is not using cobbler for that anymore | 15:22 |
| robbiew | ah | 15:23 |
| roaksoax | i'll have to check that | 15:23 |
| rvba | roaksoax: I'm not sure your fix to enum.py will be us very far. Like I said, there parameters are not meant to work with cobbler template so the fact that the names are almost compatible is mostly luck (well, and the fact the our templates are greatly inspired by cobbler's templates). | 15:26 |
| rvba | s/will be/will get/ | 15:26 |
| rvba | s/there parameters/these parameters/ | 15:26 |
| roaksoax | rvba: https://code.launchpad.net/~andreserl/maas/reenable-after-commission/+merge/114206 updated | 15:30 |
| rvba | roaksoax: the thing is that these values were disabled because there is no actual implementation behind them. | 15:32 |
| roaksoax | rvba: right, the problem is this: "power_type = ipmi" is save as power_type in cobbler, and that power type in cobbler does not exists | 15:32 |
| roaksoax | rvba: so when virsh and ether_wake where created in maas, they matched the names of power_type in cobbler | 15:33 |
| rvba | roaksoax: maybe maas enlist should simply use the default value (0) instead. | 15:33 |
| robbiew | well all I know is that if I run the command in the cobbler ipmilan template...it works :) | 15:34 |
| roaksoax | rvba: yeah but I | 15:38 |
| roaksoax | rvba: yeah but I'd need to SRU that | 15:38 |
| roaksoax | and SRU takes times :) | 15:38 |
| roaksoax | robbiew: cool, yueah so it seems that maas is not using cobbler for that anymore | 15:40 |
| robbiew | coolio | 15:40 |
| robbiew | no biggie..it's clearly trivial, so I can tell folks it's coming soon ;) | 15:41 |
| roaksoax | robbiew: i'll try to get that working within maas itself by tonight | 15:41 |
| roaksoax | rvba: removing those after commissioning actions is a feature removal and is not backwards compatible | 15:42 |
| rvba | roaksoax: even better, you can just omit the 'after_commissioning_action' parameters during enlistment and the default action will be picked up by MAAS. | 15:43 |
| rvba | roaksoax: well, not really, we simply removed the UI because there was nothing behind it. | 15:43 |
| rvba | roaksoax: if we want to enable these only to test things, then that's fine but it really doesn't make sense to enable them on trunk as there is no functionality behind it. | 15:46 |
| roaksoax | rvba: right the only problem i see is that we need to fix maas-enlist in precise | 15:49 |
| rvba | roaksoax: is that a problem only because of the SRU thingy? I confess I don't really know how much work that involves so I'll let you and Daviey be the judges on that... but I'd rather not enable an option in the UI which doesn't mean anything yet. | 15:53 |
| roaksoax | rvba: SRU in this case is really simply TBH | 15:56 |
| roaksoax | rvba: but if I wanted to add more functionality then that's a problem SRU'ing | 15:56 |
| roaksoax | or bacporting | 15:56 |
| rvba | roaksoax: so I guess we can fix maas-enlist then. | 15:57 |
| roaksoax | rvba: http://paste.ubuntu.com/1084714/ | 15:57 |
| roaksoax | rvba: that's the fix | 15:57 |
| roaksoax | rvba: thast';d be default | 15:59 |
| rvba | roaksoax: that will use 0 which happens to be the default currently but if you simply remove that parameter, then MAAS' default will be used. | 16:00 |
| roaksoax | rvba: I can do that, but, will it be useful sometime in the future? | 16:01 |
| roaksoax | passing different commissioning option | 16:01 |
| rvba | roaksoax: not sure about that, setting it to 0 is fine by me. | 16:02 |
| roaksoax | rvba: anyways, I'm more concerned about the power_management stuff rather than that | 16:02 |
| roaksoax | rvba: so what do we do about that? I think we should rename back to ipmi/ipmi_lan once cobbler is ditched | 16:04 |
| roaksoax | rvba: and add the templates that can be later renamed too | 16:04 |
| roaksoax | rvba: that's simple enough | 16:04 |
| rvba | roaksoax: we could do that but I confess I'm concerned about the fact that this still uses cobbler's template. That really is a gamble. | 16:05 |
| roaksoax | rvba: i think it is not using cobbler template. robbiew confirmed maas is not powering on a machine, but cobbler is. which means maas seems to not be telling cobbler to power it on | 16:06 |
| roaksoax | rvba: it is either, not send the power_type to be saved in cobbler, or send a power_type that cobbler support, hence the patch | 16:07 |
| rvba | roaksoax: the error you pasted was about cobbler not recognising the power setting, so MAAS talks to cobbler. | 16:08 |
| roaksoax | rvba: right, the problem is maas tells cobbler | 16:08 |
| roaksoax | " | 16:08 |
| roaksoax | "save these power options, and send power_type =ipmi" | 16:08 |
| roaksoax | rvba: cobbler says " | 16:08 |
| roaksoax | rvba: cobbler says "i don't know ipmi, but I have ipmitool ipmilan" | 16:08 |
| roaksoax | rvba: that's why it fails, but when it comes to *power on* it is not using cobbler aparently | 16:09 |
| rvba | roaksoax: right, that makes sense. | 16:09 |
| roaksoax | rvba: when the power inteface was first developed, the power types were set to the same names cobbler supports. In our case, cobbler supports virsh, ether_wake, ipmitool, ipmilan | 16:10 |
| rvba | roaksoax: I'm ok to approve the MP if it unblocks you, but we will need to work on that to get it properly working. | 16:10 |
| roaksoax | rvba: so MAAS power_types were virsh and ether_wake, to *match* cobbler supported ones | 16:10 |
| roaksoax | rvba: yes please. I'll start looking on it after I prepare a package update | 16:11 |
| rvba | roaksoax: ok | 16:11 |
| === matsubara-afk is now known as matsubara | ||
| rvba | roaksoax: ~andreserl/maas/correct_power_methods will be landed shortly. I tried to summarize our findings (well, your finding :) ) on the MP. | 16:19 |
| roaksoax | rvba: awesome, thanks! | 16:19 |
| rvba | roaksoax: thank *you* :) | 16:20 |
| roaksoax | heh :) | 16:25 |
| robbiew | RoAkSoAx: If you just wanna send me a patch to use the right cobbler template, I can roll with that....no need to do work that we have to undo once cobbler is removed | 16:30 |
| rvba | roaksoax: don't forget to regenerate (manually for now) enums.js. | 16:32 |
| roaksoax | robbiew: will do! I'll just try to fix everything to work right rather than have a temporary patch | 16:32 |
| roaksoax | robbiew: will do ;) | 16:32 |
| roaksoax | rvba: will do :) | 16:35 |
| roaksoax | rvba: ah so the power stuff is using celery then | 16:40 |
| rvba | roaksoax: yes, but we have only templates for virsh and ether_wake right now. | 16:41 |
| roaksoax | rvba: yeah i'll take care of the ipmi ones then :) | 16:41 |
| rvba | roaksoax: adding support for ipmi shouldn't be too complicated though. | 16:41 |
| rvba | roaksoax: \o/ | 16:42 |
| rvba | roaksoax: note that we can do it really :) | 16:42 |
| roaksoax | rvba: hehe no worries, I'll take care of it :) | 16:42 |
| rvba | roaksoax: all right, thanks. I need to go now but I'll review your branch(es) tomorrow (my) morning if you have branches up for review by then. | 16:45 |
| roaksoax | rvba: will do, thanks | 16:55 |
| roaksoax | robbiew: do you happen to have the output of the command "/usr/bin/ipmitool -H "$power_address" -U "$power_user" -P "$power_pass" power "$power_mode"" ? | 17:24 |
| robbiew | RoAkSoAx: you mean run it with those variables filled in..or find that output in some sort of log | 17:27 |
| roaksoax | robbiew: i'd like the output of the command if you replafce it with the real values | 17:28 |
| roaksoax | i.e. | 17:28 |
| roaksoax | > ipmitool -I lan -H 1.2.3.4 -f passfile chassis power status | 17:29 |
| roaksoax | Chassis Power is on | 17:29 |
| robbiew | ack | 17:29 |
| * robbiew turns them off first | 17:30 | |
| robbiew | RoAkSoAx: http://pastebin.ubuntu.com/1084872/ | 17:36 |
| roaksoax | robbiew: what about if you pass the "status" option | 17:40 |
| robbiew | I just turned it back off, so the status returned: Chassis Power is off | 17:41 |
| roaksoax | robbiew: ok cool thanks | 17:41 |
| roaksoax | robbiew: so in maas are you using IPMI or IPMI LAN? | 17:42 |
| robbiew | I was using IPMI LAN | 17:43 |
| roaksoax | robbiew: alright cool. I almost have a patch for you | 17:47 |
| robbiew | RoAkSoAx: okay, so I have to run to lunch...will be back in a little over an hour...then I'll have to break this stuff down and leave for Houston, but I'll try and test before | 17:47 |
| robbiew | RoAkSoAx: just email me the patch or link to pastebin...just in case ;) | 17:48 |
| roaksoax | robbiew: will do | 17:48 |
| * roaksoax food | 18:26 | |
| allenap | roaksoax: Hi there. Any ideas about http://askubuntu.com/questions/159436/default-password-for-ubuntu-12-04-maas-image/160391? Lots of people seem to be having a similar problem, where their SSH keys are not being propagated to the nodes. | 19:33 |
| * roaksoax looks | 20:00 | |
| roaksoax | allenap: seems like ssh keys were added *after node enlistment but i thought that bug didnt exist anymore* | 20:03 |
| allenap | roaksoax: Do you mean enlistment, or allocation? I don't remember this bug :_-/ | 20:09 |
| allenap | That should have just been :-/ I haven't been in an accident today. | 20:09 |
| roaksoax | allenap: can't remember TBH, or the keys have to be added before they get accepted after enlistment | 20:10 |
| roaksoax | allenap: i'll have to test | 20:10 |
| roaksoax | allenap: maybe the users accept & commission and then they add the users | 20:16 |
| allenap | roaksoax: Ah, could be. So, maybe not a bug in the conventional sense, but a fairly major usability flaw. I'll talk about it with the other squaddies tomorrow morning. Thanks. | 20:24 |
| roaksoax | allenap: cool. we';ll have to confirm though | 20:24 |
| allenap | roaksoax: Also, lp:python-tx-tftp has imported. | 20:24 |
| roaksoax | allenap: yeah I saw, I'll package it as soon as possible | 20:26 |
| allenap | Awesome, thanks :) | 20:28 |
| roaksoax | allenap: what's the license? | 21:29 |
| allenap | roaksoax: MIT. | 21:29 |
| roaksoax | allenap: coolt hanks | 21:30 |
| roaksoax | allenap: you don't happen to have his email, do you? | 21:36 |
| allenap | roaksoax: No, he won't give it up :) I've been communicating exclusively through issues and pull requests on GitHub. | 21:38 |
| roaksoax | allenap: heh.. that's weird | 21:38 |
| roaksoax | allenap: ahh dahh mark.lvov@gmail.com> | 21:45 |
| allenap | roaksoax: Where did you find that? | 21:47 |
| roaksoax | allenap: launchpad/// | 21:47 |
| roaksoax | allenap: 31. By shylent <mark.lvov@gmail.com> on 2012-07-02 | 21:47 |
| allenap | roaksoax: Hehe, tip top :) | 21:48 |
| allenap | roaksoax: Have a good evening, I'm off now. | 21:49 |
| roaksoax | allenap: you too | 21:49 |
| robbiew | RoAkSoAx: hey...so I applied the patch as you suggested and restarted celery...nothing :/ | 23:05 |
| robbiew | should I try removing the machines, and adding freshly? | 23:05 |
| roaksoax | robbiew: what's the output of /var/log/maas/celery.log? | 23:05 |
| robbiew | http://pastebin.ubuntu.com/1085367/ | 23:07 |
| robbiew | RoAkSoAx: ^ | 23:07 |
| * roaksoax looks | 23:08 | |
| roaksoax | robbiew: never seen that one before, and I don't see it in a newer version of the one I'm running. can you pastebin the whole log? | 23:20 |
| robbiew | one sec | 23:20 |
| roaksoax | robbiew: and can you also pastebin /etc/maas/celeryconfig.py and /etc/init/maas-celery.conf | 23:22 |
| robbiew | http://pastebin.ubuntu.com/1085386/ | 23:24 |
| robbiew | celery.log | 23:24 |
| robbiew | http://pastebin.ubuntu.com/1085387/ | 23:25 |
| robbiew | /etc/maas/celeryconfig.py | 23:25 |
| robbiew | http://pastebin.ubuntu.com/1085388/ | 23:26 |
| robbiew | /etc/init/maas-celery.conf | 23:26 |
| robbiew | RoAkSoAx: there ya go! | 23:26 |
| roaksoax | robbiew: can you add this "env PYTHONPATH="/usr/share/maas/" to /etc/init/maas-celery.conf right before the "exec"? should look like:http://paste.ubuntu.com/1085390/ | 23:28 |
| roaksoax | robbiew: that's already fixed and I'm preparing an update | 23:28 |
| robbiew | done | 23:29 |
| roaksoax | robbiew: so then would be "sudo service maas-celery restart" | 23:29 |
| roaksoax | robbiew: and pastebin /var/log/maas/celery.log again please | 23:29 |
| robbiew | http://pastebin.ubuntu.com/1085393/ | 23:31 |
| roaksoax | robbiew: ok it should work now | 23:32 |
| * robbiew is waiting for bootstrap to complete | 23:33 | |
| robbiew | :) | 23:33 |
| robbiew | will try soon | 23:33 |
| roaksoax | alright cool, in the meantime I'l finish preparing the update | 23:33 |
| robbiew | RoAkSoAx: http://pastebin.ubuntu.com/1085403/ | 23:39 |
| robbiew | failed :/ | 23:39 |
| * roaksoax looks | 23:43 | |
| roaksoax | robbiew: ok, so what if the ipmilan.template gets modified like this> http://paste.ubuntu.com/1085418/ (it would require restart of maas-celery maas-provision and apache2) | 23:54 |
| robbiew | oh wait...power_id is supposed to be the IP address? | 23:55 |
| robbiew | I thought it was the user id number | 23:55 |
| roaksoax | robbiew: the IP address | 23:56 |
| robbiew | ffs | 23:56 |
| roaksoax | robbiew: that might indeed be the cause of the failure | 23:56 |
| roaksoax | robbiew: I'm fixing that too cause the names are bias | 23:57 |
| robbiew | so yeah...I was wondering how the IP was determined...I assumed MAAS provided it based on the hostname lookup | 23:57 |
| roaksoax | it is actually the first time i'm looking at this so it clearly needs lots of things to get improved | 23:57 |
| robbiew | no worries | 23:57 |
| robbiew | me too :) | 23:57 |
| roaksoax | hehe :) | 23:57 |
| robbiew | but given MAAS knows the hostname | 23:58 |
| robbiew | oh wait | 23:58 |
| robbiew | nevermind | 23:58 |
| robbiew | IPMI IP address != hostname | 23:58 |
| * robbiew has to go to team dinner...I'll shoot you an email with the results of ip address change | 23:59 | |
| roaksoax | robbiew: alright, cool, and if that doesn't work try the minimized template :) | 23:59 |
| robbiew | heh | 23:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
