| dendrobates | Hey are you guys still using the cloud server I set up for you a couple years ago | 07:02 |
|---|---|---|
| dendrobates | I worked with Jussi to set it up | 07:05 |
| dendrobates | Jussi01 | 07:07 |
| dendrobates | It has been compromised and rackspace is wanting it shutdown. | 07:08 |
| MooDoo | morning all | 07:13 |
| Unit193 | What? Really? | 07:50 |
| Unit193 | tsimpson, hggdh, Pici | 07:50 |
| Unit193 | dendrobates: Yes, ubottu and a couple clones are on it. | 07:51 |
| Mikaela | meetingology too I believe, what happened to it? was it one of those not-installed security updates or did someone run something fishy or was some password bruteforced? and AlanBell should possibly also be pinged? | 07:53 |
| dendrobates | Unit193: I need to shut it down. Can you guys move those bots? | 07:53 |
| Unit193 | dendrobates: I don't have access, I pinged a few that do. | 07:54 |
| dendrobates | Rackspace stopped providing it for free about a year ago, and I've been meaning to tell you guys | 07:54 |
| Mikaela | 50 security updates | 07:55 |
| dendrobates | it seems I still have an account. The system is completely owned | 07:59 |
| Unit193 | That doesn't sound good.. | 08:00 |
| Mikaela | was it allowing password login and is there some reason for not at least having unattended-upgrades? | 08:00 |
| dendrobates | I don't maintain the server, I just provided it. | 08:01 |
| dendrobates | There are php processes running as user www-data out of /tmp/tmp | 08:02 |
| dendrobates | this system is attacking others though. We need to stop it ASAP | 08:04 |
| dendrobates | Are you guys running postfix on there? | 08:08 |
| dendrobates | and FTP? | 08:08 |
| Unit193 | jussi and tsimpson are the admins still. | 08:10 |
| Mikaela | I know only about meetingology which I have successfully scped, but the logs are larger than I exppected and I am scping them too just in case they are missed. I don't know if it's end of the world if they are lost though | 08:11 |
| dendrobates | Are you guys running any webservices on there? | 08:12 |
| Unit193 | ubottu.com | 08:13 |
| Unit193 | dendrobates: There's also a wordpress blog, but that's not as vital. | 08:13 |
| dendrobates | If you guys remove all your data, I'll create a new vm for you, until you can find new hosting | 08:14 |
| DJones | Isn't meetingology AlanBell's bot | 08:35 |
| dendrobates | there are around 4k tcp connections on the box. That seems excessive | 08:37 |
| Mikaela | DJones: I have been administrating meetingology for some months | 08:39 |
| DJones | ok, I thought alan still managed it | 08:39 |
| Mikaela | they do, but I have been helping with it and enabling SASL, CertFP etc. | 08:39 |
| Mikaela | https://wiki.ubuntu.com/mikaela tells a little more | 08:40 |
| Unit193 | DJones: Yeah that's his bot, but it runs on ubottu's server as it is the official meetingbot. | 08:43 |
| Unit193 | https://code.launchpad.net/~ubuntu-bots/ubuntu-bots/meetingology after all. | 08:44 |
| dendrobates | a php process is making connections out to thousands of http servers | 08:55 |
| Unit193 | Can you kill that and whack it out of /tmp? Perhaps disable http/php for now until one of them comes online? | 08:56 |
| dendrobates | I can kill it, but the box is owned | 08:57 |
| Unit193 | Besides email, I only have one way to contact either of them and he's not online. | 09:00 |
| dendrobates | pid 1564 is the offending process | 09:02 |
| dendrobates | I think I will kill wordpress when I do it | 09:05 |
| === mkv is now known as m4v | ||
| popey | dendrobates: who has user accounts on the box? | 09:20 |
| Mikaela | I think the accounts might be named after the bots that run on them and you must look at authorized_keys of users, but the only acount that I know surely of is meetingology | 09:22 |
| popey | look for early numbered accounts in /etc/passwd, from 1000 upwards | 09:23 |
| Mikaela | and ubottu unless there is spoofing | 09:23 |
| Mikaela | popey: tsimpson, jussi, meetingology, rclark, ubottu, jpds, seeker, ljl, pici,nalioth, topyli, steev, m4v, sven, jaksa, quassel | 09:25 |
| Mikaela | sudo:x:27:tsimpson,jussi,rclark,devel | 09:26 |
| dendrobates | I think someone got into one of the web apps. | 09:26 |
| dendrobates | I put busybox on and looked or hidden procs and files | 09:26 |
| dendrobates | It seems fine. | 09:27 |
| dendrobates | killing the apache2 service stopped all the outgoing connections | 09:27 |
| popey | whats on there? the bots that we use to log and all the html / log files we serve up over http? | 09:27 |
| dendrobates | I have no idea, but you guys need to get your data off ASAP | 09:28 |
| Mikaela | at least ubottu and meetingology, possibly some other bots | 09:28 |
| popey | dendrobates: sure, but first we need to find someone who has access | 09:28 |
| popey | because I can't get the data off it. | 09:28 |
| popey | (not having an account) | 09:30 |
| Mikaela | do they happen to be on timezones that are currently sleeping? | 09:31 |
| dendrobates | I have access. | 09:32 |
| dendrobates | I can give anyone access | 09:32 |
| AlanBell | o/ popey | 09:33 |
| popey | hey AlanBell thanks | 09:33 |
| popey | see scrollback, we have an issue with the ubotu server, dendrobates (who provided the box) has shut down the (probably owned) apache2 process | 09:33 |
| popey | if rackspace shut it down we have no access to the historical logs or the bots. | 09:34 |
| Mikaela | AlanBell: and I am running scp of meetingology, logs seem to take forever, meetingology itself is copied and also the authorized_keys | 09:34 |
| AlanBell | thanks Mikaela | 09:34 |
| AlanBell | just reading back . . . | 09:34 |
| Mikaela | I should have started this with rsync so I could have moved to tethering which is 100Mbit/s and gotten it faster, but too late now probably and I never manage to be wise in advance | 09:35 |
| popey | you can kill scp and run rsync instead, that's fine, it will carry on | 09:35 |
| popey | (assuming you do it right) | 09:36 |
| dendrobates | we have snapshots of the server, btw | 09:36 |
| Mikaela | I will try, my command is "scp -r meetingology@ubottu.com:/home/meetingology meetingology/" | 09:36 |
| AlanBell | so, there is the bantracker, the IRCC ticketing system (history is probably not that important) | 09:37 |
| AlanBell | a wiki that isn't needed | 09:37 |
| AlanBell | a pastebin that probably isn't needed | 09:37 |
| Unit193 | Well, wiki has plugin info, can be moved though. | 09:37 |
| dendrobates | syslog was scrubbed, so we can assume root compromise | 09:39 |
| AlanBell | eww | 09:39 |
| Mikaela | rsync seems to be copying in different order, but I hope it will see the previously scped logs | 09:40 |
| dendrobates | wtmp as well | 09:40 |
| dendrobates | lol sshd is configured to allow root with no password: fixing | 09:42 |
| Unit193 | >_< | 09:42 |
| Mikaela | "rsync says logs/", I hope it's just wondering what is already downloaded or what there is to download or something | 09:43 |
| Mikaela | *rsync says "logs/" | 09:43 |
| Mikaela | now it seems to download logs | 09:44 |
| * AlanBell also grabs meetingology and /home/ubottu/ubottu | 09:46 | |
| Unit193 | AlanBell: How large is that second one? | 09:47 |
| AlanBell | 3.2G | 09:48 |
| Mikaela | and meetingology is 7.1G so I could have tried downloading it forever with that copper connection | 09:49 |
| AlanBell | there is an owncloud instance (no clue what is is for) | 09:49 |
| AlanBell | several wordpress related things | 09:50 |
| AlanBell | various abandonned projects I think | 09:51 |
| ninnnu | Abandoned Wordpress, what could possibly go wrong.. | 09:51 |
| AlanBell | indeed, surprised it lasted as long as it did tbh! | 09:52 |
| Mikaela | also 50 security upgrades requiring installation | 09:52 |
| * AlanBell has 4.4G of stuff so far | 09:52 | |
| Mikaela | 3,7G | 09:53 |
| AlanBell | 5.0G now :) | 09:53 |
| Mikaela | how fast connection do you have? | 09:53 |
| AlanBell | as fast as it wants to go :) | 09:54 |
| Mikaela | I see | 09:54 |
| AlanBell | I will do a speed test when it is finished, I am not sure how quick things are at the moment | 09:56 |
| Mikaela | ok | 09:56 |
| AlanBell | so, having salvaged anything of value, what happens next? | 09:57 |
| AlanBell | udsbottu and the uds logs are there as well . . . | 09:58 |
| Mikaela | putting it running somewhere else which is hopefully a little more secure, I hope. meetingology will be missed on Monday if not sooner http://fridge.ubuntu.com/calendars | 09:58 |
| Mikaela | 5,5G of meetingology currently | 10:00 |
| popey | how big was the box? | 10:01 |
| popey | (useful data for looking for another one) | 10:01 |
| Mikaela | Filesystem Size Used Avail Use% Mounted on | 10:01 |
| Mikaela | /dev/xvda1 79G 70G 4.8G 94% / | 10:01 |
| dendrobates | AlanBell: I will kill the vm when you are done. | 10:01 |
| popey | dendrobates: how much ram does it have, how many cores? | 10:02 |
| Unit193 | AlanBell: Can you get a mysql dump? | 10:02 |
| dendrobates | AlanBell: I can create another vm, but you have to find a new home, as I can't fund it myself | 10:02 |
| popey | thanks for funding it so far dendrobates ! | 10:03 |
| dendrobates | popey: 2GB, 2vcpu | 10:03 |
| DJones | Presumably a clone of ubottu will need to be dropped into its channels once the vm is shut down | 10:03 |
| DJones | Short term anyway until new hosting/vm is set up | 10:03 |
| popey | thanks dendrobates | 10:04 |
| AlanBell | Unit193: perhaps, any particular database you are looking for? | 10:05 |
| AlanBell | there is even a sugarCRM instance kicking about on it! | 10:06 |
| Unit193 | Just thinking in terms of wiki, wp, etc, etc. | 10:06 |
| dendrobates | let me know when you are done. | 10:15 |
| AlanBell | well my downloads are done for now | 10:15 |
| AlanBell | plenty of stuff I haven't got, and I have no idea what to do with the stuff I have, but it is acquired | 10:16 |
| AlanBell | Unit193: I could get mysql stuff if I had a username/password for the interesting stuff | 10:17 |
| benonsoftware | If the IRC team is wanting to find a new host/sponsor for a server, I'll be happy to pay for an instance on DigitalOcean or some other provider if you're interested. | 10:17 |
| Mikaela | based on the current details it would be expensive on DigitalOcean | 10:18 |
| Mikaela | but no idea about anything else either | 10:18 |
| k1l_ | is there a chance to get some (v)server from the canonical side? | 10:18 |
| popey | The problem with hosting at Canonical is you get very limited access, and you're fighting for IS resource with everyone else. | 10:19 |
| Unit193 | ^ | 10:20 |
| popey | On your own box you have more control, but that also means you're more likely to get owned when nobody takes ownership :( | 10:20 |
| Unit193 | What on earth is taking all the storage on this one? | 10:20 |
| popey | (I am not pointing fingers at anyone here btw) :) | 10:20 |
| Mikaela | benonsoftware: looking at only HDD size, equivalent of ubottu.com at DigitalOcean would be 80$/month | 10:21 |
| Mikaela | but RAM would be just $20, but that is half smaller HDD | 10:21 |
| popey | Well, there's multiple things on that box. | 10:21 |
| popey | It could be split up. | 10:21 |
| popey | For example a logging bot which does just that - log to files - could well be hosted by canonical (possibly) | 10:22 |
| popey | as I imagine it doesn't need a huge amount of intervention. | 10:22 |
| popey | but something that needs a lot of TLC might be more problematic. | 10:22 |
| Unit193 | popey: ubuntulog is. | 10:22 |
| Mikaela | with meetingology we need access to install things with python/pip to $HOME/.local and also access to cron or at least these things come to my mind | 10:23 |
| Mikaela | being able to generate certificates (meetingology uses CertFP and SASL-ECDSA256-CHALLENGEAUTH) with openssl would also be nice, but that could also be done remotely and of course shell access. | 10:24 |
| popey | It seems to me we shouldn't make any rash decisions or purchase any new hosting or anything until we have a concrete list of requirements. | 10:26 |
| AlanBell | indeed | 10:27 |
| popey | Given this isn't a major emergency (in that it's a bit of a bummer, but the project won't stop because of it) | 10:27 |
| Unit193 | AlanBell: BTW, going to run the bantracker by chance? | 10:27 |
| popey | Might be worthy of a discussion over the weekend on the community team and/or irc team list | 10:27 |
| AlanBell | Unit193: I don't know how | 10:27 |
| Mikaela | are those lists open to everyone and if yes, where? | 10:27 |
| popey | yes, both are | 10:28 |
| popey | on lists.ubuntu.com | 10:28 |
| dendrobates | I'll setup a new vm for the time being. | 10:28 |
| popey | http://lists.ubuntu.com/mailman/listinfo/Ubuntu-irc | 10:28 |
| popey | http://lists.ubuntu.com/mailman/listinfo/Ubuntu-community-team | 10:28 |
| popey | probably only needs to go on one list, and -irc is probably the right one? | 10:28 |
| dendrobates | if you guys want it | 10:29 |
| Mikaela | thanks | 10:29 |
| popey | What's the most urgent thing we're missing? | 10:29 |
| popey | (from that box being dead) | 10:29 |
| Unit193 | dendrobates: I'm sure tsimpson will until a plan is made. | 10:29 |
| dendrobates | I'll be back in an hour to check on progress | 10:30 |
| AlanBell | popey: hard to assess the most urgent, there is paste prevention, meeting notes and the ban tracker/factoid stuff which will all be noticed at some point | 10:33 |
| AlanBell | along with the USDbottu stuff which will be noticed when there is a UDS | 10:33 |
| Unit193 | I'll be going in 28 minutes, will need to know if I should pop ubot93 in. | 10:33 |
| AlanBell | it would all fit perfectly cheerfully on one of these https://www.hetzner.de/de/hosting/produkte_vserver/vx6 | 10:34 |
| AlanBell | and I could in theory offer a VM on the side of one of my systems, but I am a little reluctant as it has already been a target | 10:35 |
| popey | you wouldn't put wordpress / sugarcrm / everything else php on it though? | 10:35 |
| AlanBell | nope | 10:36 |
| popey | so would be way less of a target | 10:36 |
| AlanBell | yeah, that is true | 10:36 |
| popey | Shall we co-draft a mail in etherpad to -irc? | 10:38 |
| AlanBell | good plan, but I have to pop out now | 10:38 |
| popey | ok | 10:38 |
| Unit193 | FWIW, Drone` isn't on that server, only unopaste. | 10:39 |
| Mikaela | unopaste was under meetingology | 10:40 |
| DJones | Unit193: Is it worth putting ubot93 in, but leaving it muted, that way it won't be responding to factoids and can just be unmuted if ubottu does drop out | 10:40 |
| Unit193 | DJones: Only issue with that is if someone kicks with a factoid, or uses !foo > nick | 10:41 |
| DJones | Then somebody can just unmute rather than having to wait for you to return to be able to put ubot93 in | 10:41 |
| DJones | True, I did wonder about that | 10:41 |
| Unit193 | And fwiw, tsimp is an admin in that bot. | 10:42 |
| AlanBell | I think some people were using it for irssi/quassle core as well | 10:42 |
| Unit193 | Yes, quasselcore. | 10:42 |
| === mkv is now known as m4v | ||
| dendrobates | I'm back. have you finished moving data? | 12:20 |
| dendrobates | popey: what is the status? | 12:23 |
| popey | http://paste.ubuntu.com/11855670/ was what you missed | 12:24 |
| popey | I don't know if AlanBell Mikaela have grabbed all they can... | 12:24 |
| Mikaela | I have meetingology & logs and I think AlanBell got meetingology & ubottu, no idea about mysql etc. | 12:25 |
| dendrobates | do you want me to create a new vm? | 12:25 |
| dendrobates | someone can email me an ssh key and go to town | 12:26 |
| dendrobates | I can leave the old one up, until you are have everything setup | 12:27 |
| AlanBell | dendrobates: what is the cost situation here? | 12:28 |
| AlanBell | are you being charged for this? | 12:28 |
| dendrobates | yes | 12:28 |
| dendrobates | it's less than $100/mo | 12:29 |
| dendrobates | pennies an hour | 12:29 |
| dendrobates | but over years, it adds up, but a few months is ok | 12:30 |
| AlanBell | indeed, I think it would be good to pause for a bit and decide on a long term home | 12:30 |
| AlanBell | don't really want to move things twice | 12:30 |
| dendrobates | the other option is to ask rackspace for free hosting. I think they will probably say yes | 12:31 |
| dendrobates | then the vms could be moved | 12:31 |
| dendrobates | or hp | 12:31 |
| AlanBell | ok, so clean rebuild, then we have something that can be a bit portable? | 12:32 |
| dendrobates | hopefully | 12:34 |
| dendrobates | anyway I need to work. Do you guys know how to reach me? I'm not in IRC much anymore | 12:35 |
| Mikaela | email address could be made visible to /ns info dendrobates with /ns set hidemail off | 12:36 |
| AlanBell | https://launchpad.net/~dendrobates | 12:36 |
| * AlanBell spots ways to reach dendrobates :) | 12:36 | |
| Mikaela | (or /msg nickserv...) | 12:36 |
| dendrobates | my handle is the same on twitter and github and ubuntu.com | 12:37 |
| dendrobates | someone email me a public key and I'll get e new server up | 12:38 |
| Pici | ubot5: join #ubuntu-server | 16:10 |
| ubot5 | Pici: I am only a bot, please don't think I'm intelligent :) | 16:10 |
| teward | bot fail? | 16:15 |
| Mikaela | Encyclopedia fail, I would say | 16:16 |
| Mikaela | it won't separate what commands are meant to it, what to bot and join doesn't provide feedback on successful joins | 16:17 |
| Unit193 | Pici: tsimpso seems to respond faster via email nowdays, if you haven't emailed yet. | 21:42 |
| Pici | I've copied mostly everything to a new vps, I'll working on configuring it later | 21:56 |
| Unit193 | Well, if you need any assist, I'm around. | 22:02 |
| Unit193 | Uh oh. | 22:25 |
| teward | Unit193: Pici: I was about to say... :) | 22:26 |
| Unit193 | ...Say that it is taco time? | 22:26 |
| Pici | just doing some testing ;) | 22:27 |
| Pici | Unit193: where is drone running out of? ubottu.com as well? | 22:27 |
| Unit193 | Pici: unit193.net | 22:28 |
| Pici | good | 22:28 |
| Unit193 | I offered to move it off my stuff, or whatever you want with it. | 22:28 |
| Pici | well it not being on ubottu.com means one less thing for me to worry about | 22:29 |
| Unit193 | yep. | 22:29 |
| Unit193 | kubot is there, no? | 22:29 |
| teward | Unit193: no, it's not taco tuesday. I was gonna say it's peanut butter jelly time, but i'm trying not to be a smartass here xD | 22:29 |
| Unit193 | Pici: Oh hey, if you do move it over there's some script that copies ubuntu.db (factoid db) to a location other bots can rsync from, this is vital to ubot93. | 22:42 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!