[07:02] <dendrobates> Hey are you guys still using the cloud server I set up for you a couple years ago
[07:05] <dendrobates> I worked with Jussi to set it up
[07:07] <dendrobates> Jussi01
[07:08] <dendrobates> It has been compromised and rackspace is wanting it shutdown.
[07:13] <MooDoo> morning all
[07:50] <Unit193> What?  Really?
[07:50] <Unit193> tsimpson, hggdh, Pici
[07:51] <Unit193> dendrobates: Yes, ubottu and a couple clones are on it.
[07:53] <Mikaela> meetingology too I believe, what happened to it? was it one of those not-installed security updates or did someone run something fishy or was some password bruteforced? and AlanBell should possibly also be pinged?
[07:53] <dendrobates> Unit193: I need to shut it down.  Can you guys move those bots?
[07:54] <Unit193> dendrobates: I don't have access, I pinged a few that do.
[07:54] <dendrobates> Rackspace stopped providing it for free about a year ago, and I've been meaning to tell you guys
[07:55] <Mikaela> 50 security updates
[07:59] <dendrobates> it seems I still have an account.  The system is completely owned
[08:00] <Unit193> That doesn't sound good..
[08:00] <Mikaela> was it allowing password login and is there some reason for not at least having unattended-upgrades?
[08:01] <dendrobates> I don't maintain the server, I just provided it.
[08:02] <dendrobates> There are php processes running as user www-data out of /tmp/tmp
[08:04] <dendrobates> this system is attacking others though.  We need to stop it ASAP
[08:08] <dendrobates> Are you guys running postfix on there?
[08:08] <dendrobates> and FTP?
[08:10] <Unit193> jussi and tsimpson are the admins still.
[08:11] <Mikaela> I know only about meetingology which I have successfully scped, but the logs are larger than I exppected and I am scping them too just in case they are missed. I don't know if it's end of the world if they are lost though
[08:12] <dendrobates> Are you guys running any webservices on there?
[08:13] <Unit193> ubottu.com
[08:13] <Unit193> dendrobates: There's also a wordpress blog, but that's not as vital.
[08:14] <dendrobates> If you guys remove all your data, I'll create a new vm for you, until you can find new hosting
[08:35] <DJones> Isn't meetingology AlanBell's bot
[08:37] <dendrobates> there are around 4k tcp connections on the box.  That seems excessive
[08:39] <Mikaela> DJones: I have been administrating meetingology for some months
[08:39] <DJones> ok, I thought alan still managed it
[08:39] <Mikaela> they do, but I have been helping with it and enabling SASL, CertFP etc.
[08:40] <Mikaela> https://wiki.ubuntu.com/mikaela tells a little more
[08:43] <Unit193> DJones: Yeah that's his bot, but it runs on ubottu's server as it is the official meetingbot.
[08:44] <Unit193> https://code.launchpad.net/~ubuntu-bots/ubuntu-bots/meetingology after all.
[08:55] <dendrobates> a php process is making connections out to thousands of http servers
[08:56] <Unit193> Can you kill that and whack it out of /tmp?  Perhaps disable http/php for now until one of them comes online?
[08:57] <dendrobates> I can kill it, but the box is owned
[09:00] <Unit193> Besides email, I only have one way to contact either of them and he's not online.
[09:02] <dendrobates> pid 1564 is the offending process
[09:05] <dendrobates> I think I will kill wordpress when I do it
[09:20] <popey> dendrobates: who has user accounts on the box?
[09:22] <Mikaela> I think the accounts might be named after the bots that run on them and you must look at authorized_keys of users, but the only acount that I know surely of is meetingology
[09:23] <popey> look for early numbered accounts in /etc/passwd, from 1000 upwards
[09:23] <Mikaela> and ubottu unless there is spoofing
[09:25] <Mikaela> popey: tsimpson, jussi, meetingology, rclark, ubottu, jpds, seeker, ljl, pici,nalioth, topyli, steev, m4v, sven, jaksa, quassel
[09:26] <Mikaela> sudo:x:27:tsimpson,jussi,rclark,devel
[09:26] <dendrobates> I think someone got into one of the web apps.
[09:26] <dendrobates> I put busybox on and looked or hidden procs and files
[09:27] <dendrobates> It seems fine.
[09:27] <dendrobates> killing the apache2 service stopped all the outgoing connections
[09:27] <popey> whats on there? the bots that we use to log and all the html / log files we serve up over http?
[09:28] <dendrobates> I have no idea, but you guys need to get your data off ASAP
[09:28] <Mikaela> at least ubottu and meetingology, possibly some other bots
[09:28] <popey> dendrobates: sure, but first we need to find someone who has access
[09:28] <popey> because I can't get the data off it.
[09:30] <popey> (not having an account)
[09:31] <Mikaela> do they happen to be on timezones that are currently sleeping?
[09:32] <dendrobates> I have access.
[09:32] <dendrobates> I can give anyone access
[09:33] <AlanBell> o/ popey
[09:33] <popey> hey AlanBell thanks
[09:33] <popey> see scrollback, we have an issue with the ubotu server, dendrobates (who provided the box) has shut down the (probably owned) apache2 process
[09:34] <popey> if rackspace shut it down we have no access to the historical logs or the bots.
[09:34] <Mikaela> AlanBell: and I am running scp of meetingology, logs seem to take forever, meetingology itself is copied and also the authorized_keys
[09:34] <AlanBell> thanks Mikaela
[09:34] <AlanBell> just reading back . . .
[09:35] <Mikaela> I should have started this with rsync so I could have moved to tethering which is 100Mbit/s and gotten it faster, but too late now probably and I never manage to be wise in advance
[09:35] <popey> you can kill scp and run rsync instead, that's fine, it will carry on
[09:36] <popey> (assuming you do it right)
[09:36] <dendrobates> we have snapshots of the server, btw
[09:36] <Mikaela> I will try, my command is "scp -r meetingology@ubottu.com:/home/meetingology meetingology/"
[09:37] <AlanBell> so, there is the bantracker, the IRCC ticketing system (history is probably not that important)
[09:37] <AlanBell> a wiki that isn't needed
[09:37] <AlanBell> a pastebin that probably isn't needed
[09:37] <Unit193> Well, wiki has plugin info, can be moved though.
[09:39] <dendrobates> syslog was scrubbed, so we can assume root compromise
[09:39] <AlanBell> eww
[09:40] <Mikaela> rsync seems to be copying in different order, but I hope it will see the previously scped logs
[09:40] <dendrobates> wtmp as well
[09:42] <dendrobates> lol sshd is configured to allow root with no password: fixing
[09:42] <Unit193> >_<
[09:43] <Mikaela> "rsync says logs/", I hope it's just wondering what is already downloaded or what there is to download or something
[09:43] <Mikaela> *rsync says "logs/"
[09:44] <Mikaela> now it seems to download logs
[09:46]  * AlanBell also grabs meetingology and /home/ubottu/ubottu
[09:47] <Unit193> AlanBell: How large is that second one?
[09:48] <AlanBell> 3.2G
[09:49] <Mikaela> and meetingology is 7.1G so I could have tried downloading it forever with that copper connection
[09:49] <AlanBell> there is an owncloud instance (no clue what is is for)
[09:50] <AlanBell> several wordpress related things
[09:51] <AlanBell> various abandonned projects I think
[09:51] <ninnnu> Abandoned Wordpress, what could possibly go wrong..
[09:52] <AlanBell> indeed, surprised it lasted as long as it did tbh!
[09:52] <Mikaela> also 50 security upgrades requiring installation
[09:52]  * AlanBell has 4.4G of stuff so far
[09:53] <Mikaela> 3,7G
[09:53] <AlanBell> 5.0G now :)
[09:53] <Mikaela> how fast connection do you have?
[09:54] <AlanBell> as fast as it wants to go :)
[09:54] <Mikaela> I see
[09:56] <AlanBell> I will do a speed test when it is finished, I am not sure how quick things are at the moment
[09:56] <Mikaela> ok
[09:57] <AlanBell> so, having salvaged anything of value, what happens next?
[09:58] <AlanBell> udsbottu and the uds logs are there as well . . .
[09:58] <Mikaela> putting it running somewhere else which is hopefully a little more secure, I hope. meetingology will be missed on Monday if not sooner http://fridge.ubuntu.com/calendars
[10:00] <Mikaela> 5,5G of meetingology currently
[10:01] <popey> how big was the box?
[10:01] <popey> (useful data for looking for another one)
[10:01] <Mikaela> Filesystem      Size  Used Avail Use% Mounted on
[10:01] <Mikaela> /dev/xvda1       79G   70G  4.8G  94% /
[10:01] <dendrobates> AlanBell: I will kill the vm when you are done.
[10:02] <popey> dendrobates: how much ram does it have, how many cores?
[10:02] <Unit193> AlanBell: Can you get a mysql dump?
[10:02] <dendrobates> AlanBell: I can create another vm, but you have to find a new home, as  I can't fund it myself
[10:03] <popey> thanks for funding it so far dendrobates !
[10:03] <dendrobates> popey: 2GB, 2vcpu
[10:03] <DJones> Presumably a clone of ubottu will need to be dropped into its channels once the vm is shut down
[10:03] <DJones> Short term anyway until new hosting/vm is set up
[10:04] <popey> thanks dendrobates
[10:05] <AlanBell> Unit193: perhaps, any particular database you are looking for?
[10:06] <AlanBell> there is even a sugarCRM instance kicking about on it!
[10:06] <Unit193> Just thinking in terms of wiki, wp, etc, etc.
[10:15] <dendrobates> let me know when you are done.
[10:15] <AlanBell> well my downloads are done for now
[10:16] <AlanBell> plenty of stuff I haven't got, and I have no idea what to do with the stuff I have, but it is acquired
[10:17] <AlanBell> Unit193: I could get mysql stuff if I had a username/password for the interesting stuff
[10:17] <benonsoftware> If the IRC team is wanting to find a new host/sponsor for a server, I'll be happy to pay for an instance on DigitalOcean or some other provider if you're interested.
[10:18] <Mikaela> based on the current details it would be expensive on DigitalOcean
[10:18] <Mikaela> but no idea about anything else either
[10:18] <k1l_> is there a chance to get some (v)server from the canonical side?
[10:19] <popey> The problem with hosting at Canonical is you get very limited access, and you're fighting for IS resource with everyone else.
[10:20] <Unit193> ^
[10:20] <popey> On your own box you have more control, but that also means you're more likely to get owned when nobody takes ownership :(
[10:20] <Unit193> What on earth is taking all the storage on this one?
[10:20] <popey> (I am not pointing fingers at anyone here btw) :)
[10:21] <Mikaela> benonsoftware: looking at only HDD size, equivalent of ubottu.com at DigitalOcean would be 80$/month
[10:21] <Mikaela> but RAM would be just $20, but that is half smaller HDD
[10:21] <popey> Well, there's multiple things on that box.
[10:21] <popey> It could be split up.
[10:22] <popey> For example a logging bot which does just that - log to files - could well be hosted by canonical (possibly)
[10:22] <popey> as I imagine it doesn't need a huge amount of intervention.
[10:22] <popey> but something that needs a lot of TLC might be more problematic.
[10:22] <Unit193> popey: ubuntulog is.
[10:23] <Mikaela> with meetingology we need access to install things with python/pip to $HOME/.local and also access to cron or at least these things come to my mind
[10:24] <Mikaela> being able to generate certificates (meetingology uses CertFP and SASL-ECDSA256-CHALLENGEAUTH) with openssl would also be nice, but that could also be done remotely and of course shell access.
[10:26] <popey> It seems to me we shouldn't make any rash decisions or purchase any new hosting or anything until we have a concrete list of requirements.
[10:27] <AlanBell> indeed
[10:27] <popey> Given this isn't a major emergency (in that it's a bit of a bummer, but the project won't stop because of it)
[10:27] <Unit193> AlanBell: BTW, going to run the bantracker by chance?
[10:27] <popey> Might be worthy of a discussion over the weekend on the community team and/or irc team list
[10:27] <AlanBell> Unit193: I don't know how
[10:27] <Mikaela> are those lists open to everyone and if yes, where?
[10:28] <popey> yes, both are
[10:28] <popey> on lists.ubuntu.com
[10:28] <dendrobates> I'll setup a new vm for the time being.
[10:28] <popey> http://lists.ubuntu.com/mailman/listinfo/Ubuntu-irc
[10:28] <popey> http://lists.ubuntu.com/mailman/listinfo/Ubuntu-community-team
[10:28] <popey> probably only needs to go on one list, and -irc is probably the right one?
[10:29] <dendrobates> if you guys want it
[10:29] <Mikaela> thanks
[10:29] <popey> What's the most urgent thing we're missing?
[10:29] <popey> (from that box being dead)
[10:29] <Unit193> dendrobates: I'm sure tsimpson will until a plan is made.
[10:30] <dendrobates> I'll be back in an hour to check on progress
[10:33] <AlanBell> popey: hard to assess the most urgent, there is paste prevention, meeting notes and the ban tracker/factoid stuff which will all be noticed at some point
[10:33] <AlanBell> along with the USDbottu stuff which will be noticed when there is a UDS
[10:33] <Unit193> I'll be going in 28 minutes, will need to know if I should pop ubot93 in.
[10:34] <AlanBell> it would all fit perfectly cheerfully on one of these https://www.hetzner.de/de/hosting/produkte_vserver/vx6
[10:35] <AlanBell> and I could in theory offer a VM on the side of one of my systems, but I am a little reluctant as it has already been a target
[10:35] <popey> you wouldn't put wordpress / sugarcrm / everything else php on it though?
[10:36] <AlanBell> nope
[10:36] <popey> so would be way less of a target
[10:36] <AlanBell> yeah, that is true
[10:38] <popey> Shall we co-draft a mail in etherpad to -irc?
[10:38] <AlanBell> good plan, but I have to pop out now
[10:38] <popey> ok
[10:39] <Unit193> FWIW, Drone` isn't on that server, only unopaste.
[10:40] <Mikaela> unopaste was under meetingology
[10:40] <DJones> Unit193: Is it worth putting ubot93 in, but leaving it muted, that way it won't be responding to factoids and can just be unmuted if ubottu does drop out
[10:41] <Unit193> DJones: Only issue with that is if someone kicks with a factoid, or uses !foo > nick
[10:41] <DJones> Then somebody can just unmute rather than having to wait for you to return to be able to put ubot93 in
[10:41] <DJones> True, I did wonder about that
[10:42] <Unit193> And fwiw, tsimp is an admin in that bot.
[10:42] <AlanBell> I think some people were using it for irssi/quassle core as well
[10:42] <Unit193> Yes, quasselcore.
[12:20] <dendrobates> I'm back.  have you finished moving data?
[12:23] <dendrobates> popey: what is the status?
[12:24] <popey> http://paste.ubuntu.com/11855670/ was what you missed
[12:24] <popey> I don't know if AlanBell Mikaela have grabbed all they can...
[12:25] <Mikaela> I have meetingology & logs and I think AlanBell got meetingology & ubottu, no idea about mysql etc.
[12:25] <dendrobates> do you want me to create a new vm?
[12:26] <dendrobates> someone can email me an ssh key and go to town
[12:27] <dendrobates> I can leave the old one up, until you are have everything setup
[12:28] <AlanBell> dendrobates: what is the cost situation here?
[12:28] <AlanBell> are you being charged for this?
[12:28] <dendrobates> yes
[12:29] <dendrobates> it's less than $100/mo
[12:29] <dendrobates> pennies an hour
[12:30] <dendrobates> but over years, it adds up, but a few months is ok
[12:30] <AlanBell> indeed, I think it would be good to pause for a bit and decide on a long term home
[12:30] <AlanBell> don't really want to move things twice
[12:31] <dendrobates> the other option is to ask rackspace for free hosting.  I think they will probably say yes
[12:31] <dendrobates> then the vms could be moved
[12:31] <dendrobates> or hp
[12:32] <AlanBell> ok, so clean rebuild, then we have something that can be a bit portable?
[12:34] <dendrobates> hopefully
[12:35] <dendrobates> anyway I need to work.  Do you guys know how to reach me?  I'm not in IRC much anymore
[12:36] <Mikaela> email address could be made visible to /ns info dendrobates with /ns set hidemail off
[12:36] <AlanBell> https://launchpad.net/~dendrobates
[12:36]  * AlanBell spots ways to reach dendrobates :)
[12:36] <Mikaela> (or /msg nickserv...)
[12:37] <dendrobates> my handle is the same on twitter and github and ubuntu.com
[12:38] <dendrobates> someone email me a public key and I'll get e new server up
[16:10] <Pici> ubot5: join #ubuntu-server
[16:15] <teward> bot fail?
[16:16] <Mikaela> Encyclopedia fail, I would say
[16:17] <Mikaela> it won't separate what commands are meant to it, what to bot and join doesn't provide feedback on successful joins
[21:42] <Unit193> Pici: tsimpso seems to respond faster via email nowdays, if you haven't emailed yet.
[21:56] <Pici> I've copied mostly everything to a new vps, I'll working on configuring it later
[22:02] <Unit193> Well, if you need any assist, I'm around.
[22:25] <Unit193> Uh oh.
[22:26] <teward> Unit193: Pici: I was about to say... :)
[22:26] <Unit193> ...Say that it is taco time?
[22:27] <Pici> just doing some testing ;)
[22:27] <Pici> Unit193: where is drone running out of? ubottu.com as well?
[22:28] <Unit193> Pici: unit193.net
[22:28] <Pici> good
[22:28] <Unit193> I offered to move it off my stuff, or whatever you want with it.
[22:29] <Pici> well it not being on ubottu.com means one less thing for me to worry about
[22:29] <Unit193> yep.
[22:29] <Unit193> kubot is there, no?
[22:29] <teward> Unit193: no, it's not taco tuesday.  I was gonna say it's peanut butter jelly time, but i'm trying not to be a smartass here xD
[22:42] <Unit193> Pici: Oh hey, if you do move it over there's some script that copies ubuntu.db (factoid db) to a location other bots can rsync from, this is vital to ubot93.