[00:20] i come home and my ubuntu server is hacked hard. vlc running at 99%. auth.log all removed [00:21] i can't apt-get remove/purge firefox, which was totally hijacked [00:21] what where those things doing installed on a server? [00:22] skrp: pull the plug. wipe the disks, reinstall, restore data from backup. you have know way to know what's been done. probably all kinds of rooted [00:22] *no way [00:23] any way i can snapshot and inspect what they did later? [00:24] shut it down first, then boot off a livecd if you want to look around or take an image of the disk [00:24] an alternate plan is to pull the plug, get new disks, reinstall, restore data -- and then do forensics on the hacked images to learn more about it. Keeping in mind that they might have left behind "gifts" (in both the english sense and german sense :) designed to sabotage inspection.. [00:25] I would be paranoid of restoring data [00:25] if restoring data == restoring website [00:25] that is likely how they got in to begin with [00:25] previously hacked, or just vaunerable [00:26] I'd suspect first brute-force sshd paswords, then cpanels/webpanel things, _then_ website content.. [00:26] I have seen systems hacked, and then never really abused for months or years [00:26] I always see website first [00:26] hmmm yeah [00:27] i'm going to have to wipe this one, and my pfsense box it is behind [00:27] ssh is too easy to protect against, and never install cpanel/webpanel things [00:27] patdk-lap :) [00:27] * patdk-lap notes another wordpress thing was fixed over the weekend :) [00:27] or was that last week [00:29] probably "last week" will never be wrong with wordpress... [00:55] When I try to upgrade from 10.04 to 12.04 some times I am getting "locale: /lib/libc.so.6: version `GLIBC_2.15' not found (required by locale)" [00:55] Any suggestions on this? === markthomas is now known as markthomas|away [05:53] hey guys when i set up my server, do I need to put a ".com" at the end? the reason i ask is because i need to set up postfix for sendmail, and can't seem to get it working, and was wondering if that might be why? [05:53] *i mean do i need a ".com" at the end of the hostname === ochoroch1 is now known as ochoroch === aarcane_ is now known as root________ [06:48] Good morning. === jerto1 is now known as jerto === kickinz1|afk is now known as kickinz1 [10:40] Hey guys, while trying to install apache on 12.04 I'm getting an error unable to correct problems, you have held broken packages. I've went through a lot of commands trying to solve it but none have been successful, anyone got any ideas? [10:40] I've made a fresh server and that didn't solve it either [10:41] You do know 12.04 is ancient, right? [10:42] Yep [10:42] But I was told to use 12.04, would 14.04 be a better choice? [10:43] Unless you have a requirement you really absolutely can't change, never prefer a version that is obsolete [10:43] (and even then, try to subvert the requirement) [10:43] I've got my DNS setup on 12.04 [10:43] would using 14.04 on an apache effect this [10:44] On different machines or VMs? No. [10:44] just vms [10:46] If you are setting up a new installation of Ubuntu, definitely use a current version, unless you have an extremely critical reason not to [10:50] does kernel linux-image-3.19.0-18 still under testing on 14.02 ? [10:50] how come none of my machines could boot properly under this kernel [10:57] what's preseed mean.. actually i want to make custom iso so that if new box is coming it would be install ubuntu automatically.. how to make it happen? [10:59] preseed is an installation template which "provision" your server like a stencil [11:00] fishcooker, during iso booting time, you could pointing it to a preseed config file, so you could provision millions of servers as same way [11:05] is the https://help.ubuntu.com/community/PXEInstallServer is the good point to start.. or there is another link should i follow lkthomas? [11:06] fishcooker, preseed is not very well documented, PXE is a good way to start, we are using PXE preseed as well, but I did dig into preseed codebase and search for options that I need. === hito_jp0 is now known as hito_jp [13:30] General question: What would cause the system to say "Out of Memory" when it's got 4.3GB of free memory buffered [13:30] and only when dovecot is trying to do things [13:30] fragmentation? [13:31] teward: kernel saying OOM, or a process? malloc can fail for a few reasons. [13:33] jrwren: syslog says hostname out of memory [18855] and there's no way to trace back [13:33] oh, i neglected to mention the age of the server because i'd get smacked [13:33] funny: it's EOL, and I just realized it by checking kernel version [13:33] >.> [13:33] someone give me a phaser, i need to vaporize some servers [13:34] EOl doesn't equate to out of memory though [13:34] no, but support in this channel does [13:34] it's an 8.04 server :P [13:35] teward: heh :D [13:35] even 10.04 is EOL... [13:35] shouldn't get out of memory anyway, though [13:35] but still - perhaps time to upgrade it to get some fixes? ;) [13:35] probably [13:35] dovecot was process 18855 ? [13:35] teward: Heh, we have one of those too... It has been up for replacement for years... [13:35] TJ-: couldn't tell, had to halt the system to unstick SSH [13:36] teward: that's a definite OOM [13:36] but the kernel log should show [13:36] RoyK: monitoring says it had more memory [13:36] anyway - upgrade time! [13:36] RoyK: kernel log is syslog? [13:36] or elsewhere [13:37] teward: don't remember log locations on hardy - either syslog or kern.log [13:38] well there's nothing in kern.log indicating a kernel level OOM [13:38] which makes sense since the VMware tools installed here responded and gracefully halted the system [13:38] * teward wonders if it's just Dovecot running out of its own memory [13:38] but then, the system hasn't been updated for 4+ years, so some bugs may have been fixed after that ;) [13:38] 'cause the OOM killer would've started killing things, no? [13:39] RoyK: oh, definitely, tell that to the ancient python scripts running here though [13:39] the OOM killer sometimes saves the day, but just sometimes [13:40] RoyK: this is what i don't get, vSphere shows that the memory consumption data relayed by VMware tools was under a gigabite, including the cache bits that LInux does [13:40] so i'm not sure at ALL why it failed and hung up [13:40] and I beleive the OOM killer has been improved a bit those seven years [13:40] anyway - if this is under vmware - why didn't you bring up the console? [13:42] 'cause my access creds don't give me a lot of access unfortunately [13:42] start/stop/restart and basic stats [13:43] my standard access creds anyways [13:43] * teward yawns [13:43] what we need is an overhaul here... [13:43] if it happens repeatedly, you could setup a network console... [13:45] i'm considering it [13:45] my guess though is that it's got a bug somewhere [13:45] but probably better to just setup a new VM with 14.04 and migrate the config [13:46] I wouldn't dare having something that old connected to a network [13:49] If it's PAE then even if the system has 4.3GB free, each process can only use a maximum of 2GB or 3GB depending on the kernel CONFIG_VMSPLIT_3G [13:52] migration of the config isn't the issue RoyK [13:52] migration of the custom inhouse software is [13:53] 14.04 host with a 8.04 chroot :) [13:53] lol [13:53] teward: don't you have the source? [13:54] it's all deprecated python [13:54] so no [13:54] *that* deprecated? [13:54] Python 1.x? [13:55] hihi === medz is now known as samsn [13:56] teward: internal app? [14:06] why do you think it's a OOM? [14:06] what is the exact log line? [14:07] cause it sounds like a dovecot log line, out of memory, that is normal, cause each dovecot thread has it's own limit set, normally around 256megs [14:07] and increasing it from defaults, is up to you, via dovecot config documentation [14:07] the larger the mailbox, the more memory it will need to index the messages [14:08] and then more if your using compression, or searching and other features [14:08] patdk-wk: his sshd hung too [14:08] ah, didn't see that in the text wall [14:09] it wasn't perhaps, went readonly filesystem? [14:09] ssh will stop responding if that happens, sometimes, depending on how the login is configured, due to logs and stuff unable to write [14:10] patdk-wk: never seen ssh hang because of that... [14:11] I have, can't remember exactly why [14:13] I thibk that was back when /var/run/ was not a tmpfs on /run/, if /var/ file-system went read-only, sshd had no write access to /var/run/sshd/ [14:15] I think the issue was, unable to fork [14:15] due to the system being overloaded due to webhits, or other things, just backing up due to not able to process [14:16] I've had the issue I described, many years ago [14:17] I'm sure I had that also, but I know I've had the issue on 12.04, and that uses /run on tmpfs [14:17] but I'm not saying my issue isn't specific to how I have things configured [14:17] * TJ- nods [14:25] smoser: FYI, I won't be able to attend today's meeting. Nothing particular to bring up anyway [14:27] k [14:36] jrgifford: yep, internal [14:36] patdk-lap: quite literally, in syslog: hostname out of memory [18855] [14:36] and a timestamp before it [14:37] RoyK: I figured out the SSHd hang [14:37] that was a VMware glitch [14:37] (somehow the vNIC got disconnected from the vSwitch, the restart reloaded the vNIC) [14:40] teward: strange - never seen that on vmware [14:40] seen it on hyper-v, though - far more there (that was h-v v1) [14:42] teward, seen that many times :( [14:42] it's always pisses me off [14:43] patdk-wk: which, that syslog line? [14:43] RoyK: there was a maintenance option changed by the other guy that caused a momentary blip [14:43] that one vNIC never reconnected [14:43] happened once before here [14:44] ok [14:44] that's why SSH hung - because right as it prompted for PW that change happened [14:44] so *kablooey& [14:44] teward: is the software really bound to run on python 2.5? I mean - 2.7 isn't that big a difference, is it? [14:45] no, the vnic [14:45] RoyK: i can't get approval to test [14:45] patdk-wk: ah [14:45] has happened about 10times for me, in the last two years [14:45] out of around 1400 vm's [14:45] patdk-wk: yeah things randomly asplode :/ [14:45] meh [14:45] the syslog line, is odd [14:45] no program name [14:45] it should atleast say, kernel: on it [14:46] patdk-wk: indeed, but is it really a kernel OOM error? It doesn't sound it since the resource graph showed only about 500MB used out of 4.5GB available [14:46] taking caches into account of course [14:46] that out of memory, can't be from the kernel [14:46] right [14:46] since it's missing the kernel: line [14:46] and the OOM is like a 50+ line report [14:46] teward: there are python 2.5 PPAs [14:46] the question is: was Dovecot triggering it [14:46] RoyK: i have a feeling it'll work on 14.04 [14:46] teward: you should give your boss a dork approval [14:46] if it was dovecot, it would say dovecot: :) [14:46] hmm [14:46] but i can't do anything without 5 other people's collaborative effort [14:47] since we have to migrate email data over (dovecot), postfix configurations, SMB share configs... [14:47] to insist on using software not update for over 3 years, you can't understand much [14:47] I'm guessing it is a syslog issue [14:47] RoyK: Believe me: If I had unlimited power i'd have burned this VM [14:47] but eh [14:47] i don't have time at the moment for the migration project [14:47] patdk-wk: possibly, given this is a Hardy machine [14:48] * teward shrugs [14:48] it's on the list of things I'd love to upgrade [14:48] teward, https://communities.vmware.com/message/2280260 [14:50] same type of log entry, but on esxi hmm [14:50] ooo so i can blame VMware this time [14:50] patdk-wk: that's the system that VM is on [14:50] I wonder if the VMware tools are at fault [14:50] i could remove them... [14:50] always blame VMWare teward [14:50] but we lose HA that way [14:50] jrgifford: lol [14:50] oh ya, your using older than hell tools :) [14:50] jrgifford: always blame $random [14:50] :P [14:51] patdk-wk: i could always try and update the tools but that'd go badly lol [14:51] kernel and what not [14:51] yeah, old tools will cause weird behavior, particularly if the host has upgraded throughout time. [14:52] (I'm assuming you've patched VMware at least once since Hardy was EOL...) [14:52] ya, I backport newer tools to my ubuntu boxes [14:52] jrgifford: i wouldn't know, i don't track the ESXi updates, that's the other guy's job :P [14:52] but I don't even bother porting back to 10.04, and am dropping 12.04 next month [15:11] Hi [15:13] Hi i need help setting up multiple static ip's on multiple Routers what channel would I got to get help for that [15:13] not here unless your routers are Ubuntu [15:13] Is there a routing channel [15:14] ##networking maybe, but if oyu don't know how to set static IPs on a router, you've got bigger problems [15:14] and you likely have to set it one by one on each router [15:14] I do but it can't set then up across multiple routers [15:15] I* [15:52] got a server that grinds to a halt when any operations of import are executed. recovery runs relatively fine, but sshing in takes over 3 minutes, disk operations are slow (apt-get dist-upgrade for instance), etc. RAM, Disks, Disk Array appear to check out, fsck returns 'clean' [15:52] lookin for additional debug steps [15:52] recovery mode flies === markthomas|away is now known as markthomas [16:26] teward: iotop. [16:32] how to clear apt-get autoremove package list becase that list has some know package need to be install which i do not want to uninstall [16:35] samba35: apt-mark manual [16:36] hmm ,there is around 100+ package [16:36] or for the entire list, some variation on apt-mark showauto|xargs apt-mark manual [16:37] ok Thanks [16:37] Though that will wipe out the usefuleness of autoremove really [16:37] You might as well just not use autoremove. [16:37] then [17:16] hey guys, I'm trying to install something as another user, znc --makeconf however it's trying to write to my zachary user's home directory [17:17] is sudo -u zncdev -s not enough to be another user? [17:18] whoami says I'm zncdev [17:20] [ !! ] Writing config [/home/zachary/.znc/configs/znc.conf]... [ Unable to open file ] [17:33] cluelessperson: as you can see, sudo doesn't change $HOME [17:34] TJ-, is there a way to do so? [17:35] the kernel is being held back for a minor upgrade 3.13.0.24 vs 3.13.0.59. Anybody else have an issue with this in 14.04.02 [17:35] ? [17:56] boot hangs when systemd starts in Vivid-server: http://paste.ubuntu.net/11949526/ [17:57] ^^ asked for help 2nd time [18:17] PryMar56: no luck so far, eh? guess it's time to file a bug [18:19] sarnold, I have other installs with system-219 and the same kernel version that are seamless when system starts (no delay) [18:19] systemd starts [18:19] but its debian 8 with experimental [19:14] I'm trying to learn mysql (SQL) and I'm completely new to it. I was hoping to set up a mysql server on virtualbox to play with it. Can anyone recommend a good guide on how to go about doing this? [19:17] mariano_: tada :) https://help.ubuntu.com/14.04/serverguide/databases.html [19:17] thank you. :) [19:17] mariano_: unless you need mysql for a specific reason I suggest looking at postgres instead, I like it more.. [19:20] I want to get into data management and analytics. I know stats, econometrics, ect, but I know nothing about data management. I'm just graduating so I was told that if I want to move up, I have to learn sql and hadoop. [19:20] My professor said sql is easier to grasp and that more businesses use it. [19:21] do i need to do any specific changes to libvirt dhcp server within the network file to have it access the internet with public ips? The assigning of ips work, but when attempting to ping something like google.com, etc., there is no internet access. [19:21] I have bridge working fine with static ips, but just wondering about dhcp because i do want to be able to have that option available in the future [19:22] mariano_: postgres is much closer to the enterprise class DBMS than mysql ... it emerged from what became MS SQL server [19:23] Thank you TJ. I'll start reading postgreSQL and then dive into hadoop. [19:25] really? I didn't know sybase was founded on ingres. [19:26] TJ-: thanks for the tip, I found this and it should be an interesting read: http://archive.computerhistory.org/resources/access/text/2013/05/102702565-05-01-acc.pdf [19:31] sorry, I was thinking at cross-purposes; I was referring to tpostgres and T/SQL - got buried in implementing some T/SQL functions in postgres recently, [19:42] anyone know where I can report a problem with the help.landscape.canonical.com documentation? [19:44] hi all [19:45] teward: many of the webpages have a "report a bug on this page" link in tiny text near the bottom [19:53] is midonet an OVS alternative? [20:18] jrwren: Whats "midonet" and whats "OVS"? [20:25] midonet is something new to me, so I cannot answer. OVS is Open vSwitch. [20:38] sarnold: it wasn't rendering that :/ [20:38] meh === markthomas is now known as markthomas|away [21:28] teward: did you find a place to report the bug? [21:59] Im setting up a mail server and when I send a mail using echo it bounces and looking at the log its being sent to local instead of smtp any thoughts? === Stuxweb is now known as JaguarDown [22:09] Hi all newbie here. Simple question. In order to administer my server's web pages via SFTP I've changed the permissions of /var/www directory to 775 recursively, changed owner and group to www-data, and added myself to the group www-data. (I read in a couple places this is the standard thing to do?) After a couple of kinks I am able to read/write/execute via SFTP... [22:09] ...However I decided that maybe that "Every one else/public" only needs to "read" html so I changed permissions to 774 and suddenly at the SSH terminal I can't even CD into /var/www due to "permission denied" [22:10] Even though my user account is in the sudo and www-data group [22:10] groups* [22:11] What gives? [22:11] JaguarDown: execute permission is required to traverse into a directory, ie. access anything inside of it. read permission means listing the contents (but not accessing them) [22:13] JaguarDown: not sure why that's hitting your user, though. did you exit your ssh session and log in again after adding your user to the group? if you run "groups" at the prompt does the output include www-data? [22:13] JaguarDown: just for the record, I really dislike the data being owned by www-data, since that's the usual user account that the webserver uses when running [22:13] yes, groups says I am in the www-data group [22:14] JaguarDown: .. which means when the webserver is hacked, it has write access to all the content, which makes it insanely easy for hacks to become persistent [22:14] Ah. I didn't really know what the standard acceptable convention was for ownership of it [22:14] Should I just change ownership to my one and only sudo account? [22:14] This is just a home server [22:15] Although it is serving web pages to the internet [22:15] I did not log out and back in again [22:15] sarnold: nah, realized that the doc i was looking at was 'retired' [22:16] sarnold: then realized that I can't get landscape-client to work with the custom CA that was set up [22:16] sarnold: THEN realized the easier route is to just spend $13 for a cert [22:16] JaguarDown: up to you, I'd do a different owner, but that's just my preference.. [22:16] teward: is that it these days? wow [22:16] Okay I hopped out and into the SSH terminal and it works now. [22:16] sarnold: through gandi, with Ubuntu Membership + E Rates and such, pretty cheapish for a single domain cert (landscape.domain.tld lol) [22:16] However I didn't understand what you just told me about the www-data user. I will probably change it to myself. [22:17] Or something other than www-data [22:17] teward: you can still get free certs from StartSSL [22:17] JaguarDown: for me, I'd be content to use my user account to own it; for shared-with-someone-else I'd prbably make a new 'www' account or something [22:18] Although revoking them will cost [22:18] sarnold: Okay that makes sense. I read somewhere someone used a "webmasters" group. [22:19] JaguarDown: that makes sense; it's long :) but makes sense [22:19] Cool well turns out my main problem was logging out and in. Thanks. [22:19] dasjoe: true on both cases, but the StartSSL CA certs are a little more ambiguousish [22:19] :/ [22:20] teward: I'm waiting for let's encrypt to become available, then I'll have to evaluate what's the best way for me to proceed [22:21] mmm [22:21] well my issue is that I just need it for landscape-server / landscape-client. What landscape-client does NOT want to do is accept the custom CA certificate === markthomas|away is now known as markthomas [22:36] This is probably not the best location to ask this but what do people generally do when editing html files via SFTP to be served by apache2? Do they simply edit the files via SFTP in their favorite editor and save or do they do all editing on their local machine and then upload via SFTP when they're done? [22:37] JaguarDown: rsync <3 :) [22:38] Slightly related: https://xkcd.com/378/ [22:38] dasjoe: hehhe :) [22:44] sarnold: cool! [22:45] dasjoe: favorited :) [22:51] JaguarDown: one note with rsync, it's quite picky about the last '/' in directory names. it'd be worth fiddling around with a few permutations of rsync commands on test data before you really start using it [22:52] JaguarDown: e.g. rsync -avz source/ destination/ vs rsync -avz source destination vs rsync -avz source destination/ vs rsync -avz source/ destination -- once you've seen it a few times you'll be fine :) [22:54] sarnold: Thanks! That sounds like a promising proposition, I will definitely be giving it a go. Sounds like it could greatly streamline things. [22:54] yes :) [22:56] While we are on the subject, as the default apache2 config stands, if I want to place an image on my page and store it on the webserver does that image have to be stored under the "/var/www/html" directory or can the website see it if it's under, say, /var/www/images [22:57] Sorry I am currently pretty ignorant to all of the webserver and web pages stuff. [22:57] I currently have an image in /var/www/html/images and it's working fine [22:57] I'm sure it's doable, but I'm not sure what you'd need to do to make it work [22:58] The thing is the way i have it it causes an index to be available and I would prefer not to do that. [22:58] By index I mean you can visit the folder via the website address [22:59] you can turn that off with some other directives.. [22:59] Im setting up a mail server using postfixand when i do echo "test from mail server to user@example.com" | mail -s "test" user2@example2.com the recipient is not getting the mail. im not getting any errors in var/log/mail.err and my mail.log is showing its going through postfix/pickup /postfix/cleanup postfix/qmgr postfix/local then bouncing because its not a user any idea why its not [22:59] going to postfix/smtp but going to postfix/local [23:00] JaguarDown: http://httpd.apache.org/docs/2.4/mod/mod_autoindex.html for ubuntu 14.04 LTS and newer [23:00] sarnold: thanks [23:01] abcs: is your postfix configured to accept mail for example2.com? [23:02] sarnold: no I don't want postfix to accept that mail I want to send it to someone elses mail account [23:03] * JaguarDown is now away - Reason : [23:05] Im trying to test if the mail server is set up correctly to send mail out to a relay host if theres a different way to test that i can do that [23:06] abcs: i'd next try doing things by hand: host -t mx example2.com, nc smtp ... EHLO sarnold MAIL FROM: RCPT TO: .... [23:12] host: couldnt get address for 'nc': not found returned [23:13] two options.. either your recipient has screwed up their dns records, or your dns setup is broken [23:15] http://mxtoolbox.com is a good site for testing mail servers outside your network [23:15] the dns is hosted at godaddy i wouldnt think that would be the case i can send a recieve email from windows desktop not routed through the mail server [23:15] I thuoght you were trying to debug sending? [23:15] i am [23:15] from post fix which i am currenly setting up [23:16] at the moment the email is routed through outlook that goes directly to godaddy [23:16] I need a mail server to relay it first [23:16] cryptodan_laptop: nice! [23:17] I can send email back and forth on localhost but when I try to send it somewhere where the destination is not local it bounces [23:18] to verify your dns has propagated check here https://www.whatsmydns.net/ [23:20] the domain is propageted, its been in use for 10 years [23:21] those two sites have helped me solve many issues with my own dns