| === Monthrect is now known as Piper-Off | ||
| neonixcoder | hi team, this is general question not specific to Ubuntu but this is happening on Ubunut. So I am asking it here.. | 03:26 |
|---|---|---|
| neonixcoder | We have a script which monitors network connections, if there is no network connection(3G connection) it just tries to enable networking. If this is not working out, this script will reboot the system. And these reboots happens, untill we get network connectivity back. | 03:28 |
| neonixcoder | The issue is after 5 bad pings, this script is not writing anything to log file. We tried to replicate this in lab, and everything working fine.. | 03:28 |
| neonixcoder | bit strange and I am not able to conclude what is happening with the script. | 03:29 |
| neonixcoder | any suggestions? | 03:29 |
| patdk-lap | based on that info? none at all | 03:29 |
| patdk-lap | it did not provide any useful info | 03:29 |
| neonixcoder | the file system is rw mode, so read-only mode is eliminated.. | 03:29 |
| sarnold | pastebin the script? | 03:30 |
| sarnold | pastebin how you're running the script? | 03:30 |
| neonixcoder | How are you Arnold? | 03:30 |
| neonixcoder | Give me 2 mins..! | 03:30 |
| patdk-lap | what the filesystem is and what is the reboot? | 03:30 |
| sarnold | not bad, surprised how quickly the weekend went, but not bad :) you? | 03:30 |
| * patdk-lap is just upset he debugged for 2hours why a patch wasn't working | 03:31 | |
| patdk-lap | to come to find, the patch was not installed | 03:31 |
| patdk-lap | and to find, myself from 2years ago, was smarter than I am today :) | 03:32 |
| neonixcoder | http://pastebin.com/6ymadTvb | 03:32 |
| neonixcoder | I am good Arnold, thanks for asking.. | 03:32 |
| sarnold | patdk-lap: oh yeah, been there. It's especially fun to find my own answers on stack overflow. | 03:33 |
| patdk-lap | add some syncs in there | 03:33 |
| patdk-lap | not sure what kick_watchdog.sh does | 03:34 |
| patdk-lap | but it doesn't sound filesystem friendly | 03:34 |
| patdk-lap | sounds more like, yank power | 03:34 |
| sarnold | poor dog | 03:34 |
| neonixcoder | patdk-lap: It is just a watchdog script which is poweroff the machine completely and boots after 5 seconds. | 03:36 |
| patdk-lap | yes, but exactly how? | 03:36 |
| neonixcoder | Up to 104 line it is fine, but 105 line on words I dont see any logs.. | 03:36 |
| patdk-lap | it matters if the kernel has time to write everything to disk or not | 03:36 |
| patdk-lap | watchdogs normally don't | 03:36 |
| neonixcoder | patdk-lap: We have a c code in that kick_watchdog.sh script which take care of kicking power off. | 03:37 |
| patdk-lap | so it's not friendly | 03:37 |
| patdk-lap | at the top of kick_watchdog.sh add sync command | 03:38 |
| patdk-lap | should help | 03:38 |
| neonixcoder | patdk-lap: what you mean by not friendly? | 03:38 |
| sarnold | it doesn't give the filesystem a chance to sync, umount, etc | 03:38 |
| neonixcoder | patdk-lap: We can not use sync as we deliberately delayed writting to disk by 30mins in kernel.. but when we do kick_Watchdog.sh run, it have sync included. | 03:39 |
| neonixcoder | sarnlod: Can you explain a bit? | 03:39 |
| sarnold | neonixcoder: the kernel will buffer writes until some convenient time in the future | 03:40 |
| sarnold | writes to files don't happen immediately, they happen at only certain points | 03:40 |
| neonixcoder | yes.. | 03:41 |
| patdk-lap | even worse if mapper stack is used | 03:41 |
| sarnold | so the usual thing to do is to run sync to make sure that those changes have been flushed to disk, and if you can umount the filesystem first, that's even better | 03:41 |
| neonixcoder | sarnold: my script struck up there like 2 to 3 days with out writing anything to log file. just a note, we sync data to disk every 30mins. | 03:41 |
| sarnold | btw, all these 'sudo' in here seem funny | 03:42 |
| sarnold | does this run as root? or not? | 03:42 |
| neonixcoder | :) | 03:42 |
| neonixcoder | I know it sarnlod, I am going to modify it. | 03:42 |
| sarnold | if it doesn't run as root, is sudo configured to allow it to run those commands without prompting and so no? | 03:42 |
| neonixcoder | This is written while back, if I past you actual script, you will laugh at me ;) | 03:43 |
| sarnold | note that the sudo foo >> bar may not work as you want. everyone gets tripped up with that.. | 03:43 |
| neonixcoder | sarnold: Yup, I am going to remove sudo from it. | 03:45 |
| neonixcoder | sarnold: will that helps? | 03:46 |
| sarnold | neonixcoder: unlikely | 03:48 |
| neonixcoder | yes.. this script was running for a while with out any issue. | 03:48 |
| neonixcoder | We are seeing this issue recently on couple of servers.. | 03:49 |
| neonixcoder | Our servers are connected to network through 3G, if there is no 3G network, our script tries to disable/enable for some time. If its unable to bring network, it will reboot machine. | 03:49 |
| neonixcoder | If there is no network for 10 mins, the system will be rebooted. At the time of reboot, system disks are synced. so there is no point why my script is unable to write to disk. | 03:51 |
| neonixcoder | Any suggetions? | 03:51 |
| sarnold | are you sure the machines arent hung entirely? | 03:52 |
| neonixcoder | They dont hung.. | 03:53 |
| neonixcoder | Y'day we sent a technician to see if can login. And he is able to do that. I ran some simple tests like checking network connectivity(Which is not there at the time of testing), weather my script is running or not(ps -ef | grep scriptname shows that script is running), I tried to create empty files to see file system is not mounted as readonly and I am able to create empty file as well. | 03:55 |
| neonixcoder | And the funny part is, I am unable to replicate in our lab. | 03:56 |
| neonixcoder | Our script is doing what it is meant for. | 03:56 |
| sarnold | you've got a shell ona system that's showing the issue? that's awesome; you can attach strace to it and find out what it's doing :) | 03:56 |
| sarnold | strace -f -o /tmp/strace.log -p pid | 03:57 |
| patdk-lap | you sure the pci_modem status program isn't hanging? | 03:57 |
| patdk-lap | due the the 3g issue | 03:57 |
| neonixcoder | great, sarnold. I forgot to do this. | 03:58 |
| neonixcoder | I will do this when we see this issue again. | 03:58 |
| neonixcoder | and see if I can find, what our script is doing. | 03:59 |
| sarnold | good point, if any of the called programs hang, it might be hard to tell it apart.. | 04:00 |
| patdk-lap | well, we where assuming it rebooted | 04:01 |
| patdk-lap | but if it isn't :) | 04:01 |
| sarnold | hehe | 04:01 |
| neonixcoder | patdk-lap: I too have same doubt about pci_modem status program. I will try to check next time when it happens.. | 04:02 |
| patdk-lap | add a timeout timer to it | 04:05 |
| patdk-lap | in shell script, that could be annoying though | 04:05 |
| sarnold | a watchdog for the watchdog script :) | 04:05 |
| * sarnold runms | 04:05 | |
| patdk-lap | well, the watchdog script doesn't have a proper gun | 04:06 |
| neonixcoder | patdk-lap: The status command worked fine till 4 bad pings.. have a look at my log file at http://pastebin.com/eztiyvub | 04:08 |
| neonixcoder | patdk-lap: From the 5th bad ping, it stopeed writing to log file, I see other programs are able write to same log file successfully. | 04:09 |
| === cz2 is now known as networkmanager | ||
| === networkmanager is now known as NetworkManager | ||
| === Lcawte|Away is now known as Lcawte | ||
| === Lcawte is now known as Lcawte|Away | ||
| === NetworkManager is now known as cz2 | ||
| SyncopatedFool | Having trouble setting up postfix on Ubuntu server, I have POP3 working properly, but SMTP without authentication won't allow me to send outgoing mail, and I am having troubles setting up authentication. | 12:25 |
| SyncopatedFool | mainly I have the configuration set up fine, but I have no clue how to setup the auth key to make it work | 12:26 |
| SyncopatedFool | Having trouble setting up postfix on Ubuntu server, I have POP3 working properly, but SMTP without authentication won't allow me to send outgoing mail, and I am having troubles setting up authentication. | 12:47 |
| SyncopatedFool | mainly I have the configuration set up fine, but I have no clue how to setup the auth key to make it work | 12:47 |
| SyncopatedFool | been using this guide to set up smtp auth, but failing at proving authentication http://postfix.state-of-mind.de/patrick.koetter/smtpauth/smtp_auth_mailclients.html | 12:49 |
| === Lcawte|Away is now known as Lcawte | ||
| eahmedshendy | Is that right that Ubuntu 14.04 the last version with Upstart ? I'm studying Upstart to add some scripts at the boot time, but someone told me it is deprecated with systemd? | 14:23 |
| TJ- | eahmedshendy: it's optional on 15.04, and not used on 15.10 | 14:30 |
| eahmedshendy | TJ-: Ok, now we are working on Ubuntu 14.04, and I am studying Upstart, Am I wrong? | 14:53 |
| eahmedshendy | I have this error:http://paste.ubuntu.com/13492759/ | 14:53 |
| eahmedshendy | Hi, anybody here? | 15:12 |
| teward | rbasak: ping | 15:32 |
| jgrimm | teward, fyi rbasak is out on holiday all this week | 15:33 |
| teward | ok | 15:34 |
| teward | then nevermind :) | 15:34 |
| * teward yawns | 15:34 | |
| teward | (it's only the nginx merge that's blocked because testing is in failure state due to the sbuild schroots being busted by a bad bug in ubuntu) | 15:34 |
| jgrimm | cpaelzer, feel free to add yourself to the irc meeting rotation -> https://wiki.ubuntu.com/ServerTeam/Meeting | 16:23 |
| === Lcawte is now known as Lcawte|Away | ||
| cpaelzer | jgrimm: done | 16:33 |
| jgrimm | thanks! | 16:35 |
| === Lcawte|Away is now known as Lcawte | ||
| Sazpaimon | when building an ubuntu server AMI, is there a known list of directories/files to clear/empty out in order to have a clean image? | 18:34 |
| Sazpaimon | I know stuff like things in /var/log, and maybe .bash_history, but I can't think of anything else | 18:34 |
| Sazpaimon | I also dont know if completely deleting the contents of /var/log is safe | 18:35 |
| sarnold | probably /etc/ssh stuff too, to ensure you generate a unique host key on reboot .. | 18:35 |
| helo | where does openssh-server log to? | 18:35 |
| helo | i see it is set to use AUTH syslog, but i see nothing in syslog or auth.log | 18:35 |
| sarnold | /var/log/auth.log on my 14.04 lts system | 18:36 |
| patdk-wk | heh, /var/cache | 18:41 |
| jelly | where do VMs on amazon get their entropy from? Re: > generate a unique host key on reboot .. | 19:49 |
| sarnold | jelly: in theory they could use somethinglike virtio-rng http://wiki.qemu.org/Features-Done/VirtIORNG | 19:51 |
| sarnold | jelly: another option is touse an entropy-as-a-service tool such as pollinate to grab some entropy from a service such as entropy.ubuntu.com | 19:52 |
| sarnold | jelly: another option is to use something in cloud-init to provide a seed, but I'm not sure if that's pre-made and ready to go or not... | 19:53 |
| jelly | thanks, I was not aware of half that stuff | 19:56 |
| Logos01 | Howdy, folks -- I'm trying to set up an stunnel connection between an Ubuntu 14.04 machine and a CentOS 7 machine -- I'm invoking stunnel4 as the client on the Ubuntu machine. The problem is that even though I've hard-coded the config file to use TLSv1, it's still giving me SSLv3 errors | 21:15 |
| Logos01 | As though it's not using TLSv1 at all. | 21:15 |
| Logos01 | Anyone run into this before? | 21:15 |
| Logos01 | I keep getting "SSL3_GET_RECORD:wrong version number" in /var/log/syslog | 21:17 |
| sarnold | that alone wouldn't worry me too much, there's enough similarity between sslv3 and tls1 that there's function re-use, keeping same names etc.. | 21:20 |
| sarnold | try confirming both endpoints with openssl s_client or .. wireshark? ettercap? to try to figure out which specific protocol is being used | 21:20 |
| Logos01 | Well, I've tried hard-coding tlsv1 and/or sslv3 on both ends and I keep getting that error. | 21:23 |
| Logos01 | I keep seeing disconnects as well. | 21:23 |
| denbeiren | hi all,.. i was thinking if there is a distro or tool that specifies on "booting a buggy computer and backing up all of the standard data to a networklocation" type of thing | 21:30 |
| Logos01 | Hrm... openssl s_client -connect ubuntu.machine:port -tls1 -CAfile /etc/stunnel/ca-chain.pem returns "no peer certificate available" | 21:32 |
| Logos01 | What the devil did I do ... this was working a minute ago. :( | 21:32 |
| === Lcawte is now known as Lcawte|Away | ||
| === Lcawte|Away is now known as Lcawte | ||
| sarnold | denbeiren: knoppix is popular for that | 23:03 |
| sarnold | denbeiren: ubuntu livecd, if that still works.. | 23:03 |
| * teward pokes sarnold | 23:08 | |
| sarnold | afternoon teward :) | 23:08 |
| teward | sarnold: got a few moments for a PM? | 23:08 |
| sarnold | sure | 23:09 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!