Home » Storage Spaces » Is storage expensive – part 4 – Setting up the Linux Syslog Infra

Gen2 Linux VMs, using CentOS7 are going to be part of setting up the syslog infra components of enterprise LogVault. Since the number of devices sending the syslog are high, the workload is split into multiple syslog server. I have not tried storage deduplication on Linux and hence a central windows server with NFS role is used to collected the logs from these syslog servers and apply dedup.

I typically use the Centos minimal installation ISO file as the size is small to download and copy around, as well as the installs leaves a very low foot print and bare minimal machine to add just what is requried. A small virtual disk of around 10G should be sufficient for OS deployment with a dedicated 1.5TB virtual disk for storing the syslog log files. Fixed disk, especially for log storage is highly recommended. I have Linux complaining on I/O timeout with dynamics disk even with noop as Disk scheduler (as per the Best Practices for running Linux on Hyper-V at https://technet.microsoft.com/en-us/library/dn720239.aspx ), when the syslog service starts pumping large data into logs files.

After the OS deployment and basic configuration like network, first thing to ensured that “yum” can reach internet and download the RPM packages and updates. If the network requires access to internet through proxy server, same need to be configured in /etc/yum.conf configuration file for yum.

proxy=http://myproxy.mydc.com:8080
proxy_username=myname
proxy_password=mypassword

Base installation of CentOS comes with rsyslog for syslog service. Syslog-ng is part of EPEL (Extra Packages for Enterprise Linux). Since I am going to use syslog-ng, adding EPEL support to OS and then pull the syslog-ng package from that

yum install epel-release
yum install syslog-ng tcpdump lsof

This will install the syslog-ng and other package dependency for syslog-ng. Having tcpdump ( network snooping tool ) helps to troubleshoot situation where logs are not seen on the server, to determine if syslog packets from devices are reaching to the server or not. “lsof” to check which process is listening on a particular port i.e. syslog-ng is listing on UDP port 514 or not.

 

2nd virtual disk of 1.5 TB would be usually /dev/sdb and can be verified with “fdisk -l /dev/sdb” command. Following command to add the disk to LVM(logical volume manager for linux) and create the logical volumes from it.


pvcreate /dev/sdb
vgcreate syslog /dev/sdb
lvcreate -L1.49T -nlogvolume syslog
mkfs.ext4 -L LogVolume01 -m 10 -M /LogShare /dev/syslog/logvolume
mkdir /LogShare
mount /dev/syslog/logvolume /LogShare

To ensure this volume is automatically mounted during OS boot, entries to fstab file

echo "/dev/syslog/logvolume /LogShare ext4 defaults 1 2" >> /etc/fstab

Security-Enhanced Linux (SELinux) wouldn’t allow syslog-ng to write files outside /var/log folder and hence security context need to be copied to /Logshare with the reference of /var/log

chcon --reference /var/log /LogShare

Accuracy of the time is crucial for log correlation and hence this server need to be synced with two or more NTP source. NTP sources can be configured into /etc/chrony.conf configuration file for chronyd demon, that maintain the accuracy of the system clock. Hyper-v time synchronisation should be disabled for this VM.

server ntpsrv1.mydc.com iburst

server ntpsrv2.mydc.com iburst

Iptables is the firewall in Linux and in firewalld service is the supervisor that maintain the iptables setting. Like configuring application access in windows firewall, this needs to be configured to allow syslog packets come in and seen by syslog-ng daemon. Like windows firewall, firewalld comes with zone configuration and unless not configured, default zone of the network connection would be “public”.

Copy the ssh configuration file for firewalld to /etc/firewalld/services for customer firewall rule.

cp /lib/firewalld/services/ssh.xml /etc/firewalld/services/syslog-ng.xml

Edit the customer firewall rule file /etc/firewalld/services/syslog-ng.xml to reflect syslog-ng configuration

< ?xml version="1.0" encoding="utf-8"?>

syslog-ng
Syslog-ng syslog service as part logvolt deployment.
Firewalled maintains zones and each zone have different sets of services opened for incoming access. Unless configured, default zone is public and same can be set with network configuration file.

To configure syslog, edit the /etc/syslog-ng/syslog-ng.conf configuration file.


# Remote logging

source s_remote {
tcp(ip(0.0.0.0) port(514));
udp(ip(0.0.0.0) port(514));
};

destination d_separatedbyhosts {
file(“/LogShare/firewall/$HOST/$R_YEAR-$R_MONTH/$R_DAY-messages.log” owner(“root”) group(“root”) perm(0644) dir_perm(0700) create_dirs(yes));
};

destination logcorn{udp(10.10.10.123 port(514));};

log { source(s_remote); destination(d_separatedbyhosts); };
log {source(s_remote); destination(logcorn);};

Making the service to listen on port 514, both UDP and TCP, for incoming syslog traffics. Storage the logs fine under the /Logshare folder, followed by a folder in the name of the host, followed by a folder of current year and month, and the log file name in the format of day-messages.log.

The same log is relayed back to one more host running a log correlation solution.

Now logs more than 7 days has to be moved to central logshare, a windows server NFS share with data deduplication enabled on the volume. A folder “syslog” on this volume was shared over NFS

New-NfsShare -Name syslog -Path S:\syslog -Authentication all -EnableAnonymousAccess $false -EnableUnmappedAccess $true

A new client group with syslog serves added as member was created to give NFS access

New-NfsClientgroup -ClientGroupName syslog -AddMember 10.10.10.40,10.10.10.41

A script syslog2nfs.sh inside /etc/cron.daily gets executed every day to check the file age and moves if required.

#!/bin/bash
 # Mount the NFS Share
 mount -t nfs logshare-1.mydc.com:/syslog1 /NFSroot/

# before proceeding check the mount was successful
 if [ $? -ne 0 ];then
 echo "Unable to mount the NFS share for log movement"
 exit 1
 fi

#Check the destination directory structure and create folder if required
 for LogFilesDir in `find /LogShare -type d`
 do
 destfiledir=`echo $LogFilesDir|sed "s/LogShare/NFSroot\/firewall/"`
 if [ ! -d "$destfiledir" ]; then
 mkdir -p "$destfiledir"
 fi
 done

# Now we can move the logfies
 for LogFiles in `find /LogShare -mtime +5 -type f`
 do
 destfile=`echo $LogFiles|sed "s/LogShare/NFSroot\/firewall/"`
 mv -f $LogFiles $destfile
 done

umount /NFSroot

This combination is giving over 80% data saving and expecting to be able to hold comfortably 180 days of logs, as per compliance requirement.

Leave a Reply

Your email address will not be published. Required fields are marked *

*
*