JDPFu.com 2024

My Methods and Goals

I use Ansible as a DevOps tool. That decision took a few years to solidify after trying puppet, Rexify, SaltStack and Chef. Ansible is easy to start and can grow into any necessary level of complexity needed. The greatest positive for Ansible is that it is based on ssh. That is also the negative. In just 15-30 minutes you can be managing your systems with Ansible too. If I were Facebook or of that scale, I might choose something else, but for 1000 servers or less, Ansible is easily capable out of the box.
This is NOT an article about Ansible. Previously, I used a simple bash script to repeat the initial server setups and prior to that, a text file that I’d copy/paste commands from into every new server. I like that Ansible will let me break things up into tasks and include what is relevant for a single server, group(s) of servers or all servers. Some configuration changes apply to server and desktops, others only apply to a server OR a desktop and others only apply to a specific type of machine (MariaDB, Redmind, wikimedia, etc.) and nothing else.
Regardless of where you are on the automation scale, just creating something standard is important. Don’t feel bad or good just because you copy/paste or have things scripted. It is about the results, after all – a more secure server and a properly configured base server are the goals. Consistency matters, where it makes sense.

My network has a few things running on it that I want every other server to leverage. Leveraging those services is important. Things like:

• apt-cacher-ng (package cache to avoid excess internet traffic that isn’t needed)
• NTP server (time)
• munin server (monitoring)
• email server
• backup server

Starting from a fresh Ubuntu Server installation with a deploy account (or my personal account) and ssh-server running, but nothing else. IPs are still DHCP provided.

First 5 Minutes

My account already exists on the new server, since I installed it and it is using DHCP for IPs. I do base installs manually. Bringing up a new server happens once every few months, so that isn’t a big deal and takes about 15 minutes without really watching. Automating initial server installs might be a next step if I were doing 1 a week.

Overview of Steps

Manually determine the IP for the server. This can be from the hosting provider or a console login or arp. My servers come up with DHCP addresses, so I need to ssh into them on the DHCP portion of the server subnet. Then a few manual steps are needed.

1. ssh keys (ssh-copy-id) from the workstation running Ansible (not really a server )
2. purge nano – easier than setting up vi/vim as my default, plus saves 2Kb of storage. ;) I extremely dislike nano.
3. fix the sudoers for my account (or deploy accnt)

Then I run an ansible-playbook to install, configure the next things:

1. static IPs or bridge configurations
2. personal settings, bash functions, aliases, scripts (backups, kernel-cleanup, etc)
3. setup apt to point to apt-cacher-ng leveraging an internal package cache server
4. aptitude full-upgrade (gets the latest packages AND the latest kernel) (apt is fine too)
5. aptitude install my base packages (about 25 things); more later
6. sshd_config settings (no root, force v2 only connections, disallow passwords from outside the LAN)
7. Logwatch settings (protect ssh, email address)
8. /etc/hosts file for the LAN; for small networks, this is easier than internal DNS
9. munin-node config for monitoring
10. satellite email server setup; all systems can send email.
11. email aliases so system mail is forwarded somewhere
12. Performance monitoring (still missing real alarming)
13. NTP client settings ; accurate time is very important for security AND for troubleshooting later
14. Setup networking; interfaces for static IP, jumbo frames, dns-servers, etc.
15. Restart any services due to changed config files – this will change the IP and ssh connectivity will be lost.

Now we have a static IP and can ssh in without a password.

Base Packages?

By installing certain base packages, commonly needed tools are available. The base list:

• acpid – so a client VM can receive shutdown requests from the host VM server
• openssh-server – never forget this. Might be running on a console. Should already be installed during the server installation.
• fail2ban – dynamically block any failed ssh attacks. I can’t think of any reason NOT to have this on every box. The defaults are sufficient for LAN ssh protection on port 22. If you move sshd to a different port, fail2ban config needs updates too.
• iptables-persistent – no need to manually save/restore iptable rules
• postfix – system emails need an MTA to get forwarded to a real email address; only sends email, does NOT receive from outside the machine/VM.
• aptitude – aptitude is smarter about package dependencies than apt-get. With 16.04, I started using apt as a replacement. Until I see how smart apt is for non-trivial dependencies, I want aptitude available.
• sysstat – performance capture tool library.
• munin-node – performance reporting/monitoring; setup BEFORE it is needed! Capturing this data for a few weeks before there are any issues makes trouble shooting issues much easier.
• ethtool – want jumbo frames and expanded network settings control?
• rsync – duh. MANDATORY
• rdiff-backup – my backup tool of choice Best How-To – rdiff-backup is least bad of all the choices I’ve found. Very much like rsync, but with versioning based on diffs, not hard-links. 30 days of versioned backups are less than 1.20x the original storage.
• mlocate – locate is instantaneous and replaces find for many needs (package name may have changed to locate )
• logwatch – daily reports of server activity (emailed, so MTA needed)
• lshw – create daily system hardware configs, compare today to yesterday and highlight any changes
• ntp – Keeping accurate time is a security consideration. Also, when troubleshooting complex client/server systems, having all the timestamps on logfiles match to the 0.001 sec makes finding root causes much easier. Once had to troubleshoot 50 systems, each with different timestamps. What a pain.
• tree – nice directory tree visualization – find . -type d replacement
• heirloom-mailx – provides a CLI mail program, handy for testing the send part of email.
• software-properties-common – PPA helper
• Ubuntu 16.04 has a few more dependencies – mainly for python APT support which ansible needs.
• Personal scripts for capturing system information, cleaning up old kernels and making backups. On desktops, many more scripts will be copied to my ~/bin/ from another desktop. This is manual. I’ve decided to keep 1 system of record for my ~/bin, so any new scripts I create go posted to that machine for later use – even if the script will not work on that box due to dependencies. For example, wrote a script to convert Spanish captions into English captions. There is a specific tool required for the translations which is only installed and configured on the Plex Server machine.

Too much bloat?
What is missing?

Missing and ToDo Steps

• Backups!!! Backups are the primary security tool for any computer system. My backups are special for each snowflake server, but I start with the same basic script. Currently pushing backups to a central server, but need to pull those backups to better manage backup windows and I/O contention on the backup disks.
• Firewall settings – default deny, except ssh managed by fail2ban; then open specific ports as needed; also need to block all outbound SMTP that aren’t to the primary email server
• Email front-end server firewall blocks for known spamming subnets that I just never want to allow, ever, again.
• Monitoring – Monit or munin is easy to config on nodes. The server should be easy too, just haven’t gotten there.
• Alarming – nagios or opennms are standard tools for this.
• Application settings, working AND secure configs; these are generally part of the server Ansible playbook which gets created just AFTER all these other things are handled. There is a flaw in my method – if I have great backups, then why do I need the Ansible playbook to rebuild a specific server? Well, I don’t. If I had 2-500 of the same server types, then ansible to rebuild them would make more sense. All my systems are snowflakes, so daily backups are still needed, with restores being the primary recovery process.

Category5.tv has a 3 part video on using LVM with rdiff-backup – eps 455, 461 and 465. youtube has the videos.

Automatic security updates ARE NOT performed. I’ve been burned by updates, so I want to control exactly when this happens, know what is changed (logwatch) and have a backup that is solid to be able to restore, if necessary. The scripted patching process does create detailed logs, so if something bad happens that is seen quickly, it is easy to find the culprit.

I manually run a script with apt-get that does an update and full-upgrade every Saturday morning across almost every Linux machine on the network. A log file is created. Changed from aptitude for scripted runs like this.

No DNS server – The 2nd time my small network was hacked was through BIND. I don’t run it anymore, leaving that for paid professionals.

DHCP Reservations – these are manually added for all portable devices. Nice to have them work correctly when off this network, but also have static IPs when on the LAN. Servers have static IPs configured.

Ansible

Ansible is one of those tools like sudo and ssh and rsync that is very easy to get started, but has huge depths in capabilities if we dig a little deeper.

• Simple Ansible Tutorial. Ansible is one of the few tools that I’m willing to install from source. It only gets installed on 1 machine, a workstation. The Ubuntu package version are missing some very important fixes and Ansible is under very active development, improved all the time. I use git clone to get it, then read the README.md for instructions. It is only needed on 1 machine, not any of the servers to be managed. Find the code on github.
• Another Ansible Introduction.
• More complete Ansible info simple to real-world. Some interesting techniques.
• For visual learners, like me, a good 20 min video can make things “click” and reduce the time to understanding.

More Security

This initial setup is only good for a LAN server/desktop, not enough security for an internet-facing box.There are 30-100 more things to harden it. Bob Toxen literally Wrote The Book on Linux security. Find it at bookstores and Amazon, if you’d like to know more.

Making a Linux internet server secure is a constantly moving target. Attacks change daily and we need to be prepared as administrators. If you can’t spend 60 minutes a week maintaining a running server, best to outsource that task. Maintaining an internet server is fun for a month, but after that, it is work. Internal LAN servers are much easier provided they aren’t available over the internet.

25 Linux Hardening Tips

There is the Debian Security Manual and the Debian Hardening Guide. RHEL/CentOS and those flavors of Linux are hardened in very different ways, thanks to SELinux facilities. Of course, we should never forget the SANS Security Checklist or the NSA Security Guides or the OWASP Guides.

For desktop security, the Ubuntu Basic Security page is good to start, but nothing can help with desktop security more than an educated user and the less-friendly Ubuntu Security Pages. We all have some reading to do.

Of course, there are articles here on home network and desktop security too:

• Secure Browser Settings
• Security for Desktops
• Security with the Hosts File | republished by Lifehacker
• WiFi Router Security Checklist
• Linux System Maintenance | republished by Lifehacker
• Plus lots of Backup articles
  Backups are the #1 security technique.

Methods by Other People

• Bryan Kennedy’s version.
• Fred’s version.
• Securing a Server with Ansible – I like this coverage very much! Nicely done.

I get the feeling these guys are not system administrators, rather they are programmers doing DevOps jobs. I honestly do not know. Both have repeatable, standard, processes which is the most important thing for all of us.

Other Pros Using Ansible

• DebOps
• Sovereign

Better Ways?

• What do you do?
• What did I get completely wrong? Sometimes I’m an idiot. ;)

Let me know in the comments.