Tip Your Server, Please

Ubuntu Server Install and 1st 5 Minutes

JD P

http://blog.jdpfu.com - jdp [at] algoloma

Tonight

• Upcoming Events / Meetings
• Looking for Help / Volunteers
• Setting up Linux Servers

Upcoming Events / Meetings

Meetings 2nd Thursday of the Month
Sign up for ALE's email list to stay informed - http://ale.org/

• 9/20 - InstallFest - Saturday 10am-5pm
• 10/9 - Linux Security - Gary Miller
• 11/13 - Linux Virtualization
• 12/11 - Solstice / GPG Keysigning

Other Linux Groups in Metro ATL

Volunteers Needed

• Need volunteers - 1 hr a month
• Interesting topics
• Show and Tell
• Shared experiences
• Great crowd for new speakers
• Mini-How-Tos - 5 minutes per topic

About Me

• Virtualization user / designer
• 20+ Years ProEx
• Enterprise Tech Arch / SysAdmin
• Real-time Space Shuttle GN&C Software

  Real Rocket Scientist

• Cross-Platform C/C++ Developer (12+)
• Perl WebApp Developer
• World Traveler (5 continents, 24 countries)

Linux Server Overview

• No GUI / ssh
• CLI Only / ssh
• Best Practices dictates single type of service per OS install
  • Web
  • Proxy
  • email
  • DNS
  • Gateway / VPN

These are all different services - should be on different installations.
Not just a UNIX best practice - Windows too.

All-in-One Servers

Against Security Best Practices

• For playing at home, HowToForge.org has step-by-step how-tos
• Perfect Server - Falco makes these for almost every new distro
• Don't assume these instructions are secure.

Trade-Offs

• Don't confuse High-Availability, Disaster Recovery and Backups
• Simplicity
• Complexity
• Availability
• Redundancy
• Maintainability
• Performance
• Easier Troubleshooting

Confusion

Storage Considerations

• Excellent Backups are mandatory
• Enterprise disks, fast disks, slow (Green) disks, SSDs
• RAID - HA-only
• Clustered (GlusterFS, etc)
• LVM or not? (btrfs/zfs)
• File systems (ext4/3/2, xfs, jfs, btrfs, ZFS, ReiserFS, etc.)
• Performance / USB has Queuing issues

Network Considerations

• Bandwidth Requirements - WAN/LAub
• Multiple physical connections
• Alternate subnets (backup, SAN, management, redundant)
• Failover NIC configuration
• Switch ports aren't cheap
• Enterprise it is common to see 6 ports per server used

CPU Considerations

• Most physical servers run at 13-20% utilization
• Virtualization is the defacto enterprise standard today
• vCPUs - minimal for the workload - 1 usually, perhaps 2
• vCPUs - do not over subscribe if performance is important

RAM Considerations

• Matched for the workload to avoid swapping
• Too much RAM can slow down systems ( > 32G)
• 512MB to 2G is fairly common for VMs
• Geospatial DBs need 16G+
• Avoid swapping ; servers are NOT desktops

Installation for Tonight

• VM - 1 vCPU
• No RAID - virtio
• 8G of storage - preallocated (4G can work)
• 1 NIC - virtio ( will use a bridge)
• Defaults for most things
• ssh server only - nothing else - scripts for everything else
• 15 min to install - a little longer tonight

Do Install

1st Five Minutes on Server

• What I do in the first 5 minutes
• http://blog.jdpfu.com/2014/02/28/1st-five-minutes-on-a-server
• Enterprises use automation tools to build a server
• Avoid Point-n-Click - highly inefficient and error prone

Methods / System / Documentation

• Have a system, follow it:
  • Paper
  • Script / text files
  • DevOps tool - like Ansible, Chef, CfEngine, Rexify, SaltStack, Puppet
• Constantly improve the system
• Consistency is critical - know the settings

My Script

Manually

• ssh keys
• purge nano / setup vim
• fix sudoers

Static IP / Network Settings

• Servers need a Static IP to be useful
• Blacklist IPv6 if not used

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address 172.22.22.11
  gateway 172.22.22.1
  netmask 255.255.255.0
  dns-nameservers 172.22.22.1
  dns-search jdpfu.com jdp.foo

My Script - Automatic

• Ansible-playbook that does this stuff

open the file ~/.ansible/.../ubusrv.yml

Playbook - 1

• static IPs or bridge configurations
• personal settings, bash functions, aliases, scripts (backups, kernel-cleanup, etc)
• setup apt to point to apt-cacher-ng leveraging an internal package cache server
• aptitude full-upgrade (gets the latest packages AND the latest kernel)
• aptitude install my base packages (about 25 things)

Playbook - 2

• tcp-wrappers - limit connections to LAN
• sshd_config settings (no root, force v2 only connections)
• Logwatch settings (protect ssh, email address)
• /etc/hosts file for the LAN
• satellite email server setup
• Performance monitoring (still missing real alarming)

Playbook - 3

• email aliases so system mail is forwarded somewhere useful
• NTP client settings (or ntp server)
• Setup interfaces for static IP, jumbo frames, dns-servers, etc.
• Restart any services due to changed config files – this will change the IP and ssh connectivity will be lost.

Base Packages?

* acpid – so a client VM can receive shutdown requests from the host VM server
* openssh-server – never forget this. Might be running on a console.
* fail2ban – dynamically block any failed ssh attacks. I can’t think of any reason NOT to have this on every box.
* iptables-persistent – no need to manually save/restore iptable rules
* postfix – system emails need an MTA to get forwarded to a real email address; only sends email, does NOT receive from outside the machine/VM
* aptitude – aptitude is smarter about package dependencies than apt-get.
* sysstat – performance capture tools
* ethtool – want jumbo frames and expanded network settings control?
* rsync – duh. MANDATORY
* rdiff-backup – backup tool of choice
* mlocate – locate is instantaneous.
* logwatch – daily reports of server activity (emailed, so MTA needed)
* lshw – create daily system hardware configs, compare today to yesterday and highlight any changes
* ntp – Keeping accurate time is a security consideration. Also, when troubleshooting complex client/server systems, having all the timestamps on logfiles match to the 0.001 sec makes finding root causes much easier
* tree – nice directory tree visualization – find . -type d replacement
* heirloom-mailx – provides a CLI mail program
* software-properties-common – PPA helper

Backups

• Check the backup script for the system
• Create Backup userid, setup ssh keys (if necessary)
• Run the first backup

Manual Steps

Not automated yet ...

• Firewall settings – default deny, except ssh managed by fail2ban
• Monitoring – Monit or munin is easy. SysUsage is a pretty easy to install and configure centralized monitoring suite. There are others.
• Alarming – nagios or opennms are standard tools for this.
• Application settings, working AND secure configs

Server is Ready for Internal Apps

That's all there is to bringing up a server.
Next steps are determined by your needs.

Summary

1. Have a plan / script / paper / DevOps
2. Follow it
3. Improve it every time
4. Consistency is critical
5. Don't Confuse HA, Disaster Recovery and Backups

Questions?

• Please voluteer!
• Want more help? GA-400 Linux meets EVERY Sunday

Questions?

• email: jdp (at) algoloma [dot]
• Website: http://algoloma.com
• Blog: http://blog.jdpfu.com