Last Updated: November 02, 2016
· nekwebdev

Linode Debian 7 web server setup

<a name=top></a>

Table of contents


I Basic Information

  • Set the hostname
  • Set the FQDN
  • Set the time
  • Update the server
  • Install a compiler

II Admin user setup

  • Create the admin user
  • SSH config

III Harden the network

  • sysctl
  • Prevent IP spoofing
  • Secure shared memory
  • IP tables
  • Setup fail2ban


  • Postfix and gmail
  • Apache
  • MySQL
  • PHP

V Prepare for users

  • Edit MOTD
  • Setup permissions and umask
  • Git
  • Oh My ZSH
  • Chroot jail made easy
  • /etc/skel setup

VI Maintenance

  • Logwatch
  • RKhunter and CHKRootkit
  • Tiger
  • Admin Tasks




Again, this comes from a selfish need to record my progress on a system admin project.

I am just done setting up my Linode the way I like it and I think I am about to just start over, a lot of us are like that uh, just because I feel I could have missed things here and there. And that besides the huge research that went into it, it shouldn't be too hard to reproduce. As long as I know what I am doing...

I took notes along the way knowing I would probably need to reproduce it at some point. Seems like the need is more urgent than I thought so I will do it again, with a different approach this time, and document things here.

We will setup a disk that will contain a fully setup Debian 7 web server. Note that I think you can copy paste everything in ubuntu and it will work as well.

Start by deploying a Debian 7 on your linode. Once it is booted up let's get down to business.

ssh root@linodedomain.com or IP



<a name=basicinfo></a>

Basic information

Set the hostname

echo "mysuperhostname" > /etc/hostname
hostname -F /etc/hostname


Set the FQDN

Add this line to your /etc/hosts file

<YOUR IP>  mysuperhostname.linodedomain.com  mysuperhostname

Make sure to add an A record for mysuperhostname.linodedomain.com that points to your IP.


Set the time

dpkg-reconfigure tzdata

All good.


Update the server

apt-get update
apt-get upgrade


Install a compiler

You will need it at some point...

apt-get install build-essential

<a name=admin></a>

Admin user setup

Create the admin user

apt-get install sudo
adduser adminuser
usermod -a -G sudo adminuser
groupadd admin
usermod -a -G admin adminuser
dpkg-statoverride --update --add root admin 4750 /bin/su

This creates our adminuser and adds him to the sudo and admin group which is the now the only other group that can su with root.

Ok let's harden, or even soften for now our sudoers file, but before that change the default editor. List the available editors first to make sure vim is there:

update-alternatives --list editor
sudo update-alternatives --set editor /usr/bin/vim.basic

Add this to limit the time the sudo password is remembered:

Defaults        timestamp_timeout=10

We will also make it so that our admin user will not have to enter his password on sudo, at least during setup, you can change it back after. Make sure you add this at the END of the file, so it is not over written by the %sudo line. Once our major setting up is done


You can also add this instead of adding the user to the sudo group:

[your username] ALL=(ALL:ALL) ALL

But I find it cleaner to manage what you can do via groups, which we will do again later.

We can now sudo with our new user, let's log back on:

ssh adminuser@linodedomain.com


SSH config

Generate your key on your machine then copy it over to the server:

cd ~/.ssh
ssh-keygen -t rsa -C "email@host"
scp ~/.ssh/id_rsa.pub adminuser@linodedomain.com:

Then log back in and clean up the key with the right permissions:

mkdir .ssh
mv id_rsa.pub .ssh/authorized_keys
chown -R adminuser:adminuser .ssh
chmod 700 .ssh
chmod 600 .ssh/authorized_keys

Let's create our sshlogin group and add our admin user to it

sudo groupadd sshlogin
sudo usermod -a -G sshlogin adminuser

Now we will secure SSH some more and prepare for later restrictions, note that I am making a backup of the config file before I make changes, and I like to comment out the line I changed and add in another to replace it with the changes made. You can easily refer to defaults this way and even revert to original config. Get in the habit of doing this for all config file changes.

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.back
sudo vim /etc/ssh/sshd_config

By adding this we make it so ONLY users part of sshlogin group can login through ssh

Port = XXXX
LoginGraceTime 30
PermitRootLogin no
UseDNS no
DebianBanner no
Banner /etc/issue.net
AllowGroups sshlogin
ClientAliveInterval 600
ClientAliveCountMax 0

Change the default SSH port number if you like, I choose not to. Keep it under 1024, ports under this number can only be listened to by root or root users. Tools such as Fail2Ban will keep our logs clean from SSH brute forcing so no need to obfuscate on this part.

Change /etc/issue.net to what ever banner floats your boat, I like the gov type one:

                            NOTICE TO USERS

This computer system is the private property of its owner, whether
individual, corporate or government.  It is for authorized use only.
Users (authorized or unauthorized) have no explicit or implicit
expectation of privacy.

Any or all uses of this system and all files on this system may be
intercepted, monitored, recorded, copied, audited, inspected, and
disclosed to your employer, to authorized site, government, and law
enforcement personnel, as well as authorized officials of government
agencies, both domestic and foreign.

By using this system, the user consents to such interception, monitoring,
recording, copying, auditing, inspection, and disclosure at the
discretion of such personnel or officials.  Unauthorized or improper use
of this system may result in civil and criminal penalties and
administrative or disciplinary action, as appropriate. By continuing to
use this system you indicate your awareness of and consent to these terms
and conditions of use. LOG OFF IMMEDIATELY if you do not agree to the
conditions stated in this warning.


Also, check the man for all the values you are setting, you might learn a thing or two. And google, google, google :)

Double check that you are part of sshlogin group, I got kicked out like that, not fun.

id adminuser

Restart SSH and lets try out our key :)

sudo service ssh restart

You can also create a ~/.ssh/config file on your machine to make an alias:

Host aliasname
    HostName linodedomain.com
    Port 22
    User adminuser

From now on change the SSH port if you changed it in your sshd_config.

ssh aliasname

Save yourself some pain and do this right away:

cd ~
echo "syntax on" > .vimrc

You shouldn't have a .vimrc file and this will turn on syntax highlighting.

<a name=network></a>

Harden network


sudo cp /etc/sysctl.conf /etc/sysctl.back.conf
sudo vim /etc/sysctl.conf

Last reminder to backup your config files default ;)

 # Make server reboot on low memory

# Specific to Linode, experiment.

# IP Spoofing protection
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Ignore ICMP broadcast requests
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Disable source packet routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0 
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0

# Ignore send redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Block SYN attacks
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 5

# Log Martians
net.ipv4.conf.all.log_martians = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Ignore ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0 
net.ipv6.conf.default.accept_redirects = 0

# Ignore Directed pings
net.ipv4.icmp_echo_ignore_all = 1

Add all this and reload these configs

sudo sysctl -p


Prevent IP spoofing

Add or edit the following lines in /etc/host.conf:

order bind,hosts
nospoof on


Secure shared memory

Add this to your /etc/fstab file:

tmpfs /dev/shm tmpfs defaults,noexec,nosuid 0 0

Mount it.

sudo mount -a


IP tables

Let's create a new IP tables files that we will load in ip tables to complement what fail2ban has there already.

sudo vim /etc/iptables.firewall.rules

Copy this, according to your own port settings:


#  Allow all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT ! -i lo -d -j REJECT

#  Accept all established inbound connections

#  Allow all outbound traffic - you can modify this to only allow certain traffic

#  Allow HTTP and HTTPS connections from anywhere (the normal ports for websites and SSL).
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

#  Allow ports for testing
-A INPUT -p tcp --dport 8080:8090 -j ACCEPT

#  Allow ports for MOSH (mobile shell)
-A INPUT -p udp --dport 60000:61000 -j ACCEPT

#  Allow SSH connections
#  The -dport number should be the same port number you set in sshd_config
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT

#  Allow ping
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

#  Log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

#  Reject all other inbound - default deny unless explicitly allowed policy


Save the rules and prepare a network up script

sudo iptables-restore  < /etc/iptables.firewall.rules
sudo iptables -L
sudo vim /etc/network/if-pre-up.d/firewall

Copy this in the script you created

/sbin/iptables-restore < /etc/iptables.firewall.rules

Make it executable

sudo chmod +x /etc/network/if-pre-up.d/firewall

This will make sure the IP tables are saved on a server restart. Note that tcp ports 8080 to 8090 are open for dev purposes. You might want to remove those if you do not use them.


Setup fail2ban

sudo apt-get install fail2ban
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sudo vim /etc/fail2ban/jail.local

Add or edit these lines

destemail = admin@linodedomain.com
action = %(action_mwl)s


enabled  = true
port     = ssh
filter   = sshd
logpath  = /var/log/auth.log
maxretry = 6


enabled  = true
port     = ssh
filter   = sshd-ddos
logpath  = /var/log/auth.log
maxretry = 6

Basically make sure ssh and ssh-dos are set to true, and change ssh port if needed.

sudo service fail2ban restart

Note that fail2bans adds it's own rule to iptables on it's own at boot time, no need to add them to our script.

<a name=lamp></a>


Postfix and gmail

First let's start by configuring our email server using gmail, because who runs their own mail server these days ;)

sudo apt-get install postfix mailutils libsasl2-2 ca-certificates libsasl2-modules

Choose internet site for the config, and enter any email, you'll change it later.

sudo vim /etc/postfix/main.cf

Add this at the end

relayhost = [smtp.gmail.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_CAfile = /etc/postfix/cacert.pem
smtp_use_tls = yes

Create our password file

vim /etc/postfix/sasl_passwd

And add following line:

[smtp.gmail.com]:587    USERNAME@gmail.com:PASSWORD

If you want to use your Google App’s domain, please replace @gmail.com with your @domain.com.

Fix permission and update postfix config to use sasl_passwd file:

sudo chmod 400 /etc/postfix/sasl_passwd
sudo postmap /etc/postfix/sasl_passwd

Next, validate certificates to avoid running into error. Just run following command:

cat /etc/ssl/certs/Thawte_Premium_Server_CA.pem | sudo tee -a /etc/postfix/cacert.pem

Finally, reload postfix config for changes to take effect:

sudo /etc/init.d/postfix reload

If you have configured everything correctly, following command should generate a test mail from your server to your mailbox.

echo "Test mail from postfix" | mail -s "Test Postfix" you@example.com

This whole section is pretty much a copy/paste from rtcamp.com. Credits!

Delete the /etc/postfix/sasl_passwd file you created. The postmap command converted it, delete this clear text file with your gmail password!

Now forward all root mails, edit /etc/aliases

root: username@gmail.com

Then, rebuild aliases:




sudo apt-get install apache

Optimise apache2 for a Linode 1GB:

sudo vim /etc/apache2/apache2.conf

And make sure these settings match for this section:

<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 6
MaxSpareServers 12
MaxClients 30
MaxRequestsPerChild 3000

Secure apache 2, open:

sudo vim /etc/apache2/conf.d/security

Add or edit these lines:

ServerTokens Prod
ServerSignature Off
TraceEnable Off
Header unset ETag
FileETag None

Fix some modules and restart apache2:

sudo a2enmod headers
sudo a2dismod autoindex
sudo a2dismod status
sudo a2dismod cgi
sudo service apache2 restart

We will come back to setting up virtual hosts later.



sudo apt-get install mysql-server

During the install enter the password you want for the MySQL root user. Now secure the installation:

sudo mysql_secure_installation

Optimise MySQL for a Linode 1GB:

sudo vim /etc/mysql/my.cnf

Make sure the following values are set:

max_connections = 75
key_buffer = 32M
max_allowed_packet = 1M
thread_stack = 128K
table_cache = 32

Restart MySQL:

sudo service mysql restart

Lets try to keep these sql databases in good shape:

sudo crontab -e

Add this line:

@weekly mysqlcheck -o --user=root --password=sqlrootpwd -A

Now let's automate backups:

sudo apt-get install automysqlbackup
sudo vim /etc/default/automysqlbackup

Change BACKUPDIR="/home/<your username>/sqlbackup/" and MAILADDR="admin@linodedomain.com" Also change DBNAMES to this:

DBNAMES=`mysql --defaults-file=/etc/mysql/debian.cnf --execute="SHOW DATABASES" | awk '{print $1}' | grep -v ^Database$ | grep -v ^mysql$ | grep -v ^information_schema$ | grep -v ^performance_schema$ | tr \\\r\\\n ,\ `

This will prevent false positive errors. This was already set in my Debian 7 distro.

That's it, you will receive an email in case of errors during the backup.



sudo apt-get install php5 php-pear php5-mysql
sudo service apache2 restart

php5-suhosin was removed from Debian 7 and I am keeping the install bare for now. Now let's optimise for a Linode 1GB and harden our php install:

sudo vim /etc/php5/apache2/php.ini

Make sure the following values are set:

max_execution_time = 30
memory_limit = 128M
display_errors = Off
log_errors = On
error_log = /var/log/php.log
register_globals = Off
disable_functions = exec,system,shell_exec,passthru (add those to existing list)
expose_php = Off
track_errors = Off
html_errors = Off

Restart apache.

<a name=finish></a>

Prepare for users


Open /etc/motd and just delete everything, or draw a picture, up to you :)

Add this to your ~/.profile, modify how you see fit.

# Make our ssh login useful
echo ""
w # uptime and logged in user info
echo ""
df -h -x tmpfs -x udev # disk usage, minus def and swap
echo ""


Setup permissions and umask

We want home and user home permissions to be like this:

drwxr-x--x root root /home
drwx--x--x adminuser adminuser /home/adminuser

Then in your adminuser directory:

-rw------- adminuser adminuser .bash_history
-rw------- adminuser adminuser .bash_logout
-rw------- adminuser adminuser .bashrc
drwx------ adminuser adminuser github
-rw------- adminuser adminuser .profile
drwx------ adminuser adminuser .ssh
-rw------- adminuser adminuser .viminfo

Now that we are done doing configuration stuff, let's set our umask, add umask 077 to the ~/.profile file. This will make all directories you create 700 and files 600. Thats is the most secure umask, make sure to set the right permissions on files or directories that need more rights.


For all the server stuff it is nice to have good stable versions. But for our day to day apps like git, we want the latest releases.

sudo apt-get install libcurl4-gnutls-dev libexpat1-dev gettext libzen-dev libssl-dev
mkdir github
cd github
wget https://github.com/git/git/archive/v1.8.3.2.tar.gz
tar -zxf git-
rm v1.8.3.2.tar.gz
cd git-
make prefix=/usr/local all
sudo make prefix=/usr/local install
cd ..
rm -rf git-
git clone https://github.com/git/git.git

Now when you need to update git to the bleeding edge:

cd ~/github/git
git fetch
git tag
git checkout vx.x.x
make prefix=/usr/local all
sudo make prefix=/usr/local install
git --version



sudo apt-get install zsh zsh-doc
cd ~
curl -L https://github.com/robbyrussell/oh-my-zsh/raw/master/tools/install.sh | sh

Ignore errors and run

chsh -s /usr/bin/zsh

Relog and you should be under your new zsh shell. I'll leave the customisation up to you...
Take a note to put the umask 077 in ~/.zshrc as well since you will not be autoloading .profile anymore. Try, the ys theme, really nice. Also add the login information you added earlier to ~/.profile to your zshrc file. Now both shells will have useful information at login.


Chroot jail made easy

We will have two types of web users. The one that owns and manages the main domain of this web server. linodedomain.com in our example. This user will be a regular user in our system, using a pimp zsh and with access to all commands a regular user has. No sudo. This is what the admin user is for. Once everything is setup right, the web users do not need any kind of su rights.

Our other type of user is for hosting small on the side project or sites. We do not want to give them any kind of access or control on the server. So they need a chroot jail of some sort. Easiest implementation is by far lshesll.

cd ~/github
git clone https://github.com/ghantoos/lshell.git lshell
cd lshell
sudo python setup.py install --no-compile --install-scripts=/usr/bin/

This will create a lshell group. Just add a user to this group and he will be jailed according to the rules set in /etc/lshell.conf.

I like to allow sftp, scp and make it so scp uploads go to /home/%u/uploads.

Some more setup is needed in /etc/ssh/sshd_config. At the very end, add:

Notice the last line. We are going to make it so our adminuser can not login via ssh/sftp using his password, this means that if you loose your ssh key, you are locked out. Make that security choice. I like it that the only user with sudo rights can only log with ssh keys.

# Jail our lshell group user
Match Group lshell
    AllowTCPForwarding no
    X11Forwarding no
    ForceCommand /usr/bin/lshell /etc/lshell.conf

# Disable password for the system admins
Match Group admin
    PasswordAuthentication no

Quick recap, you need to be part of group sshlogin to log via ssh. If you are part of group admin you must have ssh keys setup, so do not add a user to admin until the public key is on the server. And if you are part of the lshell group, though luck you are jailed and limited in your commands to the list set in /etc/lshell.comf

Apply your changes with sudo service ssh restart

Now to jail a user simply add him to the lshell group, I think it is nice to add these aliases in ~/.zshrc:

alias jailuser='sudo usermod -a -G lshell $1'
alias unjailuser='sudo deluser $1 lshell'
alias lsjailed='grep lshell /etc/passwd'


/etc/skel setup

The default umask for our users will be a 027 since all files that apache needs to read will be owned by the www-data group, and the permissions set to group write when needed. The folder should look something like this. uploads ils where all files uploaded via scp will go for jailed users, the other two will be owned by apache to write it's logs.

-rw------- root root .bash_logout
-rw------- root root .bashrc
drwx------ root root errors
drwx------ root root logs
-rw------- root root .profile
drwx------ root root uploads

Now we need to set the system wide umask to 0027. Open:

sudo vim /etc/pam.d/common-session
sudo vim /etc/pam.d/common-session-noninteractive

And add

session optional        pam_umask.so umask=0027


<a name=maintain></a>



sudo apt-get install logwatch

Don't bother with the config file as we will be using options in the command. Edit /etc/cron.daily/00logwatch and change the arguments to this:

/usr/sbin/logwatch --mailto admin@linodedomain.com --output mail --format html --range 'between -7 days and today'

Make it weekly and restart cron:

sudo mv /etc/cron.daily/00logwatch /etc/cron.weekly/00logwatch
sudo service cron restart


RKhunter and CHKRootkit

I doubt we got a rootkit from git or zsh, so we will setup our base system at this point.

sudo apt-get install rkhunter chkrootkit
sudo chkrootkit
sudo rkhunter --update
sudo rkhunter --propupd
sudo rkhunter --check

I don't know about putting them on a cronjob. Also only run produp once as this is when the "clean" state is taken, if I am not mistaken :)



sudo apt-get install tiger
sudo tiger


Admin tasks

So now that we have all this configured what is left for the admin to do?

MySQL is getting a weekly check and backup, with the admin getting emails if something goes wrong.

Admin will get weekly mails of the logs from logwatch

Now we need to keep the system updated and run our rootkit checks and tiger check. These we do not want to automate.

sudo apt-get update && apt-get upgrade
sudo updatedb
sudo chkrootkit
sudo rkhunter --update
sudo rkhunter --check

Thats sums it up.

Now would be a good time to remove username ALL=(ALL) NOPASSWD: ALL from /etc/sudoers using visudo.



Deploy from the master


Partition the system

First we will create 2 new drives on Linode and add them to our configuration. Make them a few gigs each, you can play with the size as time passes. Also make your root partition a few gigs bigger to account for the temp backups.

Add those two drives to your linode server configuration and check single user mode. Reboot. Check out Linode's remote access to ssh to your server in single user mode.

Let's copy our old fstab

sudo cp /etc/fstab /etc/fstab.$(date +%Y-%m-%d)

Now move our current /home and /var

mv /var /var.tmp
mv /home /home.tmp
mkdir /var
mkdir /home
mount /dev/xvdc /home
mount /dev/xvdd /var
cd /var.tmp
tar cf - * | ( cd /var; tar xfp -)
cd /home.tmp
tar cf - * | ( cd /home; tar xfp -)

Add these lines in /etc/fstab:

/dev/xvdc   /home            ext3    defaults      0 2
/dev/xvdd   /var             ext3    defaults      0 2

Reboot and go back to your Linode manager to reboot in normal mode to remove our old /var and /home

sudo rm -rf /var.tmp
sudo rm -rf /home.tmp

Taylor to client

I tried to keep this install as generic as possible, with a generic admin user name, of course I am not using adminuser in production, but something just as dull :) So remember to comment out PasswordAuthentication no in /etc/ssh/sshd_config and let the client know about the option for his admin user. Up to him.

I made a copy of the linode disk that was running this server as a master for future deployments.

Only a few changes will have to be made, mostly regarding emails and smtp settings.

Edit the admin user

Have a look at this and edit the user according to your needs. Note that with the email alias of root the first and last name of the admin user will be used in the email information sent out by the server. Use the company name for example.

Time, hostname and /etc/hosts

Set the time of the server.

dpkg-reconfigure tzdata

Set the new hostname.

echo "clienthostname" > /etc/hostname
hostname -F /etc/hostname

Edit the /etc/hosts file.

<CLIENT IP>  clienthostname.clientdomain.com  clienthostname

Make sure to add an A record for clienthostname.clientdomain.com that points to the IP.



We need to change the email for the alerts. Edit /etc/fail2ban/jail.local and change destemail = admin@clientdomain.com



Change the email and password values in /etc/postfix/sasl_passwd and check /etc/postfix/main.cf. If gmail is not used anymore you might want to reconfigure the postfix package.

Then reload the password file.

sudo postmap /etc/postfix/sasl_passwd

Once again, delete the original /etc/postfix/sasl_passwd after the conversion.

Also change the root email alias in /etc/aliases

root: username@gmail.com


SQL backups

automysqlbackups needs the admin email as well,

In /etc/default/automysqlbackup change MAILADDR=.



Change email in /etc/cron.weekly/00logwatch


Reset MySQL root password


Create main web user

I decided to make a script for creating my users. Lots of repetitive actions there, and if I stick to this master server the script will always work. First let's start by changing path in ~/.zshrc and adding export PATH=$PATH:/home/adminuser/bin now create this ~/bin directory.

This is a link to the script I'll be using: https://gist.github.com/nekwebdev/5963724



So we have a user called useradmin part of sudo, sshlogin, admin.

Being part of sudo gives super user rights.

Being part of admin forces the user to use ssh keys. Also gives access to su.

Being part of sshlogin allows login via ssh.

Being part of lshell will jail the user.

The main web user will have to be part of sshlogin.

The alternative users will have to be part of sshlogin and lshell.







1 Response
Add your response


Fantastic detail! Will you begin to automate and script these excellent processes? I'm thinking AWS CloudFormation, OpsCode Chef and the normal Continuous Delivery infrastructure as code tenets!

over 1 year ago ·