How to avoid losing your cron jobs one day

In my 14 years of working with Linux I’ve never done that… But there comes a time in every man’s life to accidentally issue crontab -r instead of crontab -e.

Yes, that removes the current crontab. Yes, the buttons "e" and "r" are conveniently located next to each other. No, it doesn’t prompt for any sort of confirmation. And yes, like the old saying goes, *nix is very user-friendly – it’s just choosy about who its friends are.

What to do:

  • Backups, obviously;
  • If you’re messing with the crontab, chances are, you’ve already edited it recently. Use a terminal emulator with an “instant replay” feature, like iTerm2, to go back in time slightly;
  • Grab the commands from the syslog. On Debian, this is a good starting point:
    grep CRON /var/log/syslog | awk '{$1=$2=$3=$4=$5=$6=$7=""; print $0}' | sort | uniq
  • Add an alias to your favorite shell: alias crontab="crontab -i"

proxy_pass based on GET args in nginx

A bit of a non-trivial task, as it turns out the location directive will not match anything after the ? in the URL. Furthermore, attempting to work around this issue by putting the proxy_pass inside an if block will likely result in an error like this one:
nginx: [emerg] "proxy_pass" cannot have URI part in location given by regular expression, or inside named location, or inside "if" statement, or inside "limit_except" block in /etc/nginx/conf.d/domain.com.conf:23

There is a workaround, it’s not pretty, but it should do the trick (giving you time to hopefully refactor the URLs in your app to something more sensible). In the example below, I needed to match a couple of MD5 hashes and proxy those requests to host B while proxying the rest of the requests to host A.

location / {
		error_page 418 = @myredir;
		if ( $args ~ "c=[0-9a-f]{32}" ) { return 418; }
		
		proxy_pass http://host_a:80;
		proxy_set_header Host yourdomain.com;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_connect_timeout 120;
		proxy_send_timeout 120;
		proxy_read_timeout 180;
}

location @myredir {
		proxy_pass http://host_b:80;
		proxy_set_header Host yourdomain.com;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_connect_timeout 120;
		proxy_send_timeout 120;
		proxy_read_timeout 180;
}

Note that the error code we are “throwing” is actually a part of a old April fool’s RFC. You can use any other code that is not a valid HTTP response code.

Quick tip: emulating plain text files directly in nginx config

Every once in a while I end up setting up another reverse proxy/rewrite engine with nginx. In order to keep such configs easily portable you could do something like this:

	location =/robots.txt {
		add_header              Content-Type text/plain;
		return 200 "User-agent: *
Disallow: /
";
	}

This eliminates the need to distribute an actual robots.txt file with the example above. If you intend to serve HTML instead, obviously change the MIME type to text/html as well.

Be careful with innodb_force_recovery

Recently, I needed to import a large SQL dump into a server running MariaDB 10.1. I was surprised to see the following message pop up in the middle of import:

ERROR 1036 (HY000) at line 1111: Table 'clients' is read only

These kinds of messages are typically encountered when DB files are copied directly and filesystem-level permissions are amiss. In my case though, I had a consistent SQL dump that just wouldn’t import. Attempting to do any inserts into the table using the command line client would fail with the same error message. It’s worth noting that the database was a mix of MyISAM and InnoDB tables, and the MyISAM tables imported fine.

As it turns out, the culprit was the following line in my.cnf:

innodb_force_recovery=3

This line was inherited from a previous setup. Recent versions of MySQL switch the InnoDB engine into read-only mode when this value is set to >=4. Apparently, MariaDB took this one step further and switched to read-only at the value of 3.

Shell scripting: “while read” loops breaking early

I stumbled upon this issue a couple of days ago while trying to debug a deployment script. The script would read entries from a CSV file and provision VMs one by one. The issue was that only the first entry from the file would get processed, after which the script would exit normally as if there were no more entries.

As it turns out, if you are using a “while read” construct to read something from a file and in the loop you have something that reads input from stdin… well, it will eat up the rest of the lines from the file, naturally, causing the while loop to break.

In my case I had a ssh command in the loop. All it took to fix it was to deny reading from stdin by adding the “ssh -n” flag:

while IFS="," read newhost newipaddr; do
  ssh -n $newhost "my remote commands"
done << newhosts.csv

Transparent access to .onion websites

This is a very basic setup but I’m sharing mine in hopes of saving someone a bunch of googling.

<insert the usual drill about TOR and anonymity and why it’s important>

So my goal was not to anonymize my every move, but rather to be able to key in an *.onion website into the URL bar of any device on the local network and have that delivered transparently. I happen to own a fancy Asus router with AsusWRT-Merlin on board, but the approach should work on pretty much any Linux box that can route traffic.

Step 1: Get TOR installed. Use the package manager available for your router/Linux box. Here’s my config from the router:

SocksPort 9050
Log notice file /tmp/torlog
VirtualAddrNetwork 10.192.0.0/10
AutomapHostsOnResolve 1
TransPort 9040
TransListenAddress 192.168.1.1
DNSPort 9053
DNSListenAddress 192.168.1.1
RunAsDaemon 1
DataDirectory /tmp/.tordb
AvoidDiskWrites 1

A client must first request the domain to be resolved through TOR’s DNS (which in our case will be available at 192.168.1.1:9053). TOR will respond with an address from the 10.192.0.0/10 subnet. The browser will then attempt to connect to said IP address, which our router should intercept and redirect to 192.168.1.1:9040.

Step 2: Figure out DNS resolution. We want to resolve *.onion domains through TOR and resolve everything else through our regular DNS server. My router uses dnsmasq, so adding this line to the config file should do the trick:

server=/.onion/192.168.1.1#9053

Kill and restart dnsmasq for this to take effect.

Step 3: Intercept and redirect. Easy:

iptables -t nat -A PREROUTING -d 10.192.0.0/10 -i br0 -p tcp -m tcp -j REDIRECT --to-ports 9040

At this point you should be able to open an *.onion website in your browser and see it just work!

Step 4: Making the changes persistent. This one will depend on how your router firmware handles that. For mine, I created two new files in the persistent partition (a.k.a. JFFS).
/jffs/configs/dnsmasq.conf.add:

server=/.onion/192.168.11.1#9053

/jffs/scripts/nat-start:

#!/bin/sh
iptables -t nat -A PREROUTING -d 10.192.0.0/10 -i br0 -p tcp -m tcp -j REDIRECT --to-ports 9040

And finally:

chmod a+rx /jffs/scripts/*

That’s it. Read more on user scripts in AsusWRT-Merlin here if you feel like it.

How to: nested LVM resize

Suppose you have a virtualization host, and each VM is assigned its own LVM partition for disk storage.

Expanding the storage

This one is trivial. You call lvresize on the host to add more space to the LVM partition. Then you shut down the VM and start it back up again so that it realizes it has some extra space now (note that a simple reboot doesn’t seem to cut it on KVM hosts, looks like the domain needs to be destroyed for the changes to take effect).
After that, from inside the VM, you can perform an online FS resize with something like:

lvresize --resizefs --size 100G /dev/mapper/my-cool-vg-MyLogVol00

Shrinking the storage

A bit more tricky, this one is. First of all, there is no such thing as online shrinking of a ext3/4 FS. Therefore, all operations need to happen while the VM is powered down, which will cause a tad more downtime.

On the host machine, use a nifty util called kpartx to create device mappings from the guest VM’s LVM partition:

kpartx -av /dev/my-host-vg/my-cool-vm

Now, the volume group inside the LVM partition should be visible on the host. Run vgscan to verify that:

[root@host ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "my-cool-vg" using metadata type lvm2
  Found volume group "my-host-vg" using metadata type lvm2

So the new volume group is there, but it’s currently marked inactive. Let’s fix that:

vgchange -ay my-cool-vg

Now, we can perform any operations with the nested LVMs. Time to shrink the partition and the FS:

lvresize --resizefs --size 50G /dev/mapper/my-cool-vg-MyLogVol00

Deactivate the volume group:

vgchange -an my-cool-vg

Now, we need to shrink the “guest” physical volume. Circle back to the kpartx output to see which mappings it added. In my setup, the first partition was the boot partition, and the second was the actual nested LVM.

pvresize --setphysicalvolumesize 51G /dev/mapper/my-host-vg-my-cool-vm2

Calculate the size carefully as this will also need to accommodate any other non-LVM partitions you have on the guest VM (in my case, I added the /boot partition here).

Now, we can get rid of the mappings that kpartx created:

kpartx -dv /dev/my-host-vg/my-cool-vm

Finally, resize the LVM partition on the host machine:

lvresize --size 52G /dev/my-host-vg/my-cool-vm

Start up the VM. Done.

Renaming the “guest” volume group

In some environments, the names of both volume groups may turn out to be the same (for example, if you didn’t consider this while writing your kickstart templates). In that case, you’ll see an error message like this:

[root@host ~]# vgscan
  Reading all physical volumes.  This may take a while...
  WARNING: Duplicate VG name VolGroup00: Existing Zclhms-H89I-VH1b-thtf-URlf-URTT-4eqwmU (created here) takes precedence over 8TlO1m-qb5b-fieh-wHDB-qzwQ-HeFe-tXIP0N
  WARNING: Duplicate VG name VolGroup00: Existing Zclhms-H89I-VH1b-thtf-URlf-URTT-4eqwmU (created here) takes precedence over 8TlO1m-qb5b-fieh-wHDB-qzwQ-HeFe-tXIP0N
  Found volume group "VolGroup00" using metadata type lvm2
  Found volume group "VolGroup00" using metadata type lvm2

In order to proceed, we’ll need to rename the “guest” volume group:

vgrename 8TlO1m-qb5b-fieh-wHDB-qzwQ-HeFe-tXIP0N my-new-vg

Problem solved? Not really, since the VM will fail to start after this.

# activate the new volume group
vgchange -ay my-new-vg
# create a temporary mount point
mkdir my-vm-temp
# mount the partitions
mount /dev/my-new-vg/MyLogVol00 my-vm-temp/
mount /dev/mapper/my-new-vg-my-cool-vm1 my-vm-temp/boot/
# update fstab
sed -i "s/VolGroup00/my-new-vg/g" my-vm-temp/etc/fstab
# update bootloader
sed -i "s/VolGroup00/my-new-vg/g" my-vm-temp/boot/grub/grub.conf
# unmount
umount my-vm-temp/boot/
umount my-vm-temp/

That was quite a detour, but now you can proceed with the actual resize. Yay.

How to: route VM traffic based on destination port

Suppose you need to route all outbound traffic to a certain destination port though a different interface (VPN, GRE, you name it). Not something you have to do every day, but if you have found this article you probably know what you are doing at this point. Suppose your virtualization host has a public IP of 111.111.111.111 and the host you want to route the traffic through has a public IP of 222.222.222.222

Set up your GRE tunnel

This section is here for the sake of completeness. If you feel confident setting up your own tunnels or VPNs, just skip ahead.
Let’s make sure the ip_gre module is present in the system and set it to autoload:

lsmod | grep gre
modprobe ip_gre
echo ip_gre >> /etc/modules

Next step, let’s create the tunnel on the VM host and assign a local IP to it:

ip tunnel add gre01 mode gre remote 222.222.222.222 local 111.111.111.111 ttl 255
ip link set gre01 up
ip addr add 10.10.10.1/24 dev gre01

Tunnel creation on the remote host is identical:

ip tunnel add gre01 mode gre remote 111.111.111.111 local 222.222.222.222 ttl 255
ip link set gre01 up
ip addr add 10.10.10.2/24 dev gre01

At this point the tunnel is up and we should be able to ping the remote host from the VM host:

ping 10.10.10.2

Time to mangle some traffic

Create a routing table on the VM host and point it to your remote host. I used number 12 here, that’s arbitrary as long as it doesn’t conflict with existing tables on your system.

ip rule add fwmark 12 table 12
ip route add default via 10.10.10.2 table 12
ip route flush cache

Add the iptables rule that will set the forwarding mark on the VM host:

iptables -t mangle -A OUTPUT -p tcp --dport 8888 -j MARK --set-mark 12

This assumes that your VMs have bridged interfaces. If you’re running a NAT setup, you’ll need to put this into the prerouting chain instead:

iptables -t mangle -A PREROUTING -p tcp --dport 8888 -j MARK --set-mark 12

Now, to make sure the remote host accepts this:

iptables -t nat -A POSTROUTING -o gre01 -j SNAT --to-source 10.10.10.1

Finally, we need to loosen up the reverse path filter on the GRE interface of the VM host. Obviously, substitute gre01 with whatever interface you have configured:

sysctl -w net.ipv4.conf.gre01.rp_filter=2

Now, just to add a masquerade rule on the remote host, and we are done:

iptables -t nat -A POSTROUTING -s 10.10.10.0/24 -j MASQUERADE

Conclusion

We have configured a set of rules that will allow our VMs “use” a different public IP address for their outbound connections depending on the destination port. The important bit is that this is completely transparent for the VMs.

You will probably want to make these changes permanent by configuring persistent rules and interfaces on both hosts. We have also used a trick called loose mode reverse filtering, which you can read up on here: http://www.slashroot.in/linux-kernel-rpfilter-settings-reverse-path-filtering