Async.fi

Mikrotik OpenVPN Server

The purpose of this post is to describe, step by step, my attempt to set up an OpenVPN server on a Mikrotik RouterBOARD 750 and create a working tunnel from an outside machine (AWS EC2 Windows Server 2008 R2) to this OpenVPN server so that an SMB server on the local network can be accessed from said outside machine. The following diagram gives an overview of the setup:

I am going to decribe how to:

  • generate certificates to be used with OpenVPN
  • set up OpenVPN server on Mikrotik router
  • set up a tunnel with OpenVPN client on Windows
I am not going to describe the following:
  • setting up and connecting to an EC2 Windows instance
  • setting up a Samba Server
A few things worth mentioning about Mikrotik OpenVPN server implementation (that will likely bite if not known in advance):
  • only supports TCP mode, UDP is not supported
  • username/password pair is also required even though certificates are being used for authentication

Generate certificates to be used with OpenVPN

root@inhouse-debian:~# apt-get install openvpn
root@inhouse-debian:~# mkdir ovpn-cert
root@inhouse-debian:~# cd ovpn-cert/
root@inhouse-debian:~/ovpn-cert# cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0/* .
root@inhouse-debian:~/ovpn-cert# emacs vars
In the file vars I set the following values: 
export KEY_COUNTRY="FI"
export KEY_PROVINCE="Etela-Suomi"
export KEY_CITY="Kotka"
export KEY_ORG="Async.fi"
export KEY_EMAIL="joni.kahara@async.fi"
export KEY_CN="kahara.dyndns.org"
export KEY_NAME="kahara.dyndns.org"
export KEY_OU="kahara.dyndns.org"
If I have understood correctly, of these only CN (Common Name) is obligatory. I may be wrong. Anyway, continuing: 
root@inhouse-debian:~/ovpn-cert# source vars
root@inhouse-debian:~/ovpn-cert# ./clean-all
root@inhouse-debian:~/ovpn-cert# ./build-ca
root@inhouse-debian:~/ovpn-cert# ./build-key-server kahara.dyndns.org
root@inhouse-debian:~/ovpn-cert# openssl rsa -in keys/kahara.dyndns.org.key -out keys/kahara.dyndns.org.pem
root@inhouse-debian:~/ovpn-cert# ./build-key ec2 
root@inhouse-debian:~/ovpn-cert# apt-get install ncftp
root@inhouse-debian:~/ovpn-cert# ncftpput -u admin 192.168.1.1 / keys/kahara.dyndns.org.crt keys/kahara.dyndns.org.pem keys/ca.crt

Set up OpenVPN server on Mikrotik router

All the stuff here can also be made through Mikrotik's admin interface; textual form without screen shots is used just to keep thing terse. 
root@inhouse-debian:~/ovpn-cert# ssh admin@192.168.1.1
[admin@MikroTik] > /certificate
[admin@MikroTik] /certificate> import file=kahara.dyndns.org.crt
[admin@MikroTik] /certificate> import file=kahara.dyndns.org.pem
[admin@MikroTik] /certificate> import file=ca.crt
[admin@MikroTik] /certificate> decrypt
[admin@MikroTik] /certificate> ..
[admin@MikroTik] > /interface bridge add name=ovpn-bridge
[admin@MikroTik] > /interface bridge port add interface=ether2-master-local bridge=ovpn-bridge
[admin@MikroTik] > /ip address add address=192.168.1.64/24 interface=ovpn-bridge 
[admin@MikroTik] > /ip pool add name=ovpn-pool ranges=192.168.1.65-192.168.1.99
[admin@MikroTik] > /ppp profile add bridge=ovpn-bridge name=ovpn-profile remote-address=ovpn-pool
[admin@MikroTik] > /ppp secret add service=ovpn local-address=192.168.1.64 name=user1 password=pass1 profile=ovpn-profile
[admin@MikroTik] > /interface ovpn-server server set auth=sha1,md5 certificate=cert1 cipher=blowfish128,aes128,aes192,aes256 default-profile=ovpn-profile enabled=yes keepalive-timeout=disabled max-mtu=1500 mode=ethernet netmask=24 port=1194 require-client-certificate=yes
[admin@MikroTik] > /ip firewall filter add action=accept chain=input disabled=no protocol=tcp dst-port=1194
[admin@MikroTik] > /ip firewall filter move 5 destination=1
That last step moves the new rule to the front of the chain; numbers ("5", "1") will likely be something else on your configuration. Firewall rule listing can be printed with the following command: 
[admin@MikroTik] > /ip firewall filter print

Setup up a tunnel with OpenVPN client on Windows

After installing OpenVPN, create a config file for it. Here it's called "kahara.dyndns.org.ovpn": 
client
dev tap
proto tcp
remote kahara.dyndns.org 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert ec2.crt
key ec2.key
verb 3
pull
auth-user-pass userpass.txt
Also, create a file called "userpass.txt" and put the following to it: 
user1
pass1
Of course in an IRL situation one should use a real password. Make sure you copied the .crt and .key files over to the Windows machine, after which you can run OpenVPN client with: 
PS C:\Users\Administrator\Desktop> openvpn.exe .\kahara.dyndns.org.ovpn
And here we have an EC2 client connected to a local SMB resource over the tunnel:

Tagged with:

Categorised as:

Note to Self: More on South

Setup being along the dev → test → prod lines, to correctly manage database migrations we first set things up at dev:

manage.py syncdb --noinput
manage.py convert_to_south <app>
manage.py createsuperuser

At this point the South migrations are being pushed to repository and pulled in at test:

manage.py syncdb --noinput
manage.py migrate
manage.py migrate <app> 0001 --fake
manage.py createsuperuser

Now, back at dev, after a change to one of the models:

manage.py schemamigration <app> --auto
manage.py migrate <app>
And, after push/pull, at test: 
manage.py migrate <app>

Tagged with:

Categorised as:

Handling Amazon SNS notifications with a Tastypie Resource

Using Django and Tastypie, we automagically respond to SNS subscription requests. After that part is handled, the notification messages start coming in and those are used to trigger an SQS polling cycle (trying to do a thorough job there which may seem like an overkill but it's not). A received SQS message is parsed and contents are passed to an external program that forks and exits which keeps the request from blocking.

from django.conf import settings
from tastypie import fields, http
from tastypie.resources import Resource
from tastypie.bundle import Bundle
from tastypie.authentication import Authentication
from tastypie.authorization import Authorization
from tastypie.throttle import BaseThrottle
import boto.sq 
from boto.sqs.message import Message
from urlparse import urlparse
import base64, httplib, tempfile, subprocess, time, json, os, sys, syslog

# Http://django-tastypie.readthedocs.org/en/latest/non_orm_data_sources.html
class NotificationObject(object):
    def __init__(self, initial=None):
        self.__dict__['_data'] = {}
        if hasattr(initial, 'items'):
            self.__dict__['_data'] = initial
    def __getattr__(self, name):
        return self._data.get(name, None)
    def __setattr__(self, name, value):
        self.__dict__['_data'][name] = value

class NotificationResource(Resource):
    sns_messageid = fields.CharField(attribute='MessageId')
    sns_timestamp = fields.CharField(attribute='Timestamp')
    sns_topicarn = fields.CharField(attribute='TopicArn')
    sns_type = fields.CharField(attribute='Type')
    sns_unsubscribeurl = fields.CharField(attribute='UnsubscribeURL')
    sns_subscribeurl = fields.CharField(attribute='SubscribeURL')
    sns_token = fields.CharField(attribute='Token')
    sns_message = fields.CharField(attribute='Message')
    sns_subject = fields.CharField(attribute='Subject')
    sns_signature = fields.CharField(attribute='Signature')
    sns_signatureversion = fields.CharField(attribute='SignatureVersion')
    sns_signingcerturl = fields.CharField(attribute='SigningCertURL')

    class Meta:
        resource_name = 'notification'
        object_class = NotificationObject
        fields = ['sns_messageid']
        list_allowed_methods = ['post']
        authentication = Authentication()
        authorization = Authorization()

    def get_resource_uri(self, bundle_or_obj):
        return ''

    def obj_create(self, bundle, request=None, **kwargs):

        bundle.obj = NotificationObject(initial={ 'MessageId': '', 'Timestamp': '', 'TopicArn': '', 'Type': '', 'UnsubscribeURL': '', 'SubscribeURL': '', 'Token': '', 'Message': '', 'Subject': '', 'Signature': '', 'SignatureVersion': '', 'SigningCertURL': '' })
        bundle = self.full_hydrate(bundle)

        o = urlparse(bundle.data['SigningCertURL'])
        if not o.hostname.endswith('.amazonaws.com'):
            return bundle

        topicarn = bundle.data['TopicArn']

        if topicarn != settings.SNS_TOPIC:
            return bundle

        if not self.verify_message(bundle):
            return bundle

        if bundle.data['Type'] == 'SubscriptionConfirmation':
            self.process_subscription(bundle)
        elif bundle.data['Type'] == 'Notification':
            self.process_notification(bundle)

        return bundle

    def process_subscription(self, bundle):
        syslog.syslog('SNS Subscription ' + bundle.data['SubscribeURL'])
        o = urlparse(bundle.data['SubscribeURL'])
        conn = httplib.HTTPSConnection(o.hostname)
        conn.putrequest('GET', o.path + '?' + o.query)
        conn.endheaders()
        response = conn.getresponse()
        subscription = response.read()

    def process_notification(self, bundle):
        sqs = boto.sqs.connect_to_region(settings.SQS_REGION)
        queue = sqs.lookup(settings.SQS_QUEUE)
        retries = 5
        done = False
        while True:
            if retries < 1:
                break
            retries -= 1
            time.sleep(5)
            messages = queue.get_messages(10, visibility_timeout=60)
            if len(messages) < 1:
                continue
            for message in messages:
                try:
                    m = json.loads(message.get_body())
                    m['return_sns_region'] = settings.SNS_REGION
                    m['return_sns_topic'] = settings.SNS_TOPIC
                    m['return_sqs_region'] = settings.SQS_REGION
                    m['return_sqs_queue'] = settings.SQS_QUEUE
                    process = subprocess.Popen(['/usr/bin/nice', '-n', '15', os.path.dirname(os.path.normpath(os.sys.modules[settings.SETTINGS_MODULE].__file__)) + '/process.py', base64.b64encode(json.dumps(m))], shell=False)
                    process.wait()
                except:
                    e = sys.exc_info()[1]
                    syslog.syslog(str(e))
                queue.delete_message(message)

    def verify_message(self, bundle):
        message = u''
        if bundle.data['Type'] == 'SubscriptionConfirmation':
            message += 'Message\n'
            message += bundle.data['Message'] + '\n'
            message += 'MessageId\n'
            message += bundle.data['MessageId'] + '\n'
            message += 'SubscribeURL\n'
            message += bundle.data['SubscribeURL'] + '\n'
            message += 'Timestamp\n'
            message += bundle.data['Timestamp'] + '\n'
            message += 'Token\n'
            message += bundle.data['Token'] + '\n'
            message += 'TopicArn\n'
            message += bundle.data['TopicArn'] + '\n'
            message += 'Type\n'
            message += bundle.data['Type'] + '\n'
        elif bundle.data['Type'] == 'Notification':
            message += 'Message\n'
            message += bundle.data['Message'] + '\n'
            message += 'MessageId\n'
            message += bundle.data['MessageId'] + '\n'
            if bundle.data['Subject'] != '':
                message += 'Subject\n'
                message += bundle.data['Subject'] + '\n'
            message += 'Timestamp\n'
            message += bundle.data['Timestamp'] + '\n'
            message += 'TopicArn\n'
            message += bundle.data['TopicArn'] + '\n'
            message += 'Type\n'
            message += bundle.data['Type'] + '\n'
        else:
            return False

        o = urlparse(bundle.data['SigningCertURL'])
        conn = httplib.HTTPSConnection(o.hostname)
        conn.putrequest('GET', o.path)
        conn.endheaders()
        response = conn.getresponse()
        cert = response.read()

        # ok; attempt to use m2crypto failed, using openssl command line tool instead

        file_cert = tempfile.NamedTemporaryFile(mode='w', delete=False)
        file_sig = tempfile.NamedTemporaryFile(mode='w', delete=False)
        file_mess = tempfile.NamedTemporaryFile(mode='w', delete=False)

        file_cert.write(cert)
        file_sig.write(bundle.data['Signature'])
        file_mess.write(message)

        file_cert.close()
        file_sig.close()
        file_mess.close()

        # see: https://async.fi/2011/10/sns-verify-sh/
        verify_process = subprocess.Popen(['/usr/local/bin/sns-verify.sh', file_cert.name, file_sig.name, file_mess.name], shell=False)
        verify_process.wait()

        if verify_process.returncode == 0:
            return True

        return False

That process.py would be something like:

#!/usr/bin/env python

import boto.sqs
from boto.sqs.message import Message
import base64, json, os, sys, syslog

if len(sys.argv) != 2:
    sys.exit('usage: %s <base64 encoded json object>' % (sys.argv[0], ))

m = json.loads(base64.b64decode(sys.argv[1]))

# http://code.activestate.com/recipes/66012-fork-a-daemon-process-on-unix/
try:
    pid = os.fork()
    if pid > 0:
        sys.exit(0)
except OSError, e:
    print >>sys.stderr, "fork #1 failed: %d (%s)" % (e.errno, e.strerror)
    sys.exit(1)

os.chdir("/")
os.setsid()
os.umask(0)

try:
    pid = os.fork()
    if pid > 0:
        sys.exit(0)
except OSError, e:
    sys.exit(1)

syslog.syslog(sys.argv[0] + ': ' + str(m))

# ...

That is, process.py gets the received (and doped) SQS message, Base64 encoded, as it's only command line argument, forks, exits and does what it's supposed to do after that on its own. Control returns to NotificationResource so the request doesn't block unnecessarily.

Tagged with:

Categorised as:

sns-verify.sh

#!/bin/sh

if [ $# -lt 3 ]; then
  echo "usage: sns-verify.sh CERT SIG MESS"
  exit 1
fi

CERT=$1
SIG=$2
MESS=$3

PUB=`/bin/tempfile`
SIGRAW=`/bin/tempfile`

# http://sns-public-resources.s3.amazonaws.com/SNS_Message_Signing_Release_Note_Jan_25_2011.pdf
/usr/bin/openssl x509 -in $CERT -pubkey -noout > $PUB
/usr/bin/base64 -i -d $SIG > $SIGRAW
RET=`/usr/bin/openssl dgst -sha1 -verify $PUB -signature $SIGRAW $MESS`

if [ X"$RET" = X"Verified OK" ]; then
  exit 0
fi

exit 1

Tagged with:

Categorised as:

Gone CloudFlare

Enabled CloudFlare on this site, with nearly every optimization thing they offer. So far it's looking good, with an empty (browser) cache it takes a moment to load initial resources but after that subsequent page loads are near-instantaneous (click around to try this out). To get SSL properly working you have to get a Pro account. Recommended!

Update: Getting fishy numbers with Pingdom (over 2000 ms), although page load times from my own machines are ok (around 500 ms or so). Investigating…

Tagged with:

Categorised as: