Async.fi

Gone DigitalOcean

In order to try something new (and save money) I switched my personal "utility" server from EC2 to DigitalOcean. As I was running just a t1.micro instance anyway, I selected DigitalOcean's cheapest offering, which costs $5/month (512MB/20GB(SSD)/1TB). Plus $1/month for regular backups (they somehow backup the whole server while the server is running, so I'm not sure how consistent said backup can possibly be but at least it's cheap).

DigitalOcean is offering just servers, and the whole experience isn't of course quite as polished as with Amazon. Nor can you, say, access S3 (or other services) at quite the same speed as when in Amazon's network. But for my use case it's a good fit.

Tagged with:

Categorised as:

Amazon DynamoDB basics with Boto

The code assumes that Boto credentials have been set up. 

      import boto.dynamodb
      from boto.dynamodb.condition import *
      
      connection =  boto.dynamodb.connect_to_region('eu-west-1')
      table = connection.get_table('table')
      
      id = '1'
      timestamp = 1234
      attrs = {
        'key1': 'value1',
        'key2': set(['value2', 'value3'])
      }
      
      # create
      item = table.new_item(hash_key=id, range_key=timestamp, attrs=attrs)
      item.put()
      
      # read
      item = table.get_item(hash_key=id)      
      key2 = list(item['key2'])
      
      # update
      item['key1'] = 'foo'
      item['key3'] = 'bar'
      item.put()
      
      # query
      table.query(hash_key=id, range_key_condition=LT(1500))
      
      # scan
      table.scan(scan_filter={'key1': EQ('foo')})
      
      # delete
      item = table.get_item(hash_key=id)
      item.delete()

Tagged with:

Categorised as:

Serving per IAM user S3 data

This isn't meant for public facing web, but a closed environment where it is necessary that each client is individually addressable (common application code, individual data). Each client has a local web server plus locally stored AWS credentials, and can therefore be fed content specific to each client. The bootstrap script is minimalistic by design, with as little moving parts as possible.

AWS credentials file (init.json below): 

    init({
      "region": "eu-west-1",
      "common_bucket": "loadres",
      "private_bucket": "697ad820240c48929dce15c25cee8591",
      "access_key": "AKIAILZCSDJEFUN3L53Q",
      "secret_key": "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"
    })

index.html: 

    <!DOCTYPE html>
    <html>
      <head>
        <meta charset="utf-8">
        <title>loadres</title>
        <script src="sha1.js"></script> <!-- https://github.com/lmorchard/S3Ajax/blob/master/js/sha1.js -->
        <script>

          // the authenticated S3 URL maker function, without STS specific parts:
          // http://www.async.fi/2012/07/s3-query-string-authentication-and-aws-security-token-service/
          var s3url = function(region, bucket, key, access_key, secret_key) {
            var expires = Math.floor(((new Date()).getTime()/1000) + 3600);
            var string_to_sign = [
              'GET\n\n\n',
              expires, '\n',
              '/', bucket, '/', key
            ].join('');
            var signature = b64_hmac_sha1(secret_key, string_to_sign) + '=';
            var url = 'https://s3-' + region + '.amazonaws.com/' + bucket + '/' + key
              + '?AWSAccessKeyId=' + encodeURIComponent(access_key)
              + '&Signature=' + encodeURIComponent(signature)
              + '&Expires=' + expires;
            return url;
          };

          var init = function(settings) {
            var head = document.getElementsByTagName('head')[0];

            // inject prod.css
            var css = document.createElement('link');
            css.setAttribute('rel', 'stylesheet');
            css.setAttribute('href', s3url(settings['region'], settings['common_bucket'], 'prod.css', settings['access_key'], settings['secret_key']));
            head.appendChild(css);

            // inject prod.js
            var js = document.createElement('script');
            js.setAttribute('src', s3url(settings['region'], settings['common_bucket'], 'prod.js', settings['access_key'], settings['secret_key']));
            head.appendChild(js);
          }
        </script>

        <!-- load AWS region and bucket info, plus credentials; this script calls init() (above) -->
        <script src="init.json"></script>
      </head>
      <body></body>
    </html>

Now in the loaded prod.js file we would bring in the application code that would fetch data specific to this client (a little repetition here): 

    var expires = Math.floor(((new Date()).getTime()/1000) + 3600);
    var string_to_sign = [
      'GET\n\n\n',
      expires, '\n',
      '/', settings['private_bucket'], '/', 'data.txt'
    ].join('');
    var signature = b64_hmac_sha1(settings['secret_key'], string_to_sign) + '=';
    var url = '/' + settings['private_bucket'] + '/' + 'data.txt'
      + '?AWSAccessKeyId=' + encodeURIComponent(settings['access_key'])
      + '&Signature=' + encodeURIComponent(signature)
      + '&Expires=' + expires;

    var r = new XMLHttpRequest();
    r.open('GET', url, true);
    r.onreadystatechange = function () {
      if(r.readyState != 4 || r.status != 200) return;
      alert("Success: " + r.responseText);
    };
    r.send();

To make this work without CORS, we're using a local proxy to handle S3 requests. In Nginx config: 

    location /697ad820240c48929dce15c25cee8591 {
      rewrite  ^//697ad820240c48929dce15c25cee8591/(.*)$ /$1 break;
      proxy_pass https://s3-eu-west-1.amazonaws.com/697ad820240c48929dce15c25cee8591;
    }

Tagged with:

Categorised as:

Gone static

Update 2: Looks like page load times, at least as reported by Pingdom, went up from what they initially were. In my own testing cached pages still load in something like 150 to 250 milliseconds but Pingdom disagees. I don't know if this is regular CloudFront performance fluctuation, some kind of impedance mismatch between Pingdom and CloudFront or "something else".

Update: That really did the trick and the estimated -90% page load time wasn't that far off:

Having been fed up with wastefulness (resource wise) and general slowness of the MySQL/PHP/WordPress/CloudFlare setup for some time, I have now moved this site to S3/CloudFront. Site is generated from an XML file (which I derived from a WordPress export dump) with a Python script that is hosted here. Commenting is obviously impossible but if you for some reason need to contact me you'll find contact details on your left.

Tagged with:

Categorised as:

Mikrotik OpenVPN Server

The purpose of this post is to describe, step by step, my attempt to set up an OpenVPN server on a Mikrotik RouterBOARD 750 and create a working tunnel from an outside machine (AWS EC2 Windows Server 2008 R2) to this OpenVPN server so that an SMB server on the local network can be accessed from said outside machine. The following diagram gives an overview of the setup:

I am going to decribe how to:

  • generate certificates to be used with OpenVPN
  • set up OpenVPN server on Mikrotik router
  • set up a tunnel with OpenVPN client on Windows
I am not going to describe the following:
  • setting up and connecting to an EC2 Windows instance
  • setting up a Samba Server
A few things worth mentioning about Mikrotik OpenVPN server implementation (that will likely bite if not known in advance):
  • only supports TCP mode, UDP is not supported
  • username/password pair is also required even though certificates are being used for authentication

Generate certificates to be used with OpenVPN

root@inhouse-debian:~# apt-get install openvpn
root@inhouse-debian:~# mkdir ovpn-cert
root@inhouse-debian:~# cd ovpn-cert/
root@inhouse-debian:~/ovpn-cert# cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0/* .
root@inhouse-debian:~/ovpn-cert# emacs vars
In the file vars I set the following values: 
export KEY_COUNTRY="FI"
export KEY_PROVINCE="Etela-Suomi"
export KEY_CITY="Kotka"
export KEY_ORG="Async.fi"
export KEY_EMAIL="joni.kahara@async.fi"
export KEY_CN="kahara.dyndns.org"
export KEY_NAME="kahara.dyndns.org"
export KEY_OU="kahara.dyndns.org"
If I have understood correctly, of these only CN (Common Name) is obligatory. I may be wrong. Anyway, continuing: 
root@inhouse-debian:~/ovpn-cert# source vars
root@inhouse-debian:~/ovpn-cert# ./clean-all
root@inhouse-debian:~/ovpn-cert# ./build-ca
root@inhouse-debian:~/ovpn-cert# ./build-key-server kahara.dyndns.org
root@inhouse-debian:~/ovpn-cert# openssl rsa -in keys/kahara.dyndns.org.key -out keys/kahara.dyndns.org.pem
root@inhouse-debian:~/ovpn-cert# ./build-key ec2 
root@inhouse-debian:~/ovpn-cert# apt-get install ncftp
root@inhouse-debian:~/ovpn-cert# ncftpput -u admin 192.168.1.1 / keys/kahara.dyndns.org.crt keys/kahara.dyndns.org.pem keys/ca.crt

Set up OpenVPN server on Mikrotik router

All the stuff here can also be made through Mikrotik's admin interface; textual form without screen shots is used just to keep thing terse. 
root@inhouse-debian:~/ovpn-cert# ssh admin@192.168.1.1
[admin@MikroTik] > /certificate
[admin@MikroTik] /certificate> import file=kahara.dyndns.org.crt
[admin@MikroTik] /certificate> import file=kahara.dyndns.org.pem
[admin@MikroTik] /certificate> import file=ca.crt
[admin@MikroTik] /certificate> decrypt
[admin@MikroTik] /certificate> ..
[admin@MikroTik] > /interface bridge add name=ovpn-bridge
[admin@MikroTik] > /interface bridge port add interface=ether2-master-local bridge=ovpn-bridge
[admin@MikroTik] > /ip address add address=192.168.1.64/24 interface=ovpn-bridge 
[admin@MikroTik] > /ip pool add name=ovpn-pool ranges=192.168.1.65-192.168.1.99
[admin@MikroTik] > /ppp profile add bridge=ovpn-bridge name=ovpn-profile remote-address=ovpn-pool
[admin@MikroTik] > /ppp secret add service=ovpn local-address=192.168.1.64 name=user1 password=pass1 profile=ovpn-profile
[admin@MikroTik] > /interface ovpn-server server set auth=sha1,md5 certificate=cert1 cipher=blowfish128,aes128,aes192,aes256 default-profile=ovpn-profile enabled=yes keepalive-timeout=disabled max-mtu=1500 mode=ethernet netmask=24 port=1194 require-client-certificate=yes
[admin@MikroTik] > /ip firewall filter add action=accept chain=input disabled=no protocol=tcp dst-port=1194
[admin@MikroTik] > /ip firewall filter move 5 destination=1
That last step moves the new rule to the front of the chain; numbers ("5", "1") will likely be something else on your configuration. Firewall rule listing can be printed with the following command: 
[admin@MikroTik] > /ip firewall filter print

Setup up a tunnel with OpenVPN client on Windows

After installing OpenVPN, create a config file for it. Here it's called "kahara.dyndns.org.ovpn": 
client
dev tap
proto tcp
remote kahara.dyndns.org 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert ec2.crt
key ec2.key
verb 3
pull
auth-user-pass userpass.txt
Also, create a file called "userpass.txt" and put the following to it: 
user1
pass1
Of course in an IRL situation one should use a real password. Make sure you copied the .crt and .key files over to the Windows machine, after which you can run OpenVPN client with: 
PS C:\Users\Administrator\Desktop> openvpn.exe .\kahara.dyndns.org.ovpn
And here we have an EC2 client connected to a local SMB resource over the tunnel:

Tagged with:

Categorised as: