Async.fi

Amazon DynamoDB basics with Boto

The code assumes that Boto credentials have been set up. 

      import boto.dynamodb
      from boto.dynamodb.condition import *
      
      connection =  boto.dynamodb.connect_to_region('eu-west-1')
      table = connection.get_table('table')
      
      id = '1'
      timestamp = 1234
      attrs = {
        'key1': 'value1',
        'key2': set(['value2', 'value3'])
      }
      
      # create
      item = table.new_item(hash_key=id, range_key=timestamp, attrs=attrs)
      item.put()
      
      # read
      item = table.get_item(hash_key=id)      
      key2 = list(item['key2'])
      
      # update
      item['key1'] = 'foo'
      item['key3'] = 'bar'
      item.put()
      
      # query
      table.query(hash_key=id, range_key_condition=LT(1500))
      
      # scan
      table.scan(scan_filter={'key1': EQ('foo')})
      
      # delete
      item = table.get_item(hash_key=id)
      item.delete()

Tagged with:

Categorised as:

Serving per IAM user S3 data

This isn't meant for public facing web, but a closed environment where it is necessary that each client is individually addressable (common application code, individual data). Each client has a local web server plus locally stored AWS credentials, and can therefore be fed content specific to each client. The bootstrap script is minimalistic by design, with as little moving parts as possible.

AWS credentials file (init.json below): 

    init({
      "region": "eu-west-1",
      "common_bucket": "loadres",
      "private_bucket": "697ad820240c48929dce15c25cee8591",
      "access_key": "AKIAILZCSDJEFUN3L53Q",
      "secret_key": "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"
    })

index.html: 

    <!DOCTYPE html>
    <html>
      <head>
        <meta charset="utf-8">
        <title>loadres</title>
        <script src="sha1.js"></script> <!-- https://github.com/lmorchard/S3Ajax/blob/master/js/sha1.js -->
        <script>

          // the authenticated S3 URL maker function, without STS specific parts:
          // http://www.async.fi/2012/07/s3-query-string-authentication-and-aws-security-token-service/
          var s3url = function(region, bucket, key, access_key, secret_key) {
            var expires = Math.floor(((new Date()).getTime()/1000) + 3600);
            var string_to_sign = [
              'GET\n\n\n',
              expires, '\n',
              '/', bucket, '/', key
            ].join('');
            var signature = b64_hmac_sha1(secret_key, string_to_sign) + '=';
            var url = 'https://s3-' + region + '.amazonaws.com/' + bucket + '/' + key
              + '?AWSAccessKeyId=' + encodeURIComponent(access_key)
              + '&Signature=' + encodeURIComponent(signature)
              + '&Expires=' + expires;
            return url;
          };

          var init = function(settings) {
            var head = document.getElementsByTagName('head')[0];

            // inject prod.css
            var css = document.createElement('link');
            css.setAttribute('rel', 'stylesheet');
            css.setAttribute('href', s3url(settings['region'], settings['common_bucket'], 'prod.css', settings['access_key'], settings['secret_key']));
            head.appendChild(css);

            // inject prod.js
            var js = document.createElement('script');
            js.setAttribute('src', s3url(settings['region'], settings['common_bucket'], 'prod.js', settings['access_key'], settings['secret_key']));
            head.appendChild(js);
          }
        </script>

        <!-- load AWS region and bucket info, plus credentials; this script calls init() (above) -->
        <script src="init.json"></script>
      </head>
      <body></body>
    </html>

Now in the loaded prod.js file we would bring in the application code that would fetch data specific to this client (a little repetition here): 

    var expires = Math.floor(((new Date()).getTime()/1000) + 3600);
    var string_to_sign = [
      'GET\n\n\n',
      expires, '\n',
      '/', settings['private_bucket'], '/', 'data.txt'
    ].join('');
    var signature = b64_hmac_sha1(settings['secret_key'], string_to_sign) + '=';
    var url = '/' + settings['private_bucket'] + '/' + 'data.txt'
      + '?AWSAccessKeyId=' + encodeURIComponent(settings['access_key'])
      + '&Signature=' + encodeURIComponent(signature)
      + '&Expires=' + expires;

    var r = new XMLHttpRequest();
    r.open('GET', url, true);
    r.onreadystatechange = function () {
      if(r.readyState != 4 || r.status != 200) return;
      alert("Success: " + r.responseText);
    };
    r.send();

To make this work without CORS, we're using a local proxy to handle S3 requests. In Nginx config: 

    location /697ad820240c48929dce15c25cee8591 {
      rewrite  ^//697ad820240c48929dce15c25cee8591/(.*)$ /$1 break;
      proxy_pass https://s3-eu-west-1.amazonaws.com/697ad820240c48929dce15c25cee8591;
    }

Tagged with:

Categorised as:

Berlin/August 2012

Here are some bits of information that I think might be useful for others travelling to this destination.

Fonic prepaid

I arranged, beforehand, two Fonic Prepaid SIMs from this guy. Please note that while during the initial activation at fonic.de you can enter any German address in your details (I gave our rented apartment address) they will send a letter to that address and if the letter returns as non-deliverable (as happened in our case) then the prepaid will be put in “nur erreichbar” state. This means that you can receive calls and SMS, but can not place calls or send SMS or activate any of the flat rate Internet options. After contacting Fonic support, they asked for the original PIN numbers printed in SIM packaging, and also a copy (scan) of my Finnish passport. After this both SIMs were usable once again and I was able to activate the “Tagesflat” Internet option while in Germany.

In addition to top-up vouchers, transferring money to your prepaid account is possible with IBAN bank transfer; see instructions here.

7-Tage-Karte

These 7 day public transport tickets can be purchased from machines in S-Bahn/U-Bahn stations and bus/tram stops, or from a Spätkauf. As far as I know, the machines only accept cash. We got these for all zones (i.e. ABC which covers Berlin and some neighbouring towns) which cost a little over 30 euros per week.

Berlin public transportation works excellently and we took a cab only occasionally (cab rides were reasonably priced too, though).

Öffi Directions

This Android app will, given two stations (or addresses), present you with a selection of best routes using any mix of the above mentioned means of transportation, often in combinations you wouldn’t have thought of by just looking at the route maps and time tables. Can not be recommended enough.

Places to visit

While in no way an exhaustive list, here are some places we went to and that I find worth mentioning.

Teufelsberg

The now abandoned joint NSA/GCHQ listening station. There was some dude claiming to represent the current owner at the gate collecting a five euro admission per person and requiring our signatures in order to avoid any legal hassles should we hurt ourselves during our exploration.

There were people in the actual radomes but we just wandered around the site as climbing didn’t seem like the thing to do at the time. There is a staircase inside the main building leading up but the door leading to that staircase was locked. The key to that lock is supposedly held by the people arranging guided tours of the site.

Fernsehturm

You should book tickets in advance, this way you’re guaranteed the next free window table at the restaurant above the observation deck (we were there on a saturday night at around eight and the place was pretty packed and we had to wait for the table for around 15 minutes). The restaurant is expensive-ish but ok, though the food is nothing to write home about. But then again people come here for the view. Old-school, professional waitresses and live music.

Sightseeing flight with Manuel

I spotted this activity from Gidsy. Our pilot Manuel flew us from the Bienenfarm field on a small Cessna plane. Good photos.

Berliner Unterwelten

These tours of the underground Berlin are arranged by a non-profit organisation. We took the "Cold War” tour which took us first to a big war-era bomb shelter near Gesundbrunnen station and then to a modern facility at Pankstraße station. Both facilities are of course decommisioned (there wouldn't be tours if they weren't) but the guide (a Brit) gave a rather lively presentation about life in the facilities if they had ever been put to use, and the effects of nuclear war in general.

Türkenmarkt

Tuesdays and fridays. Looking at the prices, you could easily stock a weeks supply of vegetables here for ten euros. At the south-east end of the market there is a fresh pasta and gnocchi vendor with an excellent product (same vendor at Hackescher Markt on saturdays).

Tagged with:

Categorised as:

S3 query string authentication and AWS Security Token Service

Getting this right took some tweaking, so:

// http://docs.amazonwebservices.com/AmazonS3/latest/dev/RESTAuthentication.html#RESTAuthenticationQueryStringAuth
// http://docs.amazonwebservices.com/STS/latest/APIReference/Welcome.html

var access_key = '…', secret_key = '…', session_token = '…';

var expires = Math.floor(((new Date()).getTime()/1000) + 3600);
var string_to_sign = [
    'GET\n\n\n',
    expires, '\n',
    'x-amz-security-token:', session_token, '\n',
    '/', bucket, '/', key
].join('');

// https://github.com/lmorchard/S3Ajax/blob/master/js/sha1.js
var signature = b64_hmac_sha1(secret_key, string_to_sign) + '=';

var url = key
    + '?AWSAccessKeyId=' + encodeURIComponent(access_key)
    + '&Signature=' + encodeURIComponent(signature)
    + '&Expires=' + expires
    + '&x-amz-security-token=' + encodeURIComponent(session_token);

Tagged with:

Categorised as: