TileStache revealed itself as a powerful and flexible product and was very good to cache tiles generated with my Django-backed mapscript-powered WMS server. Caching and serving tiles with TileStache was by far faster then running the whole mapscript WMS machinery.

But I wanted to do more: if I could just convince nginx to serve the tiles directly from disk, this would have been even faster. Of course for this to work, TileStache must be configured to cache tiles on disk, which is what you normally want.

nginx is really good at serving static files and it is luckily wery flexible and easy to configure (once you know what to do).

The following configuration uses named regular expression captures and a try_files instruction to check for the tile existance, if found, the tile is served directly, if not found the internal redirect to the @django named location takes place.
The 404 trick in the base location (/) was necessary to avoid duplication of the @django location content, but there might be a better way.

nginx configuration file

server {
    listen   88 default;
    server_name  localhost;
    access_log  /var/log/nginx/access.log;
    rewrite  "^/[a-z]{2}/(static|media)/(.*)$"  /$1/$2;
    root /www/django_app/;
 
    location  /media/ {
        alias /www/django/contrib/admin/media/;
    }
 
    location  /static/ {
        alias /www/django_app/site_media/;
    }
 
    location ~ ^/(?<lang_code>..)/resource/tiles/1\.0\.0/(?<tile_path>.*)$ {
        alias /tmp/stache/;
        add_header X-Static super;
        try_files $http_host/$lang_code/$tile_path @django;
    }
 
    location / {
        add_header X-Static hit;
        error_page 404 = @django;
        return 404;
    }
 
    location @django {
        add_header X-Static miss;
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $http_host;
    }
}

Results

niginx is listening on port 88 and the dev server (I’m using werkzeug in this case) is listening on port 8080.

ab -c 100 -n 1000 http://localhost:88/it/resource/tiles/1.0.0/track_4/0/0/0.png
 
Concurrency Level:      100
Time taken for tests:   0.084 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      7553744 bytes
HTML transferred:       7302912 bytes
Requests per second:    11902.92 [#/sec] (mean)
Time per request:       8.401 [ms] (mean)
Time per request:       0.084 [ms] (mean, across all concurrent requests)
Transfer rate:          87804.31 [Kbytes/sec] received

With TileStache only:

ab -c 100 -n 1000 http://localhost:8000/track_4/0/0/0.png
 
Concurrency Level:      100
Time taken for tests:   2.944 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Non-2xx responses:      1000
Total transferred:      145000 bytes
HTML transferred:       0 bytes
Requests per second:    339.63 [#/sec] (mean)
Time per request:       294.438 [ms] (mean)
Time per request:       2.944 [ms] (mean, across all concurrent requests)
Transfer rate:          48.09 [Kbytes/sec] received

Conclusions

Unsurprisingly, the solution with files served directly is several times faster: 35 in this test case.

In a production environment, using mod_wsgi or uwsgi, the fugures may vary, but I’m confident that the overall speed boost is guaranteed.

• part 1: Polymaps, TileStache and SVG CSS styling issues
• part 2: TileStache Django tiny wrapper

during my experiments I wanted to reach the goal of minimize both the bandwith consuption and the number of vector coordinates that the browser needs to handle, this means that if we need to show complex geometries, for example a set of mountain hiking paths for an overall of 250000 point, we absolutely need a way to simplify features at lower zoom levels.

TileStache doesn’t support this out-of-the box, so I started playing with a clone of the Vector provider and ended up with a vector-simplify branch in my fork.

What I’ve done was basically to add a new “simplify” provider configuration parameter, where you can enter the tolerance for Douglas-Peucker simplification algorhitm in pixel unit. The paramer can be set in TileStache config file as shown here:

  "layers": {
    "pmcollection":
    {
        "provider":
        {
            "name": "vector",
            "driver": "PostgreSQL",
            "parameters": {
                "dbname": "polymaps_test",
                "user": "xxx",
                "port" : "5433",                
                "table": "pm_points_pmcollection"
            },
            "verbose" : "true",
            "simplify" : 0.5
        },
        "allowed origin" : "*"
    }
 }

A picture is worth a thousand words

Some pictures might be the best way to show how this parameter affects the generated geometries:

This is a single tile, the original geometry is in red, the simplified geometry in green/yellow and an over-simplified geometry in blue.

The following overall views show all the details:

Original geometry

Over-simplified with tolerance 50

Simplified with tolerance 0.5

Open issues

The biggest issue I’ve encountered was about getting the tolerance in lat-lon WGS84 starting from pixel units, but let me explain why I choose pixel units in the first place: I think that pixel units are are the best choice because normally we don’t care about feature details which are below 1 pixel, we simply cannot see them. Using pixel units we also get the advantage of using a single number for all zoom levels, a more complicated approach would consist in using separate values for each zoom level. Translating pixel units to decimal degrees is not possible without approximations, I just hope that my approach works in most situations.

Another strange thing I noted in the library is that the bounding box used for clipping (which is a rectangle) is built using 16 points instead of 4. This means that after clipping we end up with a geometry that have many more points that it needs. I’m still waiting for a clarification I’ve asked in the TileStache ML, but in the meantime I patched my branch to use a normal 4 points rectangle and everything seems to work fine (and with smaller geojson responses).

Conclusions

TileStache confirmed to be a valuable tool for web GIS developers, it’s well written and supported and easily expandable, absolutely worths a try.

First, I would like to thank Michal for the quick patch inclusion, the Polymaps patch is still waiting for the pull and the Django admin patch is awating for somebody to take care of that 18-months-old bug.

My experiments have continued with Django integration, I just wanted to see how easy it was (and it was really easy indeed). There aren’t any clear advantages using a Django app instead of the standalone TileStache server, but writing the wrapper was a nice exercise to learn something more about TileStache and Polymaps, and having all the pieces into one place is always tempting for me (not to mention that you can dribble the CORS completely).

The wapper consists in a single router rule in urlconf:

    url(r'^tiles/(?P<layer_name>[^/]+)/(?P<z>[^/]+)/(?P<x>[^/]+)/(?P<y>[^/]+)\.(?P<extension>.+)$', 'pm_points.views.tiles', name='tiles_url'),

… a view for the tiles (get_config implements a singleton to avoid parsing TileStache config every time):

 
# proper imports go here...
 
def tiles(request, layer_name, z, x, y, extension):
    """
    Proxy to tilestache
    {X} - coordinate column.
    {Y} - coordinate row.
    {B} - bounding box.
    {Z} - zoom level.
    {S} - host.
    """
    config = get_config()
    path_info = "%s/%s/%s/%s.%s" % (layer_name, z, x, y, extension)
    coord, extension = TileStache.splitPathInfo(path_info)[1:]
    mimetype, content = TileStache.getTile(config.layers[layer_name], coord, extension)
    return HttpResponse(content, mimetype=mimetype)

… and a simple view for the map:

 
def home(request):
    """
    Pass layers by
    """
    config = get_config()
    layer_urls = []
    for l in config.layers:
        if isinstance(config.layers[l].provider, TileStache.Providers.Vector.Provider):
            layer_urls.append(reverse('tiles_url', args=[l, '{Z}', '{X}', '{Y}', 'geojson']).replace('%7B', '{').replace('%7D', '}'))
    return render_to_response('home.html', { 'layer_urls' :  layer_urls}, context_instance=RequestContext(request))

… and a template for Polymaps (I paste the js part only):

 
    var po = org.polymaps,
        map,
        layers = [],
        stylist;
        jQuery(function(){
            map = po.map()
                .container(document.getElementById("map").appendChild(po.svg("svg")))
                .center({lat: 40, lon: 0})
                .zoomRange([1, 16])
                .zoom(2)
                .add(po.interact())
                .add(po.hash())
                ;
 
            map.add(po.image()
                .url("http://s3.amazonaws.com/com.modestmaps.bluemarble/{Z}-r{Y}-c{X}.jpg"));
 
            stylist = po.stylist()
                .attr('class', function(d){
                    return d.geometry.type + '_' + d.properties.id;
                })
                .title(function(d){return d.properties.name;})
                ;
 
            {% for lay_url in layer_urls %}
            var layer = po.geoJson()
                .url("{{ lay_url|safe }}")
                .zoom(function(z){
                    return z;
                })
                .id('pmcollection')
                .on("load",  stylist)
                .on('show', stylist)
                ;
            map.add(layer);
            layers.push(layer);
            {% endfor %}
 
            map.add(po.compass()
                .pan("none"));
            map.container().setAttribute("class", "mymap");
        });

Quick and dirty, it also dynamically adds layers to the Polymap.

This story continues with: TileStache Vector simplify provider

The final goal of this experiments was the integration with Django.

Polymaps

Polymaps is a Javascript library aimed to display svg maps in a web page, recently I’ve watched to an interesting talk about a Django-powered app built on top of Polymaps:

The main reason to give Polymaps a try is the interesting support to tiled vector layers, as far as I know a unique feature.

I’ve found the documentation of Polymaps a bit obscure and still incomplete (I just hope it isn’t another facade Open Source product where the lack of documentation and community support limits the use to commercial projects).

A quick look in the code shows no comments and rather obscure variable names, debbugging this library would certainly be a nightmare if you’re not the author, if you compare the source code with OpenLayers’s code you get the idea.

BTW I managed to make it work in a few hours (with all the server stuff).

TileStache

TileStache is a python library aimed to build, cache and manage map tiles. It’s similar to TileCache in scope and uses the well known Mapnik renderer while the data provider is implemented with shapely and OGR and can read the most widely used formats (postgis, shapefile, geojson).

TileStache is the ideal companion to Polymaps because it can handle and cache vector tiles (geojson) through the Vector class (don’t even try to use the PostgreSQL class you can find in Goodies, it’s kind of unofficially deprecated and it will let you waste your time).

I had to fork the project on github and made a pull request to fix the missing “port” parameter to the DB connection since I’m using a not-standard port.

SVG CSS

I admit I am a true beginner with SVG and while I was trying to experiment with the stylist features of Polymaps  I discovered some counter-intuitive (at least for me, coming from XHTML CSS) behaviours:

You can define styles in three ways:

1. CSS rules
2. “style” attribute of the SVG element
3. individual attributes (“stroke”, “fill” etc. etc.) of the SVG element

Strange enough, the priority is top-down, while I would have expected that point 3 had a higher priority.

Another thing worths noting is that what distinguish a polygon from a linestring in the Polymaps generated “path” SVG element is just that the “d” element (which contains the geometry) ends with a “Z”.

This is a linestring:

<path d="M101.99999715555555,189.9999971369224
L75.00000142222234,219.00000190473258
L75.00000142222234,242.49999834108655
L94.49999928888906,242.99999806558844
L102.49999928888906,255.4999976705726
L101.99970702222231,256"/>;

This is a polygon:

<path d="M95.07080533333351,256.00000052462633
L0,196.67516186124612
L0,192.0000022835775
L0,128.0000001888236L0,70.92093168072711
L175.92470186666674,0.000006325172648757871
L192,0.000006325172648757871
L256,0.000006325172648757871
L256,64.00000444302805
L256,128.0000001888236
L256,160.99230434784874
L113.66593991111131,256.00000052462633Z"/>;

Styling the plygon fill without hitting linestring in a GEOMETRYCOLLECTION layer turned out to be quite troublesome.

I ended up with a CSS-3 attribute CSS rule:

.layer path {
fill: cyan;
fill-opacity: .85;
stroke: #012;
stroke-linecap: round;
}
 
.layer circle {
fill: cyan;
fill-opacity: .85;
stroke: #012;
}
 
/* Don't fill in SVG */
.layer path:not([d$="Z"]){
fill: none;
stroke-width: 3px;
}

I wonder if there is a better way…

This story continues with: TileStache Django tiny wrapper.

Zuerich QGIS hackmeeting

Last november I participated to the international QGIS programmer’s meeting in Zuerich. It was a great opportunity to meet the QGIS developers I already knew from the previous meeting in Lisbon, and to know new interesting people.

QGIS is quickly becoming one of the most important free software packages for GIS processing, I participate to the team as the lead developer of the new python plugins web application, realized with the Django framework.

During the meeting I worked on fixing bugs and I implemented some new features the most important being the new XML-RPC interface that allows to programmatically upload plugins.

The newXML-RPC  interface has been realized with RPC4Django, a cool package that allows for quick webservice creation in Django.

Looking forward to the next hackmeeting in France I wish to thank the Italian Geographic Free Open-Source Software Association for the financial aids provided.

Last week I challenged my first migration from Joomla 1.5 to 1.7. Jupgrade is a wonderful tool, it worked pretty well but leaved the categories tree in an unuseable state and made a big mess in the article’s ACL.

The symptoms of the two problems are:

1. some categories (non only in articles but also in weblinks and contacts) are not shown in the articles category drop-down list, a category tree rebuild from the control panel doesn’t solve the problem
2. users of levels different than superadminstrator cannot edit and save articles created from other users, apparently the ACL should allow this because they are set to “inherit” and the action should be allowed by the upstream ACL.

At first I managed to solve the tree problem with some SQL but this was not enough to solve the second issue: most articles (hundreds of them) still had wrong ACL set preventing administrator level users to edit or save other’s articles.

Opening and saving each single article when logged in as superadministrator solved the issue, but doing this boring procedure with hundreds of articles and having many other sites to migrate the most obvious solution was to write a small component to fix those issues.

The component performs two tasks:

1. rebuild the category tree, assigning correct depth and parent
2. reset all category and article’s ACLs to default (inherited)

No warranty and no assistance are available but if it helped you to solve a problem and to save time and money, a donation of any amount would be appreciated.

Download the component com_fixperm.

The suite was born to comply with the Italian regulation regarding web sites managed by public administrations, they must be not only accessible but also follow other technical requirements (like doctype XHTML 1.0 strict, to say one).

The project is maintained on github , all the code is released under AGPL (Affero GPL) license.

Posizioni del template Joomla! FAP 1.7

You can just right-click and save it on your bookmarks, the click on the bookmark while in the OpenCycleMap site and a popup will appear with the extent coordinates printed on screen.

Bookmarklet.

Strangely enough there is no way to download an uncompressed version from the project website, even if they provide a nice custom build form.

The recommended way is to use “jsbuild“, which turned to be a nightmare because it does not run out-of-the-box, at least on my Ubuntu lucid.

A one-liner did the trick, run it from a “build” folder under GeoExt main folder, the correct order of inclusion (which does really matter) is taken from the “single” file GeoExt.js.

for f in $(egrep '"GeoExt/' ../lib/GeoExt.js|sed -e 's/"//g'|sed -e 's/,//g') ; do cat "../lib/$f" ; done > GeoExt-debug.js

Il corso, gratuito e aperto a tutti, si svolgerà in due incontri: sabato 8 ottobre 2011 e sabato 22 ottobre 2011 dalle 14:30 alle 17:30 presso il Centro Studi e Museo d’Arte Preistorica, via Brignone 9 a Pinerolo.

Il secondo incontro, del 22 ottobre, sarà inserito nell’ambito della programmazione dell’edizione pinerolese dell’evento nazionale “Linux Day 2011″, che si terrà il 22 ottobre 2011 dalle 9 alle 18, sempre presso il Museo.

Per informazioni e iscrizioni (i posti disponibili sono limitati), invia un messaggio email a

info@pinerolo.linux.it

Sito dell’associazione: http://www.pinerolo.linux.it