TileStache revealed itself as a powerful and flexible product and was very good to cache tiles generated with my Django-backed mapscript-powered WMS server. Caching and serving tiles with TileStache was by far faster then running the whole mapscript WMS machinery.

But I wanted to do more: if I could just convince nginx to serve the tiles directly from disk, this would have been even faster. Of course for this to work, TileStache must be configured to cache tiles on disk, which is what you normally want.

nginx is really good at serving static files and it is luckily wery flexible and easy to configure (once you know what to do).

The following configuration uses named regular expression captures and a try_files instruction to check for the tile existance, if found, the tile is served directly, if not found the internal redirect to the @django named location takes place.
The 404 trick in the base location (/) was necessary to avoid duplication of the @django location content, but there might be a better way.

nginx configuration file

server {
    listen   88 default;
    server_name  localhost;
    access_log  /var/log/nginx/access.log;
    rewrite  "^/[a-z]{2}/(static|media)/(.*)$"  /$1/$2;
    root /www/django_app/;
 
    location  /media/ {
        alias /www/django/contrib/admin/media/;
    }
 
    location  /static/ {
        alias /www/django_app/site_media/;
    }
 
    location ~ ^/(?<lang_code>..)/resource/tiles/1\.0\.0/(?<tile_path>.*)$ {
        alias /tmp/stache/;
        add_header X-Static super;
        try_files $http_host/$lang_code/$tile_path @django;
    }
 
    location / {
        add_header X-Static hit;
        error_page 404 = @django;
        return 404;
    }
 
    location @django {
        add_header X-Static miss;
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $http_host;
    }
}

Results

niginx is listening on port 88 and the dev server (I’m using werkzeug in this case) is listening on port 8080.

ab -c 100 -n 1000 http://localhost:88/it/resource/tiles/1.0.0/track_4/0/0/0.png
 
Concurrency Level:      100
Time taken for tests:   0.084 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      7553744 bytes
HTML transferred:       7302912 bytes
Requests per second:    11902.92 [#/sec] (mean)
Time per request:       8.401 [ms] (mean)
Time per request:       0.084 [ms] (mean, across all concurrent requests)
Transfer rate:          87804.31 [Kbytes/sec] received

With TileStache only:

ab -c 100 -n 1000 http://localhost:8000/track_4/0/0/0.png
 
Concurrency Level:      100
Time taken for tests:   2.944 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Non-2xx responses:      1000
Total transferred:      145000 bytes
HTML transferred:       0 bytes
Requests per second:    339.63 [#/sec] (mean)
Time per request:       294.438 [ms] (mean)
Time per request:       2.944 [ms] (mean, across all concurrent requests)
Transfer rate:          48.09 [Kbytes/sec] received

Conclusions

Unsurprisingly, the solution with files served directly is several times faster: 35 in this test case.

In a production environment, using mod_wsgi or uwsgi, the fugures may vary, but I’m confident that the overall speed boost is guaranteed.

• part 1: Polymaps, TileStache and SVG CSS styling issues
• part 2: TileStache Django tiny wrapper

during my experiments I wanted to reach the goal of minimize both the bandwith consuption and the number of vector coordinates that the browser needs to handle, this means that if we need to show complex geometries, for example a set of mountain hiking paths for an overall of 250000 point, we absolutely need a way to simplify features at lower zoom levels.

TileStache doesn’t support this out-of-the box, so I started playing with a clone of the Vector provider and ended up with a vector-simplify branch in my fork.

What I’ve done was basically to add a new “simplify” provider configuration parameter, where you can enter the tolerance for Douglas-Peucker simplification algorhitm in pixel unit. The paramer can be set in TileStache config file as shown here:

  "layers": {
    "pmcollection":
    {
        "provider":
        {
            "name": "vector",
            "driver": "PostgreSQL",
            "parameters": {
                "dbname": "polymaps_test",
                "user": "xxx",
                "port" : "5433",                
                "table": "pm_points_pmcollection"
            },
            "verbose" : "true",
            "simplify" : 0.5
        },
        "allowed origin" : "*"
    }
 }

A picture is worth a thousand words

Some pictures might be the best way to show how this parameter affects the generated geometries:

This is a single tile, the original geometry is in red, the simplified geometry in green/yellow and an over-simplified geometry in blue.

The following overall views show all the details:

Original geometry

Over-simplified with tolerance 50

Simplified with tolerance 0.5

Open issues

The biggest issue I’ve encountered was about getting the tolerance in lat-lon WGS84 starting from pixel units, but let me explain why I choose pixel units in the first place: I think that pixel units are are the best choice because normally we don’t care about feature details which are below 1 pixel, we simply cannot see them. Using pixel units we also get the advantage of using a single number for all zoom levels, a more complicated approach would consist in using separate values for each zoom level. Translating pixel units to decimal degrees is not possible without approximations, I just hope that my approach works in most situations.

Another strange thing I noted in the library is that the bounding box used for clipping (which is a rectangle) is built using 16 points instead of 4. This means that after clipping we end up with a geometry that have many more points that it needs. I’m still waiting for a clarification I’ve asked in the TileStache ML, but in the meantime I patched my branch to use a normal 4 points rectangle and everything seems to work fine (and with smaller geojson responses).

Conclusions

TileStache confirmed to be a valuable tool for web GIS developers, it’s well written and supported and easily expandable, absolutely worths a try.

First, I would like to thank Michal for the quick patch inclusion, the Polymaps patch is still waiting for the pull and the Django admin patch is awating for somebody to take care of that 18-months-old bug.

My experiments have continued with Django integration, I just wanted to see how easy it was (and it was really easy indeed). There aren’t any clear advantages using a Django app instead of the standalone TileStache server, but writing the wrapper was a nice exercise to learn something more about TileStache and Polymaps, and having all the pieces into one place is always tempting for me (not to mention that you can dribble the CORS completely).

The wapper consists in a single router rule in urlconf:

    url(r'^tiles/(?P<layer_name>[^/]+)/(?P<z>[^/]+)/(?P<x>[^/]+)/(?P<y>[^/]+)\.(?P<extension>.+)$', 'pm_points.views.tiles', name='tiles_url'),

… a view for the tiles (get_config implements a singleton to avoid parsing TileStache config every time):

 
# proper imports go here...
 
def tiles(request, layer_name, z, x, y, extension):
    """
    Proxy to tilestache
    {X} - coordinate column.
    {Y} - coordinate row.
    {B} - bounding box.
    {Z} - zoom level.
    {S} - host.
    """
    config = get_config()
    path_info = "%s/%s/%s/%s.%s" % (layer_name, z, x, y, extension)
    coord, extension = TileStache.splitPathInfo(path_info)[1:]
    mimetype, content = TileStache.getTile(config.layers[layer_name], coord, extension)
    return HttpResponse(content, mimetype=mimetype)

… and a simple view for the map:

 
def home(request):
    """
    Pass layers by
    """
    config = get_config()
    layer_urls = []
    for l in config.layers:
        if isinstance(config.layers[l].provider, TileStache.Providers.Vector.Provider):
            layer_urls.append(reverse('tiles_url', args=[l, '{Z}', '{X}', '{Y}', 'geojson']).replace('%7B', '{').replace('%7D', '}'))
    return render_to_response('home.html', { 'layer_urls' :  layer_urls}, context_instance=RequestContext(request))

… and a template for Polymaps (I paste the js part only):

 
    var po = org.polymaps,
        map,
        layers = [],
        stylist;
        jQuery(function(){
            map = po.map()
                .container(document.getElementById("map").appendChild(po.svg("svg")))
                .center({lat: 40, lon: 0})
                .zoomRange([1, 16])
                .zoom(2)
                .add(po.interact())
                .add(po.hash())
                ;
 
            map.add(po.image()
                .url("http://s3.amazonaws.com/com.modestmaps.bluemarble/{Z}-r{Y}-c{X}.jpg"));
 
            stylist = po.stylist()
                .attr('class', function(d){
                    return d.geometry.type + '_' + d.properties.id;
                })
                .title(function(d){return d.properties.name;})
                ;
 
            {% for lay_url in layer_urls %}
            var layer = po.geoJson()
                .url("{{ lay_url|safe }}")
                .zoom(function(z){
                    return z;
                })
                .id('pmcollection')
                .on("load",  stylist)
                .on('show', stylist)
                ;
            map.add(layer);
            layers.push(layer);
            {% endfor %}
 
            map.add(po.compass()
                .pan("none"));
            map.container().setAttribute("class", "mymap");
        });

Quick and dirty, it also dynamically adds layers to the Polymap.

This story continues with: TileStache Vector simplify provider

The final goal of this experiments was the integration with Django.

Polymaps

Polymaps is a Javascript library aimed to display svg maps in a web page, recently I’ve watched to an interesting talk about a Django-powered app built on top of Polymaps:

The main reason to give Polymaps a try is the interesting support to tiled vector layers, as far as I know a unique feature.

I’ve found the documentation of Polymaps a bit obscure and still incomplete (I just hope it isn’t another facade Open Source product where the lack of documentation and community support limits the use to commercial projects).

A quick look in the code shows no comments and rather obscure variable names, debbugging this library would certainly be a nightmare if you’re not the author, if you compare the source code with OpenLayers’s code you get the idea.

BTW I managed to make it work in a few hours (with all the server stuff).

TileStache

TileStache is a python library aimed to build, cache and manage map tiles. It’s similar to TileCache in scope and uses the well known Mapnik renderer while the data provider is implemented with shapely and OGR and can read the most widely used formats (postgis, shapefile, geojson).

TileStache is the ideal companion to Polymaps because it can handle and cache vector tiles (geojson) through the Vector class (don’t even try to use the PostgreSQL class you can find in Goodies, it’s kind of unofficially deprecated and it will let you waste your time).

I had to fork the project on github and made a pull request to fix the missing “port” parameter to the DB connection since I’m using a not-standard port.

SVG CSS

I admit I am a true beginner with SVG and while I was trying to experiment with the stylist features of Polymaps  I discovered some counter-intuitive (at least for me, coming from XHTML CSS) behaviours:

You can define styles in three ways:

1. CSS rules
2. “style” attribute of the SVG element
3. individual attributes (“stroke”, “fill” etc. etc.) of the SVG element

Strange enough, the priority is top-down, while I would have expected that point 3 had a higher priority.

Another thing worths noting is that what distinguish a polygon from a linestring in the Polymaps generated “path” SVG element is just that the “d” element (which contains the geometry) ends with a “Z”.

This is a linestring:

<path d="M101.99999715555555,189.9999971369224
L75.00000142222234,219.00000190473258
L75.00000142222234,242.49999834108655
L94.49999928888906,242.99999806558844
L102.49999928888906,255.4999976705726
L101.99970702222231,256"/>;

This is a polygon:

<path d="M95.07080533333351,256.00000052462633
L0,196.67516186124612
L0,192.0000022835775
L0,128.0000001888236L0,70.92093168072711
L175.92470186666674,0.000006325172648757871
L192,0.000006325172648757871
L256,0.000006325172648757871
L256,64.00000444302805
L256,128.0000001888236
L256,160.99230434784874
L113.66593991111131,256.00000052462633Z"/>;

Styling the plygon fill without hitting linestring in a GEOMETRYCOLLECTION layer turned out to be quite troublesome.

I ended up with a CSS-3 attribute CSS rule:

.layer path {
fill: cyan;
fill-opacity: .85;
stroke: #012;
stroke-linecap: round;
}
 
.layer circle {
fill: cyan;
fill-opacity: .85;
stroke: #012;
}
 
/* Don't fill in SVG */
.layer path:not([d$="Z"]){
fill: none;
stroke-width: 3px;
}

I wonder if there is a better way…

This story continues with: TileStache Django tiny wrapper.

Zuerich QGIS hackmeeting

Lo scorso mese di novembre 2011, si è tenuto a Zurigo il meeting internazionale di programmatori QGIS.

Ho avuto la fortuna di partecipare, dato che mi occupo da diverso tempo della nuova infrastruttura web, basta su Django e sono autore di un paio di plugin QGIS python.

È il secondo meeting QGIS al quale partecipo ed è stata una bellissima occasione per incontrare gli altri sviluppatori, e confrontarmi con loro sul futuro del più promettente programma GIS libero.

Durante il meeting ho potuto assistere a numerose presentazioni, tra cui le più stimolanti sono state quella sul porting QGIS per android e il nuovo supporto topologico per postgis.

Il mio lavoro durante il meeting è consistito nella correzione di alcuni difetti nella nuova piattaforma web e l’implementazione di alcune nuove funzionalità, in particolare l’interfaccia XML-RPC per il caricamento dei plugin, implementata con l’ottimo pacchetto RPC4Django.

Ringrazio l’associazione GFOSS che ha contribuito  alla mia trasferta.

I’m running a VM on slicehost and I haven’t alredy found the resources to perform a full upgrade to something newer than the original Ubuntu Hardy which was installed at setup time.

Basically, I’ve compiled latest nginx from sources (older versions did not support uwsgi natively) and installed uwsgi with

sudo pip install uwsgi

not running on a virtualenv on this machine

what I’ve found really hard to fix was the wrong filesystem encoding wich lead to the famous encoding errors:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' inposition 86: ordinal not in range(128)

The fix is trivial: you must convince uwsgi that is running in an UTF-8 locale, to do so, you must set environment vars as explained in django docs   (for apache/mod_python, but the root is the same).

Having had an hard time to set env on upstart, I’ve switched to a standard init script for uwsgi:

#! /bin/bash
# /etc/init.d/uwsgi
#
 
daemon=/usr/bin/uwsgi
pid=/tmp/project-master.pid
# run as www-data
args="--uid=33 --gid=33 --ini /home/public_html/project/uwsgi.ini --pidfile $pid"
 
# Carry out specific functions when asked to by the system
case "$1" in
    start)
        echo "Starting uwsgi"
        LC_ALL='en_US.UTF-8' LANG='en_US.UTF-8' start-stop-daemon -p $pid --start --exec $daemon -- $args
        ;;
    stop)
        echo "Stopping script uwsgi"
        LC_ALL='en_US.UTF-8' LANG='en_US.UTF-8' start-stop-daemon --signal INT -p $pid --stop $daemon -- $args
        ;;
    reload)
        echo "Reloading conf"
        kill -HUP $(cat $pid)
        ;;
    *)
        echo "Usage: /etc/init.d/uwsgi {start|stop|reload}"
        exit 1
    ;;
esac
 
exit 0

Here the trick is to set env vars on the start-stop-daemon line.

The application ini is something like:

[uwsgi]
chdir=/home/public_html/project/
master=True
vacuum=True
max-requests=5000
daemonize=/var/log/uwsgi/uwsgi.log
socket=127.0.0.1:49152
env = DJANGO_SETTINGS_MODULE=settings
module = django.core.handlers.wsgi:WSGIHandler()

Last missing piece is log rotation:

"/var/log/uwsgi/*.log" {
  copytruncate
  daily
  rotate 5
  compress
  delaycompress
  missingok
  notifempty
}

Django-dag is a small reusable app which implements a Directed Acyclic Graph.

Usage

Django-dag uses abstract base classes, to use it you must create your own
concrete classes that inherit from Django-dag classes.

The `dag_test` app contains a simple example and a unit test to show
you its usage.

Example

class ConcreteNode(node_factory('ConcreteEdge')):
    """
    Test node, adds just one field
    """
    name = models.CharField(max_length = 32)
 
class ConcreteEdge (edge_factory(ConcreteNode, concrete = False)):
    """
    Test edge, adds just one field
    """
    name = models.CharField(max_length = 32, blank = True, null = True)

Diango DAG on pypi.

Django DAG github code repository with examples and tests.

GFOSS is the Italian chapter of OSGEO fundation.

I am a proud member from its birth and I participate to almost all events.

About GFOSS Day:

GFOSS day is an yearly meeting where all the people interested in geographic open source free sotware can meet to share ideas and experiences.

This year 2010  the event will last two days (18 and 19 november) with conferences, tutorials and workshops.

QGIS logo

During the last development effort for out GeoDjango-based platform, we’ve been in the need to test if GIS buffering functions were working as expected. Since everything was executed on the server without any user interaction, we ended up logging plain WKT geometries.

Of course understanding the shape and dimensions of a geometry just reading it coordinates is almost impossible, and we couldn’t find an easy way to quickly display WKT geometries in our GIS software QGIS.

This new plugin allows a user to paste WKT geometries into a dialog text field and get it displayed in QGIS inside a temporary (in-memory) layer.

You can find the new plugin named QuickWKT on QGIS the 3rd parties repository.

• Django
• postgis
• pg_routing
• ExtJS
• GeoExt
• OpenLayers

My client is a market leader in GPS wandering and track planning, who needed a complex CMS to publish and promote his activities.

The project’s goal was to develop a Geographic CMS multi-site, multi-admin (resellable) , multi-language using open source tools. Among other interesting features, we needed to support all well-known mapping providers (Google Maps, Yahoo Maps, OpenStreet Map etc.) plus an internal maps with tracks arc-node network, suitable for routing calculations.

The new website(s) are reachable at www.movimentolento.it at the time of this writing, the Italian language is the only one with some contents.

Python vs. PHP

After more than seven years of PHP programming, the language switch was a risky move, but I rapidly fell in love with Python and its vibrant users community. I usually avoid flame wars about programming languages, but I’m now deeply convinced that Python is far more advanced than PHP, even if PHP has its own advantages, expecially when deployment time comes.

(Geo)Django a powerful framework

The choice of Django as the barebone framework was dictated by Django being the one having an advanced support of Geographic extensions (GeoDjango) and by its beautiful documentation. Of course the ideal framework does not exists (yet) and is maybe impossible to create since they are always a compromise between ready-to-use functionalities and flexibility. Django authors must have had a good time providing (almost) all the necessary hooks for customization.

I’ve found Django a very interesting project for many reasons:

Lack of “magic”

While most widespread principles like DRY and convention over configuration are valuable, I’ve often found confusing for first time users the lot of under the hood features that are are triggered because you’ve named a function or a class in a certain way, I think Django has wisely chose to not implement too much of this outside of the ORM DB design phase. The result is a clean application design flow without too many conventions that you must have in the top of your head.

Powerful ORM

I really like the way you can build complex queries and retrieve related object in Django, even M2M relations can be handled in an understandable way. If you need multi-class (multi-table) references you can build them quite easily with the ContenType contribution classes.

Flexible templates

This is maybe what I love much in Django: the templating engine is so powerful and flexible that it can really save you a lot of work allowing you to put presentation logic inside the template itself. I agree that separating concerns in a pure MVC patterns (presentation and controller logic) is a must but think of Django templates like they were the presentation part of the view component and you’ve got the point.

Reusable apps

“Reusable apps” are ready to use applications that you should be able to drop into your applications, and they are reusable indeed. Even if I probably had to modify most of them in order to add multilingual and multi-site support, reusable apps can save you a lot of time.

Missing features

This is a short description of the main problems I’ve encountered when developing missing features in Django. I ended up without core modifications and with a single real patch, so it wasn’t that bad after all.

Multisite

Django comes with the “Sites” framework, it allows you to share the same DB from different websites, but you are supposed to run each website into a different Django instance. While this remains the recommended way, my clients required new sites to be added from the control panel in  an easy way.

Site ID is hardcoded in the settings and due to the fact that static class methods are shared among processes (and moreover among threads, of course) you cannot use it directly without modifications. I’ve had to modify queryst managers and to create a middleware to change Site ID on the fly based on the requested domain. A catch-all DNS record allows to create third level domains within the control panel having them online within minutes.

Something similar is available as reusable app: django-multisite.

Multilingual

Django does not support multiple language for the content (it does for the interface static strings).  I started from a reusable app (transmeta) and did some modifications to allow for optional translations and to easy control panel interface development and presentation. FTS integrations was also added.

Full text search (FTS)

DB is Postgresql which now comes with integrated FTS, I added some triggers and management commands  to support multilanguage FTS. Django doesn’t provide any utility to handle FTS.