TileStache revealed itself as a powerful and flexible product and was very good to cache tiles generated with my Django-backed mapscript-powered WMS server. Caching and serving tiles with TileStache was by far faster then running the whole mapscript WMS machinery.

But I wanted to do more: if I could just convince nginx to serve the tiles directly from disk, this would have been even faster. Of course for this to work, TileStache must be configured to cache tiles on disk, which is what you normally want.

nginx is really good at serving static files and it is luckily wery flexible and easy to configure (once you know what to do).

The following configuration uses named regular expression captures and a try_files instruction to check for the tile existance, if found, the tile is served directly, if not found the internal redirect to the @django named location takes place.
The 404 trick in the base location (/) was necessary to avoid duplication of the @django location content, but there might be a better way.

nginx configuration file

server {
    listen   88 default;
    server_name  localhost;
    access_log  /var/log/nginx/access.log;
    rewrite  "^/[a-z]{2}/(static|media)/(.*)$"  /$1/$2;
    root /www/django_app/;
 
    location  /media/ {
        alias /www/django/contrib/admin/media/;
    }
 
    location  /static/ {
        alias /www/django_app/site_media/;
    }
 
    location ~ ^/(?<lang_code>..)/resource/tiles/1\.0\.0/(?<tile_path>.*)$ {
        alias /tmp/stache/;
        add_header X-Static super;
        try_files $http_host/$lang_code/$tile_path @django;
    }
 
    location / {
        add_header X-Static hit;
        error_page 404 = @django;
        return 404;
    }
 
    location @django {
        add_header X-Static miss;
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $http_host;
    }
}

Results

niginx is listening on port 88 and the dev server (I’m using werkzeug in this case) is listening on port 8080.

ab -c 100 -n 1000 http://localhost:88/it/resource/tiles/1.0.0/track_4/0/0/0.png
 
Concurrency Level:      100
Time taken for tests:   0.084 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      7553744 bytes
HTML transferred:       7302912 bytes
Requests per second:    11902.92 [#/sec] (mean)
Time per request:       8.401 [ms] (mean)
Time per request:       0.084 [ms] (mean, across all concurrent requests)
Transfer rate:          87804.31 [Kbytes/sec] received

With TileStache only:

ab -c 100 -n 1000 http://localhost:8000/track_4/0/0/0.png
 
Concurrency Level:      100
Time taken for tests:   2.944 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Non-2xx responses:      1000
Total transferred:      145000 bytes
HTML transferred:       0 bytes
Requests per second:    339.63 [#/sec] (mean)
Time per request:       294.438 [ms] (mean)
Time per request:       2.944 [ms] (mean, across all concurrent requests)
Transfer rate:          48.09 [Kbytes/sec] received

Conclusions

Unsurprisingly, the solution with files served directly is several times faster: 35 in this test case.

In a production environment, using mod_wsgi or uwsgi, the fugures may vary, but I’m confident that the overall speed boost is guaranteed.

• part 1: Polymaps, TileStache and SVG CSS styling issues
• part 2: TileStache Django tiny wrapper

during my experiments I wanted to reach the goal of minimize both the bandwith consuption and the number of vector coordinates that the browser needs to handle, this means that if we need to show complex geometries, for example a set of mountain hiking paths for an overall of 250000 point, we absolutely need a way to simplify features at lower zoom levels.

TileStache doesn’t support this out-of-the box, so I started playing with a clone of the Vector provider and ended up with a vector-simplify branch in my fork.

What I’ve done was basically to add a new “simplify” provider configuration parameter, where you can enter the tolerance for Douglas-Peucker simplification algorhitm in pixel unit. The paramer can be set in TileStache config file as shown here:

  "layers": {
    "pmcollection":
    {
        "provider":
        {
            "name": "vector",
            "driver": "PostgreSQL",
            "parameters": {
                "dbname": "polymaps_test",
                "user": "xxx",
                "port" : "5433",                
                "table": "pm_points_pmcollection"
            },
            "verbose" : "true",
            "simplify" : 0.5
        },
        "allowed origin" : "*"
    }
 }

A picture is worth a thousand words

Some pictures might be the best way to show how this parameter affects the generated geometries:

This is a single tile, the original geometry is in red, the simplified geometry in green/yellow and an over-simplified geometry in blue.

The following overall views show all the details:

Original geometry

Over-simplified with tolerance 50

Simplified with tolerance 0.5

Open issues

The biggest issue I’ve encountered was about getting the tolerance in lat-lon WGS84 starting from pixel units, but let me explain why I choose pixel units in the first place: I think that pixel units are are the best choice because normally we don’t care about feature details which are below 1 pixel, we simply cannot see them. Using pixel units we also get the advantage of using a single number for all zoom levels, a more complicated approach would consist in using separate values for each zoom level. Translating pixel units to decimal degrees is not possible without approximations, I just hope that my approach works in most situations.

Another strange thing I noted in the library is that the bounding box used for clipping (which is a rectangle) is built using 16 points instead of 4. This means that after clipping we end up with a geometry that have many more points that it needs. I’m still waiting for a clarification I’ve asked in the TileStache ML, but in the meantime I patched my branch to use a normal 4 points rectangle and everything seems to work fine (and with smaller geojson responses).

Conclusions

TileStache confirmed to be a valuable tool for web GIS developers, it’s well written and supported and easily expandable, absolutely worths a try.

First, I would like to thank Michal for the quick patch inclusion, the Polymaps patch is still waiting for the pull and the Django admin patch is awating for somebody to take care of that 18-months-old bug.

My experiments have continued with Django integration, I just wanted to see how easy it was (and it was really easy indeed). There aren’t any clear advantages using a Django app instead of the standalone TileStache server, but writing the wrapper was a nice exercise to learn something more about TileStache and Polymaps, and having all the pieces into one place is always tempting for me (not to mention that you can dribble the CORS completely).

The wapper consists in a single router rule in urlconf:

    url(r'^tiles/(?P<layer_name>[^/]+)/(?P<z>[^/]+)/(?P<x>[^/]+)/(?P<y>[^/]+)\.(?P<extension>.+)$', 'pm_points.views.tiles', name='tiles_url'),

… a view for the tiles (get_config implements a singleton to avoid parsing TileStache config every time):

 
# proper imports go here...
 
def tiles(request, layer_name, z, x, y, extension):
    """
    Proxy to tilestache
    {X} - coordinate column.
    {Y} - coordinate row.
    {B} - bounding box.
    {Z} - zoom level.
    {S} - host.
    """
    config = get_config()
    path_info = "%s/%s/%s/%s.%s" % (layer_name, z, x, y, extension)
    coord, extension = TileStache.splitPathInfo(path_info)[1:]
    mimetype, content = TileStache.getTile(config.layers[layer_name], coord, extension)
    return HttpResponse(content, mimetype=mimetype)

… and a simple view for the map:

 
def home(request):
    """
    Pass layers by
    """
    config = get_config()
    layer_urls = []
    for l in config.layers:
        if isinstance(config.layers[l].provider, TileStache.Providers.Vector.Provider):
            layer_urls.append(reverse('tiles_url', args=[l, '{Z}', '{X}', '{Y}', 'geojson']).replace('%7B', '{').replace('%7D', '}'))
    return render_to_response('home.html', { 'layer_urls' :  layer_urls}, context_instance=RequestContext(request))

… and a template for Polymaps (I paste the js part only):

 
    var po = org.polymaps,
        map,
        layers = [],
        stylist;
        jQuery(function(){
            map = po.map()
                .container(document.getElementById("map").appendChild(po.svg("svg")))
                .center({lat: 40, lon: 0})
                .zoomRange([1, 16])
                .zoom(2)
                .add(po.interact())
                .add(po.hash())
                ;
 
            map.add(po.image()
                .url("http://s3.amazonaws.com/com.modestmaps.bluemarble/{Z}-r{Y}-c{X}.jpg"));
 
            stylist = po.stylist()
                .attr('class', function(d){
                    return d.geometry.type + '_' + d.properties.id;
                })
                .title(function(d){return d.properties.name;})
                ;
 
            {% for lay_url in layer_urls %}
            var layer = po.geoJson()
                .url("{{ lay_url|safe }}")
                .zoom(function(z){
                    return z;
                })
                .id('pmcollection')
                .on("load",  stylist)
                .on('show', stylist)
                ;
            map.add(layer);
            layers.push(layer);
            {% endfor %}
 
            map.add(po.compass()
                .pan("none"));
            map.container().setAttribute("class", "mymap");
        });

Quick and dirty, it also dynamically adds layers to the Polymap.

This story continues with: TileStache Vector simplify provider

The final goal of this experiments was the integration with Django.

Polymaps

Polymaps is a Javascript library aimed to display svg maps in a web page, recently I’ve watched to an interesting talk about a Django-powered app built on top of Polymaps:

The main reason to give Polymaps a try is the interesting support to tiled vector layers, as far as I know a unique feature.

I’ve found the documentation of Polymaps a bit obscure and still incomplete (I just hope it isn’t another facade Open Source product where the lack of documentation and community support limits the use to commercial projects).

A quick look in the code shows no comments and rather obscure variable names, debbugging this library would certainly be a nightmare if you’re not the author, if you compare the source code with OpenLayers’s code you get the idea.

BTW I managed to make it work in a few hours (with all the server stuff).

TileStache

TileStache is a python library aimed to build, cache and manage map tiles. It’s similar to TileCache in scope and uses the well known Mapnik renderer while the data provider is implemented with shapely and OGR and can read the most widely used formats (postgis, shapefile, geojson).

TileStache is the ideal companion to Polymaps because it can handle and cache vector tiles (geojson) through the Vector class (don’t even try to use the PostgreSQL class you can find in Goodies, it’s kind of unofficially deprecated and it will let you waste your time).

I had to fork the project on github and made a pull request to fix the missing “port” parameter to the DB connection since I’m using a not-standard port.

SVG CSS

I admit I am a true beginner with SVG and while I was trying to experiment with the stylist features of Polymaps  I discovered some counter-intuitive (at least for me, coming from XHTML CSS) behaviours:

You can define styles in three ways:

1. CSS rules
2. “style” attribute of the SVG element
3. individual attributes (“stroke”, “fill” etc. etc.) of the SVG element

Strange enough, the priority is top-down, while I would have expected that point 3 had a higher priority.

Another thing worths noting is that what distinguish a polygon from a linestring in the Polymaps generated “path” SVG element is just that the “d” element (which contains the geometry) ends with a “Z”.

This is a linestring:

<path d="M101.99999715555555,189.9999971369224
L75.00000142222234,219.00000190473258
L75.00000142222234,242.49999834108655
L94.49999928888906,242.99999806558844
L102.49999928888906,255.4999976705726
L101.99970702222231,256"/>;

This is a polygon:

<path d="M95.07080533333351,256.00000052462633
L0,196.67516186124612
L0,192.0000022835775
L0,128.0000001888236L0,70.92093168072711
L175.92470186666674,0.000006325172648757871
L192,0.000006325172648757871
L256,0.000006325172648757871
L256,64.00000444302805
L256,128.0000001888236
L256,160.99230434784874
L113.66593991111131,256.00000052462633Z"/>;

Styling the plygon fill without hitting linestring in a GEOMETRYCOLLECTION layer turned out to be quite troublesome.

I ended up with a CSS-3 attribute CSS rule:

.layer path {
fill: cyan;
fill-opacity: .85;
stroke: #012;
stroke-linecap: round;
}
 
.layer circle {
fill: cyan;
fill-opacity: .85;
stroke: #012;
}
 
/* Don't fill in SVG */
.layer path:not([d$="Z"]){
fill: none;
stroke-width: 3px;
}

I wonder if there is a better way…

This story continues with: TileStache Django tiny wrapper.

Zuerich QGIS hackmeeting

Last november I participated to the international QGIS programmer’s meeting in Zuerich. It was a great opportunity to meet the QGIS developers I already knew from the previous meeting in Lisbon, and to know new interesting people.

QGIS is quickly becoming one of the most important free software packages for GIS processing, I participate to the team as the lead developer of the new python plugins web application, realized with the Django framework.

During the meeting I worked on fixing bugs and I implemented some new features the most important being the new XML-RPC interface that allows to programmatically upload plugins.

The newXML-RPC  interface has been realized with RPC4Django, a cool package that allows for quick webservice creation in Django.

Looking forward to the next hackmeeting in France I wish to thank the Italian Geographic Free Open-Source Software Association for the financial aids provided.

You can just right-click and save it on your bookmarks, the click on the bookmark while in the OpenCycleMap site and a popup will appear with the extent coordinates printed on screen.

Bookmarklet.

Strangely enough there is no way to download an uncompressed version from the project website, even if they provide a nice custom build form.

The recommended way is to use “jsbuild“, which turned to be a nightmare because it does not run out-of-the-box, at least on my Ubuntu lucid.

A one-liner did the trick, run it from a “build” folder under GeoExt main folder, the correct order of inclusion (which does really matter) is taken from the “single” file GeoExt.js.

for f in $(egrep '"GeoExt/' ../lib/GeoExt.js|sed -e 's/"//g'|sed -e 's/,//g') ; do cat "../lib/$f" ; done > GeoExt-debug.js

GeoContent Plus Logo

  native!

Browse GeoContent Plus documentation!

Try GeoContent Plus demo!

If you are looking for the old GeoContent version for Joomla! 1.5, please check the original announce.

Introduction

The Map

GeoContent Plus is a Joomla! 1.7.x and 2.5.x component that let you easily embed beautiful Google Maps or OpenLayers Maps into your Joomla! 1.7.x and 2.5.x website.

Standard Joomla! articles, can be bound to geographic features such as POINTs, LINEs and POLYGONs and the resulting maps can be shown in the article page and in one or more overview maps.

A single GeoContent Item can be represented by more than one geographic type: so that you can have simultaneously a set of POINTS, some POLYGONS and some LINES for a single Joomla! article.

GeoContent Items on the map are grouped by GeoContent Layers, graphic appearance of GeoContent Items can be fully customized defining styles attributes in the GeoContent Layers configuration panel.

GeoContent Plus was created as a sort of Geographic Index to Joomla! articles, the earliest versions had a tight binding between Joomla! articles and the geometric features that GeoContent Plus was going to show on its maps. Suggestion from the users made GeoContent Plus subsequently evolve to cover more general use cases, it is now possible to createGeoContent Items which are completely unbound from existing Joomla! articles.

The Front-End editor window

The new Access Control List (ACL) features introduced by Joomla! 1.6 are now fully supported by GeoContent Plus allowing a fine grained control over the front-end and back-end user access to the different operations available in the software.

GeoContent Items are also language-aware: they can optionally make use of the same Joomla! language support system recently introduced for Joomla! core articles.

Download

ComGeocontent is open source software, distributed under AGPL software license and it is also free software, where “free” means that you can do (almost) whatever you want with this software but it does not mean that it costs nothing. We distribute the component only in bundle with paid 12 month support service that costs 45 €.

To receive a copy of GeoContent you are kindly requested to pay us 45 €, using the buttons located at the top or at the bottom of this page. You will receive a regular invoice at the end of the month. After a successful payment,  an email with the download link will be automatically sent to you PayPal’s account email address.

The money you pay will be used to further develop GeoContent and to provide professional assistance and support to those that will need it.

Together with 12 months support you will get lifetime software updates, that means that you can pay just once and use the software forever in how many Joomla installations you like.

Please use the PayPal link at the top of this page, specifying the email where the component will be sent.

Thank you for supporting GeoContent project!

Key features

QGIS logo

Last week I participated to the QGIS hackfest in Lisbon. It was a wonderful opportunity to meet a lot of interesting people, most of whom I’ve only known by email or chat.

I was also very happy to meet Tim Sutton, release manager of QGIS, I’ve spent two month within a project with him without having ever meet in person.
The meeting has been carefully organised by Giovanni Manghi and Vania Nieves, both from Faunalia PT. The faculty of letters in the university of Lisbon was the location of the meeting, a nice place to stay, just a little bit too warm in these days.
The atmosphere was that of a friend’s meeting, informal but very professional when discussing the future plans of the QGIS platform: most of the developers are not working full time on QGIS but the concentration of talented professionals in the room was really impressive.

My role in the story is related to the periferic functions of the new python plugins web application: I’ve been the main developer of that application, it is based on Django, the platform I’m working on since two years now. I’ve been lucky and in Lisbon I met two very skilled young developers which demonstrated a strong interest in helping out with polishing and completing the application. So, thanks to Richard and Ricardo (this is their name) the new QGIS web application will hopefully grow quickly and with a lot of new cool features.

The conclusion is that this Lisbon hackfest has been a very intensive and interesting meeting, I really hope that I will be able to participate at the next one, that will take place in Zurich next fall

The proposed solution involves a custom control subclass and the standard OL machinery, the MSIE oddities are handled internally by OL itself.