Posts by Lutra Consulting

Trace digitising in QGIS

Following the popularity of the AutoTrace plugin, we received a number of requests for additional features and to incorporate the tool into the main Advanced Digitising tools in QGIS.

Funding for the work was generously raised with the help of the community in 2015 and the feature landed in QGIS 2.14

How to use Trace Digitising

Check out the Trace Digitising in QGIS for detailed instructions on how to use the tool.

Key features

The trace tool:

  • uses Dijkstra’s shortest path algorithm to find traceable routes
  • can trace routes over multiple distinct features
  • can be used with Advanced Digitising tools (e.g. reshaping)
  • can be enabled and disabled by pressing T on your keyboard while digitising
  • is fast and easy to use

The following video demonstrates how to use the tracing tool:

This video shows how you can make use of tracing while reshaping a polygon:

Sponsors

The project was kindly sponsored by the QGIS community. Special thanks to all those who contributed towards the development of this tool:

  • The Royal Borough of Windsor and Maidenhead
  • Neath Port Talbot County Borough Council
  • Ujaval Gandhi
  • Surrey Heath Borough Council
  • Matias Arnold
  • Northumberland National Park Authority
  • Buccleuch Estates Limited
  • Countryscape

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows. Screenshots of the Mergin Maps mobile app for Field Data Collection
Get it on Google Play Get it on Apple store
Learn More

Using Presets, multiple styles, Atlas with overview map

Our training courses are structured to give trainees step-by-step guide using the latest QGIS features. Here is an example of one our practicals for creating professional maps.

Trainees will learn how to

  • Assign multiple style to a layer
  • Working with presets
  • Working QGIS print composer
  • Generating maps using Atlas

Creating professional maps with the Print Composer

QGIS’ Print Composer allows users to create professional maps for printing. It supports legends, frames, logos and all the other features you’d expect to see in a print quality map.

In this example, we’ll use some of the most common features required to generate a map for printing.

We’ll first load the electricity usage data created in another tutorial, style it using a simple colour ramp, then create a new map composer complete with title and legend.

Loading data in QGIS

The styling of a vector layer can be performed using a single style, or a varying style, based on the value of one or more attributes. We are going to first load the London energy usage layer prepared in an another tutorial by joining electricity usage with the LSOAs vector layer.

Start a new QGIS project

Add the MSOA_GL_ElecUsage.shp layer under extracted
downloaded folder

The vector layer should now look like the figure below.

Energy consumption in London Energy consumption in London

Next, we are going to add a shapefile of London Boroughs to identify areas of high and low usage.

Add London_boroughs.shp from extracted downloaded
folder

Open layer properties and under Style:

Set the style of the polygons to "No Brush" with a
"Dash Line".

Styles for London Boroughs vector layer (Click to enlarge)

Your map now should look like the figure below:

Energy consumption in London Energy consumption in London

In QGIS, you can apply multiple styles to a single layer. Using this method, we do not need to load the same layer multiple times. In this case, we need two styles for London boroughs layer:

  1. Without labels (as created above); to be used as an overview map

  2. With labels (next section); for the main map

We can save the current style (without label):

Right-click on London_boroughs layer from the layer
tree and select Styles > Add

For the new style, type 'no label' and click OK

We can save the new styles (with labels) for the layer:

Right-click on London_boroughs layer from the layer
tree and select Styles > Add

For the new style, type 'with label' and click OK

Changes in the next section (adding label) will be saved in with label style.

Labels

We’ll now add labels to each of the boroughs. The name of each borough is stored as the value of the NAME attribute.

Click on the London_boroughs layer in the ToC

Form the main menu, select Layer > Labelling

This window will allow us the set an attribute for label and also change font, size, rotation, buffer, etc.

Setting up labels on the London boroughs layer Setting up labels on the London boroughs layer

Enable the Label this layer with option

Set Field with labels to NAME

Under Text section, set the font to 7 pt

Under Buffer section:

  Enable Draw text buffer

  Set the Size to 0.5 mm

Click OK

You should now be able to see a label for each borough.

London boroughs with labels London boroughs with labels

Presets

In QGIS, you can create presets for layer visibility and styles. In this example we are going to create the following presets:

  • Map of LSOAs with the energy usage and borough boundaries

  • A separate preset for the London boroughs (without labels) which will be used as an “overview” for the print composer

To create the first preset:

Ensure both layers are visible

Right-click on London_boroughs and select
Style > with label

On top of the legend tree in QGIS, you can see the icon for Manage Layer Visibility image

Click on the Manage Layer Visibility icon

From the drop-down menu, select Add preset...

For the Name of the new preset insert Energy map

Click OK

To ensure the preset has been correctly set, you can change the visibility of the layer, or change the style, and select the preset from the Manage Layer Visibility menu.

To create the second preset:

Hide All Layers from Manage Layer Visibility or simply
press Ctrl+Shift+H

Turn on the visibility for London_boroughs

Right-click on London_boroughs and select
Style > no label

On top of the legend tree in QGIS, click on
Manage Layer Visibility

From the drop-down menu, select Add preset...

For the Name of the new preset insert overview map

London boroughs - without labels London boroughs

Now you should be able to switch between those two presets, and later use them as print composer frames.

Creating layout

The map is almost ready to be printed. Before printing, we’ll need to add a border, logos, legends, scale, etc.

There are quite a few steps involved in configuring the print composer so it’s recommended to regularly save your progress by saving the QGIS project.

From the main menu, select File > New Print Composer

A new window will appear for Composer title

Type Energy usage in London

The print composer title window The print composer title window

The print composer will appear. See Figure.

PrintComposerWindowEmpty The print composer window

The print composer is split into two sections. Section 1 shows the map to be outputted and section 2 shows the settings for selected items in section 1.

The map we are going to produce is an A4 landscape PNG file.

In section 2, under Composition, for Presets, select
A4(210 x 297 mm)

Now we need to add frames for the map extent and legend sections. You can add a rectangular shape by clicking on image from the main toolbar.

From the main toolbar, add a new rectangle

Click and drag a shape in section 1

Now let’s change the position and size of the rectangle:

Ensure the rectangle shape is selected in section 1

Click on Item properties in section 2

Under Shape, select Position and size

Set the option as shown in Figure

Shape options Shape options

Add second, third and fourth rectangles with the
following information

Rectangle 2:

  X: 270

  Y: 145

  Width: 43.5

  Height: 60

  Reference Point: Centre

Rectangle 3:

  X: 270

  Y: 95

  Width: 43.5

  Height: 60

  Reference Point: Centre

Rectangle 4:

  X: 270

  Y: 105

  Width: 43.5

  Height: 200

  Reference Point: Centre

Lock the position of each window by right-clicking
on each of them

Section 1 of the print composer should look like this:

The print composer with placeholders layout The print composer with placeholders layout

Adding image

Let’s add the logos.

From the main menu, select Layout > Add Image

Click within the middle rectangle on the right

An empty box will appear

In the right-hand panel (section 2), under the
Item properties tab,

	click on Main properties button

For Path, click on ... and browse to the folder where
your images are located.

Select an image

You should now see something similar to the image below.

Logos added to the composer Logos added to the composer

Adding text

Now we’ll add some text in the rectangle below the logos:

From the main menu, select Layout > Add Label  

Click somewhere within the lower right-hand box  

A new text box will appear, reading QGIS.

From the right-hand panel, click on Item properties

In Main properties, change the text to refer to the
source of the data and copyright notices.

Copyright notices Copyright notices

Adding map

Now we’ll add the map from the QGIS canvas.

From the main menu, select Layout > Add map

Drag a box in the main window

A map similar to the image below should now appear in the print composer.

To finalise the main map window, a few more changes will need to be made.

Adding map to the print composer Adding map to the print composer

From the right panel, click on the Item properties tab

  Under the Main properties, set the following values:

    Tick the box for Lock layers for map item see
image below

  Under Extents:

    Tick the box for Controlled by atlas

Adding a preset for map view Adding a preset for map view

Add scalebar

To add a scalebar:

From the main menu, select Layout > Add Scalebar

Adding a scalebar to the map Adding a scalebar to the map

A scalebar with default settings should appear on the map. See Figure [PrintComposerAddScalebar]. To change the scalebar:

Select the scalebar from the map

Click on Item Properties from the left-hand panel

Under Segments

   Set left 0 and right 2

   Set Size to 1000

   Set Height to 2 mm  

Adding legend

Next, we’ll add a legend:

From the main menu, select Layout > Add Legend

Click somewhere within the upper right-hand box

QGIS will automatically add a legend for all the layers loaded in the canvas. We can edit the legend’s settings and remove the legend entries that we do not want to show. Manual edits of the legend text can also be performed to make it more readable.

To be able to edit the legend items, you first need to untick the box for Auto update. Use icons under the Legend items to edit or change the order of the items.

The default legend The default legend

From the right-hand panel, click on the Item properties

  Under Legend items:

    Select London_boroughs and then click on the red
    minus sign


    Select MSOA_GL_ElecUsage and click on the edit
    sign, a new window will appear.

    Change the content of Item text to Electricity
	Usage

    Click on the first value 2576.2400 – 3000.0000
	and select edit sign to change the range to
	2000 – 3000   

    Repeat for the rest of the values to round them
	and remove any unwanted zeros.   

Editing legend values Editing legend values

Map with legend Map with legend

Add overview

To add an overview:

Add a new map (Map 1) to the place-holder between
the map legend and the logos

For Map 1, from the Item properties,

Under Main properties

   Select Lock layers for map item

   From the visibility preset, select overview map

Under Overviews tab  

   Click on the plus sign,
	Overview 1 should be added to the overview list

   Ensure Draw "Overview 1" overview is selected

   For Map frame, select Map 0

Setting the preset view for the overview map Setting the preset view for the overview map

The map is now almost ready. You can export it as PNG or a PDF. Or you can keep reading for even more excitement!

The completed map The completed map

Atlas generation

The map is now ready to be printed or outputted as an image. But, we are going to use this layout as a template and auto-generate energy consumption for each borough.

To set the coverage (this is already set, but make sure the option is enabled):

While Map 0 (the main map) item is selected, click on
Item properties from the right-hand panel

Select the option for Controlled by atlas generation

You can change the Margin around feature if you like, but we can keep at 10%. Further settings are also required:

Click on Atlas generation tab from the right-hand panel

Tick the box for Generate an atlas

Under Configuration, for Coverage layer select
	London_boroughs

Under Output, for Output filename expression type:
	'energy_'|| "NAME"  

The expression for output will set the name of each output file. In this example we are going to have a name with ‘energy_‘ combined with the value of NAME column (what do you think this value is?)

To generate the maps:

In the print composer, from the main menu, select
Atlas > Export Atlas as Images...

Select a folder to output maps and press Choose

The process may take a while but you will eventually have 33 maps of the energy consumption for each borough. Note the overview for the maps and also the naming of the output files. See Figure beloq as an example of the maps created.

Final energy map for Kingston upon Thames generated by the Atlas Final energy map for Kingston upon Thames generated by the Atlas

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows. Screenshots of the Mergin Maps mobile app for Field Data Collection
Get it on Google Play Get it on Apple store
Learn More

Rule-based labelling in QGIS 2.12

The new QGIS 2.12 (Lyon) will be out soon!

In this release, we have revamped the labelling engine and made it more flexible in-line with the rest of vector styling.

In this release, we have:

  • Revamped the labelling engine
  • Added support for mutually exclusive layer tree groups
  • Developed raster alignment tool

For users

In previous versions of QGIS, users can select a field value or use an expression as labels for a vector layer. In QGIS 2.12, users can additionally define rules to label vector layers. Rule-based labelling works in the similar way as “Style”. A list of rules will be defined by users and they will be applied from top-to-bottom.

Field based labelling (Click to enlarge)

Rule based labelling (Click to enlarge)

To achieve the same effect in the earlier versions of QGIS, users should add a complex expression.The example below shows the expression used in earlier versions of QGIS:

CASE WHEN  length( "htmlname" ) > 13 AND strpos("htmlname",' ') > 6  THEN  replace("htmlname",' ','  ') WHEN  length( "htmlname" ) > 20 AND "htmlname"  LIKE '%Golf Course' THEN  regexp_replace("htmlname",'Golf Course',' Golf Course') WHEN  length( "htmlname" ) > 20 AND "htmlname"  LIKE '%Nature Reserve' THEN  regexp_replace("htmlname",'Nature Reserve',' Nature Reserve') WHEN  length( "htmlname" ) > 20 AND "htmlname"  LIKE '%Church Of England%' THEN  regexp_replace("htmlname",'Church Of England',' Church Of England ')  WHEN  length( "htmlname" ) > 13 AND "htmlname"  LIKE '% Of The %' THEN  regexp_replace("htmlname",'Of The','Of The ') WHEN  length( "htmlname" ) > 13 AND "htmlname"  LIKE '% of %' AND  "fontcolour" <> 2 AND  "fontcolour" <>  4 THEN  regexp_replace("htmlname",' of ',' of  ')  WHEN "htmlname" LIKE '%/%' THEN regexp_replace("htmlname",'/','/  ') WHEN  length( "htmlname" ) > 30 THEN  replace("htmlname",' ','  ')  ELSE  "htmlname"  END

As you can see, without rule-based labelling users have to define each case and the labelling text. Additionally, with the cumbersome task of defining each case, users are limited to using specific labelling formats (e.g. font color, font size, visibility range, etc). With the new labelling engine, users can define their labels similarly to styles - in fact they can copy the rules from styling tabs and use it within the new labelling section!

Examples of label rules (Click to enlarge)

For developers

In order to facilitate addition of rule-based labelling, some internal changes were made to the QGIS labelling engine interface. The labelling is now driven by the new class QgsLabelingEngineV2 which may have several label providers associated with it. The label providers are objects derived from QgsAbstractLabelProvider and they are responsible for:

  1. providing “label features” that define properties of each label and geometry of the feature they represent; and

  2. drawing of labels at the positions determined by the engine.

Currently there are label provider implementations for diagrams, simple labelling and rule-based labelling.

The existing labelling engine (QgsPalLabeling class) is now built on top of the new labelling engine and works as a wrapper for it so that existing code that uses QgsPalLabeling still works.

As of now, the API for the new labelling engine is not considered as complete and therefore not available in Python. The idea is to make it easier to use, more polished and better prepared for the future use cases. It will be likely finished during the 2.14 release cycle and some changes may need to be postponed to QGIS 3.0 where backwards incompatible API changes will be allowed (as of now, QGIS 3.0 is being intensively discussed on QGIS developer mailing list).

Funding

This rule-based labeling has been funded by Tuscany Region (Italy). Special thanks also to the QGIS developers for their help with bug fixing.

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows. Screenshots of the Mergin Maps mobile app for Field Data Collection
Get it on Google Play Get it on Apple store
Learn More

Crayfish 2.1: New features

New features keep being added to Crayfish. Now it is possible to export time variable grid as animation, add AnuGA results and visualise vectors on user-defined grids.

Export to animation

The ground works were done in Crayfish 2.0 for this feature. You can now generate animation from contours and vectors and export them as AVI. There are two methods of exporting to animation: basic and using QGIS print template (qpt).

With the basic option, you can define a title, a legend and a progress clock. Alternatively, for a smarter solution, you can set up a print composer with the benefit of all its rich features. The composer template (qpt) can be used as a frame layout for exporting your animation.

Below is an example of a multi-frame print composer template used to generate animation from a Crayfish layer.

AnuGA support

Crayfish 2.1 now supports SWW file format generated by AnuGA</li>.

SWW files generated by AnuGA in Crayfish (Click to enlarge)

Vector on user-defined grid

With this option, users can define a grid and Crayfish will interpolate values and displays results on the custom grid. Images below show vectors on the outputted mesh and user-defined grid.

Vectors on default mesh in Crayfish (Click to enlarge)

Vectors on default mesh in Crayfish (Click to enlarge)

Vectors on a user-defined mesh in Crayfish (Click to enlarge)

Crayfish manual

With the ever-growing features in Crayfish, we have decided to dedicate a page on how to use Crayfish in QGIS. From the manual page, users can download a sample data and try the Crayfish features in QGIS.

Sponsors

We’d like to thank Maroondah City Council for sponsoring some of the great features in this release.

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows. Screenshots of the Mergin Maps mobile app for Field Data Collection
Get it on Google Play Get it on Apple store
Learn More

How to use Function Editor in QGIS Field calculator

In QGIS 2.8, there is a new option for users to add their own python function in the Field calculator. This is an extremely useful feature enabling users to populate data within the attribute table using customised python function.

Nathan wrote a blog post about the feature and how to write a python with arguments. But in QGIS 2.8, the function editor does not correctly support functions without arguments.

In the example below, we are going to calculate proportion of area for each SAC (Special Areas of Conservation) in Great Britain to the total area of the layer.

Add GB_SAC_20130918_Dissolved.shp layer to QGIS, right-click on the layer and open attribute table. Make the layer editable and click on the Field calculator. We are now going to create a new column (propArea) and populate proportionate area of each SAC to the total area of the layer.

Under Function Editor tab, click on New file and type area for the name and save the file. For the content, copy and paste the following lines:

"""
A custom function to calculate total area of the GIS layer.
This function has no arguments.
"""

from qgis.core import *
from qgis.gui import *
from qgis.utils import iface

@qgsfunction(args='auto', group='Custom')
def total_area(x, feature, parent):
	return sum( f.geometry().area() for f in iface.activeLayer().getFeatures() )

Click on Run Script to add total_area function to your Custom function list.

Now, click on Expression tab and type:

$area / total_area(0)

As you can see, we have passed 0 as an argument. If you click OK, your QGIS will freeze! As there are many features in the layer, the expression, calculates total area for each row.

Lets make the script a bit more elegant. Firstly, we need to add caching, so that area will be calculated only once and cached for the rest of operation. Secondly, we can make the script a bit more generic, so that we can use it to get the area of other loaded layers in QGIS:

from qgis.core import *
from qgis.gui import *
from qgis.utils import iface

cache = {}
def _layer_area(layer):
	""" Internal method """
	if layer.name() not in cache:
		area = sum( f.geometry().area() for f in layer.getFeatures() )
		cache[layer.name()] = area
	return cache[layer.name()]

@qgsfunction(args='auto', group='Custom')
def total_area(layer_name, feature, parent):
	for layer in iface.mapCanvas().layers():
		if layer.name() == layer_name:
			return _layer_area(layer)
	return 0

Now, click on Expression tab and type:

$area / total_area('GB_SAC_20130918_Dissolved')

This time, it should be quicker!

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows. Screenshots of the Mergin Maps mobile app for Field Data Collection
Get it on Google Play Get it on Apple store
Learn More

Crayfish 2.0: What's new!

After listening to user feedback we decided to do some major work on Crayfish. The changes include code refactoring, changes to the user interface, support for an additional file format, adding a vector and contour overlay, and a shiny new logo!

Crayfish 2.0

Time control

In the new version of Crayfish a time slider allows users to quickly browse through time. A drop-down menu allows the selection of an exact time.

Vector and contour overlay

In previous versions of Crayfish it was only possible to load contours and vectors from the same dataset. For example, it was not possible to show velocity vectors on top of depth contours. With the new Crayfish you can "unlock" the legend and choose different vectors or contours to be displayed. The video below demonstrates this in action.

Special times

Some datasets contain a special time-step outside the outputted time range. For example, Maximums and Minimums are stored at time 99999 in TUFLOW modelling package. Within the layer tree, additional time-steps items will now be shown if they exist within the dataset.

Additional file formats

We have added support for the XMDF file format. In addition, Hydro_AS 2D users should be able to open their files in the latest Crayfish.

New Python Module

We've refactored lots of code in the Crayfish library which makes it much easier to add support for further file formats and additional functionality. The Crayfish library now comes with a new Python module that allows easy manipulation with the mesh and results data - either in your custom scripts or within the QGIS Python console. For example, printing the coordinates of the nodes of a mesh together with their elevation takes just few lines of code:

import crayfish

m = crayfish.Mesh("/data/my_mesh.2dm")
o = m.dataset(0).output(0) # bed elevation data

for index, node in enumerate(m.nodes()):
print "Node XYZ: ", node.x, node.y, o.value(index)

New Profile tool plugin

If you use Profile tool plugin in QGIS, you can create a profile from the Crayfish layer and browse through the time. The profile gets updated as you change the output time.

Problems

If you have some feedback on our changes, suggestions for new functionality, or come across a bug, feel free to file a ticket on the issues page of the Crayfish github repository.

Sponsors

We’d like to thank Maroondah City Council for sponsoring some of the great features in this release.

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows. Screenshots of the Mergin Maps mobile app for Field Data Collection
Get it on Google Play Get it on Apple store
Learn More

Using layer visibility presets in QGIS

One of the cool outcomes of our recent QGIS Legend Refactoring work was the ability to add "visibility presets". This feature allows users to save and load preset layer and group visibility settings.

Main canvas

First select the layers and groups, you’d like to manage visibility presets for. Now click at the top of the Layers panel and select Add preset … A new window will appear. Enter a name for the preset and click OK. Next time you click you should see the saved presets. The presets also save the visibility of categories in each layer.

In the video below, we create two presets. One for a specific site location and another zoomed to the outline of Great Britain which will be used as an overview in the print composer.

The print composer now allows multiple map windows, each locked to a different layer visibility preset. This means a single map composer can show map windows with different layer arrangements.

In the video below, one preset is used for the main map and the other for an overview:

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows. Screenshots of the Mergin Maps mobile app for Field Data Collection
Get it on Google Play Get it on Apple store
Learn More

Crayfish 1.3: What's new!

Crayfish is becoming the must-have QGIS plugin for those working with the binary and ascii DAT/.2dm formats. Recently we had some requests from BASEMENT users to fully support the file format of their modelling package.

We have also been busy beta-testing our new illuvis flood risk communication service. Crayfish 1.3 integrates the illuvis client, allowing users to interact easily with this flood risk mapping service.

Read on for a look at some of the new features in more detail...

Advanced Styling

You can now turn off values below or above certain thresholds. This feature is very useful when you work with rainfall-runoff models and want to turn off values below e.g. 2 cm.

You can now set your colour ramps once, save them and apply them to other layers. Export to Geo-TIFF ------------------

You can now easily export outputs from Crayfish to Geo-TIFF (.tif) files. Simply select the Quantity and Output time in the Crayfish viewer then right-click on the Crayfish layer in the QGIS Layers panel and select Export to grid...

Export to illuvis -----------------

For those users who have been using illuvis, you can now upload crayfish layers directly to the flood map publishing web service rather than first having to convert results to Geo-TIFF. If you would like to become an illuvis beta-tester, please contact us.

Support for BASEMENT -------------------- Users can now view outputs from the BASEMENT modelling package.

Binary package for Ubuntu -------------------------

We love Linux and have finally managed to prepare crayfish binaries for Ubuntu 14.04. If you are using other distros, the compilation should be straight-forward using guide in the README file.

Problems -------- If you have some feedback or come across a bug, feel free to file a ticket on the issues page of the Crayfish github repository. Sponsors -------- We'd like to thank The Laboratory of Hydraulics, Hydrology and Glaciology (VAW) of ETH Zurich for sponsoring some of the great features in this release.

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows. Screenshots of the Mergin Maps mobile app for Field Data Collection
Get it on Google Play Get it on Apple store
Learn More

Loading MasterMap: Free, Easy and Fast

We've developed a new graphical loading tool for OS MasterMap data focussing on usability and performance to make it easy to load national Ordnance Survey MasterMap datasets in a matter of hours.

OS Translator II - Load OS MasterMap into PostGIS quickly, easily and for free

The tool is OS Translator II - it makes use of the excellent GDAL library and is available now in the official QGIS Plugins repository.

This blog post talks about some simple benchmarks we've carried out.

If you are interested in using this tool and not familiar with Postgresql/PostGIS, you can sign up to one of our support packages and we will be able to set you up and running within a couple of hours!

Results

National load times were as follows:

  • MasterMap Topography (National) 20 hrs 21 mins 1
  • MasterMap ITN (National) < 6 mins

Installing PostgreSQL, PostGIS and QGIS took less than 10 minutes.

1 This is the most time-consuming test which filled the SSD on the first attempt. Importing to a tablespace on the main HDD completed after 20.3 hours but showed the import of tile 1592959-TR0585-5c3268.gz to have failed with this error. Until this issue is resolved the tile would need to be loaded and de-duplicated manually (e.g. using ogr2ogr to import and a SQL query to de-duplicate) to complete the dataset. De-duplication removes duplicate features caused by the chunking / supply process.

Comparison With Other Open Source Tools

We were curious as to how OS Translator II load times compared with other open loading methods so we did some basic tests using the "SU" tile of MasterMap Topography and ITN datasets and compared it with the popular Loader scripts. The results looked like this:

OS Translator II Benchmarks

Please note that OS Translator II had an unfair advantage in these tests as it automatically takes advantage of multiple-CPU cores whereas Loader presently does not.

Hardware and Software

We used the following hardware and software configuration:

  • CPU Intel Core i7 4790K (Haswell) @ 4GHz
  • Memory 32GB PC3-12800
  • Disk(s) Samsung 840 EVO 250GB SSD and Seagate Barracuda ST2000DM001 2TB HDD2
  • OS Microsoft Windows 7 Professional (64-bit)
  • PostgreSQL 9.4.1 (x64)
  • PostGIS 2.1.5 (x64)
  • QGIS 2.6.1 (Brighton)
  • OS Translator II 1.0
  • Python 2.7.9 (win32)
  • lxml 3.2.3 (win32)
  • Loader Master (067a511313, 20th February 2014)

2 Operating system and source gml.gz files located on the SSD and default PostgreSQL tablespace stored on secondary 2TB HDD.

PostgreSQL Configuration

The following changes were made to the default PostgreSQL configuration:

  • shared_buffers 512MB
  • work_mem 16MB
  • maintenance_work_mem 128MB / 1024MB3
  • checkpoint_segments 6
  • random_page_cost 2.0
  • fsync off

3 maintenance_work_mem was set to 1024MB for the national load of MasterMap Topography layer only.

Turning fsync off is dangerous and can lead to data loss in the event of an unexpected power outage. Always switch fsync back on after loading and never use this option on a database containing critical data.

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows. Screenshots of the Mergin Maps mobile app for Field Data Collection
Get it on Google Play Get it on Apple store
Learn More

Towering QGIS 2.8 Release

The new QGIS 2.8 release (codename: Wien) is out tomorrow!

Lutra’s very own QGIS core developer Martin Dobiaš has been working hard with others on bringing you new stuff. So, what exciting new features are provided by this jauntily-angled new release? There are plenty of eagerly-anticipated additions - both big new features, and under-the-hood improvements - as well as bugfixes.

Multiple Stylings

Have you ever found yourself duplicating layers just so you could display the same data but with additional styles? Those days are now over.

You can now define multiple styles for a single layer, and easily switch between them in the layer view or layer properties dialog.

Multiple styles in the legend

Multiple styles in the layer dialog

Just so our webmapping colleagues don’t feel left out this new behaviour is extended to the WMS server too. The available multiple styles are advertised in GetCapabilities and can be used in other requests, such as GetMap.

And of course it all works in the map composer too - which means more maps with the same layers displayed in different styles.

Multiple styles in the map composer

Work carried out in cooperation with Gis3W for Regione Toscana.

Making It Snappy

Snapping is made simpler with the addition of new modes for snapping, so you can now:

  • Snap to current layer
  • Snap to all layers
  • Perform advanced per-layer snapping (previously the only mode)

New snapping modes

There’s also less confusion with snapping tolerances being to map units or layer units, so those anticipated 1 metre tolerances don’t become 1 degree tolerances without you knowing.

Snapping is not only more straightforward - it’s now much faster too thanks to using an index of geometries built when first needed. Because everybody loves faster, right?

For our developer colleagues there is an easy to use API which - amongst other things - adds fast point in polygon queries.

Funded by Ville de Vevey, SITNyon and QGIS Usergroup Switzerland.

More info here.

Simplify Simplified

QGIS has had a simplify tool for a while, but now it’s been significantly improved with lots of tasty new ingredients.

Improved simplify tool

Simplify tool tolerances used to be different for every feature - but not any more, and users can specify exact tolerances - which can be in map units or layer units.

Also new is on-the-fly reprojection support, tolerance settings are now persistent between sessions, dragging a rectangle instead of just clicking to simplify multiple features, support for multi-part features, and statistics about reduction of the number of vertices - because if you’re simplifying, it’s nice to know by how much! Oh yes, and faster too!

Work carried out in cooperation with Kartoza.

One Legend To Rule Them All

The display of rules in the legend for rule-based rendering has continued its evolution in features from QGIS 2.4:

QGIS 2.4: Good - a flat list of rules, but you can’t see the nesting.

Rules in 2.4

QGIS 2.6: Better - shown as a pseudo-tree with checkboxes.

Rules in 2.6

QGIS 2.8: Best - great for power-users who use complex styling rules to give their maps that extra zing.

Rules in 2.8

Funded by SIGE.

Long-Term Love

QGIS is also the first Long-Term Release (LTR) version - starting with 2.8 every third release is a long-term-release - maintained until the next long-term-release occurs.

Even if you’re a new-feature-junkie, you’ll appreciate the commitment by the project to both innovation and stability, and this of course underlines the burgeoning success of QGIS in enterprise environments.

So, start your engines, and get ready to download!

Making It Happen

While you’re still reeling from finding out about some of the new goodies that QGIS 2.8 brings (and check the official QGIS changelog for more!), I’ll remind you that all this cool stuff somehow needed to be paid for.

Thanks go to the organisations and individuals that sponsor or contribute to the QGIS project, and those that fund development of specific features. They make the software better for everyone, and you or your organisation could become one of them!

We at Lutra Consulting are Bronze level sponsors of the QGIS project, and use our development experience to contribute code and develop plugins for QGIS.

We tackle this in several ways, including direct approaches from clients to fund custom development, and crowdfunding campaigns. So, if you’re interested in some functionality that isn’t yet there - get in touch!

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows. Screenshots of the Mergin Maps mobile app for Field Data Collection
Get it on Google Play Get it on Apple store
Learn More