QGIS Code Sprint - FOSS4G UK - 2016

May 31, 2016 7:00 AM UTC by Lutra Consulting

The Open Source community in the UK will get together for another great FOSS4G meeting from 14 to 16 of June in Southampton.

QGIS has already established itself as one of the major Open Source GIS applications. To help the community to contribute back to the project, we are running a code sprint.

The code sprint session will not require prior knowledge of coding, so all participants are welcome to join. Make sure you bring your laptop!

There will be several power users and plugin developers to help with the code sprint. There will be 3 topics covered during the code sprint:

Documentation
Helping with the bug fixing
Contributing to qgis2web plugin

Documentation

If you are interested in helping with the QGIS documentation, we will run a short presentation to introduce you to the workflow.

Participants are required to have installed and configured the following software packages:

Latest QGIS master
Gitub account
Git software
A text editor software (Notepad++)
An image manipulation software (GIMP)

Helping with the bug fixing

For those who are interested in findings bugs, testing existing bugs and leaning up the bug queue, you will need:

An OSGeo ID
- Currently requires contacting OSGeo admins on IRC (#osgeo) in order to get the registration form (!)
Multiple versions of QGIS
- QGIS LTR (LTS)
- QGIS release
- QGIS master

If you are familiar with coding (in C++ and python) and would like to fix some bugs, you will need:

QGIS development environment
Github account

##Contributing to qgis2web plugin##

qgis2web

Help improve qgis2web by joining any of these activities:

fix bugs
develop new features
write tests
test for bugs
improve documentation

What you’ll need

laptop
QGIS, ideally current or nightly
a Github account
whatever Git/Github client you prefer
your favourite text editor

Details of some specific bugs and features are listed in a dedicated Github issue. If you are not coming to FOSS4GUK 2016, we’d still really value any help you can give. We’ll keep an eye on Github, or try the qgis2web Gitter.

This is a great opportunity to take your first steps in contributing to an open-source project. Come and help - we’ll tell you everything you need to know, and help you learn the skills you don’t yet have. Rest assured that we will be at least as grateful as you for the help you give.

If you are a beginner, @archaeogeek is also running a great workshop at FOSS4GUK 2016 on getting started in open-source: Don’t be afraid to commit

Learn More

Using OS AddressBase for Address Search in QGIS

May 27, 2016 12:00 PM UTC by Lutra Consulting

In this blog post we’ll learn how to use Ordnance Survey AddressBase data with the QGIS Discovery plugin for searching addresses.

Discovery Plugin for QGIS

Before we start

The AddressBase data will be loaded into a PostGIS table for Discovery to query. At this stage we should already have a functioning PostgreSQL / PostGIS installation.

A previous blog post describes how to quickly set up such an environment.

Creating the addressbase table

Let’s now create a table for storing the addressbase data. In the example below we’ll create a table called addressbase in the os_address schema.

The script below can be executed through pgAdminIII.

To run the script:

Open pgAdminIII
Connect to your destination database
Select Query tool from the Tools menu
Paste the code below into the Query tool
Press F5 to execute the query (it may take a few seconds to complete)

When the query has finished you should see Query returned successfully with no result in … seconds. in the Messages panel:

pgAdminIII Messages Panel

At this point we should be able to locate the new addressbase table within the os_address schema:

addressbase Table

If you can’t see the schema / table you probably need to refresh the schemas / tables views in pgAdminIII’s Object browser panel by hitting F5.

-- Create the destination schema if required
CREATE SCHEMA IF NOT EXISTS os_address;

-- Create a function which will populate the full_address and geom columns as
-- data are imported
CREATE OR REPLACE FUNCTION create_geom_and_address()
RETURNS trigger AS $$
BEGIN
  -- The geometry
  -- Set it based on the x_coord and y_coord fields
  NEW.geom = ST_SetSRID(ST_MakePoint(NEW.x_coordinate, NEW.y_coordinate), 27700);
  -- The full address
  -- Initialise it
  NEW.full_address = '';
  -- Build the full address by only including optional address components if they
  -- exist
  IF NEW.organisation_name IS NOT NULL AND length(NEW.organisation_name) > 0 THEN
	NEW.full_address = NEW.full_address || NEW.organisation_name || ', ';
  END IF;
  IF NEW.department_name IS NOT NULL AND length(NEW.department_name) > 0 THEN
	NEW.full_address = NEW.full_address || NEW.department_name || ', ';
  END IF;
  IF NEW.po_box_number IS NOT NULL AND length(NEW.po_box_number) > 0 THEN
	NEW.full_address = NEW.full_address || NEW.po_box_number || ', ';
  END IF;
  IF NEW.sub_building_name IS NOT NULL AND length(NEW.sub_building_name) > 0 THEN
	NEW.full_address = NEW.full_address || NEW.sub_building_name || ', ';
  END IF;
  IF NEW.building_name IS NOT NULL AND length(NEW.building_name) > 0 THEN
	NEW.full_address = NEW.full_address || NEW.building_name || ', ';
  END IF;
  IF NEW.building_number IS NOT NULL THEN
	NEW.full_address = NEW.full_address || NEW.building_number || ', ';
  END IF;
  IF NEW.dependent_thoroughfare IS NOT NULL AND length(NEW.dependent_thoroughfare) > 0 THEN
	NEW.full_address = NEW.full_address || NEW.dependent_thoroughfare || ', ';
  END IF;
  IF NEW.thoroughfare IS NOT NULL AND length(NEW.thoroughfare) > 0 THEN
	NEW.full_address = NEW.full_address || NEW.thoroughfare || ', ';
  END IF;

  NEW.full_address = NEW.full_address || NEW.post_town || ', ';

  IF NEW.double_dependent_locality IS NOT NULL AND length(NEW.double_dependent_locality) > 0 THEN
	NEW.full_address = NEW.full_address || NEW.double_dependent_locality || ', ';
  END IF;
  IF NEW.dependent_locality IS NOT NULL AND length(NEW.dependent_locality) > 0 THEN
	NEW.full_address = NEW.full_address || NEW.dependent_locality || ', ';
  END IF;

  NEW.full_address = NEW.full_address || NEW.postcode;

  RETURN NEW;
END;
$$ LANGUAGE 'plpgsql';

-- Drop any existing addressbase table
DROP TABLE IF EXISTS os_address.addressbase CASCADE;
CREATE TABLE os_address.addressbase
(
  -- id will be the primary key, populated automatically
  id serial NOT NULL,
  uprn bigint NOT NULL,
  os_address_toid varchar(24) NOT NULL,
  -- os_address_toid bigint NOT NULL,
  udprn integer NOT NULL,
  organisation_name varchar(60),
  department_name varchar(60),
  po_box_number varchar(6),
  sub_building_name varchar(30),
  building_name varchar(50),
  building_number smallint,
  dependent_thoroughfare varchar(80),
  thoroughfare varchar(80),
  post_town varchar(30) NOT NULL,
  double_dependent_locality varchar(35),
  dependent_locality varchar(35),
  postcode varchar(8) NOT NULL,
  postcode_type char(1) NOT NULL,
  x_coordinate numeric(8,2) NOT NULL,
  y_coordinate numeric(9,2) NOT NULL,
  latitude numeric(9,7) NOT NULL,
  longitude numeric(8,7) NOT NULL,
  rpc char(1) NOT NULL,
  country char(1) NOT NULL,
  change_type char(1) NOT NULL,
  la_start_date date NOT NULL,
  rm_start_date date NOT NULL,
  last_update_date date NOT NULL,
  class char(1) NOT NULL,
  -- the next two fields are populated automatically on insert
  full_address text NOT NULL,
  geom geometry(Point,27700) NOT NULL,
  CONSTRAINT addressbase_pkey PRIMARY KEY (id)
)
WITH (
  OIDS=FALSE
);

-- Create a pg_trgm index on the full_address column
-- This will allow super-fast, case-insensitive search on the column
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX addressbase_full_address_gin_trgm
  ON os_address.addressbase
  USING gin
  ("full_address" gin_trgm_ops);

-- Spatial index for the geometry column
CREATE INDEX addressbase_geom_gist
  ON os_address.addressbase
  USING gist
  (geom);

-- trigger to create points and addresses
-- This trigger will be executed on each row inserted, calling the function defined above
CREATE TRIGGER tr_create_geom_and_address BEFORE INSERT
  ON os_address.addressbase
  FOR EACH ROW
  EXECUTE PROCEDURE create_geom_and_address();

The script above has:

Created a table
Added any necessary indices
Created two additional, derived columns, full_address and geom

full_address will be used to store various address components into a sensible, human readable address. geom will be used to store point geometry based on address eastings/northings.

See the script comments for more information / detail.

Loading AddressBase

At this point we have an empty table ready to accept our AddressBase data. We will now import the data using pgAdminIII. Extract the CSV files for the addresses, you should end up seeing one or more CSV files, for example AddressBase_FULL_2016-03-19_001.csv

In pgAdminIII:

Locate the addressbase table
Right click it, select Import

An import dialog should appear. Select the first CSV file and set the settings in the File Options tab as shown here:

AddressBase Import Options 1

Uncheck the id, full_address and geom columns in the Columns tab as shown here:

AddressBase Import Options 2

Click Import. After a few seconds the dialog may report (Not Responding). This is nothing to worry about, be patient.

When the import process completes, close the import dialog and repeat the above steps with any remaining CSV files.

At this stage the data has been imported and the full_address field should contain sensible, human-readable addresses.

Configuring Discovery

With the data loaded in QGIS, we can now configure Discovery to make use of it.

Install the Discovery plugin if not already installed
Open Discovery’s settings using the button
Set the settings as follows, changing the Scale Expression if required

Discovery Settings for OS AddressBase

Congratulations! QGIS should now be set up to search your AddressBase data.

Learn More

QGIS Report Plugin Release

May 12, 2016 9:00 AM UTC by Lutra Consulting

We develop several public and private plugins for QGIS users and our clients. During the testing phase, they often come across some bugs. To enable them to report the issues and include the right type of information, we have developed The Report plugin to simplify the process Report plugin.

The Report plugin helps developers get better bug reports from users, by auto-populating the relevant information. It also simplifies the bug reporting process for users.

The Report plugin can automatically extract information from the plugin crash as well as other important information for developers. With just one click you can report the issue to the official plugin issue tracker. No need of searching for a plugin’s tracker on the internet or copy-pasting long tracebacks to your browser.

bug report plugin

Installation

Install Report plugin from QGIS plugin offical repository
Create a GitHub account if you do not have any existing
Create a public access token GitHub with public_repo scope
Click on the Report plugin toolbar button, Configure link and copy generated GitHub access token to the configuration dialog.
Wait for next error and report the problem to the developers

Testing

Feel free to download DevNull plugin from QGIS plugin offical repository to be able to test Report plugin
After installation fo DevNull plugin, click on the new button /dev/null in your toolbar and watch how report plugin catches the exception
Report any number of testing issues to DevNull Issue tracker, they are going to be deleted anyway!

Notes

Report plugin works only with GitHub issue trackers
Plugins must have “tracker” metadata filled in, so the Report plugin can detect the plugin issue tracker correctly

When/if QGIS issue tracker moves to GitHub, a similar tool can be added to the core to automatically report bugs and crashes.

Learn More

Setting up and Configuring Discovery plugin for QGIS

April 25, 2016 9:00 AM UTC by Lutra Consulting

As a part of migrating to Open Source GIS, the Newcastle City Council has commissioned us to create a user friendly gazetteer plugin.

In this post, we will import the OS Open Names and configure the Discovery plugin to use the data.

In the following sections, we are going to set up a PostGIS database and import the OS Open Names, freely available from here.

To use the data with the Discovery plugin, we need to first set up a PostGIS database and load the data inside the geo-databases.

If you have an existing Postgresql/PostGIS server, you can skip the next section.

Postgresql/PostGIS installation and configuration

Installation

For MS Windows users, download and install Postgresql from here (32-bit) or here (64-bit) install the software.

During the installation, select the StackBuilder to install PostGIS, under Spatial Extensions.

If your StackBuilder fails to download PostGIS (in case your proxy server blocks it), you can download and install it manually.

Preparing the database

Now that installation is successful, create a new database as osdata. Make sure you add PostGIS extension to your database. You can do that by simply running the following command in the Query editor:

CREATE EXTENSION postgis;

Create osopennames as a new schema under osdata.

Preparing data

In this example, we are focusing on the OS Open Names comes (for Great Britain only) which comes in a zip file containing several CSV files.

If you’d like to use address data for your part of the world, you can visit the OpenAddress website for the global coverage. Note that the CSV files for this dataset come with the vrt files, so you can skip the vrt creation and directly use ogr2ogr.

Once the zip file extracted, there are 2 folders, one containing the header file (DOC) and the other containing the csv files (DATA).

To be able to import all the csv files in PostGIS, we can merge all the files including the header file.

You can move the header file (OS_Open_Names_Header.csv) from the DOC folder. To ensure, the file will appear first during the merge process you can rename it to 1_OS_Open_Names_Header.csv.

You can use Windows command prompt to merge the files. The following command merges all csv files to all_open_names.csv:

 copy /b \*.csv all_open_names.csv

Loading data in PostGIS

There are several methods to import all_open_names.csv in PostGIS:

Adding it as a delimited text layer in QGIS and then load it in PostGIS
Importing it in PostGIS as a CSV and then using PostGIS’ geometry to create points
Using virtual layer and OGR2OGR library

In the example below, we explore the third option.

To create a virtual vector layer from your csv file, open a text editor, copy and paste the following lines and save it as all_open_names.vrt under DATA folder along with your csv file.

	<OGRVRTDataSource>
    	<OGRVRTLayer name="all_open_names">
        	<SrcDataSource relativeToVRT="1">all_open_names.csv</SrcDataSource>
        	<GeometryType>wkbPoint</GeometryType>
        	<LayerSRS>EPSG:27700</LayerSRS>
        	<GeometryField encoding="PointFromColumns" x="GEOMETRY_X" y="GEOMETRY_y"/>
    	</OGRVRTLayer>
	</OGRVRTDataSource>

In fact, you can use the virtual vector layer to merge all your csv files and skip the previous section!

You can then use ogr2ogr command from the OSGeo4W shell to import it to your PostGIS:

    ogr2ogr -append  -a_srs EPSG:27700 -f "PostgreSQL" PG:"host=127.0.0.1 user=postgres dbname=osdata password=postgres active_schema=osopennames" -nln osnames all_open_names.vrt

Note: if you are using the OpenAdress data make sure you assign the right EPSG (4326) in the above command.

Configuring Discovery plugin

First you need to install the plugin from the QGIS plugin repository.

Once the plugin installed, you should have a new toolbar. Click on discovery from the toolbar to open the configuration:

For Connection, select OS Data
For Shema, select opennames
For Table, select osnames
For Search Column, select Name1
Select the option to Echo Search Column in Results
For Display Columns select the followings:
1. Name2
2. DISTRICT_BOROUGH
3. POSTCODE_DISTRICT
4. COUNTRY
For Geometry Column, select geom (or other columns depending on your QGIS or OGR versions)
For BBOX Expresssion, type the following:

CASE
	WHEN "MBR_XMIN" IS NOT NULL
THEN
	MBR_XMIN || ',' ||
	MBR_YMIN || ',' ||
	MBR_XMAX || ',' ||
	MBR_YMAX
END

And you should be able to start using the gazetteer plugin after pressing OK.

For performance enhancement and other tips visit this page.

Learn More

Agenda for 5th QGIS user group – Scotland

April 21, 2016 4:27 PM UTC by QGIS UK under user groups qgis scotland

scottish thistle The 5th QGIS user group meeting in Scotland takes place next Wednesday at the University of Glasgow. It is being hosted by the School of Geographical and Earth Sciences and has been generously sponsored by thinkWhere and Ordnance Survey. You can find the draft programme of talks and presentations here: 5th-QGIS-user-group-programme

All tickets are now gone but get on the waitlist and you may be lucky.

See you all there!

Learn More

Crayfish 2.2: New features

April 14, 2016 9:00 AM UTC by Lutra Consulting

Crayfish 2.2 is out with lots of new features and new formats.

Plotting time series and cross sections

Profile tool plugin deemed not to be suitable for plotting time series. It lacks several features for visualising plots over s specific grid and generally to user interface is not very intuitive.

With the new plot options, user can plot time series at a point from the map or using an existing layer to generate plots. The tool is very flexible and allows user to export the layer to another format. In addition to time series, you can also generate cross section plots. Video below demonstrates the new feature in action:

As you can see, in the above video, more points can be added to the plot. To do so, you need to hold down CTRL key.

GRIB and NetCDF support

We have decided to expand the user base for Crayfish and allow meteorologists and oceanographers to view their temporal unstructured GRIB or NetCDF directly in Crayfish. Examples of those datasets can be found here and here

Visualising global temperature datasets (GRIB fromat) in Crayfish

Support for TELEMAC and HEC RAS 2D

We also added suppord for HEC RAS 2D and TELEMAC. Our initial benchmark shows that Crayfish is much faster than an other TELEMAC plugins we used. Plus, Crayfish is truly open source!

More vector, contour and mesh options

You can now filter vectors to display only values above/below a certain threshold.

Contour label and values in Crayfish are now similar to raster styling in QGIS. This will allow users to generate better legend directly from Crayfish for print or animation outputs.

For debugging, we have added option to allow users to label mesh and change symbology.

Mesh labels in Crayfish

We’d like to thank The Laboratory of Hydraulics, Hydrology and Glaciology (VAW) of ETH Zurich for sponsoring plotting feature.

Learn More

FOSS4G UK

April 12, 2016 10:37 AM UTC by QGIS UK under user groups foss4g uk osgeo

foss4guk_2016_logo

There is a FOSS4G UK conference, unconference, workshop, hackathon, code sprint and party happening in Southampton in June this year. If you are free from the 14th to 16th then this is a fantastic opportunity to come and find out more about free and open-source software for Geo. Have a look at the OSGeo site (http://uk.osgeo.org/foss4guk2016/) for more information. If you want to submit a talk and/or workshop then you can do so through the site. Early bird tickets will be going on sale tomorrow (13 April) and will be available through the site from Eventbrite.

Learn More

Verarbeitung von Interlis-Daten mit QGIS

March 22, 2016 12:00 AM UTC by Sourcepole

In der Schweiz werden amtliche Geodatenmodelle vorwiegend mit Interlis erstellt und häufig wird das Interlis-Transferformat für den offiziellen Datenaustausch vorgegeben. Das Interlis-Plugin für QGIS ermöglicht die einfache Verarbeitung von Interlis-Daten und bindet externe Java-Applikationen in die Processing-Toolbox ein.

Learn More

Visualising dam breach in QGIS

March 6, 2016 10:00 AM UTC by Lutra Consulting

Recently, I have come across the news about extent of damage and human loss, in case of the Mosul dam breach in Iraq.

Having all the tools, some freely available data and a free weekend, I have decided to carry out my own analysis.

Data

For the Digital Elevation Model (DEM), I used the data from JAXA Earth Observation Center (EORC).

The resolution of the DEM is approximately 30 metres. For more detailed analysis, a better dataset (e.g. LIDAR) would be ideal.

Methodology

I set up a 2-dimensional model with the following assumptions and parameters:

A square cell size (70 metres)
The dam was assumed to be full
Simulation time of 24 hours
The breach would occur after the first hour (1:00 hour in to the model run)
The dam would be completely breached after 1 hour (between 1:00 - 2:00 hour during the model run)

Results

Using Crayfish plugin for QGIS, the results can be animated to see the on-set of the flooding.

Note, that the first hour of the model run was used to initialise the water level within the dam. Hence, for the actual flood travel time, one hour should be deducted from the timing shown towards the bottom right corner of the video.

Below, you can see the flood extent generated based on my calculations and the extent shown here (red line in the main map) for Mosul.

Maximum flood extent due to the breach of the upstream dam in Mosul. (Click to enlarge)

Disclaimer: The calculation carried out for this demo was very approximate. Despite obtaining very similar results for Mosul compared to the original study, further checks and revisions are required.

Learn More

Installing Third-party Python Modules in QGIS (Windows)

March 2, 2016 12:00 PM UTC by Lutra Consulting

We’ve recently been developing a client plugin for forest management which relies on the Python SQLAlchemy module. When I installed the latest version of my colleague’s work I got a Python traceback error about a missing Python module (SQLAlchemy).

This blog is more of a note to myself for future reference of how to install third-party modules in QGIS’ Python environment.

The Problem

Installing third-party modules in Windows is usually quite straight-forward, either download an installer (which will find the Python environment from the registry) or use easy_install from the Python setuptools.

The problem is that QGIS ships with its own Python installation that these methods cannot easily add to.

(Don’t get me wrong - I like the fact that each QGIS instance has its own Python environment as it keeps each version very self-contained).

The Method

So here’s the method I successfully used to install SQLAlchemy (which I’ll use here to install the lxml module):

Click Start
Type QGIS
Wait for the QGIS version / instance you want to modify to appear:
Right-click it, select Run as administrator
In QGIS, select Python Console from the Plugins menu
Download ez_setup.py from here.
Run the following code on the python console in QGIS:

from subprocess import call
# Replace the path below to the location of ez_setup.py that
# you just downloaded
call(['python', r'C:\Users\Pete\Downloads\ez_setup.py'])

# This will install *setuptools* which is a package manager
# The previous command should return 0 on success

# Replace lxml with the package you wish to install
call(['easy_install', 'lxml'])

# Again this will return 0 on success

Restart QGIS (normally, not using the Run as administrator option)

The new module should now be available, we can test this (again on the Python console in QGIS):

import lxml

That’s it!

Learn More

QGIS Planet

Documentation

Helping with the bug fixing

What you’ll need

You may also like...

Before we start

Creating the addressbase table

Loading AddressBase

Configuring Discovery

You may also like...

Installation

Testing

Notes

You may also like...

Postgresql/PostGIS installation and configuration

Installation

Preparing the database

Preparing data

Loading data in PostGIS

Configuring Discovery plugin

You may also like...

Plotting time series and cross sections

GRIB and NetCDF support

Support for TELEMAC and HEC RAS 2D

More vector, contour and mesh options

Sponsor

You may also like...

Data

Methodology

Results

You may also like...

The Problem

The Method

You may also like...