OSGeo Statement: EU Cyber Resilience Act

QGIS Planet

(Fr) Oslandia recrute : Ingénieur(e) développement d’applications SIG ( Python / SQL / QGIS )

Sorry, this entry is only available in French.

Learn More

QGIS Abstract Connections API

 

The goal of the new API is twofold:

  1. provide a unified way to store and retrieve data provider connections in the QGIS settings
  2. provide an abstract set of methods to perform most common operation on DB data sources (e.g. browse tables, drop/create a table/schema, run arbitrary SQL commands etc.)

 

The new API is documented in https://qgis.org/api/classQgsAbstractProviderConnection.html and it provides a few specializations for DB connections (https://qgis.org/api/classQgsAbstractDatabaseProviderConnection.html) and an initial PR implementation for web service-based connections (https://github.com/qgis/QGIS/pull/33045).

 

While the whole of the desired refactoring work was too big for a single grant request, the first work package has been completed and the following data providers have been partially or totally refactored to make use of the new connections API:

  • postgres
  • geopackage (OGR)
  • spatialite

 

The new API was also used to implement the automatic loading of layer dependencies (not part of the grant program).

 

For developers interested in working with the new API, a set Python tests are available to show how to use the methods:  https://github.com/qgis/QGIS/blob/master/tests/src/python/test_qgsproviderconnection_ogr_gpkg.py (see also the postgres and spatialite companion tests).

 

There is still a large amount of work to be done in order to complete all the desired refactoring and to remove all the Python and C++ code that will be ultimately be made redundant. In particular, future work should be undertaken to:

  • port all remaining data providers to the new API
  • refactor and eliminate the remaining DB-manager connectors to make use of the abstract API
  • eliminate duplicate and untested code inside the Processing framework for working with Postgres databases and port the code to the new, stable, well-tested API
  • refactor and eliminate the remaining QGIS browser data items to make use of the abstract API 

 

For further information, the following paragraphs (taken from the original grant proposal) will provide full details about the background of this work.

Background/motivation

  • DB-Manager is an important part of the QGIS interface, which allows browsing/previews of different DB-based data sources, complex queries, management of layers, import-export etc., DB creation and deletion etc.
  • After the QGIS 3.0 release, improvements within the core browser widgets implemented in C++ have resulted in a (constantly growing) degree of overlapping functionality between the browser and db manager.
  • After QGIS 3 API improvements concerning layer import and export functionality, there are many duplicated implementations between browser and db manager – some functionality is better in browser, some functionality is better in db manager. Users are forced to choose between two competing semi-complete alternatives, instead of having one, complete, well integrated solution.
  • There are no unit tests for DB-Manager and this leads to frequent regressions, which (aside from being frustrating for users) consume a substantial part of our development time and budget during the bugfixing programs. Furthermore the nature of large Python codebases like db manager makes it very easy to accidentally break functionality with no warning or errors during development.

 

Proposed solution

We propose to start refactoring the DB-manager plugin functionality into core C++ implementation, reusing existing core API and replacing redundant duplicate functionality.

The clear advantages are:

  • no duplicate functionality, so it’s easier for users to understand and use
  • more usage of well-tested and well-maintained core C++ API
  • testability and immediate feedback on API breaks (an advantage of C++ is that the application won’t even build if an API is changed or accidentally misused)
  • better performance
  • the ability to expose database management functionality via stable PyQGIS API, allowing other plugins and scripts to utilise this functionality. In future, Processing algorithms may also be developed which would take advantage of these functions (e.g. “create schema”, “drop table”, “vacuum table” algorithms)
  • DB management functionality would be available within the main QGIS window (from the Browser panel), instead of as a separate dialog.

 

Grant proposal package

The above mentioned work is too large to be completed within a single grant, so what we propose here is to start the refactoring needed in order to have a core stable C++ API that can be used by the application and the plugins and that will be available to fully move DB manager to C++ API in the future avoiding duplication of code and functionality.

  • create an interface for databases that expose the required functions to a coherent API
  • add missing tests and documentation for the a.m. API
  • porting some basic functions from db manager to the new api:
    • create table (with native field types support)
    • create schema
    • delete table
    • Rename table

The API will be exposed through the browser and it will be used by the DB manager instead of the Python implementation that is currently used.

The post QGIS Abstract Connections API first appeared on Open Web Solutions, GIS & Python Development.

Learn More

Overview of QGIS 3.12 Mesh Features

Ready for 3D meshes, vector streamlines or contour export?

The releases of QGIS 3.12, MDAL 0.5.0 and Crayfish 3.2.1 are planned for end of February 2020. We are proud to present you few of upcoming features we implemented for this release:

  • vector trace animation
  • 3D stacked meshes
  • mesh calculator enhancements
  • export contours
  • various smaller enhancements (reference time support, resampling, export plot data, mdal_translate utility)

If you are hesitant to wait till end of February, feel free to get nightly build and test it out!

Do you want to use QGIS Mesh Layers in your projects? Read more…

Support for vector trace animation and streamlines (QGIS)

Last feature from QGIS 2.x/Crayfish 2.x series that was not ported to QGIS 3 is finally available. You would be able to visualize streamlines and particles for vector datasets in mesh layers. In QGIS main menu, under Mesh>Crayfish>Export Trace you are also able to export animation with the particle traces to various video formats

Trace animation

This feature was funded by TUFLOW

Support for 3d Stacked Meshes (e.g. TUFLOW FV format)

MDAL and QGIS now supports 3D Stacked Meshes, particularly for TUFLOW-FV format. For this release, you need to choose appropriate averaging method in the QGIS interface and you are able to browse the data similarly to any other 2D dataset. 3d stacked

In Crayfish 3.2.1, you can create plots of the profile showing the variation along Z-axis.

3d stacked plot

The technical description can be found in the following QEP

This feature was funded by TUFLOW

On the fly resampling of data defined on faces to vertices

For datasets defined on faces, one can choose to interpolate data to vertices with neighbour average method. When no data interpolation method is chosen, each pixel on a single face has a single value/color. With data on vertices, the rendering for each pixel is interpolated from the values on the vertices, making smoother figures.

Use mesh contours styling panel to switch between the data interpolation methods.

No Mesh Data Resampling Dialog Mesh Data Resampling Mesh Data Resampling Dialog

This feature was funded by Austrian Ministry of Agriculture, Forestry, Environment and Water Management

Smooth export of the contours (Crayfish processing algorithm)

We have implemented a new algorithm in QGIS’s analysis library to export directly contour lines and polygons. The method is not based on GDAL as it was in the Crayfish 2.x releases. It is both faster and with smoother shapes, matching rendered images from QGIS. You can find the new processing algorithm in Crayfish processing toolbox.

Mesh Contours

This feature was funded by Austrian Ministry of Agriculture, Forestry, Environment and Water Management

Support of datasets defined on faces in QGIS Mesh Calculator

From QGIS 3.12 you can use mesh calculator for all datasets, both defined on faces and vertices. Additionally, it allows users to store the result of mesh calculator under different name or format. This allows for example to work with FLO-2D or HEC-RAS data in the QGIS mesh calculator

This feature was funded by Austrian Ministry of Agriculture, Forestry, Environment and Water Management

Support for reference time (QGIS)

For various dataset type, for example GRIB and NetCDF, the reference time in QGIS time settings dialog is prepopulated from the raw data and does not need to be set manually. Also we fixed various bugs related to time parsing, so in QGIS 3.12 it should be possible to format and show your time in plots/animations in proper way.

Reference Time

This feature was funded by TUFLOW

Support for conversion of 2dm to UGRID mesh (mdal_translate utility)

MDAL library now has a new utility: mdal_translate. For now, use can use the utility to convert text-based 2dm mesh definition files to UGRID NetCDF/HDF5 binary-based format and save up to 80% disk and speed up loading of your mesh by similar amount.

This feature was funded by TUFLOW

Support for export of 2D plot data (processing)

With Crayfish 3.2.1 you can export your time series or cross section raw dat to CSV format for further processing.

This feature was funded by Lutra Consulting

You may also like...

Mergin Maps, a field data collection app based on QGIS. Mergin Maps makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows. Screenshots of the Mergin Maps mobile app for Field Data Collection
Get it on Google Play Get it on Apple store
Learn More

QGIS Snapping improvements

A few months ago, we proposed to the QGIS grant program to make improvements to the snap cache in QGIS. The community vote selected our project which was funded by QGIS.org. Developments are now mostly finished.

In short, snapping is crucial for editing geospatial features. It is the only way to ensuring they are topologically related, ie, connected vertices have exactly the same coordinates even if manual digitizing on screen is imprecise by nature.  Snapping correctly supposes QGIS have in memory an indexed cache of the geometries to snap to. And maintainting this cache when data is modified, sometimes by another user or database logic, can be a real challenge. This it exactly what this work adresses.

The proposal was divided into two different tasks:

  • Manage circular dependencies
  • Relax the snap cache index build

Manage cicular data dependencies

Data dependencies

Data dependency is an existing feature that allows you to configure QGIS to reload layers (and their snapping cache) when a layer is modified.

It is useful when you store your data in a database and you set up triggers to maintain consistency between the different tables of your data model.

For instance, say you have topological informations containing lines and nodes. Nodes are part of lines and lines go through nodes. Then, you move a node in QGIS, and save your modifications to the database. In order to keep the data consistent, a trigger updates the geometry of the line going through the modified node.

Node 2 is modified, Line 1 is updated accordingly

QGIS, as a database client, has no information that the line layer currently displayed in the canvas needs to be refreshed after the trigger. Although the map canvas will be up to date, because QGIS fetches data for display without any caching system, the snapping cache is not and you’ll end up with ghost snapping highlights issues.

Snapping highlights (light red) differ from real line (orange)

Defining a dependency between nodes and lines layers tells QGIS that it has to refresh the line layer when a node is modified.

Dependencies configuration: Lines layer will be refreshed whenever Nodes layer is modified

It also have to work the other way, modifying a line should update the nodes to ensure they still are on the line.

Circular data dependencies

So here we are, lines depend on nodes which depend on lines which depend on nodes which…

That’s what circular dependencies is about. This specific behavior was previously forbidden and needed a special way to deal with it. Thanks to this recent development, it is now possible.

It’s also possible to add the layer itself as one of its own dependencies. It helps dealing with specific cases where one feature modification could lead to a modification of another feature in the same layer (to keep consistency on road networks for instance).

Road 2 is modified, Road 1 is updated accordingly

This feature is available in the next QGIS LTR version 3.10.

Relax the snapping cache index build

If you work in QGIS with huge projects displaying a lot of vector data, and you enable snapping while editing these data, you probably already met this dialog:

Snap indexing dialog

This dialog informs you that data are currently being indexed so you can snap on them while you will edit feature geometry. And for big projects, this dialog can last for a really long time. Let’s work on speeding it up!

What’s a snap index?

Let’s say you want to move a line and snap it onto another one. While you drag your line with the mouse, QGIS will look for an existing geometry beneath the mouse cursor (with a certain pixel tolerance) every time you move your mouse. Without spatial index, QGIS will have to go through every geometry in your layer to check if the given geometry is beneath the cursor position. This would be very ineffective.

In order to prevent this, QGIS keeps an index where vector data are stored in a way that it can quickly find out what geometry is beneath the mouse cursor. The building of this data structure takes time and that is what the progress dialog is about.

Firstly: Parallelize snap index build

If you want to be able to snap on all layers in your project, then QGIS will have to build one snap index for each layer. This operation was made sequentially meaning that if you have for instance 20 layers and the index building last approximatively 3 seconds for each, then the whole index building will last 1 minute. We made modifications to QGIS so that index building could be done in parallel. As a result, the total index building time could theoretically be 3 seconds!

4 layers snap index being built in parallel

However, parallel operations are limited by the number of CPU cores of your machine, meaning that if you have 4 cores (core i7 for instance) then the total time will be up to 4 times faster than when the building is sequential (and last 15 seconds in our example).

Secondly: relax the snap build

For big projects, parallelizing index building is not enough and still takes too much time. Futhermore, to reduce snap index building, an existing optimisation was to build the spatial index for a specific area of interest (determined according to the displayed area and layer size). As a consequence, when you’ve done waiting for an index currently building and you move the map or zoom in/out, you could possibly trigger another snap index building and wait again.

So, the idea was to avoid waiting at all. Snap index is now built whenever it needs to (when you first enable snapping, when you move or zoom) but the user doesn’t have to wait for the build to be over and can continue what it was doing (creating feature, moving…). Snapping highlights will be missing when the index is currently being built and will appear gradually as soon as they finished. That’s what we call the relaxing mode.

No waiting dialog, snapping highlights appears as soon as snap index is ready

This feature has been merged into current QGIS master and will be present in future QGIS 3.12 release. We keep working on this feature in order to make it more stable and efficient.

What’s next

We’ll continue to improve this feature in the coming days, if you have the chance to test it and encounter issues please let us know on the QGIS tracker. If you think about a missing feature or just want to know more about QGIS, feel free to contact us at infos+data@oslandia.com. And please have a look at our support offering for QGIS.

Many thanks to QGIS grant program for funding these new features. Thanks also to all the people involved in reviewing the code and helping to better understand the existing mechanism.

 

Learn More

Tracking, calculating and merging vector changes with Input and QGIS

The latest beta release of Input (v.0.4.90) comes with smart diff support for vectors. This will allow you to use the app (current beta version) in a collaborative environment, where several users can make changes to a single survey layer (geopackage).

What does it mean?

It is often the case, where multiple users need to make changes to a single vector layer. If you work in office, this issue is usually addressed by having a central geodatabase (e.g. Postgres/PostGIS). If you want extra information (e.g. audit trails, versioning, latest changes, etc), you can modify your database, to keep track of it.

The problem arises when you want to collect data, without having access to the central geodatabase. You can do some manual work to handle this scenario, but it can easily lead into data management nightmare.

To simplify the workflow, we have developed Geodiff, a multi-platform library to keep track of changes, calculate the differences, merge and consolidate the differences.

How can I use it?

Geodiff has been integrated into the beta version of Input. This will allow you to share a project with your team and edit a single layer (geopackage format) all at the same time, even when you are offline.

To start with, you need to create a project and upload it to Mergin. You can then share the project (with write access) with your colleagues:

Sharing projects through Mergin

The project contains a survey layer (trees.gpkg). There is only one feature present within the layer:

Sharing projects through Mergin

The project is shared with two users. Both users download the project and take their devices to the field:

User 1, carried out a survey (using iPhone!), by adding a tree and editing the attribute table of the existing one. The changes were synchronised back through the Mergin:

Meanwhile, User 2 added a new feature to the survey layer. Once User 2 tries to synchronise the changes, Input automatically detects the changes not only made through User 2, but also the ones uploaded to the Mergin. The layer will be patched both locally and on the server to take all the changes into account:

The data administrator can now pull all the changes from both users in QGIS:

You can also see the history of changes to the project and the survey layer on the Mergin website. Below shows the changes from different users:

Sharing projects through Mergin

To see changes from each user, you can click on the version and it lists the changes. In this example, User 1 (jack) added a new feature and modified an existing feature:

Sharing projects through Mergin

You can also see the extended history and see where the changes have been made:

Sharing projects through Mergin

How can I test this new feature?

You can use Beta version of Input app in Android or TestFlight in iOS:

Get it on Google Play Get it on App Store

For any issues or feedback, please file a ticket on Input repository

Learn More

PostGIS Day 2019 Zürich

Am Donnerstag 14. November findet der PostGIS Day in Zürich statt! Neben topaktuellen News zu den Releases von PostGIS 3, QGIS 3.10 LTR und OpenLayers 6 gibt es Einblicke in die Responsive City Strategie der Stadt Zürich und weitere Themen wie Datentransformationen mit hale studio und Vektor Tiles.

Learn More

(Fr) Rechercher une adresse avec QGIS

Sorry, this entry is only available in French.

Learn More

Configure editing form widgets using PyQGIS

PT | EN

As I was preparing a QGIS Project to read a database structured according to the new rules and technical specifications for the Portuguese Cartography, I started to configure the editing forms for several layers, so that:

  1. Make some fields read-only, like for example an identifier field.
  2. Configure widgets better suited for each field, to help the user and avoid errors. For example, date-time files with a pop-up calendar, and value lists with dropdown selectors.

Basically, I wanted something like this:

Peek 2019-09-30 15-04_2

Let me say that, in PostGIS layers, QGIS does a great job in figuring out the best widget to use for each field, as well as the constraints to apply. Which is a great help. Nevertheless, some need some extra configuration.

If I had only a few layers and fields, I would have done them all by hand, but after the 5th layer my personal mantra started to chime in:

“If you are using a computer to perform a repetitive manual task, you are doing it wrong!”

So, I began to think how could I configure the layers and fields more systematically. After some research and trial and error, I came up with the following PyQGIS functions.

Make a field Read-only

The identifier field (“identificador”) is automatically generated by the database. Therefore, the user shouldn’t edit it. So I had better make it read only

Layer Properties - cabo_electrico | Attributes Form_103

To make all the identifier fields read-only, I used the following code.

def field_readonly(layer, fieldname, option = True):
    fields = layer.fields()
    field_idx = fields.indexOf(fieldname)
    if field_idx >= 0:
        form_config = layer.editFormConfig()
        form_config.setReadOnly(field_idx, option)
        layer.setEditFormConfig(form_config)

# Example for the field "identificador"

project = QgsProject.instance()
layers = project.mapLayers() 

for layer in layers.values():
    field_readonly(layer,'identificador')

Set fields with DateTime widget

The date fields are configured automatically, but the default widget setting only outputs the date, and not date-time, as the rules required.

I started by setting a field in a layer exactly how I wanted, then I tried to figure out how those setting were saved in PyQGIS using the Python console:

>>>layer = iface.mapCanvas().currentLayer()
>>>layer.fields().indexOf('inicio_objeto')
1
>>>field = layer.fields()[1]
>>>field.editorWidgetSetup().type()
'DateTime'
>>>field.editorWidgetSetup().config()
{'allow_null': True, 'calendar_popup': True, 'display_format': 'yyyy-MM-dd HH:mm:ss', 'field_format': 'yyyy-MM-dd HH:mm:ss', 'field_iso_format': False}

Knowing this, I was able to create a function that allows configuring a field in a layer using the exact same settings, and apply it to all layers.

def field_to_datetime(layer, fieldname):
    config = {'allow_null': True,
              'calendar_popup': True,
              'display_format': 'yyyy-MM-dd HH:mm:ss',
              'field_format': 'yyyy-MM-dd HH:mm:ss',
              'field_iso_format': False}
    type = 'Datetime'
    fields = layer.fields()
    field_idx = fields.indexOf(fieldname)
    if field_idx >= 0:
        widget_setup = QgsEditorWidgetSetup(type,config)
        layer.setEditorWidgetSetup(field_idx, widget_setup)

# Example applied to "inicio_objeto" e "fim_objeto"

for layer in layers.values():
    field_to_datetime(layer,'inicio_objeto')
    field_to_datetime(layer,'fim_objeto')

Setting a field with the Value Relation widget

In the data model, many tables have fields that only allow a limited number of values. Those values are referenced to other tables, the Foreign keys.

In these cases, it’s quite helpful to use a Value Relation widget. To configure fields with it in a programmatic way, it’s quite similar to the earlier example, where we first neet to set an example and see how it’s stored, but in this case, each field has a slightly different settings

Luckily, whoever designed the data model, did a favor to us all by giving the same name to the fields and the related tables, making it possible to automatically adapt the settings for each case.

The function stars by gathering all fields in which the name starts with ‘valor_’ (value). Then, iterating over those fields, adapts the configuration to use the reference layer that as the same name as the field.

def field_to_value_relation(layer):
    fields = layer.fields()
    pattern = re.compile(r'^valor_')
    fields_valor = [field for field in fields if pattern.match(field.name())]
    if len(fields_valor) > 0:
        config = {'AllowMulti': False,
                  'AllowNull': True,
                  'FilterExpression': '',
                  'Key': 'identificador',
                  'Layer': '',
                  'NofColumns': 1,
                  'OrderByValue': False,
                  'UseCompleter': False,
                   'Value': 'descricao'}
        for field in fields_valor:
            field_idx = fields.indexOf(field.name())
            if field_idx >= 0:
                print(field)
                try:
                    target_layer = QgsProject.instance().mapLayersByName(field.name())[0]
                    config['Layer'] = target_layer.id()
                    widget_setup = QgsEditorWidgetSetup('ValueRelation',config)
                    layer.setEditorWidgetSetup(field_idx, widget_setup)
                except:
                    pass
            else:
                return False
    else:
        return False
    return True
    
# Correr função em todas as camadas
for layer in layers.values():
    field_to_value_relation(layer)

Conclusion

In a relatively quick way, I was able to set all the project’s layers with the widgets I needed.Peek 2019-09-30 16-06

This seems to me like the tip of the iceberg. If one has the need, with some search and patience, other configurations can be changed using PyQGIS. Therefore, think twice before embarking in configuring a big project, layer by layer, field by fields.

Learn More

QGIS Versioning now supports foreign keys!

QGIS-versioning is a QGIS and PostGIS plugin dedicated to data versioning and history management. It supports :

  • Keeping full table history with all modifications
  • Transparent access to current data
  • Versioning tables with branches
  • Work offline
  • Work on a data subset
  • Conflict management with a GUI

QGIS versioning conflict management

In a previous blog article we detailed how QGIS versioning can manage data history, branches, and work offline with PostGIS-stored data and QGIS. We recently added foreign key support to QGIS versioning so you can now historize any complex database schema.

This QGIS plugin is available in the official QGIS plugin repository, and you can fork it on GitHub too !

Foreign key support

TL;DR

When a user decides to historize its PostgreSQL database with QGIS-versioning, the plugin alters the existing database schema and adds new fields in order to track down the different versions of a single table row. Every access to these versioned tables are subsequently made through updatable views in order to automatically fill in the new versioning fields.

Up to now, it was not possible to deal with primary keys and foreign keys : the original tables had to be constraints-free.  This limitation has been lifted thanks to this contribution.

To make it simple, the solution is to remove all constraints from the original database and transform them into a set of SQL check triggers installed on the working copy databases (SQLite or PostgreSQL). As verifications are made on the client side, it’s impossible to propagate invalid modifications on your base server when you “commit” updates.

Behind the curtains

When you choose to historize an existing database, a few fields are added to the existing table. Among these fields, versioning_ididentifies  one specific version of a row. For one existing row, there are several versions of this row, each with a different versioning_id but with the same original primary key field. As a consequence, that field cannot satisfy the unique constraint, so it cannot be a key, therefore no foreign key neither.

We therefore have to drop the primary key and foreign key constraints when historizing the table. Before removing them, constraints definitions are stored in a dedicated table so that these constraints can be checked later.

When the user checks out a specific table on a specific branch, QGIS-versioning uses that constraint table to build constraint checking triggers in the working copy. The way constraints are built depends on the checkout type (you can checkout in a SQLite file, in the master PostgreSQL database or in another PostgreSQL database).

What do we check ?

That’s where the fun begins ! The first thing we have to check is key uniqueness or foreign key referencing an existing key on insert or update. Remember that there are no primary key and foreign key anymore, we dropped them when activating historization. We keep the term for better understanding.

You also have to deal with deleting or updating a referenced row and the different ways of propagating the modification : cascade, set default, set null, or simply failure, as explained in PostgreSQL Foreign keys documentation .

Nevermind all that, this problem has been solved for you and everything is done automatically in QGIS-versioning. Before you ask, yes foreign keys spanning on multiple fields are also supported.

What’s new in QGIS ?

You will get a new message you probably already know about, when you try to make an invalid modification committing your changes to the master database

Error when foreign key constraint is violated

Partial checkout

One existing Qgis-versioning feature is partial checkout. It allows a user to select a subset of data to checkout in its working copy. It avoids downloading gigabytes of data you do not care about. You can, for instance, checkout features within a given spatial extent.

So far, so good. But if you have only a part of your data, you cannot ensure that modifying a data field as primary key will keep uniqueness. In this particular case, QGIS-versioning will trigger errors on commit, pointing out the invalid rows you have to modify so the unique constraint remains valid.

Error when committing non unique key after a partial checkout

Tests

There is a lot to check when you intend to replace the existing constraint system with your own constraint system based on triggers. In order to ensure QGIS-Versioning stability and reliability, we put some special effort on building a test set that cover all use cases and possible exceptions.

What’s next

There is now no known limitations on using QGIS-versioning on any of your database. If you think about a missing feature or just want to know more about QGIS and QGIS-versioning, feel free to contact us at infos+data@oslandia.com. And please have a look at our support offering for QGIS.

Many thanks to eHealth Africa who helped us develop these new features. eHealth Africa is a non-governmental organization based in Nigeria. Their mission is to build stronger health systems through the design and implementation of data-driven solutions.

Learn More

FOSS4GUK Workshop: Collecting data with QGIS, Input and Mergin

During FOSS4GUK 2019 in Edinburgh we ran a workshop for collecting data using Input. This is the content of the workshop with all the datasets.

Prerequisites

To be able to work with Input, you will need the following:

Setting up the survey project

For the purpose of this workshop, we have prepared a QGIS project. Let’s use that as a starting point:

  • Log into Mergin
  • In the top-left, click on Projects and select Explore
  • Find and click on saber/foss4guk
  • In the top panel, click on the mergin_clone icon to create a copy of the project under your own Mergin account (foss4guk_YOURNAME)

Exploring the project in QGIS

The project you have copied in Mergin, is a QGIS project with various map layers. To see the content of the project in QGIS:

  • In Mergin, in the top menu, select Projects > My projects

  • Select foss4guk_YOURUSERNAME (or the name you assigned when copying the project).

  • In the top menu, click on the mergin_download icon to download the project

  • Extract the content of the zip file

Alternatively:

The above process can be done through the Mergin plugin for QGIS. To do that:

  • Install the Mergin plugin in QGIS

  • Restart QGIS

  • In the QGIS Browser panel, right-click on Mergin and select Configure

  • Enter your Mergin username and password

  • Under My Project, right-click on foss4guk_YOURUSERNAME and select Download

  • Select a location under which the project will be downloaded to

  • Once downloaded, select Open to open the project.

Layer settings and forms

Input is based on QGIS, therefore, any layer symbology / styles you set in QGIS, will be displayed in Input. If you are using SVGs (e.g. OS MasterMap), you need to embed these in the QGIS project.

Input also supports most of the edit widgets from QGIS. Edit widgets allow you to simplify filling-in forms with drop-down options, TRUE/FALSE switches, sliders, calendar and time, default values, attachments, value relations and more. To see some of those settings:

  • From the Layers panel (in QGIS), right-click on the listed buildings (points layer) and open the Properties window.

  • From the left-hand panel, select Attributes Form. Explore the various widgets assigned to different fields.

For this layer, we have set the Photo field to use an Attachment widget. This will allow Input to make use of your mobile camera to attach photos to features.

For the Surveyor field, we have linked it to an external CSV table, to populate a drop-down option with the names of surveyors.

Input can also use a pop-up window (similar to Google Maps) to display basic information about a single feature:

To customise this pop window’s content:

  • Open the properties table, and select the Display tab

  • You can see the title is set to ENT_TITLE and there is an image tag referencing the Photo field:

      # image
  file:///[%@project_folder%]/[% "Photo" %]

Map themes

To simplify handling layer visibility, Input makes use of map themes defined in your QGIS project. In this project, there is a map theme for aerial photo (using a Bing aerial layer) and OpenStreetMap (geopackage).

[]{#anchor-5}Survey layer

In Input, any vector layer (point, line, polygon) can be edited (as long as editing that format is supported in QGIS). This could be very confusing when dealing with large numbers of vector layers in a single project (trying to figure out which one to edit).

Luckily you can set background layers (or those you don’t want to be editable in Input) to read-only:

  • In QGIS, from the main menu, select Project > Properties

  • In the new window, select the Data Sources tab from the left-hand panel

Below is the list of layers and their capability settings for the project. Layers not marked as read-only will be shown as survey layers (editable) in Input.

By default, the file paths to layers are relative. You can change that under the General tab of this window.

Using Input

To use Input, open the app on your device. On its first run, Input will show the Projects page.

  • Under Projects, select My projects
  • From the list, find YOUR_Mergin_USERNAME/foss4guk_YOURUSERNAME (e.g. saber/foss4guk_saber)
  • Tap the download icon on the right-hand side of the project to download the project (warning: if you are not connected to WiFi, this will use some of your mobile data allowance)
  • After downloading, tap Home
  • Select your downloaded project

When you open the project, you may not see all layers. This is because some of the layers have zoom-dependant visibility settings (again configured in QGIS).

Exploring the project

To switch map themes:

  • Tap More on the bottom-right side of the screen

  • Tap Map themes > aerial photo

You can also display feature details simply by tapping on them.

  • Tap on the point representing Queensberry House:

Capturing data

To capture data:

  • Tap Record

  • You can then choose the layer in which you want to record your feature, by tapping on the light green band, in the lower part of the screen, above the Input menu.

  • If you are capturing a point, by default, the suggested point to capture will be on your GPS location. You can drag the map to adjust the location of the new point. To switch back to the current GPS location, tap the GPS icon on the bottom-left of your screen.

  • After adding a point, you will be prompted to fill-in the form.

If you are recording a line or a polygon, you can either add points to define the shape of your feature or press and hold the GPS icon when in Record mode to generate a shape from your GPS track.

Editing data

You can edit the existing features on your map. For point layers, you can edit geometry and form data. For lines and polygons, you can edit only the form data.

Try it!

Let’s get out and capture some data for the Path layer!

Uploading your changes

Once you have made changes to your data, you can upload them back to Mergin:

  • In Input, tap Projects

  • Select My projects

  • Click on the sync/refresh icon to the right of your project

You can now download the project again to your desktop and see the changes in QGIS. Alternatively, you can synchronise the changes you made back to QGIS by using the Mergin plugin for QGIS (described earlier).

Learn More