Tag: qgis

The heart of the effect involves the very clever idea of using the topographic layer as a subtle opacity mask to simulate mist, fog and atmospheric haze. Have a look at the image below taken on March 17th, 2005 by NASA’s Terra satellite. This is the industrialised Po valley of northern Italy, surrounded by the Alps and Apennine Mountains that rise above the valley’s hazy pollution. The haze adds a sense of depth to the surrounding hills and mountains. It’s not uncommon to see fog and pollution in satellite imagery that gives way to the clear air in high mountains e.g. northern India and Nepal, China, Pakistan and India. Creating a similar mist effect in QGIS is actually quite simple.

Haze in northern Italy – NASA Terra Satellite

What is OpenCL?

From https://en.wikipedia.org/wiki/OpenCL:

OpenCL (Open Computing Language) is a framework for writing programs that execute across heterogeneous platforms consisting of central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs) and other processors or hardware accelerators. OpenCL specifies programming languages (based on C99 and C++11) for programming these devices and application programming interfaces (APIs) to control the platform and execute programs on the compute devices. OpenCL provides a standard interface for parallel computing using task- and data-based parallelism.

Basically, you write a program and you execute it on a GPU (or, less frequently, on a CPU or on a DSP) taking advantage of the huge parallel programming capabilities of the modern graphic cards.

Depending on many different factors, the speed gain can vary to a great extent, but it is typically around one order of magnitude.

How QGIS benefits from OpenCL?

The work I’ve done consisted in integrating OpenCL support into QGIS and writing all the utilities to load, build and run OpenCL programs.

For now, I’ve ported the following QGIS core algorithms, all of them are availabe in processing:

• slope
• aspect
• hillshade
• ruggedness

Since the framework to support OpenCL is now in place, I think that more algorithms will be ported over the time.

During this development, even if was not in scope, the hillshade renderer has been optimized for speed and it can also benefit of OpenCL acceleration.

How to activate OpenCL support

OpenCL support is optional and opt-in, to use it, you need to activate it into the QGIS options dialog like shown in the screenshot below:

How much performance gain can I expect?

Well, YMMV, but here are some figures for a big DEM raster, low values mean faster execution.

GDAL means CPU execution using the GDAL processing algorithm.

How to install the OpenCL drivers?

Of course it depends on your specific hardware and on your O.S., AMD, NVidia and Intel have different distributions channels, in general the driver for your graphic card will also provide the OpenCL driver, if your GPU is compatible, if OpenCL is not available on your current machine, try to Google for OpenCL, your O.S. and graphic card.

If there is no OpenCL support for your graphic card, you might try to install a driver for your GPU (Intel for example provides them) and you will probably have a decent acceleration even if not as much as you can get on a real graphic card.

This fact worths some more explanation: you might ask your self why running and algorithm directly on the CPU and running it on the same CPU but using OpenCL would make any difference and the reason why it is generally faster by using OpenCL is that OpenCL will run the algorithm in parallel on all cores of your CPU, while a normal application (and QGIS does not make an exception here) will use a single core.

How to build QGIS with OpenCL support on Ubuntu

Just a quick note: you’ll need to install the OpenCL headers and the ICD library:

sudo apt-get install opencl-headers ocl-icd-opencl-dev

 

Credits

I started this work as a proof of concept in my spare time (that it is not much, lately) and when I realized that it was promising, I submitted a QGIS grant proposal in order to allocate some working time to port more algorithms, write tests and polish the implementation.

This work would not be possible without all the generous sponsors and donors that feed the QGIS grant program year after year, many thanks to the QGIS community for this amazing support!

Jürgen Fischer was as usual very helpful and took care of the windows builds, now available in OSGeo4W packages.

Nyall Dawson helped with the code review and with testing the implementation on different cards and machines.

Matthias Kuhn reviewed the code.

Even Rouault pointed me to some highly efficient GDAL algorithm optimizations that I’ve been able to integrate in QGIS.

 

 

 

Why python data providers?

My main reasons for having Python data provider were:

• quick prototyping
• web services
• why not?

 

This topic has been floating in my head for a while since I decided to give it a second look and I finally implemented it and merged for the next 3.2 release.

 

How it’s been done

To make this possible I had to:

• create a public API for registering the providers
• create the Python bindings (the hard part)
• create a sample Python vector data provider (the boring part)
• make all the tests pass

 

First, let me say that it wasn’t like a walk in the park: the Python bindings part is always like diving into woodoo and black magic recipes before I can get it to work properly.

For the Python provider sample implementation I decided to re-implement the memory (aka: scratch layers) provider because that’s one of the simplest providers and it does not depend on any external storage or backend.

 

How to and examples

For now, the main source of information is the API and the tests:

• • https://github.com/qgis/QGIS/blob/master/src/core/qgsproviderregistry.h#L188
  • https://github.com/qgis/QGIS/blob/master/tests/src/python/provider_python.py
  • https://github.com/qgis/QGIS/blob/master/tests/src/python/test_provider_python.py

To register your own provider (PyProvider in the snippet below) these are the basic steps:

metadata = QgsProviderMetadata(PyProvider.providerKey(), PyProvider.description(), PyProvider.createProvider)
QgsProviderRegistry.instance().registerProvider(metadata)

To create your own provider you will need at least the following components:

• the provider class itself (subclass of QgsVectorDataProvider)
• a feature source (subclass of QgsAbstractFeatureSource)
• a feature iterator (subclass of QgsAbstractFeatureIterator)

Be aware that the implementation of a data provider is not easy and you will need to write a lot of code, but at least you could get some inspiration from the existing example.

 

Enjoy wirting data providers in Python and please let me know if you’ve fond this implementation useful!

Recently I was invited by the colleagues from OpenGIS.ch to lend a hand in a training session about QGIS server.

This was a good opportunity to update my presentation for QGIS3, to fix a few bugs and to explore the powerful capabilities of QGIS server and Python.

As a result, I published the full recipe of a Vagrant VM on github: https://github.com/elpaso/qgis3-server-vagrant

The presentation is online here: http://www.itopen.it/bulk/qgis3-server/

What’s worth mentioning is the sample plugins (I’ll eventually package and upload them to the official plugin site):

• XYZ: add simple XYZ tile server, ready to use within QGIS XYZ connections: https://github.com/elpaso/qgis3-server-vagrant/tree/master/resources/web/plugins/xyz
• custom service, does nothing but shows how to create a Python custom service by exploiting the new service capabilities available in QGIS3: https://github.com/elpaso/qgis3-server-vagrant/tree/master/resources/web/plugins/customservice
• HTTP Basic auth (how to add your auth scheme to QGIS server): https://github.com/elpaso/qgis3-server-vagrant/tree/master/resources/web/plugins/httpbasic
• more …

 

The VM uses 4 different (although similar) deployment strategies:

• good old Apache + mod_fcgi and plain CGI
• Nginx + Fast CGI
• Nginx + standalone HTTP Python wrapped server
• Nginx + standalone WSGI Python wrapped server

Have fun with QGIS server: it was completely refactored in QGIS 3 and it’s now better than ever!

 

Do you want to share your GPS data from your phone to QGIS? Here is how:   QGIS comes with a core plugin named GPS Tools that can be enabled in the Plugin installer dialog:   There are several ways to forward data from your phone and most of them are very well described in the QGIS manual page: https://docs.qgis.org/testing/en/docs/user_manual/working_with_gps/plugins_gps.html What I’m going to describe here is mostly useful when your phone and your host machine running QGIS are on the same network (for example they are connected to the same WiFi access point) and it is based on the simple application GPS 2 NET   Once the application is installed and started on your phone, you need to know the IP address of the phone, on a linux box you can simply run a port scanner and it will find all devices connected to the port 6000 (the default port used by GPS 2 NET):  

# Assuming your subnet is 192.168.9

nmap -p 6000 192.168.1.*

Nmap scan report for android-8899989888d02271.homenet.telecomitalia.it (192.168.99.50)
Host is up (0.0093s latency).
PORT STATE SERVICE
6000/tcp open X11

  Now, in QGIS you can open the plugin dialog through Vector -> GPS -> GPS Tools and enter the IP address and port of your GPS device:   Click on Connect button on the top right corner (mouse over the gray square for GPS status information)   Start digitizing!

Last week I’ve been in Madeira at the hackfest, like all the past events this has been an amazing happening, for those of you who have never been there, a QGIS hackfest is typically an event where QGIS developers and other pasionate contributors like documentation writers, translators etc. gather together to discuss the future of their beloved QGIS software. QGIS hackfest are informal events where meetings are scheduled freely and any topic relevant to the project can be discussed. This time we have brought to the table some interesting topics like:

• the future of processing providers: should they be part of QGIS code or handled independently as plugins?
• the road forward to a better bug reporting system and CI platform: move to gitlab?
• the certification program for QGIS training courses: how (and how much) training companies should give back to the project?
• SWOT analysis of current QGIS project: very interesting discussion about the status of the project.
• QGIS Qt Quick modules for mobile QGIS app

Tehre were also some mentoring sessions where I presented:

• How to set up a development environment and make your first pull request
• How to write tests for QGIS (in both python and C++)

  At this link you can find all the video recordings of the sessions: https://github.com/qgis/QGIS/wiki/DeveloperMeetingMadeira2018   Here is a link to the Vagrant QGIS developer VM I’ve prepared for the session: https://github.com/elpaso/qgis-dev-vagrant/   I’ve got a good feedback from other devs about my sessions and I’m really happy that somebody found them useful, one of the main goals of a QGIS hackfest should really be to help other developers to ramp up quicly into the project. Other than that, I’ve also find the time to update to QGIS 3.0 some of my old plugins like GeoCoding and QuickWKT.   Thanks to Giovanni Manghi and to Madeira Government for the organizazion and thanks to all QGIS sponsors and donors!   About me: I started as a QGIS plugin author, continued as the developer of the plugin official repository at https://plugins.qgis.org and now I’m one of the top 5 QGIS core contributors. After almost 10 years that I’m in the QGIS project I’m now not only a proud member of the QGIS community but also an advocate for the open source GIS software movement.

Building QGIS from sources is not hard at all on a recent linux box, but what about if you wanted to be able to step-debug into Qt core or if you wanted to build QGIS agains the latest Qt release? Here things become tricky. This short post is about my experiments to build Qt and and other Qt-based dependencies for QGIS in order to get a complete debugger-friendly build of QGIS.   Start with downloading the latest Qt installer from Qt official website: https://www.qt.io/download-qt-for-application-development choose the Open Source version.   Now install the Qt version you want to build, make sure you check the Sources and the components you might need. Whe you are done with that, you’ll have your sources in a location like /home/user/Qt/5.9.3/Src/ To build the sources, you can change into that directory and issue the following command – I assume that you have already installed all the dependencies normally needed to build C++ Qt programs – I’m using clang here but feel free to choose gcc, we are going to install the new Qt build into /opt/qt593.

./configure -prefix /opt/qt593 -debug -opensource -confirm-license -ccache -platform linux-clang

When done, you can build it with

make -j9
sudo make install

  To build QGIS you also need three additional Qt packages   QtWebKit from https://github.com/qt/qtwebkit (you can just download the zip): Extract it somewhere and build it with

/opt/qt593/bin/qmake WebKit.pro
make -j9
sudo make install

  Same with QScintila2 from https://www.riverbankcomputing.com/software/qscintilla

/opt/qt593/bin/qmake qscintilla.pro
make -j9
sudo make install

  QWT is also needed and it can be downloaded from https://sourceforge.net/projects/qwt/files/qwt/6.1.3/ but it requires a small edit in qwtconfig.pri before you can build it: set QWT_INSTALL_PREFIX = /opt/qt593_libs/qwt-6.1.3 to install it in a different folder than the default one (that would possibly overwrite a system install of QWT). The build it with:

/opt/qt593/bin/qmake qwt.pro
make -j9
sudo make install

  If everything went fine, you can now configure Qt Creator to use this new debug build of Qt: start with creating a new kit (you can probably clone a working Qt5 kit if you have one). What you need to change is the Qt version (the path to cmake) to point to your brand new Qt build,: Pick up a name and choose the Qt version, but before doing that you need to click on Manage… to create a new one: Now you should be able to build QGIS using your new Qt build, just make sure you disable the bindings in the CMake configuration: unfortunately you’d also need to build PyQt in order to create the bindings.   Whe QGIS is built using this debug-enabled Qt, you will be able to step-debug into Qt core libraries! Happy debugging!  

You may have seen my earlier tutorial where I described how to make nice looking hillshaded maps in QGIS using SRTM elevation data. Well, we don’t have to stop with just one hillshade layer on a map, it is possible to overlay multiple hillshades; a procedure that can increase the visual quality and detail. The following image is the hillshade we made before. Once you re-create a hillshade, following the previous tutorial, you can head to the next step (note that brightness and contrast settings may be different due to changes in how QGIS generates and displays hillshades).

We can improve the SRTM hillshade further by adding ESRI’s World Hillshade layer, which uses multi-directional illumination (also called a Swiss Hillshade in tribute to the celebrated Swiss cartographer Eduard Imhof). In addition, World Hillshade has a much higher resolution than SRTM 30m data in some regions of the world, it is 2m for most of the England and Wales, 10m for most of the US, 5m for Spain and 3m for Holland etc. The only drawback is that the style of this layer is somewhat controversial, some love it, some hate it, it looks like it’s illuminated from above, but mixing it with the SRTM hillshade obviates some of it criticised flaws.

To add the World Hillshade layer in QGIS go to the Layer Menu – Add Layer – Add ArcGIS MapServer Layer – click New and add the following URL:

https://services.arcgisonline.com/arcgis/rest/services/Elevation/World_Hillshade/MapServer

Notice QGIS 2.18 no longer needs a plugin to add ESRI layers, it new has this functionality built in. Also, open the url in a browser such as Firefox, it brings up a webpage that describes the layer. We also see links to other other layers. Yes, they can all be added to QGIS by simply taking the URL of the webpage that describe the layer and connecting to it via the ArcGIS MapServer Layer connector.

Name the layer World Hillshade and click Connect, then click and highlight the layer it connects to. Finally, click the Add button to add the layer to the canvas.

Next, we need to adjust the properties of the World Hillshade layer to properly overlay it above the SRTM hillshade layer. Make sure the World hillshade layer is the topmost layer. In the Layers Panel, right click Layer properties and in the window that opens up, click Style (if not visible). Next, change the Layer Blending mode (under color rendering) to Overlay. Adjust the layer’s brightness to around -20 and leave contrast at 0. If you find the scene is still too dark, brighten the SRTM Hillshade by increasing the layer’s brightness. You may also have to change (lower) the Min value of the Min – Max value boxes. Leave the contrast at 0 for the SRTM hillshade. Also, don’t brighten it too much as it might become washed out, loose detail, especially in bright areas. Play around the controls, settings may vary depending on the SRTM data you download and the version of QGIS you use.

Here’s a comparison in Ireland, a ring like structure of hills with a central peak. No, it’s not a meteorite crater. It’s a different kind of geological marvel, the Slieve Gullion Complex and its ring dyke; the deeply eroded remains of a 410 million year old Caledonian volcano. The SRTM hillshade is on the left and World Hillshade + SRTM hillshade is on the right (click on the image, it’s best appreciated full size):

We can see the World Hillshade + SRTM Hillshade layer shows much finer detail. We see a parallel array of roughly north-south orientated lines, these are fractures and faults that cut the Slieve Gullion Complex that were perhaps enhanced by glacial erosion. Also, look carefully, there seems to be some roads meandering across the landscape (hint, bottom of the map and right of the scale bar). You should get even better results with higher resolution World Hillshade data. We also notice that bending SRTM derived hillshade with World Hillshade adds a naturalistic illumination not apparent in multi-directional hillshading. So we have the best of both worlds, a high resolution hillshade and realistic looking illumination.

Hope you found this tutorial helpful.

References:

Baxter, S., 2008. A Geological Field Guide to Cooley Gullion, Mourne & Slieve Croob [pdf]. Geological Survey of Ireland, Dublin. p. 43-53.

Geologists display the dip and strike of rock layers on geological maps using a dip and strike symbol, where dip in degrees indicates the maximum angle a rock layer descends relative to the horizontal. However, it is not directly possible in QGIS 2.18, using basic label settings, to place and orient a dip label next to a dip and strike symbol.

However, there is a way around this issue using Trigonometry and editing the layer’s Attribute Table. This method may be useful for controlling the position and orientation of labels around point features in general. The first step involves adding values to the Attribute Table. First, add these two new columns:

• Angle – 0° is North and values increases clockwise up to 359°
• Distance – label distance from a point feature

You can add Angle and Distance values to these columns manually or use the Field Calculator (see below) to add values if you have lots of points. Also, I chose Map Units (not millimeters) for Symbol Size, Font Size and Distance for my map, as I prefered to keep symbol size, font size and position of labels fixed when zooming in and out.

Note – I use Strike (Angle) and Label Distance (Distance)  in my Attribute Table

The next step is to control the position of the label around the points using trigonometry. Right click the points layer and choose:

Properties – Labels – Placement

Check that Offset From Point is checked and then click the Data Defined Override next to the Offset X, Y boxes and choose Edit. The Expression String Builder will appear. Enter the following expression in the Expression String Builder window:

to_string ( ((-1) * ( “Distance” )) * cos ( radians ( “Angle” ))) ||’,’|| to_string (((-1) * ( “Distance” )) * sin ( radians ( “Angle” )) )

The expression takes the angle and distance values from the Attribute Table (edited earlier) and calculates an X, Y label position relative to the point feature. You may also optionally control the angle of a symbol or icon itself via:

Layer Properties – Style – click Data Defined Override icon – Edit

Then enter the following expression in the Data Defined Override dialogue:

“Angle” – 90

Finally, to control the rotation of label text, so text follows the orientation (angle) of a rotating symbol or icon, choose:

Layer Properties – Labels – Placement – Data Defined – Rotation

Click the Data Defined Override Icon again and then choose Edit. Enter the following expression in the Data Defined Override dialogue:

(“Angle” – 90) * -1

The following geological map of the Old Head of Kinsale in southern Ireland shows the results of the above procedure. We see that the dip labels rotate and currently follow the orientation of the dip and strike symbols (note that the points are at the intersection of the T symbol).

Geological Survey of Ireland – Creative Commons Attribution 4.0 license

You may have several different symbols, of various sizes, each requiring an appropriate label distance expressed in the Attribute Table. It took me a few tries before I found the right distances for my geological symbols, from 90 to 230 meters distance depending on the symbol size and type.

Lastly, the expressions “Angle” – 90 and (“Angle” – 90) * -1 were necessary in my case because I needed to place my labels next to the dip and strike symbol’s barb. You may need to use a different expression e.g. “Angle” and (“Angle”) * -1, or a value other than 90° depending on the symbol used and the prefered label placement location. Some trial and error is may be required to find the correct label position.

Another great QGIS hackfest is gone, and it’s time for a quick report. The location was the Linux Hotel, one of the best places where open source developers could meet, friendly, geek-oriented and when the weather is good, like this time, villa Vogelsang is a wonderful place to have a beer in the garden while talking about software development or life in general. This is a short list of what kept me busy during the hackfest:

• fixed some bugs and feature requests on the official QGIS plugin repo that I’m maintaining since the very beginning
• make the QGIS official plugin repository website mobile-friendly
• QGIS Server Python Plugin API refactoring, I’ve completed the work on the new API, thanks to the ongoing server refactoring it’s now much cleaner than it was in the first version
• attribute table bugs: I started to address some nasty bugs in the attribute table, some of those were fixed during the week right after the hackfest
• unified add layer button, we had a productive meeting where we decided the path forward to implement this feature, thanks to Boundless that is funding the development, this feature is what’s I’m currently working on these days

Thanks to all QGIS donors and funders that made yet another great hackfest possible and in particular to Boundless Spatial Inc. for funding my personal expenses.