We have been continuing our work with the Flagship sponsor of QGIS – Felt to develop their QGIS Plugin – Add to Felt  that makes it even easier to share your maps and data on the web.

What is the ‘Add to Felt’ QGIS Plugin?

The ‘Add to Felt’ QGIS Plugin is a powerful tool that empowers users to export their QGIS projects and layers directly to a Felt web map. This update introduces two fantastic features:

1. Single Layer Sharing: You can now share a single layer from your QGIS project to a Felt map. This means you have greater control over which specific data layers to share, allowing you to tailor your map precisely to your audience’s needs.
2. Map Selection: With the updated plugin, you can choose which map on Felt to add your layer to – a new map, or an ongoing project. This flexibility simplifies your workflow and ensures that your data ends up in the right place.

Businesses that rely on QGIS love how these new features provide a seamless way to view and share results, ultimately allowing them to move more quickly and stay in sync:

“Felt helps us keep each other updated on what we’ve done, what we’ve modeled, how things are progressing.” – ICON Engineering

Why is this Update Important?

Web maps are invaluable tools for sharing data with a wider audience, be it colleagues, clients, or the public. They provide creators with the ability to control data visibility, display options, and audience access, all within an easily shareable digital format. However, creating web maps can be an arduous and complex task.

Here’s where the ‘Add to Felt’ QGIS Plugin update comes to the rescue:

1. Streamlining the Process: Creating web maps traditionally involves website development, data hosting, and map application development—tasks that require a diverse skill set. This complexity can be a significant barrier, especially for smaller operations with limited resources or budget constraints.

2. Felt Simplifies Web Mapping: Felt makes it effortless to create web maps, and share them as easily as you would a Google Doc or Sheet. Simply drag and drop your data, customize the symbology to your liking, and share the map with a link or by inviting collaborators. No need to send large data files or answer questions about the map’s data sources.

3. Integration with QGIS: Now, the ‘Add to Felt’ QGIS Plugin bridges the gap between QGIS and Felt. It seamlessly imports your QGIS data into Felt, eliminating the need for manual data transfers and reducing the complexity of web map creation.

In essence, the ‘Add to Felt’ QGIS Plugin update simplifies the process of sharing and collaborating on web maps. It empowers users to harness the full potential of web-based mapping, making it accessible to everyone, regardless of their technical expertise. The update makes it even easier to share progress updates or model re-run outputs without creating a new map, or sharing a new map link.

So, if you’re a QGIS user looking to enhance your map-sharing capabilities and streamline your workflow, make sure to take advantage of this fantastic update. Say goodbye to the complexities of web map creation and hello to effortless, data-rich web maps with Felt and the ‘Add to Felt’ QGIS Plugin.

How to install and upgrade

• Open QGIS on your computer. You must have version 3.22 or later installed.
• In the plugins tab, select Manage and Install Plugins.
• Search for the ‘Add to Felt’ plugin, select and click Install Plugin.
• Close the Plugins dialog. The Felt plugin toolbar will appear in your toolbar for use.
• Sign into Felt and begin sharing your maps to the web.

If you want more features in this plugin, let us know or you’re interested in exploring how a QGIS plugin can make your service easily accessible to the millions of daily QGIS users, contact us to discuss how we can help!

The gift economy of Open Source is community driven and filled by folks with ideas that just go for it!

We at North Road are blessed that we get to join these creatives on their journey in order to get their products to you. Recently, the first QGIS flagship sponsor, Felt, engaged us to further strengthen their support for the up to 600,000 daily QGIS users to integrate their workflows between QGIS and Felt.

The result is the “Add to Felt” QGIS Plugin, which makes it super-simple to publish your QGIS maps to the Felt platform.

To get started, install the Add to Felt Plugin from the QGIS Plugin manager.

If you don’t have a free Felt account, you’ll need to sign up for one online (or from the Add to Felt plugin itself once you have installed it).

Within QGIS, users can easily publish their maps and layers to Felt. You can either:

• Publish a single layer by right-clicking the layer and selecting “Share Layer to Felt” from the Export sub-menu
• Publish your whole QGIS project/map by selecting the Project Menu, Export, “Add to Felt” action

Whilst Felt is loading up your map, you can continue working and it will let you know once your map is ready to open on Felt and share with others.

We are happy to let you know that the collaboration does not stop there! As with our SLYR tool, there is ongoing development as the requirements of the community and technology grow.  So install the Add to Felt Plugin via the QGIS Plugin manager, and let us know where you want it to go via the Add to Felt GitHub page.

Read more about it here:

• Felt’s Blog about the QGIS Plugin
• Installation Guide
• Getting Started for QGIS Users

The QGIS 3.28 release is an extremely exciting release for all users who work in mixed software workplaces, or who need to work alongside users of ESRI software. In this post we’ll be giving an overview of all the new tools and features introduced in 3.28 which together result in a dramatic improvement in the workflows and capabilities in working with ESRI based formats and services. Read on for the full details…!

Before we begin, we’d like to credit the following organisations for helping fund these developments in QGIS 3.28:

• Naturstyrelsen, Denmark
• Provincie Gelderland, Netherlands
• Uppsala Universitet, Department of Archaeology and Ancient History
• Gemeente Amsterdam
• Provincie Zuid-Holland, Netherlands

FileGeodatabase (GDB) related improvements

The headline item here is that QGIS 3.28 introduces support for editing, managing and creating ESRI FileGeodatabases out of the box! While older QGIS releases offered some limited support for editing FileGeodatabase layers, this required the manual installation of a closed source ESRI SDK driver… which unfortunately resulted in other regressions in working with FileGeodatabases (such as poor layer loading speed and random crashes). Now, thanks to an incredible reverse engineering effort by the GDAL team, the open-source driver for FileGeodatabases offers full support for editing these datasets! This means all QGIS users have out-of-the-box access to a fully functional, high-performance read AND write GDB driver, no further action or trade-offs required.

Operations supported by the GDAL open source driver include:

• Editing existing features, with full support for editing attributes and curved, 3D and measure-value geometries
• Creating new features
• Deleting features
• Creating, adding and modifying attributes in an existing layer
• Full support for reading and updating spatial indexes
• Creating new indexes on attributes
• “Repacking” layers, to reduce their size and improve performance
• Creating new layers in an existing FileGeodatabase
• Removing layers from FileGeodatabases
• Creating completely new, empty FileGeodatabases
• Creating and managing field domains

On the QGIS side, the improvements to the GDAL driver meant that we could easily expose feature editing support for FileGeodatabase layers for all QGIS users. While this is a huge step forward, especially for users in mixed software workplaces, we weren’t happy to rest there when we  had the opportunity to further improve GDB support within QGIS!

So in QGIS 3.28 we also introduced the following new functionality when working with FileGeodatabases:

FileGeodatabase management tools

QGIS 3.28 introduces a whole range of GUI based tools for managing FileGeodatabases. To create a brand new FileGeodatabase, you can now right click on a directory from the QGIS Browser panel and select New – ESRI FileGeodatabase:

After creating your new database, a right click on its entry will show a bunch of available options for managing the database. These include options for creating new tables, running arbitrary SQL commands, and database-level operations such as compacting the database:

You’re also able to directly import existing data into a FileGeodatabase by simply dragging and dropping layers onto the database!

Expanding out the GDB item will show a list of layers present in the database, and present options for managing the fields in those layers. Alongside field creation, you can also remove and rename existing fields.

Field domain handling

QGIS 3.28 also introduces a range of GUI tools for working with field domains inside FileGeodatabases. (GeoPackage users also share in the love here — these same tools are all available for working with field domains inside this standard format too!) Just right click on an existing FileGeodatabase (or GeoPackage) and select the “New Field Domain” option. Depending on the database format, you’ll be presented with a list of matching field domain types:

Once again, you’ll be guided through a user-friendly dialog allowing you to create your desired field domain!

After field domains have been created, they can be assigned to fields in the database by right-clicking on the field name and selecting “Set Field Domain”:

Field domains can also be viewed and managed by expanding out the “Field domains” option for each database.

Relationship discovery

Another exciting addition in QGIS 3.28 (and the underlying GDAL 3.6 release) is support for discovering database relationships in FileGeodatabases! (Once again, GeoPackage users also benefit from this, as we’ve implemented full support for GeoPackage relationships via the “Related Tables Extension“).

Expanding out a database containing any relationships will show a list of all discovered relationships:

(You can view the full description and details for any of these relationships by opening the QGIS Browser “Properties” panel).

Whenever QGIS 3.28 discovers relationships in the database, these related tables will automatically be added to your project whenever any of the layers which participate in the relationship are opened. This means that users get the full experience as designed for these databases without any manual configuration, and the relationships will “just work”!

Dataset Grouping

Lastly, we’ve improved the way layers from FileGeodatabases are shown in QGIS, so that layers are now grouped according to their original dataset groupings from the database structure:

Edit ArcGIS Online / Feature Service layers

While QGIS has had read-only support for viewing and working with the data in ArcGIS Online (AGOL) vector layers and ArcGIS Server “feature service” layers for many years, we’ve added support for editing these layers in QGIS 3.28. This allows you to take advantage of all of QGIS’ easy to use, powerful editing tools and directly edit the content in these layers from within your QGIS projects! You can freely create new features, delete features, and modify the shape and attributes of existing features (assuming that your user account on the ArcGIS service has these edit permissions granted, of course). This is an exciting addition for anyone who has to work often with content in ArcGIS services, and would prefer to directly manipulate these layers from within QGIS instead of the limited editing tools available on the AGOL/Portal platforms themselves.

This new functionality will be available immediately to users upon upgrading to QGIS 3.28 — any users who have been granted edit capabilities for the layers will see that the QGIS edit tools are all enabled and ready for use without any further configuration on the QGIS client side.

Filtering Feature Service layers

We’ve also had the opportunity to introduce filter/query support for Feature Service layers in QGIS 3.28. This is a huge performance improvement for users who need to work with a subset of a features from a large Feature Service layer. Unfortunately, due to the nature of the Feature Service protocol, these layers can often be slow to load and navigate on a client side. By setting a SQL filter to limit the features retrieved from the service the performance can be dramatically increased, as only matching features will ever be requested from the backend server. You can use any SQL query which conforms to the subset of SQL understood by ArcGIS servers (see the Feature Service documentation for examples of supported SQL queries).

What’s next?

While QGIS 3.28 is an extremely exciting release for any users who need to work alongside ESRI software, we aren’t content to rest here! The exciting news is that in QGIS 3.30 we’ll be introducing a GUI driven approach allowing users to create new relationships in their FileGeodatabase (and GeoPackage!) databases.

At North Road we’re always continuing to improve the cross-vendor experience for both ESRI and open-source users through our continued work on the QGIS desktop application and our SLYR conversion suite. If you’d like to chat to us about how we can help your workplace transition from a fully ESRI stack to a mixed or fully open-source stack, just contact us to discuss your needs.

Last week the first official “FOSS4G/SOTM Oceania” conference was held at Melbourne University. This was a fantastic event, and there’s simply no way I can extend sufficient thanks to all the organisers and volunteers who put this event together. They did a brilliant job, and their efforts are even more impressive considering it was the inaugural event!

Upfront — this is not a recap of the conference (I’m sure someone else is working on a much more detailed write up of the event!), just some musings I’ve had following my experiences assisting Nathan Woodrow deliver an introductory Python for QGIS workshop he put together for the conference. In short, we both found that delivering this workshop to a group of PyQGIS newcomers was a great way for us to identify “pain points” in the PyQGIS API and areas where we need to improve. The good news is that as a direct result of the experiences during this workshop the API has been improved and streamlined! Let’s explore how:

Part of Nathan’s workshop (notes are available here) focused on a hands-on example of creating a custom QGIS “Processing” script. I’ve found that preparing workshops is guaranteed to expose a bunch of rare and tricky software bugs, and this was no exception! Unfortunately the workshop was scheduled just before the QGIS 3.4.2 patch release which fixed these bugs, but at least they’re fixed now and we can move on…

The bulk of Nathan’s example algorithm is contained within the following block (where “distance” is the length of line segments we want to chop our features up into):

for input_feature in enumerate(features):
    geom = feature.geometry().constGet()
    if isinstance(geom, QgsLineString):
        continue
    first_part = geom.geometryN(0)
    start = 0
    end = distance
    length = first_part.length()

    while start < length:
        new_geom = first_part.curveSubstring(start,end)

        output_feature = input_feature
        output_feature.setGeometry(QgsGeometry(new_geom))
        sink.addFeature(output_feature)

        start += distance
        end += distance

There’s a lot here, but really the guts of this algorithm breaks down to one line:

new_geom = first_part.curveSubstring(start,end)

Basically, a new geometry is created for each trimmed section in the output layer by calling the “curveSubstring” method on the input geometry and passing it a start and end distance along the input line. This returns the portion of that input LineString (or CircularString, or CompoundCurve) between those distances. The PyQGIS API nicely hides the details here – you can safely call this one method and be confident that regardless of the input geometry type the result will be correct.

Unfortunately, while calling the “curveSubstring” method is elegant, all the code surrounding this call is not so elegant. As a (mostly) full-time QGIS developer myself, I tend to look over oddities in the API. It’s easy to justify ugly API as just “how it’s always been”, and over time it’s natural to develop a type of blind spot to these issues.

Let’s start with the first ugly part of this code:

geom = input_feature.geometry().constGet()
if isinstance(geom, QgsLineString):
    continue
first_part = geom.geometryN(0)
# chop first_part into sections of desired length
...

This is rather… confusing… logic to follow. Here the script is fetching the geometry of the input feature, checking if it’s a LineString, and if it IS, then it skips that feature and continues to the next. Wait… what? It’s skipping features with LineString geometries?

Well, yes. The algorithm was written specifically for one workshop, which was using a MultiLineString layer as the demo layer. The script takes a huge shortcut here and says “if the input feature isn’t a MultiLineString, ignore it — we only know how to deal with multi-part geometries”. Immediately following this logic there’s a call to geometryN( 0 ), which returns just the first part of the MultiLineString geometry.

There’s two issues here — one is that the script just plain won’t work for LineString inputs, and the second is that it ignores everything BUT the first part in the geometry. While it would be possible to fix the script and add a check for the input geometry type, put in logic to loop over all the parts of a multi-part input, etc, that’s instantly going to add a LOT of complexity or duplicate code here.

Fortunately, this was the perfect excuse to improve the PyQGIS API itself so that this kind of operation is simpler in future! Nathan and I had a debrief/brainstorm after the workshop, and as a result a new “parts iterator” has been implemented and merged to QGIS master. It’ll be available from version 3.6 on. Using the new iterator, we can simplify the script:

geom = input_feature.geometry()
for part in geom.parts():
    # chop part into sections of desired length
    ...

Win! This is simultaneously more readable, more Pythonic, and automatically works for both LineString and MultiLineString inputs (and in the case of MultiLineStrings, we now correctly handle all parts).

Here’s another pain-point. Looking at the block:

new_geom = part.curveSubstring(start,end)
output_feature = input_feature
output_feature.setGeometry(QgsGeometry(new_geom))

At first glance this looks reasonable – we use curveSubstring to get the portion of the curve, then make a copy of the input_feature as output_feature (this ensures that the features output by the algorithm maintain all the attributes from the input features), and finally set the geometry of the output_feature to be the newly calculated curve portion. The ugliness here comes in this line:

output_feature.setGeometry(QgsGeometry(new_geom))

What’s that extra QgsGeometry(…) call doing here? Without getting too sidetracked into the QGIS geometry API internals, QgsFeature.setGeometry requires a QgsGeometry argument, not the QgsAbstractGeometry subclass which is returned by curveSubstring.

This is a prime example of a “paper-cut” style issue in the PyQGIS API. Experienced developers know and understand the reasons behind this, but for newcomers to PyQGIS, it’s an obscure complexity. Fortunately the solution here was simple — and after the workshop Nathan and I added a new overload to QgsFeature.setGeometry which accepts a QgsAbstractGeometry argument. So in QGIS 3.6 this line can be simplified to:

output_feature.setGeometry(new_geom)

Or, if you wanted to make things more concise, you could put the curveSubstring call directly in here:

output_feature = input_feature
output_feature.setGeometry(part.curveSubstring(start,end))

Let’s have a look at the simplified script for QGIS 3.6:

for input_feature in enumerate(features):
    geom = feature.geometry()
    for part in geom.parts():
        start = 0
        end = distance
        length = part.length()

        while start < length:
            output_feature = input_feature
            output_feature.setGeometry(part.curveSubstring(start,end))
            sink.addFeature(output_feature)

            start += distance
            end += distance

This is MUCH nicer, and will be much easier to explain in the next workshop! The good news is that Nathan has more niceness on the way which will further improve the process of writing QGIS Processing script algorithms. You can see some early prototypes of this work here:

I couldn’t be at the community day but managed to knock out some of the new API on the plane on the way home. **API subject to change #FOSS4G_SotM_Oceania

cc @nyalldawson @underdarkGIS this will create a alg under the hood and check for double inputs, etc pic.twitter.com/49VpyuukGU

— Nathan Woodrow (@madmanwoo) November 23, 2018

So there we go. The process of writing and delivering a workshop helps to look past “API blind spots” and identify the ugly points and traps for those new to the API. As a direct result of this FOSS4G/SOTM Oceania 2018 Workshop, the QGIS 3.6 PyQGIS API will be easier to use, more readable, and less buggy! That’s a win all round!

Last week we kicked off the first (of hopefully many) Australian QGIS hackfests Developers Meetings. It was attended by 3 of the core QGIS development team: Nathan Woodrow, Martin Dobias and myself (Nyall Dawson), along with various family members. While there’s been QGIS hackfests in Europe for over 10 years, and others scattered throughout various countries (I think there was a Japanese one recently… but Twitter’s translate tool leaves me with little confidence about this!), there’s been no events like this in the Southern hemisphere yet. I’ve been to a couple in Europe and found them to be a great way to build involvement in the project, for both developers and non-developers alike.

In truth the Australian hackfest plans began mostly an excuse for Nathan and I to catch up with Martin Dobias before he heads back out of this hemisphere and returns to Europe. That said, Nathan and I have long spoken about ways we can build up the QGIS community in Australia, so in many ways this was a trial run for future events. It was based it in Noosa, QLD (and yes, we did manage to tear ourselves away from our screens long enough to visit the beach!).

Nathan Woodrow (@NathanW2), myself (@nyalldawson), and Martin Dobias (@wonder-sk)

Here’s a short summary of what we worked on during the hackfest:

• Martin implemented a new iterator style accessor for vertices within geometries. The current approach to accessing vertices in QGIS is far from optimal. You either have the choice of an inefficient methods (eg QgsGeometry.asPolyline(), asPolygon(), etc) which requires translations of all vertices to a different data structure (losing any z/m dimensional values in the process), or an equally inefficient QgsAbstractGeometry.coordinateSequence() method, which at least keeps z/m values but still requires expensive copies of every vertex in the geometry. For QGIS 3.0 we’ve made a huge focus on optimising geometry operations and vertex access is one of the largest performance killers remaining in the QGIS code. Martin’s work adds a proper iterator for the vertices within a geometry object, both avoiding all these expensive copies and also simplifying the API for plugins. When this work lands traversing the vertices will become as simple as

for v in geom.vertices():
   ... do something with the vertex!

• Martin is also planning on extending this work to allow simple iteration over the parts and rings within geometries too. When this lands in QGIS we can expect to see much faster geometry operations.
• Nathan fixed a long standing hassle with running standalone PyQGIS scripts outside of the QGIS application on Windows. In earlier versions there’s a LOT of batch file mangling and environment variable juggling required before you can safely import the qgis libraries within Python. Thanks to Nathan’s work, in QGIS 3.0 this will be as simple as just making sure that the QGIS python libraries are included in your Python path, and then importing qgis.core/gui etc will work without any need to create environment variables for OSGEO/GDAL/PLUGINS/etc. Anyone who has fought with this in the past will definitely appreciate this change, and users of Python IDEs will also appreciate how simple it is now to make the PyQGIS libraries available in these environments.
• Nathan also worked on “profiles” support for QGIS 3.0. This work will add isolated user profiles within QGIS, similar to how Chrome handles this. Each profile has it’s own separate set of settings, plugins, etc. This work is designed to benefit both plugin developers and QGIS users within enterprise environments. You can read more about what Nathan has planned for this here.
• I continued the ongoing work of moving long running interface “blocking” operations to background tasks. In QGIS 3.0 many of these tasks churn away in the background, allowing you to continue work while the operation completes. It’s been implemented so far for vector and raster layer saving, map exports to images/PDF (not composers unfortunately), and obtaining feature counts within legends. During the hackfest I moved the layer import which occurs when you drag and drop a layer to a destination in the browser to a background task.
• On the same topic, I took some inspiration from a commit in Sourcepole’s QGIS fork and reworked how composer maps are cached. One of my biggest gripes with QGIS’ composer is how slow it is to work with when you’ve got a complex map included. This change pushes the map redrawing into a background thread, so that these redraws no longer “lock up” the UI. It makes a HUGE difference in how usable composer is. This improvement also allowed me to remove those confusing map item “modes” (Cache/Render/Rectangle) – now everything is redrawn silently in the background whenever required.
• Lastly, I spent a lot of time on a fun feature I’ve long wanted in QGIS – a unified search “locator” bar. This feature is heavily inspired by Qt Creator’s locator bar. It sits away down in the status bar, and entering any text here fires up a bunch of background search tasks. Inbuilt searches include searching the layers within the current project (am I the only one who loses layers in the tree in complex projects!?), print layouts in the project, processing algorithms, and menu/toolbar actions. The intention here is that plugins will “take over” and add additional search functionality, such as OSM place names searching, data catalog searches, etc. I’m sure when QGIS 3.0 is released this will quickly become indispensable!

The upcoming QGIS 3.0 locator bar

Big thanks go out to Nathan’s wife, Stacey, who organized most of the event and without whom it probably would never have happened, and to Lutra Consulting who sponsored an awesome dinner for the attendees.

We’d love this to be the first of many. The mature European hackfests are attended by a huge swath of the community, including translators, documentation writers, and plugin developers (amongst others). If you’ve ever been interested in finding out how you can get more involved in the project it’s a great way to dive in and start contributing. There’s many QGIS users in this part of the world and we really want to encourage a community of contributors who “give back” to the project. So let Nathan or myself know if you’d be interested in attending other events like this, or helping to organize them locally yourself…

A lot of cartographers have a love/hate relationship with label halos. On one hand they can be an essential technique for improving label readability, especially against complex background layers. On the other hand they tend to dominate maps and draw unwanted attention to the map labels.

In this post I’m going to share my preferred techniques for using label halos. I personally find this technique is a good approach which minimises the negative effects of halos, while still providing a good boost to label readability. (I’m also going to share some related QGIS 3.0 news at the end of this post!)

Let’s start with some simple white labels over an aerial image:

These labels aren’t very effective. The complex background makes them hard to read, especially the “Winton Shire” label at the bottom of the image. A quick and nasty way to improve readability is to add a black halo around the labels:

Sure, it’s easy to read the labels now, but they stand out way too much and it’s difficult to see anything here except the labels!

We can improve this somewhat through a better choice of halo colour:

This is much better. We’ve got readable labels which aren’t too domineering. Unfortunately the halo effect is still very prominent, especially where the background image varies a lot. In this case it works well for the labels toward the middle of the map, but not so well for the labels at the top and bottom.

A good way to improve this is to take advantage of blending (or “composition”) modes (which QGIS has native support for). The white labels will be most readable when there’s a good contrast with the background map, i.e. when the background map is dark. That’s why we choose a halo colour which is darker than the text colour (or vice versa if you’ve got dark coloured labels). Unfortunately, by choosing the mid-toned brown colour to make the halos blend in more, we are actually lightening up parts of this background layer and both reducing the contrast with the label and also making the halo more visible. By using the “darken” blend mode, the brown halo will only be drawn for pixels were the brown is darker then the existing background. It will darken light areas of the image, but avoid lightening pixels which are already dark and providing good contrast. Here’s what this looks like:

The most noticeable differences are the labels shown above darker areas – the “Winton Shire” label at the bottom and the “Etheridge Shire” at the top. For both these labels the halo is almost imperceptible whilst still subtly doing it’s part to make the label readable. (If you had dark label text with a lighter halo color, you can use the “lighten” blend mode for the same result).

The only issue with this map is that the halo is still very obvious around “Shire” in “Richmond Shire” and “McKinlay” on the left of the map. This can be reduced by applying a light blur to the halo:

There’s almost no loss of readability by applying this blur, but it’s made those last prominent halos disappear into the map. At first glance you probably wouldn’t even notice that there’s any halos being used here. But if we compare back against the original map (which used no halos) we can see the huge difference in readability:

Compare especially the Winton Shire label at the bottom, and the Richmond Shire label in the middle. These are much clearer on our tweaked map versus the above image.

Now for the good news… when QGIS 3.0 is released you’ll no longer have to rely on an external illustration/editing application to get this effect with your maps. In fact, QGIS 3.0 is bringing native support for applying many types of live layer effects to label buffers and background shapes, including blur. This means it will be possible to reproduce this technique directly inside your GIS, no external editing or tweaking required!

I’ve recently spent some time optimising the performance of various QGIS plugins and algorithms, and I’ve noticed that there’s a few common performance traps which developers fall into when fetching features from a vector layer. In this post I’m going to explore these traps, what makes them slow, and how to avoid them.

As a bit of background, features are fetched from a vector layer in QGIS using a QgsFeatureRequest object. Common use is something like this:

request = QgsFeatureRequest()
for feature in vector_layer.getFeatures(request):
    # do something

This code would iterate over all the features in layer. Filtering the features is done by tweaking the QgsFeatureRequest, such as:

request = QgsFeatureRequest().setFilterFid(1001)
feature_1001 = next(vector_layer.getFeatures(request))

In this case calling getFeatures(request) just returns the single feature with an ID of 1001 (which is why we shortcut and use next(…) here instead of iterating over the results).

Now, here’s the trap: calling getFeatures is expensive. If you call it on a vector layer, QGIS will be required to setup an new connection to the data store (the layer provider), create some query to return data, and parse each result as it is returned from the provider. This can be slow, especially if you’re working with some type of remote layer, such as a PostGIS table over a VPN connection. This brings us to our first trap:

Trap #1: Minimise the calls to getFeatures()

A common task in PyQGIS code is to take a list of feature IDs and then request those features from the layer. A see a lot of older code which does this using something like:

for id in some_list_of_feature_ids:
    request = QgsFeatureRequest().setFilterFid(id)
    feature = next(vector_layer.getFeatures(request))
    # do something with the feature

Why is this a bad idea? Well, remember that every time you call getFeatures() QGIS needs to do a whole bunch of things before it can start giving you the matching features. In this case, the code is calling getFeatures() once for every feature ID in the list. So if the list had 100 features, that means QGIS is having to create a connection to the data source, set up and prepare a query to match a single feature, wait for the provider to process that, and then finally parse the single feature result. That’s a lot of wasted processing!

If the code is rewritten to take the call to getFeatures() outside of the loop, then the result is:

request = QgsFeatureRequest().setFilterFids(some_list_of_feature_ids)
for feature in vector_layer.getFeatures(request):
    # do something with the feature

Now there’s just a single call to getFeatures() here. QGIS optimises this request by using a single connection to the data source, preparing the query just once, and fetching the results in appropriately sized batches. The difference is huge, especially if you’re dealing with a large number of features.

Trap #2: Use QgsFeatureRequest filters appropriately

Here’s another common mistake I see in PyQGIS code. I often see this one when an author is trying to do something with all the selected features in a layer:

for feature in vector_layer.getFeatures():
    if not feature.id() in vector_layer.selectedFeaturesIds():
        continue

    # do something with the feature

What’s happening here is that the code is iterating over all the features in the layer, and then skipping over any which aren’t in the list of selected features. See the problem here? This code iterates over EVERY feature in the layer. If you’re layer has 10 million features, we are fetching every one of these from the data source, going through all the work of parsing it into a QGIS feature, and then promptly discarding it if it’s not in our list of selected features. It’s very inefficient, especially if fetching features is slow (such as when connecting to a remote database source).

Instead, this code should use the setFilterFids() method for QgsFeatureRequest:

request = QgsFeatureRequest().setFilterFids(vector_layer.selectedFeaturesIds())
for feature in vector_layer.getFeatures(request):
    # do something with the feature

Now, QGIS will only fetch features from the provider with matching feature IDs from the list. Instead of fetching and processing every feature in the layer, only the actual selected features will be fetched. It’s not uncommon to see operations which previously took many minutes (or hours!) drop down to a few seconds after applying this fix.

Another variant of this trap uses expressions to test the returned features:

filter_expression = QgsExpression('my_field > 20')
for feature in vector_layer.getFeatures():
    if not filter_expression.evaluate(feature):
        continue

    # do something with the feature

Again, this code is fetching every single feature from the layer and then discarding it if it doesn’t match the “my_field > 20” filter expression. By rewriting this to:

request = QgsFeatureRequest().setFilterExpression('my_field > 20')
for feature in vector_layer.getFeatures(request):
    # do something with the feature

we hand over the bulk of the filtering to the data source itself. Recent QGIS versions intelligently translate the filter into a format which can be applied directly at the provider, meaning that any relevant indexes and other optimisations can be applied by the provider itself. In this case the rewritten code means that ONLY the features matching the ‘my_field > 20’ criteria are fetched from the provider – there’s no time wasted messing around with features we don’t need.

Trap #3: Only request values you need

The last trap I often see is that more values are requested from the layer then are actually required. Let’s take the code:

my_sum = 0
for feature in vector_layer.getFeatures(request):
    my_sum += feature['value']

In this case there’s no way we can optimise the filters applied, since we need to process every feature in the layer. But – this code is still inefficient. By default QGIS will fetch all the details for a feature from the provider. This includes all attribute values and the feature’s geometry. That’s a lot of processing – QGIS needs to transform the values from their original format into a format usable by QGIS, and the feature’s geometry needs to be parsed from it’s original type and rebuilt as a QgsGeometry object. In our sample code above we aren’t doing anything with the geometry, and we are only using a single attribute from the layer. By calling setFlags( QgsFeatureRequest.NoGeometry ) and setSubsetOfAttributes() we can tell QGIS that we don’t need the geometry, and we only require a single attribute’s value:

my_sum = 0
request = QgsFeatureRequest().setFlags(QgsFeatureRequest.NoGeometry).setSubsetOfAttributes(['value'], vector_layer.fields() )
for feature in vector_layer.getFeatures(request):
    my_sum += feature['value']

None of the unnecessary geometry parsing will occur, and only the ‘value’ attribute will be fetched and populated in the features. This cuts down both on the processing required AND the amount of data transfer between the layer’s provider and QGIS. It’s a significant improvement if you’re dealing with larger layers.

Conclusion

Optimising your feature requests is one of the easiest ways to speed up your PyQGIS script! It’s worth spending some time looking over all your uses of getFeatures() to see whether you can cut down on what you’re requesting – the results can often be mind blowing!

Last week I posted regarding some thoughts I’ve had recently concerning what I perceive as a general confusion about how QGIS is developed and how users can successfully get things to change in the project. The post certainly started a lot of conversation! However, based on feedback received I realise some parts of the posts were being misinterpreted and some clarification is needed. So here we go…

1. Please keep filing bug reports/feature requests

I don’t think I was very clear about this – but my original post wasn’t meant to be a discouragement from filing bug reports or feature requests. The truth is that there is a LOT of value in these reports, and if you don’t file a report then the QGIS team will never be aware of the bug or your feature idea. Here’s some reasons why you SHOULD file a report:

• QGIS developers are a conscientious mob, and generally take responsibility for any regressions they’ve caused by changes they’ve made. In other words, there’s very much an attitude of “I-broke-it, I’ll-fix-it” in the project. So, if a new feature is buggy or has broken something else then filing bugs ASAP is the best way to make the developer aware of these issues. In my experience they’ll usually be addressed rapidly.
• As mentioned in the original post – there’s always a pre-release bug fix sprint, so filing a bug (especially if it’s a critical one) may mean that it’s addressed during this sprint.
• Filing feature requests can gain traction if your idea is innovative, novel, or interesting enough to grab a developer’s attention!

Speaking for myself, I regularly check new incoming tickets (at least once a day), and I know I’m not the only one. So filing a report WILL bring your issue to developer’s attention. Which leads to…

2. Frustration is understandable!

I can honestly understand why people get frustrated and resort to an aggressive “why hasn’t this been fixed yet?!” style reply. I believe that these complaints are caused because people have the misunderstanding that filing a bug report is the ONLY thing they can do to get an issue fixed. If filing a report IS the only avenue you have to get something fixed/implemented, then it’s totally understandable to be annoyed when your ticket gets no results. This is a failing on behalf of the project though – we need to be clearly communicating that filing a report is the LEAST you can do. It’s a good first step, but on its own it’s just the beginning and needs to be followed up by one of the methods I described in the initial post.

3. It applies to more than just code

When I wrote the original piece I focused on just the code aspect of the QGIS project. That’s only because I’m a developer and it’s the area I know best. But it applies equally across the whole project, including documentation, translations, infrastructure, websites, packaging QGIS releases, etc. In fact, some of these non-code areas are the best entry points into the project as they don’t require a development background, and eg the documentation and translation teams have done a great job making it easy to submit contributions. Find something missing in the QGIS documentation? Add it yourself! Missing a translation of the website which prevents QGIS adoption within your community? Why not sponsor a translator to tackle this task?!

4. It applies to more than just QGIS!

Again, I wrote the original piece focusing on QGIS because that’s the project I’m most familiar with. You could just as easily substitute GDAL, GEOS, OpenLayers, PostGIS, Geoserver, R, D3, etc… in and it would be equally valid!

Hopefully that helps clarify some of the points raised by the earlier article. Let’s keep the discussion flowing – I’d love to hear if you have any other suggestions or questions raised by this topic.

In part 3 of my exploration of variables in QGIS 2.12, I’m going to dig into how variables are scoped in QGIS and what layer level variables are available (you can read parts 1 and 2 for a general introduction to variables).

Some background

Before we get to the good stuff, a bit of background in how variables work behind-the-scenes is important. Whenever an expression is evaluated in QGIS the context of the expression is considered. The context is built up from a set of scopes, which are all stacked on top of each other in order from least-specific to most-specific. It’s easier to explain with an example. Let’s take an expression used to set the source of a picture in a map composer. When this expression is evaluated, the context will consist of:

1. The global scope, consisting of variables set in the QGIS options dialog, and other installation-wide properties
2. The project scope, which includes variables set in the Project Properties dialog and the auto-generated project variables like @project_path, @project_title (you can read more about this in part 2)
3. A composer scope, with any variables set for the current composer, plus variables for @layout_pagewidth, @layout_pageheight, @layout_numpages, etc.
4. A composer item scope for the picture, with item-specific variables including @item_id

The more specific scopes will override any existing clashing variables from less specific scopes. So a global @my_var variable will be overridden by an @my_var variable set for the composer:

Another example. Let’s consider now an expression set for a data-defined label size. When this expression is evaluated the context will depend on where the map is being rendered. If it’s in the main map canvas then the context will be:

1. The global scope
2. The project scope
3. A map settings scope, with variables relating to how the map is being rendered. Eg @map_rotation, @map_scale, etc
4. A layer scope. More on this later, but the layer scope includes layer-level variables plus preset variables for @layer_name and @layer_id

If instead the map is being rendered inside a map item in a map composer, the context will be:

1. The global scope
2. The project scope
3. The composer scope
4. An atlas scope, if atlas is enabled. This contains variables like @atlas_pagename, @atlas_feature, @atlas_totalfeatures.
5. A composer item scope for the map item
6. A map settings scope (with scale and rotation determined by the map item’s settings)
7. The layer scope

Using layer level variables

Ok, enough with the details. The reason I’ve explained all this is to help explain when layer level variables come into play. Basically, they’ll be available whenever an expression is evaluated inside of a particular layer. This includes data defined symbology and labeling, field calculator, and diagrams. You can’t use a layer-level variable inside a composer label, because there’s no layer scope used when evaluating this. Make sense? Great! To set a layer level variable, you use the Variables section in the Layer Properties dialog:

Setting a layer variable

Any layer level variables you set will be saved inside your current project, i.e. layer variables are per-layer and per-project. You can also see in the above screenshot that as well as the layer level variables QGIS also lists the existing variables from the Project and Global scopes. This helps show exactly what variables are accessible by the layer and whether they’ve been overridden by any scopes. You can also see that there’s two automatic variables, @layer_id and @layer_name, which contain the unique layer ID and user-set layer name too.

Potential use cases for layer-level variables

In the screenshot above I’ve set two variables, @class1_threshold and @class2_threshold. I’m going to use these to sync up some manual class breaks between rule based symbology and rule based labeling. Here’s how I’ve set up the rule-based symbols for the layer:

Rule based symbology using layer level variables

In a similar way, I’ve also created matching rule-based labeling (another new feature in QGIS 2.12):

Matching rule-based labels

Here’s what my map looks like now, with label and symbol colors matched:

*Map for illustrative purposes only… not for cartographic/visual design excellence!

If I’d hard-coded the manual class breaks, it would be a pain to keep the labeling and symbology in sync. I’d have to make sure that the breaks are updated everywhere I’ve used them in both the symbology and labeling settings. Aside from being boring, tedious work, this would also prevent immediate before/after comparisons. Using variables instead means that I can update the break value in a single place (the variables panel) and have all my labeling and symbols immediately reflect this change when I hit apply!

Another recent use case I had was teaming layer-level variables along with Time Manager. I wanted my points to falloff in both transparency and size with age, and this involved data defined symbol settings scattered all throughout my layer symbology. By storing the decay fall-off rate in a variable, I could again tweak this falloff by changing the value in a single place and immediately see the result. It also helps with readability of the data defined expressions. Instead of trying to decipher a random, hard-coded value, it’s instead immediately obvious that this value relates to a decay fall-off rate. Much nicer!

I’m sure there’s going to be hundreds of novel uses of layer-level variables which I never planned for when adding this feature. I’d love to hear about them though – leave a comment if you’d like to share your ideas!

One last thing – the new “layer_property” function

This isn’t strictly related to variables, but another new feature which was introduced in QGIS 2.12 was a new “layer_property” expression function. This function allows you to retrieve any one of a bunch of properties relating to a specific map layer, including the layer CRS, metadata, source path, etc.

This function can be used anywhere in QGIS. For instance, it allows you to insert dynamic metadata about layers into a print composer layout. In the screenshot below I’ve used expressions like layer_property(‘patron’,’crs’) and layer_property(‘patron’,’source’) to insert the CRS and source path of the “patron” layer into the label. If either the CRS or the file path ever changes, this label will be automatically updated to reflect the new values.

Inserting dynamic layer properties into a composer label

So there you go – layer level variables and the layer_property function – here in QGIS 2.12 and making your workflow in QGIS easier. In the final part of this series, we’ll explore the magical @value variable. Trust me, I’ve saved the best for last!

Following on from part 1 in which I introduced how variables can be used in map composers, I’d like to now explore how using variables can make it easier to manage your QGIS projects. As a quick refresher, variables are a new feature in QGIS 2.12 which allow you to create preset values for use anywhere you can use an expression in QGIS.

Let’s imagine a typical map project. You load up QGIS, throw a bunch of layers on your map, and then get stuck into styling and labelling them ‘just right’. Over time the project gets more and more complex, with a stack of layers all styled using different rendering and labelling rules. You keep tweaking settings until you’re almost happy with the result, but eventually realise that you made the wrong choice of font for the labelling and now need to go through all your layers and labelling rules and update each in turn to the new typeface. Ouch.

Variables to the rescue! As you may recall from part 1, you can reuse variables anywhere in QGIS where you can enter an expression. This includes using them for data defined overrides in symbology and labelling. So, lets imagine that way back at the beginning of our project we created a project level variable called @main_label_font:

Creating a variable for label font

Now, we can re-use that variable in a data defined override for the label font setting. In fact, QGIS makes this even easier for you by showing a “variables” sub-menu allowing easy access to all the currently defined variables accessible to the layer:

Binding the label font to the @main_label_font variable

When we hit Apply all our labels will be updated to use the font face defined by the @main_label_font variable, so in this case ‘Courier New’:

In a similar way we can bind all the other layer’s label fonts to the same variable, so @main_label_font will be reused by all the layers in the project. Then, when we later realise that Courier New was a horrible choice for labelling the map, it’s just a matter of opening up the Project Properties dialog and updating the value of the @main_label_font variable:

And now when we hit Apply the font for all our labelled layers will be updated all at once:

It’s not only a huge time saver, it also makes changes like this easier because you can try out different font faces by updating the variable and hitting apply and seeing the effect that the changes have all at once. Updating multiple layers manually tends to have the consequence that you forget what the map looked like before you started making the change, making direct comparisons harder.

Of course, you could have multiple variables for other fonts used by your project too, eg @secondary_label_font and @highlighted_feature_font. Plus, this approach isn’t limited to just setting the label font. You could utilise project level variables for consolidating font sizes, symbol line thickness, marker rotation, in fact, ANYTHING that has one of those handy little data defined override buttons next to it:

See all those nice little yellow buttons? All those controls can be bound to variables…

One last thing before I wrap up part 2 of this series. The same underlying changes which introduced variables to QGIS also allows us to begin introducing a whole stack of new, useful functions to the expression engine. One of these which also helps with project management is the new project_color function. Just like how we can use project level variables throughout a project, project_color lets you reuse a color throughout your project. First, you need to create a named colour in the Default Styles group under the Project Properties dialog:

Define a colour in the project’s colour scheme…

Then, you can set a data defined override for a symbol or label colour to the expression “project_color(‘red alert!’)“:

When you go back and change the corresponding colour in the Project Properties dialog, every symbol bound to this colour will also be updated!

So, there you have it. With a little bit of forward planning and by taking advantage of the power of expression variables in QGIS 2.12 you can help make your mapping projects much easier to manage and update!

That’s all for now, but we’re still only just getting started with variables. Part 3, coming soon!.. (Update: Part 3 is available now)