QGIS Planet

A year ago I have asked QGIS’s community what were their favourite QGIS new features from 2015 and published this blog post. This year I decided to ask it again. In 2016, we add the release of the second long-term release (2.14 LTR), and two other stable versions (2.16 and 2.18).

2016 was again very productive year for the QGIS community, with lots of improvements and new features landing on QGIS source code, not to speak of all the work already in place for QGIS 3. This is a great assurance of the project’s vitality.

As a balance, I have asked users to choose wich were their favorite new features during 2016 (from the visual changelogs list). As a result, I got the following Top 5 features list.

5 – Paste a style to multiple selected layers or to all layers in a legend group (2.14)

This is a productivity functionaly that I just realized that existed now, with so many people voting on it. If copy/paste styles was, in my opinion, a killer feature, being able to use it in multiple layers or even a group is just great.

4 – fTools plugin has been replaced with Processing algorithms (2.16)

While checking the Vector Menu, the tools seem the same as previous version, but it’s when you open them that you understand the difference. All vector tools, provided until now by the fTools core plugin, were replaced by equivalent processing Algoritms. For the users it means easier access to more functionality, like running the tools in batch mode, or getting outputs as temporary layers. Besides some of the tools have been improved.

3 – Virtual layers (2.14)

This is definitly one of my favourite new features, and it seems I’m not alone. With virtual layers you can run SQL queries using the layers loaded in the project, even when the layers are not stored in a relational database. We are not talking about WHERE statments to filter data, with this you can do real SQL queries, with spatial analysis, aggregations, and so on. Besides, virtual layers will act as VIEWs and any changes to any of the input layers will automatically update the layer.

2 – Speed and memory improvements (2.14)

It’s no surprise that speed and memory improvements we one of the most voted features. Lots of improvements were made for loading and managing large datasets, and this have a tremendous impact in all users. According to the changelog, zoom is faster, selecting features is faster, updating attributes on selected features is faster, and it consumes less memory. So don’t be afraid to put QGIS to the test.

1 – Trace digitising tool (2.14)

If you do lots of digitising, you better look into this new feaure that landed on QGIS 2.14. It allows you to digitize new feature by using other layers boundaries. Besides the quality improvement of layers topology, this can make digitizing almost feel pleasing and fast! Just click the first point, move your mouse around other features edged to pick up more vertex.

There were other new features that also made the delight of many users. For example, several improvements on the labeling, Georeference outputs (eg PDF) from composer (2.16), Filter legend by expression (2.14), 2.5D Renderer. Personally, the Style docker made my day/year. But you can check the full results of the survey, if you like.

Obviously, this list means nothing at all. All new features were of tremendous value, and will be useful for thousands (yes thousands) of people. It was a mere exercise as, with such a diverse QGIS crowd, it would be impossible to build a list that would fit us all. Besides, there were many great enhancements, introduced during 2016, that might have fallen under the radar for most users. Check the visual changelogs for a full list of new features.

On my behalf, to all developers, sponsors, and general QGIS contributors, once again

THANK YOU VERY MUCH FOR YOUR TREMENDOUS WORK!

I wish you a fantastic 2017.

I’ve recently spent some time optimising the performance of various QGIS plugins and algorithms, and I’ve noticed that there’s a few common performance traps which developers fall into when fetching features from a vector layer. In this post I’m going to explore these traps, what makes them slow, and how to avoid them.

As a bit of background, features are fetched from a vector layer in QGIS using a QgsFeatureRequest object. Common use is something like this:

request = QgsFeatureRequest()
for feature in vector_layer.getFeatures(request):
    # do something

This code would iterate over all the features in layer. Filtering the features is done by tweaking the QgsFeatureRequest, such as:

request = QgsFeatureRequest().setFilterFid(1001)
feature_1001 = next(vector_layer.getFeatures(request))

In this case calling getFeatures(request) just returns the single feature with an ID of 1001 (which is why we shortcut and use next(…) here instead of iterating over the results).

Now, here’s the trap: calling getFeatures is expensive. If you call it on a vector layer, QGIS will be required to setup an new connection to the data store (the layer provider), create some query to return data, and parse each result as it is returned from the provider. This can be slow, especially if you’re working with some type of remote layer, such as a PostGIS table over a VPN connection. This brings us to our first trap:

Trap #1: Minimise the calls to getFeatures()

A common task in PyQGIS code is to take a list of feature IDs and then request those features from the layer. A see a lot of older code which does this using something like:

for id in some_list_of_feature_ids:
    request = QgsFeatureRequest().setFilterFid(id)
    feature = next(vector_layer.getFeatures(request))
    # do something with the feature

Why is this a bad idea? Well, remember that every time you call getFeatures() QGIS needs to do a whole bunch of things before it can start giving you the matching features. In this case, the code is calling getFeatures() once for every feature ID in the list. So if the list had 100 features, that means QGIS is having to create a connection to the data source, set up and prepare a query to match a single feature, wait for the provider to process that, and then finally parse the single feature result. That’s a lot of wasted processing!

If the code is rewritten to take the call to getFeatures() outside of the loop, then the result is:

request = QgsFeatureRequest().setFilterFids(some_list_of_feature_ids)
for feature in vector_layer.getFeatures(request):
    # do something with the feature

Now there’s just a single call to getFeatures() here. QGIS optimises this request by using a single connection to the data source, preparing the query just once, and fetching the results in appropriately sized batches. The difference is huge, especially if you’re dealing with a large number of features.

Trap #2: Use QgsFeatureRequest filters appropriately

Here’s another common mistake I see in PyQGIS code. I often see this one when an author is trying to do something with all the selected features in a layer:

for feature in vector_layer.getFeatures():
    if not feature.id() in vector_layer.selectedFeaturesIds():
        continue

    # do something with the feature

What’s happening here is that the code is iterating over all the features in the layer, and then skipping over any which aren’t in the list of selected features. See the problem here? This code iterates over EVERY feature in the layer. If you’re layer has 10 million features, we are fetching every one of these from the data source, going through all the work of parsing it into a QGIS feature, and then promptly discarding it if it’s not in our list of selected features. It’s very inefficient, especially if fetching features is slow (such as when connecting to a remote database source).

Instead, this code should use the setFilterFids() method for QgsFeatureRequest:

request = QgsFeatureRequest().setFilterFids(vector_layer.selectedFeaturesIds())
for feature in vector_layer.getFeatures(request):
    # do something with the feature

Now, QGIS will only fetch features from the provider with matching feature IDs from the list. Instead of fetching and processing every feature in the layer, only the actual selected features will be fetched. It’s not uncommon to see operations which previously took many minutes (or hours!) drop down to a few seconds after applying this fix.

Another variant of this trap uses expressions to test the returned features:

filter_expression = QgsExpression('my_field > 20')
for feature in vector_layer.getFeatures():
    if not filter_expression.evaluate(feature):
        continue

    # do something with the feature

Again, this code is fetching every single feature from the layer and then discarding it if it doesn’t match the “my_field > 20” filter expression. By rewriting this to:

request = QgsFeatureRequest().setFilterExpression('my_field > 20')
for feature in vector_layer.getFeatures(request):
    # do something with the feature

we hand over the bulk of the filtering to the data source itself. Recent QGIS versions intelligently translate the filter into a format which can be applied directly at the provider, meaning that any relevant indexes and other optimisations can be applied by the provider itself. In this case the rewritten code means that ONLY the features matching the ‘my_field > 20’ criteria are fetched from the provider – there’s no time wasted messing around with features we don’t need.

Trap #3: Only request values you need

The last trap I often see is that more values are requested from the layer then are actually required. Let’s take the code:

my_sum = 0
for feature in vector_layer.getFeatures(request):
    my_sum += feature['value']

In this case there’s no way we can optimise the filters applied, since we need to process every feature in the layer. But – this code is still inefficient. By default QGIS will fetch all the details for a feature from the provider. This includes all attribute values and the feature’s geometry. That’s a lot of processing – QGIS needs to transform the values from their original format into a format usable by QGIS, and the feature’s geometry needs to be parsed from it’s original type and rebuilt as a QgsGeometry object. In our sample code above we aren’t doing anything with the geometry, and we are only using a single attribute from the layer. By calling setFlags( QgsFeatureRequest.NoGeometry ) and setSubsetOfAttributes() we can tell QGIS that we don’t need the geometry, and we only require a single attribute’s value:

my_sum = 0
request = QgsFeatureRequest().setFlags(QgsFeatureRequest.NoGeometry).setSubsetOfAttributes(['value'], vector_layer.fields() )
for feature in vector_layer.getFeatures(request):
    my_sum += feature['value']

None of the unnecessary geometry parsing will occur, and only the ‘value’ attribute will be fetched and populated in the features. This cuts down both on the processing required AND the amount of data transfer between the layer’s provider and QGIS. It’s a significant improvement if you’re dealing with larger layers.

Conclusion

Optimising your feature requests is one of the easiest ways to speed up your PyQGIS script! It’s worth spending some time looking over all your uses of getFeatures() to see whether you can cut down on what you’re requesting – the results can often be mind blowing!

The 6th QGIS UK user group meeting in Scotland is happening on the 3rd November 2016.  It is being hosted by the EDINA University of Edinburgh at the Informatics Forum and is sponsored by thinkWhere, Ordnance Survey, Angus Council and Cawdor Forestry.  Tickets are available through Eventbrite.

The almost final programme of presentations and lightning talks is as follows:

• Phil Taylor (CEH) – How deep is your loch?
• Fiona Hemsley-Flint – QGIS server
• University of Edinburgh – packaging and deploying QGIS
• Anouk Lang – Mapping narrative: QGIS in the humanities classroom
• Art Lembo (Salisbury University, MD) – terrain analysis with massively parallel processing techniques (embarrasingly so)
• Neil Benny (thinkWhere) – finding the heart of Scotland / viewshed analysis
• Tom Chadwin – qgis2web and coding a QGIS plugin
• Pete Wells (Lutra) – WMTS previews and XYZ support
• Stephen Bathgate – decision support system in Forestry
• Tim Manners (Ordnance Survey) – Creating an indoor routable network with QGIS and pgRouting
• Andrew Whitelee – QGIS in forestry/ecology
• Ross McDonald (Angus Council) – Them thar hills: shaded, textured and blended
• Michal Michalski (The Origins of Doha and Qatar Project) – DOHA: Doha Online Historical Atlas
• eeGeo – Using QGIS to create 3D indoor maps

Doors open from 9:00. Registration shortly thereafter. Start and welcome at 9:45 and a planned finish at 16:30. Geobeers to follow.

Last week I posted regarding some thoughts I’ve had recently concerning what I perceive as a general confusion about how QGIS is developed and how users can successfully get things to change in the project. The post certainly started a lot of conversation! However, based on feedback received I realise some parts of the posts were being misinterpreted and some clarification is needed. So here we go…

1. Please keep filing bug reports/feature requests

I don’t think I was very clear about this – but my original post wasn’t meant to be a discouragement from filing bug reports or feature requests. The truth is that there is a LOT of value in these reports, and if you don’t file a report then the QGIS team will never be aware of the bug or your feature idea. Here’s some reasons why you SHOULD file a report:

• QGIS developers are a conscientious mob, and generally take responsibility for any regressions they’ve caused by changes they’ve made. In other words, there’s very much an attitude of “I-broke-it, I’ll-fix-it” in the project. So, if a new feature is buggy or has broken something else then filing bugs ASAP is the best way to make the developer aware of these issues. In my experience they’ll usually be addressed rapidly.
• As mentioned in the original post – there’s always a pre-release bug fix sprint, so filing a bug (especially if it’s a critical one) may mean that it’s addressed during this sprint.
• Filing feature requests can gain traction if your idea is innovative, novel, or interesting enough to grab a developer’s attention!

Speaking for myself, I regularly check new incoming tickets (at least once a day), and I know I’m not the only one. So filing a report WILL bring your issue to developer’s attention. Which leads to…

2. Frustration is understandable!

I can honestly understand why people get frustrated and resort to an aggressive “why hasn’t this been fixed yet?!” style reply. I believe that these complaints are caused because people have the misunderstanding that filing a bug report is the ONLY thing they can do to get an issue fixed. If filing a report IS the only avenue you have to get something fixed/implemented, then it’s totally understandable to be annoyed when your ticket gets no results. This is a failing on behalf of the project though – we need to be clearly communicating that filing a report is the LEAST you can do. It’s a good first step, but on its own it’s just the beginning and needs to be followed up by one of the methods I described in the initial post.

3. It applies to more than just code

When I wrote the original piece I focused on just the code aspect of the QGIS project. That’s only because I’m a developer and it’s the area I know best. But it applies equally across the whole project, including documentation, translations, infrastructure, websites, packaging QGIS releases, etc. In fact, some of these non-code areas are the best entry points into the project as they don’t require a development background, and eg the documentation and translation teams have done a great job making it easy to submit contributions. Find something missing in the QGIS documentation? Add it yourself! Missing a translation of the website which prevents QGIS adoption within your community? Why not sponsor a translator to tackle this task?!

4. It applies to more than just QGIS!

Again, I wrote the original piece focusing on QGIS because that’s the project I’m most familiar with. You could just as easily substitute GDAL, GEOS, OpenLayers, PostGIS, Geoserver, R, D3, etc… in and it would be equally valid!

Hopefully that helps clarify some of the points raised by the earlier article. Let’s keep the discussion flowing – I’d love to hear if you have any other suggestions or questions raised by this topic.

I’ve been heavily involved in the open source QGIS mapping project for a number of years now. During this time I’ve kept a close watch on the various mailing lists, issue trackers, stackexchange, tweets and other various means users have to provide feedback to the project. Recently, I’ve started to come to the conclusion that there’s a lot of fundamental confusion about how the project works and how users can get changes made to the project. Read on for these insights, but keep in mind that these are just my thoughts and not reflective of the whole community’s views!..

Firstly – QGIS is a community driven project. Unlike some open source projects (and unlike the proprietary GIS offerings) there is no corporate backer or singular organisation directing the project. This means two things:

1. The bad news: No-one will do your work for you. QGIS has been created through a mix of user-led contributions (ie, users who have a need to change something and dive in and do it themselves) and through commercially supported contributions (either organisations who offer commercial QGIS support pushing fixes because their customers are directly affected or because they’ve been contracted by someone to implement a particular change). There HAS been a number of volunteer contributions from developers who are just donating their time (for various reasons), but these contributions are very much the minority.
2. The good news: YOU have the power to shape the project! (And whenever I say “you” – I’m referring directly to the person reading this, or the company you work for. Just pretend it’s in 24 point bold red blinking text.) Because QGIS is community driven (and not subject to the whims of any one particular enterprise) every user has the ability to implement changes and fixes in the program.

So how exactly can users get changes implemented in QGIS? Well, let’s take a look at all the possible different ways that changes get made and how effective each one is:

1. YOU can make the changes yourself. This implies that you have the c++/Python skills required to make the changes, are able to find your way around the source code, and push through the initial hurdles of setting up a build environment and navigating git. This can be a significant time investment, but the ultimate result is that you can make whatever changes you want, and so long as your pull request is accepted you’ll get your changes directly into QGIS. You’ll find the QGIS team is very open to new contributors and will readily lend a hand if you need assistance navigating the source or for advise on the best way to make these changes. Just ask!
2. YOU (or your employer) can pay (or “sponsor”) someone to make the changes on your behalf. Reinvesting some of those savings you’re making through using an open source program back into the program itself is a great idea, and everyone benefits. There’s numerous organisations who specialise in QGIS development (eg… my own consultancy, North Road). You can liaise with these organisations to get them to make the changes on your behalf. This is probably the most effective way of getting changes implemented. These organisations all have a history with QGIS development and this experience generally translates to much faster development then if you code it yourself. It’s also somewhat of a shortcut – if you hire a core QGIS developer to make your changes, then you can be confident that they are familiar with the coding style, policies, and long-term goals of the project and accordingly can get the changes accepted rapidly. The obvious down side of paying for changes is that, well, it costs money. Understandably, not everyone has the resources available to do this.
3. Following on from option 2 – if you can’t directly sponsor changes yourself, you could help indirectly raise funds to pay for the changes. This is a great way to get changes implemented, because everyone has the power to do this. You could seek out similar organisations/users who have the same need and pool your resources, get involved with the local QGIS user group and raise funds together, organise a crowd-funding campaign, etc.
4. Ask a developer to make the changes for you. This is not terribly effective – you’re basically asking someone to work for free, and take time away from their family/job/hobbies/social life to do work for you. That said, it does sometimes happen, and here’s a few reasons I can think of why:
   • You’ve build up enough “karma” within the project through other contributions. If someone has been heavily involved in the non-development side of the project (eg translations, documentation, helping users out on mailing lists/stackexchange, organising hackfests or user groups, etc) then developers are much more likely to want to help them out in turn.
   • You’ve got a fantastic idea which has just never occurred to anyone before. By bringing it to the attention of a developer you might trigger the “wow, I could really benefit from that too!” impulse which is hard-wired into some of us!
   • It’s a particularly interesting or challenging problem, and sometimes developers just like to extend themselves.
5. (For bugs only) File a bug report, and hope it gets picked up in one of the pre-release bug fixing sprints. This is basically the same as option 2 – expect that in this case someone else (the QGIS steering committee) is paying for the development time. There’s no way of guaranteeing that your bug will get fixed during this time though, so it’s not a particularly reliable approach if the fix is critical for you.

Finally, there’s two more very ineffective approaches:

6. File a bug report/feature request, and wait. This isn’t very effective, because what you’re doing is basically the same as 1-4 above, but just waiting for someone else to either do the work or sponsor the changes. This might happen in a week, or might take 10 years.
7. Complain about something and hope for the best. This is… not very effective. No-one is particularly motivated to help out someone who is being a jerk.

That’s it. Those are the ONLY ways changes get made in QGIS. There’s no other magical short-cuts you can take. Some of these approaches are much more effective than others, and some require skills or resources which may not be available. If you want to see something change in QGIS, you need to take a look at these options and decide for yourself which best meets your needs. But please, just don’t choose option 7!

Update: a follow up to this article was published

The 5th QGIS user group meeting in Scotland takes place next Wednesday at the University of Glasgow.  It is being hosted by the School of Geographical and Earth Sciences and has been generously sponsored by thinkWhere and Ordnance Survey.  You can find the draft programme of talks and presentations here: 5th-QGIS-user-group-programme

All tickets are now gone but get on the waitlist and you may be lucky.

See you all there!

(aka Features that ArcGIS Desktop users might not know that exists)

EN | PT

From time to time, I read articles comparing ArcGIS vs QGIS. Since many of those articles are created from an ArcGIS user point of view, they invariably lead to biased observations on QGIS lack of features. It’s time for a QGIS user perspective, so bare with me on this (a bit) long, totally and openly biased post.

“Hello, my name is Alexandre, and I have been using… QGIS“

This is something I would say at an anonymous QGIS user therapy group. I’m willing to attend one of those because being recently and temporally forced to use ArcGIS Desktop again (don’t ask!), I really really really miss QGIS in many ways.

There was a time when I have used ArcGIS on the regular basis. I used it until version 9.3.1 and then decided to move away (toward the light) into QGIS (1.7.4 I think). At that time, I missed some (or even many?) ArcGIS features, but I was willing to accept it in exchange for the freedom of the Open Source philosophy. Since then, a lot have happened in the QGIS world, and I have been watching it closely. I would expect the same have happened in ArcGIS side, but, as far I can see, it didn’t.

I’m using top shelf ArcGIS Desktop Advanced and I’m struggling to do very basic tasks that simply are nonexistent in ArcGIS. So here’s my short list of QGIS functionalities that I’m longing for. For those of you that use ArcGIS exclusively, some of this features may catch you by surprise.

Warning: For those of you that use ArcGIS exclusively, some of this features may catch you by surprise.

Transparency control

“ArcGIS have transparency! It’s in the Display tab, in the layer’s properties dialog!”

Yes, but… you can only set transparency at the layer level. That is, either it’s all transparent, or it’s not…

In QGIS on the other end, you can set transparency at layer level, feature/symbol level, and color level. You can say that this is being overrated, but check the differences in the following images.

Notice that in QGIS you can set transparency at color level everywhere (or almost everywhere) there is a color to select. This includes annotations (like the ones in the image above), labels and composers items. You can even set transparency in colors by using the RGBA function in an expression! How sweet can that be?

Blending modes

This is one of QGIS’s pristine jewels. The ability to combine layers the way you would do in almost any design/photo editing software. At layer or at feature level, you can control how they will “interact” with other layers or features below. Besides the normal mode, QGIS offers 12 other blending modes:  Lighten, Screen, Dodge, Darken, Multiply, Burn, Overlay, Soft light, Hard light, Difference, and Subtract. Check this page to know more about the math behind each one and this image for some examples

It’s not easy to grasp how can this be of any use for cartography before you try it yourself. I had the chance to play around while trying to answer this question.

A very common application for this functionality is when you want to add shadows to simulate the relief by putting a hill shade on top of other layers. In ArcGIS, you can only control the layer transparency, and the result is always a bit pale. But in QGIS, you can keep the strength of the original colors by using the multiply mode in the hill shade layer.

You can also use blending modes in the print composer items, allowing you to combine them with other items and textures. This gives you the opportunity to make more “artistic” things without the need to go post-processing in a design software.

Colour Picker Menu

Controlling color is a very important deal for a cartographer and QGIS allow you to control it like the professional you are. You can select your colours using many different interfaces. Interfaces that you might recognize from software like Inkscape, Photoshop, Gimp and others alike.

My favorite one is the color picker. Using color picker, you can pick colors from anywhere on your screen, either from QGIS itself or outside. This is quite handy and productive when you are trying to use a color from your map, it’s legend, a COLOURlovers palette or a company logo.

You can also copy/paste colors between dialogs, save and import color palettes, and you can even name a color and use it in a variable. With all this available for free, it’s hard to swallow Windows color selector :(.

Vector symbols renderers “powertools”

In ArcGIS, you have a few fancy ways to symbol your vector layers. You got: Single symbol, Unique values, Unique values many fields… and so on. At the first glance, you may think that QGIS lacks some of them. Believe me, it doesn’t! In fact, QGIS offers much more flexibility when you need to symbol your layers.

For starters, it allows you to use fields or expressions on any of the symbols renderers, while ArcGIS only allows the use of fields. Powered by hundreds of functions and the ability to create your owns in python, what you can do with the expression builder has no limits. This means, for instance, that you can combine, select, recalculate, normalize an infinite number of fields to create your own “values” (not to mention that you can tweak your values labels, making it ideal to create the legend).

And then, in QGIS, you have the special (and kinda very specific) renderers, that make you say “wooooh”. Like the Inverted polygons that allow you to fill the the outside of polygons (nice to mask other features), the Point displacement to show points that are too close to each others, and the Heatmap that will transform, ON-THE-FLY, all your points in a layer into a nice heatmap without the need to convert them to raster (and that will update as soon as you, or anyone else, edits the point features).

But I have left the best to the end. The “One rendered to Rule them all”, the Rule-based symbols. With the rule-based renderer, you can add several rules, group them in a tree structure, and set each one with a desired symbol. This gives QGIS users full control of their layer’s symbols, and, paired with expression builder and data-defined properties, opens the door to many wonderful applications.

Atlas

One of my favorite (and missed) features in QGIS is the Map Composer’s Atlas. I know that ArcGIS has its own “Atlas”, the Data Driven Pages, but frankly, it’s simply not the same.

I believe you know the basic functionally that both software allow. You create a map layout and choose a vector layer as coverage, and it will create an output panned or zoomed for each of the layer’s feature. You can also add some labels that will change according to the layers attributes.

But in QGIS, things go a little bit further…

Basically, you can use coverage layer’s attributes and geometry anywhere you can use an expression. And, in QGIS, expressions are everywhere. That way, most layers and map composer items properties can be controlled by a single coverage layer.

With the right configuration, while iterating over the atlas coverage features, you can,  choose what feature to show and what features to hide, change a theme color for your map, rotate and resize your page acording to the feature size, choose a specific logo to came along with your map, and much more. Once again, the sky is the limit.

So, if you pair Atlas it with QGIS data-defined properties, rule-based symbols and expressions, ArcGIS Data Driven Pages are no match. You don’t think so? Try to answer this question then.

Tip: If you really want to leverage your map production, using Spatialite or Postgis databases you can create the perfect atlas coverage layers from views that fit your needs. No data redundancy and they are always updated.

Label and Style dialogs

This one is more of a User Experience thing than an actual feature, but you won’t imagine how refreshing it is to have all Style and Labels options in two single dialogs (with several tabs, of course).

Using the symbol menu in ArcGIS makes me feel like if I’m in the Inception movie, at some point in time, I no longer know where the hell am I. For example, to apply a dashed outline in a fill symbol I needed to open 5 different dialogs, and then go back clicking OK, OK, OK, OK …

In QGIS, once in the properties menu, every setting is (almost) one click way. And you just need to press OK (or Apply ) once to see the result!

As an extra, you can copy/paste styles from one layer to another, making styling several layers even faster. And now you say:

“Don’t be silly! In ArcGIS you can import symbols from other layers too.”

Symbols? yes. Labels? No! And if you had lots of work setting your fancy labels, having to do the exact same/similar thing in another layer, it will make you wanna cry… I did.

(I think I will leave the multiple styles per layer vs data frames comparison for another time)

WFS

“Say what?!!”

Yup, that’s it, ArcGIS Desktop lacks support for WFS OGC standard unless you buy an extra extension: The Data Interoperability Extention.

In a GIS world that, more and more, is evolving towards Open Data, Open Standards and OGC Web Services, this reveals a very mercantile approach by ESRI. If I were an ESRI customer, I would feel outraged. <sarcasm>Or maybe not… maybe I would thank the opportunity to yet invest some more money in it’s really advanced features…<\sarcasm>

In QGIS, like everything else, WFS is absolutely free (as in freedom, not free beer). All you need to do is add the WFS server’s URL, and you can add all the layers you want, with the absolute sure that they are as updated as you can get.

Fortunately for ArcGIS users with a low budget, they can easily make a request for a layer using the internet browser :-P.

http://giswebservices.massgis.state.ma.us/geoserver/wfs?request=GetFeature&service=wfs&version=1.0.0&typename=massgis:GISDATA.TOWNS_POLY&outputformat=SHAPE-ZIP

Or they can simply use QGIS to download it. But, in both cases, be aware that the layers won’t update themselves by magic.

Expression builder

I have already mentioned the use of expressions several times, but for those of you that do not know the expression Builder, I though I end my post with one of my all time favourite features in QGIS.

I do not know enough of ArcGIS expression builder to really criticize it. But, AFAIK, you can use it to create labels and to populate a field using the field calculator. I know that there are many functions that you can use (I have used just a few) but they are not visible to the common user (you probably need to consult the ArcGIS Desktop Help to know them all). And you can create your own functions in VBScript, Python, and JsScript.

On QGIS side, like I said before, the Expression Builder can be used almost everywhere, and this makes it very convenient for many different purposes. In terms of functions, you have hundreds of functions right there in the builder’s dialog, with the corresponding syntax help, and some examples. You also have the fields and values like in ArcGIS, and you even have a “recent expressions” group for re-using recent expressions with no the need to remember prior expression.

Besides, you can create your own functions using Python (no VBScript or JsScript). For this purpose, you have a separate tab with a function editor. The editor have code highlighting and save your functions in your user area, making it available for later use (even for other QGIS sessions).

Conclusion

These are certainly not the only QGIS features that I miss, and they are probably not the most limiting ones (for instance, not being able to fully work with Spatialite and Postgis databases will make, for sure, my life miserable in the near future), but they were the ones I noticed right away when I (re)open ArcGIS for the first time.

I also feel that, based on the QGIS current development momentum, with each QGIS Changelog, the list will grow very fast. And although I haven’t tested ArcGIS Pro, I don’t think ESRI will be able to keep the pace.

“Are there things I still miss from ArcGIS?” Sure. I miss CMYK color support while exporting maps, for instance. But not as much as I miss QGIS now. Besides, I know that those will be addressed sooner or later.

In the end, I kinda enjoyed the opportunity to go back to ArcGIS, as it reinforced the way I think about QGIS. It’s all about freedom! Not only the freedom to use the software (that I was already aware) but also the freedom to control software itself and it’s outputs. Maintaining the users friendliness for new users, a lot have been done to make power users life easier, and they feel very pleased with it (at least I am).

All this being said, the winner is… QGIS!!

The End

(of a very biased post)

EN | PT

With the release of the first long term release (2.8 LTR), and two other stable versions (2.10 and 2.12), 2015 was a great (and busy) year for the QGIS community, with lots of improvements and new features landing on QGIS source code.

As a balance, I have asked users to choose wich were their favorite new features during 2015 (from the visual changelogs list). As a result I got the following Top 5 features list.

5 – Python console improvements (2.8)

Since QGIS 2.8, we can drag and drop python scripts into QGIS window and they will be executed automatically. There is also a new a toolbar icon in the plugins toolbar and a shortcut ( Ctrl-Alt-P) for quick access to the python console.

4 – Processing new algorithms (2.8)

Also in QGIS 2.8, there were introduced some new algorithms to the processing framework. If you are into spatial analysis this must have done your day (or year).

• Regular points algorithm
• Symmetrical difference algorithm
• Vector split algorithm
• Vector grid algorithm
• Hypsometric curves calculation algorithm
• Split lines with lines
• Refactor fields attributes manipulation algorithm

3 – Show rule-based renderer’s legend as a tree (2.8)

There were introduced a few nice improvements to QGIS legend. Version 2.8 brought us a tree presentation for the rule-based renderer. Better still, each node in the tree can be toggled on/off individually providing for great flexibility in which sublayers get rendered in your map.

2 – Advanced digitizing tools (2.8)

If you ever wished you could digitize lines exactly parallel or at right angles, lock lines to specific angles and so on in QGIS? Since QGIS 2.8 you can! The advanced digitizing tools are a port of the CADinput plugin and adds a new panel to QGIS. The panel becomes active when capturing new geometries or geometry parts.

1 – Rule-based labeling (2.12)

This was a very awaited feature (at least by me), and it was voted by the majority of users. Since 2.12, you can style features labels using rules. This gives us even more control over placement and styling of labels. Just like the rule based cartographic rendering, label rules can be nested to allow for extremely flexible styling options. For example, you can render labels differently based on the size of the feature they will be rendered into (as illustrated in the screenshot).

There were other new features that also made the delight of many users. For example, the Improved/consistent projection selection (2.8), PostGIS provider improvements (2.12), Geometry Checker and Geometry Snapper plugins (2.12), and Multiple styles per layer (2.8).

Don’t agree with this list? You can still cast your votes. You can also check the complete results in here.

Obviously, this list means nothing at all. I was a mere exercise as with such a diverse QGIS crowd it would be impossible to build a list that would fit us all. Besides, there were many great enhancements, introduced during 2015, that might have fallen under the radar for most users. Check the visual changelogs for a full list of new features.

On my behalf, to all developers, sponsors and general QGIS contributors,

THANK YOU VERY MUCH FOR YOUR TREMENDOUS WORK!

I wish you a fantastic (and productive) 2016.

In part 3 of my exploration of variables in QGIS 2.12, I’m going to dig into how variables are scoped in QGIS and what layer level variables are available (you can read parts 1 and 2 for a general introduction to variables).

Some background

Before we get to the good stuff, a bit of background in how variables work behind-the-scenes is important. Whenever an expression is evaluated in QGIS the context of the expression is considered. The context is built up from a set of scopes, which are all stacked on top of each other in order from least-specific to most-specific. It’s easier to explain with an example. Let’s take an expression used to set the source of a picture in a map composer. When this expression is evaluated, the context will consist of:

1. The global scope, consisting of variables set in the QGIS options dialog, and other installation-wide properties
2. The project scope, which includes variables set in the Project Properties dialog and the auto-generated project variables like @project_path, @project_title (you can read more about this in part 2)
3. A composer scope, with any variables set for the current composer, plus variables for @layout_pagewidth, @layout_pageheight, @layout_numpages, etc.
4. A composer item scope for the picture, with item-specific variables including @item_id

The more specific scopes will override any existing clashing variables from less specific scopes. So a global @my_var variable will be overridden by an @my_var variable set for the composer:

Another example. Let’s consider now an expression set for a data-defined label size. When this expression is evaluated the context will depend on where the map is being rendered. If it’s in the main map canvas then the context will be:

1. The global scope
2. The project scope
3. A map settings scope, with variables relating to how the map is being rendered. Eg @map_rotation, @map_scale, etc
4. A layer scope. More on this later, but the layer scope includes layer-level variables plus preset variables for @layer_name and @layer_id

If instead the map is being rendered inside a map item in a map composer, the context will be:

1. The global scope
2. The project scope
3. The composer scope
4. An atlas scope, if atlas is enabled. This contains variables like @atlas_pagename, @atlas_feature, @atlas_totalfeatures.
5. A composer item scope for the map item
6. A map settings scope (with scale and rotation determined by the map item’s settings)
7. The layer scope

Using layer level variables

Ok, enough with the details. The reason I’ve explained all this is to help explain when layer level variables come into play. Basically, they’ll be available whenever an expression is evaluated inside of a particular layer. This includes data defined symbology and labeling, field calculator, and diagrams. You can’t use a layer-level variable inside a composer label, because there’s no layer scope used when evaluating this. Make sense? Great! To set a layer level variable, you use the Variables section in the Layer Properties dialog:

Setting a layer variable

Any layer level variables you set will be saved inside your current project, i.e. layer variables are per-layer and per-project. You can also see in the above screenshot that as well as the layer level variables QGIS also lists the existing variables from the Project and Global scopes. This helps show exactly what variables are accessible by the layer and whether they’ve been overridden by any scopes. You can also see that there’s two automatic variables, @layer_id and @layer_name, which contain the unique layer ID and user-set layer name too.

Potential use cases for layer-level variables

In the screenshot above I’ve set two variables, @class1_threshold and @class2_threshold. I’m going to use these to sync up some manual class breaks between rule based symbology and rule based labeling. Here’s how I’ve set up the rule-based symbols for the layer:

Rule based symbology using layer level variables

In a similar way, I’ve also created matching rule-based labeling (another new feature in QGIS 2.12):

Matching rule-based labels

Here’s what my map looks like now, with label and symbol colors matched:

*Map for illustrative purposes only… not for cartographic/visual design excellence!

If I’d hard-coded the manual class breaks, it would be a pain to keep the labeling and symbology in sync. I’d have to make sure that the breaks are updated everywhere I’ve used them in both the symbology and labeling settings. Aside from being boring, tedious work, this would also prevent immediate before/after comparisons. Using variables instead means that I can update the break value in a single place (the variables panel) and have all my labeling and symbols immediately reflect this change when I hit apply!

Another recent use case I had was teaming layer-level variables along with Time Manager. I wanted my points to falloff in both transparency and size with age, and this involved data defined symbol settings scattered all throughout my layer symbology. By storing the decay fall-off rate in a variable, I could again tweak this falloff by changing the value in a single place and immediately see the result. It also helps with readability of the data defined expressions. Instead of trying to decipher a random, hard-coded value, it’s instead immediately obvious that this value relates to a decay fall-off rate. Much nicer!

I’m sure there’s going to be hundreds of novel uses of layer-level variables which I never planned for when adding this feature. I’d love to hear about them though – leave a comment if you’d like to share your ideas!

One last thing – the new “layer_property” function

This isn’t strictly related to variables, but another new feature which was introduced in QGIS 2.12 was a new “layer_property” expression function. This function allows you to retrieve any one of a bunch of properties relating to a specific map layer, including the layer CRS, metadata, source path, etc.

This function can be used anywhere in QGIS. For instance, it allows you to insert dynamic metadata about layers into a print composer layout. In the screenshot below I’ve used expressions like layer_property(‘patron’,’crs’) and layer_property(‘patron’,’source’) to insert the CRS and source path of the “patron” layer into the label. If either the CRS or the file path ever changes, this label will be automatically updated to reflect the new values.

Inserting dynamic layer properties into a composer label

So there you go – layer level variables and the layer_property function – here in QGIS 2.12 and making your workflow in QGIS easier. In the final part of this series, we’ll explore the magical @value variable. Trust me, I’ve saved the best for last!

Following on from part 1 in which I introduced how variables can be used in map composers, I’d like to now explore how using variables can make it easier to manage your QGIS projects. As a quick refresher, variables are a new feature in QGIS 2.12 which allow you to create preset values for use anywhere you can use an expression in QGIS.

Let’s imagine a typical map project. You load up QGIS, throw a bunch of layers on your map, and then get stuck into styling and labelling them ‘just right’. Over time the project gets more and more complex, with a stack of layers all styled using different rendering and labelling rules. You keep tweaking settings until you’re almost happy with the result, but eventually realise that you made the wrong choice of font for the labelling and now need to go through all your layers and labelling rules and update each in turn to the new typeface. Ouch.

Variables to the rescue! As you may recall from part 1, you can reuse variables anywhere in QGIS where you can enter an expression. This includes using them for data defined overrides in symbology and labelling. So, lets imagine that way back at the beginning of our project we created a project level variable called @main_label_font:

Creating a variable for label font

Now, we can re-use that variable in a data defined override for the label font setting. In fact, QGIS makes this even easier for you by showing a “variables” sub-menu allowing easy access to all the currently defined variables accessible to the layer:

Binding the label font to the @main_label_font variable

When we hit Apply all our labels will be updated to use the font face defined by the @main_label_font variable, so in this case ‘Courier New’:

In a similar way we can bind all the other layer’s label fonts to the same variable, so @main_label_font will be reused by all the layers in the project. Then, when we later realise that Courier New was a horrible choice for labelling the map, it’s just a matter of opening up the Project Properties dialog and updating the value of the @main_label_font variable:

And now when we hit Apply the font for all our labelled layers will be updated all at once:

It’s not only a huge time saver, it also makes changes like this easier because you can try out different font faces by updating the variable and hitting apply and seeing the effect that the changes have all at once. Updating multiple layers manually tends to have the consequence that you forget what the map looked like before you started making the change, making direct comparisons harder.

Of course, you could have multiple variables for other fonts used by your project too, eg @secondary_label_font and @highlighted_feature_font. Plus, this approach isn’t limited to just setting the label font. You could utilise project level variables for consolidating font sizes, symbol line thickness, marker rotation, in fact, ANYTHING that has one of those handy little data defined override buttons next to it:

See all those nice little yellow buttons? All those controls can be bound to variables…

One last thing before I wrap up part 2 of this series. The same underlying changes which introduced variables to QGIS also allows us to begin introducing a whole stack of new, useful functions to the expression engine. One of these which also helps with project management is the new project_color function. Just like how we can use project level variables throughout a project, project_color lets you reuse a color throughout your project. First, you need to create a named colour in the Default Styles group under the Project Properties dialog:

Define a colour in the project’s colour scheme…

Then, you can set a data defined override for a symbol or label colour to the expression “project_color(‘red alert!’)“:

When you go back and change the corresponding colour in the Project Properties dialog, every symbol bound to this colour will also be updated!

So, there you have it. With a little bit of forward planning and by taking advantage of the power of expression variables in QGIS 2.12 you can help make your mapping projects much easier to manage and update!

That’s all for now, but we’re still only just getting started with variables. Part 3, coming soon!.. (Update: Part 3 is available now)