I went hunting for a good set of Canada-wide WMS layers and (re)discovered some great work from NRCAN on the Toporama site
Good response time but also nice cartographic quality, well, at least if you want something that’s similar to the printed topographic map you may have on your wall. Good work as usual from NRCAN and co. Anyone know what they are serving it up with?
Here is their WMS connection.
Gary Sherman has done it again… Originally published as Desktop GIS, by Pragmatic Press, it quickly went out of print. Locate Press is proud to bring this work back into print by releasing this fully updated second edition. More information coming soon. See Gary’s website dedicated to his book – DesktopGISBook.com.
Leave a note here if you want to be informed when it is available.
In today’s super duper processor world if you are still only running the standard trio:
./configure make make install
.. then you’re really missing out. Well, that is unless you like having more time to drink your coffee while waiting for compilation jobs to finish.
The make command comes with an option to spawn off multiple processes while building. This is can dramatically reduce compilation time. For example, building Quantum GIS from source (for use with our upcoming Ingres Geospatial release) – took about 12 minutes to build the traditional way. But, if you take the number of cores your system has, add 1 and pass that along to the make command then you’re really cooking…:
make -j 5
.. it took only 3.5 minutes.
Try it, you’ll like it!
Years ago I started writing some scripts with the end goal of doing a wholesale migration of my data into open source spatial databases. I since left that job and didn’t have much need for the scripts, but recently picked them back up again. I never made it to the migration part, but decided instead to focus on cataloging my GIS data so I could then create other apps to use the catalog for looking things up, creating overview maps, and ultimately to fuel migration scripts. The historical name for this project is somewhat irrelevant, for now, the Catalog portion of a broader plan I had called the Network Mapping Engine (NME).
My current efforts are available here: https://github.com/spatialguru/NME/tree/master/nme/cat
Want to try it? Ensure you have the prerequisite xmlgen and elementtree_pretty libs (included on the site) as well as a recent GDAL/OGR install with Python bindings. It’s reported to work well with the OSGeo4W distribution, though I have only tested it recently on Mac OSX 10.6.
The main feature is the gdalogr_catalogue.py script but the above dependencies will also have to be available in your PYTHONPATH.
The simple command line usage only requires one parameter – pointing to the folder that you want to catalog. It will also recursively scan the sub-folders as well.
python gdalogr_catalogue.py -d /tmp/data
The result is basic XML (more about the details of output in a moment):
<?xml version="1.0" ?>
<DataCatalogue>
<CatalogueProcess>
<SearchPath>
/tmp/data
</SearchPath>
<LaunchPath>
/private/tmp/data
</LaunchPath>
<UserHome>
/Users/tyler
</UserHome>
<IgnoredStrings>
['.svn', '.shx', '.dbf']
</IgnoredStrings>
<DirCount>
2
</DirCount>
<FileCount>
16
</FileCount>
<Timestamp>
Mon Mar 19 14:42:35 2012
</Timestamp>
.... huge snip ....
Pipe the output into a new file or use the save-to-file output option:
python gdalogr_catalogue.py -d /tmp/data -f output.xml
Most metadata standards include pretty high level information about datasets, but the focus of this project is to grab as low level data as possible and make it easily consumable by other applications. For example, consider how valuable hierarchical information about all the datasets on your system or in your enterprise would be. A map viewing application could access the catalog instead of having to repetitively scan folders and open files. It could fuel part of a geospatial search engine for that matter!
The original work created CSV text files, then it produced SQL statements, but now it has final settled on producing XML. There was no existing standard for this type of metadata, so I had to create one. Because it is built on top of GDAL/OGR, much of it parallels the GDAL/OGR API, but additional information is also captured about the process of cataloguing and file statistics. It’s still changing and growing but no huge changes are expected.
There are three primary sections in the root DataCatalogue XML element, and a couple sub-elements as well:
I won’t review all the levels of elements but instead suggest you have a look at the sample_output.xml in your browser, along with the catalogue.xsl and css/catalogue.css, it renders to show some basic example data:
One of my goals is to have this XML output plug into other applications as the basis for an enterprise data catalog environment. My first goal is to add capabilities for generating a MapServer map file from the metadata. This would also allow the script to produce overview maps of the dataset extents. Another idea is to create a QGIS plugin to visualise and browse through the datasets, while also allowing option loading of the layer into QGIS.Have a better idea? Let me know and we can discuss it. Maybe I need to come up with a contest, hmm…
Updating the XML is another challenge on the horizon. I built in the creation of md5 hash based checksum so it can check for file changes without even having to open the file with GDAL/OGR, but there are no routines yet to allow it to update the XML in-place.
Obviously, there is more to come, as more people try it and let me know how they get on. Drop me a comment or email and I’d love to help you get it running if it fails in your environment!
Hidden feature alert! I guess I have been a good boy and use sudo more than I do log in as root, or I would have discovered this much earlier on. While updating some website software with a new version, I wanted to recursively copy files from a new tar ball into the old web site vhost location – while also forcing it to overwrite existing files.
Naturally, I used the cp command, but hit some interesting glitch…
cp -rf new/* old/
Now that should suppress promptings for overwriting existing files right? Well, not always. Try logging in as root on Fedora (and others?) and you will still find that you are prompted to confirm each overwrite. (This wasn’t the case on RHEL by the way.) Is it a bug in bash, or are you really just not typing it right? Neither! It’s not a bug, it’s a feature! Apparently us Linux admins need a little safeguarding from time to time. I’m sure I opt’ed in somewhere. So the default behaviour of these commands was overridden by an alias setting in .bashrc (Mr. Power Admin, meet Nanny State… 😉 ). There the cp and mv commands are set to force the –interactive feature instead of blindly accepting a -rfwithout complaint. I won’t complain, but it was a waste of a few minutes. From .bashrc:
alias rm='rm -i' alias cp='cp -i' alias mv='mv -i'
Comment out the line you detest the most and open a new terminal, and your back to nuking your system with ease. Whew… I’m pretty sure this behaviour is part of some certification quiz question somewhere, to weed us novices from the cream of the crop 🙂 Keep your crop – I just want the power.
via locatepress.com
My title was already long enough, but I should have added the subtitle “…on Windows”. Up until this week you had to be a Windows compiling guru and a GDAL/OGR master to get the driver for Ingres to build. With a wave of his hand, Frank fixed that this week and now I (a mere Windows padowan) was able to get it running. Here are a few of the things that transpired and some examples of the end result. Things are easier on Linux (what’s new?), but I needed to crack the Windows nut for a few reasons since it wasn’t quite as ready.
First off, if you are interested in hearing more about the Ingres geospatial capabilities, let me know or join our low volume, informal, list where I (am just starting to) post more info as it comes available or review our wiki (slowly getting updated). We are working, heads down, on getting the next Ingres db release out which will include geospatial capabilities in the core of the database (it’s not an add-on). This release will be backed by Actian’s subscription support services you can get 24/7 professional support for this open source product, with support spread around the world. Now onto the geek details…
Naturally you’ll need Ingres db libraries to get started. We don’t have a simple zip of files so I first pulled that together. You can likely download any of the Windows installer packs from the Ingres download pages: actian.com/downloads/ingres and it will include the library files you need. On my machine all the files I need were in two folders (lib and files) found in C:\Program Files (x86)\Ingres\IngresII\ingres\ with subfolders: lib – for libraries, and files – for headers/include.
If you also need the Ingres database too, grab a 10.1 (build 120+) installer to ensure it’s capable. If that doesn’t work then grab the windows installer from here that also included the geospatial capabilities but may be a little older. I haven’t built Ingres from source on Windows yet, so won’t attempt to walk you through that, but I’ll get to that eventually.
Grab the latest source for GDAL/OGR, the improvements were just committed this week. For those who may be new to GDAL/OGR – we are specifically interested in the OGR side of the equation as it manages the vector (not raster) data access we need.
Edit the nmake.opt file, specifically uncomment lines 356-360 and edit 356 to reflect the location of your Ingres libraries I mentioned above.
356 INGRES_DIR = C:\Users\tyler\src\ingres\ # files and lib folders are in here
357 INGRES_INC_DIR = $(INGRES_DIR)\files
358 INGRES_LIB = $(INGRES_DIR)\lib\iilibapi.lib \
359 $(INGRES_DIR)\lib\iilibutil.lib \
360 $(INGRES_DIR)\lib\libingres.lib
Then run, from the GDAL/OGR source folder:
nmake /f makefile.vc
It was that easy and I’m no pro on building on Visual Studio 2010 – having just downloaded the free express version last week.
(Note: I also built in GEOS support, it was just as easy to build from source in Windows, and all you need to do is edit nmake.opt as above but for GEOS after you have it built. You need this for our final step, if you want to hack up QGIS to use this new format from OGR.)
Check that it built in the support for Ingres by running, from the command line, some of the programs in the GDAL/OGR source\apps folder and adding the –formats option. For example:
apps\ogr2ogr --formats
...
-> "MSSQLSpatial" (read/write)
-> "Ingres" (read/write)
-> "PCIDSK" (read/write)
...
One of the reasons we really need this OGR support was for data loading and extracting. We are working on FME support but for those without FME, OGR and Geotools will remain critical!
Grab your favourite shapefiles and give ogr2ogr a try. The general syntax, to connect to a local Ingres database is like this:
ogr2ogr -f ingres @driver=ingres,dbname=[database name] [input.shp] -nln [new table name] #nln is optional
e.g...
ogr2ogr -f ingres @driver=ingres,dbname=canada roadlines.shp -nln roads
Depending on the projection/SRS of the data you may get notices complaining about not being able to INSERT into the spatial_ref_sys table. More on this at another time, but if you want to ignore it, feel free to add -skipfailures and/or -overlap – if you want to run it again easily.
The Ingres database access application is called ‘sql’ – when you run this from a command line you get an interactive session. Type ‘help \g’ (\g being like the ; on other databases) and it will list the tables that are in your database. You can also run:
ogrinfo @driver=ingres,dbname=[database name] – to see if the table was created.
There is no QGIS specific database access provider setup for Ingres.. yet. (Want to help with it? Let me know!) So the approach I thought I try, was to use the OGR Virtual Format. For this you simply create an XML file that includes the connection information you used above. This virtual format is way more powerful than that but this will get you started:
<OGRVRTDataSource>
<OGRVRTLayer name="road">
<SrcDataSource>@driver=ingres,dbname=canada</SrcDataSource>
<SrcLayer>road</SrcLayer>
<GeometryType>wkbLinestring</GeometryType>
<LayerSRS>EPSG:4326</LayerSRS>
<GeometryField>shape</GeometryField>
</OGRVRTLayer>
</OGRVRTDataSource>
Before you can use this in QGIS, consider that the GDAL/OGR that is distributed with it does not include Ingres support that we built earlier. Let the hacking begin, unless you want to build QGIS from scratch on Windows too 🙂 I was using OSGeo4W – a great distribution for Windowswith an easy to use installer for getting QGIS, GRASS and more up and running quickly. I copied these files from GEOS and GDAL into the OSGeo4W bin folder, after backing them up first of course :). Yes, this is a total hack but it saved me from rebuilding QGIS – ideally this won’t be necessary in the future if we can get Ingres support built into the base distributions of GDAL/OGR.
cd C:\OSGeo4W\bin rename gdal18.dll gdal18.dll.orig rename geos_c.dll geos_c.dll.orig copy C:\Users\tyler\src\gdal-svn\gdal19.dll gdal18.dll copy C:\Users\tyler\src\geos-3.3.2\src\geos_c.dll
Now, start up QGIS, select Layer -> Add Vector Layer. Choose Browse for a dataset. Change the drop down type filter box (bottom right) to VRT - Virtual Datasource [OGR], find your VRT file and select Open. Give it some time, I found it slow, at least on this low end laptop I was using. I didn’t get it to display attribute tables, likely due to my ignorance of the VRT format options and querying is slow.
So OGR is working well with Ingres geospatial types and I’m really glad we’ve got that ready to go on both Linux and Windows (anyone want to try Mac?). We have a queue of fixes, improvements and new features to add to the Ingres database over the upcoming months. We are also focusing on performance (any R-Tree gurus around interested in digging into the Ingres indexing code with us? 🙂 ). This initial development and testing has us off to a good start and I hope you’ll be interested in following along as we continue toward our next release. If you are interested in learning more or share thoughts I invite you to drop me a note at tyler.mitchell at actian !dot! com. Or follow me on Twitter. You can also read about my open source geo book publishing endeavours at http://locatepress.com.
Gary’s updated book is now (back) in print and listed on Amazon for sale! Covering a vast array of open source software for GIS analysis, mapping and application building. With special sections focused on Quantum GIS and GRASS GIS.
For more info see our book page or the comprehensive book website. Or just buy it on Amazon.
A few people were wondering what I’ve been up to since I stopped working for OSGeo back in November. There are two big things going on that I’m excited about.
First, I just hit the one month mark in my new position with Actian (formerly known as Ingres). Some of you will have remembered the work Andrew Ross kicked off a few years ago, adding new geospatial smarts to the existing Ingres db. I’m taking over from where he left off, but what is already there is substantial. We have OGC SFSQL support in there as well as several data access libraries able to interact with the database (OGR, Geotools so far). We’re working on other features and 3rd party toolsets so our clients can access their spatial data using a wide range of both proprietary and open source solutions. Already an Ingres db user? I’d love to hear from you about what geo needs you may have. The upside to the Ingres db is the availability of long term and 24/7 support, as well as being open source and built with familiar geo toolkits like Proj.4 and GEOS. More to come as I find my feet …
Second, and on an entirely different note, you may have noticed that my Locate Press venture is going to have its first book for sale shortly. Gary Sherman’s updated The Geospatial Desktop (formerly known as Desktop GIS) is coming out of the chute following a review of our first proofs. Sign up on the form at http://locatepress.com to be the first to hear when it is available. Visit Gary’s book site for more information as well: http://geospatialdesktop.com. We have a few other books in the works as well, including a workshop training guide on Web GIS and more.
All-in-all, a great way to start a new year! I’m looking forward to finding new and interesting ways to continue working with the range of open sourcerers friends that I’ve enjoyed working with over the past decade.
See you soon!