Category: Uncategorized

When copy with force doesn’t work in Linux bash…

Hidden feature alert! I guess I have been a good boy and use sudo more than I do log in as root, or I would have discovered this much earlier on. While updating some website software with a new version, I wanted to recursively copy files from a new tar ball into the old web site vhost location – while also forcing it to overwrite existing files.

Naturally, I used the cp command, but hit some interesting glitch…

cp -rf new/* old/

Now that should suppress promptings for overwriting existing files right? Well, not always. Try logging in as root on Fedora (and others?) and you will still find that you are prompted to confirm each overwrite. (This wasn’t the case on RHEL by the way.) Is it a bug in bash, or are you really just not typing it right? Neither! It’s not a bug, it’s a feature! Apparently us Linux admins need a little safeguarding from time to time. I’m sure I opt’ed in somewhere. So the default behaviour of these commands was overridden by an alias setting in .bashrc (Mr. Power Admin, meet Nanny State… 😉 ). There the cp and mv commands are set to force the –interactive feature instead of blindly accepting a -rfwithout complaint. I won’t complain, but it was a waste of a few minutes. From .bashrc:

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

Comment out the line you detest the most and open a new terminal, and your back to nuking your system with ease. Whew… I’m pretty sure this behaviour is part of some certification quiz question somewhere, to weed us novices from the cream of the crop 🙂 Keep your crop – I just want the power.

GFOSS Education Resources Survey

via locatepress.com

Complete our survey!
Locate Press aims to support education and instruction using open source geospatial software. As part of helping students find prospective courses, schools and contacts, we’re running this simple survey. Register your school or company and describe your course offerings and we’ll pull it together when we get a few responses. Survey here – share the link with your academic friends!

Ingres db access in QGIS via GDAL/OGR VRT

My title was already long enough, but I should have added the subtitle “…on Windows”. Up until this week you had to be a Windows compiling guru and a GDAL/OGR master to get the driver for Ingres to build. With a wave of his hand, Frank fixed that this week and now I (a mere Windows padowan) was able to get it running. Here are a few of the things that transpired and some examples of the end result. Things are easier on Linux (what’s new?), but I needed to crack the Windows nut for a few reasons since it wasn’t quite as ready.

First off, if you are interested in hearing more about the Ingres geospatial capabilities, let me know or join our low volume, informal, list where I (am just starting to) post more info as it comes available or review our wiki (slowly getting updated). We are working, heads down, on getting the next Ingres db release out which will include geospatial capabilities in the core of the database (it’s not an add-on). This release will be backed by Actian’s subscription support services you can get 24/7 professional support for this open source product, with support spread around the world. Now onto the geek details…

Ingres Libraries & Database

Naturally you’ll need Ingres db libraries to get started. We don’t have a simple zip of files so I first pulled that together. You can likely download any of the Windows installer packs from the Ingres download pages: actian.com/downloads/ingres and it will include the library files you need. On my machine all the files I need were in two folders (lib and files) found in C:\Program Files (x86)\Ingres\IngresII\ingres\ with subfolders: lib – for libraries, and files – for headers/include.

If you also need the Ingres database too, grab a 10.1 (build 120+) installer to ensure it’s capable. If that doesn’t work then grab the windows installer from here that also included the geospatial capabilities but may be a little older. I haven’t built Ingres from source on Windows yet, so won’t attempt to walk you through that, but I’ll get to that eventually.

Build GDAL/OGR

Grab the latest source for GDAL/OGR, the improvements were just committed this week. For those who may be new to GDAL/OGR – we are specifically interested in the OGR side of the equation as it manages the vector (not raster) data access we need.
Edit the nmake.opt file, specifically uncomment lines 356-360 and edit 356 to reflect the location of your Ingres libraries I mentioned above.

356 INGRES_DIR = C:\Users\tyler\src\ingres\ # files and lib folders are in here
357 INGRES_INC_DIR = $(INGRES_DIR)\files
358 INGRES_LIB = $(INGRES_DIR)\lib\iilibapi.lib \
359 $(INGRES_DIR)\lib\iilibutil.lib \
360 $(INGRES_DIR)\lib\libingres.lib

Then run, from the GDAL/OGR source folder:
nmake /f makefile.vc
It was that easy and I’m no pro on building on Visual Studio 2010 – having just downloaded the free express version last week.

(Note: I also built in GEOS support, it was just as easy to build from source in Windows, and all you need to do is edit nmake.opt as above but for GEOS after you have it built. You need this for our final step, if you want to hack up QGIS to use this new format from OGR.)

Check that it built in the support for Ingres by running, from the command line, some of the programs in the GDAL/OGR source\apps folder and adding the –formats option. For example:

apps\ogr2ogr --formats
...
-> "MSSQLSpatial" (read/write)
-> "Ingres" (read/write)
-> "PCIDSK" (read/write)
...

Convert data to Ingres Geospatial

One of the reasons we really need this OGR support was for data loading and extracting. We are working on FME support but for those without FME, OGR and Geotools will remain critical!

Grab your favourite shapefiles and give ogr2ogr a try. The general syntax, to connect to a local Ingres database is like this:

ogr2ogr -f ingres @driver=ingres,dbname=[database name] [input.shp] -nln [new table name] #nln is optional
e.g...
ogr2ogr -f ingres @driver=ingres,dbname=canada roadlines.shp -nln roads

Depending on the projection/SRS of the data you may get notices complaining about not being able to INSERT into the spatial_ref_sys table. More on this at another time, but if you want to ignore it, feel free to add -skipfailures and/or -overlap – if you want to run it again easily.

The Ingres database access application is called ‘sql’ – when you run this from a command line you get an interactive session. Type ‘help \g’ (\g being like the ; on other databases) and it will list the tables that are in your database. You can also run:
ogrinfo @driver=ingres,dbname=[database name] – to see if the table was created.

Viewing QGIS

There is no QGIS specific database access provider setup for Ingres.. yet. (Want to help with it? Let me know!) So the approach I thought I try, was to use the OGR Virtual Format. For this you simply create an XML file that includes the connection information you used above. This virtual format is way more powerful than that but this will get you started:

<OGRVRTDataSource>
   <OGRVRTLayer name="road">
    <SrcDataSource>@driver=ingres,dbname=canada</SrcDataSource>
    <SrcLayer>road</SrcLayer>
      <GeometryType>wkbLinestring</GeometryType>
      <LayerSRS>EPSG:4326</LayerSRS>
      <GeometryField>shape</GeometryField>
    </OGRVRTLayer>
</OGRVRTDataSource>

Before you can use this in QGIS, consider that the GDAL/OGR that is distributed with it does not include Ingres support that we built earlier. Let the hacking begin, unless you want to build QGIS from scratch on Windows too 🙂 I was using OSGeo4W – a great distribution for Windowswith an easy to use installer for getting QGIS, GRASS and more up and running quickly. I copied these files from GEOS and GDAL into the OSGeo4W bin folder, after backing them up first of course :). Yes, this is a total hack but it saved me from rebuilding QGIS – ideally this won’t be necessary in the future if we can get Ingres support built into the base distributions of GDAL/OGR.

cd C:\OSGeo4W\bin
rename gdal18.dll gdal18.dll.orig
rename geos_c.dll geos_c.dll.orig
copy C:\Users\tyler\src\gdal-svn\gdal19.dll gdal18.dll
copy C:\Users\tyler\src\geos-3.3.2\src\geos_c.dll

Now, start up QGIS, select Layer -> Add Vector Layer. Choose Browse for a dataset. Change the drop down type filter box (bottom right) to VRT - Virtual Datasource [OGR], find your VRT file and select Open. Give it some time, I found it slow, at least on this low end laptop I was using. I didn’t get it to display attribute tables, likely due to my ignorance of the VRT format options and querying is slow.

Summary

So OGR is working well with Ingres geospatial types and I’m really glad we’ve got that ready to go on both Linux and Windows (anyone want to try Mac?). We have a queue of fixes, improvements and new features to add to the Ingres database over the upcoming months. We are also focusing on performance (any R-Tree gurus around interested in digging into the Ingres indexing code with us? 🙂 ). This initial development and testing has us off to a good start and I hope you’ll be interested in following along as we continue toward our next release. If you are interested in learning more or share thoughts I invite you to drop me a note at tyler.mitchell at actian !dot! com. Or follow me on Twitter. You can also read about my open source geo book publishing endeavours at http://locatepress.com.

Book now selling! The Geospatial Desktop

Gary’s updated book is now (back) in print and listed on Amazon for sale! Covering a vast array of open source software for GIS analysis, mapping and application building. With special sections focused on Quantum GIS and GRASS GIS.

For more info see our book page or the comprehensive book website. Or just buy it on Amazon.

Onto Actian, Ingres db and a New Book

A few people were wondering what I’ve been up to since I stopped working for OSGeo back in November. There are two big things going on that I’m excited about.

First, I just hit the one month mark in my new position with Actian (formerly known as Ingres). Some of you will have remembered the work Andrew Ross kicked off a few years ago, adding new geospatial smarts to the existing Ingres db. I’m taking over from where he left off, but what is already there is substantial. We have OGC SFSQL support in there as well as several data access libraries able to interact with the database (OGR, Geotools so far). We’re working on other features and 3rd party toolsets so our clients can access their spatial data using a wide range of both proprietary and open source solutions. Already an Ingres db user? I’d love to hear from you about what geo needs you may have. The upside to the Ingres db is the availability of long term and 24/7 support, as well as being open source and built with familiar geo toolkits like Proj.4 and GEOS. More to come as I find my feet …

Second, and on an entirely different note, you may have noticed that my Locate Press venture is going to have its first book for sale shortly. Gary Sherman’s updated The Geospatial Desktop (formerly known as Desktop GIS) is coming out of the chute following a review of our first proofs. Sign up on the form at http://locatepress.com to be the first to hear when it is available. Visit Gary’s book site for more information as well: http://geospatialdesktop.com. We have a few other books in the works as well, including a workshop training guide on Web GIS and more.

All-in-all, a great way to start a new year! I’m looking forward to finding new and interesting ways to continue working with the range of open sourcerers friends that I’ve enjoyed working with over the past decade.

See you soon!

Open Source Desktop GIS book almost ready…

UPDATE: The Book Is Ready – Get It Here! –

This week we’re putting the final polish on Gary Sherman’s new book. For more info see Locate Press site:

Gary Sherman has done it again… Originally published as Desktop GIS, by Pragmatic Press, it quickly went out of print. Locate Press is proud to bring this work back into print by releasing this fully updated second edition. More information coming soon. See Gary’s website dedicated to his book – DesktopGISBook.com.
Leave a note here if you want to be informed when it is available.

High resolution screenshots (SOLVED!)

Ever want to take a screenshot at higher-than-your-monitor supports resolutions? I do! I laser print quality, e.g. 300-600 DPI (dots per inch) not a measly 70-90DPI like my screen supports. So if aiming for a 10″ wide print out, I’d need a 3000 pixel wide image. I don’t know about you but my monitor hasn’t gotten up to 2000 yet 🙂

I found (at least) two particular approaches that I am using now to help with this task…

I’m hoping you find this post if you are searching for this kind of solution because I found a hundred web searches that were *not* helpful or really even on this topic. What do you search for anyway? “screenshot virtual desktop” gives you a good start but you’ll get a weird mix of unrelated stuff for sure. Heck one proprietary vendor told me their product wouldn’t do it, but they also thought it really wasn’t possible. Oh my. Hint, you want virtual desktop panning or scaling.

Here’s what I found works on my Linux box, but both likely will work on other platforms as well if you have suitable hardware (and if Microsoft and Apple give you the options)…

#1 – NVIDIA X Server Settings

My NVIDIA graphics card is good enough to take on the task, yours should be too. Testing on my Red Hat Enterprise Linux (RHEL6) Desktop environment, I launched the command nvidia-settings (as root) and selected the X Server Display Settings. There is an advanced button that allows you to specify a resolution for panning. I set mine to 3000×1050 just for fun, logged out and back in to reset X Server settings and was instantly in action.

The result is that you have a huge desktop and to move around in it you slide your pointer off the edge of the screen. Similarly when you want to take a screenshot, just start up your app, maximize it (you’ll see a portion of it!) and launch your screenshot app – in my case KSnapshot is my friend. I tell it to capture the whole window and… it just works. I did find a faint ghosting effect where the KSnapshot dialog was still showing a bit on top of the window I captured – I think it’s because it takes a bit more time and energy to handle the expanded graphics – give yourself a few seconds to work with instead.

TODO: Learn X Server conf stuff enough to also enable scaling so I have a high resolution desktop that still fits on my screen – though ultimately only show 72DPI but that the system can capture at full res when needed. I looked at the Xephyr project too and it seems similar but I think it’s probably overkill for me at this stage unless I’m doing screenshot remotely.

#2 – VNC / Remote Desktop

Panning around the screen in solution #1 might not be an option for those still using mommy’s netbook for their work.. so here’s your alternative. You may even find this preferable to the above because scaling works well, at least in my VNC client app. (Hint, VNC is worth learning about if you don’t already know about it, it’s basically what Windows, Mac and even RHEL term Remote Desktop computing.) In Linux it’s simple, start a VNC Server session and a copy of your desktop starts up virtually. You don’t see anything until you connect to it using a VNC (or Remote Desktop) client program – you can connect locally or remotely through your network.

In my case I installed TightVNC (I think), but it likely doesn’t matter, just find a VNC Server and Client package to try out. I ran the command vncserver -geometry 3000×1050 – and I was off to the races. Then, even using the same computer and desktop, I ran the Remote Desktop Viewer (filed under Internet apps in my menu) and enter in my address and the display “port” number that the vnserver app mentioned after starting up – e.g. localhost:1. I also have an option in my connection dialog to turn on scaling – yes, try it, you’ll like it!

The result is really cool, nothing stupendous but will solve all your woes with high res screenshots. Need to see what a website looks like at a higher resolution than your papa’s Dell laptop supports? Give this a try. No promise on Windows and Mac as they often mess with what you can do with virtual desktops, but if a VNC Server can be run, then it just might also work.

I’m sure some off the shelf solution exists – but I haven’t found it. What I would ideally have is a tool on my desktop that I can click on – switching to ultra hi-def mode and saving a screenshot and then dropping me back to normal resolution. Should be doable – holler when you need a beta tester!

One app I will be doing some screenshots on is Quantum GIS – it will certainly be interesting to see how quickly my WMS layers fail to load at higher than normal screen resolution… so I’m sure I’ll be blogging about tilecaching sometime in the future too 😉

Tyler Mitchell
12 Nov 2011

p.s. I wrote this agggges ago – if it works for you, please comment so I know it’s not something I should address/update.

State of the Bounty?

I’m regularly asked about how users or companies can request improvements to OSGeo software or to make a specific bug go away. Short of pointing people to the ~200 OSGeo Service Providers (http://osgeo.org/search_profile) and helping them get onto the right lists to ask for help… what else can I do? Naturally some companies are more active than others in terms of providing code-level support like this, yet I know there are even more out there.

Trying to bring the two groups together through a bounty program (fee offered for a specific feature/bug solved) is always controversial. Yes, I have heard all the for/against arguments already 🙂 but I’d still like to see who out there is already doing it in this space.

So my question tonight is twofold:

1) What GFOSS or OSGeo projects have looked a bounty system and are either implementing it or hope to implement it. If you are offering up suggestions for “pay for this great new feature”, that might apply as well. I’d love to know how you are implementing it.

2) If you are a service provider or developer on GFOSS/OSGeo platforms, would you be interested in being connected to potentially paying customers (hmm, hope you answer yes so far) that have particular requests to the base.

I’m more curious than anything and wonder what we’ll find after scratching the surface! I know some local chapters have even discussed the ideas for supporting their favourite packages, etc. Please let me know if you have anything to share – I’ll summarise and re-share the results in a few days.

An OSGeo Marketplace?

I’ve mentioned before about the OSGeo Service Providers Directory. It’s nothing fancy but quite useful, especially if you’re trying to get a feel for the ecosystem of providers out there. Currently there are 196 providers registered (are you?) for 25 software projects. I haven’t done the country or language calculations, but I took a look at the cumulative number of service providers saying they support a given GFOSS project. I’ll tidy it up eventually, but provide a rough graph for your viewing pleasure below (Average 43, Max 150, Min 2).

That’s pretty astounding for me to see. Consider what it would have been 5 years ago before anyone even started collecting this data.

Now what I’d like to know is how many of those providers are VARs vs developers vs committers. Ohloh.net may help identify, for example, core committers. But it’s more of a challenge to make the connection between developers that can provide code-level support (i.e. bug fixes, new features, as hinted at in my previous post) and those user/customers who could use the help.

I wish you could group ohloh users into companies, or that we could have a flag in the directory for the various levels of service one provides. A good project to start digging into for the next rainy day.

Updated graph included fixed JUMP/OpenJump stats mistake.

Installing GeoServer on OSX (again, after 4 years)

A few years back I wrote up my experience installing GeoServer on my Macbook. I thought I’d take another stab at it and see how it goes on a newer Mac, this time running Snow Leopard (10.6.8 Macbook pro).

I’m sure some groups are packaging GeoServer into some sort of simple to install packages that I should know about, but not being daunted by the “hard way” I’m following my new user approach to getting through this: pretend I know nothing. First stop – http://geoserver.org. Then I clicked my way through Download, Stable version and selected Mac OSX Installer – imagine that! My confidence is buoyed as I feel I’m on the right track.

This is brilliant, guys, you couldn’t have made it easier. I only wondered if I should have grabbed the OS Independent download instead, but I won’t test fate by deviating from the obvious at this late hour.

Now that the geoserver-2.1.1.dmg file (56MB) has been downloaded from Sourceforge, let’s see what we got. Can it get easier than this?

GeoServer OSX Installer Package
I dragged the GeoServer icon onto the Applications icon but got an error about the Applications alias. Rather than fiddle, I opened a new Finder window and dragged it directly to the Applications folder – no problem.

Now, what do we get when we launch GeoServer from the Applications icon?

After select the Server -> Start menu it throws a few log lines up the console window and announces that GeoServer is started. Right away my web browser pops up with the GeoServer Welcome Page:

A successful, and very easy, install. You should now have a look at the GeoServer docs to get started with the real utility of the application. Good work GeoServer dev team, I was thrilled to not touch a war file or go hunting for a Tomcat install this time!

%d bloggers like this: