Category: Programming

Day 1 – Building a 3D Game using Geo Data (Hobby project)

We are a team of three guys (so far) with a design idea for a simple yet challenging racing/strategy game based on real-world geospatial data coupled with a 3D gaming engine. Tonight we started building it together in Unity after downloading a bunch of public geospatial datasets. Here’s a summary of what we’ve done and some basic screenshots. The goal is to make progress every couple days to show we is possible with a little work and a combination of creative, technical and group management skillsets.

First Images

Day 1 - Unity 3D real time rendering of geodata

Starting with Geography

The landscape of our 3D game is built using real world geographic information – aka geospatial data. We wanted this so that we could build games around real locales and market the game to local users. One side benefit is the players will learn a little about Canadian geography!

We’ve got a great collection of geo data for all of Canada available through the government Geogratis website. After we’ve decided what map tile numbers we needed, we grab the CDED data which is a TIFF elevation image file with geographic coordinates included.

We also found a BC provincial shaded relief map layer that was pretty nice as a starter. As both these data files are geolocated we can load the data into GIS (geographic information system) software from QGIS.org and combine it with any other data we have for the region. In our case we have a “populated places” point file from naturalearthdata.org that we show as stars on top of the shaded relief map.

Then we export the elevation data and the relief data (including the stars) into two files for Unity to ingest. The result is one PNG that is going to be our texture and the other will be the heightmap for the terrain.

day1_geodata

Loading into Unity

We created a terrain object with some pretty huge dimensions and height. There are mountains in the region we are working with, so it’s pretty cool to see. We apply the heightmap using a commonly used Unity script (I’ll have to put the link here when I remember where I got it!).

Then we add the shade relief map as a texture to the terrain et voila! We threw in a plane of basic water and raised it to the level where it filled the main river valleys. As our game is going to first start as a river racing game, we want to start with water from the very beginning.

We added a car, parented it to the camera and were racing down our waterways within 2 hours of starting. We spent a lot of time adjusting sizes of terrains and texture so we could try to match real-world scale. There are some further ideas we are going to try to optimise this as well as nail down a workflow for easily ingesting new geodata for other regions (as we had to manually export and adjust things in the GIS).

Reposted from our Indiedb blog, hence writing for a slightly non-geo audience: http://www.indiedb.com/members/1tylermitchell/blogs/edit/1mitchellco-day-1-cross-country-river-run-race-game

Generate terrain with flowing water from DEM in Unity

Just a quick video follow up based on a reader asking how I did what I did in my previous post with Unity (http://unity3d.com) game engine.

Using Surface Wave asset and built-in Unity Terrain generator, plus a script for taking DEMs and creating Terrains easily.  I’m really new at this by the way, but have a brilliant teacher showing me this stuff in my spare time 🙂

Video no longer available on YouTube: Adding Surface Wave waterflow to a terrain

 

Parsing GPS with Arduino – retro-post

I dug up one of my oldest blog posts from about 7 years ago.  In the post, I show how I had connected my Garmin eTrex GPS receiver to an Arduino board and used it to control a camera in a desktop application.

After pumping the data into the Arduino, I parsed the raw GPS data coming from the eTrex and streamed it out to a Python app on the desktop, via a serial port.  I was using the output to position a virtual camera in the OSSIMPlanet platform by formatting XML in Python and sending it to a listener built into the OSSIMPlanet app.  (OSSIMPlanet is a sort of pre-Google Earth, Google Earth on steroids product.)

I haven’t used it for years, but thought some of the methods from this old post may still apply if you have an Arduino + GPS or Arduino -> Python streaming requirement.   Enjoy the flashback – I know it’s inspired me to pick up a new Arduino to continue where I left off.

 

GPS Controls OSSIMPlanet - With OSSIMPlanet’s nifty camera control and listener functionality, as demonstrated in my last post, you’ve got so many neat opportunities. A couple nights ago I got a basic GPS NMEA parser working. Here’s a pic of the ultra-professional connection method I use to hook it to my arduino board 🙂 Oops, just realised that the picture … Continue reading GPS Controls OSSIMPlanet

Two wires to hook an eTrex data cable up to the Arduino
Two wires to hook an eTrex data cable up to the Arduino

Read Post

Review of 3 Recent Internet of Thing (IoT) Announcements

Working in the big data and analytics space, I’m always interested in parts of the Internet of Things (IoT) that will produce more data, require more backend systems, and help users/customers get on with their day better.

The past week has shown a few interesting announcements relating to Internet of Things topics.  Here are just a few that jumped out to me, either because they inspired me or because I was left wondering what it would really mean.

TL;DR?  Summary: While IBM is “getting started” (oops, I meant “getting serious”) and Facebook has big plans to “take over”, Amazon comes out with a consumer focused solution.

Facebook – Easing developer integration

Plans are awesome, big plans are big awesome!

This is really an announcement for app developers to drool over as it allows us to connect new things to the web easily.  “Parse for IoT, a new line of SDKs for connected devices…”

Their platform looks like it will take care of so much of the backend work that developers can focus on building out the user side of the app instead, which ultimately can connect to new devices.

Target: developers, manufacturers

IBM – Getting Serious about something

The logic behind the wording eludes me.  Are they getting serious because they weren’t before and are playing catchup?  Either way is fine, but it’s still a little confusing.  After all, other companies are still figuring out how to sell into this new market, develop products for it and, seemingly most difficult of all, trying to figure out how make announcements that are more engaging than this one.

Either I’m skim reading too fast, have an ultra low attention span, or missed something behind the headlines – because after all that I read, I’m left with this understanding:

“Yeah we can get into this too, just watch us, we’re spending lotsa cash!  And doing something with weather related energy disruption.  Watch us!”

I think I owe it to them to re-read some of the copy I saw because I may have read it before I had my morning tea.

Target audience: business, gov?

Amazon – Buying Things at the Right Time

https://plus.google.com/+TylerMitchell1/posts/ZnrdaW5ETYX

Here I’m mentioning the Dash Buttons concept – both the sort of magic wand thingy, but also the button tags to hang from appliances.

All supply chain difficulties disappear in the wake of Amazon bulk goods delivery, or does it?  I can honestly say I’ve never ordered toilet paper or laundry detergent from Amazon, though this has certainly got my thinking.  Oddly enough, refills are a mere 400m down the street from me and that’s only marginally harder than answering the door for the courier.  But I digress…

This announcement hits all the right buttons for me (sorry couldn’t resist) even though I’m not an everyday Amazon user, nor even using Prime.  They are talking about things that every person understands and deals with – the magic behind the scenes is nicely hidden away, yet we know their entire supply chain infrastructure has been around a long time and this is just optimising the “when” factor – making it possible to not postpone placing an order, but doing on impulse/necessity.

Most of us will just use a mobile app instead but think a bit longer term.  With the largest aging population in the history of the multiverse (read: transgenerational design), the younger generations aren’t going to be able to keep up with their needs without some help.  Granted, many seniors can already operate their own apps today, but for everyone else, a button on the washing machine or soap dispenser could really fill in the gaps.  Couple that with some moderation-style smart apps for caretakers (oops, granny ordered detergent three times by mistake) and I think you’ve got some really interesting future elder care provisioning.

Target: consumers and manufacturers who are ready to sell.  Today!

Who wins for best announcement this week in your books? Tweet me!

Drinking from the (data) Firehose of Terror

Between classic business transactions and social interactions and machine-generated observations, the digital data tap has been turned on and it will never be turned off. The flow of data is everlasting. Which is why you see a lot of things in the loop around real time frameworks and streaming frameworks. – Mike Hoskins, CTO Actian

From Mike Hoskins to Mike Richards (yes we can do that kind of leap in logic, it’s the weekend)…

Oh, Joel Miller, you just found the marble in the oatmeal!   You’re a lucky, lucky, lucky little boy – because you know why?  You get to drink from… the firehose!  Okay, ready?  Open wide! – Stanley Spadowski, UHF

Firehose of Terror

I think you get the picture – a potentially frightening picture for those unprepared to handle the torrent of data that is coming down the pipe.  Unfortunately, for those who are unprepared, the disaster will not merely overwhelm them.  Quite the contrary – I believe they will be consumed by irrelevancy.

If you’re still with me, let me explain.

Play video: Firehose

I agree that the tap has been turned on, maybe not at the full blast of power or under maximum control, yet the data is coming and already well beyond a trickle.  The helter-skelter implementations of big data solutions out there has, perhaps, created more of a turbulent blasting firehose than a meandering stream of flowing data.  And this is only the beginning.

Success is Still Newsworthy

We are still at the stage where designing a (successful) enterprise system built around streaming data, for example, is big news.  Why is it news?  Because building is harder than merely planning, especially when new open source projects continue to push us beyond the bleeding edge.  New tools are helping us see how we can handle more data and find more value, but they are also making the pool of data so much larger that the tools themselves are often irrelevant by the time they are adopted.

For example, MapReduce was awesome, until it began to be so widely adopted that its limitations became apparent.  It’s like finding the marble in a sandbox filled with oatmeal – not easy, but when you find it, you’re a winner!  Oh, the prize is a fierce typhoon of even more data coming your way.  Congratulations! (Sorry you didn’t prepare for that.)

So where does this leave organisations that have no ability to handle more than a trickle of data?

It’s a win or lose scenario – either you can do something about it or you can’t.  As software developers or data managers we won’t be judged along some smooth gradation of skills and capabilities.

We’ll be judged against a checklist
Yes or no.
Pass or fail.
Win or lose.
Firehose or … an icky pail to hide in the closet.

Data’s Need for Speed

Why is it a pass/fail scenario?  Consider your car – is it successful when it mostly starts in the morning?  Never.  Anything beyond fully starting is a complete failure because that is what it is designed to do.

I argue that today’s data streams are being designed to handle data at maximum velocity.  Sure many services aren’t producing millions of records per second, but as we gear up with the latest toolsets, we make the tools themselves hunger and thirst to get more and more data into their greedy little hands.  Feed the beast – or ignore it at your peril.

Systems today are designed to run at full throttle – 100% – all out – maximum overdrive. None of us would like to have pay for premium broadband and find that we only use 10% of our bandwidth.  Likewise, our systems are waiting for us to crank up the volume to see what we’ll do next.

Our data economy inherently wants to run at maximum but much of the old plumbing needs upgrades to function at that rate.  You’d be irate if the fire department ran their hoses at 10% power when trying to douse your burning home.  However, if they told you later that max pressure would burst the old hoses and you’d have no water, then you might be a little more appreciative.

Patch up those hoses so they are tested and ready.  Buckle up the survival suit.  Start digging through the oatmeal and open wide.  Be forewarned – if you are not searching for the marble, you can never find it.  If cannot find it, you’ll be sent home empty handed.

—-

p.s. If you don’t win, I highly doubt you’ll even receive a lousy copy of the home edition.

 

Google wants “mobile-friendly” – fix your WordPress site

TheNextWeb reports: “Google will begin ranking mobile-friendly sites higher starting April 21“.  It’s always nice having advance warning, so use it wisely – here’s how to tweak WordPress to increase your mobile-friendliness.

Google Mobile-Friendly Check

I use a self hosted WordPress site and wanted to make sure it was ready for action.  I already thought it was, because I’ve accessed in on a mobile device very often and it worked okay.

I even went onto the Google Web Admin tools and the mobile usability check said things were fine, but…

Google admin tool says mobile check is okay

However, all was not golden when I ran the Google mobile-friendly checker.  (Obviously two different apps here, hopefully those will merge.)

Google mobile check failure

Try it here, now!  The complaints about were that some content is wider than screens and that links were too close together.  Fair enough.

WordPress Mobile-Friendly Activation

If you’re not already using WordPress’s Jetpack features, you’re really missing out.  I use it mostly for monitoring stats but there are several other features that make it very useful, including one called Mobile Theme.

  1. From the admin sidebar select Jetpack (install it first if not already enabled).  It will show you some suggested plugins to enable, plus show you a search bar to find others.
  2. Enter “mobile” and click on the Mobile Theme item.
  3. Activate it (lower-right corner button).
  4. And you’re done!

Going back to Google’s checker it shows a different preview now and also says things are fine.  Looking at the site after making these changes, it’s obviously better.

Google mobile check fixed success

However, I still have one plugin (Crayon markup) that helps display code samples that seems to force some posts to wider than the screen.  I assume the plugin creators will fix that up, but it’s not too bad at this point.  Unless Google complains, it doesn’t matter anyway!

Geospatial Power Tools Reviews [Book]

Thinking of buying my latest book?  We’ve finally got a few reviews on Amazon that might help you decide.  See my other post for more about the book.  Buy the PDF on Locate Press.com.

Reader Reviews

Geospatial Power Tools book cover
From Amazon.com

5.0 out of 5 stars This book makes a great reference manual for using GDAL/OGR suite of command line …,
January 24, 2015 By Leo Hsu
“The GDAL Toolkit is chuckful of ETL commandline tools for working with 100s of spatial (and not so spatial data sources). Sadly the GDAL website only provides the basic API command switches with very few examples to get a user going with. I was really excited when this book was announced and purchased as soon as it came out. This book makes a great reference manual for using GDAL/OGR suite of command line utilities.
Several chapters are devoted to each commandline tool, explaining what its for, the switches it has, and several examples of how to use each one. You’ll learn how to work with both vector/(basic data no vector) data sources and how to convert from one vector format to another. You’ll also learn how to work with raster data and how to transform from one raster data source to another as well as various operations you can perform on these.”

 

“This is a great book for any GIS pro to get their hands dirty and learn how to use spatial tools to do some amazing things. The benefit of using the tools covered in this book (GDAL/OGR) is that you could learn to integrate these tools into other workflows, such as downloading the latest raster data for your organization on weekly basis and running the necessary tools against it.
The first part of the book is filled with some great examples to get you familiar with using these tools and the rest of the book covers the documentation of these tools in detail. This should be on every GIS pros bookshelf.”

 

Amazon UK:

5.0 out of 5 stars Absolute Must Have Reference.
14 Feb. 2015 By SamFranklin
“I have been a GDAL/OGR user for about 3 years and have always struggled to find readable, entry-level GDAL documentation that provides simple examples which are well explained. This I feel this is a barrier to greater adoption of GDAL/OGR.
This book 100% fills this gap and is therefore required reading for GDAL novices and veterans alike. The book is split between “Common Tasks & Workflows” and then a comprehensive guide to all utilities. I read it cover to cover and picked a number of ideas and solutions to problems with existing workflows involving GDAL, for that alone, it’s easily worth buying. The first chapter provides links to sample data which is used throughout the book. This allows the reader to replicate every example which is great for learning trickier and lesser known utilities.Cannot recommend highly enough. Great job.”
Thanks for reading, reviewing and for the encouraging comments guys.

Follow me on Twitter and follow Locate Press too for latest sales and deals!  Want to write a book too?  Contact me to discuss.

 

HBase queries from Bash – a couple simple REST examples

Learn how to do some simple queries to extract data from the Hadoop/HDFS based HBase database using its REST API.

Are you getting stuck trying to figure out HBase query via the REST API?  Me too.  The main HBase docs are pretty limited in terms of examples but I guess it’s all there, just not that easy for new users to understand.

As an aside, during my searches for help I also wanted to apply filters – if you’re interested in HBase filters, you’ll want to check out Marc’s examples here.

What docs do you find most useful?  Leave a comment.  Should someone write more books or something else?

My Use Cases

There were two things I wanted to do – query HBase via REST to see if a table exists (before running the rest of my script, for example).  Then I wanted to grab the latest timestamp from that table.  Here we go…

Does a specific table exist in HBase?

First, checking if a table exists can be done in a couple ways.  The simplest is to simply request the table name with the “exists” path after it and see what result you get back.

$ curl -i http://localhost/existing_table/exists

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: text/plain
Content-Length: 0

$ curl -i http://localhost/bad_table_name/exists
HTTP/1.1 404 Not Found
Content-Length: 11
Content-Type: text/plain

Not found

Here I use the curl “-i” option to return the detailed info/headers so I can see the HTTP responses (200 vs 404).  The plain text results from the command are either blank (if exists) or “Not found” if it does not.

Let’s roll it into a simple Bash script and use a wildcard search to see if the negative status is found:

CMD=$(curl http://localhost/table_name/exists)

if [[ $CMD = *"Not found"* ]]
 then echo "Not found"
else 
 echo "Found"
fi

Extract a timestamp from an HBase scanner/query

Now that I know the table exists, I want to get the latest timestamp value from it.  I thought I’d need to use some filter attributes like I do in HBase shell:

scan 'existing_table', {LIMIT=>1}

To do this with curl, you want to use HBase scanner techniques to accomplish this (the shortest section in the official docs it seems).

It’s a two stage operation – first you initialise a scanner session, then you request the results.  Bash can obviously help pull the results together easily for you, but let’s so go step by step:

curl -vi -H "Content-Type: text/xml" -d '<Scanner batch="1"/>' "http://localhost/existing_table/scanner"

...
Location: http://localhost/existing_table/scanner/12120861925604d3b6cf3
...

Note the XML chunk in the statement that tells it how many records to return in the batch.  That’s as simple as it gets here!

Amongst the results of this command you’ll see the Location value returned, this is the URL to use to access the results of the query.  Results are truncated and line breaked so you can see the meaningful bits:

$ curl http://localhost/existing_table/scanner/12120861925604d3b6cf3

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
 <CellSet>
  <Row key="AfrV65UNXD0AAAAs">
   <Cell column="ZXZ...mNl" 
    timestamp="1218065508852">
    L2hv...Y3N2
   </Cell>
  </Row>
 </CellSet>

Ugh, XML.. if you want JSON instead just add an ACCEPT property to the header:

$ curl -H "Accept: application/json" http://localhost/existing_table/scanner/12120861925604d3b6cf3

{"Row":[{"key":"AfrV65UNXD0AAAAs","Cell":[{"column":"ZXZ...mnl","timestamp":1218065508852,"$":"L2hv...Y3N2"}]}
...

For now we’ll hack some sed to get the to the value we want, first for the JSON response, second for the XML response.  Just pipe the curl command into this sed:

# For application/json
$ curl ... | sed 's/.*\"timestamp\"\:\([0-9]\{13\}\).*/\1/'

# For default XML results
$ curl ... | sed 's/.*timestamp=\"\([0-9]\{13\}\).*/\1/'

Now you can create a basic script the grabs the latest timestamp from the HBase query and decides what to do with it.  Here we just assign it to a variable and let you go back to implement as needed.

$ LAST_TIME=$(curl ... | sed 's/.*\"timestamp\"\:\([0-9]\{13\}\).*/\1/')
$ echo $LAST_TIME

1218065508858

Thanks for reading!

If you like this, follow me on Twitter at http://twitter.com/1tylermitchell or with any of the other methods below.

 

Noob compiling tip – “make -j” is your friend

In today’s super duper processor world if you are still only running the standard trio:

./configure 
make
make install

.. then you’re really missing out. Well, that is unless you like having more time to drink your coffee while waiting for compilation jobs to finish.

The make command comes with an option to spawn off multiple processes while building. This is can dramatically reduce compilation time. For example, building Quantum GIS from source (for use with our upcoming Ingres Geospatial release) – took about 12 minutes to build the traditional way. But, if you take the number of cores your system has, add 1 and pass that along to the make command then you’re really cooking…:

make -j 5

.. it took only 3.5 minutes.

Try it, you’ll like it!

WordPress Appliance - Powered by TurnKey Linux