Make Stellar/Star Data Maps In GIS

I have a latent interest in stargazing but haven’t done much about it for a long time until the past weekend when I had a bit of time and wondered if I could create a star map using QGIS. I found a couple tutorials and stackexchange questions that referenced David Nash’s HYG Database from www.Astronexus.com. Some of the tutorials showed equations for computing latitude and longitude from the star position values – ascension, declination specifically. However, I found that there is already an X and Y column in the latest version of the dataset which makes it easy to map, here’s how.

Download the Star Location Data

From the HYG Database page, grab the HYG version 3.0 dataset:

  • HYG 3.0: Database containing all stars in Hipparcos, Yale Bright Star, and Gliese catalogs (almost 120,000 stars, 14 MB)

The resulting 34MB CSV file contains about 20 columns and 119,615 rows.

Import CSV File into QGIS

  1. Launch QGIS (I’m using 2.10.1-Pisa)
  2. Select Layer -> Add Layer -> Add Delimited Text Layer
  3. Browse to the file location, select File Format radio button as CSV and the rest should take care of itself

Form importing the HYG Database star map CSV file into QGIS

Tweak the Map

If you use a typical WGS84 projection you’ll get a ton of points oriented in a circle.  I took a few minutes to zoom into a meaningful scale (for me it was around 1:5M), change the background colour and then use the LUM attribute to change the relative size of the stars.  I also changed the colours to be shades of white.  Here is what I was able to produce.  It’s not all that meaningful to me but I know that when I need to dig into the data further it will be available!

hygdata_quick_qgis_starmap

I’ve attached my project file here so you can get started quickly.

I’d love to hear what you would use this kind of data for and if you have any tips for making it more useful or better rendered.

Parsing GPS with Arduino – retro-post

I dug up one of my oldest blog posts from about 7 years ago.  In the post, I show how I had connected my Garmin eTrex GPS receiver to an Arduino board and used it to control a camera in a desktop application.

After pumping the data into the Arduino, I parsed the raw GPS data coming from the eTrex and streamed it out to a Python app on the desktop, via a serial port.  I was using the output to position a virtual camera in the OSSIMPlanet platform by formatting XML in Python and sending it to a listener built into the OSSIMPlanet app.  (OSSIMPlanet is a sort of pre-Google Earth, Google Earth on steroids product.)

I haven’t used it for years, but thought some of the methods from this old post may still apply if you have an Arduino + GPS or Arduino -> Python streaming requirement.   Enjoy the flashback – I know it’s inspired me to pick up a new Arduino to continue where I left off.


 

GPS Controls OSSIMPlanet - With OSSIMPlanet’s nifty camera control and listener functionality, as demonstrated in my last post, you’ve got so many neat opportunities. A couple nights ago I got a basic GPS NMEA parser working. Here’s a pic of the ultra-professional connection method I use to hook it to my arduino board 🙂 Oops, just realised that the picture … Continue reading GPS Controls OSSIMPlanet

Two wires to hook an eTrex data cable up to the Arduino
Two wires to hook an eTrex data cable up to the Arduino

Read Post

Geospatial Power Tools Reviews [Book]

Thinking of buying my latest book?  We’ve finally got a few reviews on Amazon that might help you decide.  See my other post for more about the book.  Buy the PDF on Locate Press.com.

Reader Reviews

Geospatial Power Tools book cover
From Amazon.com

5.0 out of 5 stars This book makes a great reference manual for using GDAL/OGR suite of command line …,
January 24, 2015 By Leo Hsu
“The GDAL Toolkit is chuckful of ETL commandline tools for working with 100s of spatial (and not so spatial data sources). Sadly the GDAL website only provides the basic API command switches with very few examples to get a user going with. I was really excited when this book was announced and purchased as soon as it came out. This book makes a great reference manual for using GDAL/OGR suite of command line utilities.
Several chapters are devoted to each commandline tool, explaining what its for, the switches it has, and several examples of how to use each one. You’ll learn how to work with both vector/(basic data no vector) data sources and how to convert from one vector format to another. You’ll also learn how to work with raster data and how to transform from one raster data source to another as well as various operations you can perform on these.”

 

Continue reading Geospatial Power Tools Reviews [Book]

Converting Decimal Degree Coordinates

Converting Decimal Degree Coordinates to/from DMS Degrees Minutes Seconds

cs2cs command from GDAL/OGR toolset (gdal.org) - allows robust coordinate transformations.
cs2cs command from GDAL/OGR toolset (gdal.org) – allows robust coordinate transformations.

If you have files or apps that have to filter or convert coordinates – then the cs2cs command is for you.  It comes with most distributions of the GDAL/OGR (gdal.org) toolset.  Here is one popular example for converting between degrees minutes and seconds (DMS) and decimal degrees (DD).


Geospatial Power Tools book coverThe following is an excerpt from the book: Geospatial Power Tools – Open Source GDAL/OGR Command Line Tools by me, Tyler Mitchell.  The book is a comprehensive manual as well as a guide to typical data processing workflows, such as the following short sample…


Input coordinates can come from the command line or an external file. Assuming a file containing DMS (degree, minute, seconds) style, looks like:

124d10'20"W 52d14'22"N
122d20'05"W 54d12'00"N

Use the cs2cs command, specifying how the print format will be returned, using the -f option. In this case -f “%.6f”
is explicitly requesting a decimal degree number with 6 decimals:

cs2cs -f "%.6f" +proj=latlong +datum=WGS84 input.txt

Example Converting DMS to/from DD

This will return the results, notice no 3D/Z value was provided, so none is returned:

-124.172222 52.239444 0.000000
-122.334722 54.200000 0.000000

To do the inverse, remove the formatting option and provide a list of values in decimal degree (DD):

cs2cs +proj=latlong +datum=WGS84 inputdms.txt
124d10'19.999"W 52d14'21.998"N 0.000
122d20'4.999"W 54d12'N 0.000


Geospatial Power Tools is 350+ pages long – 100 of those pages cover these kinds of workflow topic examples. Each copy includes a complete (edited!) set of the GDAL/OGR command line documentation as well as the following topics/examples:

Workflow Table of Contents

  1. Report Raster Information – gdalinfo
  2. Web Services – Retrieving Rasters (WMS)
  3. Report Vector Information – ogrinfo
  4. Web Services – Retrieving Vectors (WFS)
  5. Translate Rasters – gdal_translate
  6. Translate Vectors – ogr2ogr
  7. Transform Rasters – gdalwarp
  8. Create Raster Overviews – gdaladdo
  9. Create Tile Map Structure – gdal2tiles
  10. MapServer Raster Tileindex – gdaltindex
  11. MapServer Vector Tileindex – ogrtindex
  12. Virtual Raster Format – gdalbuildvrt
  13. Virtual Vector Format – ogr2vrt
  14. Raster Mosaics – gdal_merge

Query Vector Data Using a WHERE Clause – ogrinfo

The following is an excerpt from the book: Geospatial Power Tools – Open Source GDAL/OGR Command Line Tools by Tyler Mitchell.  The book is a comprehensive manual as well as a guide to typical data processing workflows, such as the following short sample…

Use SQL Query Syntax with ogrinfo

Use a SQL-style -where clause option to return only the features that meet the expression. In this case, only return the populated places features that meet the criteria of having NAME = ’Shanghai’:

$ ogrinfo 10m_cultural ne_10m_populated_places -where "NAME = 'Shanghai'"

... 
Feature Count: 1 Extent: (-179.589979, -89.982894) - (179.383304, 82.483323) 
... 
OGRFeature(ne_10m_populated_places):6282
 SCALERANK (Integer) = 1 
 NATSCALE (Integer) = 300 
 LABELRANK (Integer) = 1 
 FEATURECLA (String) = Admin-1 capital 
 NAME (String) = Shanghai
... 
 CITYALT (String) = (null) 
 popDiff (Integer) = 1 
 popPerc (Real) = 1.00000000000 
 ls_gross (Integer) = 0 
 POINT (121.434558819820154 31.218398311228327)

Building on the above, you can also query across all available layers, using the -al option and removing the specific layer name. Keep the same -where syntax and it will try to use it on each layer. In cases where a layer does not have the specific attribute, it will tell you, but will continue to process the other layers:

   ERROR 1: 'NAME' not recognised as an available field.

NOTE: More recent versions of ogrinfo appear to not support this and will likely give FAILURE messages instead.


Geospatial Power Tools is 350 pages long – 100 of those pages cover these kinds of workflow topic examples.  Each copy includes a complete (edited!) set of the GDAL/OGR command line documentation as well as the following topics/examples:

Workflow Table of Contents

  1. Report Raster Information – gdalinfo 23
  2. Web Services – Retrieving Rasters (WMS) 29
  3. Report Vector Information – ogrinfo 35
  4. Web Services – Retrieving Vectors (WFS) 45
  5. Translate Rasters – gdal_translate 49
  6. Translate Vectors – ogr2ogr 63
  7. Transform Rasters – gdalwarp 71
  8. Create Raster Overviews – gdaladdo 75
  9. Create Tile Map Structure – gdal2tiles 79
  10. MapServer Raster Tileindex – gdaltindex 85
  11. MapServer Vector Tileindex – ogrtindex 89
  12. Virtual Raster Format – gdalbuildvrt 93
  13. Virtual Vector Format – ogr2vrt 97
  14. Raster Mosaics – gdal_merge 107

GPS Controls OSSIMPlanet

With OSSIMPlanet’s nifty camera control and listener functionality, as demonstrated in my last post, you’ve got so many neat opportunities. A couple nights ago I got a basic GPS NMEA parser working. Here’s a pic of the ultra-professional connection method I use to hook it to my arduino board 🙂

Two wires to hook an eTrex data cable up to the Arduino
Two wires to hook an eTrex data cable up to the Arduino

Oops, just realised that the picture shows the wires in the wrong spot (it was a late night photo). Pin 2 from the serial cable goes into the Power GND on left side of board. Pin 5 from serial cable goes into the Digital Pin 0 aka RX pin on lower right.

Note that I had a lot of confusion regarding the pin-out options for the Etrex. I looked at several diagrams and couldn’t get the right pins to work. It appears that you may actually need to swap the ground/TX pins to make it work – that’s what I had to do. This is something to do with how the serial connection works.. something I won’t pretend to understand – but swapping the wires worked, that’s all I know!

Of course you could always use your nifty new bluetooth GPS receiver, or plug your receiver directly into the PC and this would still work. But for me, I will have additional devices going into the arduino, where their signals will get mixed together before being sent to the PC.

Arduino setup

The arduino doesn’t do much here, except filter the strings coming from the GPS. It grabs only the $GPGGA string that has the location info I want. I specifically used the GPGGA because the Python NME parser code example I had used that one and I didn’t want to have to learn how to handle the $GPRMC strings.

Here’s the basic code I run on the Arduino:

Tmitchell: on etrex port - pin 1 (at notch end) is for TX and pin 3 for GND
On serial cable out Pin 2 -> GND, Pin 5 as TX (hooks to Digital 0=RX on board)
*/ 
#include <string.h>

#include <ctype.h>
int ledPin = 13;                  // LED test pin
int rxPin = 0;                    // RX PIN 
int txPin = 1;                    // TX TX
int byteGPS=-1;
char linea[70] = "";
char comandoGPR[7] = "$GPGGA";
char latdms[9], londms[10], latdir[1], londir[1] = "";
int latdd, londd, heading = 0;
int cont=0;
int bien=0;
int conta=0;
int indices[13];

void setup() {
  pinMode(ledPin, OUTPUT);       // Initialize LED pin
  pinMode(rxPin, INPUT);
  pinMode(txPin, OUTPUT);
  Serial.begin(4800);
  for (int i=0;i<70;i++){       // Initialize a buffer for received data
    linea[i]=' ';
  }   
}
void loop() {
  digitalWrite(ledPin, HIGH);
  byteGPS=Serial.read();         // Read a byte of the serial port
  if (byteGPS == -1) {           // See if the port is empty yet
    delay(100); 
  } else {
    linea[conta]=byteGPS;        // If there is serial port data, it is put in the buffer
    conta++;                      
    if (byteGPS==13){            // If the received byte is = to 13, end of transmission
      digitalWrite(ledPin, LOW); 
      cont=0;
      bien=0;
      for (int i=1;i<7;i++){     // Verifies if the received command starts with $GPR
        if (linea[i]==comandoGPR[i-1]){
          bien++;
        }
      }
      if(bien==6){               // If yes, continue and process the data
        for (int i=0;i<70;i++){
          if (linea[i]==','){    // check for the position of the  "," separator
            indices[cont]=i;
            cont++;
          }
          if (linea[i]=='*'){    // ... and the "*"
            indices[12]=i;
            cont++;
          }
        }
          Serial.print(linea);  // This is the important line :)
      }
      conta=0;                    // Reset the buffer
      for (int i=0;i<70;i++){    //  
        linea[i]=' ';   
      }                 
    }
  }
}

 

I had originally hoped to do the XML prep in the arduino, but skipped it for now due to my poor understanding of variable types in the processing language. So for now I’ve still got a lot of cruft leftover in the above, that I didn’t need from the original tutorial code. But as I add more sensors I’ll want to do more mixing in the board itself.

So the arduino board just sends a raw NMEA $GPGGA string to the serial port, where Python takes over.

Python Serial Reader and OSSIM Controller

I then use a Python script that checks for strings coming from the arduino board. It does a bit of filtering, but not much error checking at the moment. This is the first time I’ve used Python to connect to the serial port, so it was fun to learn and so simple!

I found this GPGGA parser code and incorporated it into my script. I won’t paste it here as it is quite long. But here is the rest of my script – reading from Serial, parsing results, then reformatting and sending to OSSIMPlanet… comments, cruft and crummy coding.. all yours for free!

Be sure to have OSSIMPlanet running and set its Preferences to listen on port 5000 first.

# Uses PySerial
import serial

# GPGGA Parsing code next

class GPGGAParser(object):
import logging

# Open connection to Arduino
s = serial.Serial('/dev/cu.usbserial-A6001VQr', 4800, timeout=1) heading = 0
# initialising here so I can rotate the heading during each cycle.. just for fun

# Read Arduino output

while (1>0):
char = s.read(1)
while (char <> '$'):
char = s.read(1)
else:
sentence = '$' + s.read(70)
# Parse output
#############
print sentence.strip()
if (len(sentence.strip())<=30) or (sentence[1:5] <> 'GPGG'):
continue
else:
parsed = GPGGAParser(sentence)

if (parsed.latitude <> 0 and parsed.longitude <> 0):

#####################
# Reformat to XML
## Hacked method first
heading += 10 # during testing I never moved,
# so I made it hover over my location and
# slowly spin 10 degrees each second
if (heading >= 360): heading = heading-360
pitch = 70
altitude_mult = 10
ossimxml = '<Set target=":navigator" vref="wgs84"><Camera><longitude>%s</longitude><latitude>%s</latitude><altitude>%s</altitude><heading>%s</heading><pitch>%s</pitch><roll>0</roll><altitudeMode>absolute</altitudeMode></Camera></Set>' % (parsed.longitude, parsed.latitude, parsed.altitude * altitude_mult, heading, pitch)
print ossimxml # show xml being sent to ossimplanet
## Proper DOM tools second - will do this "right" at some point :)
## But not written yet

#######
# Open connection to OSSIMPlanet listener
import socket
ossim = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ossim.connect(("127.0.0.1", 5000))
# Send view updates
# Optional: connect to OSSIMPlanet broadcaster
## adjust coordinates based on current view
## I will do this when using nunchuks, but for now nothing
ossim.send(ossimxml)
ossim.close()
# Lather, rinse, repeat....

That’s about it. In OSSIMPlanet the updates are practically instantaneous, but I wait 1 sec for new GPS data to come in. When I start using the nunchuks I’ll want to do more updates faster to emulate the movement smoother. But that’s for another day…

%d bloggers like this: