Analytics Archives - spatialguru.com

Getting into large(ish) GIS datasets

It’s been a long time since a post but I wanted to highlight some of the new tooling that I’m learning while digging into the OmniSci platform – namely, extreme geospatial map rendering and analytics powered by server-side GPUs.

In this video I load a ~12M point dataset from geonames.org and explore it using both QGIS and OmniSci – just to get a feel for how easily it fits a normal kind of GIS exploration workflow.

See OmniSci in action with truly large datasets (we talk in billions usually) using some interactive demos at: https://www.omnisci.com/demos

Partitioned Data & Why It Matters

Would love to hear how you partition your spatial data..

As data volumes grow, so does your need to understand how to partition your data. Until you understand this distributed storage concept, you will be unable to choose the best approach for the job. This post gives an introductory explanation of partitioning and you will see why it is integral to the Hadoop Distributed File System (HDFS) increasingly used in modern big data architectures….

Read the full article on Make Data Useful >>

Home Energy Monitor Series – Internet of Things

Recently I started an Internet of Things series on my experiences installing, using and analysing data from a smart electrical meter. This included a BC Hydro smart meter, Eagle monitoring gateway from rainforest automation, and a cloud-based analytics service from Bidgely.

I’ve collated all the posts on the topic for you below. More will be added as I write them. Enjoy!

Continue reading Home Energy Monitor Series – Internet of Things

IoT Day 4: Bidgely Cloud Energy Monitor Dashboard

After a week of collecting smart meter readings, I’m now ready to show results in a cloud-based energy monitor system – Bidgely – complete with graphs showing readings, cost and machine learning results breaking down my usage by appliance.

This is part 4 of a series of posts about the Internet of Things applied to Home Energy Monitoring. I have a Smart Meter from BC Hydro, an Eagle energy monitor and various cloud apps helping me understand it all.

See my post from Day 1 – getting started or Day 2 – connecting to cloud services – Day 3 – viewing data

3 Value Added Parts to Bidgely

In this post I’ll show you the three parts of Bidgely that I’ve found most helpful:

Usage dashboard
Cost dashboard
Appliance breakdown (best for last!)

Usage Dashboard

Bidgely Energy Monitor Dashboard - Usage — Bidgely Energy Monitor – Usage Dashboard

Continue reading IoT Day 4: Bidgely Cloud Energy Monitor Dashboard

Review of 3 Recent Internet of Thing (IoT) Announcements

Working in the big data and analytics space, I’m always interested in parts of the Internet of Things (IoT) that will produce more data, require more backend systems, and help users/customers get on with their day better.

The past week has shown a few interesting announcements relating to Internet of Things topics. Here are just a few that jumped out to me, either because they inspired me or because I was left wondering what it would really mean.

TL;DR? Summary: While IBM is “getting started” (oops, I meant “getting serious”) and Facebook has big plans to “take over”, Amazon comes out with a consumer focused solution.

Continue reading Review of 3 Recent Internet of Thing (IoT) Announcements

IoT Day 3: Viewing data on the Eagle Energy Monitor

The Eagle energy monitor from Rainforest Automation is a very handy device. It reads the wireless signal from my electricity meter and makes it available through a web interface – both a graphical environment and a RESTful API. In this post we look at the standard graphical screens and the data download option. Next time we’ll look at the RESTful API for programmers to use.

This is part 3 of a series of posts about the Internet of Things applied to Home Energy Monitoring.

See my post from Day 1 – getting started or Day 2 – connecting to cloud services.

Continue reading IoT Day 3: Viewing data on the Eagle Energy Monitor

Plug-In Solar: Moveable Solar Power For Renters and Do-It-Yourselfers

via Green Building Elements | From brick and mortar shops to city planning, we cover sustainable trends in construction, renovation, and more..

This definitely sounds promising, anyone have experience running a similar setup?

A small company, SpinRay Energy, has announced the production of a new UL listed, grid-tied solar power system that couldn’t be easier to install. Plug-in solar power equipment may be a game changer in the arena of small residential solar power.

I see two scenarios for those in rental environments – hanging something from windows or setting up on a porch. Ultimately I guess the efficacy will be limited by size availability and cost, but size issues could be minimised if it were possible to hang PV systems in some manner without needing to screw things down into exterior walls.

Just a thought – you tried anything like this? Share any links to your favourite portable, yet substantial, systems.

IoT Day 2: Cloud Services for Energy Monitoring

Energy monitoring isn’t only about knowing what’s happening right now, but also understanding what happened historically. Often that includes knowing not just what was happening but also when and why. Enter cloud services for energy monitoring. They range from simple charts and graphs to predicting your usage over time – essentially storing, analysing and enriching your raw data.
In this post I review two cloud services that I’ve tried so far and show you what you get from them. Continue reading IoT Day 2: Cloud Services for Energy Monitoring

IoT Day 1: Home Energy Monitoring

In my next series of blog posts we explore an Internet of Things topic – Home energy monitoring – from a first person perspective. Join me as I install, use and hack a monitor (and related cloud services) in my new home.

This is part 1 of a series of posts about the Internet of Things applied to Home Energy Monitoring.

See my post from Day 2 – starting to use Cloud Services. Continue reading IoT Day 1: Home Energy Monitoring

Drinking from the (data) Firehose of Terror

Between classic business transactions and social interactions and machine-generated observations, the digital data tap has been turned on and it will never be turned off. The flow of data is everlasting. Which is why you see a lot of things in the loop around real time frameworks and streaming frameworks. – Mike Hoskins, CTO Actian

From Mike Hoskins to Mike Richards (yes we can do that kind of leap in logic, it’s the weekend)…

Oh, Joel Miller, you just found the marble in the oatmeal! You’re a lucky, lucky, lucky little boy – because you know why? You get to drink from… the firehose! Okay, ready? Open wide! – Stanley Spadowski, UHF

Firehose of Terror

I think you get the picture – a potentially frightening picture for those unprepared to handle the torrent of data that is coming down the pipe. Unfortunately, for those who are unprepared, the disaster will not merely overwhelm them. Quite the contrary – I believe they will be consumed by irrelevancy.

If you’re still with me, let me explain. Continue reading Drinking from the (data) Firehose of Terror