IoT Day 4: Bidgely Cloud Energy Monitor Dashboard

After a week of collecting smart meter readings, I’m now ready to show results in a cloud-based energy monitor system – Bidgely – complete with graphs showing readings, cost and machine learning results breaking down my usage by appliance.


Bidgely Energy Monitor Dashboard - UsageThis is part 4 of a series of posts about the Internet of Things applied to Home Energy Monitoring.  I have a Smart Meter from BC Hydro, an Eagle energy monitor and various cloud apps helping me understand it all.

 

See my post from Day 1 – getting started or Day 2 – connecting to cloud services – Day 3 – viewing data


3 Value Added Parts to Bidgely

In this post I’ll show you the three parts of Bidgely that I’ve found most helpful:

  1. Usage dashboard
  2. Cost dashboard
  3. Appliance breakdown (best for last!)

Usage Dashboard

Bidgely Energy Monitor Dashboard - Usage
Bidgely Energy Monitor – Usage Dashboard

The main part of the dashboard includes the energy usage during a given period of time – in detail and an overall amount. The usage dashboard is chosen by selecting the lightning bolt symbol in the top right.  On the top left you change the time period for the dashboard.  Changing it to Billing Cycle or Year will summarise the data into larger groups of time – days or months.

In the 24 Hour time period, the chart shows the amount of kilowatts used in 5 minute time periods.  You can mouse-over a point and get the detailed information.  This view helps you to understand what your energy usage is like and can help you identify any usage you didn’t know about.

For example, the small spikes throughout the middle of the night were confusing at first, but I figured out that it was the refrigerator running every hour or so.  This leads me to know that to reduce that usage I could get a more efficient fridge or take other actions such as placing some jugs of water in the fridge to help retain the cold better.  We’ll try that another time and see what effect it has.

Cost Dashboard

Bidgely Energy Monitor Dashboard - Cost
Bidgely Energy Monitor – Cost Dashboard

Very similar to the usage dashboard, the 24 Hour view shows the cost of the usage summarised by hour, just as the hydro company will do their billing.  This screen is turned on by the $ symbol button in the top right.  Mouse over the chart shows more details, include any different Tier usage your utility may have in place.

Appliance Breakdown

Now here is where the real secret sauce of Bidgely gets applied.  Showing charts of usage and cost isn’t rocket science, but applying machine learning to gaining further insight into your usage/costs is really useful.  Let’s look at an example in this screenshot.  Notice the bottom left shows a breakdown of usage by appliance.  When you select one, it shows it in the chart and gives a summary % in the bar below the chart:

Bidgely Energy Monitor - Appliance Breakdown
Bidgely Energy Monitor – Appliance Breakdown

 

This breakdown magically appears when their algorithms and machine learning analysis start to see patterns of usage in your data.  Just like I could instinctively see my refrigerator usage, data analysis can detect similar patterns and make certain assumptions.  For example, my heating system draws 1.5kW, so when there is a 1.5kW increase in usage, the assumption can be that my heater turned on.  That’s a fairly simple example that isn’t going to gain any venture capital funding.

Things get way more interesting when engineers can determine what appliances are running simultaneously.  When a second appliance begins running, say my fridge, at another 0.8kW how can it figure this out?  This takes a combination of factors – additional guestimates about typical fridge usage, as well as frequency and duration – together these produce a super smart system that can determine these factors.

The only downside I see so far is that this analysis happens after-the-fact.  The initial analysis took about 3-4 days, then they emailed me to tell me they had discovered my “always on” usage, heating system, fridge and clothes dryer.  I’m super curious if it will also find my water heater and another space heater I have.  Time will tell.

Wish List

Real-time Analytics

I’m also curious if it can become faster at identifying these components.  Not an easy feat but I’m sure it’s on their TODO list already.  [Perhaps they need a higher performance analytic platform like we produce at Actian!]  The closer to real-time that any analytics can get, the better for everybody and the wider the lead Bidgely will have on its competitors.

Manual Training

The other aspect to Bidgely I’d like to see is more ability to manually train their model.  My premise here is that the user has innate knowledge about what is going on that a system might never really understand – this should be taken advantage of.

I mentioned in an earlier post that their Appliance Analyzer app helped train the model, but I’m not so sure now.  I think it was only to give you a rough idea of your usage in a manual way (which is what they said it did).  But it really should be used to train my house and could be done two ways.

First, just like their little wizard does – you identify what device you are using, select Start, turn it on and it tells you when it sees the spike – done.  Second, show the current usage and allow the user to select from a checklist what is currently running.

Now, using all of that data in real-time on multiple devices that turn on/off over varying durations of time – that’s still the secret sauce this team brings to the table.  I look forward to seeing how they move forward as they continue to develop.  It is sure interesting so far!

Offline Updates

I prefer to power down my computer equipment in the evenings, though I keep my energy monitor running.  When I power it back on it the morning Bidgely won’t update with the data collected earlier.   This is not a unique problem to Bidgely of course, other cloud providers have the same issue, but my Eagle does support downloading older data – so hope they could take advantage of this sync/refresh approach in the future.

VIDEO: Kibana 3 Dashboard – 3 Use Cases Demonstrated

Kibana dashboards, from the Elasticsearch project, can help you visualise activity and incidents in log files. Here I show 3 different types of use cases for dashboards and how each can be used to answer different questions depending on the person.  Video and details follow.

Video no longer available on YouTube: Kibana 3 Dashboard - 3 Use Cases

Text Search Dashboard

The first example is the simplest UI I could imagine: a query/search box, a histogram, and a table.  In this instance any user, at any level of curiosity, can find textual data in the logs using a keyword match.

Kibana 3 Dashboard: Text Search

Then they can also see the relative number of records that occur at given times within the time window of all the data available.  These are aggregate counts of all records that have some match to the query keyword or algebra.

Likewise, the table reflects the subset of data provided by the records, with the ability to only show fields of interest.

Process Details

A slightly more advanced use is to focus on a particular process (i.e. application) running on a machine that’s being logged.  Here we can then take a particular metric, i.e. CPU usage, and graph it instead of just a simple histogram.

Kibana 3 Dashboard: Process Details

A typical user may be in charge of a particular set of services in a system.  Here they can see how they perform and yet still dig into the details as desired.

I also do some cool “markers” to subtly show when events coincide with other process metrics.

All Events

The data example shown here has process, performance and event logging information.  I combine multiple queries and having them drive different parts of the dashboard – a pie chart, summary table, histogram, sparkline and other charts based on numeric data.

Kibana 3 Dashboard: All Events

These can then all be filtered based on time windows that are interactively selected.  This is really the typical picture of a dashboard – giving more densely packed information about a variety of metrics, ideal for system managers to get a handle on things.

Streaming Pipeline

The data are generated by Windows servers using a custom C# application that pushes data in a Kafka topic in a Hadoop cluster running in EC2. The data stream is then read from the topic using Actian DataFlow platform and pushed into Elasticsearch for Kibana to use at the end of the pipeline.  There are other reasons I have this kind of pipeline – namely that DataFlow can simultaneously also feed other outgoing part of the pipeline – RDBMS, graph, etc.   More on that in a future video/post.

Next Steps

  • My next plans are to show you Kibana version 4 in action, replicating some of what I’ve shown here.
  • If you haven’t seen it already, see this link and my video with some tips and tricks for using Kibana 3.
  • Tell me more about your interests in dashboards and I’ll consider focusing on them too.

Analytics Dashboard – Kibana 3 – a few short quick tips

Video no longer available on YouTube: Analytics Dashboard - Kibana 3 - a few short quick tips

After you’ve loaded log files into elasticsearch you can start to visualize them using the Kibana web app and build your own dashboard. While using Kibana for a week or so, I found it tricky to find the docs or tutorials to get me up to speed quickly with some of the more advanced/hidden features.

In this Kibana dashboard video:

  1. build TopN automated classification queries
  2. view the TopN values of a particular column from the table panel
  3. manually create multiple queries to appear as series in your charts
WordPress Appliance - Powered by TurnKey Linux