spatialguru.com

Web console for Kafka messaging system

Running Kafka for a streaming collection service can feel somewhat opaque at times, this is why I was thrilled to find the Kafka Web Console project on Github yesterday. This Scala application can be easily downloaded and installed with a couple steps. An included web server can then be launched to serve it up quickly. Here’s how to do all that.

For a quick intro to what the web console does, see my video first. Instructions for getting started follow below, including the quick video.

Video no longer available on YouTube: Kafka Web Console Introduction

Kafka Web Console Project – Download

The main repository for this project is available on Github (claudemamo/kafka-web-console). However, I wanted the ability to add and remove Kafka topics so I use a forked repository that has a specific branch with those capabilities. (These are planned to be added to the main project but have not been yet.)

Download the ZIP archive file and unzip.

Before doing anything further, we need another application to build and launch the console.

Download Play Framework

A program called Play with Activator is used to build and launch the web console. It’s a Java app for launching Scala apps.

Download it here, unzip it and add it to the system path so you can execute the activator command that it provides.

Build/Launch

Now back to the Kafka web console code. Enter the top level directory and execute the Play Activator start command (with one special option):

cd kafka-web-console-topic-add-remove
activator start -DapplyEvolutions.default=true

 [info] Loading project definition from /home/demo/src/tmp/kafka-web-console-topic-add-remove/project
 [info] Set current project to kafka-web-console (in build file:/home/demo/src/tmp/kafka-web-console-topic-add-remove/)

 (Starting server. Type Ctrl+D to exit logs, the server will remain in background)

 Play server process ID is 8528
 [info] play - database [default] connected at jdbc:h2:file:play
 [info] play - Starting application default Akka system.
 [info] play - Application started (Prod)
 [info] play - Listening for HTTP on /0:0:0:0:0:0:0:0:9000

The first time it runs will take some time to build the source and then launch the web server.

In the last line, above, you see it launches by default on port 9000.

Configuring the Console

Kafka Web Console - Zookeeper Register Form — Step 1 – Register your Zookeeper instance

Now you can launch the web console and start using the application. Step 1 figure shows the basic form for your Zookeeper configuration details. This is the only setup step required to get access to your Kafka brokers and topics.

The remained of the steps/figures are just showing the different screens. See the video for watching it in action.

Kafka Web Console - Broker List Table — Step 2 – Brokers tab shows list of Kafka Brokers

Kafka Web Console - Topic List Table — Step 3 – Topics tab lists the Kafka topics. Note that this forked version provides the red “delete” button for each topic, as well as the additional “create topic” tab above.

Kafka Web Console - Topic Feed Table — Step 4 – Selecting a topic allows you to see an active topic stream.

Drinking from the (data) Firehose of Terror

Between classic business transactions and social interactions and machine-generated observations, the digital data tap has been turned on and it will never be turned off. The flow of data is everlasting. Which is why you see a lot of things in the loop around real time frameworks and streaming frameworks. – Mike Hoskins, CTO Actian

From Mike Hoskins to Mike Richards (yes we can do that kind of leap in logic, it’s the weekend)…

Oh, Joel Miller, you just found the marble in the oatmeal! You’re a lucky, lucky, lucky little boy – because you know why? You get to drink from… the firehose! Okay, ready? Open wide! – Stanley Spadowski, UHF

Firehose of Terror

I think you get the picture – a potentially frightening picture for those unprepared to handle the torrent of data that is coming down the pipe. Unfortunately, for those who are unprepared, the disaster will not merely overwhelm them. Quite the contrary – I believe they will be consumed by irrelevancy.

If you’re still with me, let me explain.

Play video: Firehose

I agree that the tap has been turned on, maybe not at the full blast of power or under maximum control, yet the data is coming and already well beyond a trickle. The helter-skelter implementations of big data solutions out there has, perhaps, created more of a turbulent blasting firehose than a meandering stream of flowing data. And this is only the beginning.

Success is Still Newsworthy

We are still at the stage where designing a (successful) enterprise system built around streaming data, for example, is big news. Why is it news? Because building is harder than merely planning, especially when new open source projects continue to push us beyond the bleeding edge. New tools are helping us see how we can handle more data and find more value, but they are also making the pool of data so much larger that the tools themselves are often irrelevant by the time they are adopted.

For example, MapReduce was awesome, until it began to be so widely adopted that its limitations became apparent. It’s like finding the marble in a sandbox filled with oatmeal – not easy, but when you find it, you’re a winner! Oh, the prize is a fierce typhoon of even more data coming your way. Congratulations! (Sorry you didn’t prepare for that.)

So where does this leave organisations that have no ability to handle more than a trickle of data?

It’s a win or lose scenario – either you can do something about it or you can’t. As software developers or data managers we won’t be judged along some smooth gradation of skills and capabilities.

We’ll be judged against a checklist
Yes or no.
Pass or fail.
Win or lose.
Firehose or … an icky pail to hide in the closet.

Data’s Need for Speed

Why is it a pass/fail scenario? Consider your car – is it successful when it mostly starts in the morning? Never. Anything beyond fully starting is a complete failure because that is what it is designed to do.

I argue that today’s data streams are being designed to handle data at maximum velocity. Sure many services aren’t producing millions of records per second, but as we gear up with the latest toolsets, we make the tools themselves hunger and thirst to get more and more data into their greedy little hands. Feed the beast – or ignore it at your peril.

Systems today are designed to run at full throttle – 100% – all out – maximum overdrive. None of us would like to have pay for premium broadband and find that we only use 10% of our bandwidth. Likewise, our systems are waiting for us to crank up the volume to see what we’ll do next.

Our data economy inherently wants to run at maximum but much of the old plumbing needs upgrades to function at that rate. You’d be irate if the fire department ran their hoses at 10% power when trying to douse your burning home. However, if they told you later that max pressure would burst the old hoses and you’d have no water, then you might be a little more appreciative.

Patch up those hoses so they are tested and ready. Buckle up the survival suit. Start digging through the oatmeal and open wide. Be forewarned – if you are not searching for the marble, you can never find it. If cannot find it, you’ll be sent home empty handed.

—-

p.s. If you don’t win, I highly doubt you’ll even receive a lousy copy of the home edition.

OSX Open Command – Launch Custom Application

The OSX “open” command line tool is very useful. Use it to launch a URL or point to a folder and the web browser or Finder pops up automatically. But what about when you want to launch a particular app to handle a resource you provide? It can do that as well. Easily.

In this example I’m setting up my environment to launch several Windows remote desktop client sessions using the wonderful CoRD application. The idea is that I can put the CoRD app in a folder with a shell script and have an easily transportable launcher package to give to others – without them having to install a RDP client, etc.

Because I’m not install the CoRD application globally, the RDP protocol doesn’t get associated with the app. Therefore, when using the open command I need to explicitly tell it what application to launch it with, using the “-a” flag:

 open -a /User/tyler/CoRD.app/Contents/MacOS/CoRD rdp://winserver1:3389

The benefit to using the open command here is that it won’t launch new CoRD instances, but will add them as sessions to the existing one. Likewise, open will start the process and return you to the command line – so you can list several open statements in a script and it will run them all without having to do any funky backgrounding step.

Thanks to this old OSXDaily post for pointing me in the right direction. I had given up on the command because I didn’t realise there were more options. Next time I’ll read the manual first!

VIDEO: Kibana 3 Dashboard – 3 Use Cases Demonstrated

Kibana dashboards, from the Elasticsearch project, can help you visualise activity and incidents in log files. Here I show 3 different types of use cases for dashboards and how each can be used to answer different questions depending on the person. Video and details follow.

Video no longer available on YouTube: Kibana 3 Dashboard - 3 Use Cases

Text Search Dashboard

The first example is the simplest UI I could imagine: a query/search box, a histogram, and a table. In this instance any user, at any level of curiosity, can find textual data in the logs using a keyword match.

Then they can also see the relative number of records that occur at given times within the time window of all the data available. These are aggregate counts of all records that have some match to the query keyword or algebra.

Likewise, the table reflects the subset of data provided by the records, with the ability to only show fields of interest.

Process Details

A slightly more advanced use is to focus on a particular process (i.e. application) running on a machine that’s being logged. Here we can then take a particular metric, i.e. CPU usage, and graph it instead of just a simple histogram.

A typical user may be in charge of a particular set of services in a system. Here they can see how they perform and yet still dig into the details as desired.

I also do some cool “markers” to subtly show when events coincide with other process metrics.

All Events

The data example shown here has process, performance and event logging information. I combine multiple queries and having them drive different parts of the dashboard – a pie chart, summary table, histogram, sparkline and other charts based on numeric data.

These can then all be filtered based on time windows that are interactively selected. This is really the typical picture of a dashboard – giving more densely packed information about a variety of metrics, ideal for system managers to get a handle on things.

Streaming Pipeline

The data are generated by Windows servers using a custom C# application that pushes data in a Kafka topic in a Hadoop cluster running in EC2. The data stream is then read from the topic using Actian DataFlow platform and pushed into Elasticsearch for Kibana to use at the end of the pipeline. There are other reasons I have this kind of pipeline – namely that DataFlow can simultaneously also feed other outgoing part of the pipeline – RDBMS, graph, etc. More on that in a future video/post.

Next Steps

My next plans are to show you Kibana version 4 in action, replicating some of what I’ve shown here.
If you haven’t seen it already, see this link and my video with some tips and tricks for using Kibana 3.
Tell me more about your interests in dashboards and I’ll consider focusing on them too.

Google wants “mobile-friendly” – fix your WordPress site

TheNextWeb reports: “Google will begin ranking mobile-friendly sites higher starting April 21“. It’s always nice having advance warning, so use it wisely – here’s how to tweak WordPress to increase your mobile-friendliness.

Google Mobile-Friendly Check

I use a self hosted WordPress site and wanted to make sure it was ready for action. I already thought it was, because I’ve accessed in on a mobile device very often and it worked okay.

I even went onto the Google Web Admin tools and the mobile usability check said things were fine, but…

However, all was not golden when I ran the Google mobile-friendly checker. (Obviously two different apps here, hopefully those will merge.)

Try it here, now! The complaints about were that some content is wider than screens and that links were too close together. Fair enough.

WordPress Mobile-Friendly Activation

If you’re not already using WordPress’s Jetpack features, you’re really missing out. I use it mostly for monitoring stats but there are several other features that make it very useful, including one called Mobile Theme.

From the admin sidebar select Jetpack (install it first if not already enabled). It will show you some suggested plugins to enable, plus show you a search bar to find others.
Enter “mobile” and click on the Mobile Theme item.
Activate it (lower-right corner button).
And you’re done!

Going back to Google’s checker it shows a different preview now and also says things are fine. Looking at the site after making these changes, it’s obviously better.

However, I still have one plugin (Crayon markup) that helps display code samples that seems to force some posts to wider than the screen. I assume the plugin creators will fix that up, but it’s not too bad at this point. Unless Google complains, it doesn’t matter anyway!

iPhone cable – loose connection?

iPhone cable – mysterious loose connection bothering you? Before buying a new gold plated cord or adapter, clean out the port with a toothpick. You will be amazed!

Kafka Consumer – Simple Python Script and Tips

[UPDATE: Check out the Kafka Web Console that allows you to manage topics and see traffic going through your topics – all in a browser!]

When you’re pushing data into a Kafka topic, it’s always helpful to monitor the traffic using a simple Kafka consumer script. Here’s a simple script I’ve been using that subscribes to a given topic and outputs the results. It depends on the kafka-python module and takes a single argument for the topic name. Modify the script to point to the right server IP.

from kafka import KafkaClient, SimpleConsumer
from sys import argv
kafka = KafkaClient("10.0.1.100:6667")
consumer = SimpleConsumer(kafka, "my-group", argv[1])
consumer.max_buffer_size=0
consumer.seek(0,2)
for message in consumer:
 print("OFFSET: "+str(message[0])+"\t MSG: "+str(message[1][3]))

Max Buffer Size

There are two lines I wanted to focus on in particular. The first is the “max_buffer_size” setting:

consumer.max_buffer_size=0

When subscribing to a topic with a high level of messages that have not been received before, the consumer/client can max out and fail. Setting an infinite buffer size (zero) allows it to take everything that is available.

If you kill and restart the script it will continue where it last left off, at the last offset that was received. This is pretty cool but in some environments it has some trouble, so I changed the default by adding another line.

Offset Out of Range Error

As I regularly kill the servers running Kafka and the producers feeding it (yes, just for fun), things sometimes go a bit crazy, not entirely sure why but I got the error:

kafka.common.OffsetOutOfRangeError: FetchResponse(topic='my_messages', partition=0, error=1, highwaterMark=-1, messages=)

To fix it I added the “seek” setting:

consumer.seek(0,2)

If you set it to (0,0) it will restart scanning from the first message. Setting it to (0,2) allows it to start from the most recent offset – so letting you tap back into the stream at the latest moment.

Removing this line forces it back to the context mentioned earlier, where it will pick up from the last message it previously received. But if/when that gets broke, then you’ll want to have a line like this to save the day.

For more about Kafka on Hadoop – see Hortonworks excellent overview page from which the screenshot above is taken.

Web Mapping Illustrated – 10 year celebration giveaway [ENDED!]

web-mapping-tyler-mitchell-large — My O’Reilly, 2005 book

Update: All copies are gone! If you want Geospatial Desktop or Geospatial Power Tools – go to LocatePress.com – quantity discounts available. For Web Mapping Illustrated go to Amazon.

I’m giving away a couple copies of my circa 2005 classic book. Details below… When O’Reilly published Web Mapping Illustrated – Using Open Source GIS Toolkits – nothing like it existed on the market. It was a gamble but worked out well in the end.

Primarily focused on MapServer, GDAL/OGR and PostGIS, it is a how-to guide for those building web apps that included maps. That’s right, you couldn’t just use somebody else’s maps all the time – us geographers needed jobs, after all.

To help give you the context of the times, a couple months before the final print date, Google Maps, was released. I blithely added a reference to their site just in case it became popular.

The book is still selling today and though I haven’t reviewed it in a while, I do believe many of the concepts are still as valid as when it was written. In fact, it’s even easier to install and configure the apps now due to packaging and distribution options that didn’t exist back then. Note this was also a year before OSGeo.org’s collaborative efforts started to help popularise the tools further.

In celebration of 10 years of sales I have a couple autographed copies as giveaways to the first two people who don’t mind paying only for the shipping (about USD$8) and who drop me a note expressing their interest.

Additionally, I have some of Gary Sherman’s excellent Geospatial Desktop books as giveaways as well. Same deal, pay actual shipping cost only from my remote hut in northern Canada. Just let me know you’d like one of them and I’ll email you the PayPal details. Sorry, not autographed by Gary, though I was editor and publisher, so could scribble on it for you if desired.

Neo4j Cypher Query for Graph Density Analysis

Graph analysis is all about finding relationships. In this post I show how to compute graph density (a ratio of how well connected relationships in a graph are) using a Cypher query with Neo4j. This is a follow up to the earlier post: SPARQL Query for Graph Density Analysis.

Installing Neo4j Graph Database

In this example we launch Neo4j and enter Cypher commands into the web console…

Neo4j is a java application and requires OracleJDK 7 or OpenJDK 7 to be on your system.
Download the Community Edition of Neo4j. In my case I grabbed the Unix .tar.gz file and unzipped the files. Install on Windows may vary.
From command line and within the newlycreatedneo4j-community folder, start the database:
1. ```
bin/neo4j start
```
Use web console at: http://localhost:7474 – the top line of the page is a window for entering Cypher commands.
Load sample CSV data using a Cypher command – I cover this in a separate post here. Be sure the path to the file is the same on your system, matching where you saved the CSV files to.

Quick Visualization

Using the built-in Neo4j graph browser, you can easily see all your relationships as nodes and edges. For a query, return results that include all the objects:

MATCH (p:Person)
RETURN p

Compute Graph Density

Graph Density requires total number of nodes and total number of relationships/edges. We do them both separately, then pull them together at the end.

Compute the number of unique nodes in the graph

MATCH (p:Person)
RETURN count(p)

count(p)
21
Returned 1 row in 49 ms

This tells us that there are 21 people as Subjects in the graph. (I’m not sure how this differed from the 20 I had in my other post – perhaps part of the header from the CSV came in?)

Therefore, a maximum number of edges between all people would be 21² (we’ll use only 21×20 as a person might not link to themselves in this example).

Compute the number of edges in the graph

Here we only select subjects that are “Person” and only where they have a relationship called “HAS_FRIEND”.

(The “p:”, “r:” and “f:” prefixes act like the “?…” variables references in SPARQL – you set them to whatever you want as a pointer to the data returned from the MATCH statement.)

MATCH (p:Person)-[r:HAS_FRIEND]-(f:Person)
RETURN count(DISTINCT r)

count(DISTINCT r)
57

When we defined the data from CSV, we set up a relationship that I thought would be just one-way. But you can see, if you don’t provide the DISTINCT keyword, that you’ll get double the record counts, so I’m assuming it’s treating the relationships as bi-directional.

Total edges is 57. Do the quick math and see that the ratio is then:

57 / (21 x 20)
= 57 / 420
= 0.136...

Rolling it all together

We can do all this in a single query and get some nice readable results, even though the query looks a bit long. Note that I snuck in an extra “-1” to account for that stray record I didn’t account for.

MATCH (p:Person)-[r:HAS_FRIEND]-(f:Person) 
RETURN count(DISTINCT p) as nrNodes, 
 count(DISTINCT r) as nrEdges,
 count(DISTINCT r)/((count(DISTINCT p)-1) * (count(DISTINCT p) - 1.0)) AS graphDensity

nrNodes nrEdges graphDensity
21      57      0.1425

Challenge: Why did I get different results than in the SPARQL query example?

In the earlier post I had only 20 nodes, but in this one got 21. Can you explain why?

Future Post

What’s your favourite graph analytic? Let me know and I’ll try it out in the future post.

One of things I have planned is to do some further comparisons between graph analytics in SPARQLverse and other product like Neo4j but with large amounts of data instead of these tiny examples. What different results would you expect?

Code snippet: SPARQL Query Graph Density

I’m testing out sharing SPARQL code snippets using Github Gist features. I’ll be adding more as I work through more graph-specific examples using SPARQLverse, but here is my first one:

 SELECT (?nrEdges/(?nrNodes *(?nrNodes - 1.0)) AS ?graphDensity)
 FROM <graphName>
 WHERE {
  { SELECT (COUNT (*) AS ?nrEdges) (COUNT (DISTINCT ?person) AS ?nrNodes)
    WHERE { ?person <knows> ?anotherPerson . }
   }
 }

Ideally we’d have a common landing place for building up a library of these kinds of examples.