HDHomeRun Wire Protocol

HDHomeRun

HDHomeRun is a ATSC/QAM Digital TV Tuner with Ethernet Interface. This device has two different models, one is a single tuner and the other a dual tuner, but the functionality is the same. The is control software available for Windows, Mac and Linux and best of all the software is open source allowing you to embed it in your own solution. This is the easiest way I know of to take the digital video signals from over the air or cable and get them onto your IP network where you can do interesting things with them.

Now that you have your HDHomeRun and established IP freedom for your digital video, what are you going to do next? One easy way to enjoy the newfound freedom is to grab a copy of VLC and your hdhomerun_config utility and start watching you favorite shows. So get your network setup and have at it.

OK, well that’s fun but what you really want is to record your shows and watch them whenever (and wherever) you want. Hmm, this sounds like a good fit for a highly concurrent, fast IO, superb networking language that doesnt quit unless you nuke it from orbit because let’s face it the last season of Lost is really that important. I’m of course talking about Erlang, which is a programming language that was designed at the Ericsson Computer Science Laboratory. Erlang is a functional programming language which is ideal for the development of highly available soft realtime systems.

The output from the HDHomeRun is a single or multiprogram transport stream depending on if the program number is set, that is sent via unicast UDP to the destination specified in the target destination to the tuner. The Video and Audio codecs that are used are dependent on the broadcast source, for ATSC it’s MPEG-2 Video and AC3 (A52) Audio. I’m not going to get into transcoding, so we will just capture the broadcast as is to disk.

For Erlang this is simply spawning a process that opens a udp socket, opens the output file then goes into a loop receiving packets on the socket and writing them out to disk. There is a little more complexity around the timing between the control commands to the HDHomeRun and the listener process, basically something has to be listening on the destination socket before HDHomeRun will start sending packets. The other complexity is that Erlang has a robust process supervisor model, so that if things like your listener process dies it can be restarted, but lets say you put a timer into the listener process that stops the process if it does not receive data for 1000 milliseconds. If the listener process exits because of a timeout, then you probably want to restart the listener then immediately retune the HDHomeRun to get the data flowing again. I’m not going to address the supervisor model either, since my focus is on the control commands to HDHomeRun.

We could issue the control commands from Erlang by calling external scripts or creating a port process, but Erlang has very nice bit syntax and binary matching that make it ideal for network programming. As mentioned above the C code that is available for download provides the complete implementation of the control program, but one of the files hdhomerun_pkt.h covers some details of the actual network protocol. The packet format is described below the first two bytes are the Packet type, the next two are the payload length, then the payload followed by a 32 bit CRC of the entire packet.

/*
  The discover protocol (UDP port 65001) and control protocol (TCP port 65001)
  both use the same packet based format:
   uint16_t    Packet type
   uint16_t    Payload length (bytes)
   uint8_t[]   Payload data (0-n bytes).
   uint32_t    CRC (Ethernet style 32-bit CRC)
*/

The Packet type is one of the following six: Discover Request, Discover Reply, Get/Set Request, Get/Set Reply, Upgrade Request or Upgrade Reply. The Upgrade Request and Reply are used to upgrade the firmware on the HDHomeRun box, since this is a sensitive operation and does not occur during normal operation, I will not try to replicate that with the Erlang code.

#define HDHOMERUN_TYPE_DISCOVER_REQ 0x0002
#define HDHOMERUN_TYPE_DISCOVER_RPY 0x0003
#define HDHOMERUN_TYPE_GETSET_REQ 0x0004
#define HDHOMERUN_TYPE_GETSET_RPY 0x0005
#define HDHOMERUN_TYPE_UPGRADE_REQ 0x0006
#define HDHOMERUN_TYPE_UPGRADE_RPY 0x0007

To be continued…

HTTP Live Streaming

HTTP Live Streaming

HTTP Live Streaming is a standard implemented Apple, to support streaming of video on the iPhone, iPad and iPod touch. Apple has submitted the specification to the IETF as IETF Internet-Draft of the HTTP Live Streaming specification.

HTTP Live Streaming competes with the current incumbent, RTMP, Real-Time Messaging Protocol from Adobe. While recently Adobe has published the standard for RTMP, it has been proprietary for years but had been reverse engineered by several groups including the Red5 server. RTMP requires the Adobe Flash Player on the client, which has a reputation as being buggy on many platforms causing high cpu usage and fast battery drain. Apple has not included support for Adobe Flash on their mobile platforms.

HTML5

The server side of video delivery is only part of the story, the other piece is the client, which until now has primarily been the Adobe Flash Player. Video delivery over the Internet is finally becoming a first class citizen with the introduction of HTML5. HTML5 defines a standard way to embed video in a web page, using a <video> element. Apple Safari, which is the native browser on Apple’s mobile devices, provides Video playback through the HTML5 Video element. This native support allows the browser to optimize the Video Decoder for the platform being used, this can include offloading video decoding to specialized video chips such as Nvidia, reducing the CPU load on the main processor. This should result in full definition 1080P HD content playback right from the browser. There is no way at present for the video to be shown fullscreen, since it’s inside the browser and not running as a plugin like the flash player.

HTTP Live streaming has many advantages over RTMP, the main one being that it’s a pure HTTP solution so common HTTP servers can be used to deliver Video. This also means that streaming is automatically supported by nearly all edge servers, media distributors, caching systems, routers, and firewalls. Many internal networks are behind firewalls and proxies that could interfere with video delivery using different protocols, these devices do not cause a problem with HTTP Live streaming.

HTTP Live streaming has support for multiple bit rate encoding, also know as adaptive streaming. This helps when the throughput of downstream client is not known, so faster clients can get high quality, high bit rate encoding but a slower client can get a lower quality, lower bit rate encoding. Over the open Internet or mobile network this is the only way to reliably deliver a live stream. There is also support for Encryption using AES-128, with key management over HTTPS and session-based authentication.

Encoding

The media encoder takes a real-time signal from an audio-video device, encodes the media in h.264, and encapsulates it for transport. Encoding should be set to a format supported by the client device, such as H.264 video and HE-AAC audio. Currently, the supported delivery format is MPEG-2 Transport Streams for audio-video.

MPEG-2 transport streams should not be confused with MPEG-2 video compression, MPEG-2 transport streams is the container format and is the standard for ATSC broadcast delivery and can be used with a number of different compression formats. Only MPEG-2 transport streams with H.264 video and AAC audio are supported at this time for audio-video content.

The file is then cut into segments using a segmenter program, the size of the segments are determined by the duration and bit rate, a typical duration is 10 seconds. The segmenter also creates an index file containing references to the individual media files. Each time the segmenter completes a new media file, the index file is updated. The index is used to track the availability and location of the media files. The media files are stored as .ts files and index files are saved as .M3U8 files, an extension of the .m3u format used for MP3 playlists.

I think the time is right for a new standard to emerge in the live video streaming space and I believe this is it. With the momentum of the iPhone, Apple carries a lot of weight in this space, and developers want easy standards based solutions, HTTP Live streaming wins on both accounts. RTMP had it’s day, but it’s hard to implement and was really designed for a different era, one in which multiplexing information across one socket was needed. This is not the case anymore, even inside a browser window there can be many events going on, with Javascript coordinating the information flow. HTML5 provides a general purpose websocket interface which defines a full-duplex communications channel that operates over a single socket and is exposed via a JavaScript interface. Between these HTML5 Video and HTML5 websocket the functionality that RTMP provided is no longer needed,

There are still issues to overcome, for one this is an Apple standard and is not supported by other browsers except Safari. There are issues in general around the Video player in HTML5 because different browsers support different encoding formats and containers. While h.264 is recognized as a great encoding format, there are licensing issues around the technology so open technologies like Theora video, which is free, were introduced. There are other formats like Dirac for video encoding and Matroska-MKV as a container, that are gaining ground. In the end once a Open Source, license free video encoding and container format are agreed upon the industry will move towards that. The licensing corporation around h.264 may realize that and come up with a solution, but they have a built in monopoly with OTA and other broadcast uses.

Document Oriented Database Schema Design

Interest in NoSQL is on the rise and for good reason, traditional RDBMS systems have many strengths like ACID guarantees but they also have many weaknesses. The most associated trait of the NoSQL movement is the ability to scale writes and to a lesser extent reads (master/slave replication on RDBMS can handle high read requirements to a large extent). One of the often overlooked aspects in NoSQL is how well data maps into a non-relational store, but that is what I want to cover today.

In relational database design Database normalization is the recognized approach for schema design. A crude synopsis of Database normalization, is don’t replicate data. For example if you were designing a names and addresses, a fully normalized design would look something like this:

This will give you the data consistency you are looking for, so when Topeka changes it’s name to Google you are only updating one record. In the real world cities and states do not change their names very often, and a 4 table join to get the full contact information would have performance problems, so a selective Denormalization step would occur. The resulting database schema would look like this:

With a Document Oriented Database like MongoDB the Denormalization process would even go further and the address records would get embedded into the Person, resulting in one document. In order to show this concept more clearly, I will use the mongoid gem for ruby to create a Person and Address class. I then setup the associations using the has_many and belongs_to macros.

require 'rubygems'
require 'mongoid'

Mongoid.configure do |config|
  name = 'contacts'
  host = '192.168.1.19'
  config.master = Mongo::Connection.new(host, 27017).db(name)

end

class Person
  include Mongoid::Document
  field :first_name
  field :last_name
  has_many :addresses
end

class Address
  include Mongoid::Document
  field :street_number
  field :street_name1
  field :street_name2
  field :city
  field :state
  belongs_to :person, :inverse_of => :addresses
end

person = Person.new(:first_name => "Walt", :last_name => "Disney")
person.addresses << Address.new(:street_name1 => 'P.O. Box 10000',:city => 'Lake Buena Vista', :state => 'FL')
person.addresses << Address.new(:street_name1 => 'P.O. Box 10040',:city => 'Lake Buena Vista', :state => 'FL')
person.save

Once you run this program the resulting document is shown below.

  { "_id" : "4bb333c44954b30ff6000001", "first_name" : "Walt", "last_name" : "Disney", "_type" : "Person", "addresses" : [
    {
        "street_name1" : "P.O. Box 10000",
        "city" : "Lake Buena Vista",
        "state" : "FL",
        "_id" : "4bb333c44954b30ff6000002",
        "_type" : "Address"
    },
    {
        "street_name1" : "P.O. Box 10040",
        "city" : "Lake Buena Vista",
        "state" : "FL",
        "_id" : "4bb333c44954b30ff6000003",
        "_type" : "Address"
    }
  ] }

In many ways this is the feature of a NoSQL database that is most appreciated, the ability to more accurately represent real world data relationships in a natural manner. I would recommend you look at MongoDB, if you have similar data models.

Solaris Truss

In Sun Solaris there is an incredibly useful command for tracking down low level issues, the command is called truss. truss allows you to trace system calls and signals, this means you can watch a compiled program including the Java JVM while it makes system calls.

For example while trouble shooting a JReports reporting issue, all I had to go on was this exception message.

I can tell based on the stack trace that it was trying to make a socket connection that failed, but unfortunately that is not a lot to go on. So I used the truss command with the following options, -t tells it which system calls to follow (I was only interested in connect since that is whats failing), the -v and -x allows me to see all the arguments getting passed to these system calls. The -p tell it which pid to watch and 4079 happened to be the pid of the Glassfish instance running JReports

#truss -t connect -v connect -x connect -fp 4079
4079/85485:     connect(367, 0xA977CDC8, 16, SOV_DEFAULT)       Err#146 ECONNREFUSED
4079/85485:             AF_INET  name = 127.0.0.1  port = 80

This tells me that JReports was trying to connect to localhost(127.0.0.1) on port 80, which is a problem since I have the reports server bound to a specific interface not all interfaces. Now I at least know the problem and can work on a solution thanks to truss.

If you are using Linux then check out strace.

Ruby Book I would like to see

I follow Ruby Best Practices on twitter and he asked for feedback on Ruby in the Enterprise Book he was thinking of writing. I really like the concept, but gave him the following feedback.

Where would the target audience becoming from?

I assume primarily Java or .NET I'm not qualified for .NET, but most the questions I get when you mention using Ruby (primarily Rails) instead of Java (J2EE) is they equate that to the View Layer, not the model and controller part. So I would pick a sample application that would really show the entire stack.

Here would be some ideas:

Large Retailer: System allows customers to be notified when product X is back in stock at their local store. The Ruby system must be able to talk back with Mainframe for order shipments and a J2EE application for pricing and customer records. In this case the only model is the “in stock request” the login information for the customer may or may not reside in the Ruby system

Airline: System to send SMS messages about Flight Delays based on Frequent Flyer information. Assume the airlines current Web interface will be where the customer enables this, so there is no user focused View layer instead it’s primarily a web interface that gets toggled for a particular customer. The phone number is stored in the Ruby Model, but the rest of the information about a customer is in Mainframe for flight information and database (Terradata) for other Frequent Flyer information.

Electric Power Company: System to dispatch work crews based on trouble calls. System should provide Web interface, SMS interface and integrate with Voice Response system to allow customers to report electric outage, (using phone number, customer id, street address, ect). System integrates with Mainframe for billing information and proprietary UNIX application for power line layout (GIS) information. The power line layout application is used to pinpoint problem location based on aggregate trouble reports.

Gas Company: System to monitor gas pressure across footprint based on network of sensors using cellular communications as the backchannel. Sensors are used to detect leaks. System communicates with proprietary VAX application to close certain vales in the event of leak.

These are just samples to show a real world integration effort between the various types of systems that a “Enterprise Company” would have. The segmentation of data is also important, when do you need to replicate and when can you just use the other systems data. How does data change notification work (triggers, polling, etc)

I think the other big topic would be how do you deploy a Ruby application Network, Hardware, etc.

Focus would be:

• Linux/UNIX
• HAProxy, Keepalived (or other VRRP solution) for HA
• Nginx or Apache and Mongrel, Phusion, Unicorn, etc
• Capistrano Chef or Puppet
• JRuby/Glassfish if you want to go there
• DB: Mysql, Postgres, Oracle, etc (possibly include a Key/Value store like MongoDB or CouchDB if the problem domain fits)

The thing here is that most "enterprise" developers are not concerned with the deployment environment because there is a DBA Group, a Sys Admin Group, possibly a Hardware Group, A network group, etc that handles the deployment. The developer produces a WAR file and a set of instructions, the rest is done by someone else. To pitch a Ruby project they are going to have to be more knowledgeable on a typical deployment architecture.

One of the things I have run into as a developer is that Ruby still does not have the depth of libraries of Python and I have to go back to using python for some stuff. Notable examples are Telcos which use Corba and Sun RPC (ONC-RPC) in their network, both of these are weak or non-existent in Ruby but available in Python (or Java)

One other thing that may be nice is to highlight good uses of Ruby in the Enterprise vs poor choices, although the lines get blurry

Rails Model Annotation

One of the things I like about datamapper over activerecord is that all the attributes of the model are listed in the model itself. I found something today that can help to bridge that gap in a small way.

Have a look at annotate-models you can install it as a gem

gem install annotate-models

Then go into your project directory and run annotate, you will then have something like this at the top of each model.

# == Schema Information
# Schema version: 20090130210328
#
# Table name: profiles
#
#  id               :integer(4)      not null, primary key
#  user_id          :integer(4)
#  first_name       :string(255)
#  last_name        :string(255)
#  website          :string(255)
#  blog             :string(255)
#  flickr           :string(255)
#  about_me         :text
#  location         :string(255)
#  is_active        :boolean(1)
#  last_activity_at :datetime
#  created_at       :datetime
#  updated_at       :datetime
#

Github from behind a firewall

If you are like me you are slowly using github more and more, as some of the top upcoming frameworks like Merb and Rails are being hosted there.

If you are behind a firewall at work, you might not be able to clone a repository because the port used for the git protocol is 9418, so issuing a “git clone” command will timeout. You can change the protocol to http, so instead of git clone git://github.com/path/to/repo you would do git clone http://github.com/path/to/rep, sounds easy enough but it’s slower when it works and many times it will not work. I have gotten output like this:

walk 2707ebbbc5837619dcc51af43db18ac31380c4a5
walk 743917a6de6464dd2fa5e064f1881adba218cae6
walk 2a348104ddb2c1b2090a8ca80d21258846afd250
walk 8a9cbc1af1343bfd32ea4b473341dff364b67353
walk 1bc43720343856280ad6801562f75b3239ca4ce2
walk e9ea1557e703f2f238192d72bd2714b0af425b7c
walk fbc886b9adecad2e6e9c9a13f67349cd21c01ffa
walk c4dc40df63fc5cebd2f3b2cceb7c8ea20ac86776
walk 64db048595a07b70057b98731d6e957406bca913
walk 012e0629863b3ce4b98c7eebb3f5ef4ed6d7b1e8
walk 109ec6b85d66f959a83048af9e173f612b1d4009
Getting alternates list for http://github.com/sam/extlib.git
Getting pack list for http://github.com/sam/extlib.git
error: Unable to find abb467803fa74dfdc3740f31b4fde5544e6a4100 under http://github.com/sam/extlib.git
Cannot obtain needed tree abb467803fa74dfdc3740f31b4fde5544e6a4100
while processing commit 109ec6b85d66f959a83048af9e173f612b1d4009.
fatal: Fetch failed.

I assume this is fetching each commit and processing it, but has problems

SSH Tunnel Solution

The solution is pretty simple if you have ssh access outbound from work to a machine that is not fire walled. You would setup a ssh tunnel with the following command ssh -L 9418:github.com:9418 (username)@(machine.domain) Then your clone command would be changed to git clone git://localhost/path/to/repo

Ruby on Rails content_for

One of the mantras in the Rails community is DRY (Don’t repeat Yourself), this is nothing new if your a software developer you should always look for ways to refactor your code to avoid copy and paste. Trying to maintain code with logic that has been copied and pasted to different locations is a nightmare, and one that unfortunately most of us have been exposed too.

In Rails the view part of MVC is handled by ActionView and the default templates are ERb. The views are further broken down into layouts and views. Layouts are called first for each controller then the view corresponding to the action. For the most part the layout is going to contain header elements for the xml, but what if you need some special javascript in one of you actions but not others.

You could just add it to the layout and say tough you are going to have the client load it no matter what, besides taking up extra bandwidth this can cause javascript errors for things like Google Maps where there is no associated div.

Rails gives you a better way though, you can use content_for to specify content associated with a particular yield command. For example if you wanted to use YM4R (Yellow maps 4 Ruby) to helps with your Google Maps mashup project, you could do something like this:

You would add a yield item to your layout.

<head>
  <%= stylesheet_link_tag :all %>
  <%= javascript_include_tag :defaults %> 
  <%= javascript_include_tag "popup" %> 
  <%= yield :head %>
</head>

Then in you view for the action you want to use the Google Map you would do something like this:

  <% content_for :head do %>
    <%= GMap.header %>
    <%= @map.to_html %> 
  <% end %>

In your controller you need something that actually sets up your map, for example…

 @map = GMap.new("map_div")
 @map.control_init(:large_map => true,:map_type => true)
 @map.center_zoom_init([75.5,-42.56],4)
 @map.overlay_init(GMarker.new([75.6,-42.467],:title => "Hello", :info_window => "Info! Info!"))

That’s it, your layout stays DRY and your single view gets the Google Map.