Inaugural: Twitter streaming with node.js and HTML5 WebSockets
The idea of having a full javascript stack with code reuse and the removal of language impedance between client and server has always interested me.
It seemed that a full javascript stack was an utopia, waiting in the distant future, but not anymore. Now there are a few powerful javascript solutions for the server and with CouchDB I can aim for a full javascript stack.
One of those solutions is node.js. It’s an high performance non-blocking event driven framework. It is brilliantly designed, it feels right, and the performance is impressive to say the least. I’ve not been so excited with a new technology since Rails appeared on my radar a few years ago.
If you’re unfamiliar with it, go watch Ryan Dahl presentation at JSConf.
WebSockets and Twitter Real Time Streaming
Yesterday I stumbled upon Ruben Fonseca excellent
blog post
on a Ruby Twitter Streaming example using AMQP, EventMachine and HTML5
WebSockets (basically it streams real time tweets to the browser without the use of polling).
Ruben’s experiment was inspired by Ilya Grigorik
post on WebSockets
and Ruby.
I decided to follow on Ruben’s post and do the exact same thing with node.js, only IMO much simpler. Ruben’s architecture was based on the following:
Twitter Stream → Filter → RabbitMQ → AMQP → Eventmachine → WebSocket → Browser
The node.js version:
Twitter Stream → node.js → WebSocket → HTML5 Browser
Let’s take a look at the code:
The Server
var sys = require('sys'),
http = require('http'),
ws = require("./vendor/ws"),
base64 = require('./vendor/base64'),
arrays = require('./vendor/arrays');
// Command line args
var USERNAME = process.ARGV[2];
var PASSWORD = process.ARGV[3];
var KEYWORD = process.ARGV[4] || "iphone";
if (!USERNAME || !PASSWORD)
return sys.puts("Usage: node server.js ");
// Authentication Headers for Twitter
var headers = [];
var auth = base64.encode(USERNAME + ':' + PASSWORD);
headers['Authorization'] = "Basic " + auth;
headers['Host'] = "stream.twitter.com";
var clients = [];
// Connection to Twitter's streaming API
var twitter = http.createClient(80, "stream.twitter.com");
var request = twitter.request("GET", "/1/statuses/filter.json?track=" + KEYWORD, headers);
request.finish(function (response) {
response.setBodyEncoding("utf8");
response.addListener("body", function (chunk) {
// Send response to all connected clients
clients.each(function(c) {
c.send(chunk);
});
});
});
// Websocket TCP server
ws.createServer(function (websocket) {
clients.push(websocket);
websocket.addListener("connect", function (resource) {
// emitted after handshake
sys.debug("connect: " + resource);
}).addListener("close", function () {
// emitted when server or client closes connection
clients.remove(websocket);
sys.debug("close");
});
}).listen(8080);
Notice how everything looks familiar, and at first glance it seems we’re writing typical
client side javascript.
I find it to be very elegant and simple solution, and with a lot less dependencies. The javascript
style callbacks and closures are well suited for an evented non-blocking framework like node.js.
I’m using Jacek Becela very cool minimal web sockets
library that creates the websocket server with a similar interface to the native
tcp.createServer.
The Client
$(document).ready(function(){
if(!("WebSocket" in window)) {
alert("Sorry, the build of your browser does not support WebSockets");
return;
}
ws = new WebSocket("ws://localhost:8080/");
ws.onmessage = function(evt) {
data = eval("(" + evt.data + ")");
var p = $("");
if($('#tweets div.tweet').size() > 15) {
$('#tweets div.tweet:last').slideDown(100, function() {
$(this).remove();
});
}
$('#tweets').prepend(p);
p.slideDown(140);
}
});
Here for comparison sake (and because I’m lazy) I copied Ruben’s client side script.
You can check out the project and installation instructions on the github repo.