Node.js Experiments

So I played with Node.js for a couple of months, and we even deployed a widget for MIT's OpenCourseWare at OpenStudy with it. It was a remarkably fun experience, not the least thanks to CoffeeScript, which is a fabulously awesome syntax layer over JavaScript that gets rid of many of the lamest aspects of JS sytnax (for my money, the biggest change is the ability to abbreviate function() to ->).

But, ultimately, we ran into a few issues. The biggest one was that our real-time push strategy of choice, socket.io, ended up behaving rather poorly under the kinds of load characteristics we subjected it to. MIT's pages see thousands of hits a day (that's not ginormous, by the way -- about 3 hits per minute) and users tend to stay for a few minutes.

The first issue is that we park our site behind nginx, and nginx does not speak HTTP 1.0 when proxying. The result is that WebSocket connections can't be proxied through nginx (indeed, WebSocket proxy support is currently less than spectacular, and WebSocket has since proven to have some security issues). Nothing to worry about, socket.io falls back on alternate mechanisms on the (many) browsers that are missing WebSocket support, so we should still have been okay. Moreover, we were okay for this experiment with removing ourselves from being behind nginx.

The next issue is that the static serving aspect of Node.js is not quite fleshed out yet. We were using express and connect, which come with a middleware to gzip outgoing requests. Unfortunately, it has serious performance issues, so using it at any scale is not really a good idea. Still no problems, for the purposes of our experiment, we were willing to disable gzip.

Finally, we found ourselves in a situation where Node.js would randomly quit on us with timeout errors (this appears to be a timeout firing without an associated handler or some such). After some investigation, it appeared that this was a known issue that was difficult to fix. We tried to apply some fixes to socket.io to handle this, but we were still getting the issues. Given chronic crashing, then, we couldn't keep up the experiment. We even tried switching back from node 0.3 (which may not have been supported by socket.io) to node 0.2, but this led to strange CPU spikes we didn't really have time to experiment with.

Ultimately, we turned our attention back to Lift and the improvements made to it since the 2.0 release. Between the designer-friendly views, CSS selector binding, and the inclusion of lift-mongodb, Lift is still treating us super-nicely. The compilation wait is still annoying, but in proper style I dealt with it by getting a quad-core Core i7 iMac, which tears through it a bit faster.

This shouldn't be taken to mean that I'm saying Node.js isn't ready for primetime. Obviously there are plenty of people employing it in production and loving every minute of it. It was fantastic to be able to code in JS both server- and client-side, and going back to a dynamic language was fun and exciting after a while in the statically typed world. Then again, going back to Lift after Node was also kind of cool, so I guess it's just change that I'm a fan of.

For the most part, Node treated us well. With CoffeeScript, the syntax was awesome, and the developers who are working on Node and its libraries are the kinds of passionate, fast-paced people you expect to find around surging technology. It just didn't happen to work out for us. But! All was not wasted. I did throw together a quick library for doing something very similar to Lift's view-first, CSS-selector based transformations in Node, so look for that over the next couple of days.

ADDENDUM: The timeout issues we faced with node.js are being tracked at https://github.com/joyent/node/issues/378 , where it's been mentioned that they may be gone with Node 0.4.0.

Errors and callbacks in Node.js's node-mongodb-native

node-mongodb-native has the usual error handling mechanism of Node.js: we pass in a callback to handle results, and the first parameter is an error that will be null if no error occurred. For example:

Most such structures in Node seem to execute the callback outside of an error context. However, in node-mongodb-native, the toArray and nextObject callbacks (and by extension a couple of others) are wrapped in a try-catch that triggers the error callback. What does this mean? If your callback throws an exception, the same callback will be invoked with the error parameter set. Thus your callback may be invoked twice: once with successful data, and once with an error corresponding to the exception thrown within your own callback.

So if you're seeing some strange double-invocations of your callback, that's probably what's cracking.

UPDATE: See issue 81 on the node-mongodb-driver github repository to track the resolution of this issue.

OpenStudy's Latest Features

We pushed several new features to OpenStudy today, and I wanted to give a brief rundown of a couple of them really quick.

Public Access

First off is what we call `public access'. Almost all content on the site is now accessible without logging on to OpenStudy. For a few months now, since we first started our beta, OpenStudy has been a login-only site, where login is required to view any of the study groups, studypads, or users on the site. When we were in closed beta, this made sense, and in the run up to our open beta we had a ton of other features we wanted to get in for our users to play with. Now, it's time to throw the doors open even further.

When you aren't logged in, you will see a banner on every page:

You can interact with the page as usual, or click any of these buttons to log in, sign up, or connect with facebook. Moreover, if you want to join a study group or follow a studypad or user, you can go ahead and click the appropriate buttons and you will be prompted to log in before the action is taken.

Facebook Like and Tweet Buttons

With public access also come facebook like and tweet buttons. Now, if you are on a study group or studypad, you can like it on facebook or tweet it to your followers. Just look for the two buttons in the sidebar!

Everyone who sees the links will be able to get to the page and have a look around before deciding whether they want to sign up or not!

Help

We've been working for a while on some great help videos to explain some of the features of OpenStudy and how they can be used. Starting today, you'll find a link to the help videos in the OpenStudy header on any page:

Click on the link to view our 6 videos about OpenStudy (with full HTML5 video support, thanks to Vimeo).

We're always working on cool new stuff, so keep an eye out for the next batch of new features!

A Passing Note About Facebook Like Buttons

In Facebook's documentation of the like button, they mention that the like button tag needs an href attribute that specifies the URL you are liking. A little further down, in the section on Open Graph tags, when discussing the og:url Open Graph tag, it says:

og:url - The canonical, permanent URL of the page representing the entity. When you use Open Graph tags, the Like button posts a link to the og:url instead of the URL in the Like button code.

Now, the problem here is that the iframe version of the like button does not, in fact, post a link to the og:url—it still requires the href attribute¹. If you have the href attribute set correctly, after someone likes the page, Facebook will crawl it and properly import your Open Graph tags; however, the href attribute on the iframe's URL must be correct for this to work.

I spent a while banging my head against the wall wondering why the like button wasn't working before I figured this one out. The symptoms the like button shows are flashing briefly to a `1 person' like message and then immediately reverting to the 0 people `Like' button. Also note that the way to debug this is to pull up Firebug or the WebKit Inspector and watch the AJAX that goes back in forth. Look for the response to the facebook AJAX call and check out the body to see what error message you received.

¹ – I'm not sure whether or not the XFBML version of the button correctly uses the og:url tag or not, as I didn't try using it.

GA Amendment 1: If All You Did Was Post About It, It's Your Fault

I blame myself for amendment 1 passing. What I did to fight it was post on facebook and tweet/retweet on Twitter. 90-95% of the people there *already knew about it* and *already disagreed with it*. What should have happened was a real movement to prevent it from passing, on the order of magnitude of the movements we create to elect officials we want in office.

Amendment 1 was worded in a way that introduced bias, and the movement to evade that perception was lackluster and based on exactly the types of media where people were already likely to be opposed or not to vote at all.

There was no ground game, there was no organized phone opposition. The facebook page just put out blog posts. Excellent, since I agreed enough to like the page, I must need more convincing. Maybe it was to forward by email to your friends, to whom you've probably already spoken to at length. Or maybe it was so you could repost it on facebook yourself, to reach all your friends who may not even have seen it for the noise in their feed.

Obviously, it didn't work. Current numbers have Amendment 1 passing by 67.5%, which in any election would be a landslide. I suspect the 32.5% of people who voted against Amendment 1 were the members of the facebook page, and the voters who are actually on Twitter.

So let this be a lesson to us all. Next time there's an amendment that we are as vehemently opposed to as we seem to have been to this one, let's actually do what you do when you want a vote to go your way, instead of messing around like college kids trying to tell their friends what their date was like or hackers trying to tell everyone about their favorite new programming language.

Pet projects, Node.js, and Scala

My post yesterday implied that all I'd be blogging about would be OpenStudy and what we're doing. This isn't entirely true, as I'll be trying to blog about the cool technology stuff I get around to playing with in my spare time. Next on the radar is definitely the ever-increasing hotness of Node.js -- if only out of curiosity. If nothing else, it feels like a nice homecoming back to some of the Ruby world's approaches to things, which is rather refreshing.

The one thing I've gotten tired of when using Scala is the compile times. It's entirely possible that as developers we ultimately gain time by having a compiler that will help us avoid type errors, and Scala's type system is beautiful and extremely powerful. For comprehensions are some of the nicest syntactic sugar I've seen in a while, and the layers that Lift's Box class adds on top of them are even nicer.

That said, I can't help but feel like the compile times slow me down. Even though this may be all perception, and I may actually develop effectively faster, the mind gets discouraged when it thinks it's getting less done, not just when it's actually getting less done. In addition, the pain of recompiling every time a change happens makes even continuous iterative testing feel like a slowdown. No longer can you just code and wait one second for your test to run -- you have to change gears after waiting for your code, wait for the compile, and then see the results. This seriously cramps the iterative test-driven process, since adding a test, waiting for it to fail, and then adding the code to make it pass becomes a much longer cycle.

The Node.js experiment is to find out about some of the various apparent positives it offers, including the ability to use Javascript on both client and server. There are a lot of goodies in Lift and in Scala that are enticing, however, so it'll be a tough fight.

Time for some new life

It's been an incredibly busy few months since I last posted, but it's time I got back into the swing of things. As the Chief Software Engineer at OpenStudy, I've been hacking day in and day out to get OpenStudy closer and closer to our vision for the future of collaborative studying online. One thing we've been a bit short on, unfortunately, is blog posts explaining what we're up to behind the scenes.

Recently, our designer and user experience architect, Siddharth Gupta, started blogging it up at http://siddharthgupta.net/, posting cool tidbits of upcoming designs and ideas that improve his workflow day-to-day. This blog, then, is meant to be the face of the developer side of OpenStudy, with summaries of new features when we deploy new stuff, as well as posts about what's coming up and teasers regarding what we're working on. I'm hoping this will get people a little more interested in and excited about the stuff that's coming and how we're improving OpenStudy every day to make it better.

In certain cases, blog posts may appear both here and on the main OpenStudy blog at http://blog.openstudy.com/ . I've got some posts cooking quietly that should be coming out in the next couple of days, so come back soon!