Pivotal Blabs

In a previous post I discussed how to unit-test client-side AJAX JavaScript code using JsUnit. In that post I dealt with raw XmlHttpRequests. What if you're using a library such as Prototype, which provides a layer between your JavaScript and the raw request? Let's look at how you can hook into Prototype's AJAX functionality in order to mock out the raw request.

In Prototype, there's an object called Ajax that has a method called getTransport, which returns an XmlHttpRequest. In our test page, we can replace getTransport with a different implementation:

Ajax.getTransport = createMockTransport;

var request;
function createMockTransport() {
  request = new MockXmlHttpRequest();
  return request;
}

That's basically it: we now have a mock request that looks to Prototype like a real one. There are a couple of gotchas, though.

setRequestHeader

Protoype calls setRequestHeader on the XmlHttpRequest. The latest version of JsUnit's MockXmlHttpRequest mocks out setRequestHeader, but some earlier versions did not. If you're using a version earlier than 2.2alpha22, you'll need to add this method to your mock request:

request.setRequestHeader = function(label, value) {
}

(You can hold on to the request headers, of course, if you want to test them.)

abort

If your request ever gets aborted by your code, Prototype may encounter issues. To simulate an abort in your mock request that makes Prototype happy, try adding the following:

request.aborted = false;

request.abort = function() {
  this.aborted = true;
  this.readyState = 4;
  this.onreadystatechange();
  this.responseIsSuccess = function() {
    throw new Error("simulating crash when attempting to access request.transport.status");
  }
}

By Steve Conover and Brian Takita

Peer-to-Patent, one of Pivotal Labs' clients, got Slashdotted last week, and we had no trouble handling the load. The site was just as responsive as it always is, and we didn't come close to having a scale problem.

Moral of the story: the technology for serving static web pages is old, boring, and extremely scalable. If you have the type of site that can be page-cached, do so aggressively, starting with the front page and any pages likely to be linked to. We got a huge payoff for the engineering time that we invested in our page-caching strategy.

Highlights:

• We moved away from Rails page-caching and developed our own "holeless cache", which uses a symlink trick (see below) to instantly and "holelessly" switch to a new version of a cached page. (The cache "hole" is the time between the expiration or purge of a cached page and the time when it's regenerated. The danger is that in that time your Mongrels can be saturated with requests - something we proved to ourselves could easily happen.)

• Here's our symlink trick, using the front page as an example:

  1. Have index.html point to index.html.current
  2. If (index.html.current is >= 20 minutes old)
     1. Copy index.html.current to index.html.old
     2. Point index.html to index.html.old
     3. Rewrite index.html.current by asking Rails for the page (using the process method)
     4. Repoint index.html back at index.html.current
  3. Repeat step 2 every minute using a cron job.

• For cache expiration that's model-based, we make a call from the model observer class to our holeless cache routine, instead of using Rails cache sweepers. So, instead of just deleting the cached page we regenerate it in place.

• It was important to write tests that proved that the HTML we generated for cached pages looked exactly the same in different "modes" (user logged in vs not, for example). This forced us to push modal decision logic out of Markaby templates and into JavaScript, meaning that view-oriented Rspec tests asserting modal differences became useless. We rewrote them as Selenium tests.

• Performance/load testing: we tried several tools and approaches and found that a simple Ruby script that launches wget requests (that write to /dev/null) in many separate threads worked best for us.

• We send down exactly one .js and one .css file. If you are sending down more than one of each of these to the browser, you have a performance problem. Fix it with asset packager.

Update: one clarification about the cron job: we deploy this "automatically" using capistrano.

Interesting Things

• As mentioned in the 06/19/207 post, Rspec 1.0.6 fixed the Controller-exception-swallowing problem, but now you can't call flash.stub!(:foo) -- this throws an error.
• Check out Google Website Optimizer, which "helps online marketers increase visitor conversion rates and overall visitor satisfaction by continually testing different combinations of site content (text and images)."
• After some research we found that you can cache static web pages in memcache, at least with the Nginx web server, making this a very interesting component of a page caching strategy.

Interesting Things

• RSpec gotcha: Controller errors do not automatically "re-raise" to cause specs to fail in v1.0.5. This is fixed in v v1.0.6.

Interesting Things

• "Do long file uploads lock up Mongrel processes?" Evidence seems to show that, at least with some web servers like nginx, the web server will handle all of the uploading, leaving the Mongrel processes to continue serving. Perhaps Apache does the same?

Ask for Help

• "What is the preferred means for running Ruby/Rails processes on a scheduled basis?" People suggested the following:
  • cron/crontab -- our old friend.
  • drb -- Distributed Ruby
  • Ruby Daemon Module

Interesting Things

• Rails has a handy convention/method-hook for turning an object into an URL parameter: implement to_param. ActiveRecord already does this by having to_param return the result of id.

Commonly the JavaScript side of AJAX ends up untested, tightly coupled to the server-side code, and difficult to read. In a previous post, we saw how testability led to the ability to refactor our code to make it readable. This time we'll focus on ways to test AJAX (which is a fairly lengthy topic in itself) and take it as read that once our code is tested we'll be in a good position to refactor towards readability and to generally tame the complexities of client-side AJAX code.

Example

Suppose we have a very simple calculator application: the user enters a number into a textfield and presses a button labeled "Double". The application sends the number to the server, and the server sends back a response containing twice the value of the number. The doubled value is then displayed in a second textfield. Of course, this is an artificial example, but the structure of the code is common enough to demonstrate practices for testing AJAX. Let's use raw XmlHttpRequests here; I'll post again separately with how to adapt this if you're using prototype.js. Our code might look like this:

<script language="javascript">
var request;
function calculate() {
  var enteredValue = document.entryfield.value;
  if (window.XMLHttpRequest)
    request = new XMLHttpRequest();
  else
    request = new ActiveXObject("Microsoft.XMLHTTP");
  request.onreadystatechange = callback;
  request.open("GET", "/calculate?value=" + enteredValue, true);
  request.send(null);
}

function callback() {
  if (request.readyState == 4) {
    if (request.status != 200) {
      document.outputfield.value = "Error: " + request.status;
      return;
    }
    var responseValue = request.responseText;
    document.outputfield.value = responseValue;
  }
}

</script>

<input type="text" name="entryfield">
<input type="button" onclick="calculate()" value="Double">
<input type="text" name="outputfield">

When the user presses the "Double" button, an AJAX request is sent to a servlet, passing the value the user entered in the "entryField" text field. When the server responds, the text in the response is displayed in the "outputField" text field. If the server didn't respond successfully, we display an error message in "outputField" containing the error code.

Starting a Test Page

Let's try to write a Test Page for our code:

<script language="javascript" src="/path/to/calculate.js"></script>
<script language="javascript">
function testClickCalculate() {
  document.entryfield.value = "5";
  calculate(); //but wait - we don't have a server
  //now what?
}
</script>

<input type="text" name="entryfield">
<input type="text" name="outputfield">
</body>

Hmm - how do we proceed? As things are, when we run our test, our request will get sent off to a server that isn't running. We don't have enough control over our environment to continue with the test.

So, how can we test AJAX?

The most important thing to bear in mind is that we are trying to write unit tests for our JavaScript - tests that exercise just a unit of our JavaScript code at a time. So we certainly don't want to bring a server into the picture - that's way out of scope for our JavaScript unit tests. Instead, what we want to do is insulate ourselves from the machinery of the request/response server interaction. The typical point at which we set up our insulation is at the level of the XmlHttpRequest: rather than a real request, we will use a mock version in our test - a pretend version of the request that we control.

Testing the request

Our first task, then, is to set things up so that our test uses a mock request and our code uses a real one:

function calculate() {
  ...
  request = createRequest();
  ...
}

function createRequest() {
  if (window.XMLHttpRequest)
    return new XMLHttpRequest();
  else
    return new ActiveXObject("Microsoft.XMLHTTP");
}

but in our Test Page, we implement createRequest differently. JsUnit comes with a library called jsUnitAjax.js, which contains a mock implementation of XmlHttpRequest.

<script language="javascript" src="/path/to/jsunit/lib/jsUnitAjax.js"></script>

function createRequest() {
  return new MockXmlHttpRequest();
}

Our Test Page's implementation overrides the real implementation of createRequest. Good: now we are able to call calculate without worrying about a real request getting sent to the server. Let's go back and continue with our test.

function testClickCalculate() {
  document.entryfield.value = "5";
  calculate();
  assertEquals("GET", request.method);
  assertEquals("/servlet?value=5", request.url);
  assertTrue(request.isAsync);
  assertTrue(request.sendCalled);
  assertNull(request.data);
  assertEquals(callback, request.onreadystatechange);
}

Notice that we're testing that calling calculate() sends the request, and how the request gets set up - its method, its URL, etc - by examining the mock request. We also verify that the correct callback method has been set up for when the server responds. We aren't using a real server, and we don't care (in this test) about a response.

Testing the response

So, that's half the story. How about testing the response? We need to simulate the server responding to the request. We've already tested that the correct callback is set up; let's take advantage of that now.

function testValidServerResponse() {
  request = new MockXmlHttpRequest();
  request.readyState = 1;
  callback();
  assertEquals("", document.outputfield.value);

  request.readyState = 2;
  callback();
  assertEquals("", document.outputfield.value);

  request.readyState = 3;
  callback();
  assertEquals("", document.outputfield.value);

  request.readyState = 4;
  request.status = 200;
  request.responseText = "50";
  callback();
  assertEquals("50", document.outputfield.value);
}

We go through each readyState, ensuring that nothing happens until state 4. For state 4, we give the mock request a status of 200 and a responseText of "50", and then verify that calling callback() populates the outputfield correctly. Notice how we've tested just the response logic, without using a real server and without the need to set up a meaningful request.

Simulating a server-side error

We have one more test - we need to test what happens when the server responds unsuccessfully:

function testInvalidServerResponse() {
  request = new MockXmlHttpRequest();
  request.readyState = 4;
  request.status = 500;
  callback();
  assertEquals("Error: 500", document.outputfield.value);
}

This time we give the mock request a status of 500, and we verify that the output field contains the expected error message.

Interesting Things

• Brown-bag lunch topic today: "Smalltalk is dead -- long live Smalltalk!"

  Ask for Help

• "How can we get a drop-down list/select box to change our browser's URL? We want people to see forum posts from the 'Last 7 Days', so the URL should change when they choose that from the select box." One option is to wrap it in a form and use the onChange() JavaScript event hook to submit the form:

<code>
<% form_tag(:controller => forum, :action => :filter) do %>
  <%= select_tag('recent', options_for_select(filters]), {:onchange => "this.form.submit();"}) %>
<% end %>
</code>

IntelliJ IDEA has a great feature: if you hit control-shift-V you see a list of the ten most recent selections you cut or copied onto the clipboard. Here are two ways to get the same thing on all Mac OS X apps.

Quicksilver's "Clipboard" and "Shelf" plugins

• Video demo of the QuickSilver clipboard
• Using the Shelf module

Bottom line:

1. In QS preferences, go to (top menu) Plugins / (left menu) AllPlugins
2. Check the 'Clipboard Module' and the 'Shelf Module' so that they get installed
3. Bounce QS
4. Go back into QS preferences and go to (top menu) Preferences / (left menu) Clipboard to tweak your clipboard size and behavior
5. Now copy some text from some app
6. Now hit Command-Space, then immediately afterwards once QS comes up Command-L to see the Clipboard History window pop up for you.

I think the Shelf module lets you store clips permanently, but I haven't figured out how to use it yet.

JumpCut

• JumpCut project at SourceForge

A scissors icon will appear in your menu bar. Whenever you cut or copy a text item, it'll be added to that menu. Clippings can also be accessed by a hotkey (default is Control-Option-V.) A little window like the one you see when using the application switcher or the brightness controls will appear. While holding the modifier keys , use the arrow keys to scroll through the stack.

Interesting Things

• Rails gotcha: Date.parse and Time.parse do not react the same when passed data that can't be parsed:

<code>
$ script/console
Loading development environment.
>> Time.parse "monkey"
=> Thu Jun 14 08:43:10 PDT 2007
>> Date.parse "monkey"
ArgumentError: 3 elements of civil date are necessary
        from /usr/local/lib/ruby/1.8/date.rb:650:in `new_with_hash'
        from /usr/local/lib/ruby/1.8/date.rb:695:in `parse'
        from (irb):2
>>
</code>

• Rails gotcha #2: in functional tests, assert_redirected_to can fail if you are not consistent in your "strings vs symbols" notation. Example:

<code>
class MonkeyWranglersController < ActionController
    def index
            redirect_to :controller "monkey_wranglers", :action => :list
    end
end
</code>

Now, the assertions:

<code>
get :index
assert_redirected_to :controller => "monkey_wranglers", :action => :list # passes
###
get :index
assert_redirected_to :controller => "monkey_wranglers", :action => "list" # fails
</code>

• More View helpers: ActionView::Helpers::TextHelper::cycle helps you with the "every-other-row-should-be-a-different-color" pattern usually seen in tabular data.