Mild-Mannered Canadian Fury

Doug Stephen is Politely Peeved

Arbitrary Dynamic Routing in Sinatra for Crazy People; or, How to solve problems that nobody else has


Thu, 04 Oct 2012 Ā«permalinkĀ»

So I haven’t talked about it in a while, but some of you might remember that one of my many side-projects involves Arduinos and home networks. The project itself is related to home automation. I’m lucky in that I get the chance to expand on the project through classes I’m taking at the university; specifically, some Natural Language Processing researching I’m doing for an Artificial Intelligence project. This doesn’t have to do with NLP, though. But bare with me. I think it’s something pretty cool, if not a little hacked together.

The project is intended to be very modular; the interfaces can be many and arbitrary, not strictly Natural Language. The assorted interfaces only need to conform to an API that lives in a central brain which facilitates communication with different nodes in the network. You can think of each node as an individual Arduino, for now. Currently, I’m exposing the API to the brain over HTTP, and I’m using Sinatra to make prototyping the endpoints very quick.

One of the key features that I wanted was the ability to create and remove nodes from the network without having to restart the Sinatra web server process; but Sinatra doesn’t seem to have an immediately obvious way to create and remove routes on the fly. Additionally, because these routes need to be flexible and possibly even re-definable, the challenge was even hairier. I basically needed a way to dynamically load and unload Ruby code inside of a single process without reinvoking the interpreter. This is a little discussion on how I went about solving this. It’s the text-book definition of a hack, and there are probably more robust ways to go about doing what it is that I want to do.

Arbitrary, Dynamic Routes. Jesus.

If you aren’t familiar with Sinatra, it’s a Domain Specific Language built on top of Ruby that allows for the rapid creation of simple HTTP handlers. It allows you to define responses to HTTP methods like GET, POST, etc. very easily with a simple syntax. The Sinatra “Hello World” looks something like this:

Sinatra Hello World
1
2
3
4
5
require 'sinatra'

get '/' do
  'Hello world!'
end

You can see the DSL part coming in to play with the use of get to define how the app should respond to an HTTP GET request that is issued to the root URI. This is where one would specify routes. You can see that it’s a little non-trivial to create routes on the fly, especially when the routes map to arbitrary Classes with their own behavior and method invocations available to them.

The first step here was to go about finding a way to create a new node. The creation of a node is a task that could be exposed in multiple ways; direct user input, the introduction of an Arduino to the network that gets picked up, etc. For our purposes, lets say that the user wants to add a new node themselves. We’ll keep it simple here, and just allow for the creation of a new node based on its name, and ignore the part where we define behavior. At the end, the Sinatra stack itself should be responsible for exposing the node creation functionality as an API endpoint. First, I create a simple static page (no need to get fancy) that displays a text field, and it submits the the information back to the Sinatra stack using a basic AJAX request.

Data attributes are based on jQuery Mobile, which I use to generate the widgets (it looks nice and it’s easy to use).

new_node.html
1
2
3
4
5
6
7
<body>
  <div id="container">
    <label for="new-node-name">New Node Name:</label>
    <input type="text" name="new-node-name" id="new-node-name" value="Enter new node name..."/>
    <a href="#" data-role="button" id="submit-name">Send New Name To Node Creator</a>
  </div>
</body>

Which gives us:

And the accompanying JavaScript to issue the request:

JavaScript For POSTing to Make Node
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<script type="text/javascript">
$(document).bind("pageinit", function(event){
  console.log("Document ready");
      
  $('a[data-role="button"]').buttonMarkup({corners: false, inline: true, icon: "refresh"});
      
  $('#submit-name').unbind('click').click(function(event){
      event.preventDefault();
      event.stopPropagation();
                      
      var text_field = $('#new-node-name');
      var str = 'name=' + text_field.val();
      text_field.val("");
                      
      $.ajax({
          url: "make_node",
          type: "POST",
          data: str,
          cache: false,
      });

      return false;
  });
});
</script>

So once we put a name in the text field and submit it, we issue a post request to the URI http://brain.local/make_node?name= with the name of the node.

Now, let’s write the Sinatra code to handle the POST request. POST requests sent to the make_node route need to be handled in a way that allows the app to:

  1. Pull the name of the new node out from the request
  2. Create the appropriate file based on this name
  3. Write out the file in a manner that is useful

For my project specifically, the long-term model will have this particular endpoint write out a definition file, and a separate process will monitor a directory of definition files that will generate new Ruby files based on the behavior defined. For this demonstration we’re just going to write out a Ruby file directoy, since we aren’t actually defining any behavior for our nodes.

Note that we’re not performing any input sanitation; this is obviously a terrible thing. The only reason that this is the case is because:

  1. I’m prototyping this stuff really, really quickly to meet some deadlines,
  2. This is not production code. It’s experimental code. The stuff that will go in to the real project (which I’ve already started) looks nothing like this. This is just for the purposes of this demo.
  3. I have complete control over all input; and by the time the project is mature, all of the input will be managed similarly.

None of those are exceptionally good excuses, but just be cognizant of the fact that you’ll obviously want to restrict the input if you do something like this in any project that’s remotely important.

Let’s look at the route handler:

Handle POST Requests at make_node
1
2
3
4
post '/make_node' do
  node_name = "#{params[:name]}"
  NodeTools.make_node(node_name)
end

Looks easy. Except NodeTools is a class I wrote, so we’ll have to dig in to NodeTools to see what make_node does. This is just a demonstration, though, of how easy Sinatra makes it to manage routes for requests. It’s much closer to the metal than Rails, and much more lightweight. For this reason, Sinatra is fantastic if your “web app” is primarily an API service instead of a full-blown web app.

Let’s look at NodeTools, specifically, the definition of make_node:

NodeTools.make_node
1
2
3
4
5
6
def self.make_node(node_name)
  File.open("./nodes/" + node_name.downcase + ".rb", "w") do | node |
    self.write_requires node
    self.write_node_class_def node, node_name
  end
end

Easy enough; I chose to downcase all of the letters when creating the file. I’m working under the assumption that all of the text input will have no spaces, and only contain letters and underscores. This sort of input restriction should be enforced on the interface side, not on the API side, so in our specific example, the submission handler should refuse to POST given bad input. That said, the API should still fail gracefully in the event that it does receive bad input. Shame on us for not making sure this is so. Anyway. Moving on.

We open a file, stick it in a folder called nodes, and name it after the name of the node. Next, we write to me file. NodeTools.write_requires isn’t very interesting, it just writes out a list of require statements. The real meat is in NodeTools.write_node_class_def(node, node_name):

NodeTools.write_node_class_def(node, node_name)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
def self.write_node_class_def(node, node_name)
  raise TypeError unless node.is_a? File
  camel_case_node_name = String.new

  node_name.split("_").each do |token|
    camel_case_node_name << token.capitalize
  end

  node.puts "class " + camel_case_node_name
  node.puts "\tdef default"
  node.puts "\t\treturn \"Dynamic " + camel_case_node_name + "!\""
  node.puts "\tend"
  node.puts "end"
end

So recapping: Given a node name, we create a file modeled after the name, and then write out to it. Our method for writing out tokenizes an underscore-delimited list of words in to camel case, then uses it to write out the definition for a class named after the node whose only method is default, which spits out “Dynamic <class_name>”.

A node with the name foo would create the file foo.rb describing class Foo. It looks like this:

foo.rb
1
2
3
4
5
class Foo
  def default
    return "Dynamic Foo!"
  end
end

So far, so good. Hacked together, brute force, inelegant, unsafe, but works.

Now, we want to be able to use Foo, and invoke Foo.default. This is the fun part. We want to be able to get to Foo in a meaningful manner, so we want Sinatra to be able to respond to requests issued to a sensible route. We’re going to use http://brain.local/node/foo. Note I didn’t say nodes; I’m not directly providing access to the nodes folder on the server’s filesystem. The route is just a clever alias that, for lack of a better description, invokes a Remote Procedure Call (that turns out to be not Remote in any sense of the word).

First, we have to get Sinatra prepared to handle arbitrary requests to /node/blahblahblah. Thankfully, routes can be described using Regular Expressions. On top of that, given that you’re running a new enough version of Sinatra and Ruby (Ruby 1.9+ specifically), you can use a handy little feature called named captures that allows you to reference regular expression groups by a name stuffed in to a parameters array.

So, we’ll describe a route handler for GET requests issued to “/node/blahblahblahblah”:

Handling Arbitrary Get Requests For Nodes
1
2
3
4
5
6
7
get %r{/node/(?<node_name>\w*)/?} do
  node_name = "#{params[:node_name]}"

  newNodeObject = NodeTools.load_node(node_name)

  return newNodeObject.send("default")
end

You can see the syntax for named captures; I’m honestly not a fan of the syntax, if only because it uses the ? which I always associate with the “zero or one” quantifier, but to each their own. The rundown on the named capture syntax is that, immediately after the opening paren for a capture group, you stick a ?<capture_name> before the pattern.

We then pull the node name out of the URI, use our old friend NodeTools to load a node object up, and then we use the Object.send method to invoke the default method by name. If you weren’t aware of the Object.send trick, don’t worry, neither was I, but it turns out that Ruby has very flexible String-based reflection capabilities. We’re going to leverage those some more in NodeTools.load_node, which is where the real fun happens:

NodeTools.load_node
1
2
3
4
5
6
7
8
9
10
def self.load_node(node_name)
  load "./nodes/"+node_name+".rb"
  class_name = String.new

  node_name.split("_").each do |token|
    class_name << token.capitalize
  end

  return Object.const_get(class_name).new
end

First, we load the file. Note, that I specifically choose to load and not require the file. I’m doing this on purpose. We’re not writing something designed to scale and handle a million requests; it’s a tiny isolated brain program that will only handle a few requests every few hours. load causes the corresponding file to get reloaded every time, whereas require doesn’t reload a file if it is required more than once. Since we want our arbitrary routes to be flexibly redefinable at runtime without having to restart the web server process, we use load every time an arbitrary node is asked for to make sure we have the most up-to-date version of the Ruby class corresponding to the node. We create an empty string that is going to represent the class name, which we’ll generate from the node name. It’s easier than parsing the text file. We use the same tokenize and camel case method as before.

It’s the last line where the secret sauce lives; once we’ve called load on the Ruby file, the classes defined within are now part of our running Ruby environment. Here’s the cool part: Class names are simply top-level constants in Ruby. All objects inherit a const_get method that allows them to access defined constants by name; since Class names are top-level constants, we can interact with classes by name using Object.const_get. Kapow. NodeTools.load_node gives us back an object corresponding to the given node name, and we can then use Object.send to invoke the default method by name.

Pretty cool, huh?

So if we go to http://brain.local/node/foo in our browser, we should see a simple page that says “Dynamic Foo!”. Do we?

Well, I’ll be damned. It works.

BUT WAIT! THERE’S MORE!

Let’s prove to ourselves that we can alter the behavior of routes at runtime. And we’re not just going to change what default prints out. We’re going to add new method definitions (manually) to our class, while Sinatra is still running. I’m going to dig in to foo.rb and I’m going to add a new method.

foo.rb
1
2
3
4
5
6
7
8
9
class Foo
  def default
    return "Dynamic Foo!"
  end

  def bar
    return "Dynamic Foo Bar!"
  end
end

Now, we need to differentiate between Foo.default and Foo.bar. I’m still not going the route of URI parameters; In the real project, I use those for method arguments. To invoke a method, we’re just going to modify our arbitrary routing scheme. Now, we’re gonna use something like this: http://brain.local/node/class/method.

Let’s take a look at the tweaked GET handler:

Handling Arbitrary Routes With Method Invocation
1
2
3
4
5
6
7
8
9
10
11
12
get %r{/node/(?<node_name>\w*)/?(?<method_call>\w*)} do
  node_name = "#{params[:node_name]}"
  method_call = "#{params[:method_call]}"

  newNodeObject = NodeTools.load_node(node_name)

  unless method_call.empty?
    return newNodeObject.send(method_call)
  end

  return newNodeObject.send("default")
end

We add a second capture to our regexp, and extract it as the method we want to call. Loading up the object happens exactly the same way. Now we just do a check to see if we have an explicitly named method call, and if we do, we pass it in to Object.send.

So we go to node/foo/bar in our browser, and:

Bang. Never once did we have to shut down our Sinatra process and spin it back up.

Below are the complete source files for my Sinatra file and NodeTools.

Please, for the love of all that is holy, do not use these in production. So much needs to be added to them before they are clean. I’m not even using the code in this current state in my prototypes. These are simply concept snippets that you can use as a jumping-off point if you really do find this interesting for whatever reason. I’m pretty sure this technique solves a problem that nobody actually has, but it was useful for me, and I wanted to share because it encompasses some pretty cool tricks. I’m sure that there are better ways to do what I wanted to do, and if so, please go up top and hit the contact link and drop me a line. But until then, in all of its glory, here you go:

Sinatra brain:

NodeTools: