So I haven’t talked about it in a while, but some of you might remember that one of my many side-projects involves Arduinos and home networks. The project itself is related to home automation. I’m lucky in that I get the chance to expand on the project through classes I’m taking at the university; specifically, some Natural Language Processing researching I’m doing for an Artificial Intelligence project. This doesn’t have to do with NLP, though. But bare with me. I think it’s something pretty cool, if not a little hacked together.
The project is intended to be very modular; the interfaces can be many and arbitrary, not strictly Natural Language. The assorted interfaces only need to conform to an API that lives in a central brain which facilitates communication with different nodes in the network. You can think of each node as an individual Arduino, for now. Currently, I’m exposing the API to the brain over HTTP, and I’m using Sinatra to make prototyping the endpoints very quick.
One of the key features that I wanted was the ability to create and remove nodes from the network without having to restart the Sinatra web server process; but Sinatra doesn’t seem to have an immediately obvious way to create and remove routes on the fly. Additionally, because these routes need to be flexible and possibly even re-definable, the challenge was even hairier. I basically needed a way to dynamically load and unload Ruby code inside of a single process without reinvoking the interpreter. This is a little discussion on how I went about solving this. It’s the text-book definition of a hack, and there are probably more robust ways to go about doing what it is that I want to do.
Arbitrary, Dynamic Routes. Jesus.
If you aren’t familiar with Sinatra, it’s a Domain Specific Language built on top of Ruby that allows for the rapid creation of simple HTTP handlers. It allows you to define responses to HTTP methods like
POST, etc. very easily with a simple syntax. The Sinatra “Hello World” looks something like this:
1 2 3 4 5
You can see the DSL part coming in to play with the use of
get to define how the app should respond to an HTTP
GET request that is issued to the root URI. This is where one would specify routes. You can see that it’s a little non-trivial to create routes on the fly, especially when the routes map to arbitrary Classes with their own behavior and method invocations available to them.
The first step here was to go about finding a way to create a new node. The creation of a node is a task that could be exposed in multiple ways; direct user input, the introduction of an Arduino to the network that gets picked up, etc. For our purposes, lets say that the user wants to add a new node themselves. We’ll keep it simple here, and just allow for the creation of a new node based on its name, and ignore the part where we define behavior. At the end, the Sinatra stack itself should be responsible for exposing the node creation functionality as an API endpoint. First, I create a simple static page (no need to get fancy) that displays a text field, and it submits the the information back to the Sinatra stack using a basic AJAX request.
Data attributes are based on jQuery Mobile, which I use to generate the widgets (it looks nice and it’s easy to use).
1 2 3 4 5 6 7
Which gives us:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
So once we put a name in the text field and submit it, we issue a post request to the URI
http://brain.local/make_node?name= with the name of the node.
Now, let’s write the Sinatra code to handle the POST request. POST requests sent to the
make_node route need to be handled in a way that allows the app to:
- Pull the name of the new node out from the request
- Create the appropriate file based on this name
- Write out the file in a manner that is useful
For my project specifically, the long-term model will have this particular endpoint write out a definition file, and a separate process will monitor a directory of definition files that will generate new Ruby files based on the behavior defined. For this demonstration we’re just going to write out a Ruby file directoy, since we aren’t actually defining any behavior for our nodes.
Note that we’re not performing any input sanitation; this is obviously a terrible thing. The only reason that this is the case is because:
- I’m prototyping this stuff really, really quickly to meet some deadlines,
- This is not production code. It’s experimental code. The stuff that will go in to the real project (which I’ve already started) looks nothing like this. This is just for the purposes of this demo.
- I have complete control over all input; and by the time the project is mature, all of the input will be managed similarly.
None of those are exceptionally good excuses, but just be cognizant of the fact that you’ll obviously want to restrict the input if you do something like this in any project that’s remotely important.
Let’s look at the route handler:
1 2 3 4
Looks easy. Except
NodeTools is a class I wrote, so we’ll have to dig in to
NodeTools to see what
make_node does. This is just a demonstration, though, of how easy Sinatra makes it to manage routes for requests. It’s much closer to the metal than Rails, and much more lightweight. For this reason, Sinatra is fantastic if your “web app” is primarily an API service instead of a full-blown web app.
Let’s look at NodeTools, specifically, the definition of
1 2 3 4 5 6
Easy enough; I chose to downcase all of the letters when creating the file. I’m working under the assumption that all of the text input will have no spaces, and only contain letters and underscores. This sort of input restriction should be enforced on the interface side, not on the API side, so in our specific example, the submission handler should refuse to POST given bad input. That said, the API should still fail gracefully in the event that it does receive bad input. Shame on us for not making sure this is so. Anyway. Moving on.
We open a file, stick it in a folder called nodes, and name it after the name of the node. Next, we write to me file.
NodeTools.write_requires isn’t very interesting, it just writes out a list of
require statements. The real meat is in
1 2 3 4 5 6 7 8 9 10 11 12 13 14
So recapping: Given a node name, we create a file modeled after the name, and then write out to it. Our method for writing out tokenizes an underscore-delimited list of words in to camel case, then uses it to write out the definition for a class named after the node whose only method is
default, which spits out “Dynamic <class_name>”.
A node with the name
foo would create the file
class Foo. It looks like this:
1 2 3 4 5
So far, so good. Hacked together, brute force, inelegant, unsafe, but works.
Now, we want to be able to use
Foo, and invoke
Foo.default. This is the fun part. We want to be able to get to Foo in a meaningful manner, so we want Sinatra to be able to respond to requests issued to a sensible route. We’re going to use
http://brain.local/node/foo. Note I didn’t say nodes; I’m not directly providing access to the nodes folder on the server’s filesystem. The route is just a clever alias that, for lack of a better description, invokes a Remote Procedure Call (that turns out to be not Remote in any sense of the word).
First, we have to get Sinatra prepared to handle arbitrary requests to
/node/blahblahblah. Thankfully, routes can be described using Regular Expressions. On top of that, given that you’re running a new enough version of Sinatra and Ruby (Ruby 1.9+ specifically), you can use a handy little feature called named captures that allows you to reference regular expression groups by a name stuffed in to a parameters array.
So, we’ll describe a route handler for
GET requests issued to “/node/blahblahblahblah”:
1 2 3 4 5 6 7
You can see the syntax for named captures; I’m honestly not a fan of the syntax, if only because it uses the ? which I always associate with the “zero or one” quantifier, but to each their own. The rundown on the named capture syntax is that, immediately after the opening paren for a capture group, you stick a
?<capture_name> before the pattern.
We then pull the node name out of the URI, use our old friend
NodeTools to load a node object up, and then we use the
Object.send method to invoke the default method by name. If you weren’t aware of the
Object.send trick, don’t worry, neither was I, but it turns out that Ruby has very flexible String-based reflection capabilities. We’re going to leverage those some more in
NodeTools.load_node, which is where the real fun happens:
1 2 3 4 5 6 7 8 9 10
load the file. Note, that I specifically choose to
load and not
require the file. I’m doing this on purpose. We’re not writing something designed to scale and handle a million requests; it’s a tiny isolated brain program that will only handle a few requests every few hours.
load causes the corresponding file to get reloaded every time, whereas
require doesn’t reload a file if it is
required more than once. Since we want our arbitrary routes to be flexibly redefinable at runtime without having to restart the web server process, we use
load every time an arbitrary node is asked for to make sure we have the most up-to-date version of the Ruby class corresponding to the node. We create an empty string that is going to represent the class name, which we’ll generate from the node name. It’s easier than parsing the text file. We use the same tokenize and camel case method as before.
It’s the last line where the secret sauce lives; once we’ve called
load on the Ruby file, the classes defined within are now part of our running Ruby environment. Here’s the cool part: Class names are simply top-level constants in Ruby. All objects inherit a
const_get method that allows them to access defined constants by name; since Class names are top-level constants, we can interact with classes by name using
NodeTools.load_node gives us back an object corresponding to the given node name, and we can then use
Object.send to invoke the default method by name.
Pretty cool, huh?
So if we go to
http://brain.local/node/foo in our browser, we should see a simple page that says “Dynamic Foo!”. Do we?
Well, I’ll be damned. It works.
BUT WAIT! THERE’S MORE!
Let’s prove to ourselves that we can alter the behavior of routes at runtime. And we’re not just going to change what
default prints out. We’re going to add new method definitions (manually) to our class, while Sinatra is still running. I’m going to dig in to foo.rb and I’m going to add a new method.
1 2 3 4 5 6 7 8 9
Now, we need to differentiate between
Foo.bar. I’m still not going the route of URI parameters; In the real project, I use those for method arguments. To invoke a method, we’re just going to modify our arbitrary routing scheme. Now, we’re gonna use something like this:
Let’s take a look at the tweaked
1 2 3 4 5 6 7 8 9 10 11 12
We add a second capture to our regexp, and extract it as the method we want to call. Loading up the object happens exactly the same way. Now we just do a check to see if we have an explicitly named method call, and if we do, we pass it in to
So we go to
node/foo/bar in our browser, and:
Bang. Never once did we have to shut down our Sinatra process and spin it back up.
Below are the complete source files for my Sinatra file and NodeTools.
Please, for the love of all that is holy, do not use these in production. So much needs to be added to them before they are clean. I’m not even using the code in this current state in my prototypes. These are simply concept snippets that you can use as a jumping-off point if you really do find this interesting for whatever reason. I’m pretty sure this technique solves a problem that nobody actually has, but it was useful for me, and I wanted to share because it encompasses some pretty cool tricks. I’m sure that there are better ways to do what I wanted to do, and if so, please go up top and hit the contact link and drop me a line. But until then, in all of its glory, here you go: