Build Your Own Rails Plugin Platform with Desert
While it is easy to include plugins in your Rails projects, it isn't easy to extend and customize the plugin for your own application. Desert solves that limitation/complication by making it just as easy to extend or modify a plugin class as it would be with any other class. In this post we will go over how Desert provides an easy way to manage and extend your plugins.
At Pivotal, we offer an integrated platform of Rails plugins named Socialitis.
Socialitis is an internal project that grew out of the observation that many of our start-up clients needed to build the same non-differentiating features; user management, friends/contacts, activity feeds, on-site messaging, etc.
The Socialitis platform is broken up into number of plugins that extend the Rails app in specific ways. These plugins may have dependencies on other plugins.
One of the major design goals of Socialitis is easy, drop-in, integration into existing Rails apps. This means using convention over configuration and removing as much integration responsibility from the user of the plugin as possible.
Another design goal was to provide sensible defaults and make each plugin easy to customize for your app.
We used Desert to achieve these goals, and so can you for your own platform.
The major features that Desert provides are:
- Defining a Rail's like directory structure into your plugin (models, views, controllers, helpers)
- Plugin dependencies
- Seamless overriding of classes and modules defined by parent plugins
- Plugin migrations
- Plugin routing
Desert provides a similar feature set to the Radient plugin system and the now defunct Appable Plugins framework.
For a simple example, lets say you have two plugins, name User and Messaging. The User plugin provides basic authentication and login features, and the Messaging plugin allows Users to send Messages to each other. The Message plugin depends on the User plugin.
The directory structure of the full Rails app looks like:
|-- app
| |-- controllers
| | |-- application.rb
| | `-- blogs_controller.rb
| |-- helpers
| | |-- application_helper.rb
| | `-- blogs_helper.rb
| |-- models
| | `-- user.rb
| `-- views
| |-- blogs
| |-- layouts
| | `-- users.html.erb
| `-- users
| |-- index.html.erb
| `-- show.html.erb
|-- db
| `-- migrate
| `-- 001_migrate_users_to_001.rb
|-- lib
| `-- current_user.rb
|-- spec
| |-- controllers
| | `-- blogs_controller_spec.rb
| |-- fixtures
| |-- models
| |-- spec_helper.rb
| `-- views
| `-- blogs
`-- vendor
`-- plugins
`-- user
|-- app
| |-- controllers
| | |-- logins_controller.rb
| | `-- users_controller.rb
| |-- helpers
| | |-- logins_helper.rb
| | `-- users_helper.rb
| |-- models
| | |-- login.rb
| | `-- user.rb
| `-- views
| |-- logins
| | |-- edit.html.erb
| | |-- index.html.erb
| | |-- new.html.erb
| | `-- show.html.erb
| `-- users
| |-- edit.html.erb
| |-- index.html.erb
| |-- new.html.erb
| `-- show.html.erb
|-- config
| `-- routes.rb
|-- db
| `-- migrate
| `-- 001_create_users.rb
|-- init.rb
|-- lib
| `-- current_user.rb
|-- spec
| |-- controllers
| | `-- user_controller_spec.rb
| |-- fixtures
| | `-- users.yml
| |-- models
| | `-- user.rb
| |-- spec_helper.rb
| `-- views
| `-- users
`-- tasks
`-- message
|-- app
| |-- controllers
| | `-- message_controller.rb
| |-- helpers
| | |-- message_helper.rb
| | `-- user_helper.rb
| |-- models
| | |-- message.rb
| | `-- user.rb
| `-- views
| `-- messages
| |-- edit.html.erb
| |-- index.html.erb
| |-- new.html.erb
| `-- show.html.erb
|-- config
| `-- routes.rb
|-- db
| `-- migrate
| `-- 001_create_messages.rb
|-- init.rb
|-- spec
| |-- controllers
| | |-- message_controller_spec.rb
| | `-- user_controller_spec.rb
| |-- fixtures
| | |-- messages.yml
| | `-- users.yml
| |-- models
| | |-- message_spec.rb
| | `-- user_spec.rb
| |-- spec_helper.rb
| `-- views
| `-- messages
`-- tasks
The User plugin introduces the various User and Login Rails objects. The Message plugin introduces its respective Message objects. Notice that the Message plugin also reopens some of the User objects to insert functionality.
For example, vendor/plugins/users/app/models/user.rb looks something like:
class User < ActiveRecord::Base
has_many :logins
end
The Message plugin would then reopen User in vendor/plugins/message/app/models/user.rb:
class User < ActiveRecord::Base
has_many :messages_received
has_many :messages_sent
end
Meanwhile, the main application can also reopen User in app/models/user.rb
class User < ActiveRecord::Base
def custom_app_method
# custom app logic #
end
end
Desert allows you to utilize Ruby's ability to repoen classes to layer on functionality in your plugins and application. At Pivotal, we have had success in sharing code across multiple client applications using this technique.
Another thing to note is normally the Message plugin would be loaded before the User plugin. Desert allows you to create plugin dependencies. So in vendor/plugins/message/init.rb:
require_plugin 'user'
require_plugin 'will_paginate'
This means you no longer need to define plugin load order inside of environment.rb. Your plugins can take care of that. Desert works with practically all plugins. That means you can have a plugin dependency on any existing Rails plugin.
To see more examples & documentation, take a look at the Desert project at http://github.com/pivotal/desert.
You mention that Desert can manage plugin dependencies. In the example you give, the Messages plugin would seemingly depend on the User plugin. How would you specify that so that Desert ensures the User plugin gets loaded first?
remove
@Adam - Thanks for the comment.
Inside of vendor/plugins/message/init.rb you would include:
I updated the article.
remove
So, I assume you had a reason to go this route rather than use Rails Engines. What would that reason be?
remove
Looks interesting -- but we're using Engines currently; just wondering if you'd be able to compare it to that? It does look broadly similar -- it's the details that are important to me I guess.
remove
@Dave & @Simon - We have been using Desert for over one year. We originally started with Rails Engines, then went to Appable Plugins, and finally created Desert. At that time, I believe Rails Engines did not support either multi-loading or the app directory within plugins (I don't remember exactly which at this moment).
That prompted us to use Appable Plugins. Unfortunately, we had frequent issues with its file loading and getting the tests to run. Also, there were separate plugins for plugin migrations and plugin routes.
So we created Desert, which proved to have solid file loading. We also merged in the plugin migration and plugin routes into Desert. This made it easier to manage all of the plugin extension features.
We have been happily using Desert since. It seems that in the mean time, Rails Engines has evolved to more or less provide the same functionality as Desert. Regretfully, I have not been up to date with Engine's evolution.
One major difference that I can see is the multi file loading mechanism is different.
For example, when requiring users_controller.rb, Rails Engines will look through the entire $LOAD_PATH and load any users_controller.rb file that is on the $LOAD_PATH.
Desert keeps a separate load path that contains all app & plugin routes. When requiring a file, Desert will attempt to match all files on the Desert load path. If there is at least one match, those files are loaded. If there are no matches from the Desert load path, then Desert will delegate to Ruby's normal require method which only loads the first file match on $LOAD_PATH.
Rails Engines loading mechanism has an issue with Models.
Desert's loading mechanism allows models incorporated into the multi-file loading fold.
If it has not yet already been considered, I think that Rails Engines should consider using a similar approach. I'd be happy in helping out with a patch.
With this improvement in mind, I think Engines deserves another look at Pivotal. Ideally there would be one project rather than two that offers almost identical functionality. That is assuming that there are no other major differences.
remove
Brian,
This is somewhat similar to what Django does with apps. e.g. Admin feature in django is a separate app and can be installed in other apps.
I have been looking for such a feature in Rails for long.
remove
Nice article Brian!
(just spotted it from the Radiant mailing list).
One question, though: it also manages the overloading/overwriting of plugin views and controllers into the main application?
A silly question, but will look into the code later today :-D
Thanks for your hard work and for sharing this with the community!
remove
@Luis - Thank you.
Yes, Desert does support overloading/overwriting of plugin views and controllers.
remove
I was informed by James Adam that Rails Engines does not support plugin dependencies, since he recommends using gemified plugins.
remove
I have played with desert a bit and am having trouble figuring out the best way to bootstrap it for unit testing. What is your recommendation for making the plugins easy to test by themselves and within another project?
remove
@Gabe - You can create a test suite within the plugin itself and integration tests within your application.
If the plugin you are making is simple, you may not need it to be within the context of a Rails application.
If the plugin is more complex and requires the Rails application to work, it would be easier to use a test Rails application that uses the plugin.
There are a couple of common patterns that I know of:
Now your applications that use the plugins should have logic dependent on the plugin working correctly. If this is the case, you are implicitly testing the integration of the plugin withing your applications.
In rare cases, you would need to have explicit integration tests for the plugin.
remove
Thanks for the reply; I think Engines did do the stuff you wanted -- we've been using it since about March 2007. I've actually already written a plugin to do model sharing, and I've submitted a patch. I'm not sure what happened with that; it may have been taken up in a recent version, but we're sort of using different versions in different apps and I can't honestly remember.
Desert looks interesting though, I might have to evaluate it a bit more. Engines doesn't seem to play as nicely with Rails 2.0 as it used to with 1.2, so I'd be interested to see how Desert works.
You don't happen to know how the autoloading stuff interacts with Passenger/mod_rails do you?
remove
Just wanted to give it a try, but it failed. I created a new Rails 2.1 project and followed the instructions of the readme. After adding the "require 'desert'" to environment.rb Rails won't startup any longer (e.g. script/console). It keeps on doing something with the filesystem, seems like scanning my whole disk (used lsof to track what's going on).
I'd really like to try it out since I have class-reloading issues with Rails Engines.
Any clue?
remove
@Christian - Its possible that there is a circular dependency with the constant loading.
For example:
The User constant is not added to Object when it is evaluated in something_special.rb. When User is encountered in the SomethingSpecial module, Object.const_missing is invoked and user.rb is loaded again. Also SomethingSpecial did not yet get added either, so something_special.rb will get loaded again.
A fix for this issue is something like:
This would work because the User constant is added to Object when SomethingSpecial is loaded.
remove
@Simon - I havn't tried Desert with mod_rails yet so I'm not sure about all of the details.
One thing to note is Desert will lazily load your files and mod_rails uses fork. This may cause a performance issue if the files are not loaded in environment.rb, because each forked process would need to do the same work to load the files over and over again.
To fix that issue (if there is one), you would need to show the constants or require the files within environment.rb.
For example:
remove
Hi.
I just started to look into desert, cause it really looks like a great thing to use for sharing code between projects. Especially when you can mange them as a plugin with git/svn and piston, pretty neat.
However, i ran into problems pretty fast. I've followed the few install instructions, but I can't get the desert_plugin generator to work. "script/generate desert_plugin foobar" just give me the errormsg
"Couldn't find 'desert_plugin' generator"
I can manually copy an application to the plugin-folder, and it works as it should, but it would nicer to use the generator to have things set up automaticly. Any thoughts?
/MAttias
remove
I am also getting the following error.
"Couldn't find 'desert_plugin' generator"
Dan
remove