LiquidFoot » General

Deploying Omeka

Wayne — Thu, 12 Nov 2009 00:10:40 +0000

At work, we’ve been doing a lot of development using the Omeka framework. We’ve implemented plugins to enable the software to interact with our Fedora repository, work with geo-rectified images, build in a search interface with Solr, and another one (Adam’s not quite ready to announce this one yet). However, as good as the project is, one thing I’ve noticed is that the ways to deploy the software are a bit heavy on the user knowing what they’re doing. They’ve done a good job of guiding users through the setup after all the pieces are in place, but I’m a big fan of having automated ways of deploying software.

One of my favorite deployment tools is Capistrano. I love being able to check changes to the software to our Subversion server, and from my local machine type a simple command and update the remote source. We actually have been using this for several of our instances on our staging servers. The secret for deploying a PHP project with Capistrano is with Lee Hambley’s railsless-deploy gem. The first time you set one of these up, it’s a real pain, but you can quickly develop a template for deploying these.

I’ve been paying attention to some of the musings on Twitter regarding setting Omeka up. One of the big stumbling blocks is with MySQL. I wrote a quick helper script, and in one of my “how can I make this more beneficial” moments, I wrote a spec file for generating an rpm to install it.

I hadn’t worked on packaging in Linux for a while and forgot how painful it could be. Compounding my efforts was the fact that I was running Fedora Core to build it from VirtualBox. Normally this runs fine, but the fact that I was using rpmbuild was causing VirtualBox to eat up most of my available memory (I only have 4GB). It was just slow going because the rpmlint was taking so long.

Anyway, I have a functional rpm that can how be installed. I haven’t tested it much (read that at all), but I believe I’ve worked the bugs out of the build process. The idea here is to deploy it in a slightly different manner than you get out of the box. The RPM version creates /etc/omeka where it stores its database configuration and then defaults to placing the directory tree in /var/www/omeka. This helps in separating the configuration from the actual web application, but I’m debating with myself if everything but the archive folder should go into /usr/local/omeka.

Anyway, we’ll have an update soon, along with more code!

Capistrano Deployment for Omeka

Wayne — Fri, 28 Aug 2009 15:44:09 +0000

We’re currently starting a project at the Scholars’ Lab in which we’re going to use Omeka. We’re a Rails shop and use Capistrano to deploy projects to our staging and production servers all the time. Since I had read that Capistrano didn’t need to be for Rails, I thought it would help out in deploying the PHP for our Omeka work.

It took a little work, but I finally stumbled across Lee Hambley’s railsless-deploy project over on Github.

The install is pretty easy (assuming you have rubygems installed).

gem sources -a http://gems.github.com/ # make sure you have github sources installed
gem install leehambley-railsless-deploy

After this, for rails apps, you just capify the project, but this command didn’t seem to work, so I just manually added the Capfile in the root of the project directory with the snip provided on the project page:

load 'deploy' if respond_to?(:namespace) # cap2 differentiator
# Dir['vendor/plugins/*/recipes/*.rb'].each { |plugin| load(plugin) }

require 'rubygems'
require 'railsless-deploy'
load    'config/deploy'

After that, it was just a matter of using the recipes we typically use. Now, after we get the plugins we’re developing ready to be deployed, all we have to do is

cap deploy

Capistrano makes life so much easier!

Because I’m a Geek

Wayne — Mon, 26 Jan 2009 00:01:51 +0000

Saw this shirt and it made me laugh. Then I got out a pencil (because I’m a geek) and solved it

Who say’s math majors can’t be funny!

rdesktop and Tcl/tk

Wayne — Fri, 16 Jan 2009 16:36:17 +0000

This is something I did a few years ago and has been very useful. I’m a *nix admin that occasionally needs to log on to a Windows server. I used rdesktop for a long time to do this. I wanted, though, a nice GUI that would allow me to pick which server I wanted (and an excuse to play a bit with Tcl). This is the script I came up with to do just this:

#! /usr/bin/wish

wm title . "Remote Desktop Launcher"

frame .top -borderwidth 10
pack .top -side top -fill x

button .top.quit -text Quit -command exit
set but [button .top.run -text "Launch" -command Run]
pack .top.quit .top.run -side right

label .top.l -text Server: -padx 0
entry .top.cmd -width 20 -relief sunken -textvariable server
pack .top.l -side left
pack .top.cmd -side left -fill x -expand true

bind .top.cmd <Return> Run
focus .top.cmd

menubutton .mb -text Server -menu .mb.menu
pack .mb -padx 10 -pady 10

set m [menu .mb.menu -tearoff 1]
$m add radio -label server1 -variable server -value server1.address.edu -command Run
$m add radio -label server2 -variable server -value server2.address.edu -command Run
$m add radio -label server3 -variable server -value server3.address.edu -command Run
$m add radio -label server4 -variable server -value server4.address.edu -command Run
$m add radio -label server5 -variable server -value server5.address.edu -command Run

proc Run { } {
	global server
	exec rdesktop -u <user_name> -p <password> -g 1280x800 -x b $server
}

All you need to do is plugin “real” values for the server and addresses. You can also auto-login by giving values for the username and password (may also want to change the geometry of the window unless you have a big screen).

I also have bash aliases for these, but that’s a different post…

What Is a Database?

Wayne — Tue, 06 Jan 2009 20:13:01 +0000

After working on a couple of different projects over the years that lent themselves to using a database, it seems a lot of people have some misconceptions of what exactly a database is…or what it does. Many have confused data stores and databases (when they really mean a database management system — but not necessarily in a relational model flavor). When they want to build a database, I’ve found many using the wrong tools (e.g. spreadsheets) and find themselves heading down a road full of headaches that can be time consuming to deal with.

Most of the people I’ve worked with have a basic understanding of the fact that a database is a store of information. While Excel (and other spreadsheet programs) can do really useful things like crunch numbers and even filter information and do basic lookups (like a database system), there are a few other things they can’t do that makes this perhaps not the best way to proceed in a project that needs to collect a lot of information. I’ll start this as a first installment on using database management systems for research.

If you can believe it, database systems still adhere to the 13 rules outlined in E. F. Codd’s paper “A Relational Model of Data for Large Shared Data Banks” written in 1969 while Codd was at IBM (and republished in 1970 in Communications of the ACM). These rules are commonly referred to as Codd’s 12 rules (numbered 0 – 12). So, here are the rules (I’ll explain them in a bit more detail later).

Rule 0: The system must qualify as relational, as a database, and as a management system.
Rule 1: All information in the database is to be represented in one and only one way, namely by values in column positions within rows of tables.
Rule 2: All data must be accessible with no ambiguity.
Rule 3: The database management system must allow each field to remain null (or empty). Specifically, it must support a representation of “missing information and inapplicable information” that is systematic, distinct from all regular values, and independent of data type.
Rule 4: The system must support an online, inline, relational catalog that is accessible to authorized users by means of their regular query language.
Rule 5: The system must support at least one relational language that
1. Has a linear syntax
2. Can be used both interactively and without application programs
3. Supports data definition operations (including view definitions), data manipulation operations (update and retrieve), security and data integrity constraints, and transaction management operations.
Rule 6: All views that are theoretically updateable must be updateable by the system.
Rule 7: The system must support set-at-a-time insert, update, and delete operators.
Rule 8: Changes to how the data is stored (its physical level) must not require a change to an application based on the structure.
Rule 9: Changes to the logical level (tables, columns, rows, etc) must not require a change to an application based on the structure.
Rule 10: Integrity constraints must be specified separately from application programs and stored in the catalog.
Rule 11: The distribution of portions of the database to various locations should be invisible to users of the database.
Rule 12: If the system provides a low-level (record-at-a-time) interface, then that interface cannot be used to subvert the system.

There are a bunch of different database managers out there. Most likely you even have at least one already on your computer (Firefox uses SQLite). Depending on what you need to do, and how much money you’re willing to invest, and the scope of your project, there are an amazing array of products of there. This is a brief list of some of the most popular:

MySQL (free): Very popular with web developers; was initially developed for speed over features, but has become very feature rich, though it does require programming to develop an interface
PostgreSQL (free): Feature-rich database system great for large-scale projects that will handle millions of connections easily; usually overkill for small projects
Apache Derby/Java DB (free): Pure Java relational database management system;
Microsoft Access: In a lot of corporate Office deployments; not good for projects with multiple users though
OpenOffice Base (free): Comes bundled with HSQLDB, but also allows you to use MySQL and PostgreSQL.
CouchDB (free): Kind of different than the rest in that it is a document-oriented database; it doesn’t contain a specific schema, but allows you to create ad-hoc, distributed, RESTful access to information. It’s not a relational database management system, but does provide some extra flexability to add “stuff” to a data store quickly.

Next up, designing tables so you can get stuff back out easily…

It Works!

Wayne — Thu, 18 Dec 2008 03:39:13 +0000

With my departure from W&M we talked about the deployment of the second Sun X4500 for the library. My original plan was to wait until the 10/08 system came out (which had the SAMBA stuff in the kernel) and then deploy the hardware with ZFS and Z-RAID. However, no one else really has any experience with either administering SAMBA or Solaris. I had read that Windows could be installed on these servers. It’s a bit of a sacrilege, but it should help out until they find a replacement.

The Sun Installation Assistant CD really took all the work out of setting the thing up…it includes all the required drivers for the hard drives and NIC. Took a while since I was redirecting the KVM through the ILOM and sharing my ROM drive. Everything went very well, but I did notice that the OS I installed (Windows 2003 R2 Standard) has a limit of 4GB. I couldn’t for the life of me figure out what happend to the other 12GB. Been in *nix too long and forgot that Microsoft puts limits on the amount of RAM a server can support. I just need to reinstall with Enterprise edition. Not a huge deal, but it’s one of those things I wish I would have known/remembered.

I did set up a “test” share (it’s primarily to give our media center access to a lot of storage), but I ran into an issue(s) with OS X. Seems the mount would take, but you can’t see any of the files. You can move files to the server, but they don’t show up.

I think the bandwidth should be ok coming from the media center. There’s full gigE from the desktop to the server and I ganged the NICs.

I wished I could have used the Z-RAID, and I’m not sure that the Windows Dynamic RAID-5 will do quite the same job, but it’s a step forward in getting some storage (and not attached hard drives) for media production.

GMail Ninjas

Wayne — Fri, 21 Nov 2008 03:23:43 +0000

I really dig the new themes in Google…I finally have ninjas

EndNote Migration

Wayne — Wed, 19 Nov 2008 20:03:24 +0000

I’ve been working with the folks over at the Center for Archaeological Research with a bibliography of their research reports. It’s a bit of an offshoot of the NEH Slave Quarters research I’ve been working on, but I wanted to have a way to work through their research reports, just to have an idea of what they have available.

The first thing I did is get them to give me the file in the crappy EndNote XML format. I was having issues bringing it into Zotero, so I installed the bibutils package (sudo apt-get install bibutils) and then ran the endx2xml program on it. This created a nice MODS file for me to start crunching on. What I wanted to do is make a tag cloud using some of the categories they created. I created a database with two tables to store the citation and author information. I’m on my development box, so this is MySQL


--

-- Create schema archaeology

--
CREATE DATABASE IF NOT EXISTS archaeology;

USE archaeology;
--

-- Definition of table `archaeology`.`authors`

--
DROP TABLE IF EXISTS `archaeology`.`authors`;

CREATE TABLE&nbsp; `archaeology`.`authors` (

`id` char(36) NOT NULL,

`citation_id` char(36) NOT NULL,

`given` varchar(50) NOT NULL,

`family` varchar(50) NOT NULL,

`role` varchar(50) NOT NULL,

PRIMARY KEY&nbsp; (`id`)

) ENGINE=InnoDB DEFAULT CHARSET=latin1;
--

-- Definition of table `archaeology`.`citations`

--
DROP TABLE IF EXISTS `archaeology`.`citations`;

CREATE TABLE&nbsp; `archaeology`.`citations` (

`id` char(36) NOT NULL,

`ref_id` varchar(25) NOT NULL,

`title` varchar(500) NOT NULL,

`year` int(11) NOT NULL,

`publisher` varchar(500) NOT NULL,

`added` timestamp NOT NULL default CURRENT_TIMESTAMP,

`genre` varchar(50) NOT NULL,

`subject` varchar(500) NOT NULL,

PRIMARY KEY&nbsp; (`id`)

) ENGINE=InnoDB DEFAULT CHARSET=latin1;

I next needed to write the XSLT to actually transform the modsCollection into the schema. I haven’t worked with XSLT in a while and was (very) rusty Here’s a sampling of the output of converting the EndNote XML to MODS

Heritage Education and Interpretive Plan for the Smith River Blueway and Pedestrian Trails, Henry County, Virginia

Elizabeth
Andre

author

Stephanie
Sapp

author

2007
William and Mary Center for Archaeological Research, Williamsburg, Virginia.      Submitted to Virginia Department of Historic Resources, Roanoke

text
report

06-25/L-WMCAR Reports/Henry Co/Interp/18th/19th/20th/

Andre2007a

Architectural Evaluation of the Anderson Farm (047-0034-0010) for the Proposed Anderson-Hughes Development Project in Norge, Virginia

Elizabeth
M
Andre

author

2007
William and Mary Center for Archaeological Research, Williamsburg, Virginia.      Submitted to Anderson-Hughes, LLC, Newport News, Virginia

text
report

07-22/L-WMCAR Reports/ARCeval/19th/20th/James City Co/

Andre2007b

What I wanted to do is take each citation and insert it into the database, then relate the last-entered citation with its authors. At first I thought I could use MySQL’s last_insert_id() function…turns out those are only for auto-increment fields and I always use UUIDs for primary keys. What I ended up doing is adding in the added field with a timestamp (with a default value of CURRENT_TIMESTAMP). Then, I just went through the nodes and put the data where it needed to go.









DELETE FROM citations;

DELETE FROM authors;

INSERT INTO citations ( id, ref_id, title, year, publisher, genre, subject)

VALUES (

UUID(),

'',

'







',

'',

'







',

'',

'







'

);


INSERT INTO authors (id, citation_id, given, family, role)

VALUES(

UUID(),

(SELECT id FROM citations ORDER BY added desc limit 1),



' ',

'',

''



);









'




''

I noticed that I also had to write a template to escape single quotes in the title, publisher, and topic because of some possesive language in those fields. It took the longest time to actually write the code for that (like I said, I was really rusty and forgot all about XSL branching).

In the end, what gets generated is a nice SQL script (this one pushes 30,000 lines) like this


DELETE FROM citations;

DELETE FROM authors;
INSERT INTO citations ( id, ref_id, title, year, publisher, genre, subject)

VALUES (

UUID(),

'Andre2007a',

'Heritage Education and Interpretive Plan for the Smith River Blueway and Pedestrian Trails, Henry County, Virginia',

'2007',

'William and Mary Center for Archaeological Research, Williamsburg, Virginia. Submitted to Virginia Department of Historic Resources, Roanoke',

'report',

'06-25/L-WMCAR Reports/Henry Co/Interp/18th/19th/20th/'

);
INSERT INTO authors (id, citation_id, given, family, role)

VALUES(

UUID(),

(SELECT id FROM citations ORDER BY added desc limit 1),

'Elizabeth ',

'Andre',

'author'

);
INSERT INTO authors (id, citation_id, given, family, role)

VALUES(

UUID(),

(SELECT id FROM citations ORDER BY added desc limit 1),

'Stephanie ',

'Sapp',

'author'

);
INSERT INTO citations ( id, ref_id, title, year, publisher, genre, subject)

VALUES (

UUID(),

'Andre2007b',

'Architectural Evaluation of the Anderson Farm (047-0034-0010) for the Proposed Anderson-Hughes Development Project in Norge, Virginia',

'2007',

'William and Mary Center for Archaeological Research, Williamsburg, Virginia. Submitted to Anderson-Hughes, LLC, Newport News, Virginia',

'report',

'07-22/L-WMCAR Reports/ARCeval/19th/20th/James City Co/'

);
INSERT INTO authors (id, citation_id, given, family, role)

VALUES(

UUID(),

(SELECT id FROM citations ORDER BY added desc limit 1),

'Elizabeth ',

'Andre',

'author'

);

There’s still a lot of cleaning up to do (normalizing the authors table, exploding the subjects, generating the tags), but it did give me pause to remember how beautifully maddening XSLT can be.

I’ll post some screenshots when this gets a bit further along, but I thought this may save someone some time at some point.

Virtual Box

Wayne — Mon, 21 Jul 2008 13:22:19 +0000

On several of the projects I work on, it’s very handy to have multiple operating systems installed to test out my code. For a while I’ve been using VMWare. It’s been working pretty nicely and I didn’t have many complaints. I did experience some headaches when I upgraded to Ubuntu 8.04 because of the kernel packages, but after a couple of bug fixes, recompiling, and reconfiguring, everything was happy.

I’ve got a Mac Pro on the desk next to me that we’ve been running Parallels on to run some of our Windows-only software (e.g. the client for our library catalog). Phil saw that there was support for Windows on OS X 10.5, so we threw Virtual Box on it to try it out. Here are my observations thus far running the software on Ubuntu and Leopard:

Virtual Box seems to be a bit faster. I have no tests to back this up, it just seems faster.
The VDIs are nice (here’s a longer list of VDIs)
Permissions for the kernel driver can cause a small headache (be sure to add your user account to the vboxusers group).
Virtual Box doesn’t require me to go get keys when I rebuild a computer and forget to write down the key

So far, I really am liking Virtual Box…

Rotten Luck

Wayne — Fri, 13 Jun 2008 13:23:15 +0000

I’m currently at a conference at Lehigh University, and it’s given me pause to really reconsider air travel. Every time I travel, seems something goes awry. Coming back from a conference in California about a month ago, the last leg of my flight was canceled because the plane needed a new tire, and all of Charlotte didn’t have a tire for that particular aircraft. I could swallow that since the aircraft wasn’t safe to fly, but the line to actually reschedule my flight took about 2 hours, and they couldn’t get me back to Newport News (where my car was) until almost midnight the next day. Fortunately, my wife came to get me in Norfolk. I was significantly peeved to not choose US Airways for my next flight to Vegas that I was going on in the next couple of weeks.

Coming back from Las Vegas (on United), the company had issued some “cost saving measures” that meant that the pilots couldn’t have the engines on before we boarded on the flight from Dulles to Richmond. It was hot that day, so they turned the fans on for us, which drained the auxiliary power, which meant the engines couldn’t be started. Well, they brought some field units on, but after the first three didn’t work, we disembarked the aircraft to wait for maintenance to bring a fourth unit from another unit to get the aircraft started.

In my latest trip (unfortunately back on US Airways), I got to Lehigh yesterday afternoon, but my bags only made it as far as Boston. I was first told my bags would be on the 4:00pm flight, but when I called at 6:30 to check, they told me they’d be arriving at 7:00pm. I called at 9:00 to check, and they told me my bag would be delivered by midnight last night. Since I wasn’t woken up by my phone last night, I called again this morning. I was informed that the delivery service began at 8:00am, and they had a 4 – 6 hour window for delivery.

I’m thinking that the next time I need to travel, Amtrak is looking a lot better…