Over the last couple of years, I’ve been working on-and-off on improving the Database of Early Chesapeake Architecture. I’d recently gotten the data (again) from Swem Library and had been working on pushing it to a more open framework (it was ColdFusion + MSSQL; now Rails + Postgresql). What I wanted to be able to add in is an extensible method of citation for the various buildings and sites used (check out the work-in-progress). At the time, I tried to use RefWorks (with very little success) to handle the citations, but with the improvements to Zotero (and it’s Group ability), I thought I’d go down this road and test if this is a viable method of citation management.

I first took a look at Jeremy’s phpZotero code. The first thing I noticed is that there’s a pretty funky way that you have to construct URLs for the “API” (I think the Zotero API is more of an RSS decorator than an API, but hopefully they’re working on that). Essentially you construct a URL requests to the Zotero API server with a user ID. Unfortunately, most people don’t know their user_id, so the PHP code goes and looks this up on a different server, parses out the user_id, then constructs an HTTP request for RSS data from the API server. I kind of brace a bit at the thought of every request needing to go to two servers (multiple points of failure, speed, etc.). When I started creating a ruby wrapper for these calls, I made the object use the user_id (Integer) and not the user_name (String). Knowing folks might not know their user_id, I whipped up a quick Sinatra app on heroku to look up Zotero User ids: zotero-id-finder (source on github).

Next, I needed something to not have to use Nokogiri all the time to grab info off the Zotero server. I made a basic Gem to wrap these requests named rzotero. This is really helpful, but what I really wanted to do is add citations to ActiveRecord models a la acts_as_taggable. To that end, I started mocking up acts_as_citable which will allow you to add in citations identifiers to be used with rzotero plugin to record and display items in a collection in ActiveRecord contexts.

As a warning, all of these are in a pre-alpha-ish state and may (badly) break your code (especially since it’s lightly tested right now). As things progress, I’ll have more posts here, and over on the Scholars’ Lab site.

We have a few historic maps from our McGregor collection available through our Geoserver installation, so I looked up their layer ids and manually added them into a page with OpenLayers. Nothing fancy here, just a test to see what it looks like.

var lat, long layer;
var zoom = 10;

var lat, lon, layer;
 var zoom = 10;

 map = new OpenLayers.Map('map');
 map.addControl(new OpenLayers.Control.LayerSwitcher());

 layer1 = new OpenLayers.Layer.WMS("Map of the Southern States",
 "http://lat.lib.virginia.edu:8080/geoserver/gwc/service/wms",
 {layers: "McGregor:000003056_00011"}
 );
 layer2 = new OpenLayers.Layer.WMS("Nova Virginae tabula",
 "http://lat.lib.virginia.edu:8080/geoserver/gwc/service/wms",
 {layers: "McGregor:000003482_00011"}
 );
 layer3 = new OpenLayers.Layer.WMS("A map of the most inhabited part of Virginia",
 "http://lat.lib.virginia.edu:8080/geoserver/gwc/service/wms",
 {layers: "McGregor:000003012_st1"}
 );
 layer4 = new OpenLayers.Layer.WMS("Carte de la campagne en Virginie du Major General Mis. de la Fayette",
 "http://lat.lib.virginia.edu:8080/geoserver/gwc/service/wms",
 {layers: "McGregor:000003013_st1"}
 );
 layer5 = new OpenLayers.Layer.WMS("Carte de la campagne en Virginie du Major General Mis. de la Fayette",
 "http://lat.lib.virginia.edu:8080/geoserver/gwc/service/wms",
 {layers: "McGregor:000003019_00011"}
 );

 map.addLayer(layer1);
 map.addLayer(layer2);
 map.addLayer(layer3);
 map.addLayer(layer4);
 map.addLayer(layer5);

if(navigator.geolocation){
 navigator.geolocation.getCurrentPosition(function(position){
 lat = position.coords.latitude;
 lon = position.coords.longitude;
 map.setCenter(new OpenLayers.LonLat(lon, lat), zoom);

 var vectorLayer = new OpenLayers.Layer.Vector("Overlay");
 var feature = new OpenLayers.Feature.Vector(
 new OpenLayers.Geometry.Point(lon, lat),
 {some:'data'},
 {externalGraphic: 'http://geocoder.ca/marker.png', graphicHeight: 64, graphicWidth: 48});
 vectorLayer.addFeatures(feature);
 map.addLayer(vectorLayer);
 });
 }

map.addControl(new OpenLayers.Control.PanZoomBar());

This actually worked surprisingly well the first time I launched it. With Firefox 3.5 I tested it on my laptop. At first I wasn’t using the geowebcache we had set up, so it was a bit slow. I changed to the geowebcache WMS service and things in the browser were noticeably faster.

At lunch, I handed an iPhone to Schuyler Erle who’s first instinct was to try to drag things around and zoom-in and out. Well, there was a fail as the controls you normally use in OpenLayers were far too small to actually be useful and it just didn’t feel right.

I then attempted to add in a touchHandler function to help with the interactions from the user. I found Ross Boucher’s post “iPhone Touch Events in JavaScript” and used his code to get the sliding working:

function touchHandler(event)
{
   var touches = event.changedTouches,
   first = touches[0],
   type = "";

   switch(event.type)
   {
     case "touchstart": type = "mousedown"; break;
     case "touchmove":  type="mousemove"; break;
     case "touchend":   type="mouseup"; break;
     default: return;
 }

 // initMouseEvent(type, canBubble, cancelable, view, clickCount,
 // screenX, screenY, clientX, clientY, ctrlKey,
 // altKey, shiftKey, metaKey, button, relatedTarget);

    var simulatedEvent = document.createEvent("MouseEvent");
    simulatedEvent.initMouseEvent(type, true, true, window, 1,
    first.screenX, first.screenY,
    first.clientX, first.clientY, false,
    false, false, false, 0/*left*/, null);

    first.target.dispatchEvent(simulatedEvent);
    event.preventDefault();
 }

 document.addEventListener("touchstart", touchHandler, true);
 document.addEventListener("touchmove", touchHandler, true);
 document.addEventListener("touchend", touchHandler, true);
 document.addEventListener("touchcancel", touchHandler, true);

Now the scrolling worked, but you could no longer zoom into the map (and we want real zoom, not just making the image bigger). I haven’t found a fix for this yet, but I have to say that this is a kind of cool way of interacting with these materials.

Something I need to talk with Joe about next week making your location a query to Geonetwork and then using the top n maps returned to build this interface. It’s a little like a coverage map for your current location.

What I’m going to do here is go through the setup for building the RPM and the spec file. This should work on just about any OS that uses RPM for installing software, but this isn’t tested very extensively.

So I guess the first thing to cover is what exactly an RPM is. RPM is the Redhat Package Manager and is an implementation of a container for other files. It allows you to quickly install software, including necessary programs to run the program, provides information about the program you are installing, informs users about potential conflicts between your program and the software currently installed on the system, and provides a path to install, upgrade, and remove the software.

Now, in the case of Omeka, you don’t really need a package manager if you’re a seasoned system administrator. However, if you’re wanting to run this yourself, and you have to convince others to manage the software for you (e.g. you’re running it at an institution), have an RPM that your systems administrator installs will save a lot of headaches, especially since upgrades to the software can be integrated into scheduled maintenance with kernel and other software upgrades.

Building RPMs requires two things, the source you want to install and a specification (spec) file that tells the software what to do. You’ll need to first set your system up to have the tools to build software which you can do with a couple of lines (as the root user):

yum groupinstall "Development Tools"
yum install rpmdevtools

Next we need the directory hierarchy for building the RPMs. You can do this manually, but there’s a handy tool for helping out called rpmdev-setuptree. In the terminal, if you run this command, it will create a new folder in your home directory named “rpmbuild” with the correct directories made for you.

rpmdev-setuptree

If you want these in a different location, say omeka_rpm, you can manually create the following hierarchy:

mkdir -p ~/omeka_rpm/BUILD
mkdir -p ~/omeka_rpm/BUILDROOT
mkdir -p ~/omeka_rpm/RPMS
mkdir -p ~/omeka_rpm/SOURCES
mkdir -p ~/omeka_rpm/SPECS
mkdir -p ~/omeka_rpm/SRPMS

For our purposes, we’re really going to be dealing in the SOURCES and SPEC directories. Let’s start with the setup for the source files. First you need to get the source files from Omeka’s website and put it into your SOURCES folder. We have to do some additional work with this file since it’s packaged using zip and the RPM is expecting a gzipped tarball. The following code snip assumes you’re using the default structure from the rpmdev-setuptree command and if a release after Omeka 1.1 is the most current, you will need to update the revision numbers.

cd ~/rpmbuild/SOURCES
wget http://omeka.org/files/omeka-1.1.zip
unzip omeka-1.1.zip
mv omeka-1.1 omeka
tar -czvf omeka-1.1.tar.gz omeka

Now if take a look at the directory, you should see a new file named “omeka-1.1.tar.gz” which is in the correct format for using it to build our RPM. There are two more files we need in here, an httpd configuration file and a readme for its use. Let’s just touch these so they exist and we can come back and add content to them a little later after I discuss them.

cd ~/rpmbuild/SOURCES
touch fedora-http-conf
touch README.fedora.omeka

Ok, let’s look at the real meat of this now. You need to create an omeka.spec file in the SPECS directory. This is the script that tells the rpmbuild tool how to set up the system.

I use gedit when I have a GUI, but emacs or vi work just as well (better if you’re good at the commands).

The idea behind this setup is to set up Omeka much like other software on the system is set up. This assumes that there will be only one installation of Omeka running on the system and takes into account that MySQL may be running on another server, so only the php >= 5.2.4, httpd, php-mysql, and ImageMagick packages are required. The configuration for the database connection is set in /etc/omeka/db.ini and uses a symbolic link to the application files in /usr/share/omeka (though this isn’t quite working yet). Themes and the archive folder shouldn’t get overwritten, so those directories stay in place. An httpd configuration for omeka (which aliases “/omeka”) is added into the httpd.conf directory. Since this software does not actually need to be compiled for a processor architecture type, the build archicture is set to noarch.

Summary: Omeka web publishing software
URL: http://omeka.org
Name: omeka
Version: 1.1
Group: Application/Publishing
Release: 1%{?dist}
License: GPL
Source0: %{name}-%{version}.tar.gz
Source1: omeka-http-conf
Source2: README.fedora.omeka
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
Requires: php >= 5.2.4, httpd, php-mysql, ImageMagick
BuildArch: noarch

%description
Omeka is a free and open source collections based web-based publishing platform for scholars, librarians, archivists, museum professionals, educators, and cultural enthusiasts.

%prep
%setup -q -n omeka

%install
 mkdir -p ${RPM_BUILD_ROOT}%{_datadir}/omeka
 mkdir -p ${RPM_BUILD_ROOT}%{_sysconfdir}/omeka
 install -m 0644 -D -p %{SOURCE1} ${RPM_BUILD_ROOT}%{_sysconfdir}/httpd/conf.d/omeka.conf
 cp -pr * ${RPM_BUILD_ROOT}%{_datadir}/omeka
 #cat db.ini | sed -e "s|dirname(__FILE__).'/'|'/usr/share/omeka/!|g" > ${RPM_BUILD_ROOT}%{_sysconfdir}/omeka/db.ini
 #/bin/ln -sf ../../../etc/omeka/db.ini ${RPM_BUILD_ROOT}%{_datadir}/omeka/db.ini
 /bin/cp %{SOURCE2} ./README.fedora
 # Remove empty files for rmplint
 find ${RPM_BUILD_ROOT} -empty -exec rm -f {} \;
 # These are docs; remove and docify later
 rm -f ${RPM_BUILD_ROOT}%{_datadir}/omeka/{readme.txt,license.txt,release.txt,changelog.txt}

%clean
 rm -rf ${RPM_BUILD_ROOT}

%files
 %defattr(-,root,root,-)
 %config(noreplace) %{_sysconfdir}/httpd/conf.d/omeka.conf
 %dir %{_datadir}/omeka
 %{_datadir}/omeka/admin
 %{_datadir}/omeka/application
 %{_datadir}/omeka/archive
 %doc changelog.txt
 %config(noreplace) %{_sysconfdir}/omeka/db.ini
 %{_datadir}/omeka/index.php
 %doc license.txt
 %{_datadir}/omeka/paths.php
 %{_datadir}/omeka/install
 %{_datadir}/omeka/plugins
 %doc readme.txt
 %doc release.txt
 %{_datadir}/omeka/robots.txt
 %{_datadir}/omeka/themes
 %dir %{_sysconfdir}/omeka

%changelog
* Mon Nov 09 2009 Wayne Graham <wayne dot graham at virginia dot edu> - 1.1
- Initial build

Now we can add in the httpd configuration that will get written. Open up SOURCES/omeka-http-conf and add

Alias /omeka /var/www/omeka

<Directory /var/www/omeka>
 AllowOverride Options
</Directory>

And README.fedora.omeka

Omeka is a database driven web publishing program.

Once this package is installed, there are a few configuration items which need
to be performed before Omeka is usable.  First, you need to establish a
username and password with which to connect to your MySQL database. You also need to make both
MySQL and Omeka aware of this configuration.  Let's start by creating the database and the
username / password inside MySQL first:

 # mysql
 mysql> create database omeka;
 Query OK, 1 row affected (0.00 sec)

 mysql> grant all privileges on omeka.* to omeka identified by '1superPrivateOmekaPassword!';
 Query OK, 0 rows affected (0.00 sec)

 mysql> flush privileges;
 Query OK, 0 rows affected (0.00 sec)

 mysql> exit
 Bye
 #

Under certain curcumstances, you may need to run variations of the "grant"
command:
mysql> grant all privileges on omeka.* to omeka@localhost identified by 'omeka';
 OR
mysql> grant all privileges on omeka.* to omeka@'%' identified by 'omeka';

This has created an empty database named 'omeka', created a user named
'omeka' with a password of '1superPrivateOmekaPassword!', and given the 'omeka' user total
permission over the 'omeka' database.  Obviously, you'll want to select a
different password, and you may want to choose different database and user
names depending on your installation.  The specific values you choose are
not constrained, they simply need to be consistent between the database and the
config file.

Next, you need to edit your /var/www/omeka/db.ini file to reflect the
values you've chosen.  These values will go in the appropriate places at the
beginning of that file.

Once that's done and the database server and web server have been started,
open a web browser to http://localhost/omeka/install/ and
follow the instructions given to you on the pages you see to set up the
database tables and begin publishing your Omeka instace.

Now everything is in place. Let’s try to build this thing. Go in to the SPEC directory and build the binary for the rpm.

cd ~/rpmbuild/SPECS
rpmbuild -bb omeka.spec

This takes a while, and if you’re running this through VMWare of VirtualServer, this process will demand some pretty hefty system resources and your system will slow down.

If everything has gone correctly, you should now have a nice new RPM package named omeka-1.1.fc11.noarch.rpm in the RPMS/noarch directory. You can test it out by installing the package with

rpm -Uvh omeka-1.1.fc11.noarch.rpm

What’s Next

For packaging RPMs like this, a few more infrastructure elements need to be in place. The following outlines some of them (if you have more, leave a comment).

• Put the build files into SCM (currently only in our private SVN repo)
• Auto-install dependencies
• Set up an RPM repository to work with yum
  • Target specific Linux builds
• Automate build
• Run MySQL setup script after initial install
• Build aptitude package (using alien?)

Additional Resources

• Chapter 4. Building RPM Packages
• How To Build RPM Packages on Fedora
• RPM scriptlet recipes
• Package Naming Guidelines
• Packaging Guidelines

One of the things I never liked the way I handled was describing the basic location. At first the architectural historians wanted to use UTM coordinates. I switched to latitute/longitude and wrote some conversion code to switch between the two. I was bothered, though, that describing a “site” by a point. Obviously a site is an area that contains some number n buildings. Even worse, the points were really just the centroids of the county/city of the information that was entered as the archaeologists didn’t have have information. There had to be a better way to approach the description of this information over time.

I started working at UVa earlier this spring and one of their big pushes has been GIS in the humanities. A light went off and I saw a rather nice use of using the concepts of GIS to better describe and interact with my information. I also really wanted to migrate the information to a more open format as well as really spend some time envisioning what the application should look like.

I’ve started working on this migration and have posted it on Github. To give a brief explanation of how the application. I’m using PostGIS as the backend. On OS X, getting this ready took a bit, but I think I got it explained in the README file decently. It took a while to get the correct templates ready in PostgreSQL.

I’m using OpenLayers for interacting with the maps (through map_layers), and the spatial_adapter plugin and the GeoRuby gem.

I’m expecting there to be some refactoring of the data from MSSQL, but I think PostgreSQL will be a much better solution for my open source leanings.

I’ll try to keep this updated with my progress…at least until our development blog gets off the ground.

A few years ago, I had written a bunch of Sherlock plugins for searching various library catalogs. Those pretty much lay dormant until I was talking with my student worker this last semester and he got quite excited about these, and wondered if there might be a way to also search the databases the library subscribes to, and provide off-campus access to these through our proxy server. This code is in a very basic stage right now, but essentially what I did was create a database driven application in PHP that generates OpenSearch XML for various targets to search against.

Right now, this only supports HTTP GET requests (POST parameters require an additional table), but here’s the DDL:

CREATE TABLE `plugins` (
  `id` char(36) NOT NULL,
  `short_name` varchar(50) NOT NULL,
  `description` varchar(255) NOT NULL,
  `engine` varchar(25) NOT NULL,
  `image` varchar(25) NOT NULL,
  `input_encoding` varchar(10) NOT NULL,
  `url` varchar(550) NOT NULL,
  `method` varchar(4) NOT NULL default 'get',
  `proxy` tinyint(1) NOT NULL default '1',
  KEY `engine_index` (`engine`),
  KEY `short_name_index` (`short_name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

You don’t need all of these fields for every record, and here is some useful records…

INSERT INTO plugins (id, short_name, engine, input_encoding, method, proxy, url)
VALUES (UUID(), ' JStor', 'jstor', 'UTF-8', 'get', 1, 'http://www.jstor.org/action/doBasicSearch?Query=');

INSERT INTO plugins (id, short_name, engine, input_encoding, method, proxy, url)
VALUES (UUID(), 'Google Your University', 'google', 'UTF-8', 'get', 1, 'http://www.google.com/univ/wm?hl=en&ie=ISO-8859-1&btnG=Google+Search&q=');

INSERT INTO plugins (id, short_name, engine, input_encoding, method, proxy, url)
VALUES (UUID(), 'Google Scholar', 'googlescholar', 'UTF-8', 'get', 1, 'http://scholar.google.com/scholar?&hl=en&lr=&btnG=Search&q=');

INSERT INTO plugins (id, short_name, engine, input_encoding, method, proxy, url)
VALUES (UUID(), 'Ebrary', 'ebrary', 'UTF-8', 'get', 1, 'http://site.ebrary.com/lib//Top?layout=search&nosr=1&sch=sch&frm=smp.x&p00=');

INSERT INTO plugins (id, short_name, engine, input_encoding, method, proxy, url)
VALUES (UUID(), 'EbscoHost', 'ebsco', 'UTF-8', 'get', 1, 'http://search.ebscohost.com/login.aspx?db=aph&direct=true&bQuery=');

INSERT INTO plugins (id, short_name, engine, input_encoding, method, proxy, url)
VALUES (UUID(), 'LexisNexis', 'lexisnexis', 'UTF-8', 'get', 1, 'http://www.lexisnexis.com/us/lnacademic/search/homesubmitForm.do');

INSERT INTO plugins (id, short_name, engine, input_encoding, method, proxy, url)
VALUES (UUID(), 'Proquest UMI', 'umi', 'UTF-8', 'get', 1, 'http://proquest.umi.com/pqdweb?RQT=305&FT=1&DBId=414&SQ=');

INSERT INTO plugins (id, short_name, engine, input_encoding, method, proxy, url)
VALUES (UUID(), 'Science Direct', 'sciencedirect', 'UTF-8', 'get', 1, 'http://www.sciencedirect.com/science/quicksearch?query=');

Apparently I wrote the queries with MDB2, I’ve since moved most of my interactions to Zend Db (you should be able to write that part pretty quickly) and just dump the names of the plugins in the tableon a view:

foreach($plugins as $plugin):

    echo('<li>' . $plugin['short_name'] . ': ');
       echo(' <a href="openSearch.php?id=' . $plugin['id'] . '"><img src="images/a9.png" title="OpenSearch" /></a>'); 

        if(strlen($plugin['url']) > 0){
        	echo(' <a href="search.php?engine=' . $plugin['engine'] . '&q=regulators">test search');
            echo(' | <a href="#" title="Ref: ' . $plugin['engine'] . '" onClick="addOpenSearch(\'' . $plugin['engine'] . '\', \'gif\', \'Education\', \'' . $plugin['id'] . '\'); return false;">add</a>');
        }

        echo('</li>);

');
endforeach;

The add code is straight-forward

$result = $result[0];

$short_name = $result['short_name'];
$description = $result['description'];
$engine = $result['engine'];
$id = $result['id'];

$xml = <<<EOT
<?xml version="1.0" encoding="UTF-8"?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/"
xmlns:moz="http://www.mozilla.org/2006/browser/search/">
<ShortName>$short_name</ShortName>
<Description>$description</Description>
<Url type="text/html" method="get" template="http://<your_server>/plugins/proxy.php?engine=$engine&q="/>
<Contact>libsys@wm.edu</Contact>
<Image width="16" height="16">http://<your_server>/favicon.ico</Image>
<Developer>Wayne Graham</Developer>
<InputEncoding>UTF-8</InputEncoding>
<moz:SearchForm>http://<your_server>/plugins/search.php?engine=$engine</moz:SearchForm>
<moz:UpdateUrl>http://<your_server>/plugins/opensearch.php?$id</moz:UpdateUrl>
<moz:IconUpdateUrl>http://swem.wm.edu/favicon.ico</moz:IconUpdateUrl>
<moz:UpdateInterval>7</moz:UpdateInterval>
</OpenSearchDescription>
EOT;
header("Content-Type: text/xml");
echo($xml);

You just need a little JavaScript to add this in to IE and Firefox:

function addOpenSearch(name,ext,cat,pid,meth)
{
  if ((typeof window.external == "object") && ((typeof window.external.AddSearchProvider == "unknown") || (typeof window.external.AddSearchProvider == "function"))) {
    // See bugs 430058/430067/430070 for Camino
    if (((typeof window.external.AddSearchProvider == "unknown") || (window.navigator.vendor == 'Camino'))&& meth == "p") {
      alert("This plugin uses POST which is not currently supported by your browser's implementation of OpenSearch.");
    } else {
      window.external.AddSearchProvider("http:///plugins/install.php?id=" + pid);
    }
  } else {
    alert("You will need a browser which supports OpenSearch to install this plugin.");
  }
}

If I get some time soon, I’ll wrap this up into a package; rudimentary, but a decent start to your own app!

If you’re not one of the of the cool kids who uses iGoogle, you’ll need to set yourself up to use this code. I just looked at the developer’s guide, and it seems a bit more fleshed out than it was when I first took a whack at the code The really cool thing about these gadgets is that they are essentailly just an XML file with some JavaScript/HTML thrown in.

I thought it might be a nice feature to just provide a search box for people to launch a catalog search. This is just a simple HTML form.

<form target="_parent" name="searchform" action="http://lion.wm.edu/uhtbin/cgisirsi/0/0/057/5" id="searchform">
    <input id="searchInput" style="width: 100%;" name="searchdata1" type="text" accesskey="f" />
    <input id="library" name="library" type="hidden" value="SWEM" />
    <input id="user_id" name="user_id" type="hidden" value="SWEPUB" />
    <input id="sourceid" name="sourceid" type="hidden" value="gadget" />
</form>

This is just HTML code that you would use on any search page. Nothing fancy. Now on to creating the gadget..

There are two parts to the XML code for the spec. The root node (Module) has two elements, ModulePrefs and Content. The ModulePrefs element is relatively straight-forward:

<ModulePrefs
    title="Swem Catalog Search"
    height="40"
    description="Search Swem's online catalog from iGoogle"
    author="Wayne Graham"
    author_email="dontspamme@gmail.com" screenshot=""
    author_location="Williamsburg, Virginia"
    title_url="http://swem.wm.edu">

    <Require feature="analytics" />

</ModulePrefs>

The attributes of the ModulePrefs element (title, height, description, etc.) should be pretty self explanitory. I wanted to be able to track usage of the gadget, so I eneabled Google analytics for the widget by requiring the analytics feature. Now for the meat.

What I did was write a short CSS definition and some JavaScript to actually display a form in the Content element of the Module.

<Content type="html"><![CDATA[
<style>
    #subLibrary {
        font-family:arial, sans-serif;
        font-size:10px;
        color:#676767;
    }
    #link {
        font-family:Garamond, serif;
        font-size:32px;
        color:black;
        text-decoration:none;
    }
    #searchGoButton {
        font-weight:bold;
    }
</style>
<script>
function displaySearchBar (prefs) {
    var lang = prefs.getString("mylang");

    if (lang == "") {
        lang = prefs.getString(".lang");
    }

    var html =
        '<form target="_parent" name="searchform" action="http://lion.wm.edu/uhtbin/cgisirsi/0/0/0/57/5" id="searchform">' +
        '<table style="width:100%">' +
        '<tr>' +
        '<td style="padding-bottom:4px">' +
        '<a target="_parent" id="link" href="http://lion.wm.edu/uhtbin/webcat">Catalog</a>' +
        '<span style="vertical-align: sub;">Swem</span>' +
        '</td>' +
        '<td width=100%>' +
        '<input id="searchInput" style="width:100%" name="searchdata1" type="text" accesskey="f" value="" />' +
        '<input id="library" name="library" type="hidden" value="SWEM" />' +
        '<input id="user_id" name="user_id" type="hidden" value="SWEPUB" />' +
        '<input id="sourceid" name="sourceid" type="hidden" value="gadget" />' +
        '</td>' +
        '<td>' +
        '<input type="submit" name="go" id="searchGoButton" value="Search" />' +
        '</td>' +
        '<td>' +
        ' ' +
        '</td>' +
        '</tr>' +
        '</table>'
        '<form>';
    document.write(html);
}

displaySearchBar(new _IG_Prefs(__MODULE_ID__));
_IG_Analytics("analytics-id", "/google-gadget");
</script>
]]>
</Content>

Once you’re done, you can test everything with the directions on the Developer Tools page. There are even really nice resources to tell you how to host this on Google Code.

As a finished product, this is what the gadget looks like in iGoogle:

For the folks that use iGoogle as their home page, this can be valuable tool for them. Next up, I need to brush off my code for generating Sherlock/Opensearch extensions for Firefox and IE…

I went back in and modified the XSLT that was processing the given names to now read:

<xsl:variable name="ref_id">
    <xsl:value-of select="@ID" />
</xsl:variable>
...
<xsl:for-each select="name">
INSERT INTO authors (id, ref_id, given, family, role)
VALUES(
    UUID(),
    <!--(SELECT id FROM citations ORDER BY added desc limit 1),-->
    '<xsl:value-of select="$ref_id"/>',
    <xsl:for-each select=".">
        '<xsl:for-each select="namePart[@type='given']">
            <xsl:value-of select="."/>
            <xsl:text> </xsl:text>
        </xsl:for-each>',

        '<xsl:value-of select="namePart[@type='family']"/>',
        '<xsl:value-of select="role/roleTerm"/>'
    </xsl:for-each>
    );
</xsl:for-each>

It looked like the SQL to grab the last item didn’t always return a record for whatever reason (a locking issue?). Since there was already a key in the field, I just switched over to use that in the inserts. Still need to go through and regularize the fields so MySQL isn’t attempting to pull a foreign key index on a variable width field, but this is an update to yesterday’s code.

The first thing I did is get them to give me the file in the crappy EndNote XML format. I was having issues bringing it into Zotero, so I installed the bibutils package (sudo apt-get install bibutils) and then ran the endx2xml program on it. This created a nice MODS file for me to start crunching on. What I wanted to do is make a tag cloud using some of the categories they created. I created a database with two tables to store the citation and author information. I’m on my development box, so this is MySQL

</p>
<p>--<br>
-- Create schema archaeology<br>
--</p>
<p>CREATE DATABASE IF NOT EXISTS archaeology;<br>
USE archaeology;</p>
<p>--<br>
-- Definition of table `archaeology`.`authors`<br>
--</p>
<p>DROP TABLE IF EXISTS `archaeology`.`authors`;<br>
CREATE TABLE&amp;nbsp; `archaeology`.`authors` (<br>
`id` char(36) NOT NULL,<br>
`citation_id` char(36) NOT NULL,<br>
`given` varchar(50) NOT NULL,<br>
`family` varchar(50) NOT NULL,<br>
`role` varchar(50) NOT NULL,<br>
PRIMARY KEY&amp;nbsp; (`id`)<br>
) ENGINE=InnoDB DEFAULT CHARSET=latin1;</p>
<p>--<br>
-- Definition of table `archaeology`.`citations`<br>
--</p>
<p>DROP TABLE IF EXISTS `archaeology`.`citations`;<br>
CREATE TABLE&amp;nbsp; `archaeology`.`citations` (<br>
`id` char(36) NOT NULL,<br>
`ref_id` varchar(25) NOT NULL,<br>
`title` varchar(500) NOT NULL,<br>
`year` int(11) NOT NULL,<br>
`publisher` varchar(500) NOT NULL,<br>
`added` timestamp NOT NULL default CURRENT_TIMESTAMP,<br>
`genre` varchar(50) NOT NULL,<br>
`subject` varchar(500) NOT NULL,<br>
PRIMARY KEY&amp;nbsp; (`id`)<br>
) ENGINE=InnoDB DEFAULT CHARSET=latin1;</p>
<p>

I next needed to write the XSLT to actually transform the modsCollection into the schema. I haven’t worked with XSLT in a while and was (very) rusty Here’s a sampling of the output of converting the EndNote XML to MODS

</p>
<p><?xml version="1.0" encoding="UTF-8"?><br>
<modsCollection><br>
<mods ID="Andre2007a"><br>
<titleInfo><br>
<title>Heritage Education and Interpretive Plan for the Smith River Blueway and Pedestrian Trails, Henry County, Virginia</title><br>
</titleInfo><br>
<name type="personal"><br>
<namePart type="given">Elizabeth</namePart><br>
<namePart type="family">Andre</namePart><br>
<role><br>
<roleTerm authority="marcrelator" type="text">author</roleTerm><br>
</role><br>
</name><br>
<name type="personal"><br>
<namePart type="given">Stephanie</namePart><br>
<namePart type="family">Sapp</namePart><br>
<role><br>
<roleTerm authority="marcrelator" type="text">author</roleTerm><br>
</role><br>
</name><br>
<originInfo><br>
<dateIssued>2007</dateIssued><br>
<publisher>William and Mary Center for Archaeological Research, Williamsburg, Virginia. &amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; Submitted to Virginia Department of Historic Resources, Roanoke</publisher><br>
</originInfo><br>
<typeOfResource>text</typeOfResource><br>
<genre>report</genre><br>
<subject><br>
<topic>06-25/L-WMCAR Reports/Henry Co/Interp/18th/19th/20th/</topic><br>
</subject><br>
<identifier type="citekey">Andre2007a</identifier><br>
</mods><br>
<mods ID="Andre2007b"><br>
<titleInfo><br>
<title>Architectural Evaluation of the Anderson Farm (047-0034-0010) for the Proposed Anderson-Hughes Development Project in Norge, Virginia</title><br>
</titleInfo><br>
<name type="personal"><br>
<namePart type="given">Elizabeth</namePart><br>
<namePart type="given">M</namePart><br>
<namePart type="family">Andre</namePart><br>
<role><br>
<roleTerm authority="marcrelator" type="text">author</roleTerm><br>
</role><br>
</name><br>
<originInfo><br>
<dateIssued>2007</dateIssued><br>
<publisher>William and Mary Center for Archaeological Research, Williamsburg, Virginia. &amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; Submitted to Anderson-Hughes, LLC, Newport News, Virginia</publisher><br>
</originInfo><br>
<typeOfResource>text</typeOfResource><br>
<genre>report</genre><br>
<subject><br>
<topic>07-22/L-WMCAR Reports/ARCeval/19th/20th/James City Co/</topic><br>
</subject><br>
<identifier type="citekey">Andre2007b</identifier><br>
</mods><br>
<modsCollection></p>
<p>

What I wanted to do is take each citation and insert it into the database, then relate the last-entered citation with its authors. At first I thought I could use MySQL’s last_insert_id() function…turns out those are only for auto-increment fields and I always use UUIDs for primary keys. What I ended up doing is adding in the added field with a timestamp (with a default value of CURRENT_TIMESTAMP). Then, I just went through the nodes and put the data where it needed to go.

</p>
<p><?xml version="1.0" encoding="UTF-8"?><br>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"></p>
<p><xsl:output method="text" /><br>
<xsl:strip-space elements="*"/></p>
<p><xsl:template match="/"></p>
<p>DELETE FROM citations;<br>
DELETE FROM authors;</p>
<p><xsl:for-each select="modsCollection/mods"></p>
<p>INSERT INTO citations ( id, ref_id, title, year, publisher, genre, subject)<br>
VALUES (<br>
UUID(),<br>
'<xsl:value-of select="@ID" />',<br>
'<xsl:call-template name="escapesinglequotes"><br>
<xsl:with-param name="arg1"><br>
<xsl:value-of select="titleInfo/title"/><br>
</xsl:with-param><br>
</xsl:call-template>',<br>
'<xsl:value-of select="originInfo/dateIssued"/>',<br>
'<xsl:call-template name="escapesinglequotes"><br>
<xsl:with-param name="arg1"><br>
<xsl:value-of select="originInfo/publisher"/><br>
</xsl:with-param><br>
</xsl:call-template>',<br>
'<xsl:value-of select="genre"/>',<br>
'<xsl:call-template name="escapesinglequotes"><br>
<xsl:with-param name="arg1"><br>
<xsl:value-of select="subject/topic"/><br>
</xsl:with-param><br>
</xsl:call-template>'<br>
);</p>
<p><xsl:for-each select="name"><br>
INSERT INTO authors (id, citation_id, given, family, role)<br>
VALUES(<br>
UUID(),<br>
(SELECT id FROM citations ORDER BY added desc limit 1),<br>
<xsl:for-each select="."><br>
'<xsl:value-of select="namePart[@type='given']"/> ',<br>
'<xsl:value-of select="namePart[@type='family']"/>',<br>
'<xsl:value-of select="role/roleTerm"/>'<br>
</xsl:for-each><br>
);<br>
</xsl:for-each><br>
</xsl:for-each><br>
</xsl:template></p>
<p><xsl:template name="escapesinglequotes"><br>
<xsl:param name="arg1" /></p>
<p><xsl:variable name="apostrophe">'</xsl:variable></p>
<p><xsl:choose><br>
<xsl:when test="contains($arg1, $apostrophe)"><br>
<xsl:if test="string-length(normalize-space(substring-before($arg1, $apostrophe))) > 0"><xsl:value-of select="substring-before($arg1, $apostrophe)" disable-output-escaping="yes"/>''</xsl:if><br>
<xsl:call-template name="escapesinglequotes"><br>
<xsl:with-param name="arg1"><xsl:value-of select="substring-after($arg1, $apostrophe)" disable-output-escaping="yes"/></xsl:with-param><br>
</xsl:call-template></p>
<p></xsl:when></p>
<p><!-- No quotes found in string, just print it --><br>
<xsl:when test="string-length(normalize-space($arg1)) > 0"><xsl:value-of select="normalize-space($arg1)"/></xsl:when></p>
<p></xsl:choose><br>
</xsl:template></p>
<p></xsl:stylesheet></p>
<p>

I noticed that I also had to write a template to escape single quotes in the title, publisher, and topic because of some possesive language in those fields. It took the longest time to actually write the code for that (like I said, I was really rusty and forgot all about XSL branching).

In the end, what gets generated is a nice SQL script (this one pushes 30,000 lines) like this

</p>
<p>DELETE FROM citations;<br>
DELETE FROM authors;</p>
<p>INSERT INTO citations ( id, ref_id, title, year, publisher, genre, subject)<br>
VALUES (<br>
UUID(),<br>
'Andre2007a',<br>
'Heritage Education and Interpretive Plan for the Smith River Blueway and Pedestrian Trails, Henry County, Virginia',<br>
'2007',<br>
'William and Mary Center for Archaeological Research, Williamsburg, Virginia. Submitted to Virginia Department of Historic Resources, Roanoke',<br>
'report',<br>
'06-25/L-WMCAR Reports/Henry Co/Interp/18th/19th/20th/'<br>
);</p>
<p>INSERT INTO authors (id, citation_id, given, family, role)<br>
VALUES(<br>
UUID(),<br>
(SELECT id FROM citations ORDER BY added desc limit 1),<br>
'Elizabeth ',<br>
'Andre',<br>
'author'<br>
);</p>
<p>INSERT INTO authors (id, citation_id, given, family, role)<br>
VALUES(<br>
UUID(),<br>
(SELECT id FROM citations ORDER BY added desc limit 1),<br>
'Stephanie ',<br>
'Sapp',<br>
'author'<br>
);</p>
<p>INSERT INTO citations ( id, ref_id, title, year, publisher, genre, subject)<br>
VALUES (<br>
UUID(),<br>
'Andre2007b',<br>
'Architectural Evaluation of the Anderson Farm (047-0034-0010) for the Proposed Anderson-Hughes Development Project in Norge, Virginia',<br>
'2007',<br>
'William and Mary Center for Archaeological Research, Williamsburg, Virginia. Submitted to Anderson-Hughes, LLC, Newport News, Virginia',<br>
'report',<br>
'07-22/L-WMCAR Reports/ARCeval/19th/20th/James City Co/'<br>
);</p>
<p>INSERT INTO authors (id, citation_id, given, family, role)<br>
VALUES(<br>
UUID(),<br>
(SELECT id FROM citations ORDER BY added desc limit 1),<br>
'Elizabeth ',<br>
'Andre',<br>
'author'<br>
);</p>
<p>

There’s still a lot of cleaning up to do (normalizing the authors table, exploding the subjects, generating the tags), but it did give me pause to remember how beautifully maddening XSLT can be.

I’ll post some screenshots when this gets a bit further along, but I thought this may save someone some time at some point.

A while back I got a couple of Sun Thumpers (24TB each) to help out with our storage needs in the library. I’ve gotten to the point to deploy them. I have to admit I really dig the Solaris admin tools (web based ILOM) and the ZFS configuration tools.

The idea for the first Thumper is to make that the primary storage for of content for our institutional repository (we’re running DSpace). The last couple of days I’ve been getting down to a really geeky level in systems administration. I created an NFS share, but the question was on how to mount it. We use nfs3 in the integrated graphics labs to mount the high performance computing clusters, so I did this for DSpace and added an fstab entry

<server>:/dspace /dspace_nfs nfs rw,nosuid,hard,intr,rsize=32768,wsize=32768,tcp 0 0

I then ran some test to see what kind of throughput I could get.

time dd if=/dev/zero of=/dspace_nfs/dspace/testfile bs=16 count=16384

I got a result of

262144 bytes (262 kB) copied, 0.088866 seconds, 2.9 MB/s

Then, just to compare it against normal disk I/O, I did the same on the local hard drive

time dd if=dev/zero of=~/testfile bs=16 count=16384

This was quite a bit faster (as you probably would guess)

262144 bytes (262 kB) copied, 0.041865 seconds, 6.3 MB/s

To test the read speed, I took the same file and brought it back.

time dd if=/dspace_nfs/dspace/testfile of=/dev/null bs=16

On the NFS mount I got

262144 bytes (262 kB) copied, 0.017822 seconds, 14.7 MB/s

and on the local file system I got

262144 bytes (262 kB) copied, 0.015867 seconds, 16.5 MB/s

I was kind of surprised at how close the reads were. It’s not a huge file it’s moving (our larger files are around 350MB), but it’s reasonably consistence with an arbitrary file size of the PDFs we have in the collection.

I was corresponding with a colleague who’s been doing this way longer than I have and he suggested using a larger blocksize and writing/reading some files larger than the RAM size. I also wanted to test against NFS4 (I was using NFS3 in the fstab entry).

To do this, I just created a mount point and mounted the filesystem.

mkdir /mnt/nfs4
mount -t nfs4 <server>:/dspace /mnt/nfs4

I ran the same test on the new mount and got a result of

262144 bytes (262 kB) copied, 0.055023 seconds, 4.8 MB/s

and for the read

262144 bytes (262 kB) copied, 0.014205 seconds, 18.5 MB/s

So, it looks like NFSv4 is much improved over NFSv3!

CoreContainer cc = new CoreContainer(solrCoreDir, configFile);
solrCore = cc.getCore(solrCorename);

Ok, so that was more of an update, but it got me to reindex the our marc content. Our 001 fields don’t always have info, so our Systems Librarian told me just to use the 035a for the id field. There was only one problem…they had a “(Sirsi)” as a prefix to the actual number. This wouldn’t be that big of an issue, but it was causing the queries sent to Solr to be off, and I wasn’t getting any results for a detail view.

Fortunately, Solrmarc already has a facility to deal with this: PatternMaps! To pull out just the number, I changed the id definition in the vufind.properties file to

id = 035a, (pattern_map.id), first

This basically maps the id field in the schema.xml file to the 035a, using a patternMap named pattern_map.id, and only takes the first of the 035a if there are multiples. I know, there shouldn’t be, but I’ve seen funky things

Now, to add the PatternMap definition:

pattern_map.id.pattern_0 = \\(Sirsi\\)\\ (.*)=>$1

It’s just a simple regex pattern definition that takes out the “(Sirsi)” and the space between it and the actual id we’re using.