It’s been a long while since I posted anything here. Things have been a bit hectic with the launching of a new institutional repository service (using DSpace), taking a computer graphics course (linear algebra, ray tracing, scientific visualization, etc.), and finishing up my book on the Facebook API.

This is the first book I’ve written and I have to admit it was far more of a task than I had first thought it would be. Not that the subject matter was very dense, but there were at least four major changes in the API that required an almost total rewrite of the code base for the examples. There were even a couple of sections that had to get cut because Facebook “fixed” their code because of security and user concerns (plus a lot of folks “abuse” some of the things you could do with the API).

There are some great Coldfusion resources for building Facebook applications in ColdFusion. The “un-official” ColdFusion client library for Facebook apps is at nearpersonal. There is also a RIAForge starter project for FBML (FBML is a tag based language for Facebook) named Facebook FBML Starter Kit.

So, if you’re wanting a short book that highlights some of the more common elements of developing Facebook applications, be sure to check out my book!

I’ve been knocked out by a really bad cold the last couple of weeks and I’m just starting to get things back to normal. I did want to send a quick post on the status of the SolColdfusion project. After seeing Ray’s Seeker project, it reminded me that I hadn’t set up a project at RIAForge yet, so I took care of that last night. You can now access the official project at http://solcoldfusion.riaforge.org/.

Since the project site is up-and-running, I also submitted it to the Solr project SOLR-404 (I know it’s a coincidence that it’s number 404, but it makes me wonder if it’s some type of bad omen …). Anyway, if you think the client should get included in the project, be sure to vote for it!

I’ve also been working on a generic interface, much like Erik Hatcher’s Solr Flare. It’s been a bit slow coming as I’ve not had a lot of time to work on these projects, but hopefully things will settle down shortly so I can devote a bit more time to them. ]]> http://www.liquidfoot.com/2007/10/31/solcoldfusion-update/feed/ 0 http://www.liquidfoot.com/2007/10/05/solr-schema/ http://www.liquidfoot.com/2007/10/05/solr-schema/#comments Fri, 05 Oct 2007 20:27:02 +0000 Wayne http://www.liquidfoot.com/?p=178 If you’ve ever worked on a project that involved Coldfusion’s bundled version of Verity, you’ve no doubt run into the issue of trying to confine your fields into the structure that Verity imposes, and those custom fields are really precious in these instances. About 6 months ago, I ran into an issue with a search project where I had about 125,000 documents to index. Since we also wanted to be able to use the indexes for some other projects, I was a bit nervous to commit almost the entire allotment of indexable objects to one collection. This launched me into writing a custom search engine and indexer using Lucene and slapping Coldfusion around the responses to do things that Verity did. However, once the projects were complete, I never really got around to making it easy to use. It does cool stuff like search across multiple collections, context highlighting, relevancy calculations, term vector calculations, “did you mean”, etc. Essentially everything I think all good search engines need to be able to do. Something this system lacked was an easy way to define the fields that you wanted indexed (along with a knowledge of Java to actually make the changes).

The ability to create any number of fields to index in different ways (along with faceting) is a real strong point of Solr. Not only can you add fields and choose how that data is analyzed, you can create your own field types that process the information in your index the way you want them to be.

This is done in the $SOLR_HOME/config/schema.xml file. The first section (<types>) defines the types of fields that you will be using, and how Solr should process them with Lucene. If you look at some of the fieldtypes, you’ll get an idea of what’s possible. For instance, the fieldtype for “string” is an untokenized field that doesn’t normalize the fields and sorts missing information last.

However, if you need a more robust fieldtype, look at the fieldtype for “text”. This uses a whitespace tokenizer (splits words with whitespace) and uses the stopwords defined in the stopwords.txt file. It does some other processing (filters words, converts them to lowercase, runs a porter stemmer, and then removes duplicates). This fieldtype also defines what to do when a query is passed to it (uses the same filters). This is slightly different than the defined “textTight” which does not perform any further analysis on the text when being queried. You’ll probably find that most of these work for most instances, but if you need to, you can build your own fieldtype that has very specific indexing and query filters.

The next section contains the actual fields you want to use in the aptly named “fields” element. This is where you actually define the fields that will be in your index, the type of analysis to perform on the field, if it should be indexed, stored, have term vectors, or be multivalued (have multiple instances of the same field in the index).

Let’s say you’re wanting to develop an indexing schema for books (hey, I work in a library). At a very basic level, you’d want a field for an id, title, author, reviews, and a set of topics (or tags). Your fields element would contain something along the lines of:

You’ll notice that I have a couple of extra fields for title, author, and topic, these are for the faceting info and are just untokenized fields to make the calculations for facets a little more efficient.

Now, we’re almost done with creating the schema. We just need to declare a unique key, default search field, and default search operator.

Remember when I made the fields with the “Str” suffix? We can use a really cool feature of Solr called a “copyField” that literally copies the information from one field to another.

It’s worth mentioning here that Solr indexes are not databases! While there are some similarities in the way that Solr allows you to add, update, select, store, and delete information from the index, Solr isn’t an RDBMS. I’ve seen a few discussions where there is some confusion as to why Solr can’t do the equivalent of a stored procedure, or some other function of a database.

Now, your index server is ready to receive documents to search against. The server, in with the above example as the schema, will expect information to be in the following format:

Next week when I get some time, I’ll write about creating facet queries…

To get up and running with Solr, you’ll need some type of Servlet container. Typically when folks start talking about servlet containers, they’re talking about Tomcat or Jetty. In fact, Solr comes with Jetty 6.1.3 (they haven’t upgraded to 6.1.5 yet in the distribution). You may also hear about Resin, but in my experience, it runs a bit slower than Jetty and Tomcat. As a small note, servlet containers are different than J2EE application servers like JRun, Geronimo, GlassFish, and JBoss (which use servlet containers like Tomcat and Jetty, but also have EJB containers and can handle other types of logic). If you have a J2EE application server running, you can easily use Solr, and if not, consider using Jetty or Tomcat as your container server.

Since your environment can be as varied as there are IT departments, I won’t try to cover everything. Essentially you need to have at least the Java 1.5 JRE. However, I would strongly suggest the most current Java JDK (and not the JRE) as it has performance enhancements to run in server mode (with -server). If you don’t already have this Java version installed on your server (assuming this is the same server running CF), don’t worry, ColdFusion will still work if you install the required Java runtime.

Essentially the process for deploying Solr, once you have a servlet container up-and-running is to drop the solr.war file into the webapps directory on the server. It won’t do anything at this point as you need to set the configuration files for Solr. The easiest way to do this is copy the files from example/solr into a new directory (which I will refer to now as solr_home).

You can tell Java about the home directory by setting the solr.solr.home (-Dsolr.sol.home), set the JNDI lookup (“java:comp/env/solr/home”), or just throw it into the JVM’s working directory (the default path is ./solr). Now you just need to make sure everything is running. Just point your browser to http://<server>:<port>/solr/admin. You should then see the administration interface (you may need to restart your servlet container to get everything working properly), but it’s not an administrative interface like you get in CFAdmin. This is more of an informational administration panel. You can make sure everything is running, that there are documents in your index is set up properly, check out the schema and configuration files, and thread information. Really the only thing you can administer here is the log level.

For some more specific notes on intalling Solr in Tomcat and Jetty, check out Solr’s wiki. In particular, if you’re going to need multiple instances of Solr to run, pay attention to the sections on Multipe Solr apps on those wiki pages. ]]> http://www.liquidfoot.com/2007/10/05/solr-and-coldfusion-setting-up/feed/ 0 http://www.liquidfoot.com/2007/10/04/coldfusion-solr-client-solcoldfusion/ http://www.liquidfoot.com/2007/10/04/coldfusion-solr-client-solcoldfusion/#comments Thu, 04 Oct 2007 22:19:47 +0000 Wayne http://www.liquidfoot.com/?p=182

As I hinted at yesterday, I was close to having some code in the pipeline to abstract using Solr. I’ve finished the initial code with the following built in. Here’s a brief setup guide to start playing with the code.

First, you’re going to need to grab the latest release version of Solr (currently 1.2). The only real requirement to run this software is that you have a JRE of 1.5 or higher. Untar/zip the file somewhere convenient and open a command prompt. Get to the example directory in the apache-solr.1.2.x folder (cd /example). To start up the sample server running Jetty, just issue the following command:

This will start a new instance of the Solr server on your computer on port 8983. You can make sure this is running by navigating to http://localhost:8983/solr (NOTE: this is a link to your computer. If you get an error, it’s because your computer isn’t running an instance of Solr on port 8983).

At this point, it’s probably good to send you over to the Solr website to take a look at their tutorial. Go ahead. I’ll wait…

…

Great, you’re back.

You’ve seen some basic inserting, deleting, and querying of Solr index data. You may have also noticed that there are clients for PHP, Ruby, Python, and Java…no ColdFusion. I want to do a little more testing on this before I submit the patch, but I’ve added the initial code as an encosure here to do updating, deleting, and searching in Coldfusion.

The CFC SolColdfusion should be in the path org/apache/client (at least that’s where I’m putting in for the purposes of this initial demonstration). The initialization takes one required parameter (the Solr host) and then has two optional parameters (port and path).

To set this up, create an instance with

Now, there are a lot of different parameters you can send to Solr to perform different queries. And, since some of these key names can repeat, I chose to implement sending these parameters as an array. So, let’s set this up.

These parameters are basically what are the defaults that Solr will return back to you. If you want highlighting, you would need to add two additional row vectors with ‘hl = on’ and ‘hl.fl = ‘.

Searching is straight forward, taking a query, the start row, number of rows to return, and the array of parameters:

This searches all fields and all content and returns back an XML document with the search results in it.

In the result node, you’ll see that Solr returns an xmlAttribute of

of 0 (assuming you don’t have anything in the index). Let’s add an example document from the documents that come with Solr.

You’ll notice I used a commit and optmize statement. Neither of these statements are necessary every time you add a document, but be aware that Solr caches documents and won’t flush the new documents to disk unless you either commit the documents or the mergefactor setting you used in your solrconfig.xml file has been reached.

Now, let’s delete this document…

Don’t forget to commit deletions to the index!

There’ll be more soon (add multiple documents, delete by queries). In the mean time, try it out. If you have any comments, questions, concerns, whatever, let me know.

I’ve spent the last few months working on some projects that didn’t really have anything to do with ColdFusion (lots of Java and PHP). One of the projects I’ve been working with (Vufind.org) uses Solr as it’s indexing/search engine. That’s starting to get picked up by some pretty big companies (Netflix just relaunched their search using Solr this week).

I’ve been working with Solr in Java for a bit now, and I wanted to start to build an interface for using it as a search engine (my Lucene code is stuck in open source limbo) in Coldfusion. One of the cool things about Solr is that it returns results back through HTTP (in XML, JSON, or ruby).

As soon as I get the code finished, I’ll post it as a patch in Solr.

Ok, it’s been a while. I’ve been working on some non-ColdFusion projects so I’ve been a bit remiss in keeping things up-to-date here. But big news (at least for William and Mary). Her Royal Majesty, Queen Elizabeth II will be visiting William and Mary this Friday as part Jamestown’s 400th anniversary.

Normally this wouldn’t be related to ColdFusion at all, but last week it was announced that she would be coming here and thus ensued a mad rush to get an online exhibit of the Queen’s 1957 visit up that would make available pretty much everything we have stored in our Special Collections from her previous visit. I got to use Model-Glue after a long time of not looking at it. I had almost forgotten how painless Model-Glue makes putting projects like this together.

I just wanted to give a big thanks to Joe Rinehart and Doug Hughes for bringing the “rapid” back into rapid application development!

We recently made the move from an IIS Windows web server to an Apache *nix based web server as part of our efforts to consolidate our library’s server infrastructure. And for reasons I won’t expound upon, we had one MS Access DSN that didn’t get migrated to MSSQL and that needed to be used still. Since ColdFusion uses a Windows only driver for MS Access, I needed to figure out a way around this. I found a couple of JDBC drivers for Access (Easysoft’s JDBC-ODBC Bridge and HXTT’s Access Pure Java JDBC Drivers), but these seemed to be a bit on the expensive side for the short amount of time that I’d need to keep Access in production.

I did notice on Easysoft’s website that they were using the JdbcOdbc bridge, so after a little bit more digging, I found the syntax to use configure ColdFusion to use MS Access through the JdbcOdbc Bridge; the JDBC URL is

and the Driver Class

For the very basic inserting of data from a seldom-used web form into a single table, this band aid fix has been doing pretty good!

I’ve been playing with some form validation stuff for CF. I had been usign , but I wanted the HTML interface to act a bit more like the Flash interface, but I don’t really want to use Flash. I’ve also been doing a lot more work with some of the DHTML libraries that AJAX has made popular, so I figured there had to be a relatively elegent way to do form validations with something like Ajaxian about easy form validation and decided to give it a try. The article on Dexagogo shows how they created a library to handle form-validations that doesn’t require any other work than creating a form. This was just what I was looking for!

Basically, you just need the latest files from script.aclo.us with the latest prototype version (the 1.5 release candidate is included in the latest script.aculo.us lib folder), and the validation library. Convienently, they’re all included in the demo file on the site.

To use this, you really only need prototype and the validation library (script.aculo.us adds a nice effect – much like the Flash format in cfform). For me, I made these calls:

This is slightly different than the example on their page; they load the effects.js file directly, I’m calling the library via script.aculo.us with the load parameter. This isn’t really a big deal for one library, but it is convenient when you want to use several, but not all, of the libraries (e.g. scriptaculous.js?load=effects,dragdrop,slider).

Anyway, to actually use this, you need to create a form with an id attribute:

Now, we add some fields and use the class attribute to call the validator:

There are 11 options for use in the validation library (this is directly off their page):

• required (not blank)
• validate-number (a valid number)
• validate-digits (digits only)
• validate-alpha (letters only)
• validate-alphanum (only letters and numbers)
• validate-date (a valid date value)
• validate-email (a valid email address)
• validate-url (a valid URL)
• validate-date-au (a date formatted as; dd/mm/yyyy)
• validate-currency-dollar (a valid dollar value)
• validate-one-required (At least one textbox/radio element must be selected in a group)

This is really nice, because if you want to allow an optional field, but validate it, you can do:

There’s one more piece of the pie…to call the validation library. At the bottom of your page add:

The first argument is the id attribute of the form you’re wanting to validate. The second tells the Validator object what to do. This particular example enables validation on each field as you leave it (which I find useful). Some of the other options are:

• stopOnFirst (boolean): Stop on the first validation failure; default: false
• onSubmit (boolean): Override the default behavior of adding an even listener to the onsubmit event (set to false if you want to make sure your onsubmit method gets called no matter what); default: true
• immediate (boolean): validate when the cursor leaves the field; default: false
• focusOnError (boolean): place the focus on the first field with an error; default: true
• useTitles (boolean): make field validators use form element title attributes as error advice message; default: false
• onFormValidate (string function): call a function when the form is validated
• onElementValidate (string function): call a function when an element is validated

What I thought was really cool was the ability to add custom validation types via an API. Say you only want folks to use capital letters for their names, you simply add a new validation type like:

Want to add several? You can do that too:

This should be all of the normal items included in (plus a couple extra for good measure). Now the only thing left is to make it look pretty. One of the nice things about the Flash format in is that it color codes required fields with the different halo effects. To obtain a similar effect in the forms, we’ll use style sheets instead.

This is a rather light stylesheet, but it’ll give you something to start with (based on the default haloGreen skin):

I made a short example of some of the validations at http://swem.wm.edu/blogs/waynegraham/examples/validation/. I have to say that I’ve found this to be a bit better solution (at least for my needs) than using ! ]]> http://www.liquidfoot.com/2006/09/21/form-validation/feed/ 0 http://www.liquidfoot.com/2006/08/23/real-life-xslt-20-transformations/ http://www.liquidfoot.com/2006/08/23/real-life-xslt-20-transformations/#comments Wed, 23 Aug 2006 18:56:31 +0000 Wayne http://www.liquidfoot.com/?p=208

I ran into a bit of a situation that was really blowing my mind. I have a rather large XML file (around 20,000+ lines) marked up in TEI that I wanted to do some transformations on (a day book and ledger from the 1850s). Essentially the code follows the format

What I wanted to accomplish was group all this together in separate divs for HTML output (ok, I actually need to write each page to its own file, but this is pretty much just one more step).

I just could not find a way to group this info this way using XSLT 1 without wrapping each page within its own div structure. I didn’t really want to go back and do this, so I asked the TEI-L list. David Sewell pinged me back with some XQuery code that recursively recalls the document structure for a given node.

He also mentioned that it would be pretty easy to write an XSLT 2 transformation that groups these nodes together. I did a little bit of digging and came up with

This transformed the pages to what I was wanting

The XSLT processor for ColdFusion doesn’t support XSLT 2.0 (it’s still a draft spec). However, Saxon does (specifically Saxon 8). For more on doing XSLT transformations, see XSLT 2.0 in ColdFusion. ]]> http://www.liquidfoot.com/2006/08/23/real-life-xslt-20-transformations/feed/ 0