The ability to create any number of fields to index in different ways (along with faceting) is a real strong point of Solr. Not only can you add fields and choose how that data is analyzed, you can create your own field types that process the information in your index the way you want them to be.

This is done in the $SOLR_HOME/config/schema.xml file. The first section (<types>) defines the types of fields that you will be using, and how Solr should process them with Lucene. If you look at some of the fieldtypes, you’ll get an idea of what’s possible. For instance, the fieldtype for “string” is an untokenized field that doesn’t normalize the fields and sorts missing information last.

However, if you need a more robust fieldtype, look at the fieldtype for “text”. This uses a whitespace tokenizer (splits words with whitespace) and uses the stopwords defined in the stopwords.txt file. It does some other processing (filters words, converts them to lowercase, runs a porter stemmer, and then removes duplicates). This fieldtype also defines what to do when a query is passed to it (uses the same filters). This is slightly different than the defined “textTight” which does not perform any further analysis on the text when being queried. You’ll probably find that most of these work for most instances, but if you need to, you can build your own fieldtype that has very specific indexing and query filters.

The next section contains the actual fields you want to use in the aptly named “fields” element. This is where you actually define the fields that will be in your index, the type of analysis to perform on the field, if it should be indexed, stored, have term vectors, or be multivalued (have multiple instances of the same field in the index).

Let’s say you’re wanting to develop an indexing schema for books (hey, I work in a library). At a very basic level, you’d want a field for an id, title, author, reviews, and a set of topics (or tags). Your fields element would contain something along the lines of:

You’ll notice that I have a couple of extra fields for title, author, and topic, these are for the faceting info and are just untokenized fields to make the calculations for facets a little more efficient.

Now, we’re almost done with creating the schema. We just need to declare a unique key, default search field, and default search operator.

Remember when I made the fields with the “Str” suffix? We can use a really cool feature of Solr called a “copyField” that literally copies the information from one field to another.

It’s worth mentioning here that Solr indexes are not databases! While there are some similarities in the way that Solr allows you to add, update, select, store, and delete information from the index, Solr isn’t an RDBMS. I’ve seen a few discussions where there is some confusion as to why Solr can’t do the equivalent of a stored procedure, or some other function of a database.

Now, your index server is ready to receive documents to search against. The server, in with the above example as the schema, will expect information to be in the following format:

Next week when I get some time, I’ll write about creating facet queries…

I ran into a bit of a situation that was really blowing my mind. I have a rather large XML file (around 20,000+ lines) marked up in TEI that I wanted to do some transformations on (a day book and ledger from the 1850s). Essentially the code follows the format

What I wanted to accomplish was group all this together in separate divs for HTML output (ok, I actually need to write each page to its own file, but this is pretty much just one more step).

I just could not find a way to group this info this way using XSLT 1 without wrapping each page within its own div structure. I didn’t really want to go back and do this, so I asked the TEI-L list. David Sewell pinged me back with some XQuery code that recursively recalls the document structure for a given node.

He also mentioned that it would be pretty easy to write an XSLT 2 transformation that groups these nodes together. I did a little bit of digging and came up with

This transformed the pages to what I was wanting

The XSLT processor for ColdFusion doesn’t support XSLT 2.0 (it’s still a draft spec). However, Saxon does (specifically Saxon 8). For more on doing XSLT transformations, see XSLT 2.0 in ColdFusion. ]]> http://www.liquidfoot.com/2006/08/23/real-life-xslt-20-transformations/feed/ 0 http://www.liquidfoot.com/2006/08/21/getting-xml-from-mssql-server/ http://www.liquidfoot.com/2006/08/21/getting-xml-from-mssql-server/#comments Mon, 21 Aug 2006 18:49:57 +0000 Wayne http://www.liquidfoot.com/?p=210 I’ve been playing with all the AJAX stuff that’s been coming out lately. I suppose that like a lot of folks, I was creating a query, then having a generic function that created the XML in a proxy file for the JavaScript (Ray Camden has a really nice function for transforming a query to XML).

Last week I was doing some research to find a way to do some XML searching and stumbled upon the FOR XML statement. I knew that most RDBMSs were capable of dealing with XML record sets, but it’s been years since I’ve even looked at any of the XML stuff for MSSQL.

The FOR XML statement returns a query result and transforms rows into XML elements. There are three arguments that this can take:

• RAW: Transforms each row into an element with a generic identifier (<row/>) as the element tag.
• AUTO: Returns the results in a simple nested XML tree
• EXPLICIT: Allows you to define the XML tree returned