LiquidFoot » lucene

Luke for Lucene 2.4

Wayne — Thu, 21 Aug 2008 19:13:37 +0000

On the Vufind project, I’ve been migrating the Solrmarc codebase to use Solr 1.3. It’s got a lot of nifty changes (the multi-core and DataImportHandler are really nice additions), including moving to Lucene 2.4 for the actual indexing.

On the down side, this means that Luke doesn’t have the correct analyzers or format interpretors (you get an incorrect format excpetion).

The fix is pretty quick. Just download the source tarball/zip for Luke (at the bottom of the page) and extract them somewhere. Then, grab a build of the Lucene 2.4 core, analyzers, and snowball analyzer (I grabbed mine from the Solr nightly build).

Just throw the jars into Luke’s lib folder (e.g. ~/luke-src-0.8-dev/lib) and then rebuild with Ant (ant). This will build the jars in the dist folder. You should be able to check out your index to make sure everything is good in the index.

If you’re lazy and just want the one I created, here you go: luke-08-dev

Coldfusion Solr Client – SolColdfusion

Wayne — Thu, 04 Oct 2007 22:19:47 +0000

As I hinted at yesterday, I was close to having some code in the pipeline to abstract using Solr. I’ve finished the initial code with the following built in. Here’s a brief setup guide to start playing with the code.

First, you’re going to need to grab the latest release version of Solr (currently 1.2). The only real requirement to run this software is that you have a JRE of 1.5 or higher. Untar/zip the file somewhere convenient and open a command prompt. Get to the example directory in the apache-solr.1.2.x folder (cd /example). To start up the sample server running Jetty, just issue the following command:

java -jar start.jar

This will start a new instance of the Solr server on your computer on port 8983. You can make sure this is running by navigating to http://localhost:8983/solr (NOTE: this is a link to your computer. If you get an error, it’s because your computer isn’t running an instance of Solr on port 8983).

At this point, it’s probably good to send you over to the Solr website to take a look at their tutorial. Go ahead. I’ll wait…

…

Great, you’re back.

You’ve seen some basic inserting, deleting, and querying of Solr index data. You may have also noticed that there are clients for PHP, Ruby, Python, and Java…no ColdFusion. I want to do a little more testing on this before I submit the patch, but I’ve added the initial code as an encosure here to do updating, deleting, and searching in Coldfusion.

The CFC SolColdfusion should be in the path org/apache/client (at least that’s where I’m putting in for the purposes of this initial demonstration). The initialization takes one required parameter (the Solr host) and then has two optional parameters (port and path).

To set this up, create an instance with

“component”, “org.apache.solr.client.SolColdfusion”).init(“http://localhost“, “8983″, “/solr”) />

Now, there are a lot of different parameters you can send to Solr to perform different queries. And, since some of these key names can repeat, I chose to implement sending these parameters as an array. So, let’s set this up.

1) />

“indent”>
“on” />
“wt”>
“standard” />
“fl” />
“*,score” />
“qt” />
“standard” />
“wt” />
“standard” />

These parameters are basically what are the defaults that Solr will return back to you. If you want highlighting, you would need to add two additional row vectors with ‘hl = on’ and ‘hl.fl = ‘.

Searching is straight forward, taking a query, the start row, number of rows to return, and the array of parameters:

“*:*”, 0, 10, params) />

This searches all fields and all content and returns back an XML document with the search results in it.

“#results#” />

In the result node, you’ll see that Solr returns an xmlAttribute of

numFound

of 0 (assuming you don’t have anything in the index). Let’s add an example document from the documents that come with Solr.

“sample”>

“id”>F8V7067-APL-KIT
“name”>Belkin Mobile Power Cord for iPod w/ Dock
“manu”>Belkin
“cat”>electronics
“cat”>connector
“features”>car power adapter, white
“weight”>4
“price”>19.95
“popularity”>1
“inStock”>false

“id:F8V7067-APL-KIT”, 0, 10, params) />

“#xmlParse(results)#” />

You’ll notice I used a commit and optmize statement. Neither of these statements are necessary every time you add a document, but be aware that Solr caches documents and won’t flush the new documents to disk unless you either commit the documents or the mergefactor setting you used in your solrconfig.xml file has been reached.

Now, let’s delete this document…

“F8V7067-APL-KIT”) />

Don’t forget to commit deletions to the index!

There’ll be more soon (add multiple documents, delete by queries). In the mean time, try it out. If you have any comments, questions, concerns, whatever, let me know.

ColdFusion and Solr

Wayne — Wed, 03 Oct 2007 22:38:43 +0000

I’ve spent the last few months working on some projects that didn’t really have anything to do with ColdFusion (lots of Java and PHP). One of the projects I’ve been working with (Vufind.org) uses Solr as it’s indexing/search engine. That’s starting to get picked up by some pretty big companies (Netflix just relaunched their search using Solr this week).

I’ve been working with Solr in Java for a bit now, and I wanted to start to build an interface for using it as a search engine (my Lucene code is stuck in open source limbo) in Coldfusion. One of the cool things about Solr is that it returns results back through HTTP (in XML, JSON, or ruby).

As soon as I get the code finished, I’ll post it as a patch in Solr.

Fun with PDFs

Wayne — Fri, 18 May 2007 00:12:25 +0000

I’ve been working with a lot of PDF files lately for a few different projects (see The FlatHat and Card Catalog). With our special collections cards, when you got a result back, Acrobat viewer would blow up the image to around 600%, making for a rather ugly image. For the FlatHat, I really wanted to be able to open a PDF and have the search terms highlighted, so I started hunting for ways to actually do this.

I’ve been using PDFBox to extract text from our PDFs to index with Lucene, so I started there and they clued me in to Adobe’s PDF Open Parameters. This really killed a few birds with one stone.

When I was working on the Flat Hat newspaper, I was originally only returning back the page that the search result was on. I had some misgivings about this (like what if the story was on more than one page), but being able to pass the search query from the engine into the PDF is really nice since the user doesn’t have to search through the entire issue to find the the context they are searching for (e.g. whistle bait — when I saw that term, I cracked up; definitely a different era).

Basically, the PDF Open Parameters allow you to pass commands into a PDF to allow you to control how the PDF is opened. They’re passed with a “#” after the filename (e.g. filename.pdf#zoom=100). You can string commands together with an ampersand (&) with a few caveats:

only one digit after a decimal is retained
parameters and their values can only be 32 total characters long
you can’t use reserved characters (=, #, and &) to escape special characters
if you turn bookmarks off for a PDF that had bookmarks showing, they won’t go away until the PDF has been rendered

Anyway, here are some examples of what you can do: