Why Reference and Instruction Librarians Hate Federated Searching and NextGen Catalogs
Confession: this presentation is therapy for me. In the past six years, I have implemented federated search twice in academic libraries more or less on my own. I also have spent much of 2009 chairing and serving on a NextGen catalog product selection and implementation committee; we went live with WorldCat Local this fall semester. Discovery products such as these are often billed by vendors as the Holy Grail in the age of Google and Amazon. Perhaps this has raised our collective professional expectations of federated search and NextGen products to an unreasonably high level, because, in my experience at least, the reception among reference and instruction librarians can be described as lukewarm at best. In this pool of dissatisfied librarians, I include my former reference and instruction librarian self, my colleagues at the University of Colorado Denver’s Auraria Library, and, more scientifically, the respondents of a survey conducted by Lynn Lampert and Katherine Dabbour at California State University Northridge.1 [SLIDE 2] The short answer for the lukewarm reception is that expectations of discovery tools are built on expectations of the products that they are designed to work with. The longer answer to this question envelops these technical issues and also the dissonance between what federated searching and NextGen catalogs purport to do and the pedagogical roles of reference and instruction staff in the library.
While much has changed in the realm of federated searching in recent years, there were still a handful of technical shortcomings that were hard to swallow in 2007 when we chose 360 Search at Auraria. Note, however, that the following discussion does not target any specific products here; my personal experiences with products and customer support with WebFeat (2003) and Serials Solutions 360 Search (2007) were both very positive, in spite of my initial distaste for federated searching generally. The shortcomings discussed here—whether real or perceived—were more or less endemic to the products regardless of the brand at the time they were implemented. These include lack of features, inability to search all databases, speed, and unmet performance expectations.
Date and peer review filters are now standard on most online databases, and these features have, logically, become embedded in reference and instruction routines. While Serials Solutions’ 360 Search and other products are technically capable in and of themselves of applying these limits to searches, limitations in the metadata that database vendors provided rendered date and peer-reviewed filters useless. Serials Solutions’ 360 Search supported peer-reviewed filtering, but technical support recommends avoiding it because filtering a list of results for peer-reviewed articles usually results in zero results. Further, users may include a single specific year in their search terms; however, searching a range of dates is not yet supported for the same reason that the peer-reviewed filter does not work. (To reiterate, these were not flaws in the 360 Search product itself but conditions of the technology at the time.)
Secondly, in spite of what product literature may claim, no federated searching product can—or in some cases, should—currently search every online resource. Again, this is not necessarily a reflection on the quality of the search product itself, and whether or not to federate all databases and resources is a philosophical question all its own. Four common reasons that a database may be excluded from a federated search include: vendor prohibition, no existing “translator” for the product, an license agreement that stipulates a limited number of concurrent users for a resource, or sources that are priced in a pay-by-search model.
First, notable holdouts among vendors who did not permit their clients to federate some or all of their products in 2007 included Hoover’s, InfoUSA, and content giant LexisNexis. At the time Auraria went live with 360 Search, only LexisNexis Academic could be made available through a federated interface. Libraries sold on the concept of federated search have expressed their dismay about exclusion to holdout vendors, and they have also considered looking for equivalent content in other online resources that will allow inclusion.
Secondly, the nature of the electronic resources market creates a demand for constant creation by federated search vendors of the “translators” that allow a resource to be included in federated search. Therefore, while it may be technologically possible for the federated search product to work with a given resource, if the vendor has not developed a translator yet, that resource will be effectively wait-listed. If the resource is local or highly specialized, the client library may have to wait until more client libraries request inclusion of a resource to increase the demand for the translator. In the case of the Auraria Library, this meant exclusion of Prospector, the unified catalog for the Colorado Alliance of Research Libraries consortium, which is used heavily for interlibrary loan.
Third and fourth, technical support generally recommends that resources for which a library has a limited number of concurrent users or which are priced by pay for search be excluded from federated searching. If a subscription to a resource with limited concurrent users is included, it is unlikely that any user would ever be able to successfully access the product. A resource that is charge-by-use will be rapidly depleted of the allowed number of searches. This is because “use” of the database in this context begins not when a user clicks a link to an item in one of these resources, but when the resource is included in a federated search. In some cases, changing the subscription to the electronic resource to unlimited use may be an option; however, for other resources, this is prohibitively expensive. Some favorite high-quality resources of reference and instruction staff fell into this third category, as purchasing a resource with a low number of concurrent users may be a factor in licensing an expensive product at all.
Deciding how to handle all of the orphaned resources created by vendor exclusion, lack of an existing translator, a limited number of concurrent users, or limited number of searches in a customized implementation of a federated search product can be quite difficult. Links to the excluded resources’ native interfaces can be included in an A-Z list within the federated environment or on appropriate subject guides or other pertinent web pages, but even so, they may be easily overlooked by patrons seeking a quicker—that is to say, federated—method of searching, such as selecting a bundle of resources grouped around a subject area, e.g., “Art & Architecture.” From a reference and instruction perspective, it can be difficult to market and encourage use of a new search feature that omits some of the best and most recommended resources.
A further issue with both federated searching tools and NextGen catalogs is search speed. With the explosion in the number of online resources made available in recent years, many libraries have gradually outgrown their network infrastructure, in a worst-case scenario, network capabilities can be pushed to the breaking point because of the increase in traffic. However, even in a healthy network environment, the simple fact that a federated search product is doing more work than a search in a database’s native interface also accounts for this extended search time. Waiting a minute or more for a search to grind away can create an awkward lull at the reference desk or in a classroom setting does little to instill confidence in the minds of the library staff and patrons. Even though we can say that this search grinding away takes less time than searching each native interface individually, we’ve all become impatient due to high speed of the interwebs.
Finally, there are behaviors with discovery products that are simply unexpected. A particular use of the software that sounds brilliant in theory sometimes does not prove effective in practice. For example, reference and instruction staff at Auraria were asked to draw up a list of ten or so resources that would be included in a general-focus “Quick Search” box on the Library’s home page. Eleven databases plus the library catalog were chosen for inclusion, and staff were excited by the potential of offering results to general queries from these resources from a search box on the home page. However, in practice, the result was disappointing. The results returned from the fastest resource were the results on top of the pile, and of the twelve resources chosen, PsycINFO routinely returned results first. Reference and instruction staff felt that this skewed the results for a general query; therefore, the feature was gradually reduced to three databases plus the catalog, and then simply three databases.
While these current technical shortcomings are a large part of the dissatisfaction in the reference and instruction department, there are philosophical and pedagogical issues as well. One of the primary concerns of reference and instruction staff is that discovery tools dumb down the research process. All of the controlled vocabulary and carefully constructed indexes behind individual online resources are tossed out the window; the results returned from a certain resource via the federated interface may be of a lesser quality than those returned from a search in that resource’s native interface. In the words of a colleague, in an academic environment, federated searching “removes many kinds of academic research drills and routines one or more steps from reality.”
My response to that question is, “Whose reality?” A librarian’s? Because we know how to walk both ways to school uphill in the snow, our patrons have to, too? Just earlier this week at KLA, Rick Anderson from the University of Utah called for the slaughter of five sacred cows of librarianship, the third of which was reference. He noted that reference is not scalable, and that our goal should be to make libraries so easy to use that reference service is not necessary. Auraria Library is, in fact, moving to an information desk model that refocuses the main duties of reference librarians to consultation, liaison, and outreach duties. If the tools we choose to offer our patrons are easier to use, reference librarians can go beyond the library’s walls and increase our presence in our communities in these ways. (Thank you whoever tweeted about Rick Anderson’s presentation, by the way.)
Further, federated searching and NextGen products bring no content into either the physical or virtual library. Reference and instruction librarians quite understandably crave content with which to fulfill their reference and instruction duties. With product price tags in the tens of thousands and budgets shriveling, buying a tool that does not stand up to staff expectations and brings no more content into the library seems foolish. I find this a bit ironic, however, because libraries spend tens of thousands of dollars on online resources that have terribly unfriendly user interfaces, even for information professionals, yet are the sole online provider for crucial resources.
Finally, in terms of personal use—whether for conducting one’s own research or while assisting patrons—using discovery tools feels like putting the training wheels back onto the bicycle. Reference librarians know or can surmise which resources will likely yield good results for a given query, and they proceed to what they know are “the usual suspects” in the lineup of electronic resources. Because of this expertise, it is difficult to use federated searching instinctively in reference and instruction. One reference librarian told me that she never taught federated search in her classes because she hated it.
While I am now a systems librarian and no longer work at the reference desk, an undergraduate asked me in passing shortly after we implemented 360 Search how to find a “professional article” in a biology journal. I found myself directing her to the “biology” drop down menu in our homegrown directory of databases, ultimately offering her a choice of BioOne, BIOSIS, ScienceDirect, and Web of Science. I added as a footnote that she could search multiple resources simultaneously from the Biology subject guide—but not Web of Science because it was not included in our federated search setup because of our limited number of concurrent users. Not to mention the peer-reviewed filter issue so that she’d get a “professional” article. And there’s the rub: explaining the shortcomings of the federated search box on the Biology guide was more difficult than simply pointing her to a couple of “best bet” choices in the first place.
Given the above, why are federated search products still on the market, and why are libraries still contracting with vendors? What has changed my own mind in the last five years, transforming me from a reluctant community college reference librarian fighting Webfeat tooth and nail to a web librarian petitioning the university’s budget priorities committee for extra funds to pay for 360 Search? Two things: usability testing and developments in discovery tools themselves.
Web usability testing has finally opened our eyes to the fact that patrons have vastly different mental models of the world of information than librarians. For me personally, this became painfully clear in 2004 while observing a graduate student at Georgetown University during a usability testing session. [SLIDE 3] He timed out after three minutes while trying to decipher from the library’s home page where to find a scholarly article about Descartes. He vacillated between the “catalog” and “database” links on the library’s home page for 180 agonizing seconds and, in the end, he never made a choice.
Libraries have historically had difficulty marketing and presenting in an intuitive fashion what is the heart of the virtual library: subscribed electronic content, which accounts for the lion’s share of our annual resource budgets. While not inexpensive, discovery tools are now offered with a number of pricing options. Even if a library chooses to federate as many resources as possible, the annual price tag will still likely be less that one percent of the total annual expenditure for electronic resources—a small price to pay for what can be a large return on a very large investment. Additionally, implementing federated searching as a discovery tool can restore patrons’ faith in their ability to find what they need when they come to their library’s web site versus the open Internet. We will only continue to offer more and more online content in the coming years, and discovery tools provide a way to present sensible options for patrons as the number of online resources continues to grow.
The products themselves, to include the implementation process, have also evolved quite a bit in recent years. Early vendor offerings of federated search tools were clunky behemoths that required a local server installation and took months, sometimes years, to prepare for patron use. Now, vendors typically offer more lightweight hosted options that are ideal for libraries whose local technology resources are limited or lacking. Serials Solutions’ promised—and, in the case of the Auraria Library, delivered—six-to-eight-week turnaround from contract to launch is a dramatic improvement over the years-long implementation time. Generally, vendor support during implementation is better, with the vendor doing more of the setup work, depending upon the products and the options chosen. And this year, because of a new amendment passed recently in the state of Colorado, it actually took us longer to secure our contract with WorldCat Local than to implement it.
In terms of technical improvements with federated search tools, vendors have found effective ways of deduplicating results, which was an early Achilles’ heel. The practice of HTML screen scraping—analyzing the output of a database by “reading” the HTML of the results page—is being replaced by more highly structured XML Gateway technology, which improves the quality of the results returned. Additional features like clustered results, integration with other tools, supported web integration services, and impressive administration and statistics modules are now available among the various products. Notable examples in 2007 included Serials Solutions’ clustered results feature in 360 Search and Info-Graphic’s administrative module for their AGent product.
Further, customization options allow discovery tools to overlay an entire web site, unifying many resources to a single access point. Some librarians wanted to put both 360 Search and WorldCat Local on the databases page; there’s not much point spending money on a discovery tool if it’s put on a web page that no one understands in the first place. In addition to providing a single-search box on the library’s home page, many libraries are using federated search products to enhance the more traditional and usually home-grown A-Z/subject directories of databases as well as library guides and pathfinders. Almost any combination of resources and audience is possible. For example, an “English 101” bundle in an academic library could include EBSCO Academic Search Premier, Gale’s General OneFile, and LexisNexis Academic. Code for this particular search could be embedded on a class guide for English composition courses so that less time could be spent training students on the individual interfaces. More time could then be spent discussing topic selection and refinement, Boolean logic—which is supported by federated search products—and other important research concepts.
I’m pleased to report that I have found myself instinctively using WorldCat Local as launched on our home page. [SLIDE 4] There were a few of issues that arose during implementation, not the least of which included labels of the tabs on the single search box. We based our search box on the University of Washington’s home page display of WorldCat Local, copying the tab labels “books,” “articles,” “dvds” and “cds.” Then the debate began. The top issue was that librarians were concerned that students would think that they were searching all articles, so the label was briefly “Select Articles.” This was problematic, however, in that it sounded as though “select” was a verb. Then “selected articles” was proposed, but this gave the incorrect impression that this material was somehow vetted by library staff. A compromise was struck by adding a “research tip” on each of the tabbed areas of the search box. [SLIDE 5] This alerted users to additional information in a way that did not sound like an apology about what librarians were dissatisfied with. “All formats” was briefly suggested for the “all” tab, but this was rejected in favor of single-word tab labels, and also because “format” is an empty, jargony librarian word.
Are discovery tools the Holy Grail? To the Knights of the Round Table, the Grail represented an unattainable ideal that compelled the Knights to pursue it, in some cases to their own destruction. Sometimes, things fail; Auraria discontinued 360 Search this summer because OCLC is adding federated search to WorldCat Local. [SLIDE 6] I don’t think that the big picture future is so bleak for discovery tools, however. [SLIDE 7] At Computers in Libraries this past Spring, SirsiDynix Vice President Stephen Abram noted, “Criticizing federated searching is like looking at your child learning to crawl and declaring, ‘He’s a horrible accountant, I’d never hire him.’” He further noted, “It takes a village to raise this stuff.” We are finally listening to what our customers want; vendors are listening more to what we want and partnering with libraries during development. Opening these channels of communication and collaboration is a very good thing. New products that seem to be neither federated search nor NextGen catalogs, such as Serials Solutions’ Summon are emerging onto the market, and open source and other small vendor offerings will be interesting to watch in the coming years.
These nascent discovery tools are proving that they can successfully connect patrons with subscribed online content; they are more tools that we can put in our patrons’ research toolboxes, even if librarians still prefer not to use them until future developments, hopefully, will make improvements in what reference and instruction librarians find lacking. Even with all of its current technical shortcomings, discovery tools provide a means of presenting our hyperstructured universe, with all of is semi-secret classification schemes and codes, to our customers in a way that they not only understand, but have come to expect.
1. Lampert, Lynn D. and Katherine S. Dabbour, “Librarian Perspectives on Teaching Metasearch and Federated Search Technologies,” in Federated Search: Solution or Setback for Online Library Services, ed. Christopher Cox (Binghamton, NY: Haworth, 2007) 253-78.