Tag: feedback

Grouse about your next-generation catalogue – LibCamp@Brunel

A journey to the the wild wild west (of London)

On Saturday 28th January I attended LibCamp@Brunel, a library unconference generously hosted by the library at Brunel University in Uxbridge. I’d not been this far west in London as a destination before and on arriving I was pleased to recognise the tube station at Uxbridge as one of Charles Holden’s designs, which I took as a good omen for the day.

At the opening introduction and pitching, I pitched a session about staff perception versus library user perception of  next-generation library catalogues. As the unconference attendees were by and large library workers, I also wanted to invite everyone to come and grouse about problems they’d had with these systems. And let’s be honest, “Grouse about your next-gen catalogue” is going to be fun.

I had modest expectations for this session but it was very well attended, so much so our allotted space was too small and we had to move somewhere roomier. As I was facilitating I couldn’t live-tweet the session and following a few requests from people who couldn’t attend I decided to expand on the points made to give you a flavour of the discussion.

Perceptions of the catalogue

For some time I’ve been trying to understand problems readers have with the catalogue, and had wondered if it was possible to generalise this to talk about staff versus reader perception of Encore and next-generation systems. I hoped we could work towards this in discussion. As well as Encore, Aquabrowser, VUfind, and Summon were mentioned in discussion.

We’ve come a long way. I expected I would have to define next-generation catalogue in the session, but I was delighted when one of the graduate trainees present explained what I call next-gen was simply what she expected from a normal library catalogue. I had to give a really quick potted history of four generations of catalogue interfaces. (This is how to make your systems librarian feel old…)

I explained our experience of implementing Innovative Interfaces Encore at Senate House Library, and particularly how different I have found the perspectives of the library staff versus our readers. To be clear, my colleagues were almost entirely positive towards the new catalogue. I was pushing at an open door implementing a catalogue that offers a much better experience to readers used to using modern Web sites compared with the previous catalogue, relatively little changed since the 2000s.

However, I think it’s important to answer criticism and deal with objections as there could easily be problems I’d overlooked, and there’s a need to have these arguments as one step in bringing people with you.

Andrew, you can’t implement without feature x…

In the early days pre-implementation I heard various objections to Encore along the lines of it being feature-incomplete compared with the previous catalogue. Some of my colleagues were hopeful that it would be possible to put off implementing Encore on this basis: we should wait until the next release, or the next-plus-one release, where these issues would be resolved…

It is correct that the new catalogue:

  • Doesn’t generate any left-to-right phrase indexes as our old catalogue did. Everything is indexed as keywords.
  • Doesn’t deal with classmarks for most of our multitude of classification schemes at all. At all. It doesn’t index them as classmarks and doesn’t allow you to browse by classmark.
  • Has fewer options for presenting a ‘scoped’ view of the catalogue limited to just a particular library or collection.
  • In the version we launched with, didn’t offer an advanced search with pre-limits and didn’t support boolean operators at all. (This has been added since.)

Having already done some user testing of the new catalogue I was reasonably confident none of the missing features were a show-stopper for implementation. If there were problems for some readers, we had a simple solution: allow everyone to continue using the old catalogue in parallel to the old one.

One of the Library Campers had pointed out in advanced this is an unusual approach. I explained further in discussion this was partly by necessity as the ‘patron’ features – the ability to log in to view your loans, place a reservation and renew loans – were still based in the old catalogue anyway.

I was asked about how we make sure readers find and use Encore. To drive reader uptake of the new catalogue I wanted to offer Encore as the default option on that places that really matter to us – on the Senate House Library homepage and on the old catalogue homepage. The latter uses some JavaScript to redirect your search depending on what options you select, but if you keep the default ‘Quick Search’ you get Encore. It was important to me that by following the path of least resistance readers would end up with the new catalogue.

I have said before and I stand by it: if you want to buy and implement a new system you should have the courage in your convictions and implement it properly. It amazes me to see libraries that offer their new discovery interfaces as an “alternative search” that can be ignored, or that requires special effort to find and use. I do see the value in doing this during a public beta test or preview, as the British Library did with Primo (branded as Explore the British Library), but absolutely not when you’ve made it live.

As of January 2012 we see slightly more use of the new catalogue in terms of visits, ~56% of the combined total based on Google Analytics data (I said ~50% based on data from Q4 2011 in the session). I consider this a reasonable start.

In the eight months since going live with the new catalogue several types of problem have emerged with Encore.

Longer term: how staff use the catalogue

It’s surprised me how many unusual uses of the old catalogue interface our staff have built up over time and the extent to which the catalogue has taken on functions I wouldn’t expect. For example, making use of the way classmarks are indexed to produce a list of everything from a particular classmark, particularly useful for Special Collections where the classmark might be used to describe what collection something is in. Or a need to produce a list that represents everything related to some sub-set of our catalogue – that is, a search strategy that you can be confident represents 100% true positives!

Much of this has been presented to me in good humour in a playful spirit of showing me how Encore can be “beaten” by a particular use case.

There are uses of the old catalogue that are simply impossible in the Encore catalogue, but my answer is first they don’t tend to represent realistic use cases our readers make, second they can more or less easily be moved to the staff client for our library system. Apart from Encore, Katharine Schopflin and Graham Seaman discussed how next-generation systems can have problems with known item searching and in attempting to present a search interface biased towards too much towards browsing and subject searching can be actively unhelpful when you have specific items in mind. I explained I think Encore is quite good for known item search, in particular the way it prioritises exact hits from MARC field 245 $a, my favourite examples are journals like Text and Agenda.

Generally I don’t think we should aim every discovery tool only at our most expert users, information professionals with great experience with our collections, when they have working alternatives available. I explained in response to a question there is no staff-specific view of Encore if you sign in using a staff account. I think this is right and proper from a “dogfooding” point of view, but I confess I daydream about a catalogue that is this flexible enough to offer a different interfaces with different features for novice to expert as required…

Longer term: you need to sort out your metadata

It’s become a truism that because next-generation systems make better use of our bibliographic data they force us to sort out existing problems with our metadata. We’ve certainly found our fair share of these problems since launching Encore, but not all of them are fixable.

The first we’ve tried to address is the way different types of material were described in our catalogue, the combination of print monographs (er, books) and print periodicals (um, journals) into a single material type termed “printed material”. Cue amused smiles from the Library Campers! Since then we’ve split them into books and journals as I explain on a blog post on our Encore blog – ‘Helping you find print journals more easily’.

The general problem is Encore can only act on the metadata it has available, but realistically you won’t always have time and money to do the work required to make it good. Encore does useful things like provide facets based on geographical names in your subject headings, or dates of publication, or languages. The problem is the data being missing or coded ‘undetermined’.

We know there are some very good items in our collection that are not findable during subject searching by readers because they have a record that’s not very good. Graham Seaman mentioned a problem in Summon in the way dates can be described in different ways, understandable by humans but not machines. For example you could refer to things from the same time period as ’16th century’, ‘1500–1525’, or ‘Renaissance’ and so miss out on relevant items.

These are problems that existed with our old catalogue but which the next-generation catalogue brings into sharper relief.

Thoughts on usability testing the next-gen catalogue

This is the second post in a series exploring user understanding of next-generation catalogues:

What I like about usability testing

I have always found usability testing library systems enjoyable – as a participant, facilitator, and manager – and gotten useful things out of it. My preferred style is Steve Krug’s “Lost our lease, going-out-of-business-sale usability testing” from Don’t make me think (2006) with about five subjects and a very focused idea about what I wanted to get out of the process. By that I mean specific problems that needed user feedback to inform our judgments.

What I like best about this method is it represents effective action you can take quickly on a shoestring. You can short-circuit the endless librarians-around-a-table discussions you can get into about Web stuff: let’s test this out rather than just talking about it! I have defended using this method with small groups, as even testing a few users tells you something about what your users are actually doing whereas testing no-one tells you nothing at all. In writing that I realised I was paraphrasing Jakob Nielsen, “Zero users give zero insights”.

We’ll likely employ this method when we rework the Senate House Library Web site next year.

What I don’t

I think there are some problems with this style of testing as a methodology so have been looking into other methods for investigating Encore.

My main problem is the artificial nature of the test. Putting a person in your usability “lab” with a camera recording and asking them to do various tasks does not produce a natural experience of using your library catalogue. Your methods of observing the test will alter the users behaviour: these are observer effects you cannot hope to control for. In my dissertation interviews I tried to temper this by focusing on subject searching, browsing, and exploration of the next-generation catalogue interface rather than asking for subjects to complete tasks. I used a form of close questioning to explore participants’ understanding of Encore. This relies on asking probing questions along the lines of:

  • How?
  • Why?
  • What?
  • What if?

Ultimately this is based on a contextual inquiry approach described by Beyer and Holtzblatt in Contextual design (1998), but done with the understanding that it was taking place in an artificial environment not “the field”.

In truth the usability testing-style part of the investigation was meant as a warm-up towards comparisons between two or more catalogues using the repertory grid technique. I thought this worked reasonably well. The usability test section yielded up a good deal of qualitative data and certainly worked to get participants to the right frame of mind for grid construction.

It also produced useful results about for tweaks we could make to improve Encore as a better fit to readers’ expectations of a library catalogue. That is, it worked as usability testing.

However as I did the work I was aware of the artificial nature of the process affecting how my subjects behaved and their problems engaging with the process in anything like a natural way. The cognitive walkthrough style is difficult on two levels: it feels odd and a bit embarrassing to do it as a subject, but also it makes you think about what you are doing and how you should express yourself which affects your behaviour. Several participants picked up on this during their interviews and criticised it.

I’ve found our readers experience of the catalogue is deeply affective, and think we need to dig deeper into that affective layer to understand the user experience. I think ethnographic methods like the contextual inquiry approach is the way to go here, and will return to this in my next post.

Final point. I know our vendor has done their own usability testing on the Encore interface including informing changes to the current look and feel, in use on our catalogue. I have no reason to doubt its effectiveness or rigour. We could do usability testing of Encore, but I doubt we would add much beyond what the vendor already knows.

References

Beyer, H. and Holtzblatt, K. (1998) Contextual design. London: Morgan Kaufmann.

Krug, S. (2006) Don’t make me think. 2nd edn. Berkeley, CA: New Riders.

User feedback and problems with the next-gen catalogue

This is the first post in a series exploring user understanding of next-generation catalogues:

Our situation

We made our next-generation catalogue / discovery interface, Encore by Innovative Interfaces live in June 2011. Since then I’ve been trying to better understand the causes of the problems readers have with it.

As a starting point I’ve been doing this through the lens of the mental models theory; but I’m trying not to see every problem in terms of one particular theory just because that’s what I’m looking for. Sometimes a missing feature is just a missing feature, to paraphrase something attributed to Freud.

I’d expected some experienced users would find problems moving to a next-gen catalogue because their “bibliographic retrieval” mental model fitted to a traditional library catalogue would not fit so well to a “web search” style catalogue. To view this in reverse, and much more fairly blame the catalogue than the user: I had thought the next-gen catalogue in trying to be like a Web search engine would cause some problems for experienced users. I’m looking at this as mental models failure or mismatch, not implying it’s people not wanting to change.

I expected problems would surface easily as we positioned Encore aggressively, making it the default search (named Quick Search) when you visit the Senate House Library catalogue front page. This was meant as a nudge: you can select the old WebPAC catalogue but you’re not offered it as the default and it’s a little bit of effort to choose it.

Because of this my staff training for Encore focused on helping staff better explain how Encore works, with a view to building better models in the minds of readers.

User feedback

I’ve been gathering feedback reported via staff, Twitter and Facebook, and our online feedback form. Broadly they fit into these categories:

  • A general I like it (~10%) / I don’t like it or really don’t like it (~20%)
  • I can’t work out how to do x like you can in the old catalogue / It lacks feature x the previous catalogue has (~20%).
  • Suggestion for an enhancement (my personal favourite) (~9%)
  • Questions or feedback not about Encore (~40%)

The numbers of comments not about Encore suggests the first thing I should do is put an easy-to-find “Ask us a question” and a “Report problem with this record” link on each page! We don’t have that on the old catalogue so I expect we’re missing out on picking up potential enquiries there.

A good chunk of problems related to “I can’t work out how to do x” represent application of mental models from the old catalogue onto the new catalogue. For example an expectation of being able to browse based on phrase indexing of fields like title. It simply doesn’t exist in Encore, and it can be baffling if this is what you expected. I also got some interesting comments about the look and feel of Encore as “cluttered” or “busy”, which affected the user’s perception of the catalogue functionality way beyond what you’d expect. Innovative have since released a new Encore skin which is subjectively much cleaner and pretty much nails that problem.

I’ve been pleasantly surprised to receive positive comments at all as my expectation in a customer service situation is people are more likely to spend the time if they want to say something negative. I think twitter helps with this as it’s much easier to say something immediately by microblogging than marshalling your thoughts and filling in a ponderous official-looking form. Overall I’m happy to get a 1:2 ratio of positive to negative, and of course each negative comment is an opportunity to say something about Encore and better explain it.

By the way, some of the positive comments are wonderful such as this tweet:

Next steps

It’s useful to get any feedback about what you’re doing, but passively collecting data is not going to get us where we need to be. The question I want to answer is along the lines of: how can the library make this catalogue better support readers and improve their experience of Senate House Libraries? This is going to be more complex than answering usability-type questions about the Encore interface versus the old WebPAC, or comparing the difference in tweaking around the edges of the Encore configuration. Not to say I haven’t done plenty of that already…

To do this we need to actively gather data on our readers’ experiences with Encore and how they make use of it during information seeking. More to come on this later.