The Idea of Reference

There is no source of reference remotely as authoritative as the Encyclopaedia Britannica. There is no brand as venerable and as veteran as this mammoth labour of knowledge and ideas established in 1768. There is no better value for money. And, after a few sputters and bugs, it now comes in all shapes and sizes, including two CD-ROM versions (standard and deluxe) and an appealing and reader-friendly web site. So, why does it always appear to be on the brink of extinction?

The Britannica provides for an interesting study of the changing fortunes (and formats) of vendors of reference. As late as a decade ago, it was still selling in a leather-imitation bound set of 32 volumes. As print encyclopaedias went, it was a daring innovator and a pioneer of hyperlinked-like textual design. It sported a subject index, a lexical part and an alphabetically arranged series of in-depth essays authored by the best in every field of human erudition.

When the CD-ROM erupted on the scene, the Britannica mismanaged the transition. As late as 1997, it was still selling a sordid text-only compact disc which included a part of the encyclopaedia. Only in 1998, did the Britannica switch to multimedia and added tables and graphs to the CD. Video and sound were to make their appearance even later. This error in trend analysis left the field wide open to the likes of Encarta and Grolier. The Britannica failed to grasp the irreversible shift from cumbersome print volumes to slender and freely searchable CD-ROMs. Reference was going digital and the Britannica's sales plummeted.

The Britannica was also late to cash on the web revolution - but, when it did, it became a world leader overnight. Its unbeatable brand was a decisive factor. A failed experiment with an annoying subscription model gave way to unrestricted access to the full contents of the Encyclopaedia and much more besides: specially commissioned articles, fora, an annotated internet guide, news in context, downloads and shopping. The site enjoys healthy traffic and the Britannica's CD-ROM interacts synergistically with its contents (through hyperlinks).

Yet, recently, the Britannica had to fire hundreds of workers (in its web division) and return to a pay-for-content model. What went wrong again? Internet advertising did. The Britannica's revenue model was based on monetizing eyeballs, to use a faddish refrain. When the perpetuum mobile of "advertisers pay for content and users get it free" crumbled - the Britannica found itself in familiar dire straits.

Is there a lesson to be learned from this arduous and convoluted tale? Are works of reference not self-supporting regardless of the revenue model (subscription, ad-based, print, CD-ROM)? This might well be the case.

Classic works of reference - from Diderot to the Encarta - offered a series of advantages to their users:

1. Authority - Works of reference are authored by experts in their fields and peer-reviewed. This ensures both objectivity and accuracy.

2. Accessibility - Huge amounts of material were assembled under one "roof". This abolished the need to scour numerous sources of variable quality to obtain the data one needed.

3. Organization - This pile of knowledge was organized in a convenient and recognizable manner (alphabetically or by subject)

Moreover, authoring an encyclopaedia was such a daunting and expensive task that only states, academic institutions, or well-funded businesses were able to produce them. At any given period there was a dearth of reliable encyclopaedias, which exercised a monopoly on the dissemination of knowledge. Competitors were few and far between. The price of these tomes was, therefore, always exorbitant but people paid it to secure education for their children and a fount of knowledge at home. Hence the long gone phenomenon of "door to door encyclopaedia salesmen" and instalment plans.

Yet, all these advantages were eroded to fine dust by the Internet. The web offers a plethora of highly authoritative information authored and released by the leading names in every field of human knowledge and endeavour. The Internet, is, in effect, an encyclopaedia - far more detailed, far more authoritative, and far more comprehensive that any encyclopaedia can ever hope to be. The web is also fully accessible and fully searchable. What it lacks in organization it compensates in breadth and depth and recently emergent subject portals (directories such as Yahoo! or The Open Directory) have become the indices of the Internet. The aforementioned anti-competition barriers to entry are gone: web publishing is cheap and immediate. Technologies such as web communities, chat, and e-mail enable massive collaborative efforts. And, most important, the bulk of the Internet is free. Users pay only the communication costs.

The long-heralded transition from free content to fee-based information may revive the fortunes of online reference vendors. But as long as the Internet - with its 2,000,000,000 (!) visible pages (and 5 times as many pages in its databases) - is free, encyclopaedias have little by way of a competitive advantage.

E(merging) Books

A novel re-definition through experimentation of the classical format of the book is emerging.

Consider the now defunct BookTailor. It used to sell its book customization software mainly to travel agents - but this technology is likely to conquer other niches (such as the legal and medical professions). It allows users to select bits and pieces from a library of e-books, combine them into a totally new tome and print and bind the latter on demand. The client can also choose to buy the end-product as an e-book. Consider what this simple business model does to entrenched and age old notions such as "original" and "copies", copyright, and book identifiers. What is the "original" in this case? Is it the final, user-customized book - or its sources? And if no customized book is identical to any other - what happens to the intuitive notion of "copies"? Should BookTailor-generated books considered to be unique exemplars of one-copy print runs? If so, should each one receive a unique identifier (for instance, a unique ISBN)? Does the user possess any rights in the final product, composed and selected by him? What about the copyrights of the original authors?

Or take BookCrossing.com. On the face of it, it presents no profound challenge to established publishing practices and to the modern concept of intellectual property. Members register their books, obtain a BCID (BookCrossing ID Number) and then give the book to someone, or simply leave it lying around for a total stranger to find. Henceforth, fate determines the chain of events. Eventual successive owners of the volume are supposed to report to BookCrossing (by e-mail) about the book's and their whereabouts, thereby generating moving plots and mapping the territory of literacy and bibliomania. This innocuous model subversively undermines the concept - legal and moral - of ownership. It also expropriates the book from the realm of passive, inert objects and transforms it into a catalyst of human interactions across time and space. In other words, it returns the book to its origins: a time capsule, a time machine and the embodiment of a historical narrative.

E-books, hitherto, have largely been nothing but an ephemeral rendition of their print predecessors. But e-books are another medium altogether. They can and will provide a different reading experience. Consider "hyperlinks within the e-book and without it - to web content, reference works, etc., embedded instant shopping and ordering links, divergent, user-interactive, decision driven plotlines, interaction with other e-books (using Bluetooth or another wireless standard), collaborative authoring, gaming and community activities, automatically or periodically updated content, ,multimedia capabilities, database, Favourites and History Maintenance (records of reading habits, shopping habits, interaction with other readers, plot related decisions and much more), automatic and embedded audio conversion and translation capabilities, full wireless piconetworking and scatternetworking capabilities and more".

Deja Googled

The Internet may have started as the fervent brainchild of DARPA, the US defence agency - but it quickly evolved into a network of computers at the service of a community. Academics around the world used it to communicate, compare results, compute, interact and flame each other. The ethos of the community as content-creator, source of information, fount of emotional sustenance, peer group, and social substitute is well embedded in the very fabric of the Net. Millions of members in free, advertising or subscription financed, mega-sites such as Geocities, AOL, Yahoo and Tripod generate more bits and bytes than the rest of the Internet combined. This traffic emanates from discussion groups, announcement (mailing) lists, newsgroups, and content sites (such as Suite101 and Webseed). Even the occasional visitor can find priceless gems of knowledge and opinion in the mound of trash and frivolity that these parts of the web have become.

The emergence of search engines and directories which cater only to this (sizeable) market segment was to be expected. By far the most comprehensive (and, thus, less discriminating) was Deja. It spidered and took in the exploding newsgroups (Usenet) scene with its tens of thousands of daily messages. When it was taken over by Google, its archives contained more than 500 million messages, cross-indexed every which way and pertaining to every possible (and many impossible) a topic.

Google is by far the most popular search engine yet, having surpassed the more veteran Northern Lights, Fast, and Alta Vista. Its mind defying database (more than 1.3 billion web pages), its caching technology (making it, in effect, one of the biggest libraries on earth) and its site ranking (by popularity and links-over) have rendered it unbeatable. Yet, its efforts to integrate the treasure trove that is Deja and adapt it to the Google search interface have hitherto been spectacularly unsuccessful (though it finally made it two and a half months after the purchase). So much so, that it gave birth to a protest movement.

Bickering and bad tempered flaming (often bordering on the deranged, the racial, or the stalking) are the more repulsive aspects of the Usenet groups. But at the heart of the debate this time is no ordinary sadistic venting. The issue is: who owns content generated by the public at large on computers funded by tax dollars? Can a commercial enterprise own and monopolize the fruits of the collective effort of millions of individuals from all over the world? Or should such intellectual property remain in the public domain, perhaps maintained by public institutions (such as the Library of Congress)? Should open source movements gain access to Deja's source code in order to launch Deja II? And who owns the copyright to all these messages (theoretically, the authors)? Google, as Deja before it, is offering compilations of this content, the copyright to which it does not and cannot own. The very legal concept of intellectual property is at the crux of this virtual conflict.

Google was, thus, compelled to offer free access to the CONTENT of the Deja archives to alternative (non-Google) archiving systems. But it remains mum on the search programming code and the user interface. Already one such open source group (called Dela News) is coalescing, although it is not clear who will bear the costs of the gigantic storage and processing such a project would require. Dela wants to have a physical copy of the archive deposited in trust with a dot org.

This raises a host of no less fascinating subjects. The Deja Usenet search technology, programming code, and systems are inextricable and almost indistinguishable from the Usenet archive itself. Without these elements - structural as well as dynamic - there will be no archive and no way to extract meaningful information from the chaotic bedlam that is the Usenet environment. In this case, the information lies in the ordering and classification of raw data and not in the content itself. This is why the open source proponents demand that Google share both content and the tools to access it. Google's hasty and improvised unplugging of Deja in February only served to aggravate the die-hard fans of erstwhile Deja.

The Usenet is not only the refuge of pedophiles and neo-Nazis. It includes thousands of academically rigorous and research inclined discussion groups which morph with intellectual trends and fashionable subjects. More than twenty years of wisdom and erudition are buried in servers all over the world. Scholars often visit Usenet in their pursuit of complementary knowledge or expert advice. The Usenet is also the documentation of Western intellectual history in the last three decades. In it invaluable. Google's decision to abandon the internal links between Deja messages means the disintegration of the hyperlinked fabric of this resource - unless Google comes up with an alternative (and expensive) solution.

Google is offering a better, faster, more multi-layered and multi-faceted access to the entire archive. But its brush with the more abrasive side of the open source movement brought to the surface long suppressed issues. This may be the single most important contribution of this otherwise not so opportune transaction.

The Disintermediation of Content

Are content brokers - publishers, distributors, and record companies - a thing of the past?

In one word: disintermediation.

The gradual removal of layers of content brokering and intermediation - mainly in manufacturing marketing - is the continuation of a long term trend. Consider music for instance. Streaming audio on the internet ("soft radio"), or downloadable MP3 files may render the CD obsolete - but they were preceded by radio music broadcasts. But the novelty is that the Internet provides a venue for the marketing of niche products and reduces the barriers to entry previously imposed by the need to invest in costly "branding" campaigns and manufacturing and distribution activities.

This trend is also likely to restore the balance between artists and the commercial exploiters of their products. The very definition of "artist" will expand to encompass all creative people. One will seek to distinguish oneself, to "brand" oneself and to auction one's services, ideas, products, designs, experience, physique, or biography, etc. directly to end-users and consumers. This is a return to pre-industrial times when artisans ruled the economic scene. Work stability will suffer and work mobility will increase in a landscape of shifting allegiances, head hunting, remote collaboration, and similar labour market trends.

But distributors, publishers, and record companies are not going to vanish. They are going to metamorphose. This is because they fulfil a few functions and provide a few services whose importance is only enhanced by the "free for all" Internet culture.

Content intermediaries grade content and separate the qualitative from the ephemeral and the atrocious. The deluge of self-published and vanity published e-books, music tracks and art works has generated few masterpieces and a lot of trash. The absence of judicious filtering has unjustly given a bad name to whole segments of the industry (e.g., small, or web-based publishers). Consumers - inundated, disappointed and exhausted - will pay a premium for content rating services. Though driven by crass commercial considerations, most publishers and record companies do apply certain quality standards routinely and thus are positioned to provide these rating services reliably.

Content brokers are relationship managers. Consider distributors: they provide instant access to centralized, continuously updated, "addressbooks" of clients (stores, consumers, media, etc.). This reduces the time to market and increases efficiency. It alters revenue models very substantially. Content creators can thus concentrate on what they do best: content creation, and reduce their overhead by outsourcing the functions of distribution and relationships management. The existence of central "relationship ledgers" yields synergies which can be applied to all the clients of the distributor. The distributor provides a single address that content re-sellers converge on and feed off. Distributors, publishers and record companies also provide logistical support: warehousing, consolidated sales reporting and transaction auditing, and a single, periodic payment.

Yet, having said all that, content intermediaries still over-charge their clients (the content creators) for their services. This is especially true in an age of just-in-time inventory and digital distribution. Network effects mean that content brokers have to invest much less in marketing, branding and advertising once a product's first mover advantage is established. Economic laws of increasing, rather than diminishing, returns mean that every additional unit sold yields a HIGHER profit - rather than a declining one. The pie is getting bigger.

Hence, the meteoric increase in royalties publishers pay authors from sales of the electronic versions of their work (anywhere from Random House's 35% to 50% paid by smaller publishers). As this tectonic shift reverberates through the whole distribution chain, retail outlets are beginning to transact directly with content creators. The borders between the types of intermediaries are blurred. Barnes and Noble (the American bookstores chain) has, in effect, become a publisher. Many publishers have virtual storefronts. Many authors sell directly to their readers, acting as publishers. The introduction of "book ATMs" - POD (Print On Demand) machines, which will print every conceivable title in minutes, on the spot, in "book kiosks" - will give rise to a host of new intermediaries. Intermediation is not gone. It is here to stay because it is sorely needed. But it is in a state of flux. Old maxims break down. New modes of operation emerge. Functions are amalgamated, outsourced, dispensed with, or created from scratch. It is an exciting scene, full with opportunities.

Invasion of the Amazons

The last few months have witnessed a bloodbath in tech stocks coupled with a frantic re-definition of the web and of every player in it (as far as content is concerned).

This effort is three pronged:

Some companies are gambling on content distribution and the possession of the attendant digital infrastructure. MightyWords, for example, stealthily transformed itself from a "free-for-all-everyone-welcome" e-publisher to a distribution channel of choice works (mainly by midlist authors). It now aims to feed its content to content-starved web sites. In the process, it shed thousands of unfortunate authors who did not meet its (never stated) sales criteria.

Others bet the farm on content creation and packaging. Bn.com invaded the digital publishing and POD (Print on Demand) businesses in a series of lightning purchases. It is now the largest e-book store by a wide margin.

But Amazon seemed to have got it right once more. The web's own virtual mall and the former darling of Wall Street has diversified into micropayments.

The Internet started as a free medium for free spirits. E-commerce was once considered a dirty word. Web surfers became used to free content. Hence the (very low) glass ceiling on the price of content made available through the web - and the need to charge customers less than 1 US dollars to a few dollars per transaction ("micro-payments"). Various service providers (such as Pay-Pal) emerged, none became sufficiently dominant and all-pervasive to constitute a standard. Web merchants' ability to accept micropayments is crucial. E-commerce (let alone m-commerce) will never take off without it.

Enter Amazon. Its "Honour System" is licensed to third party web sites (such as Bartleby.com and SatireWire). It allows people to donate money or effect micro-payments, apparently through its patented one-click system. As far as the web sites are concerned, there are two major drawbacks: all donations and payments are refundable within 30 days and Amazon charges them 15 cents per transaction plus 15(!) percent. By far the worst deal in town.

So, why the fuss?

Because of Amazon's customer list. This development emphasizes the growing realization that one's list of customers - properly data mined - is the greatest asset, greater even than original content and more important than distribution channels and digital right management or asset management applications. Merchants are willing to pay for access to this ever expanding virtual neighbourhood (even if they are not made privy to the customer information collected by Amazon).

The Honour System looks suspiciously similar to the payment system designed by Amazon for Stephen King's serialized e-novel, "The Plant". Interesting to note how the needs of authors and publishers are now in the driver's seat, helping to spur along innovations in business methods.