The National Intellectual Property Researchers Association - NIPRA - calls for the immediate launching of a major study of the U.S. Patent and Trademark Office's (PTO's) automated patent search system by a competent outside organization. The purpose of the proposed study shall be to ascertain the overall effectiveness of present PTO patent searching systems, and to guide future PTO automation efforts for the time frame of 2001 to 2005.

In the view of NIPRA's members who perform hundreds if not thousands of patentability searches every working day, as developed in close consultation with a large body of PTO examiners over a several year period, the proposed study shall: (A) be of at least one year duration; (B) be funded by the PTO at the level of between ½% and 1% of the total PTO automation budget for the year of inception; (C) be awarded to an outside private organization - such as a major university, a Federal Contract Research Center (FCRC), or the like - responsive to a detailed request for proposal (RFP); (D) be competitively awarded to a private organization that has a well established history of conducting unbiased evaluations of complex major systems under adversarial conditions; and (E) produce a final, or annual, report upon completion which is made available via wide public dissemination.

NIPRA asserts that it is long overdue for the PTO to commission an outside organization to carry out a comprehensive review and assessment of its automated patent search systems. The automation budget for the year 1999 has been stated as being well over $100 million ($138 million according to a recent POPA newsletter), and is projected to remain at that level for the next five years. This annual level of expenditure has been approximately the same for the past five years, giving a total expended and projected spending on automation of well over one billion dollars.

All of this expenditure has taken place without one single dollar spent on having independent outside experts evaluate the usefulness and efficiency of the existing or proposed systems. This appears to be one of the most reckless uses of public money in the annals of federal procurement. No group within the PTO has evaluated, measured or quantified in any ongoing and systematic way just how good the automated patent search systems are with respect to specific patentability results the examiners are producing on a day-to-day basis. This is especially pertinent in view of the PTO's recent replacement of the APS/Messenger patent search system with the BRS/East-West systems (BRS/EW). It has been estimated - based mostly on documented examples of system operational flaws, and some anecdotal information - that the BRS/EW systems now provide about 40% of the capability of its predecessor systems. This despite the expenditure of many tens of millions of dollars to shut down APS, start up BRS/EW, and install the many quick fixes in an attempt to have a lawn mower powered BRS search engine replace the 18 wheeler powerhouse APS system.

The resulting decrease in user search efficiency and the low system confidence level expressed by both public searchers and examiners speak volumes on the need to evaluate the existing automated patent search systems approaches. This evaluation must be done with a view towards producing rigorous, quantifiable measures and definitions of both how the existing systems are performing and how future systems should migrate over the next 2-5 years to overcome the present low searching efficiencies and user confidence levels.

In carrying out these several tasks, the outside organization performing the systems evaluations should use the techniques described in the 1985 ACM paper by Blair and Maron as their starting point and build on that type of analysis. Due to the early date of that work, and the fact that significant software and hardware changes have continuously occurred in the online, full-text database searching world, it is possible that a new set of key parameters well beyond 'recall' and 'precision' will emerge as the primary criteria for evaluation. Even so, once the key search system parameters have been established by the outside organization, the customary analyses should be carried out. Analyses such as: (1) how well does BRS/EW respond to real life patentability cases as measured by the key search system parameters; (2) what changes are needed in BRS/EW to improve its performance in real life patentability cases; (3) how can the present responses by BRS/EW be compared with its best predecessor systems; (4) in what time frame can the needed improvements of (2) be realistically accomplished; (5) what is the quantifiable effect on real life patentability searching due to the low level of Foreign Patent Access System (FPAS) integration into the USPT database; and (6) based on the findings of (2) and (4), which presently underway or proposed automation patent search system efforts should be accelerated, delayed or stopped.

Pulling together all of the above, it is clear that any statement of work for the prospective outside organization must include at least the following five deliverables.

I - Starting with the 1985 Blair et al ACM paper as a guide, develop a set of key search system parameters best encompassing the present PTO patentability search requirements, and best describing the performance of present and near-future patentability search systems in terms of those parameters.

II - Starting with the 1997 Cottone JPOS paper as a guide, develop a suitable number of compilations of real life PTO cases (sanitized) quantifying the effectiveness of various search types presently available to PTO examiners. Extrapolating from these quantified compilations, develop measures of effectiveness for the various search types.

III - Investigate all available reasonably well documented, user-described BRS/EW failures or confidence reducing anomalies with a view towards identifying their systemic causes. Thereafter, propose changes or corrections at the appropriate systems levels to correct or eliminate recurrence of the failures or anomalies.

IV - Conduct ongoing interviews with a wide cross-section of PTO examiners (see Note A) for the purpose of determining the 'center of gravity' of their automated patentability searching requirements. From these interviews identify their specific, well-defined needs, the relative frequency of the specific needs, the relative importance of the specific needs, and alternate means for meeting these specific needs when full BRS/EW (and FPAS) capabilities are not available.

V - Publish a final report in printed form detailing the customary scholarly content, including: what was done, why it was done, the special or general methodologies employed, specific findings and recommendations, recommendations for follow-on work - all complete with charts, graphs, tables, statistics and bibliographies.

NIPRA Proposal for Combined Search Facility