An Environment for Merging and Testing Large
Ontologies
Deborah L. McGuinness
Richard E. Fikes
James Rice
Steve Wilder
Knowledge Systems Laboratory
Computer Science Department Stanford University
www.ksl.stanford.edu
Abstract
Large-scale ontologies are becoming an essential component of many applications
including standard search (such as Yahoo and Lycos), e-commerce (such as Amazon and
eBay), configuration (such as Dell and PC-Order), and government intelligence (such as
DARPA’s High Performance Knowledge Base (HPKB) program. The ontologies are
becoming so large that it is not uncommon for distributed teams of people with broad
ranges of training to be in charge of the ontology development, design, and maintenance.
Standard ontologies (such as UN/SPSC) are emerging as well which need to be
integrated into large application ontologies, sometimes by people who do not have much
training in knowledge representation. This process has generated needs for tools that
support broad ranges of users in (1) merging of ontological terms from varied sources
(2) diagnosis of coverage and correctness of ontologies (3) maintaining ontologies over
time. In this paper, we present a new merging and diagnostic ontology environment tool
called Chimaera, which was developed to address these issues in the context of HPKB.
We also report on some initial tests of its effectiveness in merging tasks.
Corresponding Author:
Deborah L. McGuinness
Gates Building 2A #241
Stanford University
Stanford, CA 94305
[email protected]
(650) 723 9770
Topics:
Implemented KR&R Systems (Reports, Evaluations);
Knowledge-Based Systems
1 Introduction
Ontologies are finding broader demand and acceptance in a wide array of applications. It
is now commonplace to see major web sites include taxonomies of topics including
thousands of terms organized into five-level or deeper organizations as a browsing and
navigation aid. In addition to broader use of ontologies, we also observe larger and more
diverse staff creating and maintaining the ontologies. Companies are now hiring “chief
ontologists” along with “cybrarian” staffs for designing, developing, and maintaining
these ontologies. Many times the team leader may have knowledge representation or
library science training, however the staff may not have much or any formal training in
knowledge representation. The broader demand for ontologies along with greater
diversity of the ontology designers is creating demand for ontology tools.
The average ontology on the web is also changing. Early ontologies, many of which
were used for simple browsing and navigation aids such as those in Yahoo and Lycos,
were taxonomies of concept names. The more sophisticated ontologies were large and
multi-parented. More recently, mainstream web ontologies have been gaining more
structure. Arguably driven by e-commerce demands, many class terms also have
properties associated with them. Early commerce applications, such as Virtual
Vineyards, included a handful of relations, and now many of the consumer electronics
shopping sites are including tens or hundreds of role names, sometimes associated with
value restrictions as well. We now see more complicated ontologies even in applications
that are only using ontologies to support smart search applications. Additionally,
ontologies are being used more to support reasoning tasks in areas such as configuration
and intelligence talks. A decade ago, there were a modest number of ontology-supported
configurators such as PROSE/QUESTAR[McGuinness and Wright,1998; Wright et. al.,
1993], however now web-based configurators are quite common. There are even spin
offs of configurator companies handling special areas of configuration such as PC-Order
for PC configuration. Configuration ontologies not only have class, role, and value
restriction information, but they typically have cardinality information and disjointness
and reason with contradictions. Thus, we claim that ontologies are becoming more
common, the designers come from more diverse backgrounds, and ontologies are
becoming larger and more complicated in their representational and reasoning needs.
Simultaneously, there appears to be a stronger emphasis on generating very large and
standardized ontologies. Areas such as medicine began this task many years ago with
SNOMED[Spackman, et. al., 1997] and UMLS[McCray and Nelson, 1995]. Recently
broader and shallower efforts have emerged like the joint United Nations/Dunn and
Bradstreet effort to create an open coding system for classifying goods and services. The
goal of standard ontologies is to provide a highly reusable, extensible, and long-lived
structure. Large ontologies in concert with the challenges of multiple ontologies, diverse
staffing, and complicated representations strengthens the need for tools.
1
In this paper, we address two main areas. The first is merging different ontologies that
may have been written by different authors for different purposes, with different
assumptions, and using different vocabularies. The second is in testing and diagnosing
individual or multiple ontologies. In the rest of this paper, we will give two project
descriptions that served as motivation for our work on merging and diagnostic tools. We
will then describe an ontology environment tool that is aimed at supporting merging and
testing ontologies. We will describe the tool’s use in our work on DARPA’s high
performance knowledge base (HPKB) program[Cohen, et. al., 1998]. Finally, we will
describe an evaluation of the merging capabilities of the tool and discuss future plans.
2 Two Motivating Problems
In the last year, one or more of the authors took were involved in each of two major
ontology generation and maintenance efforts. We gained insight from the tasks that was
used to help shape our resulting ontology tool efforts. Subsequently, we have used
[McGuinness, 1999] as well as licensed the tools on other academic and commercial
ontology projects. We will describe the tasks briefly and present an abstraction of the
problem characteristics and needs with relation to ontology tools.
2.1 Merging the High Performance Knowledge Base Ontologies
The first problem was in the High Performance Knowledge Base program. This program
aimed to generate large knowledge bases quickly that would support intelligence experts
in making strategic decisions. The KBs had a reasonably broad subject area including
terrorist groups, general world knowledge (such as that contained in the CIA World Fact
Book[Frank, et. al., 1998]), national interests, events (and their results) in the middle east,
etc. The types of questions that an analyst might ask of a KB might be simple, including
straight “look up” questions like finding the leader of an organization or the population of
a country. Other questions might be quite complex including asking about the possible
reaction of a terrorist group to a particular action taken by a country. Knowledge bases in
this program tended to have a high degree of structure, including many roles associated
with classes, value restrictions on most roles, fillers on many roles, minimum cardinality
restrictions on roles, disjoint class information, etc. The knowledge bases were typically
designed by people trained in knowledge representation and usually populated by those
literate but not expert in artificial intelligence.
In the first year of the program, many individual knowledge bases were created in order
to answer particular “challenge problem questions”. These questions were aimed to be
typical of those that a government analyst would ask. Two competitive research and
integration teams were evaluated on the quality of the answer that their knowledge bases
returned. Many of the year one challenge problems were answered in particular contexts,
i.e., with only a subset of the knowledge bases loaded. In the second program year, some
teams, including ours, needed to be prepared to answer questions in any portion of the
domain. We needed to load all of the knowledge bases simultaneously and potentially
2
reason across all of them. Thus, we needed to load a significant number of KBs
(approximately 70) that were not originally intended to be loaded together and were
written by many different authors. Our initial loading and diagnosis step was largely
manual with a number of ad hoc scripts. This was a result of two issues: time pressure in
concert with the expectation that this was a one-time task. Some of the findings from the
merging and diagnosis task follow:
(1) Large ontology merging projects may require extensive systematic support for
pervasive tests and changes. Our final ontology contained approximately 100,000
statements (and the forward chained version of the ontology after rules had been
processed contained almost a million statements). Even though the individual
ontologies all shared an “upper ontology”, there was still extensive renaming that
needed to be done to allow all the ontologies to be loaded simultaneously and to
be hooked together properly. There were also pervasive checks such as checks
for comment and source field existence as well as argument order on functions.
We discovered that different authors were using arguments in differing orders and
thus type constraints were being violated across ontologies.
(2) Large ontologies require a tool that focuses the attention of the editor in particular
portions that are interconnected (and in need of repair). There were many places
where taxonomic relationships were missing when multiple ontologies were
loaded together. For example, a class denoting nuclear weapons was related to
the weapon class but not to the weapon of mass destruction class, nor to the
disjoint partition of classes under weapon. A tool that showed (just) the relevant
portions of the taxonomies and facilitated taxonomy and partition modifications
later turned out to be extremely valuable for editing purposes.
(3) Ontologies may require small, yet pervasive changes in order to allow them to be
reused for slightly different purposes. In our HPKB task, we found a number of
roles that needed to be added to classes in order to make the classes useful for
other purposes. Also, we found a large number of roles that were inverses of
other roles but were not related by an explicit role inverse statement. Without the
inverse information, the inverse roles were not being populated and thus were not
useful for question answering even though the information appeared to be in the
knowledge base. We needed to support users in finding the connections that
needed to be made to make ontologies more useful.
2.2 Creating Class Taxonomies from Existing Web Ontologies
In a noticeably different effort, we crawled a number of web taxonomies, including
Yahoo! Shopping, Lycos, Topica, Amazon, and UN/SPSC to mine their taxonomy
information and to build CLASSIC[Borgida et. al., 1989; Brachman, et. al., 1999] and
OKBC (Open Knowledge Base Connectivity) [Chaudhri, et. al, 1998] ontologies. Our
goals for this work were to (1) “generate” a number of naturally occurring taxonomies for
3
testing that had some commercial purpose. (2) build a larger cohesive ontology from the
“best” portions of other ontologies. (“Best” was initially determined by a marketing
organization as portions of ontologies that had more usage and visibility.)
Our ontology mining, merging, and diagnosis effort had little emphasis on reasoning, but
instead was centered on choosing consistent class names and generating a reasonable and
extensible structure that could be used for all of the ontologies. The expected use of the
output ontology was for web site organization, browsing support, and search (in a manner
similar to that used in FindUR [McGuinness, 1998]). All of the same findings were
repeated in this effort, however with different instantiations.
We also found that extensive renaming was required. We also found the unique names
assumption was systematically violated within individual web ontologies and thus class
names needed their own contexts in order to be useful. Thus, systematic treatment was
required to put individual ontology branches into their own name-space and to separate
terms like steamers (under clothing appliances) from steamers (under kitchen appliances).
We also found much more need for ontological re-organization. Thus, we still required
focusing an editor’s attention on pieces of the ontology. Additionally, we found need for
more diagnostic checks with respect to ontological organization. For example, there were
multiple occurrences of cycles within class graphs, thus checks for cycles were
introduced into our diagnostics. There was also no partition information in these
ontologies, but there were multiple places where it appeared beneficial to add it.
These two experiences along with a few decades of staff experience with building
knowledge representation and reasoning systems and applications led us to design and
implement an ontology merging and diagnosis tool that we will describe next.
3 An Ontology Environment Supporting Merging and Testing
Chimaera is a new knowledge base editing, merging, and diagnosis ontology tool
developed by the Stanford University Knowledge Systems Laboratory. Its initial design
goal was to provide substantial assistance with the task of merging KBs produced by
multiple authors in multiple settings. It later took on another goal of supporting testing
and diagnosing ontologies as well. Finally, inherent in the goals of supporting merging
and diagnosis come requirements for ontology browsing and editing. We will define the
tasks of merging and diagnosis as used in our work, and then we will introduce the tool.
We consider the task of merging two ontologies to be one of combining two or more
ontologies that may use different vocabularies and may cover overlapping content. The
major two tasks are to (1) coalesce two semantically identical terms in different
ontologies so that they are referred to by the same name in the resulting ontology. (2)
identify terms that should be related by subsumption or instance relationships and provide
support for introducing those links. There are many auxiliary tasks inherent in these
tasks, such as identifying the points for editing, performing the edits, identifying when
4
two terms could be identical if they had small modifications such as a further
specialization on a value restriction, etc. We will remain at the high level of changes for
our discussion however.
The general task of merging can be arbitrarily difficult, requiring extensive (human)
author negotiation. However, we believe that merging tools can significantly reduce both
labor costs and error rates. We will support these beliefs with the results from some
initial evaluation tests.
We address the task of diagnosing single or multiple ontologies by producing a test suite
that evaluates (partial) correctness and completeness of the ontologies. The major tasks
involve finding and reporting provable inconsistencies, possible inconsistencies, and
areas of incomplete coverage. As with merging, diagnosis can be arbitrarily complex,
requiring extensive human analysis to identify all problems and present them in an order
appropriate to the problem importance. Tools built to provide the first level of analysis
however, can greatly reduce human labor cost as well as improving the consistency of the
analysis. In our diagnostic test suite, we do not attempt to find all problems; we just
choose a subset that is computationally viable and motivated by usefulness of the reports.
3.1 Chimaera
Chimæra is a Web-based browser-based editing, merging, and diagnosis tool whose
design and implementation is based on our experience developing other UIs for
knowledge applications such as the Ontolingua ontology development environment
[Farquhar, et al, 1997], the Stanford CML editor [Iwasaki, et al, 1997], the Stanford
JAVA Ontology Tool (JOT), the Intraspect knowledge server [Intraspect 1999], a few
web interfaces for CLASSIC [McGuinness, et. al., 1995; Welty, 1996], and a
collaborative environment for building ontologies for FindUR[McGuinness, 1998].
Chimaera has a web-based UI that is optimized for Netscape and MSIE browsers. It is
written in HTML, augmented with Javascript where necessary to support niceties like
spring-loaded menus and drag and drop editing. Our goal was to make it a standardsbased generic editing, merging, and diagnosis tool. When Ontolingua’s editor was first
developed, there was no standard API for knowledge-based systems. Since then the
OKBC API has been developed by Stanford’s Knowledge Systems Laboratory (KSL)
and SRI International’s AI Lab. OKBC, which allows us to develop tools that can merge
KBs in any OKBC-compliant representation system either on the same machine or over
the network. Chimæra was designed from the ground up to be a pure OKBC application.
Our typical editing environment Ontolingua, but this is not a requirement.
5
Figure 1: A view of Chimæra's user interface
The UI for the current version of Chimæra is shown in Figure 1. The interface consists of
a set of commands on spring-loaded menus (the command activates as soon as the menu
selection is made). Like most GUIs, the user selects operands by clicking on them, and
selection is shown by the selected operands being displayed in boldface. Applicable
commands are then available on the menus, and inapplicable commands are also
displayed showing the reason why they are inapplicable. The UI contains amongst its
seventy odd commands a fairly full-featured taxonomy and slot editor as well as
commands more obviously associated with the KB merging task, e.g., the “Merge
Classes” command. It also contains 17 diagnostic choices along with options for their
modification. The current UI is not a general-purpose editing environment for
ontologies. It only addresses classes and slots; non-slot individuals and facets are not
displayed. Similarly, there is no support for the editing of axioms. We plan to extend the
functionality of the tool in later versions to include all object types. In contrast to two
other merging efforts [Noy and Musen, 1999; and Chalupsky, et. al., 1997], our
environment also supports creating and editing disjoint partition information and includes
an extensive diagnostic emphasis.
The restricted nature of the UI allows us to present a view of the KB to the user that is not
cluttered by any extraneous commands, widgets or KB content. This is very important to
the design of the UI, since focus of attention is vital in the KB merging task. The user
may never be able to make merging decisions if the classes to be considered are many
screens apart. There are, therefore (currently) no fewer than 29 different commands in
6
the View menu that affect the way the KB is displayed to the user. We have worked to
identify thoughtful default choices for those commands.
Chimaera currently addresses only a portion of the overall ontology merging and
diagnosis tasks. Even though it may be viewed as an early design in terms of a complete
merging and diagnostic tool, we have found significant value in it to date. We now
describe some experiments designed to evaluate its usefulness in merging.
The experiments we have run only make use of those features in Chimaera designed to
support the merging of class-subclass taxonomies. Chimaera includes support for
merging slots and in the future, will support merging of facets, relations, functions,
individuals, and arbitrary axioms. Similarly, the diagnosis functions only include domain
independent tests that showed value in our experiments to date. These tests allow
limited user input for modifications to the tests. In our future environment, we expect to
include a diagnostic testing language that allows users to dynamically add new tests to
the test suite, and thus support more domain-dependent diagnostics as well.
3.2 Merging and Evaluation
Chimaera generates name resolution lists that help the user in the merging task by
suggesting terms from multiple ontologies that are merging candidates. Figure 2 shows a
suggestion for merging Mammalia from ontology “Test2” with Mammal from ontology
“Test1”. The suggested merging candidates may be names of classes or roles.
Candidates are put on the list by a “vigor” metric which progressively looks for more
potential merging pairs. It considers term names, presentation names (called “pretty
names” in Ontolingua), term definitions, and possible acronym and expanded forms, etc.
Chimaera also generates a taxonomy resolution list where it suggests taxonomy areas ripe
for reorganization. It uses a number of strategies for finding edit points for taxonomies.
One looks for areas of the taxonomy that contain elements from more than one ontology
(since those are likely to need some reorganization). It also looks for names of the form
“y” and “x-y” (since many terms following that pattern such as “vehicle” and “combatvehicle” are related by the subclass relationship).
We ran four experiments aimed at evaluating Chimaera’s merging effectiveness. They
are in the areas of helping to (1) coalesce ontology names (2) perform taxonomic edits (3)
identify ontology edit points (4) test overall effectiveness in a substantial merging task.
Because of space constraints here, we describe our high level findings and only describe
one of the experiments. We include tables with numerical values on our findings from all
of the experiments in the appendix. A long version of the merging experiment write-up is
available from http://www.ksl.stanford.edu/yearindex.html.
7
Figure 2: Chimæra in name resolution mode suggesting a merge of Mammal and
Mammalia
4 Experiment Findings
We conducted a set of experiments scoped to be within our resource budget that were
designed to produce a measure of the performance of Chimæra. We also compared those
results to the performance of other tools that a KB developer might reasonably use to do
such a merge, absent the KB merging tool itself. At each stage in the experiment, our
goal was to control for as many factors as possible and to assure that the experimental
settings correspond closely to the settings in which the tool would actually be used.
Our first step was to perform a set of three calibration experiments in which we
determined the number of steps and time required to do specific types of operations that
would be performed while doing a merging task using Chimæra, a representative KB
editing tool (Ontolingua), and a representative text editing tool (Emacs). These studies
were designed to provide quantitative “rate” comparisons in that they indicated which
steps in the merging task Chimæra speeds up and by how much, and to provide
qualitative indications of the steps for which Chimæra provides substantial improvements
in reliability. Using the results of these calibration experiments, we then performed a
larger merge task using only Chimæra. The calibration experiments were then used to
estimate the comparative utility of Chimæra over this larger task.
The primary results of these experiments are the following:

Merging two or more substantial ontologies was essentially not doable in a time
effective manner using a text-editing tool, primarily because of the difficulty of
examining the taxonomy of any non-trivial ontologies using only a text editor.
8

Chimaera is approximately 3.46 times faster than an ontology editing tool
(Ontolingua) for merging substantial taxonomies. Moreover, for the portion of the
taxonomy merging task for which Chimaera’s name resolution heuristics apply,
Chimaera is approximately 14 times faster than an ontology editing tool (Ontolingua).

Almost all of the operations performed during a taxonomy merge would be more
error-prone if they were performed using an ontology editing tool (Ontolingua), and
the critical “merge class” operations would be extremely error-prone if performed
using a KB editing tool.
The ontology merging task is only an interesting problem when one tries to merge large
ontology fragments. Chimaera has proved to provide considerable utility in non-trivial
merging tasks. The other tool options tried were so poor at this task that it became
impractical to perform a head-to-head experiment against other tools because the other
tools simply were not able to merge reasonably large ontologies in a reasonable amount
of time. We feel safe in concluding, therefore, that Chimæra, even though it addresses
only a portion of the overall merging task, makes a significant qualitative difference in
one’s ability to build large ontologies using fragments derived from a number of sources.
4.1 Experiment 3: Finding Edit Points
Our third experiment may be the most instructive thus we describe it here. While
optimally, we would have run the test on many pairs of similar users over many pairs of
ontologies that made sense to merge, we did not have many users nor many comparable
KBs so we attempted to control as many variables as we could with limited resources.
We asked two KR-literate users to find edit points in two ontologies. Edit points include
finding two terms that should be merged, subsumption relationships that should be added
or deleted, and partition information that should be added or deleted. (We asked our
subjects NOT to introduce new terms in an effort to control the valid edit points they
could generate). The ontology editing comparison tool was Ontolingua. The Ontolingua
user was heavily experienced in using Ontolingua as a browser and somewhat
experienced at using it as an editor. The Chimaera user had never used Chimaera as an
editor and had only used it superficially as a browser and was given an hour of training
on the tool. Both subjects were allowed to test their tool on a different ontology as
training. The ontologies were one fragment of the IKB from Cycorp on Agents and the
Agents ontology that was used by the entire SAIC team of HPKB. The results, in Figure
2, show that the Chimaera user was vastly more productive initially and was noticeably
more productive throughout the experiment. The initial segment is when the Chimaera
user is directly working off of the name resolution menu. This accurately suggested
9
Experiment 3: Chimæra vs. Ontolingua editor
Cumulative operations
100
80
60
Chimæra
Ontolingua Editor
40
20
0
0
400
800
1200
1600
2000
2400
2800
3200
3600
Time (s)
Figure 3: Chimæra proved to be superior to the Ontolingua editor at finding
candidate edit points
many merges with similar names and also suggested some acronym expansions. In this
initial phase, the Chimaera user was 14 times as productive as the Ontolingua user, and
throughout the experiment, she was at least 3-4 times as productive.
5 Diagnostics Tests
Chimaera also has a set of diagnostics that can be run selectively or in their entirety.
Routinely when we obtain knowledge bases now, we run the diagnostics and invariably
find issues with our incoming KBs. The current list of diagnostics was derived as a
retrospective analysis of the most useful domain independent tests that we needed to run
on the HPKB and on the crawled web ontologies. They group into four areas:
(1) simple checks for incompleteness (missing argument names, missing
documentation strings, missing sources, missing type constraints, missing term
definitions);
(2) syntactic analysis (incidence of words (or sub-strings), possible acronym
expansion);
(3) taxonomic analysis (redundant super classes, redundant types, trivial instances or
subclasses of THING, definition extensions from included ontologies), and
(4) semantic evaluation (slot
domain/range mismatch).
value/type mismatch, class definition cycle,
10
This is obviously not everything that could be checked. The current diagnostic suite does
not connect to the full theorem prover so there is only limited consistency checking. The
current testing environment also does not give users the power to add their own,
potentially domain-specific, checks. Even with the limited power of the diagnostics set
though, we successfully used it to provide initial correctness and completeness checks of
all incoming HPKB knowledge bases for our final team evaluation. Possibly more
importantly, its output was usable by people with little training in knowledge
representation. Also, the tool takes multiple input formats, thus we were able to allow
people to use it who had no familiarity with OKBC or Ontolingua. We had some SNARK
and KIF-literate users load in their ontologies in the input format they were familiar with,
run diagnostics, and debug their knowledge bases with little intervention from us. We
also used this toolset to check for problems in our semi-automatically generated
ontologies from web crawls. The tests found a surprising number of things that would
have been tedious or difficult for us to find ourselves, such as class cycles and
inconsistency in naming in Amazon’s ontology.
6 Conclusion
We have presented an ontology editing, merging, and diagnostic environment developed
to meet the emerging needs of representation and reasoning tasks on the web and
ontology creation and maintenance tasks. We have briefly overviewed the merging and
diagnostics components and presented some evaluation results on the merging side and
some anecdotal reports on the diagnostics side. While our tool is in its early stages, we
believe that it makes significant improvements in productivity and quality of ontology
development and maintenance. We believe this from our own personal use of the tool,
from evaluations of the tool, and from demand from commercial and academic needs
after the tool was built. We are continuing to develop the tool focusing in particular on
extending its reasoning capabilities, its extensibility, and usability by non-experts.
Acknowledgement
The authors are indebted to DARPA for their support of this research under contract N66001-97-C-8554,
Large-Scale Repositories of Highly Expressive Knowledge. We would also like to thank Cycorp for kindly
providing knowledge base fragments for some merging experiments and Bob Schrag at IET for his support
in the design and conducting of the experiments.
11
Bibliography
1. Alex Borgida, Ronald J. Brachman, Deborah L. McGuinness, and Lori Alperin
Resnick. CLASSIC: A Structural Data Model for Objects, Proceedings of the 1989
ACM SIGMOD International Conference on Management of Data, Portland, Oregon,
June, 1989, pp. 59--67.
2. Ronald J. Brachman, Alex Borgida, Deborah L. McGuinness, and Peter F. PatelSchneider. "Reducing" CLASSIC to Practice: Knowledge Representation Theory
Meets Reality. To appear in the Artificial Intelligence Journal, 1999.
3. Hans Chalupsky, Eduard Hovy, Tom Russ. Progress on an Automatic Ontology
Alignment Methodology. Proceedings of the ANSI ad hoc group on ontology
standards, 1997. Available from http://ksl-web.stanford.edu/onto-std/ .
4. Vinay Chaudhri, Adam Farquhar, Richard Fikes, Peter Karp, and James Rice.
OKBC: A Programmatic Foundation for Knowledge Base Interoperability.
Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI98). AAAI Press. (http://www.ksl.stanford.edu/KSL_Abstracts/KSL-98-08.html)
5. Paul R. Cohen, Robert Schrag, Eric Jones, Adam Pease, Albert Lin, Barbara Starr,
David Easter, David Gunning, and Murray Burke. The DARPA High Performance
Knowledge Bases Project. In Artificial Intelligence Magazine. Vol. 19, No. 4, pp.2549, 1998.
6. Adam Farquhar, Richard Fikes, and James Rice. The Ontolingua Server: a Tool for
Collaborative Ontology Construction. International Journal of Human-Computer
Studies 46, 707-727, 1997. (http://www.ksl.stanford.edu/KSL_Abstracts/KSL-9626.html)
7. Gleb Frank, Adam Farquhar, and Richard Fikes. Building a Large Knowledge Base
from a Structured Source: The CIA World Fact Book. IEEE Intelligent Systems, Vol.
14, No. 1, January/February 1999. Also, KSL Technical Report KSL-98-16.
8. Intraspect
Knowledge
Server,
Intraspect
(http://www.intraspect.com/product_info_solution.htm)
Corporation,
1999.
9. Y. Iwasaki, A. Farquhar, R. Fikes, & J. Rice. A Web-based Compositional Modeling
System for Sharing of Physical Knowledge. Morgan Kaufmann, Nagoya, Japan, 1997.
(http://www.ksl.stanford.edu/KSL_Abstracts/KSL-98-17.html).
10. A.T. McCray and S.J. Nelson. The representation of meaning in the UMLS. Methods
of Information in Medicine. 1995; 34: 193-201.
12
11. Deborah L. McGuinness. Ontologies for Electronic Commerce, Proceedings of the
AAAI Artificial Intelligence for Electronic Commerce Workshop, Orlando, Florida,
July, 1999.
12. Deborah L. McGuinness. Ontological Issues for Knowledge-Enhanced Search. In the
Proceedings of Formal Ontology in Information Systems, June 1998. Also in
Frontiers in Artificial Intelligence and Applications, IOS-Press, Washington, DC,
1998.
13. Deborah L. McGuinness and Peter Patel-Schneider. Usability Issues in Knowledge
Representation Systems. In Proceedings of the Fifteenth National Conference on
Artificial Intelligence, Madison, Wisconsin, July, 1998. This is an updated version of
Usability Issues in Description Logic Systems published in Proceedings of
International Workshop on Description Logics, Gif sur Yvette, (Paris), France,
September 1997.
14. Deborah L. McGuinness, Lori Alperin Resnick, and Charles Isbell. Description
Logic in Practice: A CLASSIC: Application. In Proceedings of the 14th International
Joint Conference on Artificial Intelligence, Montreal, Canada, August, 1995.
15. Deborah L. McGuinness and Jon Wright. An Industrial Strength Description Logicbased Configurator Platform. IEEE Intelligent Systems, Vol. 13, No. 4, July/August
1998, pp. 69-77.
16. Deborah L. McGuinness and Jon Wright. Conceptual Modeling for Configuration: A
Description Logic-based Approach. In the Artificial Intelligence for Engineering
Design, Analysis, and Manufacturing Journal - special issue on Configuration, 1998.
17. Natalya Fridman Noy and Mark A. Musen. An Algorithm for Merging and Aligning
Ontologies: Automation and Tool Support in AAAI-99 Workshop on Ontology
Management. Also, SMI Technical Report SMI-99-0799.
18. K.A. Spackman, K.E. Campbell, and R.A. Cote. SNOMED-RT: a reference
terminology for health care. Proceedings of the 1997 AMIA Annual Fall Symposium.
1997; 640-644.
19. Chris Welty. An HTML Interface for CLASSIC. Proceedings of the 1996 International
Workshop on Description Logics. AAAI Press. November, 1996.
13
14