The MedKAT Pipeline

Document ingestion identifies sections, subsections, etc., into annotations and simultaneously adds derived information, such as the number of sections and subsections, header information, and correlations between disjoint pieces of text describing the same tissue specimen. The derived information is based both on textual labels and visual (formatting) cues within the document.

The pipeline can use any tokenizer that has been written as a UIMA annotator. For optimal performance, all textual resource used within the pipeline, such as terminologies and ontologies, are expected to be tokenized in the same fashion as the documents that are analyzed.

The determination of sentence boundaries in the provided pipeline is done using OpenNLP, and then additional annotators are executed to adjust previously determined sentence annotations to take into account the structure of medical documents and their implied meaning. Examples of potential issues for a general sentence detector are list and header processing, parenthesis processing and non-standard use of punctuation symbols. Similarly, part-of-speech (POS) tagging is done using the OpenNLP POS tagger, and then a few domain-specific tokens are corrected.

In several of the algorithms the context plays an integral part. Context is defined here as the range of text within a document used to determine the semantic meaning of a word or phrase. To determine context, the pipeline uses the OpenNLP shallow parser.

Concept finding is one of the most critical components in our system. It maps textual mentions to terminology concepts to create codified information. This task is directed by ConceptMapper ^[1] , which creates candidate matches between concept structures based on a terminology and unstructured text. A complete description of ConceptMapper is provided in the document The ConceptMapper Annotator . The mapping of textual mentions to concepts depends on the final intended application, and for our purposes, ConceptMapper has been configured to generate many potential matches, allowing for overlapping and interwoven results, and a subsequent set of rule-based annotators filter out overgenerated matches depending numerous criteria. For example, the snippet “hepatic flexure of the colon” can be codified using ICD-O as “hepatic” (C22.0), “colon” (C18.9) and “hepatic flexure of the colon” (C18.3). Note that the ICD-O entry for C18.3 actually reads as “hepatic flexure of colon” (note the absence of “the” before “colon”), which demonstrates why exact string matching is not sufficient to codify unstructured text. ConceptMapper finds all possible mappings between the terminology and the free text (C22.0, C18.9, C18.3), and the subsequent filters mark the ones to be ignored due to a potential term subsumption (C22.0, C18.9). Of course, while using the longest match heuristic would avoid the need for such filtering in this simplified case, there are more complex examples where that is not sufficient.

The “hepatic flexure of the colon” example described earlier showed the necessity for some filtering. For example, one such filtering rule is the subsumption rule, which specifies whether contained annotations, e.g. “hepatic” should be exposed or not. Other filters mark annotations based on particular values of one of their attributes and another removes duplicate and identical annotations. One filter discovers subsumed generic anatomical sites or histologies and marks them. There is also a filter that handles nominal ellipsis. Consider for example the phrase “Colon, ascending and transverse.” One interpretation is that it refers to two sites (ascending colon and transverse colon), another interpretation is that the physician described three sites, colon, ascending colon and transverse colon, as denoted by the use of the comma, and this is the interpretation that is assumed by MedKAT/P.

MedKAT/P also contains regular expression-based annotators which, in conjunction with a terminology, discover textual mentions describing dimensions and sizes, dates, number of excised and positive lymph nodes and stage.

The negation detector is a generalized algorithm as reported in [ChapmanEtAl2001] . Negation trigger words (e.g. “no,” “ruled out”) are specified in a user modifiable dictionary. The trigger words become the anchors around which negated phrases are discovered within a user specified window. Here we report on results assuming that the window is a sentence. Within the predefined window, the algorithm examines generalized noun phrases starting to the right of the negation keyword. If none is found, then it continues examining phrases to the left. When a generalized noun phrase is found, all semantic entities are marked as negated.

The next step in the pipeline is to discover the relationship between the appropriate leaf classes (e.g. histology, anatomical sites, size, grade) to populate container classes such as the primary and metastatic tumor classes, the lymph node class and the gross description part class. The relations between the classes are “contains” and “is-part-of.” There is a common methodology in filling container classes, coupled with certain class specific rules as outlined in the next paragraphs. We will first outline the common methodology applied to instantiate the primary and metastatic tumor classes, the lymph node class and the gross description part class and then provide more specific details and examples. At this point in the processing, the Concept Finding portion of the pipeline has already identified the leaf classes of anatomical site, histology, grade value, stage and dimension and the container class size.

The first step is to determine which section of a document should be considered for instantiating a container class. For instance, in general gross description part classes are populated only from the gross description section of a pathology report. The tumor and lymph node classes take information from the final diagnosis section. The relation between class and document section is specified in the configuration file associated with the annotator responsible for discovering a particular class. Second, certain classes are categorized according to multiple criteria (e.g., primary vs. metastatic vs. benign, tumor size vs. margin size). Third, we determine which mentions refer to each other (i.e., are co-referring). Fourth, we determine which instances of classes (e.g. histology or anatomical site) should be considered candidates for populating each of the container classes (e.g. primary tumor class or lymph node class). In the final fifth step container classes are merged or split according to class specific rules.

In step two described above, some classes are categorized. One categorization labels classes as positive or negated with respect to a particular class. A class whose negation attribute is set to true by the negation detector is negated with respect to all classes. An example of a negated histology is the phrase “tumor free,” where tumor is a histology that our negation algorithm (previously described) marked as such. A class which is negated with respect to a particular class is referred to as excluded with respect to a class. Exclusion states that an instance of a class can be part of only a single container class. For instance, an anatomical site mentioned as part of an invasion class is excluded from consideration for filling a tumor class. Anatomical sites are categorized into originating sites, lymph nodes, invasion sites and other sites. Histology classes are categorized as metastatic and non-metastatic. Size mentions are categorized as tumor sizes and other sizes. Our categorization algorithm is based on a set of trigger phrases (specific to a particular categorization) and the noun phrase hierarchy previously described. For each class instance to be categorized, the algorithm checks whether an appropriate trigger word is co-occurrent with the mention attribute of the class. Co-occurrence is defined based on the noun hierarchy, which means that noun phrases, noun phrase lists and prepositional noun phrases are inspected in turn. In addition, ICD-O codes are used for categorization of histologies and anatomical sites.

Although pronominal anaphora resolution is not required for analysis of pathology reports, co-reference resolution is critical in populating the CDKM. Co-reference is based on codes associated to a concept, such as ICD-O. The methodology for discovering co-referenced generic histology classes is similar to pronoun resolution [KennedyBoguraev1996] . For each histology H , examine each generic histology GH mentioned after H and which is categorized equivalently to H . Only generic histologies GH occurring between H and a subsequent equally categorized histology H1 are considered. The set of generic histologies GH is co-referenced with H . The following examples may clarify the algorithm some more. Let HM1 and HM2 be two metastatic histologies, HN1 a non-metastatic histology, and GHM1 , GHM2 and GHN1 be generic metastatic and non-metastatic histologies occurring in the following sequence:

		HM1 GHM1 GHM2 HN1 GHM3 GHN1 HM2.

Here GHM1 GHM2 and GHM3 will be co-referenced with HM1 and GHN1 with HN1 respectively.

Besides generic rules for populating class instances, we used class-specific rules as well. For example, the gross description part class may contain one or more anatomical sites and a size. Processing of the document starts with an initial anatomical site within the gross description section and continues with all the other anatomical sites within the same context (i.e. hierarchy of noun phrases) until either a size is found, or the hierarchy is exhausted. At this point, the size expression is parsed to determine whether a single size or a range of sizes was specified. In the latter case, two gross description part classes are instantiated, both having the same anatomical sites but different sizes. This is a class specific implementation of step (5) of the general algorithm.

The primary and metastatic tumor classes are populated simultaneously by a single TumorModelAnnotator . The assumption is that tumor classes are populated with information within a user-defined portion of the document – the Tumor Context TC . The algorithm iterates through multiple steps. First it identifies all non-negated histologies within TC . Second, for all identified histologies, it examines the noun phrase containing the histology for all occurrences of any of these three classes: anatomical sites, grade values and sizes and associates them with the histology. It is noteworthy that each of these classes can be associated with only a single histology and hence, once an association is found, it is removed from further consideration. Third, for histologies missing one or more of these associations, step two is repeated, but for noun phrase lists instead of noun phrases. Fourth, step two is repeated for any histology missing any associations within the context of a sentence. Ultimately, tumor classes which have co-referenced histologies are merged into a single instance. Classes which refer to the same exact anatomical site(s), grade value(s) and sizes and differ only in the histologies are merged as well. An artifact of pathology reports is that anatomical sites are at times implied to be the same as the sites mentioned in the gross description. To account for this, for any non-negated histology that has no anatomical site associations, we extend the context to the gross description part of the document. For the particular pathology reports used for this study, as is the norm, the first sentence of the tumor context TC is considered to be the gross description. The final step is to instantiate the tumor classes based on the categorization of the histology and anatomical sites. It is important to consider all histologies (including benign ones) and all anatomical sites in the process to identify associations correctly, but neither benign histologies, nor histologies with an associated anatomical site that is a lymph node, are considered for primary or metastatic tumor classes. The category of the remaining histologies (metastatic or non-metastatic) determines which type of tumor class is instantiated.

Lymph nodes classes have attributes that are anatomical sites, histologies, and lymph node expressions ( LNE ). In particular, only anatomical sites which have been categorized as lymph nodes ( AS-L ) are considered. LNE 's describe either the general state (positive/negative) of the lymph nodes or provide more detail in terms of number of positive lymph nodes and excised nodes (from which the state is deduced). For each AS-L , the algorithm for instantiating lymph node classes determines the histologies and LNE ’s co-occuring with the anatomical site ( AS-L ) in the same sentence. If they are not found, the context is expanded to sentences within the same section. A set of rule-based filters are applied to derive the correct associations, taking the categorization of histologies and anatomical sites into positive and negative classes into account.

To populate the gross description part class, we introduced two new syntactic structures – the ParenthesisSeparatedNoun-phrase ( PSN ) and the ParenthesisPhrase ( PPH ). A sequence of a noun phrase, a parenthesis, a noun phrase followed by another parenthesis is called a PSN . Any expression enclosed with matching parenthesis is a PPH . We define a hierarchy of syntactic constructs consisting of the following levels: noun phrase, PSN , generalized noun phrase and PPH . The algorithm for populating a gross description part examines the syntactic hierarchy in order, with noun phrases being at level 0 and PPH being at level 4. If anatomical site(s) and size expression(s) co-occur in the same syntactic structure, one or more gross description part classes are instantiated. The number of instantiated classes depends on the type and number of size expressions found. If an anatomical site AS occurs without a size expression within a syntactic structure, a set of rules determines whether the AS should be associated with an already existing gross description part class or a new class be instantiated. The rules depend on the lexical ordering of the anatomical site and size mentions.

The requirements on the design of the ConceptMapper dictionary were that it be easily extensible and that arbitrary properties could be associated with individual entries. Additionally, the set of properties could not be fixed, but rather customizable for any particular application.

The structure of a ConceptMapper dictionary is quite flexible and is expressed using XML (see Example 3.1, “Sample dictionary entry”). Specifically, it consists of a set of entries, specified by the <token> XML tag, each containing one or more variants (synonyms), the text of which is specified using by the "base" feature of the <variant> XML tag. Entries can have any number of associated properties, as needed. Individual variants (synonyms) inherit features from their parent token (i.e., the canonical form), but can override any or all of them, or even add additional features.

In the following sample dictionary entry, there are 6 variants, and according to the rules described earlier, each inherits the all attributes from the canonical form (canonical, CodeType, CodeValue, and SemClass), though the variants “colonic” and “colic” override the value of the POS (part of speech) attribute:

			<token canonical="colon, nos"
			       CodeType="ICDO" CodeValue="C18.9"
			       SemClass="Site" POS="NN">
				<variant base=”colon, nos”/>
				<variant base=”colon”/>
				<variant base="colonic" POS="JJ" />
				<variant base="colic" POS="JJ" />
				<variant base="large intestine" />
				<variant base="large bowel" />
			</token>

The result of running ConceptMapper are UIMA annotations, and there are two configuration parameters that are used to map the attributes from the dictionary (see AttributeList) to features of UIMA annotations (see FeatureList).

The entire dictionary is loaded into memory, which, in conjunction with an efficient data structure, provides very fast lookups. As stated earlier, dictionaries with millions of entries have been used without any performance issues. The obvious drawback to storing the dictionary in memory is that large dictionaries require large amounts of memory; this is partially mitigated by the fact that the dictionary is implemented as a UIMA shared resource (see DictionaryFile). This means that multiple annotators, such as multiple instances of ConceptMapper that are set up using different parameters, can all access it without having to load it more than once. The dictionary loader is specified in the external resource section of the descriptor, and is expected to implement the interface org.apache.uima.conceptMapper.support.dictionaryResource.DictionaryResource. Two implementations are included in the distribution, org.apache.uima.conceptMapper.support.dictionaryResource.DictionaryResource_impl, the standard implementation, which loads an XML version of a dictionary, and org.apache.uima.conceptMapper.support.dictionaryResource.CompiledDictionaryResource_impl which loads a pre-compiled version, for faster loading. The compiler is supplied as org.apache.uima.conceptMapper.dictionaryCompiler.CompileDictionary, which takes two arguments, a ConceptMapper analysis engine descriptor that loads the dictionary using the standard dictionary loader, and the name of the output file into which to write the compiled dictionary.

Input documents are processed on a token-by-token basis, so it is important that the dictionary entries are tokenized in the same way as the input documents. To accomplish this, ConceptMapper allows any UIMA analysis engine to be specified as the tokenizer for the dictionary entries. See parameter TokenizerDescriptorPath for details.

As stated earlier, input documents are processed on a token-by-token basis. Tokens are processed one span (e.g., a sentence or a noun phrase) at a time. Token annotations are specified by the parameter TokenAnnotation, while span annotations are specified by the parameter SpanFeatureStructure. By default, all tokens within a span are considered, and it is the text associated with each token that is used for lookups. ConceptMapper can also be configured to consider tokens differently:

The ability to skip tokens during lookups based on particular feature values makes it easy to skip, for example, all tokens with particular part of speech tags, or with some previously computed semantic class. For example, given the text below in Example 3.2, “Sample Input Text”:

				Infiltrating mammary carcinoma

Assume each word is a token that has a feature SemanticClass, and that feature for the token “mammary” contains the value “AnatomicalSite”, while the tokens “Infiltrating” and “carcinoma” do not. It is then possible to configure ConceptMapper to indicate that tokens that have a particular feature, in this case SemanticClass, equal to one of a set of values, in this case “AnatomicalSite”, should be excluded when performing dictionary lookups (see parameters ExcludedTokenClasses and ExcludedTokenTypes). By doing this, for the purposes of dictionary lookup, the example text would effectively appear to be:

				Infiltrating carcinoma

In addition to the set of feature values that indicate their associated token are to be excluded during lookup, there are also configuration parameters that can be used to specify a set of feature values for inclusion (see parameters IncludedTokenClasses and IncludedTokenTypes). The algorithm for selecting annotations to include during lookup is as follows:

				if there is an includeList but no excludeList
				include annotation if feature value in includeList

				else if there is an excludeList
				exclude annotation if feature value in excludeList

				else
				include annotation

This provides a simple way to restrict the selection of pre-classified tokens, whether that pre-classification is done via previous instances of ConceptMapper or some altogether different annotator. See TokenTextFeatureName

The actual dictionary lookup algorithm is controlled by three parameters. One specifies token-order independent lookup (OrderIndependentLookup). For example, a dictionary entry that contained the variant:

			<variant base='carcinoma, infiltrating'/>

would also match against any permutation of its tokens. In this case, assuming that punctuation was ignored, it would match against both “infiltrating carcinoma” and “carcinoma, infiltrating”. Clearly, this particular setting must be used with care to prevent incorrect matches from being found, but for some domains it enables the use of a more compact dictionary, as all permutations of a particular entry do not need to be enumerated.

Another parameter that controls the dictionary lookup algorithm toggles between finding only the longest match vs. finding all possible matches (FindAllMatches). For the text:

			... carcinoma, infiltrating ...

If there was a dictionary entry for “carcinoma” as well as the entry for “carcinoma, infiltrating”, this parameter would control whether only the latter was annotated as a result or both would be annotated. Using the setting that indicates all possible matches should be found is useful when subsequent disambiguation is required.

The final parameter that controls the dictionary lookup algorithm specifies the search strategy (SearchStrategy), of which there are three. The default search strategy only considers contiguous tokens (not including tokens from the stop word list or otherwise skipped tokens, as described above), and then begins the subsequent search after the longest match. The second strategy allows for ignoring non-matching tokens, allowing for disjoint matches, so that a dictionary entry of

			    A C

would match against the text

			    A B C

This can be used as alternative method for finding “infiltrating carcinoma” over the text “infiltrating mammary carcinoma”, as opposed to the method described above, wherein the token “mammary” had to have been have been somehow pre-marked with a feature and that feature listed as indicating the token should be skipped. On the other hand, this approach is less precise, potentially finding completely disjoint and unrelated tokens as a dictionary match. As with the default search strategy, the subsequent search begins after the longest match.

The final search strategy is identical to the previous, except that subsequent searches begin one token ahead, instead of after the previous match. This enables overlapped matching. As with the setting that finds all matches instead of the longest match, using this setting is useful when subsequent disambiguation is required.

Output is in the form of new UIMA annotations. As previously discussed, the mapping from dictionary entry attributes to the result annotation features can also be specified. Given the fact that dictionary entries can have multiple variants, and that matches could contain non-contiguous sets of tokens, it can be useful to be able to be able to know exactly what was matched. There are two parameters that can be used to provide this information. One allows the specification of a feature in the output annotation that will be set to the string containing the matched text. The other can be used to indicate a feature to be filled with the list of tokens that were matched. Going back to the example in figure 2, where the token “mammary” was skipped, the matched string would be set to “Infiltrating carcinoma” and the matched tokens would be set to the list of tokens “Infiltrating” and “carcinoma”.

Another output control AE descriptor parameter can be used to specify a feature of the resultant annotation to be set to contain the span annotation enclosing the matched token. Assuming, for example, that the spans being processed are sentences, this provides a convenient way to link the resultant annotation back to its enclosing sentence.

It is also possible to indicate dictionary attributes to store back into each of the matched tokens. This provides the ability for tokens to be marked with information regarding what it was matched against. Going back to the example in figure 2, one way that the SemanticClass feature of the token “mammary” could have been labeled with the value “AnatomicalSite” was using this technique: a previous invocation of ConceptMapper had “mammary” as a dictionary entry, that entry had the SemanticClass feature with the value “AnatomicalSite”, and SemanticClass was listed as an attribute to write back as a token feature. If, instead of “mammary” the match was against a multi-token entry, then each of the multiple tokens would have that feature set.

Detect sentence boundaries and create sentence annotations that span these boundaries. Uses the OpenNLP MaxEnt Sentence Detector. Create annotation of type uima.tt.SentenceAnnotation on sentence boundaries.

MedKAT_NLP/descriptors/analysis_engine/primitive/MedKATSentenceDetector.xml

Tokenize the text and create token annotations that span the tokens. The tokenization is performed using the OpenNLP MaxEnt tokenizer, which tokenizes according to the Penn Tree Bank tokenization standard. In general, tokens are separated by white space, but punctuation marks (e.g., ".", ",", "!", "?", etc.) and apostrophe'd endings (e.g., "'s", "'nt", etc.) are separate tokens. Create annotation of type uima.tt.TokenAnnotation on token boundaries .

MedKAT_NLP/descriptors/analysis_engine/primitive/MedKATTokenizer.xml

Reset sentence boundaries to exclude trailing spaces.

MedKATp/descriptors/analysis_engine/primitive/SentenceShortener.xml

Divide document into sections based on parameter settings.

MedKATp/descriptors/analysis_engine/primitive/SectionFinder.xml

Divide document sections into sub-sections based on parameter settings.

MedKATp/descriptors/analysis_engine/primitive/SubsectionDetector.xml

Divide document sub-sections into sub-sub-sections based on parameter settings.

MedKATp/descriptors/analysis_engine/primitive/SubSubsectionDetector.xml

Create annotations that enclose matching parentheses, as "(" and ")" or "[" and "]" or "{" and "}".

MedTKATp/descriptors/analysis_engine/primitive/SyntacticUnitFinder.xml

Creates annotations at location of the "\n" character.

MedKATp/descriptors/analysis_engine/primitive/NewLineSentenceAnnotator.xml

None.

Finds the Diagnosis and Gross Description section annotations and splits it into subsections and possibly a bullet list entries.

MedKATp/descriptors/analysis_engine/primitive/TextDocParser.xml

Finds sentences that overlap sections and breaks them into 2 smaller sentence annotations. This annotator detects sentences that cross section boundaries, and breaks those sentences into smaller ones within each section. It deletes the original annotation and leaves only the new ones. Uses only annotations specified in the descriptor file.

MedKATp/descriptors/analysis_engine/primitive/SentenceBreakDetector.xml

Defined by parameters.

Combines any sentence annotations within parentheses into a single sentence. This is designed for use with Mayo pathology documents where short phrases, which may include periods, are parenthesized. It combines them. It will not do what you want if there are two actual sentences within the parentheses.

MedKATp/descriptors/analysis_engine/primitive/ParenSentenceCombiner.xml

An aggregate to parse the document and create phrasal and clausal annotations over the text. Uses the OpenNLP MaxEnt parser. Assigns POS tags to tokens as part of the processing.

MedKAT_NLP/descriptors/analysis_engine/aggregate/MedKATNLP_RunSP.xml

Assigns part of speech tags to tokens using the OpenNLP MaxEnt part of speech tagger. Requires that sentence and token annotations have been created in the CAS. Updates the POS field of each token annotation with the part of speech tag.

MedKAT_NLP/descriptors/analysis_engine/primitive/MedKATPOSTagger.xml

Parse the document and create phrasal and clausal annotations over the text. Uses the OpenNLP MaxEnt parser. This analysis engine takes a parameter called "ParseTagMapping" which maps each parse tag to a syntax annotation type. The parse tags come from the standard Penn Tree Bank phrase and clause tags (produced by the OpenNLP parser), and each syntax annotation type must be defined in the type system and have a corresponding JCas Java class.

MedKAT_NLP/descriptors/analysis_engine/primitive/MedKATParser.xml

Map dictionary entries onto input documents using ConceptMapper. See Chapter 3, The ConceptMapper Annotator for detailed documentation of ConceptMapper.

MedTKATp/descriptors/analysis_engine/primitive/ConceptMapperInitalPass.xml

Map dictionary entries onto input documents using ConceptMapper. See Chapter 3, The ConceptMapper Annotator for detailed documentation of ConceptMapper.

MedTKATp/descriptors/analysis_engine/primitive/ConceptMapperSecondPass.xml

Map dictionary entries onto input documents using ConceptMapper. See Chapter 3, The ConceptMapper Annotator for detailed documentation of ConceptMapper.

MedTKATp/descriptors/analysis_engine/primitive/ConceptMapperLymph.xml

Map dictionary entries onto input documents using ConceptMapper. See Chapter 3, The ConceptMapper Annotator for detailed documentation of ConceptMapper.

MedTKATp/descriptors/analysis_engine/primitive/ConceptMapperNegator.xml

Map dictionary entries onto input documents using ConceptMapper. See Chapter 3, The ConceptMapper Annotator for detailed documentation of ConceptMapper.

MedTKATp/descriptors/analysis_engine/primitive/ConceptMapperTumorOrigin.xml

Set of filters to mark named entities variously, to prevent further processing of them. Markers are defined as an integer mask, with the following values:

MedTKATp/descriptors/analysis_engine/aggregate/DictTermFilters.xml

None.

None.

Mark entities as "Subsumed", if necessary (see DictTermFilters for description of markers).

Entities containing commas are not processed. Otherwise, entities are marked as Subsumed if they have fewer tokens than another term and begin and end indices either match or fall between another entity's begin and end indices.

MedTKATp/descriptors/analysis_engine/primitive/SimpleSubsumptionFilter.xml

Mark entities as "Modifier", if necessary (see DictTermFilters for description of markers).

If more than one term in span, and the span has no commas, mark all but last one as a modifier.

MedTKATp/descriptors/analysis_engine/primitive/ModifierTermFilter.xml

Mark entities as "Subsumed", if necessary (see DictTermFilters for description of markers).

Marks terms as "Disallowed" if they contain a term that is:

MedTKATp/descriptors/analysis_engine/primitive/CommaAndDisallowedFilter.xml

Mark entities as "Ignored", if necessary (see DictTermFilters for description of markers).

Entities are marked as Ignored if they have multiple tokens or if their code value matches an entry in the IgnoredTermCodes parameter.

MedTKATp/descriptors/analysis_engine/primitive/IgnoredTermFilter.xml

Mark entities as "Duplicate", if necessary (see DictTermFilters for description of markers).

Entities with the same AttributeValue and begin and end indices are considered duplicates.

MedTKATp/descriptors/analysis_engine/primitive/DuplicateTermFilter.xml

Mark entities as "Subsumed", if necessary (see DictTermFilters for description of markers).

Mark duplicates (duplicates are defined as having equal begin, end and semantic class)

Entities containing commas are not processed if their SemClass is specified in the CommaOverridesSubsumption parameter. Otherwise, entities are marked as Subsumed if they have fewer tokens than another term and begin and end indices either match or fall between another entity's begin and end indices, depending on the setting of the TokensMatchCriterion parameter.

MedTKATp/descriptors/analysis_engine/primitive/SubsumptionFilter.xml

Finds and annotates dates.

MedTKATp/descriptors/analysis_engine/primitive/DateFinder.xml

None.

Using negation terms (as specified by "NoTermType" annotations) in conjunction with noun-phrase chunking information, find negated tokens and mark them as such (current implementation assume tokens are of type "org.ohnlp.medkat.taes.conceptMapper.DictTerm"). Initially, tokens are considered following the negation term, and if none are found, then prior to the negation term. See negation algorithm for more details.

MedTKATp/descriptors/analysis_engine/primitive/DrNo.xml

Finds and annotates sizes or ranges of sizes, up to three dimensions, including unit expressions. Units are limited to 'cm' and 'mm'

MedTKATp/descriptors/analysis_engine/primitive/DimensionFinder.xml

None.

Finds co-referring diagnoses, as defined in the discussion of the co-referencing algorithm, above. Makes two assumptions:

A new version of this should be created without these assumptions.

MedTKATp/descriptors/analysis_engine/primitive/DiagnosisCoreferencer.xml

Finds co-referring sites, as defined in the discussion of the co-referencing algorithm, above, above. Makes two assumptions:

A new version of this should be created without these assumptions.

MedTKATp/descriptors/analysis_engine/primitive/SiteCoreferencer.xml

Create annotations that enclose matching parentheses, as "(" and ")" or "[" and "]" or "{" and "}".

MedTKATp/descriptors/analysis_engine/primitive/GradeDetector.xml

Converts instances of internal MedKAT named entity types to SCRxxx external types. The conversion is performed for anatomical sites, diagnoses and generic named entities. Instances of internal types that contain coreferenced objects are also converted to the corresponding external types.

MedKATp/descriptors/analysis_engine/primitive/SCRNamedEntityTypeConverter.xml

None

Matches regular expressions in document text.

MedKATp/descriptors/analysis_engine/primitive/SizeLocationRegExAnnotator.xml

Disambiguates between different size types.

MedKATp/descriptors/analysis_engine/primitive/Disambiguator.xml

Parses lymph nodes expressions and populates appropriate attributes in annotations of the corresponding type set by parameters.

MedKATp/descriptors/analysis_engine/primitive/LymphStatus.xml

For related NP annotations of different types produces a single annotation that includes all relevant pieces.

MedKATp/descriptors/analysis_engine/primitive/NPMerger.xml

None

Creates annotations that contain information about lymph nodes described in the report. (See Lymph nodes model algorithm for the algorithm overview.

MedKATp/descriptors/analysis_engine/primitive/LymphNodesAnnotator.xml

Creates annotations that contain information about gross description parts described in a report. (See Gross description model algorithm for the algorithm overview.

MedKATp/descriptors/analysis_engine/primitive/GrossDescriptionAnnotator.xml

Creates annotations that contain information about primary and metastatic tumors. (See tumor model algorithm) for the algorithm overview.

MedKATp/descriptors/analysis_engine/primitive/TumorModelAnnotator.xml