Query Resiliency for Cognitive Search: Enabling Natural Language Questions

By Margot Hughan, Product Manager, Kyndi

Companies generate a lot of content to enable their customers and employees to find a path forward and answer questions. This improves the productivity of an organization by enabling employees to find solutions in real time, and also frees up live customer support resources for novel questions.

However, a lot of this content goes unused since it is difficult to access through natural language questions. Users naturally have different ways of phrasing a question. Additionally, they may use short hard references or introduce typos. Machine Learning solutions that require extensive expertise, time, and data to develop and are optimized around a rigid, pre-defined set of queries are too limiting and restrictive. A true cognitive search needs to be able to adapt to any user and return relevant results. 

Kyndi cognitive search enables your users to ask or investigate what they are looking for in the way that is natural to them, taking on the burden of understanding the intent of a query and finding the most relevant content, regardless of how it is phrased. Below are some of the cognitive abilities that Kyndi uses to make Kyndi’s cognitive search truly resilient to variations in a query.

 

  • Out-of-the-box cognitive abilities for query resiliency
    • SDFM: Connect words in a query to common synonyms or words with semantic similarity.
    • LEMMA: Connect words in a query to lemma variations in a corpus.
    • MORPH: Connect words in a query to words that are closely associated based on shared morphemes.
    •  LEXEME: Connect words in a query to words that are closely associated based on a shared stem.
    • TYPO: Connect misspellings in a query to a correctly spelled version.
    • ANAPHOR: Connect short and long-form versions of names of people.
  • Opportunities to extend query resilience
    • CANON: Connect words in a query to user-defined equivalent terms, including acronyms.
    • VARIATION: Connect words in a query to domain-specific related words.

 

Let’s think about a common problem of users trying to answer questions about how to file their taxes. A user may have the question “What’s the maximum earned income tax credit if I file with my husband?” A different user may phrase this as “How much is the EITC limmit if I file with my partner?”.

For either variation, Kyndi returns this sentence as the top result: “Earned Income Tax Credit: The maximum credit for filing jointly as a married couple and claiming three or more qualifying dependents amounts to $6,660 in 2021, with a phaseout for the credit beginning at $56,844 of adjusted gross income (AGI).

To return this result, Kyndi is performing complex word-form analysis and bringing to bear a number of the cognitive abilities above to connect the user’s query to the desired results. Let’s break down some of the cognitive qualities Kyndi used to make it resilient to changes in the user’s query.

For every query, Kyndi extends the question to look for any lemma, morphological, or lexeme variations that might be present in the corpus. For example, here Kyndi connected “file” from the query to “filing” in the result.

Additionally, Kyndi uses its proprietary Semantic Distance Field Model to find content that is semantically similar to a user’s query, such as connecting “limit” to “maximum” and understanding that the references to “husband” and “partner” indicate filing jointly as a “married couple”.

Even before forming these connections, Kyndi is able to resolve typos within the query. For example, for this query, Kyndi first resolves “limmit” to “limit” and then extending that to find the semantically similar concept “maximum”.

Finally, you may have noticed the user’s use of the acronym “EITC” in reference to “Earned Income Tax Credit”. For domain-specific terminology such as this, Kyndi offers the opportunity to extend Kyndi’s understanding of terms or phrases that are unique to your business or industry. Kyndi provides token-level explainability for each token in a query to identify what it is connected to and why. Your developers and administrators can use this information to debug missed connections and perform last-mile connections.

Contact Kyndi to discuss how to get started on a game-changing cognitive search solution, powered by AI, which works by understanding the user intent.