By Ryan Welsh
In 2018 I was an applied math and quantitative expert that worked for a prestigious law firm in New York City. The Friday before Lehman Brothers filed for bankruptcy and kicked off the 2008 global financial meltdown, I was tasked with reading around three years’ worth of information in three days to understand our exposure to Lehman and the financial implications. Bleary-eyed, exhausted and under immense pressure, I wasn’t sure I was finding everything but had to press on, reading and reading and reading.
All I could think about afterwards was that the technology in the world today, there has to be a better way.
Two years later, I started Kyndi, an Artificial Intelligence (AI) company focused on natural language processing (NLP), knowledge representation and machine learning for text. We’re automating advanced reading tasks to make knowledge workers smarter, faster and more fulfilled.
The speed at which humans read hasn’t changed much over the last hundred years, but the requirements to read more, faster continues to multiply exponentially in almost every industry. Modern knowledge workers consume tsunamis of text in emails, documents, applications, web pages and more to analyze, understand and make good decisions. Sci-fi movies have long fantasized about a future where humans could read and analyze at hyper-accelerated rates or download knowledge directly into their brain. While the later may still be fantasy, the former is here as reading automation powered by text AI takes flight.
Artificial intelligence (AI) and related technologies have made huge strides already in helping people quickly consume and find insights in STRUCTURED data (information in tables and databases). But UNSTRUCTURED text (natural language sentences and paragraphs) has been a much more complicated puzzle. Yet 80% of the data in most organizations is unstructured. And as Digital Transformation projects accelerate, the amount of accessible text continues to grow exponentially. Hiring more and more humans to read it probably isn’t viable. There is a massive problem to be solved.
Today most people rely on search engines to help them find a term or document relevant to their idea, interest or problem. But search engines are notoriously limited in their ability to provide relevant results and pointers to the right information. In large part this is because they can’t infer interrelated words and meanings, reveal patterns in the text or understand the contextual nature of the search. Search engines offer limited capability for people to refine their results to continue narrowing in. They also are limited in their ability to explain their prioritization and point to exact areas of text — leaving many researchers to sort through endless search results to read, analyze and find the answers they seek.
There are four core functions we believe Reading Automation will fully address over the next few years and around which we already offer or intend to build products:
– Analyze & Discover – People want to quickly scan thousands of documents to find items like people, places, organizations or any “named entity” that a user defines. They want to be informed if closely related terms they didn’t search for exactly may also be helpful in navigating the text. This step of processing and prioritizing content is the core of reading automation. Today analysts on some topics may have to read through hundreds of documents just to understand which are most important, have the relevant material and should be reviewed further.
– Categorize, Visualizate & Iterate – People want the ability to see interrelated topics, patterns and trends within text so they can refine and exclude to get closer to the meaning they seek. Kyndi has approached this by representing the terms visually on a knowledge graph that helps people see interrelated concepts, exclude or include what’s important and refine results.
– Review What’s Relevant – While reading automation can process, analyze, categorize and answer many questions, people are still going to want to see into the documents to understand or validate the results and potentially to read further to add additional human insight. Reading automation solutions will need to expose the prioritized documents and text in an easily consumable way to help users see what’s important for review. Kyndi has received much recognition for this aspect of our reading automation in a world where AI is mostly a “black box” and not explainable.
– Extract Key Information – We’re not here yet, but we know that once people have identified a named entity, they might want the system to quickly read, locate and extract that data across many documents into a table (making unstructured data structured!). For instance, a loan manager may want to quickly review interest and principal payments in many 300 page documents across their portfolio to create a table of different clients and their rates.
Reading automation AI, as a key aspect of the hyperautomation trend, promises to help readers find actionable insights and prioritize where they look to interpret, re-read and defend those insights with unprecedented speed and accuracy thanks to advances in text AI.
Because we expect that reading automation will be a function people want to plug in across different interfaces, we’ve built Kyndi’s Reading Automation Platform as a set of engines that can be quickly and easily added into internal or commercial applications where documents may live, where people are doing their work and where processes are in flow. We empower companies to imbed the most advanced reading automation into their applications without needing a team of expensive, scarce AI or NLP PhDs.
We’ve also created reading automation to map to the timelines in which people need to read vs the 6-9 month timelines that many AI and machine learning solutions require to ‘learn’ the data. Kyndi doesn’t require manual labeling and the system can learn from as few as 10 documents. This gives companies freedom to ask strategic and emerging questions even from smaller text data sets. We expect all Reading Automation vendors will have to follow suit as reading automation that takes more than a day or two will be mostly irrelevant to the researcher.
We have already seen great traction across the government, pharmaceutical and manufacturing spaces where analysts and researchers are deep in projects like identifying terrorists, preparing for FDA approval or addressing quality control problems. Knowledge workers in these industries dig through thousands of deep, long and complex documents to find and review the information they need. With Kyndi, they are able to quickly and easily find, interrelate, visualize names, places, facts and figures and read up on hyper-relevant documents and passages. In many instances, one analyst can now analyze troves of documents that would have previously required many different people do the same work.
Reading Automation is in the first inning of a much longer game. The idea of automating many steps of reading with AI is completely new for most people. They don’t even know they have a problem for which they could buy a solution because they’ve been reading just fine most of their adult lives.
But a great wave of change is coming. Reading automation could be as revolutionary as the invention of the printing press. It’s hard to imagine today that all books were written by hand, by humans, one page at one time. It used to take many YEARS for a monk to transcribe just one copy of the bible. One day we’ll look back in amazement at people who had to read every document themselves, the whole way through to do their work. A new standard of productivity is coming, and the first reading automation solutions are already here.