Artificial Intelligence is here and it’s changing every aspect of how business functions.

Now that machine learning is becoming more and more mainstream, some design patterns are starting to emerge. As the CEO of CrowdFlower, I’ve worked with many companies building machine learning algorithms and I’ve noticed a best practice in nearly every successful deployment of machine learning on tough business problems. That practice is called “human-in-the-loop” computing. Here’s how it works:

First, a machine learning model takes a first pass on the data, or every video, image or document that needs labeling. That model also assigns a confidence score, or how sure the algorithm is that it’s making the right judgment. If the confidence score is below a certain value, it sends the data to a human annotator to make a judgment. That new human judgment is used both for the business process and is fed back into the machine learning algorithm to make it smarter. In other words, when the machine isn’t sure what the answer is, it relies on a human, then adds that human judgment to its model…