I founded and lead the centralized Evaluations org of research scientists and engineers that is responsible for evaluation strategy, development, tooling, and execution for all GenAI foundation models (e.g. Llama). Our work spans all modalities (text, image, audio, video, etc.), capabilities (safety, reasoning, intelligence, agentic behaviors, etc.), and evaluation types (automated, AI-assisted, human, etc.).

Building Snorkel Flow, the AI data development platform for the enterprise.
For more details on the company, visit snorkel.ai.
For more details on my role at the company, visit my LinkedIn profile.

Snorkel is a system for rapidly creating, modeling, and managing training data. It is the flagship implementation of the data programming paradigm for supporting weak supervision resources. Collaborators and active users include over a forty major technical and medical organizations (e.g., Google, Microsoft, Intel, Toshiba, JPL, Alibaba, Stanford Medicine, etc.).

We extend Snorkel to multi-tasking supervision and multi-task learning (MTL). In particular, we are interested in the massive multi-task learning regime where we have large numbers of tasks and labels of varying types, granularities, and label accuracies. Using Snorkel MeTaL, we achieved new state-of-the-art scores on the GLUE Benchmark and four of its component tasks.

Driven by accuracy improvements and deployment advantages, many organizations have begun to shift toward learning-centered software stacks. That is, "Software 1.0" code with explicit instructions written by programmers is being replaced by "Software 2.0" code that is written in the weights of neural networks. In this paradigm, training data becomes the primary interface through which developers interact with their Software 2.0 systems. This requires a new level of scalability, control, and efficiency when it comes to generating and shaping training sets. We are exploring the ramifications of this new programming model and building the tools to support it.

Most of the conversations a chatbot sees over its lifetime happen after it's deployed. These conversations aren't typically useable as training data, but give the chatbot the right tools and it can learn from those too! We introduced a multi-task "self-feeding" chatbot that knows how to extract new training examples from the conversations it participates in to improve itself further after it's deployed.

We introduce a framework for generating titles for tables that are displayed out of their original context. We use a pointer-generator network, a sequence-to-sequence model that is capable of both generating tokens and copying tokens from the input (such as rare and out-of-vocab words), resulting in titles that are both relevant and readable and reducing hallucinations.

We explore collecting natural language explanations for why annotators give the labels they do and parsing these into executable functions, which can then be used to generate noisy labels for large amounts of unlabeled data. The resulting probabalistically labeled training dataset can then be used to train a powerful downstream discriminative model for the task at hand. We find that utilizing these natural language explanations allows real-world users to train classifiers with comparable F1 scores up to 100 times faster than when they provide just labels.

We introduce an information extraction framework that utilizes multiple representations of the data (structural, tabular, visual, and textual) to achieve state-of-the-art performance in four real-world extraction taks. Our framework is currently in use commercially at Alibaba and with law enforcement agencies fighting online human trafficking.

Using the multi-modal parsing and extraction tools from Fonduer and learning and inference tools from Snorkel, we construct a knowledge base of genotype/phenotype associations extracted from the text and tables in ~600 open-access papers from PubMed Central. Our system expands existing manually curated databases by approximately 20% with 92% precision.

We use topic models to correlate social media messages with survey outcomes and to provide an interpretable representation of the data. Rather than rely on fully unsupervised topic models, we use existing aggregated survey data to inform the inferred topics, a class of topic model supervision referred to as collective supervision.

With an internal committee of 20 MIT and DoD researchers, I spearheaded the construction of this report, which formalizes the components and complexities of recommender systems and surveys their existing and potential uses in the Department of Defense and U.S. Intelligence community.