Sponsor detection

Architecture Overview

The system follows a multi-stage pipeline:

1. Text Processing & Chunking
2. Named Entity Recognition (NER)
3. Semantic Embedding Generation
4. Similarity Matching
5. Post-processing & Filtering

1. Text Processing & Chunking

The system first processes incoming text through intelligent chunking by configuring chunk size. This is to handle the long articles that exceed OpenAI's token limits.

2. Named Entity Recognition (NER)

This is the core extraction phase using OpenAI's GPT-4 model. We crafted a prompt that instructs the AI to extract organizations, extract people, handle variations with some examples, and output the json structured list.

3. Semantic Embedding Generation

The system uses OpenAI's text-embedding-3-large model to convert both extracted entities and sponsor names into high-dimensional vectors.

4. Similarity Matching

We use cosine similarity as the core matching algorithm, with a threshold of 0.94 determined through iterative testing. Cosine scores were computed for all possible entity–sponsor pairs using NumPy.

5. Post-processing and Filtering

We apply post-processing and filtering to handle edge cases that arise from the nondeterministic behavior of LLMs.

How did we decide this was the best approach?

Our approach was highly iterative and driven entirely by metrics. I used MLflow as the central stack for tracking metrics, which allowed us to systematically evaluate different methods with the goal of achieving over 90% in precision, accuracy, and recall. Here are some of the approaches we experimented with: (1) Extracting entities using alternative models such as spaCy, (2) Testing different n-gram combinations, (3) Applying fuzzy search with existing names, (4) Adjusting cosine similarity thresholds, and (5) Designing and refining LLM prompts. By leveraging MLflow for detailed metric tracking, we were able to compare approaches effectively and select the solution that best met our requirements. This iterative process proved invaluable in optimizing our results.

Performance Optimizations

1. Batch Embedding: Processes multiple entities simultaneously
2. Caching: Local sponsor data caching
3. Parallel Processing: Multi-threading for similarity calculations
4. Adaptive Chunking: Dynamic text segmentation based on API limits

API Integration

The system is deployed as an AWS Lambda function. I used AWS SAM and AWS cloudformation to manage the deployment. I also added github actions CI/CD pipelines to track the metrics while creating pull requests. The metrics are also sent to our mlflow server.

Monitoring

All the logs are configured to send to aws cloudwatch.

Note on a unique error

Modern tools -> modern bugs! When Facebook is mentioned as a social media, it will not recognize it as an organizational entity. Example, “We will serve until we run out,” the Inn posted in a Facebook invitation. Thus, the LLMs do not extract Facebook as an organizational entity, until we are explicit about social media platforms in our prompt.