How AI Relevance Scoring Works: Filtering Signal from Noise

The Problem: Alert Fatigue

If you set up a tracker on a competitor's blog, you want to know when they announce a new product. You do not want to receive an email when they fix a typo, update a copyright year, or change the alt-text on a logo. In legacy systems, this generated massive "Alert Fatigue," causing teams to eventually mute or ignore the tool entirely.

The solution is AI Relevance Scoring. Here is how it operates under the hood at Watchflare.

1. The Semantic Content Hash

Before AI is invoked, computational efficiency is paramount. Watchflare strips away the HTML styling (classes, scripts, dynamic IDs) focusing purely on the markdown content. It calculates a SHA-256 hash. If the hash hasn't changed, the pipeline stops (saving compute costs). If it has changed, we proceed to AI.

2. The Intelligence Context Injector

The extracted text is bundled with the user's custom "Intelligence Context." This is the user's master directive, for instance:

"Only alert me regarding C-level executive departures, funding rounds, or strategic geographic expansions."

3. The Multi-Model Evaluation Chain

The payload is sent to an advanced LLM (e.g., Qwen 3.6 or Gemma 3 via OpenRouter). The model is instructed to act as a harsh critic. It evaluates the new content against the user's prompt and outputs a JSON object containing a score between 0 and 100, and a summary.

4. The Fallback Mechanism

AI APIs occasionally fail or return malformed JSON. Watchflare utilizes an atomic fallback chain. If Qwen fails to respond correctly within the specified timeout, the request instantly cascades to Llama 4 Maverick. This ensures the scoring pipeline is never brittle.

5. Threshold Execution

Once the integer score is returned, it hits the notification router. If the user set their alert threshold to >80, and the AI returned a 12 (because it was just a typo fix), the finding is silently recorded in the database, but no emails or webhooks are dispatched.

Conclusion

AI Relevance Scoring is the differentiator between a "scraper" and an "intelligence platform." It replaces boolean logic with semantic reasoning, ensuring your team only spends time reviewing data that actually impacts your bottom line.