Entity Resolution at Scale: Matching Products Across Amazon, Reddit, and RTINGS

"AirPods Pro 2," "Apple AirPods Pro (2nd Generation)," "AirPods Pro USB-C" u2014 same product, three different names.

Entity resolution u2014 figuring out that different strings refer to the same real-world thing u2014 is one of the hardest problems in product data engineering. At SmartReview, we match products across 50+ review sources, each with its own naming conventions, categorization, and data formats.

Here's how we solved it without spending six months building a custom ML model.

The Problem Space#

Consider matching products across these sources:

All five refer to the same product. A naive string match would treat them as five different products.

Our Three-Layer Approach#

We use three complementary techniques, each catching matches the others miss.

Layer 1: Brand + Model Normalization#

The first pass normalizes brand names and extracts model identifiers:

interface NormalizedProduct {
  brand: string;           // "apple"
  modelFamily: string;     // "airpods pro"
  generation: string;      // "2nd gen"
  variant: string;         // "usb-c"
  rawName: string;         // original string
}

function normalizeProductName(raw: string): NormalizedProduct {
  let name = raw.toLowerCase().trim();
  
  // Remove common noise
  name = name
    .replace(/\b(new|latest|best|review|vs\.?)\b/g, "")
    .replace(/[-u2013u2014]/g, " ")
    .replace(/\s+/g, " ")
    .trim();

  // Extract brand (lookup against known brand list)
  const brand = extractBrand(name);
  
  // Extract generation markers
  const genPatterns = [
    /(?:gen(?:eration)?\s*)(\d+)/i,
    /(\d+)(?:st|nd|rd|th)\s*gen/i,
    /\b(\d+)\b(?=\s|$)/,  // trailing number often = generation
  ];
  const generation = extractPattern(name, genPatterns);

  // Extract variant (color, connectivity, size)
  const variant = extractVariant(name);

  // What remains is the model family
  const modelFamily = extractModelFamily(name, brand, generation, variant);

  return { brand, modelFamily, generation, variant, rawName: raw };
}

This handles ~60% of matches u2014 the straightforward cases where brand + model + generation aligns.

Layer 2: Fuzzy String Matching#

For the remaining 40%, we use Levenshtein distance with a category-aware threshold:

function fuzzyMatch(
  a: NormalizedProduct,
  b: NormalizedProduct,
  threshold: number = 0.85
): boolean {
  // Brand must match exactly (after normalization)
  if (a.brand !== b.brand) return false;

  // Compare model family with fuzzy matching
  const similarity = stringSimilarity(
    a.modelFamily,
    b.modelFamily
  );

  if (similarity < threshold) return false;

  // If generations are both present, they must match
  if (a.generation && b.generation && a.generation !== b.generation) {
    return false;
  }

  return true;
}

The key insight: brand must match exactly, but model name can be fuzzy. This prevents false positives like matching "Sony WH-1000XM5" with "Sony WF-1000XM5" (over-ear vs in-ear u2014 completely different products with similar names).

Layer 3: Cross-Reference Validation#

For edge cases, we validate matches against external canonical sources:

async function crossReferenceValidate(
  candidates: NormalizedProduct[]
): Promise<ProductCluster[]> {
  const clusters: ProductCluster[] = [];

  for (const candidate of candidates) {
    // Search for the product on a canonical source
    const canonicalResults = await tavily.search(
      `${candidate.brand} ${candidate.modelFamily} specifications`,
      { searchDepth: "basic", maxResults: 3 }
    );

    // Extract canonical product identifier
    const canonicalId = extractCanonicalId(canonicalResults);

    // Group by canonical ID
    const existing = clusters.find(c => c.canonicalId === canonicalId);
    if (existing) {
      existing.members.push(candidate);
    } else {
      clusters.push({
        canonicalId,
        canonicalName: candidate.rawName,
        members: [candidate],
      });
    }
  }

  return clusters;
}

Handling the Hard Cases#

Product Lines vs Individual Products#

"Roomba" could mean the brand, the product line, or a specific model (Roomba j7+). We use context clues:

• If a review discusses specific features ("self-emptying base"), it's likely a specific model
• If it's a general comparison ("Roomba vs Roborock"), it's the product line
• We maintain a hierarchy: Brand u2192 Line u2192 Model u2192 Variant

Regional Name Differences#

The same product sometimes has different names in different markets. The Samsung Galaxy S24 is called "Galaxy S24" everywhere, but some accessories have region-specific names. We maintain an alias table for known cases.

Discontinued vs Current Models#

When someone searches "AirPods Pro vs Sony," do they mean the current or previous generation? We default to current unless the query specifies otherwise, but we keep both generations in our database with clear generation markers.

Performance at Scale#

Our entity resolution pipeline processes ~5,000 product mentions daily across all review sources:

The 1.8% false positive rate is acceptable because our trust score system (covered in Part 5) catches anomalies u2014 if a "product" suddenly has wildly inconsistent ratings, it's likely a merge error.

Lessons Learned#

1. Don't build ML first. Our three-layer heuristic approach handles 94%+ of cases. ML would marginally improve accuracy but massively increase complexity.
2. Brand matching must be exact. Fuzzy brand matching creates catastrophic false positives.
3. Generation numbers are treacherous. "AirPods 3" and "AirPods Pro 3" are different products. Always match model family before generation.
4. Maintain a manual override table. Some matches are just weird. "Galaxy Buds2 Pro" vs "Galaxy Buds 2 Pro" (note the space) u2014 keep a list of known aliases.
5. Log everything. When a match seems wrong in production, you need the matching pipeline's decision trail to diagnose why.

What's Next#

We're exploring embedding-based matching for the long tail u2014 products where our heuristics fail because names are too dissimilar. Early experiments with product description embeddings show promise for matching across languages.

See entity resolution in action on aversusb.net u2014 every comparison page unifies data from multiple sources under a single canonical product identity.

---

Part 8 of our "Building SmartReview" series. Previous: Part 7: People Also Ask Content Discovery