Getting Real Data: Scraping Shufersal Products

Real Prices or Nothing

From the beginning, I was committed to using real supermarket prices. Made-up numbers would have been easier, but the entire appeal of PriceGuess is that you're testing your actual knowledge of Israeli grocery prices. That meant I needed a reliable way to get product data from Shufersal, Israel's largest supermarket chain.

The CRON Worker

Cloudflare Workers support scheduled triggers via CRON expressions. I set up a daily scrape worker that runs at midnight Israel time. The worker fetches product data from Shufersal's online catalog, processes it, and stores it in D1:

// workers/daily-scrape.ts
export default {
  async scheduled(event: ScheduledEvent, env: Env) {
    const products = await fetchShufersal(env);

    for (const product of products) {
      await env.DB.prepare(`
        INSERT OR REPLACE INTO products
        (id, name, price, image_url, category, updated_at)
        VALUES (?, ?, ?, ?, ?, datetime('now'))
      `).bind(
        product.id,
        product.name,
        product.price,
        product.imageUrl,
        product.category
      ).run();
    }

    // Select tomorrow's challenge
    await selectDailyChallenge(env.DB);
  }
};

Storing in D1

Cloudflare D1 is essentially SQLite at the edge. The schema is simple: a products table, a challenges table that maps dates to products, and a results table for player scores. D1's SQL interface made queries straightforward, and the free tier gives 5GB of storage, which is far more than I need for product data.

The products table holds around 500 items at any given time. Each product has a name (in Hebrew), a price in shekels, an image URL, and a category. I also track when each product was last updated, so I can filter out stale data.

Selecting Daily Challenges

The challenge selection algorithm was more nuanced than I initially expected. I needed to ensure variety: no repeats within 30 days, a mix of categories (dairy, produce, snacks, household), and a range of price points from cheap items under 10 shekels to expensive ones over 80. The selection function weights these factors:

async function selectDailyChallenge(db: D1Database) {
  // Get products not used in last 30 days
  const candidates = await db.prepare(`
    SELECT p.* FROM products p
    WHERE p.id NOT IN (
      SELECT product_id FROM challenges
      WHERE date > date('now', '-30 days')
    )
    AND p.price > 5
    AND p.image_url IS NOT NULL
    ORDER BY RANDOM()
    LIMIT 10
  `).all();

  // Pick one with category diversity in mind
  const recent = await getRecentCategories(db);
  const selected = candidates.results.find(
    p => !recent.includes(p.category)
  ) || candidates.results[0];

  await db.prepare(`
    INSERT INTO challenges (date, product_id)
    VALUES (date('now', '+1 day'), ?)
  `).bind(selected.id).run();
}

Handling Edge Cases

Products disappear from stores. Prices change between scrapes. Images get removed. I had to handle all of these gracefully. If a product is removed before its challenge date, the system has a fallback queue of pre-selected alternatives. If a price changes between when the challenge was scheduled and when it's played, I use the price from the moment the challenge was created, not the current price. This prevents a situation where the "correct" answer shifts mid-day.

Ethical Scraping

I want to be transparent about the scraping aspect. I respect rate limits, I only scrape once daily, and I only collect publicly available product information (names, prices, images that are shown on the public website). I'm not bypassing any authentication or accessing private data. The data I use is the same information any customer sees when browsing Shufersal's website.

I also keep the scraping footprint minimal. One request batch per day, during off-peak hours. It's less load on their servers than a single person browsing the website for five minutes.

The Midnight Rotation

Every day at midnight Israel time (UTC+2), the challenge rotates. The new product is already selected and cached. When a player opens the app, they get the day's challenge instantly from D1 without any computation. This pre-selection approach keeps response times fast and ensures every player sees the same product at the same time.

With the data pipeline running smoothly, I had a working daily game. But I wasn't done. The next big feature was the one I was most excited about: real-time multiplayer.