Hack Price Monitoring with IFTTT and ScraperWiki

At WisePricer, we’re always trying to develop new ways to help online merchants stay ahead of their competition. Today, I want to share a way to keep track of Amazon’s prices for multiple products. I’ll be mashing up ScraperWiki and IFTTT to show you how to setup a daily “Price Monitoring” email delivered straight to your inbox.

In addition to this tactic, the combination of these two tools offers you endless possibilities for price monitoring. Later, I will share some of these ideas which you can implement yourself (and share with others in the comments section below!)

The Players
ScraperWiki is a platform that allows you to make data do things. It is a platform for writing and scheduling screen scrapers, and for storing the data they generate. ScraperWiki is useful for both for programmers who want to write screen scrapers with less fuss, and for journalists, activists and the general public who want to discover and reuse interesting data.

IFTTT is  a service that lets you create powerful connections with one simple statement:

If “this,” then “that.”

Using this principle, you can construct your own IFTTT Recipe using the following basic principles and ‘ingredients’:

“Channels” are the basic building blocks of IFTTT and include Twitter, Facebook, Email, Weather, etc. The “this” part is the trigger for the recipe. For example,  “I post a tweet”, or “I’m tagged in a photo on Facebook”. The “that” part of the recipe is the action element, including examples like “send me a text message or “update my Facebook status”.

A full recipe looks like this:

Following this pattern requires some basic coding skills. I believe it is essential for everyone to learn the basics or at least try. It’s a useful skill.

If Bloomberg is willing to give it a try, you certainly have no excuse…

Step 1: Setup
To start, create your own ScraperWiki account https://scraperwiki.com/login/#signup, then copy my code https://scraperwiki.com/scrapers/competitorsmonitor/ into your account.

You’ll be able to edit that code (i.e. changing the list of Amazon ASINs) and run the scraper for the first time.

After running the scraper (be patient, it can take a few moments), go back to the scraper dashboard. Under ‘Schedule’ click the Edit button to set up a daily schedule, and then click “Explore with API”.

Step 2: Edit Schedule

Step 3: Configure Settings
Set the feed type as “rss2” and change the SQL query to “select * from `swdata` order by pubDate desc limit 10”, and click Copy. The link you have acquired can be used in any RSS feed reader, or as a trigger in IFTTT.

That’s it! You’re done with ScraperWiki and can move on to IFTTT. If you don’t already have an IFTTT account, go ahead and open one. You’ll be amazed with what they’ve done with their UX experience. Once you’re setup with an account, create a recipe that will get your feed updates sent to your email account (or Facebook, Twitter, etc). Try it for yourself.

Here’s a screenshot of the daily email taken from my Gmail account:

Don’t be afraid to experiment! Here are some more ideas for you to try my “Scrape ‘n Notify” technique:

  • Find out if any of your competitors are having a promotion\sale by scraping their website landing page.
  • Set an email alert for current Amazon best sellers in categories within your industry.
  • Check if your main competitor is out of stock, to seize retail opportunities.

Are you ready to scale up?
Get a free trial account with WisePricer to monitor your competitors on thousands of products, and seamlessly sync from your store. It’s a cinch.

About Wiser

Wiser provides a complete suite of solutions to give retailers, brands, and manufacturers the edge to stay both competitive and most importantly, profitable.

Wiser’s core product is WisePricer, full-featured dynamic pricing and merchandising engine that monitors, analyzes and reprices retail products in real-time. WisePricer enables retailers to boost profit margins and revenue, price with confidence, and improve merchandising through powering the development of a sound pricing strategy.

Ready to try it out for yourself? Schedule a Demo or Sign up for a 14-day Trial online – No obligation. Learn how to win the pricing battle.

Get your free copy of our success kit: The New Rules: What the Winners of Online Retail Already Know.


Roey Brecher

  • http://lastdropofink.co.uk/ Matthew Ogborne

    Howdy Roey, Arie,

    How sweet is that guide!

    If I continue to read the article, I’ll end up loosing a week to coding apps to automate different tasks :)


  • http://lastdropofink.co.uk/ Matthew Ogborne


    I’ve only read the first 3 paragraphs and got distracted with ScraperWikia and IFTTT, you know they have a link up to Gmail so that you can script emails? The one thing that is the major diference between exchange/outlook and gmail? https://ifttt.com/gmail

    Sick! *That’s my term for cool)

    By the way, how did you get on with extracting the bets sellers from Amazon? Also keeping track of the latest eBay deals might be worth doing too.


    • http://www.wisepricer.com/ Roey Brecher

      Howdy Matt!
      IFTTT has many other channels and the combinations between them will get you many cool recipes.
      Amazon has a “Best Sellers” list for every category, for example this list http://amzn.to/Sjsjbw.


  • http://motyar.blogspot.com/ Motyar

    Nice post Roey, Was using IFTTT at its fullest, Was writing my own php scraper (to fetch Tumblr liked post img url etc ). Thanks for letting us know about
    ScraperWiki. ll start using it soon.

    So we don’t need our own servers, do we? Anyone who know basics of programming can create amazing applications using these services. This is real “Hacking web work for you”.

    – Motyar

    • http://www.wisepricer.com/ Roey Brecher

      Hi Motar, glad you enjoyed it.
      I’m not sure of how ScraperWiki will scale up for scraping many pages on a daily basis. But it sure is a useful tool for hacking small scraping tasks, and when combined with IFTTT it’s just pure awesomeness.

  • Pingback: Semantic Analysis | Pearltrees()