Solved – How to determine if a price decrease is historically significant

econometricsmachine learningstandard deviationstatistical significancetime series

I am creating a program to collect and analyze prices for goods and alert the user when the price decreases so that they can buy it on "sale." I know a little statistics (e.g. standard deviation) but am looking for the best way to determine if a price drop is significant compared to historical prices – i.e. if I should bother alerting the user to the price decrease.

Let's say, for example, that my program is collecting prices for books. It searches the web, say, on a daily or weekly basis for prices for the book (both used and new), and stores this data in a database. I can make a graph of the prices over time (both lowest price, mean price, etc.) and visually determine when the price drops a significant amount – this is easy for a person to do. But what is the best statistical algorithm to determine this?

I can think of a few cases in which a person would buy a product when the price drops:

  1. If the price drops below a certain static barrier. For example, if the customer determines that they would buy the product at < 15, then whenever the price dips below that amount they should be alerted to that. This scenario is based on the idea that the customer believes that the price will actually eventually get that low, at some point in the future.
  2. If the price drops a significant amount below the current price. This is more realistic in some scenarios. For example, if there is a rare and relatively expensive book (for example, it was printed by an academic publisher with only a few thousand copies, all in hardcover, which sold for 150 when new and there are only a few copies on the market) – the book may be available (used) at the current time for an average price of 85. The price may never drop to 15, because the supply is so low. However, it may drop to a significantly lower price (40 or 50), but the customer doesn't have a specific price where they will buy the book.
  3. If the price of the book drops very low. A book may go today for 20, but the customer may want to wait and see if a bookseller is going to put it on sale for 5-8, because they don't need this book right now – but it's on their "wish list."

I am interested in this second and third scenario. Essentially I want to be able to determine when an item is available for a price that is "notable." I will store all of the historic prices in a database, and can compare this week's prices with historic prices.

What are your suggestions for statistical analytic tools which can be used to study this type of data and make automated recommendations?

Best Answer

Personally, for my wishlists with different companies, I'm interested in a percentage drop, particularly if it is large (e.g. 80% drop for software price). A couple of things spring to mind:

  1. I am interested in current pricing comparisons, for example if I get notified that a price has dropped 15% but competitors are selling it even cheaper, then I will buy from the competitor. So the analysis should take into account current pricing from competitors as well, particularly if this is easy to check using one of those online price comparison sites.
  2. A comparison with initial price is probably not the correct measure to use as items remain on the market over time. As items are on the market for longer, they tend to come down in price. How will you calculate the percentage change, especially if the customer has the item on their wishlist for a reasonable length of time? Will you update the baseline against which the price drop is measured?
  3. Some items are marked down dramatically just before a new edition is released (books, software). If that is the reason for the price decrease, you may wish to add some text to the customer notification that a new release is coming out.
  4. The pricing from suppliers can be incorrect, for example there has been discussion on some boards about ridiculously high pricing for books, such as $999 for an out-of-print but not particularly interesting technical book. Cost inputs like will interfere with your algorithm, so you'll need to run some type of outlier detection to exclude these values.
  5. You may wish to use different algorithms for rare versus common items. Getting market pricing for the former may be very difficult.

Update: if you have sales data, you may notice an upwards change in sales trend once price hits a certain level. This is another way you could do notifications, however I'm assuming you don't have sales information.

This could probably be answered by an econometrician, which I am not, as I think you're after some type of utility function.

Related Question