Solved – How to build a recommendation system based on customer purchase data instead of standard ratings

data transformationrecommender-system

I have learned some basics about using collaborative filtering to build a recommendation system and would like to try it out on a dataset which are a large number of customer purchases in around a few hundred products.

However I have some challenges

the user-item value is not on the same scale like 1-5 of a movie rating so measuring similarity does not seem straight-forward.
some customers are much larger than the rest. Their total purchases amount is larger than average customer.
some products are much more popular than the rest. The total product purchases amount is larger than average product.

What are some recommended ways to deal with this type of data before applying the standard SVD and collaborative filtering algorithm?

Best Answer

Maybe create a score based on the number of items purchases, the amount purchases and normalized it.
For that, use the magnitude. Thus, a customer who had like 10 items is more valuable than one who have rated 90% of the items. To calculate the magnitude see an example on: Item-Item Collaborative Filtering
Collaborative Filtering methods do not propose the most popular items.

Related Solutions

Solved – should one always perform svd before doing KNN

The problem with directly using the sparse matrix or large dimensional to do the nearest neighbor computation is the curse of dimensionality. The curse of dimensionality is the general term for all the problems related to large dimensions. In your case, the distance function on high correlated (less information) will not give contrast to the distance.

Sparsity peeps in as well, which also makes the data not relevant to be used. SVD is helpful in reducing the matrix to a rank of the most relevance. If your data is N x M though it contains only rank r information, the dimensionality reduction using the SVD will give you mostly relevance features up to N x r. A similar effect is achieved using other dimensionality reduction techniques too such as PCA.

So, it's better for you to replace the zero with the mean of the data which will be non-extra information added by your pre-processing which anyhow will be removed by SVD. so that we do not lose any other relevant information on the same sample.

and then perform KNN (or other) to do an instance-based search of the related samples.

Solved – Designing a simple tag-based recommender system

I think there are 2 things to think about:

How to deal with implicit feedback.

There is a famous paper called "Collaborative Filtering for Implicit Feedback Datasets" [1] which can help you. You can just use this type of method on your "user X purchases" table and see what happens.

You have products but also tags of that products

You can transform your "user X products" matrix into a "user X tags" matrix. Then you can apply some function to add more weight to the "purchase" event. Then you mix (apply a weight and then just sum) with your original "user X tags" matrix. Imagine this scenario: users A and B, products X, Y and Z. User A purchased product X and product Z. Product X has tags "m" and "n", product B has tags "o" and "p" and product Z has tags "n" and "q". Your purchases user/products matrix is:

   X  Y  Z
A [1  0  1]
B [0  0  0]

This can be transformed into a purchase user/tags matrix:

   m  n  o  p  q
A [1  1  0  0  1]
B [0  0  0  0  0]

Instead of another binary matrix you can set values that represents a combination of possible interactions between a user and a product. All of that will be properly handled by a method prepared for implicit feedback.

I've already built a similar(in some sense) recommender system. Imagine travel packages and cities. The clients purchased travel packages but we ended up recommending cities and then applying some rules to find the best travel package to that city to that client. We could have recommended travel packages but this choice would gave us us a much more sparse and bigger matrix.

[1] http://yifanhu.net/PUB/cf.pdf

Best Answer

Related Solutions

Solved – should one always perform svd before doing KNN

Solved – Designing a simple tag-based recommender system

Related Question