Solved – Difference between multitask learning and transfer learning

Question

I am reading Caruana (1997) Multitask learning (pdf). In the definition of multi task learning, the author states that;

Usually, we do not care how well extra tasks are learned; their sole purpose is to help the main task be learned better.

This is also the goal of transfer learning right? What is the difference then? Are these two terms used interchangeably (I think not)?

Answer 1

Single Task Learning: Giving a set of learning tasks, t1 , t2 , …, t(n), learn each task independently. This is the most commonly used machine learning paradigm in practice.

Multitask Learning: Giving a set of learning tasks, t1 , t2 , …, t(n), co-learn all tasks simultaneously. In other words, the learner optimizes the learning/performance across all of the n tasks through some shared knowledge. This may also be called batch multitask learning. Online multitask learning is more like lifelong learning (see below).

Transfer Learning (or Domain Adaptation): Giving a set of source domains/tasks t1, t2, …, t(n-1) and the target domain/task t(n), the goal is to learn well for t(n) by transferring some shared knowledge from t1, t2, …, t(n-1) to t(n). Although this definition is quite general, almost the entire literature on transfer learning is about supervised transfer learning and the number of source domains is only one (i.e., n=2). It also assumes that there are labeled training data for the source domain and few or no labeled examples in the target domain/task, but there are a large amount of unlabeled data in t(n). Note that the goal of transfer learning is to learn well only for the target task. Learning of the source task(s) is irrelevant.

Lifelong Learning: The learner has performed learning on a sequence of tasks, from t1 to t(n-1). When faced with the nth task, it uses the relevant knowledge gained in the past n-1 tasks to help learning for the nth task. Based on this definition, lifelong learning is similar to the general transfer learning involving multiple source domains or tasks. However, some researchers have a narrower definition of lifelong learning. They regard it as the learning process that aims to learn well on the future task t(n) without seeing any future task data so far. This means that the system should generate some prior knowledge from the past observed tasks to help new/future task learning without observing any information from the future task t(n). The future task learning simply uses the knowledge. This definition makes lifelong learning different from both transfer learning and multitask learning. It is different from transfer learning because transfer learning identifies prior knowledge using the target/future task labeled and unlabeled data. It is different from multitask learning because lifelong learning does not jointly optimize the learning of the other tasks, which multitask learning does.

All content lifted from https://www.cs.uic.edu/~liub/IJCAI15-tutorial.html.