I am looking for machine learning algorithms that can be used with panel data, and that are available in Python. Scikit does seem to contain anything relevant for panel data.
Solved – machine learning for panel data in Python
machine learningpanel datapythonscikit learn
Related Solutions
In my view, Python is a good choice for building the machine learning part (you don't say anything about the rest of your application, so I can't comment of that).
NumPy is powerful and mature, and has lots of numerical packages built on top of it.
For example, SciKits is a suite of such packages. It incorporates scikit-learn, which is
a Python module integrating classic machine learning algorithms in the tightly-knit scientific Python world (numpy, scipy, matplotlib). It aims to provide simple and efficient solutions to learning problems, accessible to everybody and reusable in various contexts: machine-learning as a versatile tool for science and engineering
With regards to performance, native NumPy operations are on par with their BLAS counterparts (they basically are wrappers around BLAS). Thus, NumPy code that can be expressed in terms of vector/matrix operations tends to be as fast as comparable C/Fortran code.
On the flip side, code expressed as Python loops can be slow. Additionally, it is hard to speed things up by using multiple threads. However, there are ways around both of these shortcomings: using multiprocessing instead of threading, numexpr, Cython and so on.
When you have panel data, there are a different tasks that you can try to solve, e.g. time series classification/regression or panel forecasting. And for each task, there are numerous approaches to solve it.
When you want to use machine learning methods to solve panel forecasting, there are a number of approaches:
Regarding your input data (X), treating units (e.g. countries, individuals, etc) as i.i.d. samples, you can
- bin the time series and treat each bin as a separate column, ignoring any temporal ordering, with equal bins for all units, the bin size could of course simply be the observed time series measurement, or you could upsample and aggregate into larger bins, then use standard machine learning algorithms for tabular data,
- or extract features from the time series for each unit, and use each extracted feature as a separate columns, again combined with standard tabular algorithms,
- or use specialised time series regression/classification algorithms depending on whether you observe continuous or categorical time series data, this includes support vector machines with special kernels that compare time series with time series.
Regarding your output data (y), if you want to forecast multiple time points in the future, you can
- fit an estimator for each step ahead that you want to forecast, always using the same input data,
- or fit a single estimator for the first step ahead and in prediction, roll the input data in time, using the first step predictions to append to the observed input data to make the second step predictions and so on.
All of the approaches above basically reduce the panel forecasting problem to a time series regression or tabular regression problem. Once your data is in the time series or tabular regression format, you can also append any time-invariant features for users.
Of course there are other options to solve the panel forecasting problem, like for example using classical forecasting methods like ARIMA adapted to panel data or deep learning methods that allow you to directly make sequence to sequence predictions.
Best Answer
If you are considering to apply machine learning to temporal (i.e. panel data) then I recommend to use a recurrent neural network (RNN) for the tasks at hand.
Python offers several excellent neural networks libraries, such as Caffe, Brainstorm and Theano.
Note that when applying neural networks it is of importance that you have sufficient data available. If this is not the case then I do not recommend machine learning techniques, but in stead would recommend ARMA based models