First, mind that deep learning is a buzz term. There is not even a consensus of a formal definition in the research community. A discussion of the term does not lead anywhere, really. It's just a word.
That being said, convolutional nets are deep because they rely on multiple layers of feature extraction, as you said. They extract features from the input to predict an outcome.
What you refer to is a "generative" approach, i.e. the features are used to create the observation (a picture, not a class label). That is what made deep learning popular, but it is in no way limited to that.
In general, binary data types are used to represent membership in particular categories. Suppose you have some data in a row-column format, so that the columns are features and rows are observations. Perhaps you're interested in the relationship between height and gender. So your data look like
Height Gender
6 ft Male
5 ft 4 in Female
And so on. Clearly, there's not a great numerical representation of "male" and "female." But another way to look at the problem is as a question of membership, to answer the question "is this person male?" or, equivalently, "is this person female?" In this way, we can take the categories "male" and "female" and translate them into binary, numerical quantities (conventionally, $1$ and $0$). Whichever we choose to treat as 1 or 0 is irrelevant from a mathematical standpoint. Importantly, it's standard practice to expand a variable with $k$ categories into $k-1$ binary columns. This is because you don't want to make your columns linearly dependent with an intercept column of all $1$s, and make it impossible to uniquely estimate these quantities. I don't know what the particular details of your so-called "deep learning" regression are, but it probably has something like an intercept.
You've asked @Sheep
Do you know the underlying process for these kind of variables?
But this is a fundamentally unanswerable question without actually doing the analysis. Run the regression and find out!
Does it make sense to estimate a variable( my target variable) based on categorical variables?
Maybe. If the two quantities are completely unrelated -- for example, political party in power in a given territory and number of hurricanes over the Atlantic ocean -- then it probably wouldn't make sense. Instead of looking at your research as a purely rote, quantitative exercise, I would encourage you to think critically about what your data represent, and what the underlying causal mechanism is. Is there a biological process at work? Are people acting according to their self-interest? What do we know about the climate that casues hurricanes?
Best Answer
For standard regression I recommend using Multi-Layer Perceptron (MLP) with Mean Square Error (MSE) loss function. The model can be defined in Keras as follows:
See this article for a complete example. For a TensorFlow implementation, consider the following code.