It depends on whether the force field is conservative or not.
Example of a conservative force is gravity. Lifting, then lowering an object against gravity results in zero net work against gravity.
Friction is non-conservative: the force is always in the direction opposite to the motion. Moving 10 m one way, you do work. Moving back 10 m, you do more work.
As @lemon pointed out in a comment, this is expressed by writing the work done as the integral:
$$W = \int \vec F \cdot d\vec{x}$$
When $F$ is only a function of position and $\vec \nabla\times \vec F = 0$, this integral is independent of the path and depends only on the end points; but if it is a function of direction of motion, you can no longer do the integral without taking the path into account.
From a mathematical point of view (ignoring integrals for now), we know that the work is defined to be
$$W = \vec{F}\cdot \vec{x}$$
By definition, the dot product of two vectors is a scalar. So that should be enough to convince you mathematically.
From a more intuitive point of view, remember that scalars can be negative or positive - this alone does not mean they are defining a direction. As you stated (correctly), the work does depend on the direction of the force. But this does not mean it is a vector itself (just look at the dot product above to convince yourself). Try to think of the force and displacement as more of a cause and effect type of a relationship though. You seem to be implying in your question that these entities are completely separate. If you push on an object in a certain direction, it is going to accelerate in that direction, unless some frictional force balances your push, so the net force is actually 0...then it will not accelerate at all.
If the displacement is perpendicular to the direction of the force then work is zero.
If there is zero work, then will be no displacement (assuming 0 initial velocity). This is not because there is inherently no displacement...it is rather that there is no net force. Again, imagine pushing on a block downwards, so it does not move at all. This is because the normal force is pushing back at you, so there is no net force on the object at all. That is in fact why it does not move.
So depending on what direction you push on an object (and subsequently accelerates), you can define work to be positive or negative. This depends on your choice of coordinate system. But to reiterate - work will only be positive or negative. It will never have a direction associated with it.
Best Answer
Work is the dot product of a vector force and a vector displacement, hence a scalar.
Knowing just the scalar distance isn’t enough to calculate work. That distance might be in the same direction as the force, but it might be perpendicular or even opposed. All of those would give different values for the work done.