The difference is that in the case of wormholes, it is not certain that closed timelike curves (and thus violations of causality) actually occur, and if they occur, in which form. There are different approaches to the issue, for an interesting read (which also treats other kinds of spacetimes that potentially violate causality), see this. The general idea is that causality violations might be suppressed by quantum effects.
Ok, so let's say you had a wormhole. How long would it take to get from point A to point B using it? Let's say it's instantaneous. A traveller would arrive the same moment they left, spend some time at point B (it's really a nice place; the B-ian people are friendly and the food is great), then use the wormhole to go back to point A. No problem right? No violation of causality? Perhaps, but you need to ask yourself "when is right now at point B?" Consider this diagram:
This is a Minkowski diagram. The red axes represent the reference frame we're in and the green axes represent a reference frame at some high velocity relative to ours. So now ask yourself that question, when is right now? By the red frame, right now is the x-axis. The wormhole could take you to any point along it. But what if I enter the wormhole travelling fast enough to be in the green frame? In that frame, instantaneous travel is anything along the x'-axis. Notice that accordingly, that would put me into the red frame's future (we're just looking at the first quadrant). So you say "well that's simple, the wormhole isn't moving in my frame so it would make use of my definition of instantaneous". Here's the bigger problem. Now if I'm in the green frame and enter the wormhole, I travel along the x-axis and the point I end up is actually in my past (trace a line from somewhere on the x-axis back to the ct'-axis that is parallel to the x'-axis, it leads to the past).
How does this explain how I can send a message back to 2004? Say I have the wormhole, I enter it in the red frame (let's assume that's the Earth frame). Then I get to point B, accelerate to be in the green frame and go back through the wormhole to our point A at x=0. So let's run through this. I start at x=0, I used the wormhole to travel to some point on the x-axis, I speed up (so shift the green frame so that the green origin is on the x-axis at our chosen point), then I return through the wormhole to x=0 except remember I'm travelling along the x'-axis now. Voila, I'm in 2004.
But hold on, you say. Didn't I already establish that the wormhole uses the definition of instantaneous from its own frame; the red one? True, I did say that. But what if at point B I find another wormhole travelling in the green frame that links back to x=0? Or what if I found a way of speeding up the other end of the wormhole? Then I could certainly travel back to 2004.
The only way to prevent me from using a wormhole to travel to the past is to make all wormholes exist in the same reference frame and transport objects using that frame's definition of instantaneous. But that would mean there is a preferred reference frame in the universe. And that would be a matter for another question on this site.
Best Answer
It's true that traversable wormholes would theoretically allow for travel into the past according to general relativity (though this may be ruled out by quantum effects in whatever theory of quantum gravity eventually replaces general relativity, see the chronology protection conjecture). But the only way this would happen is if one mouth of the wormhole experiences time dilation relative to the other, either because it's taken on a journey at relativistic speed relative to the other mouth, or because it's moved closer to a source of gravity where it experiences gravitational time dilation. If the two mouths have clocks moving alongside them which were initially synchronized, then if some time dilation has accumulated, when the two mouths are brought back together an outside observer looking at them side-by-side will see the clocks showing different times. But, as explained by physicist Kip Thorne (who discovered traversable wormholes as a theoretical possibility in general relativity) in his book Black Holes and Time Warps, time threads differently through the wormhole, in such a way that if you look through one mouth at the clock which is alongside the other mouth, and compare it with the clock alongside the mouth you're looking through, then the clocks will still be synchronized (assuming the time for light to travel through the wormhole from the clock to your eyes is negligible). This means for example that if a side-by-side comparison by an outside observer (one who was not looking through either wormhole) showed mouth #1's clock to read 2015 and mouth #2's to read 2010, then if you jumped through mouth #2 you would exit mouth #1 when its clock also read 2010, not 2015. In some cases this would make it so that when you jump through a wormhole, you could end up in a region of spacetime where it would be possible to send a signal to your own younger self, which would be received at an earlier time then it was sent according to a clock you carried along with you (i.e. in terms of your own proper time).
But you don't mention anything about time dilation--if one mouth of the wormhole is moved slowly from A to B, so that no time difference accumulates, then there will be no time travel in your scenario. One thing to keep in mind is that in relativity, simultaneity is not generally defined in terms of when you see the light from events--for example, in an inertial frame in special relativity, if I see the light from an explosion 10 light-years away in the year 2020 (in the space and time coordinates of that frame), I subtract out the light travel time and say the event actually took place in 2010 in my frame. And wormhole spacetimes can be asymptotically flat, meaning to a good approximation you can treat them as localized distortions of spacetime moving around in an otherwise flat spacetime, so you can still set up something very close to an inertial frame in the region outside the wormholes. In this case, we might say that in a frame where A and B are at rest, the event of the clocks at A and B each reading 12:00:00 AM on Jan 1. 2020 are simultaneous. Then if I jump into mouth B at that moment, and exit from mouth A 4 seconds later according to the time coordinates of this frame, we can predict that I will see the clock at A reading 12:00:04 AM, Jan. 1 2020. If I immediately look back through the wormhole I will see the clock at B reading the same time of 12:00:04 AM, Jan. 1 2020, but if I immediately look at B through normal space using a telescope, due to the 10-light-year distance I will see the clock at B reading 12:00:04 AM, Jan 1. 2010. Assuming I had been living near B until I jumped through that mouth in 2020, looking through my telescope will also allow me to see myself when I was ten years younger.
But the main thing to realize is that there's no way I can actually send a signal to reach my younger self in the past in this scenario, as was possible in the other scenario where time dilation had created a time difference between clocks next to each mouth. If I send it through the wormhole, then (assuming a negligible travel time for a light signal) it exits mouth B 12:00:04 AM Jan. 1 2020, whereas I had jumped through mouth B 4 seconds earlier at 12:00:00 AM Jan. 1 2020. Meanwhile, if I send a light signal through normal space at 12:00:04 AM Jan 1 2020, in my frame it takes 10 years to get there, and another 10 years for the light from the event of someone receiving it at B to get back to me. This means I won't see anyone at B receive it (at least not if I am looking through normal space) until my clock reads 12:00:04 AM Jan 1 2040, and I will see the clock at B reading 12:00:04 AM Jan. 1 2030 when someone next to it receives my signal. So regardless of whether I send a signal through normal space or back through the wormhole, in this scenario the signal will arrive at B when the clock there shows a later time than the moment I jumped through the wormhole to travel from B to A.