It may not suit your goals, but one approach is to enroll in a masters program before entering a doctoral program. This could help you get back into the groove of academic life,
and also give you a chance to meet new professors who could write letters for your application to a more high-powered doctoral program. (I once advised a student who had spent quite a long time, maybe 8 years, in the software industry before returning to academia, and
this is the route she took. I think it served her well; because of the masters, which involved a mixture of coursework and a small thesis, she was very solidly prepared for her doctoral work, and was one of the strongest students in her cohort.)
Nobody seems to have mentioned much about teaching--- perhaps because the original question itself makes no mention of teaching having anything to do with the desire to return to academia. This is a kind of elephant in the room.
I should admit: I'm on the academic side, I have not personally tried to make this kind of transition, and I have never been in a position to evaluate somebody making this kind of transition. But it seems to me that if you're reasonably current with your research area, and publishing papers, and meeting people (as suggested elsewhere), your biggest obstacle may be teaching.
Presumably you have no teaching experience over the last n years, and depending on your grad school experience, you may not have had much then (or it may have been a different sort from what professors do). This may matter. I don't know how to begin building a teaching history, while working a full-time job.
You may need to overcome the suspicion that will find teaching low-level service courses boring for the same reasons you find your current job in industry boring. Imagine the skeptic on the search committee who asks, rhetorically, "Who wouldn't be an academic if it were all just learning, writing papers, and talking to enthusiastic people with the same interests?"
Even with stellar references and a personal connection or three in the department, someone will ask: can you teach? Do you want to? What's the answer, and how do you convey it on your CV?
I don't have specific advice in this area, because it depends on where you want to work, and your own background. If it is possible to do pedagogical things in your current job, or service/outreach to non-specialists or students, perhaps that would help. Maybe actual teaching (on a per-course basis, not as tenure-track faculty) or volunteering would help. My feeling is that you need to do something to address these issues head-on, to confront both any genuine gaps in your CV, and the biases and prejudices you may face simply because you are changing careers.
Best Answer
[This is certainly a biased view, but too long for a comment, and hopefully these are some helpful starting points. ]
I'd say it's active but in the US not huge (more elsewhere in the world). A few things to look up:
Computational number theory. Of computer algebra/symbolic computation, this may be the area most active in the US (the other areas have some activity in the US, more elsewhere in the world).
Computational group theory (in the US, a little bit myself, moreso James Wilson, and Alexander Hulpke come to mind, all in sunny Colorado ;), Peter Brooksbank). Dmytro Savchuk @ USF maintains the GAP package for working with automata groups. More generally I'd say look at contributors to GAP, MAGMA, macaulay2, and similar software packages and see where people are (again, most aren't in the US, but some are!)
Computational commutative algebra / Gröbner bases (Hal Schenck in the US comes to mind).
The journals suggested in the comments by Peter Taylor are great, but in CS lots of publications happen in conferences (if you're not used to "publication in a conference" or that sounds like an oxymoron...just go with it). e.g. here are some conferences on symbolic computation: ISSAC, FPSAC, CASC, SNC
There's also probably closely related work happening in automated/interactive theorem provers e.g. the people trying to formalize large bodies of math in Lean etc., but I don't know who/where to point you to for that (hopefully others could chime in).