The law of reflection also holds for non-plane mirrors, provided that the normal at any
point on the mirror is understood to be the outward pointing normal to the local tangent plane of the mirror at that point
While the law of reflection is based in physics, interpreting it purely mathematically seems useful. Though this produces a problem for surfaces where the tangent is undefined. In real life, I don't think such a surface could exist. But in a world where surfaces with undefined tangents can exist, I have no idea what would happen.
What would happen if a light/billiard hit a point with an undefined tangent? Is it undefined?
Best Answer
I am not sure how appropriate this is for the math section, but here it is.
I think an insightful approach is to use Huygens principle for refraction. Use this video and the many others that show rather than compute how things work in this model. It is not the ray-based model of optics that you mentioned, and it is not the best model that there is in physics, but for the purpose of this question I think it is good enough to get the right picture.
So if you look at a couple of animations with Huygens principle you will notice that the sources of light generate some wave-like things that propagate with a spherical symmetry around the source point. Put a bunch of these source points on a surface and let them generate their wave-like patterns and you get a superposition of multiple waves that travel through the medium in question. Now if those waves happen to "hit" an object, on each point on the surface of that object you will get a "secondary" source, in the sense that each point is treated as starting to generate waves itself. There are some observations to be made here, such as the phase difference between two secondary sources due to a difference in the distance between the main source and the two secondary sources, but what matters for now is that the surface of an object hit by waves becomes a secondary source and starts generating waves itself.
Now back to your question, in this model you can imagine that at the sharp tip of your mirror surface ideally you have only one point. If it is hit by the wave from a main source, it will become a secondary source and start generating a spherical wave. Since in the image you uploaded you have two surfaces that create that sharp edge, the main contribution to the superposition of all the secondary waves will be given by the two plane surfaces and a negligible amount from the tip.
So the take away is that given the more physical nature of the question, the answer is based on physics also, in the sense that you don't apply the ray model for this scenario, but a more general one namely the wave-based one. Sure, the two can be linked via approximations, but the idea of this answer was to give a direction and maybe a mental image of a possible solution to your question.