This is best performed by using linear transformations on a coordinate system, where the origin is placed at the vertex of the scaling box that does not move. In other words, if you "grab" the vertex in the lower left, the origin will be the vertex that remains stationary--the one in the upper right. You calculate all of the coordinates of the vertices relative to that origin, and then the scaling transformation would be of the form $(x, y) \mapsto (cx, cy)$ for some scaling constant $c$ that depends on the location of the mouse pointer.
For instance, the coordinates of the large box would be in counterclockwise order $(0,0)$, $(-100,0)$, $(-100,-100)$, $(0,-100)$. Let us assume that prior to the scaling the smaller box's coordinates are $$\{(-40,-50), (-90,-50), (-90,-100), (-40,-100)\}.$$
Then if you choose a scaling factor $c$ based on how the corner at $(-100,-100)$ is moved, e.g., $c = 1/2$, the new coordinates of the outer box become $$\{(0,0), (-50,0), (-50,-50), (0,-50)\}.$$ The inner box becomes $$\{(-20, -25), (-45, -25), (-45,-50), (-20,-50)\}.$$ This furnishes a computationally simple way to rescale.
The only remaining part is to figure out how to recalculate the coordinates based on the choice of vertex to move. If you had moved instead the lower right vertex, the origin would need to be located at the upper left, and the outer box would be $$\{(100,0), (0,0), (0,-100), (100,-100)\}$$ with the inner box coordinates modified accordingly. This is not difficult to do; if in general your scaling box has vertices at $$\{(x_1, y_1), (x_2, y_2), (x_3, y_3), (x_4, y_4)\},$$ where there are restrictions on the $x_i$ and $y_i$ such that these points define a rectangular region with sides parallel to the coordinate axes, then a simple translation can remap all the coordinates based on any choice of vertex; e.g., if I wanted to make the point $(x_2, y_2)$ the origin, I simply subtract $(x_2, y_2)$ from all of the vertices in this list:
$$\{(x_1 - x_2, y_1 - y_2), (0,0), (x_3-x_2, y_3-y_2), (x_4-x_2, y_4-y_2)\}.$$
Best Answer
If the center of resizing (which it sounds like is the center of your outer rectangle—the one point that is not moved by the resizing) is $(x_c,y_c)$ and you're resizing by a factor of $r_x$ in the $x$-direction ($r_x=\frac{83}{185}$ in your example) and $r_y$ in the $y$-direction ($r_y=\frac{330}{185}$ in your example), then $$(x_\text{new},y_\text{new})=(x_c+r_x(x_\text{old}-x_c),y_c+r_y(y_\text{old}-y_c)).$$