Let's suppose that we start with two copies of the same plane, both having the same notion of what is an open set. From one, we color the square (and its interior -- a square is a collection of vertices and line segments, but I gather you intend to include its interior) and say that it is no longer present. (This makes some of the statements below a bit easier to make.)
The plane without a square has a boundary: the points on the border of the square. (In fact, all the points of the square are boundary points. None of the points in the square's interior are boundary points.) The unmodified plane has no boundary. Boundary points are discussed in point-set topology.
The presence of this boundary makes the plain plane and the plane without a square nonhomeomorphic. Homeomorphism is the general type of equivalence of topological spaces, again studied in point-set topology. The number of boundary components of a space is invariant under homeomorphism, so these two spaces cannot be homeomorphic.
We also encounter boundary points when studying manifolds. Since we have deleted a closed set from the plane, the plane minus square is still an (open) manifold (without boundary, because the boundary points are all deleted with the square). We continue to study manifolds in algebraic topology where we study how circles (and in higher dimensions, spheres and their higher dimensional analogues) can be drawn on the manifold. For the plane without square, a circle can be drawn that does not encircle the deleted square. Circles can also be drawn that go clockwise one or more times around the square and others that go one or more times anticlockwise around the square. These circles are not contractible to a point (the hole where the square was obstructs shrinking). In fact, by pushing the drawings around the plane without leaving it, we find that two circles encircling the square the same number of times in the same direction can be tweaked to be the same path and for each integer number of encirclings, this sliding gives one circle family (all related by this tweaking of segments of the path) per integer for each integer. We can even sensibly describe adding two such circle paths together and either getting more orbits around the missing square or (if there is cancellation) fewer orbits -- these circles add the same way their associated integers do. This allows us to identify the fundamental group of the plane without square -- it's (isomorphic to) the integers under addition. (The fundamental group is a homotopy group. The other homotopy groups capture properties of connectedness and higher-dimensional behaviour, so are not so useful for the two spaces we are discussing.) For the unmodified plane, all circles are contractible to a point -- there's nothing to obstruct this contraction -- so the fundamental group of the plane is the trivial group with one element (corresponding to the family of circles that contract to a point). Fundamental group is also a homeomorphism invariant, so these two spaces cannot be homeomorphic.
Generically, fundamental groups can be a little hard to compute from a "random" description of a space. Somewhat easier is homology groups, although for the two spaces under discussion, each space's first homology group is identical to that space's fundamental group. That these space's homology groups are different shows that the two spaces are not homeomorphic.
One homeomorphic equivalent to the plane without square is the punctured plane. In this case, we imagine a sort of surface tension which collapses the deleted portion to a single point, the "puncture". The punctured plane is "simpler" than the plane without square, because we don't need any details about the square. In fact, the punctured plane is homeomorphic to a plane without any connected, simply connected region (basically, any kind of blob with no gaps in its interior). This includes the plane without a disk, the plane without a triangle, the plane without a rectangle, the plane without a trapezoid, the plane without a dodecagon, and so on. These are all homeomorphic, so the homeomorphism invariants we discussed above are the same for all of them ... and different from the same invariants for the unmodified plane.
Switching gears, suppose the plane and the plane without square were covered in hair. We could comb the hair in various ways, obtaining vector fields on the space. Notice that if we comb the hair on the plane to have curl there will be (at least) one point where the field drops to zero. If we imagine the vectors indicating flow, the curl field is a rotational flow and the point in the middle, at the eye of the storm, has zero flow. However, in the plane without square, we can arrange for that point to land somewhere in the square -- that is, the vector field has no zero in the plane without square. One can study the continuously varying vector fields that can be put on a space to study the space: smooth sections of tangent bundles. Two of these are usually considered equivalent if one can be smoothly stretched and squeezed into the other. Then there are algebraic invariants of the sets of vector fields and these invariants are different for the plane and the plane without square.
To your third question: The usual numbers for counting (unordered) quantity (zero, one, two, three, ...) are cardinals. There is not a sensible notion of infinite cardinals that differ by a finite amount -- it turns out that two such cardinals are actually the same cardinal. However, the usual numbers for counting (ordered) sequences (first, second, third, ...) are ordinals and infinite ordinals can differ by finite amounts. For instance, there is an ordinal for the integers and an ordinal that comes one before it. These two infinite ordinals differ by $1$. An extension of this is the surreal numbers, which can represent infinite numbers differing by finite amounts and also differing by nonzero infinitesimal amounts.
It would be weird to try to distinguish the plane and the plane without square by ordering all their points and trying to see a difference in their corresponding ordinals. This would also fail -- the two sets would yield the same ordinals.
Measure theory attempts to capture the notions of length and area (and volume, and higher dimensional analogues) without introducing paradoxes, although some paradoxes seem baked into these notions in three and higher dimensions. Measure theory will tell you the area of a square (with its interior), and can with a little effort tell you the area of the deleted region in the plane without square. However, the measure of the plane with or without the square is infinity, so this does not distinguish the "sizes" of the two spaces.
We can, however, choose to view the plane as a stereographic projection of a sphere missing one point (the center of the projection). Then any region in the plane can be carried backwards along the projection to a region on the sphere and we can calculate its area there. Since the area of the sphere is finite ($4 \pi$ times the square of its radius, the one missing point not changing the area), the un-projected area of the plane and the un-projected area of the plane without square will be different finite numbers. This is a very sneaky way to get at your question -- aggressively distort the two infinities so that they contribute finite areas, so that the finite difference becomes discernible.
Best Answer
When dealing with a sum of nonnegative terms $\sum_{n=1}^\infty a_n$, there are only two things such a sum can do:
In case (1.), if we don't care to mention (or don't know) the value $L$, we can write $\sum_{n=1}^\infty a_n < \infty$. This is perfectly standard notation, used in almost any text of analysis.
This also applies to integration of a nonnegative function with respect to a positive measure.