At school and in all my math books from school (all the way from elementary to high school), the only kind of parenthesis which I have seen used to control the order of operations (like how the parenthesis in $(a+b)\cdot c$ makes the addition come before the multiplication) has been the curved ones: '(' and ')', whereas the box-shaped parenthesis: '[' and ']' only has been used to indicate intervals (like this $[42;\infty[$).
But in the (mostly college/university level) physics and mathematics material I have found on the internet and read on my own, I often find that the box parenthesis is used instead of the curved parenthesis, where my school math book, for instance, would have written something like $(f(x)+g(x))^2$, I find that others have written $[f(x)+g(x)]^2$ (neither examples are directly taken from a specific source, but they show the general difference)
My question is, therefore as follows: if both kinds of parenthesis are equally valid, why do my math books consequently use only the one; if the parenthesis are not equally valid, when should I use the one or the other; does this have to do with national conventions (My math books are all Danish, and the other physics/mathematics material I read are English), and finally why did all my schools only teach me to use one specific parenthesis.
Best Answer
Sometimes, specific types of brackets have specific meanings. As you noted, interval notation is one of these. As another example, set notation is almost always done with curly braces.
When brackets are simply used as grouping symbols, it matters a lot less. Some people just use round parentheses universally. Some like to use parentheses for the innermost set in a nesting, and then square brackets outside of those, and then curly braces. This is also fine. Sometimes, when I'm teaching, I'll use parentheses around the argument of a function, and square brackets around the argument of an operator. (E.g., $f(x)$, but $\frac{d}{dx}[f-g]$.
Generally, just do what you like, but remain alert and flexible. When someone defines a specific notation in a specific context, follow suit in that context. When someone doesn't define a specific notation, be aware that these symbols are used differently by different writers, and be prepared to adapt if a usage seems confusing.