I need to draw first a rectangle with curvy looking edges.
Then I need a rectangle with curvy looking edges and also rounded corners. But the corners need to be inverted (pointing inwards).
I have made a handmade drawing for illustration:
[Tex/LaTex] tikz rectangle with curved edges and inverted rounded corners
tikz-pgf
Related Solutions
The problem is that the rounded corners option is getting applied to the result of the decoration. So you need to turn it off after the initial path is constructed but before the decoration is applied. This cannot be done in the decoration key because the options for that are actually processed when this key is originally parsed (ie at the start of the path). So putting /tikz/sharp corners in the decoration simply negates the rounded corners.
Decorating the path is the very next thing done after the path has been processed so it's hard to stick something in between the path being constructed and being decorated! However, it is possible and I'll present two ways. The first works by splitting the construction into two pieces. First we construct the rounded rectangle. We do not draw it, but we save it. Then we decorate it in a completely different command. This separates the construction from the decoration and means that the scope of the rounded corners option is over by the time we get to the decoration.
The second way works because TikZ (or PGF) constructs a path as it reads it, rather than saving everything for the end. This means that it is possible to insert options in the middle of the path and have them take effect from that point on (this only applies to options that affect how the path is constructed, not how it is rendered (so colour doesn't work)). We basically want to turn off rounded corners at the end of the path so we do precisely that: at the end of the path we put [sharp corners]. Then by the time the decoration is processed, we no longer round corners.
The second method is simpler, but the first is more powerful. Indeed, it's the idea behind my spath library (still in alpha - indeed, looking at the code for this answer showed that that library lacks a few essential pieces): namely that a path can be defined and used in two different places.
With all that, here's the code (note that the \makeatletter ... \makeatother piece is only needed for the first of the two solutions):
\documentclass[a4paper]{article}
%\url{http://tex.stackexchange.com/q/38989/86}
\usepackage{tikz}
\makeatletter
\tikzset{
use path/.code={\pgfsyssoftpath@setcurrentpath{#1}}
}
\makeatother
\usetikzlibrary{decorations.pathmorphing}
\begin{document}
\begin{tikzpicture}
% First method: define and then use
\path[rounded corners=30mm,save path=\rectpath]
(0,0) rectangle (8,8);
\draw[use path=\rectpath,
decorate,decoration={snake,amplitude=5,segment length=5}] (0,0);
% Second solution: sharp corners at the end
\draw[rounded corners=30mm,
decorate,decoration={snake,amplitude=5,segment length=5}]
(0,9) rectangle +(8,8) [sharp corners];
\end{tikzpicture}
\end{document}
Here's the result:
As I said in my comment, I hope that even if they fix this then they provide a way to turn it back on as the original picture - whilst not what you want - is quite fun! I especially like the fact that the snake decoration turns into a birds nest.
Make the radius of the rounded corners smaller. Remove =10mm and you will see the instant effect. Also, in your case, I find no need to load the xcolor package since it is used internally by tikz, unless, as noted by Alain Matthes, you use some of its options for naming your colors. You can uncomment it if you like. The libraries are not needed here, too, so I commented them out.
Here is your figure when the 10mm is dropped.
And here is one with scaling (not really necessary, just for demonstration purposes).
%\documentclass{article}
\documentclass[border=5]{standalone}
\usepackage{tikz}
%\usepackage{xcolor}
%\usetikzlibrary{arrows,shapes,snakes,automata,backgrounds,petri}
\begin{document}
\begin{tikzpicture}[scale=3]
\draw [rounded corners,fill=gray!20] (0,0)--(1,0)--(0.5,1)--cycle;
\end{tikzpicture}
\end{document}
Here I have scaled the figure up to 3 since your figure is too small.
To make a point, see what happens when I scale the figure up by 5 and set rounded corners=10mm.
Best Answer
Note that I think I ought not answer 'do-it-all-for-me's such as this one. Also, be aware that I am much less likely to be responsive to follow-ups in such cases, unless your follow-up happens to appeal to me. If you want any adjustments, I therefore recommend that you desire the addition of one or more of the following: