Actually, after some manipulations, you can use the master theorem! Let us see how. First, let us prove the following lemma:
Lemma: The function $T$ is non-decreasing, i.e. $T(n) \leq T(n+1)$ for all $n \in \mathbb{N}$.
Proof. By strong induction on $n \in \mathbb{N}$.
Base cases: For all $0 \leq n \leq 6$, one has $T(n) = 1 = T(n+1)$. Moreover, $T(7) = 1 < 2\,T(0) + 8^{\pi/2} = T(8)$, as $\big\lfloor \frac{8}{\sqrt{2}} \big\rfloor =5$.
Inductive step: Let $n > 7$. The strong induction hypothesis is $T(k) \leq T(k+1)$ for all $0 \leq k < n$. The goal is to prove that $T(n) \leq T(n+1)$.
By definition,
\begin{align}
T(n) &= 2\,T\big(\big\lfloor \frac{n}{\sqrt{2}} \big\rfloor - 5\big) + n^\frac{\pi}{2}
&
T(n+1) &= 2\,T\big(\big\lfloor \frac{n+1}{\sqrt{2}} \big\rfloor - 5\big) + (n+1)^\frac{\pi}{2}\,.
\end{align}
According to the properties of the floor function, $\big\lfloor \frac{n+1}{\sqrt{2}} \big\rfloor \leq \big\lfloor \frac{n}{\sqrt{2}} \big\rfloor + \big\lfloor \frac{1}{\sqrt{2}} \big\rfloor + 1 = \big\lfloor \frac{n}{\sqrt{2}} \big\rfloor + 1$, and $\big\lfloor \frac{n}{\sqrt{2}} \big\rfloor \leq \big\lfloor \frac{n}{\sqrt{2}} \big\rfloor + \big\lfloor \frac{1}{\sqrt{2}} \big\rfloor \leq \big\lfloor \frac{n+1}{\sqrt{2}} \big\rfloor$, since $\sqrt{2} > 1$.
Therefore, there are only two cases:
- either $\big\lfloor \frac{n+1}{\sqrt{2}} \big\rfloor = \big\lfloor \frac{n}{\sqrt{2}} \big\rfloor$ and then $T(n) \leq 2\,T\big(\big\lfloor \frac{n}{\sqrt{2}} \big\rfloor - 5\big) + (n+1)^\frac{\pi}{2} = T(n+1)$, where the inequality holds because from $\frac{\pi}{2} > 0$ it follows that $n^\frac{\pi}{2} < (n+1)^\frac{\pi}{2}$ ;
- or $\big\lfloor \frac{n+1}{\sqrt{2}} \big\rfloor = \big\lfloor \frac{n}{\sqrt{2}} \big\rfloor + 1$; we can apply the strong induction hypothesis to $T \big(\big\lfloor \frac{n}{\sqrt{2}} \big\rfloor- 5 \big)$ because $\big\lfloor \frac{n}{\sqrt{2}} \big\rfloor - 5 < n$ (indeed, $n + 5 > n = \lfloor n \rfloor \geq \big\lfloor \frac{n}{\sqrt{2}} \big\rfloor $ since $\lfloor \cdot \rfloor$ is non-decreasing), so $T \big(\big\lfloor \frac{n}{\sqrt{2}} \big\rfloor- 5 \big)\leq T \big(\big\lfloor \frac{n}{\sqrt{2}} \big\rfloor- 5 + 1\big) = T \big(\big\lfloor \frac{n + 1}{\sqrt{2}} \big\rfloor- 5 \big)$ and hence $T(n) \leq 2\,T\big(\big\lfloor \frac{n}{\sqrt{2}} \big\rfloor - 5\big) + (n+1)^\frac{\pi}{2} = T(n+1)$. $\qquad\square$
As $T$ is non-decreasing by the lemma above (and $\big\lfloor \frac{n}{\sqrt{2}} \big\rfloor - 5 < \big\lfloor \frac{n}{\sqrt{2}} \big\rfloor \leq \frac{n}{\sqrt{2}}$), for $n > 7$ one has $T(n) = 2\,T\big(\big\lfloor \frac{n}{\sqrt{2}} \big\rfloor - 5\big) + n^\frac{\pi}{2} \leq 2\,T\big(\frac{n}{\sqrt{2}}\big) + n^\frac{\pi}{2}$. Therefore, if we set
\begin{align}
S(n) =
\begin{cases}
1 &\text{if } n = 0 \\
2\,S\big(\frac{n}{\sqrt{2}}\big) + n^\frac{\pi}{2} & \text{otherwise}
\end{cases}
\end{align}
then $T(n) \leq S(n)$ for all $n \in \mathbb{N}$ and so, for any function $g$, $S(n) \in O(g(n))$ implies $T(n) \in O(g(n))$, i.e. the fact that $S$ grows asymptotically no faster than $g$ implies that $T$ grows asymptotically no faster than $g$.
The point is that we can use the master theorem to find a $g$ such that $S(n) \in O(g(n))$. Using the same notations as in Wikipedia article:
\begin{align}
a &= 2 & b&= \sqrt{2} & c_\text{crit} &= \log_\sqrt{2} 2 = 2 & f(n) &= n^{\pi/2}
\end{align}
thus, $S(n) \in O(n^2)$ by the master theorem (since $\pi/2 < 2 = c_\text{crit}$), and hence $T(n) \in O(n^2)$.
This answer to your question may not have a closed form, but it is not that complicated. Define the real constants
$$ w :=\! \sqrt{3}, \;
p :=\! (-1\!+\!w)/2, \;
q :=\! (-1\!-\!w)/2. \tag{1} $$
Define the recurrence function
$$ f(x) := 2x^2/(1-2x^2). \tag{2} $$
Notice that $\,p\,$ is a repelling while $\,q\,$ is
an attracting fixed point of $\,f(x).\,$
Define the convergent power series
$$ s(x) \!:=\! x \!+\!\! \left(\frac12\!-\!\frac56w\right)\!x^2
\!+\!\! \left(\frac73\!-\!\frac23w\right)\!x^3 \!\!+\! O(x^4).\!
\tag{3} $$
Then the following functional equation holds
$$ f(q+s(x)) = q+s(-2px) \tag{4} $$
where the the coefficients of $\,s(x)\,$ are
uniquely determined so that the functional
equation holds.
Thus the convergent ($\;|\!-\!2p|<1\;$) sequence
for any $\,x_0\,$ defined by
$$ a_n := q + s((-2p)^n x_0) \tag{5} $$
satisfies $\,a_n \to q\,$ as $\,n\to\infty\,$
and the recursion
$$ a_{n+1} = f(a_n). \tag{6} $$
For your information here is a PARI/GP
code to calculate the power series for $\,s(x)\,$
w = quadgen(12); p = (-1+w)/2; q = (-1-w)/2;
f(x) = 2*x^2/(1-2*x^2);
nxt(n) = {my(s, c='c);
s = truncate(stx + O(x)^n) + c*x^n*(1 + O(x));
s = truncate(f(q+s) - (q+subst(s, x, -2*p*x)))/x^n;
sx -= x^n * polcoeff(s, 0, c)/polcoeff(s, 1, c)};
sx = x ; for(n=2, M=9, nxt(n)); print(sx + x*O(x)^M);
Best Answer
Let's check if \begin{equation} T(n) = \frac{15}{7}n \end{equation} is a solution for our recursion. \begin{align} T(n) &= \frac{15}{7}n \\ T(\frac{n}{3}) &=\frac{1}{3} \frac{15}{7}n\\ T(\frac{n}{5}) &=\frac{1}{5} \frac{15}{7}n \end{align} So \begin{equation} T(\frac{n}{3}) + T(\frac{n}{5}) + n = \frac{1}{3} \frac{15}{7}n + \frac{1}{5} \frac{15}{7}n + n \end{equation} which is \begin{equation} T(\frac{n}{3}) + T(\frac{n}{5}) + n = \frac{3}{7}n + \frac{5}{7}n + n = \frac{8+7}{7}n = \frac{15}{7}n = T(n) \end{equation} So \begin{equation} T(n) = \frac{15}{7}n \end{equation} is indeed a solution.