I'll write my solution.
Let's count how many ways to sit Smith and Sykes together there are. We can put them in $16$ ways (choose the two chairs and then the order). No matter where they sit down, there are $8!$ ways to put the other people.
The probability is
$$\frac{8!\cdot16}{10!}=\frac8{45}$$
However, the probability that they are not next to each other is
$$1-\frac8{45}=\frac{37}{45}$$
Your solution is wrong, but did you misread the book, too?
You asked in comments how I got $181440$ or $14963$. The former is $9! / 2$ which is the number of possible arrangements at the second sitting, after taking into account rotations and reflections. Just taking into account rotations it would be $9! = 362880$
The number with no duplicated neighbours I got with the following R code, using the combinat package to generate all $362880$ possibilities with the person $1$ in the first place, and counting:
library(combinat)
seated <- 10
perms <- matrix(unlist(permn(seated-1)), ncol = seated-1, byrow = TRUE)
permsextended <- cbind(1, perms+1, 1)
pairs <- 100 * permsextended[,-(seated+1)] + permsextended[,-1]
originalpairs <- c(100*(1:(seated-1)) + (2:seated), 100*seated + 1,
100*(2:seated) + (1:(seated-1)), 100*1 + seated)
dupes <- matrix(pairs %in% originalpairs, ncol=seated)
totaldupes <- rowSums(dupes)
nodupes <- permsextended[totaldupes==0, -(seated+1)]
That gives
> nrow(nodupes)
[1] 29926
> nrow(nodupes) / factorial(seated-1)
[1] 0.08246803
and I divided $29926$ by $2$ to get $14963$.
These are the first few examples found
> head(nodupes)
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,] 1 7 2 9 3 10 4 6 8 5
[2,] 1 7 2 10 3 9 4 6 8 5
[3,] 1 7 2 9 3 8 4 6 10 5
[4,] 1 7 2 10 3 8 4 6 9 5
[5,] 1 7 2 8 3 10 4 6 9 5
[6,] 1 7 2 8 3 9 4 6 10 5
There are further curiosities in the data. For example if you number the first sitting from $1$ to $10$, those with even numbers then are more likely to be sitting directly opposite person $1$ in the second sitting, i.e. in the sixth relative position:
> table(nodupes[,6])
2 3 4 5 6 7 8 9 10
4318 2844 3186 3048 3134 3048 3186 2844 4318
If instead of a full count, I do a simulation (no longer constraining player $1$ in the second sitting), I get something similar with
set.seed(1)
cases <- 1000000
seated <- 10 # should be less than 100
originalpairs <- c(100*(1:(seated-1)) + (2:seated), 100*seated + 1,
100*(2:seated) + (1:(seated-1)), 100*1 + seated)
runningcount <- 0
for (i in 1:cases){
example <- sample(seated)
examplextend <- c(example, example[1])
examplepairs <- 100 * examplextend[-(seated+1)] + examplextend[-1]
runningcount <- runningcount + (sum(examplepairs %in% originalpairs)==0)
}
getting
> runningcount / cases
[1] 0.082199
Best Answer
Pick two seats for the tallest and shortest (out of the six that are not end seats,) then pick randomly which one sits in those two seats. Then pick $6!$ different ways to seat the rest of the people.
This gives $\binom{6}{2}\cdot 2!\cdot 6!$ different possibilities.
Then there are only $5$ different ways to choose the two seats in the first step so that they are together, so there are $5\cdot 2!\cdot 6!$ different ways for them to be seated together.
So for (a) we should get:
$$\frac{5}{\binom{6}{2}}=\frac{1}{3}$$
For (b), you do similarly: How many ways are there to choose the two seats so that they have a single seat between them?
There are actually $2\cdot 7!$ ways for the tallest person to be at either end: $7!$ for each end. So your formula should be:
$$P=\frac{6\cdot5\cdot 5!\cdot2!}{8!-2\cdot 7!-2\cdot 7!+2!6!}$$
Which also gives $\frac{1}{3}$.