There may be contexts where the first definition is appropriate, but it does
seem somewhat pathological in that a sequentially compact subspace may not be
relatively sequentially compact. In this respect it differs
from the second definition.
For example, take the Tychonoff plank
$X = ([0, \omega_1] \times [0, \omega]) \setminus (\omega_1, \omega)$
and the subspace $A = [0, \omega_1) \times [0, \omega]$.
Then $A$ is sequentially compact, since any sequence is confined to a
subspace $[0, \alpha] \times [0, \omega]$ with $\alpha < \omega_1$, which
is compact and first countable. On the other hand $\overline{A} = X$,
which is not sequentially compact since $\{ (\omega_1, n) \}_{n=0}^\infty$
has no cluster point.
To decide which is the most useful definition would probably involve looking
at a large number of applications, which I don't have available. I could
even imagine some use for a definition $1\frac12$: a subspace $A$ of a
topological space $X$ is relatively sequentially compact if there is a
sequentially compact $B \subset X$ such that $A \subset B$. This is strictly
weaker than definition 1 and stronger than definition 2, although I don't
know if it is strictly stronger than definition 2.
In any space an open set is sequentially open by definition.
The main observation for the cofinite topology (on an infinite set) is the following: there are exactly three types of sequence:
a. $(x_n)$ has infinitely many terms, so $\{x_n: n \in \mathbb{N}\}$ is infinite.
Then $\lim (x_n) =X$, i.e. every $p \in X$ is a limit of this sequence.
b. $(x_n)$ has finitely many distinct terms and is eventually constant, i.e. there is some $N$ and some $x \in X$ such that $\forall n \ge N: x_n = x$ and in that case $\lim (x_n) = \{x\}$.
c. $(x_n)$ has finitely many terms and is not eventually constant. Then this sequence has no limit, so $\lim_n (x_n) = \emptyset$.
Now suppose that $U$ is sequentially open, so for any sequence $(x_n)$ in $X$ we have that if $\emptyset \neq \lim_n (x_n) \subseteq U$ then $(x_n)$ is eventually in $U$. Note that the non-emptyness is important (see also wikipedia for another formulation of this definition: for all sequences $(x_n)$ and all $x$, if $x_n \to x$ and $x \in U$ then $(x_n)$ is eventually in $U$.)
We want to show that $U$ is open. If $U$ is empty, we are done, so assume $U\neq \emptyset$. Suppose we have a sequence with $\lim_n (x_n) \subseteq U$. If it is of type a. then $\lim (x_n) = X$ and $X = U$ and we are done. If it is of type b., we are also done as in particular $x \in U$ and $(x_n)$ is eventually $x$. Type c. is not an allowed "test sequence" for sequential openness. So in all cases $U$ is open and we're done.
We could also test (more easily) sequentially closed sets: suppose $F$ is infinite and sequentially closed. If $F \neq X$ pick $p \notin F$ and countably many different $x_n \in F$ for all $n$. Then $(x_n)$ would be a sequence in $F$ that converges to $p$ while $;p \notin F$, so $F$ would be not be sequentially closed at all. This contradiction came from assuming $F \neq X$, so $F=X$. Hence all sequentially closed sets are finite or $X$, as required.
So indeed the cofinite topology is a sequential space.
Best Answer
Let $X$ be an uncountable set with the co-countable topology; a sequence $\langle x_n:n\in\Bbb N\rangle$ in $X$ converges to a point $x\in X$ if and only if there is an $m\in\Bbb N$ such that $x_n=x$ for all $n\ge m$. Let $A$ be an uncountable subset of $X$ with uncountable complement; then $A$ is not open, but any sequence converging to a point of $A$ is eventually in $A$.
Of course that space is only $T_1$, but we can do better. Let $\alpha$ be any ordinal greater than $\omega_1$, and let $X$ be $\alpha$ with the order topology. $X$ is quite nice, being hereditarily normal. Let $A=\{\xi\in X:\omega_1\le\xi\}$; $A$ is not open in $X$, but $A$ is sequentially open, since every sequence frequently in $X\setminus A$ either converges to a point of $X\setminus A$ or fails to converge.