Based on our discussion in the comments, you created a new .bst file using the custom-bib package. This .bst file then needs to be modified to be used with the jabbrv package.
It seems that the .bib files created by custom-bib package are a bit more sophisticated than the basic .bst files assumed by the package, and the modification suggested in the package web site doesn't work. Thanks to some discussion in chat with Joseph Wright, here's a modified function that works:
FUNCTION {format.journal}
{journal duplicate$ empty$
'pop$
{
"\JournalTitle{" swap$ * "}" *
emphasize
output
}
if$
}
This function is added to the .bst file and the following line in the {article} function of the .bst file
journal "journal" output.check
is changed to
format.journal "journal" output.check
Name the new .bst file jabbrv_thesis. I've put a copy here:
Modified .bst file
Now you can use the jabbrv package to do the abbreviations. To do this you use:
\usepackage[warnundef]{jabbrv}
\bibliographystyle{jabbrv_thesis}
If there are particular journal names that you don't want abbreviated (e.g. Nature) then you need to edit the jabbrv-ltwa-en.ldf file (located in the jabbrv folder) and comment out the \DefineJournalAbbreviation lines for those names. If there words that aren't abbreviated, but should be, (these will show up in the warnings) then you need to add lines like the following to your document.
\DefineJournalAbbreviation{<full name>}{<abbreviation>} % no final '.'
Thanks to all the comments above, I ended up going for biblatex. (vancouver.bst doesn't do what I was hoping for.)
I thought I would post my solution as an answer, in case anybody stumbles upon this question and like me doesn't know what to do.
However, it is my first time using biblatex, so my soltion is most likely not optimal, and might involve some re-inventing the wheel (as some of the commands might already exist, seeing as it's all new to me).
\usepackage[style=authoryear]{biblatex}
\DeclareNameAlias{sortname}{first-last}
\bibliography{BibFile}
%new cite command: "Vancouver Short"
\DeclareCiteCommand{\citeVS}
{\usebibmacro{prenote}}
{\usebibmacro{author}, \usebibmacro{title}}
{\multicitedelim}
{\usebibmacro{postnote}}
% new cite command: "Vancouver Short Collection" - necessary when referencing whole collections.
\DeclareCiteCommand{\citeVSc}
{\usebibmacro{prenote}}
{\usebibmacro{editor}, \usebibmacro{title}}
{\multicitedelim}
{\usebibmacro{postnote}}
Any improvements/suggestions/edits (to this answer) are welcome.
Best Answer
"It shows the order used in my text." is the default behaviour of the Vancouver system rather than an error. Instead of
\bibliographystyle{vancouver}, use e.g.\bibliographystyle{plain}that is a numbered and alphabetically sorted citation style.