It sounds like xindy is unable to determine how to sort one or more of your entries, but without a minimal example it's hard to tell exactly. The default is for xindy to sort on the entry name unless you have specified a sort value. However, xindy ignores (La)TeX commands so if the sort key only contains commands, xindy is left with a blank sort key and issues a warning. Earlier versions of xindy gave a somewhat cryptic error:
ERROR: CHAR: index 0 should be less than the length of the string
Newer versions give a slightly more understandable warning:
WARNING: Would replace complete index key by empty string, ignoring
Since the first warning isn't very easy to understand, makeglossaries searches for instances of it and if found gives the warning:
Possible cause of problem: Sort key required for entries only containing command names
Here's a minimal example illustrating the problem:
\documentclass{article}
\usepackage[xindy]{glossaries}
\makeglossaries
\newglossaryentry{P}{name={\P},description={Paragraph symbol}}
\newglossaryentry{S}{name={\S},description={Section symbol}}
\begin{document}
Reference the terms: \gls{S} \gls{P}.
\printglossaries
\end{document}
In this case the first term has the sort key \P and the second term has the sort key \S. In both cases, once xindy has stripped out the commands, it's left with an empty sort string and doesn't know how to compare the entries. Should \P come before or after \S? The solution is to provide a sort key for these entries:
\documentclass{article}
\usepackage[xindy]{glossaries}
\makeglossaries
\newglossaryentry{P}{name={\P},sort={P},description={Paragraph symbol}}
\newglossaryentry{S}{name={\S},sort={S},description={Section symbol}}
\begin{document}
Reference the terms: \gls{S} \gls{P}.
\printglossaries
\end{document}
If all your terms are like this (for example they are all symbols) then it may not make any sense trying to sort them alphabetically. In which case it might be better to sort them by definition (package option sort=def, as you mentioned in your comment) or by use in the document. In either of these cases, glossaries manufactures a numerical sort key for each term, so the issue no longer arises.
Edit: There's a GUI diagnostic tool for the glossaries package called makeglossariesgui which can provide further details by analysing the .aux, .log and associated glossary files. The example below has three problematic entries and a working one for comparison:
\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[xindy]{glossaries}
\makeglossaries
\newglossaryentry{sample}{name={sample},
description={an example}}
\newglossaryentry{S}{name={\S},
description={section symbol}}
\newglossaryentry{alpha}{name={\ensuremath{\alpha}},
description={alpha}}
\newglossaryentry{beta}{name={$\beta$},text={\beta},
description={beta}}
\begin{document}
Test: \gls{sample}, \gls{S}, $\gls{alpha}$, $\gls{beta}$.
\printglossaries
\end{document}
A LaTeX run is first required to generate the .aux file (although this is usually already present if you're trying to work out why xindy has failed). The .aux file can then be opened in makeglossariesgui using the File > Open menu item.
For the above MWE, an error dialog is displayed to show that xindy failed:
Once that dialog box is dismissed, the "Diagnostics" tab lists each problem:
The text displayed is:
Xindy won't accept the sort value \S (for entry S) as it's treated as
an empty string. This is because xindy ignores (La)TeX commands within
the sort field, and once these commands have been stripped the sort
field becomes empty. You will need to explicitly set the sort field
using the sort key in the entry definition.
Xindy won't accept the sort value \ensuremath {\alpha } (for entry alpha) as it's treated as
an empty string. This is because xindy ignores (La)TeX commands within
the sort field, and once these commands have been stripped the sort
field becomes empty. You will need to explicitly set the sort field
using the sort key in the entry definition.
Xindy won't accept the sort value $\beta $ (for entry beta) as it's treated as an empty
string. This is because xindy ignores (La)TeX commands within the sort
field, and once these commands have been stripped the sort field
becomes empty. You will need to explicitly set the sort field using
the sort key in the entry definition.
Xindy reported the following:
WARNING: Would replace complete index key by empty string, ignoring
#<ordrule-regexp: '\\[a-zA-Z@]+ *' => '' :again NIL :only-at-start NIL
ERROR: CHAR: index 0 should be less than the length of the string
So this explicitly identifies each problematic entry by its label (which neither xindy nor makeglossaries do).
The "General Information" tab contains a "Details" link, which when clicked on opens a window with a summary of all the entries. The second column shows the sort key, which is displayed in red if there's a problem with it.
Once the problems have been fixed, you can switch back to using makeglossaries or explicitly xindy.
Incidentally, the glossaries-extra package, provides the command \glsxtrnewsymbol when the symbols package option is used. This internally uses \newglossaryentry but sets the sort value to the label rather than the name field. For example:
\glsxtrnewsymbol[text={\beta},description={beta example}]{beta}{$\beta$}
This is equivalent to
\newglossaryentry{beta}{name={$\beta$},text={\beta},
sort={beta},
type=symbols,
category=symbol,
description={beta example}}
This seems to be a problem with the way xindy sorts sub-entries. It isn't specific to the glossaries package.
Analysis
The following is a simpler illustration:
\documentclass{article}
\usepackage{fontspec}
\usepackage{makeidx}
\setmainfont{Liberation Serif}
\makeindex
\newcommand*{\lettergroupDefault}[1]{\lettergroup{#1}}
\begin{document}
α\index{α}
ω\index{ω}
r\index{r}
s\index{s}
\printindex
\end{document}
Assuming this document is called test.tex then:
lualatex test
texindy -L english -C utf8 -o test.ind test.idx
lualatex test
produces:
This is because xindy has been instructed to sort according to the English alphabet, which is a Latin alphabet, so the non-Latin characters are put in the "default" group because the english module has no rule for them.
If instead you were to do:
lualatex test
texindy -L greek -C utf8 -o test.ind test.idx
lualatex test
then the index looks like:
Now xindy has been instructed to sort according to Greek alphabet, which has no rules for Latin characters, so they're put in the "default" group.
Suppose now you try:
lualatex test
texindy -L russian -C utf8 -o test.ind test.idx
lualatex test
then all the Latin and Greek characters end up in the "default" group:
This is because xindy has now been instructed according to the Cyrillic alphabet and has no rules for the Latin or Greek characters.
Returning to glossaries, here's the index example rewritten:
\documentclass{article}
\usepackage{fontspec}
\usepackage[xindy]{glossaries}
\setmainfont{Liberation Serif}
\makeglossaries
\newglossaryentry{α}{name={α},description={alpha}}
\newglossaryentry{ω}{name={ω},description={omega}}
\newglossaryentry{r}{name={r},description={r}}
\newglossaryentry{s}{name={s},description={s}}
\begin{document}
\gls{α}
\gls{ω}
\gls{r}
\gls{s}
\printglossary[style=indexgroup]
\end{document}
Now:
lualatex test
makeglossaries test
lualatex test
produces:
because it's using the English sorting rules.
This is fine so far, but sub-entries seem to be behaving differently. A slight adjustment to the above example illustrates this:
\documentclass{article}
\usepackage{fontspec}
\usepackage[xindy]{glossaries}
\setmainfont{Liberation Serif}
\makeglossaries
\newglossaryentry{d}{name={d},description={parent}}
\newglossaryentry{α}{name={α},description={alpha},parent=d}
\newglossaryentry{ω}{name={ω},description={omega},parent=d}
\newglossaryentry{r}{name={r},description={r},parent=d}
\newglossaryentry{s}{name={s},description={s},parent=d}
\begin{document}
\gls{α}
\gls{ω}
\gls{r}
\gls{s}
\printglossary[style=indexgroup]
\end{document}
This results in the strange ordering you noticed. I don't know why it puts the r entry before the Greek letters within the sub-entries, but it behaves as expected if I switch the language back to Greek:
\documentclass{article}
\usepackage{fontspec}
\usepackage[xindy={language=greek,glsnumbers=false}]{glossaries}
\setmainfont{Liberation Serif}
\makeglossaries
\newglossaryentry{d}{name={d},description={parent}}
\newglossaryentry{α}{name={α},description={alpha},parent=d}
\newglossaryentry{ω}{name={ω},description={omega},parent=d}
\newglossaryentry{r}{name={r},description={r},parent=d}
\newglossaryentry{s}{name={s},description={s},parent=d}
\begin{document}
\gls{α}
\gls{ω}
\gls{r}
\gls{s}
\printglossary[style=indexgroup]
\end{document}
This produces:
So for some reason, the english xindy module seems to have a slightly different way of sorting the sub-entries. Again, this isn't specific to glossaries as it can be demonstrated with \index:
\documentclass{article}
\usepackage{fontspec}
\usepackage{makeidx}
\setmainfont{Liberation Serif}
\makeindex
\newcommand*{\lettergroupDefault}[1]{\lettergroup{#1}}
\begin{document}
α\index{d!α}
ω\index{d!ω}
r\index{d!r}
s\index{d!s}
\printindex
\end{document}
which produces:
The problem persists even if I define a letter group for the Greek characters:
% arara: lualatex
% arara: xindy: {language: english, codepage: utf8, modules: [basename, texindy]}
% arara: lualatex
\documentclass{article}
\usepackage{filecontents}
\usepackage{fontspec}
\usepackage{makeidx}
\setmainfont{Liberation Serif}
\makeindex
\newcommand*{\lettergroupDefault}[1]{\lettergroup{#1}}
\begin{filecontents*}{\jobname.xdy}
(define-letter-group "Greek"
:prefixes ("α" "ω")
:after "Z")
\end{filecontents*}
\begin{document}
1\index{d!1}
+\index{d!+}
α\index{d!α}
a\index{d!a}
z\index{d!z}
ω\index{d!ω}
r\index{d!r}
s\index{d!s}
1\index{1}
+\index{+}
a\index{a}
r\index{r}
s\index{s}
z\index{z}
α\index{α}
ω\index{ω}
\printindex
\end{document}
This produces:
The main (top) level entries have been correctly sorted, but not the sub-entries.
Recommendation
I think the simplest solution is to either use a number instead of the Greek letters in the sort key (1 for α etc) or use a prefix that is sorted correctly (such as +). For example:
% arara: lualatex
% arara: xindy: {language: english, codepage: utf8, modules: [texindy]}
% arara: lualatex
\documentclass{article}
\usepackage{fontspec}
\usepackage{makeidx}
\setmainfont{Liberation Serif}
\makeindex
\newcommand*{\lettergroupDefault}[1]{\lettergroup{#1}}
\begin{document}
α\index{d!+α@α}
a\index{d!a}
z\index{d!z}
ω\index{d!+ω@ω}
r\index{d!r}
s\index{d!s}
\printindex
\end{document}
Translating this back to glossaries:
% arara: lualatex
% arara: makeglossaries
% arara: lualatex
\documentclass{article}
\usepackage{fontspec}
\usepackage[xindy]{glossaries}
\setmainfont{Liberation Serif}
\makeglossaries
\newglossaryentry{d}{name={d},description={parent}}
\newglossaryentry{α}{name={α},sort={+α},description={alpha},parent=d}
\newglossaryentry{ω}{name={ω},sort={+ω},description={omega},parent=d}
\newglossaryentry{r}{name={r},description={r},parent=d}
\newglossaryentry{s}{name={s},description={s},parent=d}
\begin{document}
\gls{α}
\gls{ω}
\gls{r}
\gls{s}
\printglossary[style=indexgroup]
\end{document}
This produces:
Best Answer
It is a bug in glossaries. It calls xindy with an empty -C value as the code page is not set:
Report the bug. As a work around load either fontspec or set the code page manually: