I have the indefinite quadratic form $q(x,y,z) = 19 x^2 + 5 y^2 – z^2.$ It's not my fault. I find, on reflection, that I have no idea how to describe the orthogonal group of this over the integers. The thing is isotropic in $\mathbb Z,$ for example $19 + 45 = 64, $ and $76 + 5 = 81.$ Even if that were not the case, the orthogonal group would contain those for the binary forms $19 x^2 – z^2$ and $5 y^2 – z^2.$ Note that there is an answer for the original Pythagorean problem, Orthogonal group of quadratic form and to some extent http://en.wikipedia.org/wiki/Tree_of_Pythagorean_triples Oh, note that i found all null vectors of the form, https://math.stackexchange.com/questions/483496/pythagorean-triples-with-additional-parameters/483510#483510 which took two recipes because of an annoying odd even thing.
Anyway, given the diagonal matrix $F$ with diagonal entries $19,5,-1,$ here are a number of integral matrices $P$ that solve $P^t F P = F,$ and so are called automorphs or members of the (automorphism, orthogonal, isometry, rotation) group. As we are in odd dimension, we do not much care about the determinant and allow $\pm 1.$
Note also that a recent Bulletin included two articles on the integral orthogonal group for an indefinite quaternary form, one by Kontorovich and one by Fuchs, anyway see http://www.ams.org/journals/bull/2013-50-02/
Um, let's see, oddity, you can take all entries of $P$ positive if you wish, so I'm just printing those. Since we can negate any row or any column at our leisure, it hurts nothing.
QUESTION: can someone please give generators for the entire group, that is every isometry can be written as a word in said generators? I'm hoping I've given enough information to do that. I'm guessing it should be no more than about five matrices, with any luck in this list or evident products of same.
EDIT: Reading Keith's eight-page note at http://www.math.uconn.edu/~kconrad/blurbs/ called Orthogonal Group of $x^2 + y^2 – z^2.$ Maybe I can do this myself, given time. Not sure. I did learn a bit about reflections for the "Euclidean Forms" project. Alright, Keith gives his five reflections at the bottom of page 2. So, let's see. need to collect a bunch of vectors where the quadratic form evaluates to one of $-2,-1,1,2.$ Do that tomorrow, easy enough.
List of some vectors with quadratic form equal to -2,-1,1,2 for 19 x^2 + 5 y^2 – z^2
x y z
-----------------------------------------
-2 21 34 119
-2 21 170 391
-2 31 18 141
-2 39 110 299
-2 39 206 491
-2 59 186 489
-2 59 282 681
-2 131 270 831
-2 159 278 931
-2 201 94 901
-2 229 18 999
-2 231 70 1019
-----------------------------------------
-1 0 0 -1
-1 0 0 1
-1 0 4 9
-1 0 72 161
-1 1 1 5
-1 1 4 10
-1 1 11 25
-1 1 29 65
-1 1 76 170
-1 1 199 445
-1 4 8 25
-1 4 28 65
-1 4 172 385
-1 10 10 49
-1 10 106 241
-1 10 286 641
-1 14 10 65
-1 14 170 385
-1 15 4 66
-1 15 27 89
-1 15 39 109
-1 15 85 201
-1 15 113 261
-1 15 228 514
-1 15 300 674
-1 20 8 89
-1 20 76 191
-1 20 80 199
-1 20 284 641
-1 21 18 100
-1 21 23 105
-1 21 77 195
-1 21 87 215
-1 21 213 485
-1 21 238 540
-1 24 4 105
-1 24 84 215
-1 24 104 255
-1 25 1 109
-1 25 53 161
-1 25 56 166
-1 25 160 374
-1 25 167 389
-1 30 42 161
-1 30 266 609
-1 35 23 161
-1 35 46 184
-1 35 115 299
-1 35 161 391
-1 39 0 170
-1 39 85 255
-1 39 255 595
-1 45 15 199
-1 45 77 261
-1 45 122 336
-1 45 246 584
-1 49 11 215
-1 49 91 295
-1 49 124 350
-1 49 284 670
-1 50 46 241
-1 55 11 241
-1 55 104 334
-1 55 137 389
-1 71 8 310
-1 71 143 445
-1 71 167 485
-1 74 94 385
-1 79 28 350
-1 79 77 385
-1 79 133 455
-1 79 217 595
-1 85 53 389
-1 85 115 451
-1 85 274 716
-1 99 94 480
-1 99 99 485
-1 101 29 445
-1 101 91 485
-1 101 106 500
-1 101 179 595
-1 101 266 740
-1 105 77 489
-1 105 129 541
-1 111 15 485
-1 111 220 690
-1 111 265 765
-1 115 91 541
-1 115 134 584
-1 120 276 809
-1 125 94 584
-1 125 151 641
-1 129 87 595
-1 129 167 675
-1 130 134 641
-1 139 85 635
-1 139 190 740
-1 146 262 865
-1 151 56 670
-1 151 251 865
-1 154 182 785
-1 155 10 676
-1 155 106 716
-1 155 199 809
-1 171 77 765
-1 171 267 955
-1 174 186 865
-1 174 274 975
-1 179 133 835
-1 179 167 865
-1 179 182 880
-1 179 218 920
-1 185 29 809
-1 185 80 826
-1 185 293 1039
-1 190 218 961
-1 200 220 1001
-1 204 228 1025
-1 211 10 920
-1 211 115 955
-1 214 190 1025
-1 215 91 959
-1 220 4 959
-1 220 28 961
-1 221 143 1015
-1 221 293 1165
-1 224 124 1015
-1 225 57 989
-1 225 85 999
-1 230 122 1039
-1 245 137 1111
-1 249 104 1110
-1 251 151 1145
-1 251 179 1165
-1 275 77 1211
-1 276 76 1215
-1 295 8 1286
-1 295 167 1339
-----------------------------------------
1 0 1 -2
1 0 1 2
1 0 17 38
1 10 9 48
1 10 111 252
1 10 273 612
1 18 7 80
1 18 257 580
1 28 39 150
1 28 249 570
1 30 31 148
1 60 7 262
1 72 191 530
1 78 1 340
1 78 199 560
1 80 159 498
1 98 225 660
1 118 297 840
1 168 193 850
1 172 9 750
1 180 199 902
1 210 193 1012
1 210 263 1088
1 228 121 1030
1 270 191 1252
1 282 73 1240
1 298 105 1320
-----------------------------------------
2 3 0 13
2 3 52 117
2 7 16 47
2 7 44 103
2 13 16 67
2 13 124 283
2 17 28 97
2 17 32 103
2 17 124 287
2 17 136 313
2 27 16 123
2 27 92 237
2 27 120 293
2 37 28 173
2 37 104 283
2 37 196 467
2 43 76 253
2 47 32 217
2 57 180 473
2 57 272 657
2 87 24 383
2 87 180 553
2 93 60 427
2 93 196 597
2 147 24 643
2 163 280 947
2 173 52 763
2 173 76 773
2 193 44 847
2 197 92 883
2 197 260 1037
2 207 16 903
2 217 52 953
2 223 44 977
2 247 244 1207
2 277 52 1213
-----------------------------------------
x y z
LIST of some orthogonal matrices, P solves P^t F P = F:
-1 0 0
0 1 0
0 0 1
1 0 0
0 -1 0
0 0 1
1 0 0
0 1 0
0 0 -1
1 0 0
0 1 0
0 0 1
1 0 0
0 9 4
0 20 9
1 0 0
0 161 72
0 360 161
39 10 10
38 11 10
190 50 49
39 110 50
38 101 46
190 530 241
39 290 130
38 299 134
190 1430 641
39 1990 890
38 1829 818
190 9590 4289
39 40 20
152 151 76
380 380 191
39 40 20
152 161 80
380 400 199
39 10 10
418 101 106
950 230 241
39 110 50
418 1211 550
950 2750 1249
39 10 10
1102 299 286
2470 670 641
170 0 39
0 1 0
741 0 170
170 780 351
0 9 4
741 3400 1530
210 140 79
76 49 28
931 620 350
210 320 151
76 119 56
931 1420 670
210 20 49
532 49 124
1501 140 350
210 20 49
1216 119 284
2869 280 670
360 250 139
494 341 190
1919 1330 740
360 530 251
494 731 346
1919 2830 1340
360 130 101
950 341 266
2641 950 740
550 480 249
228 201 104
2451 2140 1110
550 660 321
228 271 132
2451 2940 1430
550 60 129
1824 201 428
4731 520 1110
609 380 220
76 49 28
2660 1660 961
609 980 460
76 119 56
2660 4280 2009
609 320 200
1216 641 400
3800 2000 1249
609 20 140
1444 49 332
4180 140 961
759 730 370
646 619 314
3610 3470 1759
759 830 410
646 709 350
3610 3950 1951
759 220 200
836 241 220
3800 1100 1001
780 250 211
38 11 10
3401 1090 920
780 1970 899
38 101 46
3401 8590 3920
780 10 179
950 11 218
4009 50 920
780 670 349
1102 949 494
4199 3610 1880
780 950 461
1102 1339 650
4199 5110 2480
1500 310 371
494 101 122
6631 1370 1640
1500 130 349
1178 101 274
7049 610 1640
Best Answer
Edit : this is a new answer, after more computations.
Let $H$ be the subgroup of your orthogonal group that preserve globally each connected component of the (two-sheeted) space $q(x,y,z)=-1$.
Up to this action, there is a single isometry class of isotropic vectors.
One representant is $(-1\ -3\ 8)$ and its stabilizer is the infinite dihedral group generated by
The group $H$ can be described as follows : let us write
All the matrices $Ai$ and $Bi$ have order 2, while $T$ has infinite order.
Let $K$ be the free product of all the subgroups generated by these element (thus it is isomorphic to $\mathbf Z/2^{\star 11}\star \mathbf Z$.
Let $R$ be the subgroup of K generated by $[A1,A2]$ , $[A1,A3]$, $[A3,B1]$, $[A2,A4]$, $[A5,B3]$, and $T.A4.T^{-1}.A5$.
Then $R$ is the kernel of the obvious representation of $K$ in $H$, which is an epimorphism.
The group $\mathrm H_1(H,\mathbf Q)$ has dimension 1, thus $H$ is not a reflective group. I hope to be able to draw a Coxeter Diagram for its reflection subgroup soon. If this happens, I will edit this answer once more.