LZ's progression

And · **#261**

gogui-twogtp V1.4.9, leela-zero-0.15-cpuonly 20 games ok. I previously unpacked the network. I have a weak video card now, but I'll try to run the gpu version

And · **#262**

I started the gpu version (using sabaki) with f438268e - it plays normally, Ms5 - it works, but it makes stupid moves. It seems that Ms5 has problems with gpu

And · **#263**

40 games f438268e - Ms5 28-12 (10 sec/move, no pondering, LZ0.15-cpuonly, twogtp V1.4.9)

And · **#264**

60 more games, total
100 games f438268e - Ms5 63:37 (10 sec/move, no pondering, LZ0.15-cpuonly, twogtp V1.4.9)
Ms5 disappointed

Vargo · **#265**

I heard networks bjiyxo f438268 (192x15) and LM_GX89 (192x15) were very strong. OK, I added two players : LZ_Elfv1 (224x290) and LZ#202 (256x40) to run a 4 players tournament.

Short time parity (2 min/game per player , with gpu:1x1080) no pondering , komi 7.5, every player playing 10 games against every other one.

Results :

202-Elf 3-7
202-f43 5-5
202-x89 2-8
x89-Elf 5-5
x89-f43 7-3
Elf-f43 5-5

I'm very much surprised by the poor result of #202 (too little time for 40b ?) and by the excellent result of GX89.

It's about 3h for the 60 matches. I'll try to run the same thing tomorrow, to see if I get similar results. Maybe (probably) I should let it run for a lot more games.

Complete tournament code :

Code:

rem 202-elf

gogui-twogtp -black "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\202.gz --noponder" -white "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\elf2.txt --noponder" -games 10 -sgffile 202elf -time 2 -auto -komi 7.5 -alternate

gogui-twogtp -analyze 202elf.dat

rem  202-f43

gogui-twogtp -black "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\202.gz --noponder" -white "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\f438268.gz --noponder" -games 10 -sgffile 202f43 -time 2 -auto -komi 7.5 -alternate

gogui-twogtp -analyze 202f43.dat

rem 202-X89

gogui-twogtp -black "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\202.gz --noponder" -white "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\LM_GX89.txt --noponder" -games 10 -sgffile 202x89 -time 2 -auto -komi 7.5 -alternate

gogui-twogtp -analyze 202x89.dat

rem X89-elf

gogui-twogtp -black "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\LM_GX89.txt --noponder" -white "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\elf2.txt --noponder" -games 10 -sgffile x89elf -time 2 -auto -komi 7.5 -alternate

gogui-twogtp -analyze x89elf.dat

rem X89-f43

gogui-twogtp -black "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\LM_GX89.txt --noponder" -white "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\f438268.gz --noponder" -games 10 -sgffile x89f43 -time 2 -auto -komi 7.5 -alternate

gogui-twogtp -analyze x89f43.dat

rem elf-f43

gogui-twogtp -black "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\elf2.txt --noponder" -white "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\f438268.gz --noponder" -games 10 -sgffile elff43 -time 2 -auto -komi 7.5 -alternate

gogui-twogtp -analyze elff43.dat

Attachment:

all the stats.zip [32.08 KiB]
Downloaded 560 times

Attachment:

60 games.zip [49.23 KiB]
Downloaded 530 times

Mike Novack · **#266**

Vargo wrote:

Short time parity (2 min/game per player , with gpu:1x1080) no pondering , komi 7.5, every player playing 10 games against every other one.

......

I'm very much surprised by the poor result of #202 (too little time for 40b ?) and by the excellent result of GX89.

I think this sort of thing requires REASONABLE time controls. Nothing faster than human blitz play. Saying that this irrelevant, that all that matters is equality, misses an important point.

Think of a race. Which is faster, a human or a race car?

a) Down a quarter mile straightway
b) Ten times around two pylons 50 feet apart

splee99 · **#267**

Well, a "long game" on my computer (GTX1050) with 20 seconds per move maybe just same as the "2 minutes" game on a GTX1080. #202 lost again to LM-GX89. Sometimes it seems bigger doesn't mean better in the AI world.

lichigo · **#268**

Is it possible to use the gx-89 with lizzie?

Uberdude · **#269**

lichigo wrote:

Is it possible to use the gx-89 with lizzie?

Should be, I've used GX88 with Lizzie.

lichigo · **#270**

Oh great, could you share the link. I would like to try it ^^

Vargo · **#271**

lichigo wrote:

could you share the link

HERE

Mike Novack wrote:

this sort of thing requires REASONABLE time controls

With 2min/game and good gpu, these networks are already (very) strong, I don't see why it's not reasonable.

I've let it run for 6x20 more games.

After 6x30 games, results are :

202-Elf 12-18
202-f43 17-13
202-x89 12-18
x89-Elf 12-18
x89-f43 14-16
Elf-f43 17-13

I'll let it run up to 6x50 games, results in the evening.
_________________________________________________________

Assigning arbitrarily 4000 Elo to each network at the start of the tournament, after 30 games, I get

1)Elf_ v1 4047
2)LM_GX89 3994
3)f438268 3981
4)LZ_#202 3978

(and yes, I know it's questionable ;-)

)

And · **#272**

when using gpu 1080, these networks play very strongly and at 2 min/game. even using cpu only, will many be able to defeat them?
but the results are different if you use only cpu, for example:

100 games f438268e - elf.v1 62:38 (5 sec/move, no pondering, LZ0.15-cpuonly, twogtp V1.4.9)

for an elf there is not enough computational power, it is probably better to take a 15 block network for a CPU only (with little time control)
that is, when the strength of the networks does not differ much, the results of the matches depend not only on the number of games (if there are few of them, then the result is doubtful), but also from time to move + power gpu / cpu

Vargo · **#273**

After 6x50 games, results are :

202-Elf 21-29
202-f43 21-29
202-x89 28-22
x89-Elf 22-28
x89-f43 26-24
Elf-f43 27-23

A little oddity : usually, all games are by resignation, but there was one won by 1.5 after... 423 moves

After 6x50 games, starting with 4000 for each network, I get relative Elos :

1)Elf_ v1 4034
2)LM_GX89 4003
3)f438268 3985
4)LZ_#202 3978
but 6x50 games is too few to be really meaningful.

Mike Novack · **#274**

Maybe I should make clearer what I am saying.

The whole point of this is determining the question is A stronger than B. It is entirely possible that A runs on less resources than B. In other words, at short enough time control, A "has enough to be strong" while B does not. However while A would be stronger given more time, the rate at which it gets stronger might be less than the rate at which B gets stronger with more time.

That is why am suggesting should be reasonable time controls (no faster than human blitz)

There is ANOTHER question that we could be asking. There is trade off between strength of hardware and time. We MIGHT be asking "what is the strongest program, playing at reasonable time controls, but on weaker hardware". THAT we can model at shorter time controls (like being used here) on this powerful hardware. The "weaker hardware" question IS important to many of us. We just need to be clear about what question we are asking.

Vargo · **#275**

Hardware, parameters and time control are all reasonable, and you know it. Other tests with weaker or stronger hardware, with different networks, or different time controls are obviously possible. Feel free to show us your results, it would be interesting.

Satorian · **#276**

Of course everyone is free to do as desired with their own processing power and time, but I agree with Mike that a super short playing/move time has a bias towards smaller/quicker networks by allowing them more playouts within a generally low-playout regime, while those networks would plateau much quicker with longer, more reasonable time controls. I also don't find these super short time controls very interesting. Using an engine for review I would probably give it at least 30-60 minutes total or 10-15s per move, same for playing against it.

Uberdude · **#277**

How many playouts a move is 40 block LZ getting in these 2 minute test games on a 1080 Ti? If it's at least 3k (what I get in 20 seconds on my 1060, and would take 5 minutes on my crappy laptop) then I think that's about enough for a test where the deeper network can show its strength and not be handicapped by being unable to do basic reading (there was a nice chart on the LZ github showing this I linked a while back).

Satorian · **#278**

At 120 seconds per player, I would expect this to be 1s per move or less. I don't think the 1080 Ti outperforms the 1060 at 20:1 to allow the 40b network to reach 3k playouts on a consistent basis. Perhaps Vargo could provide some stats on average/median/min/max playout count and time per move.

And · **#279**

Anyone with a good gpu can test the minigo network?
2019-01-10 12:49 50b98433(40x256) VS d13c4099(elf 20x224) 266 : 134 (66.50%) - elf 33.5%
2019-01-24 03:56 b9efeafa(minigo 19x256) VS ffe8ba44(40x256) 30 : 53 (36.14%) - minigo 36.14%
http://zero.sjeng.org

Minigo v15-990 was stronger than ELFv1 at 10s and 120s (10s/ 120s was at least 10k/ 120k-po on ELFv1)/ a move.
https://github.com/gcp/leela-zero/issues/2179

Uberdude · **#280**

Satorian wrote:

At 120 seconds per player, I would expect this to be 1s per move or less. I don't think the 1080 Ti outperforms the 1060 at 20:1 to allow the 40b network to reach 3k playouts on a consistent basis. Perhaps Vargo could provide some stats on average/median/min/max playout count and time per move.

I asked a similar question a while back so we can estimate it:

Uberdude wrote:

Vargo wrote:

20 game match with GPU 1x1080 , 5 min per side and per game, no pondering.

#184 v. ELF v1
ELF wins 14 : 6 (8 wins as B, 6 wins as W)

So that's about 800 playouts for #184 and 1600 for Elfv1?

Vargo wrote:

Uberdude wrote:

So that's about 800 playouts for #184 and 1600 for Elfv1?

Something like 800 visits for #184 and 2000 visits for Elfv1

So that was 5 mins per side per game, these test are 1 minute. 184 is a 40 block so (assuming there isn't an LZ 0.16 fp16 speedup between the two tests, and there may be) 800/5 = 160 so a lot short of the several thousand I like to give the 40 blocks to show their strength. If there is fp16 difference that'll help at most a factor of 2.

LZ's progression

Who is online