Life In 19x19
http://prod.lifein19x19.com/

katago Handicap games
http://prod.lifein19x19.com/viewtopic.php?f=18&t=17219		 Page 1 of 1
Author:  Vargo [ Tue Jan 21, 2020 9:27 am ]
Post subject:  katago Handicap games
Katago 1.3.1 wins every time at H9 against GnuGo (~5k), so, I tried higher handicaps.
At H10, it won by 20+
At H11 it won by resignation
At H12 it won by 2
at H13 (!) it lost by 10, then lost by 15, and won the 3rd game (below)

(Sabaki 0.43.3, laptop with GTX 1660Ti, komi=0, time_settings 0 7 1 (7sec/move) network=g170-b20c256x2-s1039565568-d285739972.txt.gz )

Against CrazyStoneDL (set at 5 dan)
it won at H6, H7, and H8 (below)
woaw !!!

Attachments:
CS5d_H8_kata131_7s.sgf [2.98 KiB]
Downloaded 2331 times
gnu_k131H13_Kw.sgf [2.91 KiB]
Downloaded 2311 times
Author:  Vargo [ Wed Jan 22, 2020 1:49 am ]
Post subject:  Re: katago Handicap games
Same kind of results with gtp4zen7
Sabaki 0.43.3, laptop with GTX 1660Ti, komi=0
time_settings 0 7 1 (7sec/move) network=g170-b20c256x2-s1039565568-d285739972.txt.gz for katago 1.3.1

According to this site , gtp4zen parameters are :

zen7 5K : -t1 -r5 -s800 -n1 -o2.4 -p0.3
zen7 1D : -t1 -s1800 -n1 -o1.3 -p0.35 (and -r5)
zen7 5D : -t1 -s2700 -n2 -o1.0 -p0.55 (and -r5)

with these settings,
zen5k loses by resignation at H12
zen1d loses by resignation at H9
zen5d loses by resignation at H8

The games :
I'll try higher handicaps, and upload lost games too.

Attachments:
k131z71DH9.sgf [2.63 KiB]
Downloaded 2314 times
k131z7_5kH12.sgf [2.29 KiB]
Downloaded 2258 times
k131z75DH8.sgf [3.58 KiB]
Downloaded 2291 times
Author:  Vargo [ Wed Jan 22, 2020 7:37 am ]
Post subject:  Re: katago Handicap games
Two handicap games with free placement of the handicap stones :
Gnugo plays n consecutive moves before the first white move.

GnuGo seems stronger if it can choose the places for the handicap stones.
At H12, GnuGo wins 3 times, with katago resigning (too?) soon. So I changed the resignThreshold to -0.9999
But katago lost this game :

At H10, with free placement by GnuGo, katago wins :


Attachments:
gnu_k131H10_Kwins15.sgf [2.77 KiB]
Downloaded 2278 times
gnu_k131H12_Gwins13.sgf [3.26 KiB]
Downloaded 2230 times
Author:  Vargo [ Fri Jan 24, 2020 6:48 am ]
Post subject:  Re: katago Handicap games
After several 20-game matches at very high handicap (katago v1.3.1 v. Gnugo 3.8)
I find the results very surprising...
For example, at H17 (!) Katago wins ~20% of the games (twogtp 1.5.1, 3sec/move and resignThreshold = -0.9999 for Katago, desktop with GTX 1080)
Here is one :

At H15, on a laptop, katago wins about 50% of its games on Sabaki (7 sec/move with a 1660 Ti)

Is Katago absolutely incredible, is Gnugo getting weaker, or is it me, doing something stupid :scratch: ?
Could someone please try it, thank you.

If you want the same places as I had for the handicap stones, download h15.sgf
1) In Sabaki, "File", "Open..." (h15.sgf)
2) "Engines" "Attach..." , Gnugo for B, Katago for W , and "OK"

Attachments:
h15.sgf [126 Bytes]
Downloaded 625 times
kata131gnu_h17-11.sgf [1.87 KiB]
Downloaded 2135 times
Author:  gennan [ Sat Jan 25, 2020 4:13 am ]
Post subject:  Re: katago Handicap games
If GnuGo is 5k EGF, then 15 stones handicap for 50% looks quite reasonable to me. It suggests KataGo is 10-11d EGF, meaning it should be able to give strong pros 2-3 stones handicap, which is consistent with actual game results of strong pros vs strong AI.

I'm even pleasently surprised by this consistency.
Author:  gennan [ Sat Jan 25, 2020 4:35 am ]
Post subject:  Re: katago Handicap games
Experimenting with KataGo 1.3.1, I updated my table of point values of handicaps (without komi). (previous instance: https://lifein19x19.com/viewtopic.php?f=15&t=16992&p=249931&hilit=handicap#p249931)
Also in this respect, I'm pleasantly surprised by the improved linearity even including higher than 9 stones handicap.

Code:
Handicap  Estimated  Ideal
1           6          7
2          20         21
3          34         35
4          48         49
5          60         63
6          74         77
7          87         91
8         103        105
9         115        119
10        127        133
11        139        147
12        153        161
13        170        175
14        186        189
15        206        203
16        219        217
17        222        232

The estimated points value is based on KataGo's evaluation.
The ideal points value is based on the assumption that perfect komi is 7 points.
I used traditional fixed handicap placement and for handicaps higher than 9 I used 3-3 points for handicaps 10-13 and 10-3 points for handicaps 14-17.

The discrepancy from perfect linearity is less than 6% (except for the 1st entry, but there it's only 1 point).

To me this suggests there exists a nice and simple probability distribution of errors made by players of a specific rank. This might be useful to create a nice statistical model for rating systems based on ranks. It may even be possible to estimate a player's rank by analyzing a few of his games to find his typical error distribution.

The anchor of such a rating system would be perfect play (no errors). We don't know exactly which rank that is, but if we assume that "God" is barely able to give strong AI a 2 stone handicap, she would be about 12d-13d EGF (EGF rating 3250).

Actually I've been working on such a statisical model lately.
Author:  Vargo [ Tue Jan 28, 2020 3:35 am ]
Post subject:  Re: katago Handicap games
According to this site, there are several patterns for very large handicaps.
With the Japanese pattern, Katago can win some games up to 20 stones :o against Gnugo.
At 21 stones , even with different parameters, I've not seen it win.

Katago 1.3.1 , Gnugo 3.8 , Gogui-twogtp 1.5.1, maxVisits=1000 and resignThreshold=-0.999 for Katatgo

Two games won by Katago :

H19 :
H20 :


Attachments:
kata131gnu_h20japan-14.sgf [1.94 KiB]
Downloaded 2037 times
kata131gnu_h19japan-1.sgf [2.2 KiB]
Downloaded 2039 times
Author:  moha [ Tue Jan 28, 2020 6:57 am ]
Post subject:  Re: katago Handicap games
gennan wrote:
To me this suggests there exists a nice and simple probability distribution of errors made by players of a specific rank. This might be useful to create a nice statistical model for rating systems based on ranks. It may even be possible to estimate a player's rank by analyzing a few of his games to find his typical error distribution.

The anchor of such a rating system would be perfect play (no errors). We don't know exactly which rank that is, but if we assume that "God" is barely able to give strong AI a 2 stone handicap, she would be about 12d-13d EGF (EGF rating 3250).

You may also be interested in this, your idea sounds similar.
Author:  And [ Tue Jan 28, 2020 9:04 am ]
Post subject:  Re: katago Handicap games
Katago 1.3.1 "g170 20 block s1.42G" (maxVisits=300, resignThreshold=-0.999) & AmiGo1.8 (~11k) H25:

https://sourceforge.net/projects/amigogtp/

Attachments:
amigo - katago.sgf [1.83 KiB]
Downloaded 2050 times
Author:  Vargo [ Sun Feb 23, 2020 2:26 am ]
Post subject:  Re: katago Handicap games
A chain of H9 games (komi 0.5, Sabaki 0.43.3)

KataGo 1.3.2 (16k visits, 20bs191) wins against KataGo 1.3.2 (1 playout, 20bs191) W+R

KataGo 1.3.2 (1 playout, 20bs191) wins against GnuGo 3.8 W+20.5

GnuGo 3.8 wins against AmiGo W+194.5
And maybe a very weak bot would lose at H9 against Amigo ;-)

Attachments:
kata132_16kv_kata132_1pH9.sgf [2.48 KiB]
Downloaded 1861 times
gnu_kata132H9.sgf [1.56 KiB]
Downloaded 1854 times
GnuGo-AmiGoH9.sgf [1.94 KiB]
Downloaded 1834 times
Author:  And [ Sun Feb 23, 2020 9:15 am ]
Post subject:  Re: katago Handicap games
Vargo wrote:
And maybe a very weak bot would lose at H9 against Amigo ;-)

may be LZ 008 0.1k? (https://github.com/breakwa11/GoAIRatings), but against him LZ 004 0.01k?
Author:  And [ Thu Apr 09, 2020 10:04 am ]
Post subject:  Re: katago Handicap games
Katago 1.3.5 "g170 30 block s2.27G" (maxVisits=50, resignThreshold=-0.999) & AmiGo1.8 (~11k) H41:


Attachments:
Katago & AmiGo.sgf [1.93 KiB]
Downloaded 1708 times
Author:  And [ Sat Apr 11, 2020 5:18 am ]
Post subject:  Re: katago Handicap games
It is interesting that KataGo networks 15, 20 and 40 blocks do not want to play with such a handicap H41. pass and immediately or after a certain number of moves because of this lose. 30 block network somehow differently "sees"
Page 1 of 1 All times are UTC - 8 hours [ DST ]
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/