Life In 19x19
http://prod.lifein19x19.com/

Playing a bot without proper handicap
http://prod.lifein19x19.com/viewtopic.php?f=18&t=11606		 Page 1 of 1
Author:  Pippen [ Sun Mar 15, 2015 9:09 am ]
Post subject:  Playing a bot without proper handicap
Yesterday I did beat my first KGS-3d player ever in an even game: a bot. After reviewing the game I came to wonder if I can really be proud of such a win or if I took advantage of playing a 3d bot without proper handicap. Does a bot play with full strength in an even game, regardless of his opponent, or does he take the rank of a player into account, so that e.g. if I play it as a 1d in an even game it will play "as if" having a 2 stone handicap? If you'll look at the game you will see how sloppy the bot played at times, that's what made me think.
Author:  tentano [ Sun Mar 15, 2015 9:57 am ]
Post subject:  Re: Playing a bot without proper handicap
Uh, I dunno, if that's really 3d, I need to upgrade my self-assessment.

For a large part, you were winning purely on reading. Nothing fancy, just reading more solidly.

By the end you have 50+ captured stones, which is really quite silly for an even game against someone two ranks stronger. Even sillier when you didn't give away anything anywhere near that big in exchange.

In fact, the only thing you really threw out was the five stones in the top left corner. It's a pretty overwhelming victory.

I don't think any AI is made to take a rank into account, though. It would open them up to abuse if the stated rank of an opponent was a factor. You didn't take any advantage, instead you took a disadvantage, which should have yielded a far more difficult game.

I really suspect that this bot might have had some sort of problem. I guess if the machine it was running on was also doing something else at the time, it would be unable to deliver full power within the same time constraints. The rating graph for that specific bot doesn't seem to indicate instability. Or maybe you hit one of its weak spots?
Author:  Bantari [ Sun Mar 15, 2015 7:38 pm ]
Post subject:  Re: Playing a bot without proper handicap
I have commented on that before, but I don't think my comments were accepted gracefully.

The point is - I cannot imagine a 3d who would play like the bot played in that game. Maybe if the 3d was drunk... but then - bots don't get drunk.

I really suspect most of the bot ranks on KGS and other servers. I have played a few of them myself, and this just confirmed my conclusion. Are the bot ranks only hype, marketing gimmick?
Author:  xed_over [ Sun Mar 15, 2015 9:46 pm ]
Post subject:  Re: Playing a bot without proper handicap
Bantari wrote:
I really suspect most of the bot ranks on KGS and other servers. I have played a few of them myself, and this just confirmed my conclusion. Are the bot ranks only hype, marketing gimmick?

Ranks are a result of their win/loss records of games played. If the bot is not deserving of a 3d rank, then I suspect it speaks more to its opponents, than to the bot itself.
Author:  Mike Novack [ Mon Mar 16, 2015 8:14 am ]
Post subject:  Re: Playing a bot without proper handicap
You need to consider that a bot might not always be running on its usual hardware or that the hardware at some moment might be running something else (so the bot has less of the machine).

In other words, the ranking of these bots might be the result of the average situation.

Some of the bot "owners" might take their bot down during times the full regular machine isn't available but others might (temporarily) be substituting less powerful hardware. Probably none of these bot "owners" can afford dedicating hardware to be just running the bot for us.

I think we'd need to be asking the individual bot owner/teams.
Author:  Uberdude [ Mon Mar 16, 2015 8:54 am ]
Post subject:  Re: Playing a bot without proper handicap
xed_over wrote:
Bantari wrote:
I really suspect most of the bot ranks on KGS and other servers. I have played a few of them myself, and this just confirmed my conclusion. Are the bot ranks only hype, marketing gimmick?

Ranks are a result of their win/loss records of games played. If the bot is not deserving of a 3d rank, then I suspect it speaks more to its opponents, than to the bot itself.


I've lost to AyaMC5 (also 3d, dunno if parameters/hardware different) when playing silly blitz. Usually I kill some stones in a semeai and get bored and then it plays some gote moves in the semeai and I tenuki, then it plays some sente moves in the semeai and I still tenuki :D .
Author:  Pippen [ Mon Mar 16, 2015 10:25 am ]
Post subject:  Re: Playing a bot without proper handicap
Mike Novack wrote:
You need to consider that a bot might not always be running on its usual hardware or that the hardware at some moment might be running something else (so the bot has less of the machine).

In other words, the ranking of these bots might be the result of the average situation.

Some of the bot "owners" might take their bot down during times the full regular machine isn't available but others might (temporarily) be substituting less powerful hardware. Probably none of these bot "owners" can afford dedicating hardware to be just running the bot for us.

I think we'd need to be asking the individual bot owner/teams.


Well, of course also humans have some up and down's. I am in peace with the answer that bots do not take the actual rank of its opponent into account when playing. My theory is that Aya got confused with the Ko-war in the middle of the game, wouldn't be the first time that happens to a machine^^.
Author:  quantumf [ Mon Mar 16, 2015 11:19 am ]
Post subject:  Re: Playing a bot without proper handicap
Mike Novack wrote:
You need to consider that a bot might not always be running on its usual hardware or that the hardware at some moment might be running something else (so the bot has less of the machine).

In other words, the ranking of these bots might be the result of the average situation.

Some of the bot "owners" might take their bot down during times the full regular machine isn't available but others might (temporarily) be substituting less powerful hardware. Probably none of these bot "owners" can afford dedicating hardware to be just running the bot for us.

I think we'd need to be asking the individual bot owner/teams.


Possibly.

I believe the answer is simple, that bots always play to their strength, while humans frequently do not (see Uberdude's comment). This accounts for at least one stone, perhaps 2 or 3 (at least, around the 3k-3d range).
Author:  skydyr [ Mon Mar 16, 2015 12:20 pm ]
Post subject:  Re: Playing a bot without proper handicap
Pippen wrote:
Mike Novack wrote:
You need to consider that a bot might not always be running on its usual hardware or that the hardware at some moment might be running something else (so the bot has less of the machine).

In other words, the ranking of these bots might be the result of the average situation.

Some of the bot "owners" might take their bot down during times the full regular machine isn't available but others might (temporarily) be substituting less powerful hardware. Probably none of these bot "owners" can afford dedicating hardware to be just running the bot for us.

I think we'd need to be asking the individual bot owner/teams.


Well, of course also humans have some up and down's. I am in peace with the answer that bots do not take the actual rank of its opponent into account when playing. My theory is that Aya got confused with the Ko-war in the middle of the game, wouldn't be the first time that happens to a machine^^.


I don't know. The play in the lower right and center early on by the bot seems kyuish to me, to say the least, despite the strength white builds outside.
Author:  topazg [ Mon Mar 16, 2015 1:47 pm ]
Post subject:  Re: Playing a bot without proper handicap
skydyr wrote:
I don't know. The play in the lower right and center early on by the bot seems kyuish to me, to say the least, despite the strength white builds outside.


Welcome to bot land. I've played chess engines around the 1500-1600 ELO mark who suddenly spot a complicated mate in 8 and boom. You can't expect a 3d bot to play like a 3d human.
Author:  emeraldemon [ Mon Mar 16, 2015 2:51 pm ]
Post subject:  Re: Playing a bot without proper handicap
I did a little hunting on kgs.gosquares.net (http://kgs.gosquares.net/index.rhtml.en?id=ayamc3&r=1), found some kinda interesting things.

1) ayamc3 often gets repeat opponents, sometimes quite often: go4funonly[2d] has played 189 games against the bot, with a 43% win rate. Many examples are less extreme, but I definitely notice a trend of humans wanting to play till they win, losing a few but then beating Aya and then stopping.

Just looking at a sampling of Aya's opponents from the last few months:
test123[2d] has gone 0-33 against Aya, and lost a lot against several other bots besides. Is this intentional feeding?
miyabi[2d] is 1-8 against aya
pascala[4k] is 26-34 (43%) taking 6 stones mostly
floatcloud[2d] is 9-12
xz[1k] is 19-26 (42%) taking 3 stones

I don't know if you can draw any trend from this, but I wouldn't be surprised if humans approached bot games differently.

2) Of the 41372 rated games aya has played, only 8085 (about 20%) have been against higher-rated opponents. We could speculate about why mostly weaker players challenge Aya, but it seems to offer a boost: it has a 68% win rate against lower-rated opponents, compared to a 48% win rate against higher-rated opponents. 46% of its games are giving a handicap.

This is pure speculation, but I personally think the monte-carlo evaluation style is good for playing against handicap: in the beginning it will evaluate the board, decide it has a low chance of winning, and pick complicated variations. If it pulls ahead it will gradually start picking safer moves. Obviously it isn't perfect, but it's an approach not too different from how humans play handicap games.
Author:  Mike Novack [ Mon Mar 16, 2015 3:08 pm ]
Post subject:  Re: Playing a bot without proper handicap
emeraldemon wrote:

This is pure speculation, but I personally think the monte-carlo evaluation style is good for playing against handicap: in the beginning it will evaluate the board, decide it has a low chance of winning, and pick complicated variations. If it pulls ahead it will gradually start picking safer moves. Obviously it isn't perfect, but it's an approach not too different from how humans play handicap games.


Not how it's done. The MCTS algorithm works very poorly if the probability of winning the game is very low, as it would be at the start of a handicap game were both players playing equally. Equally badly I mean, since the playouts don't assume good play, just equal play.

So how is it done? With adjustable "added komi" which is adjusted downward as the game progresses. In other words, komi that is invisible to you but used by the evaluator to decide which side won that playout. The people who design these programs have to do much experimentation to determine the proper size of this komi and its decay rate to get the programs to play properly against various handicaps.

Remember, the MCTS evaluator is doing only ONE thing, selecting the move that came out on top when the playout tree was scored. It is NOT "thinking", not picking a strategy, etc. That the best thing to do is play wildly if behind and steadily if ahead is a consequence of "best chance not to lose the game".
Page 1 of 1 All times are UTC - 8 hours [ DST ]
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/