A while back I posted a very long analysis about judging practices in bull riding. I mentioned various schemes that are used in bull riding by different organizations. To recap,
There are other schemes in which the high and low score is dropped or an outlying score is dropped. Dropping a score from the high or low side of the score distribution obviously adds to the effect of suppressing extreme high and low scores by moving all scores closer to the mean or average. This is an important point to remember - the primary effect of adding judges, adding precision or dropping extreme scores is they all move the resulting final score closer to the mean of the distribution of judges scores. This is what I mean when I say systems that favor the mean.
So let's dig a little deeper into judging schemes and their effects. The PBR recently introduced a novel one. How does it work and what effects is it going to have?
After the Winston-Salem PBR Teams event, PBR implemented a new scheme apparently aimed at raising overall scores. At the Team events, they have been using 4 judges for scoring, and they have 2 extra for rules/fouls/times/review. They decided that starting in Oklahoma City, they would have all 6 of these judges score the rides and bulls, and drop the lowest 2 on each out. In the previous essay I mentioned that most schemes that
tend to favor the mean, or move final scores closer to the average of all the judges. The PBR's new scheme does this PLUS it shifts the mean upwards.
In practice, dropping the 2 lowest judges every time will result in higher scores, yes. But it is bound to have secondary effects as well. Judging is subjective, and cannot really be 100% systematic. Over time, scores are cyclical. You may see a 92 point ride, then ten years later a nearly identical ride will be scored 90 or possibly 96. The sport of bull riding has seen inflated scores a lot, because management, marketing and production in just about every governing organization always want higher scores, and they have power to hire judges that will score higher or pass rules that will push scores higher. Thus there was an event in 2012 where 44% of the qualified rides were 90 points or higher. There were reride options awarded for bulls scoring 43 points out of 50.
There's a psychological angle to this too. Judges are hired labor - usually independent, and typically do not have a great deal of power in the hiring organization. They are also human beings, and all of them understand that they are way more likely to get fired for scoring in too low a range than scoring too high. Before any rules or schemes, there's always a mild amount of pressure to keep scores up.
Over the past few decades, the PBR has done a decent job of managing this relationship between pro-level judges and the organization. For this to work well there needs to be a balance between marketing / production and the competition side of the sport. Someone will always be "judging the judges" and at PBR this management role has been left to the competition department more often than not. The competition department has had power at PBR as well. This relatively stable balance of power has resulted in the best overall judging in the history of the sport. The highest level of PBR competition has been the gold standard in bull riding for a long time.
The introduction of Team competition and recent changes at PBR have had some effect on this balance of power. Marketing and production interests have increased in number. Each team has their own management, and they all have some interest in seeing higher scores. They are also all biased. So at the moment, marketing and production have more influence on judging and competition than they did a year or two ago, AND that influence contains more bias as well. Thus, we get a rule / scheme that is guaranteed to result in higher (possibly to the point of inflated) scores.
As an aside, individual score values have less impact on the Teams competition than any other competition format. The Team who stays on the most bulls will always win the game. PBR has done an outstanding job of setting even pens for the Team games so far, and so long as that is the case, every Team has an equal opportunity to dominate by just staying on for 8 seconds more often. Judging is less of a factor in Teams competition, period.
So the new scheme of dropping the two lowest judges will result in higher scores, yes. It does introduce bias into a system that was already working pretty well. It doesn't necessarily solve any problems long term. Each team will still be biased in that they want higher scores for themselves - but now the scores they want are 2 points closer to 100 than the scores they wanted 3 weeks ago.
It sends a clear message to judges that they should definitely try not to be the lowest guy, and we can already see the effect of this. At Ridgedale, MO there were 129 scored outs, and the group of judges there included 3 guys who have a ton of experience and arguably a high level of job security alongside 3 others who have less experience and job security. For each out there are two judges whose scores get dropped and do not count toward the official score. So how did this shake out? If we split the judging crew in half by experience,
most experienced
This is not necessarily the judges consciously making a decision to score higher. If you are one of the less experienced judges here and you want to get hired to judge more of these what would you do? The organization is sending a clear message that they value judges who score higher over those who score lower, and this message is going to get through to the judges and affect their scores. Rides that would have been 86 points are probably going to be 88 now, etc.
Is this going to ruin the sport? No. But, if the PBR's goal is to have the most competent and accurate judging possible, they are sidestepping that with this scheme. They have always leaned towards judging accuracy, and have seen good results. This is leaning toward higher scores, and will necessarily result in a higher incidence of inflated scores than usual.
