This was posted on reddit about 15 minutes ago by Loquori (SEMC lead platform engineer), thought I’d paste it here too ~HS
(https://www.reddit.com/r/vainglorygame/comments/828sop/5v5_matchmaker_sees_some_measurable_improvements/)
5V5 Matchmaker sees some measurable improvements this past week, but we’re still tuning and improving
The 5V5 Matchmaker is almost all new and different from the 3v3 Matchmaker. Improving and tuning the 5V5 Matchmaker has been a weeks-long operation, and we’re quickly approaching the same quality of matchmaking as 3V3. We are continuing to tune and improve as we prepare for launching 5V5 Ranked mode. While the experience has not been what we wanted for a lot of people, we appreciate everyone who kept playing, rated their matches, and helped us tune and improve the new algorithm.
This process started late last year as we realized that the 3V3 algorithm was not going to be able to handle groups of 10 players. Even with only trying to create groups of six players, the Matchmaker sometimes struggled to keep up in our largest regions, especially SEA. We needed a more efficient algorithm. Our most important priority was a Matchmaker that didn’t get overloaded and crash. The best matches in the world don’t matter, if the Matchmaker is dead. After that, we wanted to balance optimizations for wait time against quality of matchups.
Prior to Update 2.12, we spent months testing and confirming the scalability of the new Matchmaker. We created large volumes of test players with a range of skill metrics and dumped them all into the Matchmaker at once. We tried different combinations of party sizes. We found crashes and fixed them. We found slow steps in the process and fixed them. All of our metrics and tuning were geared toward measuring and improving the stability and efficiency of the new Matchmaker.
When we launched 2.12 and 5V5 Early Access, the metrics confirmed our testing. We had a stable, efficient Matchmaker, but we also started getting individual reports of really bad matchups and long wait times. The bad matchups were not consistent with the intent of the algorithm; something was busted but only part of the time. The long wait times were a consequence of copying some of the tuning data from 3V3 Casual mode. So, we relaxed some criteria to reduce wait times. The only thing worse than getting a bad matchup is waiting 20 minutes to get a bad matchup. Unfortunately, as most of our metrics were around stability and efficiency, we didn’t have enough visibility into the matchups to understand why the algorithm was creating these bad groups out of the available pool of players. This investigation and the prep work for releasing Update 3.0 consumed all of the time for 2.12, and so this was our state going into 3.0.
When we launched Update 3.0 and Vainglory 5V5 to the general public, we quickly made some tuning changes to try to alleviate the pain while we continued investigation. With full availability of 5V5, the small Early Access pool size was no longer a concern. We had plenty of players, so we tightened up the criteria we had loosened for long wait times in 2.12. We also tuned some of the inputs that go into the Matchmaker about new players to account for most 5V5 players being new to 5V5 but not to Vainglory. Tuning efforts are ongoing with updates happening every few days as we review metrics.
As part of Update 3.0, we included some additional metrics that we couldn’t get into 2.12 safely. This gave us two very important pieces of information: One clue was from player reports that skill tiers were all mixed up in a single match. We expected some amount of this as your 5V5 Casual skill metrics are seeded from a variety of sources and only in part from your 3V3 Ranked MMR, which is itself only somewhat related to your skill tier. With that much error, it didn’t surprise us that players from very different skill tiers were being mixed into the same matches, but we now had data that showed that the 5V5 Casual skill metrics were indeed seeded poorly. This was an oversight on my part. We focused too heavily on the stability and not enough on verifying the correctness of the information the Matchmaker was using to create groups. Just after the first weekend of 3.0, we went through and recreated everyone’s Casual skill metrics.
At the same time, we also had enough data to show that even had the skill metrics correctly represented the actual skill of our players, the Matchmaker was creating bad matchups. The win chance of most pairings was within a 60/40 rough tolerance, but some were terrible with 80/20 splits. So, while one engineer was working on fixing skill metrics, another engineer was replaying bad matchups harvested from Live and trying to reproduce the bad decisions in the Matchmaker. Through that investigation, we learned that the algorithm had two critical defects, which together would cause the two teams to drift apart as we added more players to each team, instead of drawing closer. The fix for that was rolled out during Week 2 of 3.0. The result was that we saw win chances in 5V5 come in line with 3v3.
While win chance is a very important measure of the quality of the Matchmaker, it’s not the only one we care about. In addition to fair teams, we also want teams that are as close together in skill as possible, even within a team. In 5V5, we are seeing more spread in the parties that join the Matchmaker than we did with 3V3. So, as a result, the Matchmaker has been doing its best to compensate by building opposing teams that match the spread in these parties. This is one key difference between the 3V3 Matchmaker and the 5V5 one. The 3V3 Matchmaker considers a party to be roughly one homogenous unit, but the 5V5 one considers more of the individual variations between the different party members. So, the 3V3 Matchmaker will often take a widely spread party and match it against a tightly grouped bunch of players as long as they are all within an acceptable skill range, but the 5V5 Matchmaker will try to create teams that look like each other as much as possible, including wide skill range spreads, if necessary. Currently, this is creating more spread within teams than we think is healthy. We are continuing to look at ways we can improve this without creating unbalanced teams.
So, we hit our initial goal of building a stable and scalable Matchmaker for 5V5. We’ve improved our ability to observe the Matchmaker’s decisions. We’ve fixed a bunch of bad data. We’ve done a lot of fine tuning. We are very close to comparable quality with the 3V3 Matchmaker in almost all metrics, except for spread within teams, and that’s our current priority. The word on the street that we’ve heard is that the Matchmaking for most people feels significantly better than the days right after 5V5 landed. There’s still a little ways to go before I’m willing to sign off on this as our Matchmaker for 5V5 Ranked, but I’m confident that we can get there.
The best way all of you can help is to rate the Matchmaker at the end of the match. After every tuning iteration and fix, one of the key metrics I look at to determine to keep the change or throw it out is to compare the ratio of high star to low star ratings. I can look at groups of similar matches with low ratings, see how much impact the Matchmaker had in creating that sad experience, observe how the Matchmaker made its decisions in those cases, and tune and tweak it to make better decisions in the future. The star rating is a key tool for us. So, please rate your matches, and I’ll see you on the Rise … just as soon as I make one more tweak.
Thanks,
Loquori
Lead Platform Engineer, Super Evil Megacorp