Whatever you think a stat called non-conference strength of schedule is meant to measure, odds are you were probably incorrect as to what it actually does signify. D3sports.com photo illustration; source images by Ronnie Glover and Pete Meshanic, d3photography.com

By Ryan Scott
D3hoops.com

It’s that time of year again. Regional rankings leading to Selection Sunday for the men’s and women’s Division III NCAA Tournament. For basic information on how these tournaments are structured and who gets in, check out the D3hoops.com Tournament FAQ page. This piece will be diving deeper into the numbers used for ranking and selection and what precisely they measure.

Most of the primary criteria that the committees use to rank teams and select at-large playoff teams are straightforward numbers: win-lost percentage, head to head competition, and results against common opponents. The men’s basketball committee has expressed a willingness to dive deeper into these numbers and explore context a bit more than the women’s committee, but ultimately they are what they are.

• NCAA Tournament FAQ
• Men's basketball conference tournament tracker
• Women's basketball conference tournament tracker

Similarly, results vs. regionally ranked opponents will be parsed ad nauseum, especially this year when the number of ranked teams has increased by 30%. The difference in relative quality between regions and between the top and bottom within regions is larger than ever, but those discussions are equally transparent. Every team’s schedule is public and available.

Where things remain pretty opaque are the calculated numbers. It has been notoriously difficult to get precise calculations from the NCAA, so we’ve often relied on outside statisticians who attempt to duplicate the NCAA’s published final numbers to reverse-engineer the formulae.

Strength of schedule (SOS) plays a significant part in both ranking and selection – and the basic calculation is public knowledge. You add 2/3rds of the winning percentage of a team’s opponents (minus the games played against the team being evaluated) to 1/3rd of the winning percentage of the opponents’ opponents’ winning percentage to establish an SOS.

The women’s committee uses this number straight up, which means there is no benefit or penalty for where the game is played. This incentivizes playing and beating good teams wherever you can schedule them. The committee may choose to value road games over home games, but that would be part of the committee discussion of head-to-head, common opponents, or contests against regionally ranked team, and not included in the numerical calculation.

The men’s committee applies multipliers to SOS components, 1.25 for road games and .75 for home games, with the intention of boosting the value of road games and reducing the value of home contests. There are further advantages and disadvantages to the multipliers based on how they’re applied.

Currently, if you play a 16-4 team on the road, both the wins and losses are multiplied by 1.25, so that team looks like 20-5 on your SOS – you get extra credit for playing that team on the road. The downside, though, is if you play a 4-16 team on the road; with the multiplier, that record becomes 5-20, further affecting your SOS in a negative direction.

The opposite is true for home games: playing a good team at home benefits your SOS less, because a 16-4 team becomes 12-3, but playing a 4-16 at home hurts you less, since they become 3-12 for the purposes of SOS.

Matt Snyder, one of the statisticians who help us understand what these calculated numbers mean and who keeps his own ranking system at tomaroonandgold.blogspot.com, sums it up this way, “You only want to play bad teams at home, never on the road.”

There are some real pros to this system — you can’t benefit just by scheduling any road game. Road contests against weak opponents actually hurt you overall. Coaches who understand how these numbers work have a real advantage in scheduling. You still need to win games, but you can balance your chance of winning with which opponents bring the most SOS advantage.

The basic formula for calculating SOS is outlined in the Pre-Championship Manual for each sport, and although there’s no explicit explanation for how the multiplier is applied, Snyder and others have been able to verify the procedure mathematically. What remains a mystery, however, is the Non-Conference Strength of Schedule (NCSOS), a secondary criteria metric that’s become a de facto tiebreaker for teams the committee cannot distinguish from one another using primary criteria.

NCSOS showed up in the criteria in 2017 and since that time, virtually the whole of Division III basketball was under the impression that it was essentially the same as SOS, but with conference opponents removed for the team being evaluated. This was touted as a means by which to negate the relative strength or weakness of a team’s conference and measure the quality of their non-conference opponents.

This was the method Snyder used to approximate this number for his site. Using Snyder’s number to evaluate the first, alphabetical list of regionally ranked teams for 2022, one decision stood out as anomalous. Guilford was included on the list for Region 6 while Oglethorpe was not. Both teams have very similar primary criteria, with Oglethorpe having a slight advantage in winning percentage.

Moving to secondary criteria, Snyder’s NCSOS calculation had Guilford and Oglethorpe still locked together, within a few hundredths of a point. The NCAA data sheet, however, put Guilford’s NCSOS number almost a full tenth higher, a massive difference for this metric and clearly the explanation for Guilford’s inclusion over Oglethorpe. However, it raised questions about what exactly the official NCSOS was actually measuring.

Through conversations with the basketball committees and the NCAA, we now understand the NCSOS to be calculated without any conference game data included at all – not for the team being evaluated, their opponents, nor their opponents’ opponents.

For example: If New England College plays against Williams, NEC’s SOS number would include a significant bump from Williams’ participation in the strong NESCAC schedule, however, NEC’s NCSOS number would not reflect any of Williams’ conference games and show only data from their non-conference schedule.

What this means in practical terms: Playing a team in a strong conference helps improve your primary criteria SOS. It only improves your secondary criteria NCSOS if that power conference opponent also schedules well out of conference.

The men’s basketball committee has been assured by the NCAA stats team that these numbers have been calculated and applied consistently since their inclusion in the criteria and across all sports, so it’s not an issue of something changing other than the general understanding of what’s being measured.

Different doesn’t mean bad, necessarily, but it’s incredibly frustrating for coaches who’ve worked to schedule based on incorrect expectations of what opaque numbers were thought to be incentivizing. Further, it decreases the benefit to teams from weaker conferences that NCSOS has been thought to provide. Scheduling a 20-4 team does very little for your NCSOS if those four losses all came out of conference.

“We need to know what the numbers mean if we’re going to schedule to improve them,” says Lycoming coach, Mike McGarvey, whose whole roster returned for this year and will remain largely intact next season, as well. “We want to put ourselves in a position to be selected, but it feels like a guessing game.”

On the positive side, calculating the NCSOS in this manner limits the amount of data that overlaps with the SOS calculation. SOS and NCSOS are measuring very different things. The question is, when it comes to selection, is a NCSOS number that only includes a team’s non-conference opponent’s non-conference data more or less helpful than one which includes your non-conference opponents’ full schedule?

In other words, is the purpose of the NCSOS to remove the potential harm/benefit that comes from playing in a particularly weak/strong conference or is the purpose of NCSOS to incentivize teams to play opponents who also play tough opponents? Neither is wrong, per say, but it’s essential that the answer be something either the sport specific or championships committee has chosen intentionally.

The current formulation doesn’t correct for the relative strength or weakness of a team’s conference – it ranks them based on how well their opponents schedule out of conference.

Even at the end of the season, NCSOS may be taking into account as few as five games for each of your opponents. Is that metric worth its place as a vital tiebreaker for rankings and selection?

Wheaton men’s basketball coach and men’s committee chair, Mike Schauer is both pragmatic and diplomatic in his response: “We can have discussions about whether we should be using different criteria or using our criteria differently, but my role is to follow the criteria as stated, so that’s what I’m going to do. As long as the numbers are applied equally to all teams, we have to use what we’re given.”

This is all predicated on our perception of what NCSOS is measuring being correct. We’ve obviously been wrong about it for five years and Snyder’s not yet been able to duplicate the numbers as proof of how it’s calculated. Our understanding of what’s being measured and how its being measured may still change moving forward.

This raises the question of who’s overseeing the process. Who has the authority to decide what’s being measured and what’s being incentivized? What is the procedure for evaluating and potentially changing calculations for criteria like multipliers or NCSOS?

Typical procedure for adding or changing a selection criterion would be for the Championships Committee, which oversees championships for all Division III sports, to poll the various sport specific committees for feedback on improvements or changes to the selection process. When they decided to adopt the NCSOS in 2017, they would have worked with the NCAA stats department to ensure they could provide a uniform metric across all sports.

It’s likely, when the NCSOS number showed up in basketball committee meetings, there was an incorrect assumption of what was being measured that was never corrected and we’ve been working with a faulty definition ever since.

No statistic is ever going to be universally embraced across 450 member institutions, let alone the various Division III sports. The committee structure is set up to make changes and improvements over time, but the real key is effectively communicating what the numbers mean and how they’re calculated so every interested party can schedule and select with the best possible knowledge.

For some of the metrics currently being used, it's not clear that that has been done.