BenchmarkersPareto Frontier

Pareto Frontier Mechanism

TIG employs a Pareto Frontier mechanism that determines which solutions qualify for rewards. The Pareto Frontier evolves so that as more compute enters the TIG network, we target larger and more difficult instances of the challenge. This mechanism is employed per block and for each challenge individually.

Pareto Dominance and the Pareto Front

Pareto dominance is a concept used to compare solutions across multiple objectives or dimensions. A solution AA is said to “Pareto dominate” a solution BB if:

  • Solution AA is at least as good as solution BB in all objectives/dimensions.
  • Solution AA is strictly better than solution BB in at least one objective/dimension.

In other words, AA dominates BB if AA is better or equal in every way, and actually better in at least one way.

Given a set of solutions, the Pareto frontier is the subset of those solutions that are not dominated by any other solution. These are the “optimal” solutions where you cannot improve one objective without making at least one other objective worse.

Determining Qualifiers

Challenges in TIG have difficulty parameters that serve as the objectives/dimensions attached to solutions in respect to the Pareto Dominance. Parameters for each challenge can be found in Challenges. At each block we can use this to determine which solutions qualify for rewards.

There is a threshold of 5000 solutions that can qualify for rewards per challenge. If the number of qualifiers exceeds this threshold, then solutions with higher difficulty have greater priority over lower difficulty solutions when determining which solutions are rewarded.

This is implemented using Pareto Frontier Mechanism:

  1. Initialize: All benchmarks for a challenge are sorted by difficulty. Set the total number of qualifiers to total_qualifiers=0{total\_qualifiers}=0, for each Benchmarker ii set their qualifiers to qualifiersi=0{qualifiers}_i=0

  2. From the remaining benchmarks, identify those with difficulties on the current Pareto Frontier and add them to a processing list.

  3. For each Benchmarker ii, let qiq_i be the number of solutions for Benchmarker ii in the current processing list:

    • Set qualifiersi+=min(cutoffiqualifiersi,qi)\text{qualifiers}_i+=\min(\text{cutoff}_i-\text{qualifiers}_i,q_i).
    • Set total_qualifiers+=iqualifiersi\text{total\_qualifiers}+= \sum_i \text{qualifiers}_i
  4. Remove the benchmarks from the current processing list from consideration for subsequent iterations (these benchmarks remain active and available for the next block’s qualifier determination).

  5. Repeat steps 2-4 until the number of qualifiers reaches at least threshold_num_qualifiers{threshold\_num\_qualifiers}



Note: There can be more than threshold_num_qualifiers{threshold\_num\_qualifiers} qualifiers per challenge, threshold_num_qualifiers{threshold\_num\_qualifiers} is currently set to 5000, Only the qualifying solutions will affect difficulty and earn rewards. The term cutoffi{cutoff}_i is the cutoff of Benchmarker ii for this challenge.

Difficulty Adjustment

TIG automatically adjusts challenge difficulty range based on the total number of qualifying solutions found by the Benchmarkers. This process happens at every block, such that when a Benchmarker references a block they must choose a difficulty setting within that blocks difficulty range.

The scaling factor determines how difficulty should change:


scaling_factor=min(total_qualifiers5000,1.125)\text{scaling\_factor} = \min\left(\frac{\text{total\_qualifiers}}{5000}, 1.125\right)

The difficulty adjustment is then a two step process:

Step 1: Identify the Easiest Qualifying Frontier:

Find qualifying solutions that form the lowest difficulty Pareto frontier, these are solutions dominated by all other solutions but not by each other. This set of points is defined as the easiest_qualifying_frontier{easiest\_qualifying\_frontier}.

Step 2: Create Base and Scaled Frontiers:

The frontiers are sets of points where each point represents difficulty coordinates. When we scale a frontier, we multiply each coordinate of every point in the set by that scale.

  • If scaling_factor1{scaling\_factor} \geq 1:

Set

base_frontier=easiest_qualifying_frontier\text{base\_frontier} = \text{easiest\_qualifying\_frontier}

Then:

scaled_frontier=scaling_factor×base_frontier\text{scaled\_frontier} = \text{scaling\_factor} \times \text{base\_frontier}
  • If scaling_factor<1{scaling\_factor} < 1:

Set

base_frontier=scaling_factor×easiest_qualifying_frontier\text{base\_frontier} = \text{scaling\_factor} \times \text{easiest\_qualifying\_frontier}

Update

scaling_factor=min(1scaling_factor,1.125)\text{scaling\_factor} = \min\left(\frac{1}{\text{scaling\_factor}}, 1.125\right)

Then:

scaled_frontier=scaling_factor×base_frontier\text{scaled\_frontier} = \text{scaling\_factor} \times \text{base\_frontier}

When precommitting to a benchmark, a Benchmarker’s difficulty setting is considered valid if it:

  • Does not Pareto dominate the scaled_frontier{scaled\_frontier} (not too easy)
  • Is not Pareto dominated by the base_frontier{base\_frontier} (not too hard)

This creates a “difficulty corridor” that automatically adjusts based on network performance, ensuring challenges remain appropriately challenging as the compute in the network grows.