SOTA Evaluations and Benchmarking

Our tig-SOTA-metrics repository provides a straightforward way to evaluate TIG algorithms on relevant academic datasets and compare them to state-of-the-art (SOTA) methods. We have made this highly accessible via a one-click interactive Jupyter notebook for each challenge that downloads academic dataset(s), runs the evaluations, and generates clear visualizations of the results.

Innovators can get started by either:

• Running the SOTA Benchmarks on Google Colab.

• Running the SOTA Benchmarks locally on their machine.

Running SOTA Benchmarks on Google Colab

To begin, navigate to the tig-SOTA-metrics repository and select a challenge evaluator. Each challenge folder contains a dedicated interactive notebook and all the necessary files to run the evaluation.

Head over to the README.md file for your challenge and click on the Open in Colab button. This would launch a Jupyter notebook in Google Colab, where you can run the evaluation directly in your browser without needing to set up anything locally.

Note: You will need to be signed into Google to run the notebook.

To run the evaluations, select Run all from the top menu in the notebook:

Running SOTA Benchmarks Locally

To run the SOTA Benchmarks locally, you will need to have Python and Jupyter installed on your machine.

• You can install Jupyter using pip:

  pip install jupyterlab

• Clone the tig-SOTA-metrics repository:

  git clone https://github.com/tig-foundation/tig-SOTA-metrics.git && cd tig-SOTA-metrics

• Once installed, you can now run Jupyter Lab in the same directory:

  jupyter lab

• This will open Jupyter Lab in your default web browser. You can now navigate to the challenge you want to evaluate your algorithm against, and open the quick_start.ipynb notebook. Select Run all Cells from the Run tab to run all the cells in the notebook.