Expansion of Remote Labor Index Dataset

The Remote Labor Index (RLI), a benchmark designed to measure the capabilities of artificial intelligence models against complex, real-world tasks, has been updated to include new projects from Fable and Astra. According to data from remotelabor.ai, the index tracks a diverse array of professional workflows, including architecture, data analysis, video animation, product design, and game development.

The dataset comprises projects that represent over 6,000 hours of professional labor, with a total valuation exceeding $140,000. These projects vary significantly in scale and complexity; some individual tasks involve costs surpassing $10,000 and require more than 100 hours of human labor to complete. All cost and time metrics are derived directly from the human professionals who performed the original work.

Methodology and Evaluation Standards

The RLI employs a rigorous evaluation framework to test the end-to-end capabilities of AI models. In this testing protocol, a model is provided with a single prompt containing a specific set of requirements and is tasked with completing the project in its entirety.

Following the model’s attempt, human experts evaluate the output. To achieve a successful score, the model must produce work that is comparable in quality to that of human peers and must fulfill every specified requirement. According to discussions on the social platform Reddit, the benchmark maintains a strict binary scoring system: partial completions are not credited. A model that achieves 90% or 99% completion is still categorized as a failure within the index.

Performance and Scope

The benchmark focuses on “frontier” models, a decision driven by the significant resources required to conduct human-led evaluations. Due to this intensive process, the index does not currently evaluate every available model.

Unverified commentary from users on Reddit suggests that the success rate for complete project execution remains low, with one user noting that only 21% of projects in the index were completed from a single prompt. The discussions also highlighted that the index’s current scope is limited to high-level models, as the effort required to evaluate non-frontier models is substantial. While there has been speculation regarding future plans to expand the range of evaluated models, no official confirmation has been provided by the RLI organizers.