Join a fast‑growing AI firm to help build evaluation datasets for large language models tackling real software engineering tasks. The role blends hands‑on Ruby development with AI research, all on a fully remote freelance basis.
What You’ll Do
- Analyze and triage GitHub issues from trending open‑source libraries.
- Set up and Dockerize code repositories, configuring development environments.
- Evaluate unit‑test coverage and overall test quality.
- Modify and run codebases locally to benchmark LLM bug‑fix performance.
- Collaborate with researchers to select challenging repositories and issues.
- Lead junior engineers on task automation and dataset creation.
What You Need
- Minimum 3 years of professional software engineering experience.
- Strong proficiency in Ruby (or another listed language).
- Hands‑on experience with Git version control.
- Practical knowledge of Docker and CI pipeline setup.
- Ability to navigate and understand complex codebases.
- Experience running, modifying, and testing real‑world projects locally.
Good to Have
- Prior involvement in LLM research or evaluation projects.
- Experience building or testing developer tooling or automation agents.
- Contributions to open‑source projects or libraries.
The Opportunity
You’ll work fully remote with a flexible 20‑40 hour weekly commitment and a required 4‑hour PST overlap. This contractor role offers exposure to cutting‑edge AI research without traditional employment benefits.