Business Analyst 2
The Judge Group Inc.

Job Info

---

Location: Sunnyvale, CA
Salary: $60.00 USD Hourly - $65.00 USD Hourly
Description: Our client is currently seeking a Business Analyst 2

We are seeking a highly skilled data quality expert with a strong understanding of Large Language Models (LLMs) to join our team as a Quality Analyst for Human-created Data. In this role, you will liaise closely with the highly technical modeling teams as well as vendor leads, understand the data needs of LLM modeling teams, and ensure that the data produced by expert content creators meets the highest quality standards. You'll need a strong ability to understand LLM use cases, have a nuanced understanding of good versus great content, and be able to use quantitative and qualitative techniques to analyze LLM datasets.

Responsibilities:

Ensure high quality during task development

Define data requirements based on a deep understanding of the modeling team goals and study of model loss patterns

Design detailed instructions for content creator teams, illustrating nuanced differences with clear examples

Rapidly iterate with content-production vendors on small batches of data to arrive at desired data quality

Measure and demonstrate data quality during data collection

Drive processes to deliver benchmark datasets for assessing data quality

Benchmark vendor data quality against competitor models

Use quantitative techniques to analyze vendor-produced data to assure high quality and drive rater pool optimization

Audit datasets for quality issues and develop tools for accelerating qualitative analysis

Use modeling and experimentation techniques to demonstrate data impact and identify blind spots

Communicate on data quality

Send out regular reports to project stakeholders on data quality

Track key metrics such as iteration time and vendor quality audit results

Author best practices and guideline documents (e.g. guidelines for teams wanting to work with vendors for human data, best practices for instruction design, etc)

Skill/Experience/Education

Mandatory* Bachelor's degree in English, Creative Writing, Linguistics, Science, Computer Science, or a related field.

* 3+ years of experience in Technical Writing, Quality Assurance or a related field

* Excellent writing and editing skills, with the ability to adapt to different styles and tones.

* Strong analytical and problem-solving skills.

* Ability to work independently and as part of a team.

Desired* Understanding of LLM capabilities and limitations.

* Understanding of LLM processes like Pre-training, RLHF, SFT, Evals, etc

* Knowledge of NLP concepts and techniques.

* Familiarity with human evaluation methods and tooling for LLMs.

* Experience in a research or academic setting.

* Experience with training/tuning models, prompt engineering and evaluating LLM outputs is a plus.

* Experience in survey design, standardized assessment development, or a similar quantitative field

Contact: nkumar03@judge.com

This job and many more are available through The Judge Group. Find us on the web at www.judge.com

---