PSV Sample Quality Validation and Verification Tool

A Serendipitous Discovery: PSV Sample Quality Validation Can Be This Automated

Just the other day, I was casually browsing GitHub, hoping to find some tools that could improve dataset quality, and then I stumbled upon this skill called prk-psv-qa. To be honest, at first I thought it was something related to audio or video, but upon clicking in, I realized it was about PSV sample validation. So, what exactly is a PSV sample? It's essentially a standardized sample used to evaluate the capabilities of large language models. Each sample contains elements like task context, prompt, output, and ground truth. Previously, I had to check these samples manually, and it was genuinely exhausting, especially when the number of samples exceeded a few hundred.

The emergence of this skill was a lifesaver for me. It can be run directly from the command line, and with just a simple command, it automatically performs quality checks on all PSV samples within a YAML file. I tried running it, and within minutes, it produced a complete validation report—showing which samples had issues, their scores, and where changes were needed, all at a glance. This move is absolutely YYDS, truly top-notch!

What Exactly Can prk-psv-qa Do?

In simple terms, this skill acts as a quality inspector for PSV samples, and a rather strict one at that. It reviews each sample from six dimensions:

• Format Validation: Checks whether all required fields are present, whether data types and ranges are correct, and whether the format of external_id is standardized.
• Content Quality Scoring: The weight distribution here is quite reasonable—task context clarity accounts for 20%, prompt authenticity for 30%, output consistency for 25%, and ground truth accuracy for 25%.
• Ground Truth Calibration: Assesses whether the sample's rating is justified, whether the reasoning is clear and educational, and whether it strictly adheres to the PECAM rating definitions.
• Technical Prompt Leakage Detection: This design is quite clever—it specifically checks whether the task context implicitly contains evaluation guidance or mentions specific technique names, preventing "cheating" in the prompts.
• Realism Check: Ensures that the prompts sound like something a normal human would say, and that the outputs resemble what a real model would generate, with natural and fluent language.

After all checks are completed, the skill updates the qa_status, qa_score, and qa_feedback fields for each sample in the YAML file, and also generates a summary file called validation_report.txt. Looking at the output format, it's quite ceremonial—there's a total score, pass rate, quality breakdown, distribution statistics, and finally a list of problematic sample IDs along with their reasons.

Hands-On Experience: Installation and Running

The installation process is fairly straightforward—it only requires a single command:

npx skills add https://github.com/promptranks/prk-psv-flow --skill prk-psv-qa

After execution, you can simply type prk-psv-qa in the terminal to run it. If you want to specify a particular YAML file, you can add a parameter:

prk-psv-qa --file=psv_samples_batch2.yaml

By default, it automatically selects the most recently modified YAML file, which is a thoughtful little detail.

I tested it by running a YAML file containing 10 samples, and the output looked like this:

========================================
PSV Sample Validation Report
========================================
Total Samples: 10
✓ PASSED: 8 (80%)
✗ REVISED: 2 (20%)

Average Score: 8.2/10
...
Issues Found:
- PSV-CORE-0003 (6.5): Ground truth too high
- PSV-CORE-0007 (6.8): Technique hint detected
========================================

Seeing those two REVISED samples, I immediately went to check the detailed report. Sure enough, PSV-CORE-0003 had a ground truth rating that was set too high, while PSV-CORE-0007 had a technique name quietly embedded in its task context. Thanks to this tool, I wouldn't have been able to spot these subtle issues on my own.

Some Usage Insights and Cautions

From my usage, I believe the greatest value of this skill lies in how it quantifies the concept of "quality." Previously, I judged whether a sample was good based purely on intuition; now I can rely directly on scores. Moreover, its calibration of ground truth is particularly rigorous, which reminds me to be more careful when writing the reasoning for samples—there's no room for sloppiness.

There are a few points I'd like to remind everyone about: First, this tool is only responsible for checking, not for making corrections. If you find samples scoring below 7.0, you'll need to revise them yourself or use the companion prk-psv-revise skill for handling. Second, ensure that the YAML file format is standardized before running, otherwise parsing errors might occur. Third, although its checks are fairly comprehensive, it's not infallible, so a final manual review is still essential.

Summary

prk-psv-qa is a tool worth bookmarking for anyone currently building PSV benchmark sets. It transforms the tedious process of quality checking into an automated workflow, saving both time and effort, and the results are quite reliable. If you're also struggling with sample quality control, give this skill a try—it might just save you a few strands of hair 😄