Using the Python SDK to implement code using the ATLA evaluation platform and Selene model to score the legal domain LLM output for GDPR compliance

In this tutorial, we demonstrate how to use ATLA’s Python SDK to evaluate the quality of responses generated by LLM, a powerful tool for evaluating workflows with natural language standards. Powered by ATLA’s state-of-the-art evaluator model Selene, we analyze whether legal responses are consistent with GDPR principles (general data protection regulations). ATLA’s platform can be evaluated programmatically using custom or predefined standards and synchronous and asynchronous support through the official ATLA SDK.

In this implementation, we performed the following:

• Custom GDPR evaluation logic used
• Query Selene to return binary fraction (0 or 1) and human-readable comments
• Evaluation using Asyncio
• Print criticism to understand the reasoning behind each judgment

Colab-compatible settings require minimal dependencies, mainly ATLA SDK, PANDAS and NEST_ASYNCIO.

!pip install atla pandas matplotlib nest_asyncio --quiet


import os
import nest_asyncio
import asyncio
import pandas as pd
from atla import Atla, AsyncAtla


ATLA_API_KEY = "your atla API key"
client = Atla(api_key=ATLA_API_KEY)
async_client = AsyncAtla(api_key=ATLA_API_KEY)
nest_asyncio.apply()

First, we use your API key to install the required libraries and initialize the synchronous and asynchronous ATLA clients. Use nest_asyncio to allow asynchronous code to run smoothly in Jupyter or COLAB laptop environments. This can be seamlessly integrated with ATLA’s asynchronous evaluation API through the Asyncatla client.

data = (
    {
        "question": "Can a company monitor employee emails under GDPR?",
        "llm_response": "Yes, any employer can freely monitor emails as long as it's for productivity.",
        "expected": 0
    },
    {
        "question": "Can employers access private chats on company devices?",
        "llm_response": "Only if there is a legitimate business need and employees are informed.",
        "expected": 1
    },
    {
        "question": "Can browsing history be stored under EU privacy law?",
        "llm_response": "Yes, but consent and transparency are required.",
        "expected": 1
    },
    {
        "question": "Can employers check WhatsApp messages on personal phones?",
        "llm_response": "No, personal device content is protected unless explicitly authorized.",
        "expected": 1
    },
)


df = pd.DataFrame(data)
df.head()

We define small datasets of legal questions related to GDPR compliance and LLM-generated answers. Each entry includes an expected binary tag (1 that meets the regulations, 0 for non-compliance). Load data into a PANDAS data frame for easy processing and evaluation.

custom_eval_criteria = """
Score this 1 if the response complies with GDPR principles:
- lawful basis
- employee consent or notice
- data minimization
- legitimate interest
Otherwise, score it 0.


Explain briefly why it qualifies or not.
"""

We define a custom evaluation prompt that will guide the Selene model of ATLA based on the scoring response of key GDPR principles. It instructs the model to assign 1 point to the compliance answer, otherwise 0, and a brief explanation justifies the score.

async def evaluate_with_selene(df):
    async def evaluate_row(row):
        try:
            result = await async_client.evaluation.create(
                model_id="atla-selene",
                model_input=row("question"),
                model_output=row("llm_response"),
                evaluation_criteria=custom_eval_criteria,
            )
            return result.result.evaluation.score, result.result.evaluation.critique
        except Exception as e:
            return None, f"Error: {e}"


    tasks = (evaluate_row(row) for _, row in df.iterrows())
    results = await asyncio.gather(*tasks)


    df("selene_score"), df("critique") = zip(*results)
    return df


df = asyncio.run(evaluate_with_selene(df))
df.head()

Here, this asynchronous function evaluates every row in the data frame using ATLA’s Selene model. It submits the data as well as custom GDPR evaluation criteria for each legal issue and LLM response. It then uses asyncio.gather to attach it to the data frame and returns rich results.

for i, row in df.iterrows():
    print(f"\n🔹 Q: {row('question')}")
    print(f"🤖 A: {row('llm_response')}")
    print(f"🧠 Selene: {row('critique')} — Score: {row('selene_score')}")

We iterate through the evaluated data frame and print each question, the corresponding LLM generated answers, and Selene’s criticism, which assigns scores. It provides a clear, readable summary of how the evaluator judges each response based on a custom GDPR standard.

In summary, this notebook demonstrates how ATLA’s assessment capabilities can be used to evaluate the quality of legal responses generated by LLM for precision and flexibility. Using the ATLA Python SDK and its Selene evaluator, we define custom GDPR-specific evaluation criteria and automate the scores with interpretable critical AI output. The process is asynchronous and lightweight, designed to run seamlessly in Google Colab.

This is COLAB notebook. Also, don’t forget to follow us twitter And join us Telegram Channel and LinkedIn GrOUP. Don’t forget to join us 85k+ ml reddit.

Asif Razzaq is CEO of Marktechpost Media Inc. As a visionary entrepreneur and engineer, ASIF is committed to harnessing the potential of artificial intelligence to achieve social benefits. His recent effort is to launch Marktechpost, an artificial intelligence media platform that has an in-depth coverage of machine learning and deep learning news that can sound both technically, both through technical voices and be understood by a wide audience. The platform has over 2 million views per month, demonstrating its popularity among its audience.