Skip to main content

Evaluating Thanoy The Thai Legal AI Assistant Performance

· 3 min read
Kobkrit Viriyayudhakorn
Kobkrit Viriyayudhakorn
CEO @ iApp Technology

The following evaluation report assesses Thanoy, the Thai Legal AI Assistant powered by OpenThaiGPT, which is designed to provide accurate and reliable legal advice across various legal documents and queries. Trained on over 10,000 Thai legal articles and regulations, Thanoy offers an advanced solution for legal professionals and general users seeking legal guidance.

1. Introduction to Thanoy​

Thanoy is an AI-powered assistant developed to enhance access to Thai legal information and advice. It leverages OpenThaiGPT to analyze and respond to user queries, offering insights into Thai laws and regulations. Key features include its availability through a LINE chatbot interface, ensuring users can access legal advice anytime. Thanoy is designed to ensure its responses are based on a comprehensive understanding of Thailand's legal landscape, making it an invaluable tool for both professionals and non-experts alike.

2. Evaluation Approach​

For this evaluation, an automatic assessment approach was used, employing OpenAI’s API to evaluate Thanoy’s responses. The evaluation focused on the alignment between the user query, context retrieved, and the legal advice provided by Thanoy. The model used for evaluation was ChatGPT-4o, set with a temperature of 0, to ensure that responses were precise and focused on factual accuracy.

3. Evaluation Dataset and Metrics​

We analyzed over 100,000 chat logs in JSON-Lines format to extract 1,000 random triplets (user queries, context, and responses) for the evaluation. The key metrics assessed included the Average Relevant Score, which measures how well Thanoy’s responses align with the user’s question and the retrieved context, and the Standard Deviation of Relevant Score, which indicates the consistency of Thanoy’s performance across different queries.

4. Key Findings and Results​

  • Average Relevant Score: Thanoy achieved a high average relevant score, indicating that its responses are generally well-aligned with the user’s queries and the legal context.
  • Standard Deviation of Relevant Score: The standard deviation was relatively low, suggesting that Thanoy’s responses are consistent and reliable across various types of queries.

These results demonstrate that Thanoy is performing at a high level in terms of delivering accurate and relevant legal advice, though some areas may still benefit from refinement in edge cases.

Thanoy is not just a tool for immediate legal advice but also a foundation for future advancements in AI-driven legal services. As AI technology evolves, the capabilities of assistants like Thanoy are expected to improve, particularly in terms of understanding complex legal language and providing even more precise insights. The feedback and performance from this evaluation are essential in driving future improvements and ensuring Thanoy can meet the growing demand for accessible legal assistance in Thailand.

Conclusion​

This evaluation highlights Thanoy’s strength as a Thai Legal AI Assistant, demonstrating its ability to provide accurate, relevant, and consistent legal advice. As AI continues to transform industries, Thanoy represents a significant step forward in the legal sector, offering real-time, reliable access to legal information and supporting users in navigating Thailand’s legal system.