If you care about rigorous evaluation of agentic systems, give it a look at MASEval! The harness is an important element of agents. MASEval makes it straightforward to change its components and evaluate their impact. MASEval is our first software! parameterlab.github.io/MASEval/ ⬇️

‼️New paper from Parameter Lab! ⛓️‍💥 We identify privacy collapse, a silent failure mode of LLMs: LLMs fine-tuned on seemingly benign data can lose their ability to respect contextual privacy norms. Done by @anmolgoel.bsky.social during his internship! Check-out 👇

👏 Proud to share that the paper that Ahmed Heakl authored during his internship at Parameter Lab was accepted at #ICLR2026! See how 🩺Dr.LLM increases accuracy and decreases inference computations of frozen LLMs: www.linkedin.com/posts/ahmed-...

Our #EMNLP2025 paper Leaky Thoughts 🫗 shows that Large Reasoning Models (LRMs) can unintentionally leak sensitive information hidden in their internal thoughts. 📍 Come chat with Tommaso at our poster on Friday 7th, 10:30–12:00 in Hall C3 📄 aclanthology.org/2025.emnlp-m...

🫗 An LLM's "private" reasoning may leak your sensitive data! 🎉 Excited to share our paper "Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers" was accepted at #EMNLP main! 1/2

🔎Does Conversational SEO actually work? Our new benchmark has an answer! Excited to announce our new paper: C-SEO Bench: Does Conversational SEO Work? 🌐 RTAI: researchtrend.ai/papers/2506.... 📄 Paper: arxiv.org/abs/2506.11097 💻 Code: github.com/parameterlab... 📊 Data: huggingface.co/datasets/par...

Excited to share that our paper "Scaling Up Membership Inference: When and How Attacks Succeed on LLMs" will be presented next week at #NAACL2025! 🖼️ Catch us at Poster Session 8 - APP: NLP Applications 🗓️ May 2, 11:00 AM - 12:30 PM 🗺️ Hall 3 Hope to see you there!

👥 We're Hiring: Senior/Junior Data Engineer! 📍 Remote or Local | Full-Time or Part-Time At ResearchTrend.AI, we’re building a platform that connects researchers and AI engineers worldwide—helping them stay ahead with daily digests, insightful summaries, and interactive events.

🔎 Wonder how to prove an LLM was trained on a specific text? The camera ready of our Findings of #NAACL 2025 paper is available! 📌 TLDR: longs texts are needed to gather enough evidence to determine whether specific data points were included in training of LLMs: arxiv.org/abs/2411.00154

We are delighted to announce that our research paper on the scale of LLM membership inference has been accepted for publication in the Findings of #NAACL2025! 🎉

There's an internship opening at @parameterlab.bsky.social : parameterlab.de/careers The research outputs have been quite successful so far: researchtrend.ai/organization...

🎉We’re pleased to share the release of the models from our Apricot🍑 paper, accepted at ACL 2024! At Parameter Lab, we believe openness and reproducibility are essential for advancing science, and we've put in our best effort to ensure it. 🤗 huggingface.co/collections/... 🧵 bsky.app/profile/dnns...

🚨📄 Exciting new research! Discover when and at what scale we can detect if specific data was used in training LLMs — a method known as Membership Inference (MIA)! Our findings open new doors for using MIA as potential legal evidence in AI. 🧵 https://arxiv.org/abs/2411.00154

📄 We’ve been working on the calibration of confidence score for black-box LLMs. See the thread below for an overview of the 🍑 Apricot paper, proudly accepted at #ACL24! #uncertainty

Check out one of our latest papers about LLM fingerprinting!

We are excited to join Bluesky! At Parameter Lab, we're committed to enhancing AI safety and trustworthiness. Our research addresses privacy, copyright, and security challenges in foundational models. Follow us for insights and updates on trustworthy AI research!