Thrilling, indeed! Looking forward to working together!

snowflake.com/blog/snowflake…

Excited to be working with Nandan Thakur Jimmy Lin alongside the wonderful folk Snowflake including Daniel Campos Vivek Raghunathan et al. and in the coming year on TREC RAG and BEIRv2 among many interesting things to come!

Excited to partner with Jimmy Lin to help build the next generation of open retrieval benchmarks!

World-class search needs world-class metrics 🚀

And great metrics need to be constantly evolved to avoid overfitting 🤟

As we build out a great search experience at Snowflake AI research, we are excited to join forces with Jimmy Lin and the University of Waterloo 🙌

Our

We've made it even easier to play with the MS MARCO V2.1 corpora for the TREC 2024 RAG Track with Anserini prebuilt indexes. Literally, install via 'wget' and copy/paste a command to get BM25 results! github.com/castorini/anse…

If you want to play with the new MS MARCO V2.1 Corpus for TREC 2024 RAG trec-rag.github.io - we have Anserini prebuilt indexes ready! Instructions at github.com/castorini/anse…

The TREC RAG website (trec-rag.github.io/annoucements/2…) has been updated with links to download the updated corpora and topics + qrels from prior years! More topic details to come soon.

Oh and one more thing ...
We have task descriptions out too! As always, we open it up to discussions to get the best version of the track for all of you!

This is an awesome competition 😀

Terrible timing announcing this today... but bring your #LLaMA3 models over to the TREC 2024 RAG track to tackle the all important question: 'How often should you take your toddler to the potty when potty training?'

Orgs wishing to participate in TREC RAG 2024 should respond by submitting an application! To apply, use the new Evalbase web app at ir.nist.gov/evalbase. Create an account + profile, then register a participating org from the main page! More here trec.nist.gov/pubs/call2024.…!

We have geared up the three tasks - (i) R, (ii) AG and (iii) RAG to be held this year in TREC RAG 2024.

Fun fact: Wondered how to potty train your toddlers? 💩
In the potty training example we show toddlers' bathroom habits might hold the key …🚽 #RAG

trec-rag.github.io/annoucements/2…

We're bringing you the MS MARCO V2.1 document corpus and its segmented version! Deduped and prepped with care for the TREC RAG and your retrieval needs! Corpora should be accessible soon along with all our Anserini/Pyserini Indexes + RankLLM baselines!

RAG is all the RAGe these days, but we (still) don't quite know how to evaluate it properly... This year, we are taking a stab at it in the context of TREC, building on 30+ years of experience in evaluating IR systems. trec-rag.github.io

🧐 Exploring deduped MS MARCO V2 as the corpus of choice for TREC 2024 RAG 🚀. Props to NIST folk for a resolution to near dupes in the original corpus. Potential memorization issues acknowledged. You can trust us for baselines soon 😉 Let's discuss! 🔄

#RAG #IR #TREC_RAG