Building Hack the North’s AI Goose Guide Bot
Written by: Alex Aumais
The best part of being an organizer at Canada’s largest Hackathon is how empowered every member is to bring ideas to life. We’re always working on ways to innovate and make Hack the North transformative for our hackers, volunteers, sponsors and, of course, our organizers!
This year, Goose Guide Bot served as an excellent example of an innovative project that met all the criteria. My name’s Alex, and I led the development of that bot! I’ll walk you through the experience of developing it with the team, and its impact on Hack the North this year!
This was my first year at Hack the North as a backend developer. There’s certainly a lot to learn about our technical systems, event planning, and Hack the North as an organization, but it was also amazing to discuss the event with all the other organizers.
One night, I was chatting with new, returning, and even past organizers about the experience of working support shifts during the event. Some quotes came up and really stuck with me:
Help desk’s always flooded with people, and plus I spend the whole event on slack answering questions.
It’s unfortunate, because most of the answers to people’s questions are in the welcome booklet, but many hackers don’t read it and instead just ask in #questions. And it can take from 10 minutes to half an hour for someone to get around to answering a simple question.
I would get tons of direct messages on Slack during my sleep break, so I wouldn’t get to them until much later.
This made me think: Huh, this seems like the type of knowledge retrieval challenge where AI could truly be helpful.
Throughout the night, we brainstormed ideas of how we could use AI to support hackers at our event, and tackled key questions about user experience, platform choice, and information retrieval. We settled on using Retrieval Augmented Generation (RAG) to look up information in our Hacker Handbook and formulate answers for hackers.
Over the next couple of weeks, we formalized our problem statement: Hackers were waiting too long for accurate answers to simple questions, and organizers were overwhelmed with the task of answering them. Our solution? Use a RAG-based AI bot to automatically answer hacker’s day-of questions posted in the #questions Slack channel.
With this solution in mind, we dove into the design and technical decisions:
User Experience and Functionality
- We started with single-answer responses, with plans to explore conversational versions in future project iterations.
- Questions were collected from both the #questions channel and direct messages to accommodate user preferences and handle more private inquiries.
- We aimed for a 10-second response time, and used a 🤖 emoji to give users some quick feedback that their question was being processed.
Information Management and Accuracy
- We’re a real time event! Information evolves over time, (like how our ceremonies were pushed back by half an hour this year), so out-of-date information could cause more harm than good. That’s why we gave Goose Guide Bot the ability to also pull information from the #announcements channel in our Slack, so that when we posted public updates, they could be reflected in its automated answers.
- We also didn’t want Goose Guide Bot to go rogue, telling you to eat rocks and put glue on your pizza, or make up activities that we weren’t running. So to mitigate this, we set up some anti-hallucinations checks and filters.
- We finally implemented a monitoring channel that let organizers keep an eye on the questions being asked and how the bot was responding. Goose Guide Bot scored itself on its answers, and if they were low confidence, or it completely failed to give you an answer, it would leave a notification so an organizer could manually answer it. This came in pretty handy during the event, and here’s an example of what this looked like while we were testing.
Evaluation and Improvement
- We also set up comprehensive logging to analyze usage trends and improve the bot for future events, and you’ll see some of the data we collected with charts below!
On a technical level, we partnered with our sponsor Voiceflow to bring this project to life using their platform to build a conversational AI agent. The setup was fast, we had our first prototype ready in under an hour and could share a link with the organizer team right away. Voiceflow was also a good choice because it allowed us to build the complexity we wanted while simplifying the challenges of creating a Retrieval-Augmented Generation (RAG) based agent.
Goose Guide Bot’s Knowledge Base (the collection of documents the bot can read to generate answers) included a variety of resources, such as our public Welcome Handbook, internal documents, event-day plans, transcribed maps, menus, and travel itineraries. Voiceflow made it really easy to go in and add more information to the knowledge base on the fly. Finally, as mentioned before, Goose Guide Bot also collected extra information from all the announcements sent in real time, so its answers were up-to-date.
We developed the Slack Bot through the Bolt Framework for TypeScript, and Goose Guide Bot is open source for you to learn from, or even use for your own event!
https://github.com/hackthenorth/goose-guide-bot
Plus, if you want a technical deep dive on how it works, you can find a series on the Voiceflow YouTube channel here: https://youtu.be/4VubbpLFcO4
Finally, Goose Guide Bot was ready to launch. It was amazing to see the project I spent so long on have its first users, as hackers were sent their Slack invitations.
I really believe that having a strong connection with your work and directly seeing its impact on people is one of the best aspects of being a Hack the North organizer. You’re not just working on a side project to pad your resume — you’re developing cool tech that goes into production and reaches over a thousand users!
So, let’s get into some stats.
During the event, Goose Guide Bot answered 835 unique questions from 298 distinct hackers, so that’s an average of 2.8 questions per user. Those 835 questions, at 1 minute per question on average, would have taken organizers 14 hours straight of answering questions on Slack, so the impact was clear. The bot also knew a ton about the event. As a member of our leadership team aptly put it:
“Goose guide bot knows a LOT more about the event than I do haha”
Of those 835 questions, 83% were considered successfully answered (either a success, emergency, or filtered out as irrelevant).
On average, Goose Guide Bot’s answers got a score of 3.71 out of 5 when graded on a rubric for helpfulness, where 5 is going above and beyond, and 0 is not answering the question at all.
It answered questions 24/7 throughout the event, and held up the fort while most organizers were asleep late at night.
And all those questions were answered on average in 8.95 seconds, which is an impressive improvement from our human average of about 10 minutes in 2023.
Using Cohere’s semantic clustering tool, we see that Goose Guide Bot was asked a variety of questions, and this data will be used to improve its knowledge base in future years.
Goose Guide Bot was definitely a success. It achieved its mission of answering hacker questions, and I was over the moon when I came into Organizer HQ to bump into the advisor I had originally dreamt this project up with, and they told me how Goose Guide Bot was saving their life. 🤩 It’ll undoubtedly return in some form in the future, with some improvements.
And who knows, maybe even an internal organizer version of Goose Guide Bot to help us during the year 👀
If you want to join a team of ambitious organizers, working on cool projects from goose guide bot to logistics, marketing and more, consider joining our organizing team! Follow us on social media for updates on when organizer applications open in the new year!
— Team Hack the North 💙⚙️
Editors: Agamjot S., and Catherine Y.
