Llama Impact Hackathon

Start

Brightening our reality towards the future of fully autonomous laboratory

Auto Science Llama

garyyjn

Matthew Mo

80% of scientific publication cannot be replicated at Pfizer, $28 Billion+ wasted each year on bad science. Experimental procedures require a new medium of communication rather than texts-based publications. We bring experiment instructions to Augmented Reality and blend those instructions into the laboratory environment. This is extensively enabled by AI and LLM pipelines that breaks down text based instructions into specific actionable steps. During this hack, we made pipeline even more robust via Llama models capable of image and vide understanding, because often text instructions lack specificity in technical or geometric-dependent steps. We demonstrated this by having Llama vision, with help of RAG from OpenTrons documents, understanding of a pseudo-incision made on a plastic bottle and guiding a student to make the same incision. We also tested Llama vision's understanding on considerable amount of video demonstration data on Jove and documented it being capable at technical video understanding. 

Using Llama Vision to understand demonstrated experiment videos and assert reproducibility(doing experiments correctly) as a an overwatch agent via Augmented Reality. 

"Auto Science Llama team on Llama Impact Hackathon Hackathon"

Team Idea

Submission