Skip to main content

Georgia Tech College of Design

Formerly known as the College of Architecture


Shimon Plays CODA Building Grand Opening, Wows Research Neighborhood

A photo of Shimon the marimba-playing robot, far left, plays with saxophonist Richard Savory, percussionist Chris Moore, and keyboardist Gil Weinberg in the lobby of the new CODA building.
Shimon the robot improvises on the marimba with the Center for Music Technology robotic musicianship team in the lobby of the new CODA building. | Photo: Parker Coye/College of Design

It felt appropriate that a Georgia Tech research project should be a star attraction at the grand opening of the Southeast's premier innovation neighborhood.

In the sunlit lobby of the CODA building, Shimon the robot's jazzy marimba drew a crowd of stunned onlookers. To be fair, it's still a novelty to see a robot bobbing its head to the beat and jamming with live musicians.

"This is something I like to do," said Gil Weinberg, director of the Center for Music Technology and head of the robotic musicianship team that built and improves Shimon. "I look at Shimon and get inspired to write new pieces, especially for Shimon to play. And hopefully every time I play these pieces, each performance will be a little different in a good way."

Shimon is the only robot of its kind: a robotic marimba player that can listen to, understand, and collaborate with human counterparts. It uses artificial intelligence and deep learning -- and creative algorithms -- to push musical experiences and outcomes to uncharted domains.

Weinberg and Ph.D. students at the Center have taught Shimon large datasets from a variety of composers. Shimon studies the data and then produces and performs its own original compositions. Shimon recently even scored an entire film.

The CODA performance began with a short demo, featuring Shimon improvising with Weinberg on keyboards; Richard Savery, a Ph.D. student in the School of Music, on saxophone; and Chris Moore, the coordinator for the Bachelor of Science in Music Technology program at Georgia Tech, on drums.

"The first piece is a demo we normally do," Weinberg said. "Shimon is going to listen to me, mostly, and also to Richard playing saxophone. And respond with its own machine-learning based improvisation."

"The second piece is a standard called “Things Are Getting Better”, and the last piece is a piece I wrote called “Iltur” which is improv in Hebrew," he said.

Research, Performances, and a Globetrotting Robot

Shimon the robot plays marimba with Ph.D. student Richard Savery on saxophone and School of Music faculty member Chris Moore on drums.

Shimon the robot looks at Ph.D. student Richard Savery while improvising during the CODA building grand opening performance. | Photo: Parker Coye/College of Design

Playing the CODA building's swanky grand opening was no small feat for Weinberg and team. Logistically, Shimon is difficult to move. The robot has three parts: the head, hands, and the marimba.

Altogether, Shimon is seven feet wide and five feet tall. When it's moved, the robot is disassembled and packed into three custom-made, 4-foot by 6-foot black boxes. Shimon's computer and all the other equipment is packed into a 3- by 3-foot box.

Even without its luggage, Shimon is heavy. At least two people are needed to load and unload the robot. After Shimon is transferred into boxes, it is loaded onto a truck and moved to a local performance venue in this case, or to the airport to fly to its next destination.

Invented in 2008, Shimon is an avid traveller. Together with Weinberg, Shimon has criss-crossed the globe, performing in the Netherlands, Shanghai, China; Berlin, Germany; and Istanbul, Turkey.

In the U.S., they spent the last decade performing coast-to-coast. Its first Deep Learning composition debuted at the Consumer Electronics Show Nvidia Keynote presentation in Las Vegas; its first ever live TV gig, was the "Today Show" in New York City; and Shimon played to its first standing-room-only concert at the Kennedy Center, in Washington, D.C., commemorating the 25th anniversary of the signing of the Americans with Disabilities Act.

Shimon doesn't get performance jitters, Weinberg said, but robots can be temperamental. Even before the CODA performance, Shimon didn't play perfectly.

"We had a bunch of issues just like every gig. We just need it to work for the 20 minutes we’ll be on. We did a bunch of 20 minutes the morning of the show that luckily wasn’t public," he said.

Upgrades and Academics

Shimon the robot is motionless at the marimba before the performance.

Shimon the robot at the marimba, before performance. | Photo: Parker Coye/College of Design

As onlookers marveled at Shimon' performance, some of them must have considered the implications of an improvising robot. Several new tenants of the building and Georgia Tech's Executive Vice President for Research, Chaouki Abdallah, watched as Shimon jammed with the robotic musicianship team.

“Shimon exemplifies what music at Georgia Tech is all about,” says Jason Freeman, chair of Georgia Tech’s School of Music.

“This awe-inspiring project created by music technology students and faculty leverages recent advances in fields ranging from mechatronics to deep learning," he said. "It suggests new ways in which humans and machines can collaborate to create innovative and expressive music while also pushing forward research in fields such as robotics, machine learning, and human-computer interaction.”

The next project for Shimon includes significant upgrades to the hardware and software, Weinberg said.

"The updates will hopefully allow it to play much more interesting music, because it will have new strikers that will allow it to play much faster and create more virtuosic performances."

"It will also be able to sing," he said with a smile.