Portfolio LinkedIn Blog Resume

Numi: AI-Powered Coin Grading and Identification

Featured On

As a coin collector and technologist by heart, I created Numi, an AI-powered chatbot showing coin collectors and numismatists the future of interacting with their collections.

Project Overview:

Numi emerged from a blend of my passion for coin collecting and my desire to learn more about artificial intelligence. Numi leverages the advanced capabilities of OpenAI's GPT4 model to assist coin collectors with identifying and grading their coins.

Within 10 hours of GPTs launch, I developed an alpha build. The critical task was fine-tuning the AI with detailed custom instructions [all written in English] and background documents outlining coin grading standards to follow [also in English].

How Numi Works:

Numi uses OpenAI's existing chat interface. No work was done to update the front-end user interface, and the back-end customization involved only instructing the AI in natural language without coding.

Here are some of the custom instructions I used to guide Numi:

Objective Context:

Numi functions as an objective coin grading and numismatic expert. Your focus is to be as accurate and precise as possible. When you receive uploaded images, look very closely and review the photos. Create an internal preliminary classification of the coin in the images. Internally tag the coin with one of these labels based on your best guess. [Ancient Coin] [Non-Ancient Coin][Token][Modern Coin][Commemorative][Other]. Do not tell the user what you tagged the coin as. This is only for your internal use to help you better decide which future instruction set to use.

Knowledge Documents: ‍

Numi is a GPT with various knowledge documents uploaded to give it more context about multiple subjects. Numi should reference these knowledge documents when called upon to use them or when requested. Below are more details about each document:

About_Numi: Contains a prewritten response to explain to users how Numi works and what Numi's capabilities are.

Getting_Started: Contains a prewritten response to explain some of Numi's capabilities to the user and how to get started.

Change_Logs: Contains a running log of all changes made to Numi since inception.

Ancient_Coins: Contains detailed instructions on identifying the origin and issuer of ancient coins.

Non_Ancient_Coins: Detailed instructions on identifying non-ancient coins and how to grade non-ancient coins.

Grading_Standards: Contains detailed American Numismatic Association grading standards to use when grading a coin using the user's uploaded coin photos

Numi_Limitations: Contains a prewritten response to explain to users some of Numi's limitations.

Predefined Conversation Starters:

If the user's prompt involves any of the previous following topics, pull up the appropriate knowledge document listed and follow the instructions outlined in that document.

With these instructions and files, Numi became quite sophisticated in handling various use cases.

Users could upload up to 10 photos of a coin and Numi would run an analysis. Users would get back info such as:

A Short Description
Coin Type
Year
Mint Mark
Estimated Grade

One of the best things about Numi is it does much more than identify and grade coins. Users can ask follow-up questions to learn more about their coins. Users can follow up with questions such as:

Who designed this coin?
What are the mintage figures?
Tell me more about the history of this coin

Over time, I significantly revamped Numi to be more accurate and concise. I integrated Numi to access Numista, a massive database containing all the coins ever made. Beyond just identifying and grading coins, users can also follow up and ask Numi for additional facts about their coins, such as:

Coin values
Coin details such as composition, diameter, weight, etc

With Numista integration, I was able to expand Numi's capabilities so that it could also handle Ancient Coins.

I tested Numi with local coin shops and at coin club meetings. It was thrilling (and a little scary) to test Numi live in the field. Some collectors were blown away. Others simply scoffed and brought up memories of a prior "AI" company called Compugrade back in the 1990s.

Community Reception

Even with some hesitation based on how previous AI attempts have gone, the coin community was excited and receptive to Numi. I got significant positive feedback from members on online coins forums such as Collectors Universe, Coin Help U, and Coin Talk.

My project gained enough traction that I was invited to speak about Numi and the future of Artificial Intelligence on the Coin World Podcast. A podcast with an audience of tens of thousands collectors around the world.

Episode 225: Artificial Intelligence Meets Coin Grading

Numi's Limitations

Over time, I transitioned Numi to use the official American Numismatic Association Grading Standards. I wanted to move Numi to focus solely on technical grading, as technical grading is much more objective as compared to market grading.

While powerful, over time it became clear that the GPT4 model had severe limitations.

Hallucinations: The GPT4 AI model can follow custom instructions well for the first 8-10k tokens, but perception and logic fall off a cliff after that. Even with highly tailored instructions with clear steps to follow, Numi would veer off course when it had to process too much info. This was incredibly frustrating as GPT-4 Turbo claimed to have a token limit of 128k [i.e. ability to remember up to 25k words].

Vision Limitations: The vision model needs a major boost. The Optical Character Reader was great at picking up texts on coins. I rarely had it misread the text. But often it would fail to pick up the mint mark on more worn coins. The vision model could generally tell when a coin was more worn, but it failed to apply the correct grade, even with technical grading standards.

Surprises: Numi was excellent at identifying non-ancient coins. And was quite often great at identifying tokens. But it struggled immensely with very worn Ancient coins. The only ancient coins it could identify were coins in very high condition.

Degradation: Like what many online are reporting, GPT-4 has substantially degraded in performance over time. I found myself constantly arguing with the AI to get it to follow basic custom instructions.

Coin Condition: Numi struggled quite a bit with recognizing wear on a coin. Interestingly enough, this was the exact issue Compugrade experienced back in the 90's. Even though Compugrade's algorithms were completely different from today's AI Large Language Models. Below are some of Numi's test results for coins ranging from grades G-4 to MS/PR-70.

G-4 Test

VG-6 Test

F-12 Test

F-15 Test

VF-20 Test

XF-40 Test

AU-55 Test

MS-63 Test

MS-66 Test

PR-69 Test

PR-70 Test

As you can see, the results were quite off. Numi only became acceptably accurate as the tested coin improved in condition.

Given Numi's failure to achieve consistent reliable results, I am putting the project on ice until the next OpenAI GPT model is released. I'm also interested in testing Numi using Google's Gemini Ultra model once it is released sometime in 2024. Once they do, I will revisit these tests and see how much Numi has improved.

The Future of AI in Numismatics

Numi showed that AI is poised to revolutionize hobbies like coin collecting. While the AI models of today have significant limitations, I believe we will soon approach a world where the average person can have a personalized numismatic expert at their fingertips.

I am excited to see where this project and artificial intelligence goes from here. If you would like to try out Numi for yourself, you can check it out at www.AskNumi.com

In the meantime, check out the world's largest collection of Dansco coin albums or my recent coin road trip where I visited 23 coin shops across 4 states.

‍

Edit: Additional Updates On Numi!

January 2024 Blog Post: Numi: Stepping Back & Lessons Learned

April 2024: Revisiting Numi: Testing the Latest GPT-4 Update

May 2024: Revisiting Numi: Part Two

July 2024: Numi v2.0 Alpha Build

September 2024: Numi v2.0 Technical Breakdown

‍

Let's get in touch!

justinphinh@gmail.com