How I Build AI Projects From Scratch (Steal My Workflow)

If you're trying to get a job in AI, you've already heard that you're supposed to build projects. The hard part is that there's a huge gap between following a tutorial where somebody else has already made every decision, and building a legit project from scratch.

So in this guide I'm going to walk you through the actual process I use to build AI systems, from figuring out what's worth building in the first place, all the way through testing, deployment, and leveraging it for job opportunities once it's live.

This is the same basic process I've used building production AI and ML systems at companies like Amazon, but I'm going to break down every step assuming you've never built an AI product before.

The workflow at a glance

1. Know why you're building. Learning, demonstrating skills, or getting real users.
2. Pick a problem you understand. Then work backwards to the technology.
3. Scope it before you build. Who it's for, constraints, data, success metrics, and whether AI even makes sense.
4. Work with AI the right way. Spec-driven development, so you make the decisions, not Claude.
5. Set up the foundations. Data storage, API keys, environments.
6. Build an MVP that answers your biggest questions. Run it like an experiment.
7. Build in phases. Tackle the riskiest parts first.
8. Evaluate on two layers. Normal software tests plus an AI evaluation set.
9. Monitor it. Log every model call so you know what changed.
10. Deploy it. Then present it well and get real users.

Why build projects?

Before we actually build anything, we need to get really clear on why we're building a project in the first place.

Because a project can do three different things:

1. It can teach you a skill.
2. It can demonstrate those skills to potential employers.
3. And, if real people actually use it, it can start becoming experience you can put on your resume or even a product you monetize.

Those are three very different goals. If I build something just because I want to learn how to call a vision model, that's a perfectly good project for learning. But if I'm putting that same project on my resume and expecting it to help convince somebody that I can build production AI systems, the bar is higher.

Not only do I need to build something more robust, I also need to show that I used good judgement throughout, and that Claude didn't just vibe code some slop for me.

In this guide, I'm going to show you how to build a project that accomplishes all three of these goals: learning, demonstrating skills, and building experience by getting real users.

As an example, I'll walk through how I rebuilt an app that I originally created last year. I call it ShelfScanner. The idea is that you tell it what kinds of books you like, you take a photo of a bookshelf, like at the library or a bookstore, and it recommends books from that shelf that you're going to enjoy.

It's a really cool idea, but the original implementation... left something to be desired. So I rebuilt it step-by-step, starting from a totally blank repo.

What makes a good project?

Coming up with an idea for what to build is where a lot of people get stuck.

Every project idea either feels too simple to be impressive, or so ambitious that you have no idea how you'd ever finish it. And I think part of the problem is that people often start with the technology they want to use. They learn about RAG or agents or some model that just came out, and then they go looking for a problem they can jam in there.

I suggest working in the opposite direction. Start with a problem you understand, and then work backwards to figure out what technology actually makes sense for solving it.

You can look at problems in your own life, problems people around you have, or problems inside a community you're already part of, even a Reddit or Discord for a hobby you're into. Your own life is usually the easiest place to start, because you already understand what's annoying, what a useful solution would look like, and what good looks like when you're done.

The best portfolio projects are not always the most technically impressive. Instead, they're the ones you have a personal connection with, because it'll keep you motivated throughout the process, and that enthusiasm really comes across in interviews. Almost every hiring manager I spoke with for my book on ML careers said the same thing: they deeply look for enthusiasm and decision-making, which you can only really show if you came up with the idea yourself.

Do not copy the app I'm using as an example here. This is meant to give you a framework, not a project to make. That kinda defeats the point.

Now, on to the problem I was trying to solve in my own life: bookstores.

I love reading, but when I walk into a bookstore and I'm looking at a shelf full of books I've never heard of, I have basically no idea what I should buy. So the first version of the problem statement I wrote down for this project was basically:

"When I go to the bookstore, I can't figure out what to buy unless I recognize a title."

And the thing I wanted was pretty simple: take a picture of the books that are physically in front of me, give the system some information about what I like, and have it tell me which ones to buy.

Initial scoping

At this point it is very tempting to open a coding assistant, make a new project, and start building.

But before I build anything, I need to turn "an app that recommends books from a picture" into a problem specific enough that I can make decisions around it. Not Claude.

For a personal project, this can feel a little ridiculous because it's just you. But I want a record of what I believed before I started building, what constraints I was designing around, and what eventually changed. The project becomes more valuable because I approached it like a real product.

I use a scoping template that I've been developing over the course of my professional career. You can download a blank copy here.

There's a lot in it, because it's meant to cover the kinds of things that might come up in a production AI system: product requirements, data, models, evaluation, infrastructure, security, deployment, monitoring, and lots of other fun stuff.

But I am absolutely not filling all of that out at the start. Most of it stays blank on purpose. We are staying focused on the problem we are trying to solve. "Recommend books from a photo" leaves way too many questions unanswered:

• Who is this for?
• What kind of photo?
• Can it recommend anything it thinks I'd like, or only books it can actually see?
• What happens if it can only read part of the shelf?

I start by filling out the initial scoping questions that help me understand what problem I'm solving and for whom. I think about things like who the user is, what constraints I might have, what the input and output data looks like, and what success metrics might make sense. I even have a nifty checklist to fill out so that I make sure I stay on track.

Some technical decisions I can make right away. For example, I already have a pretty good idea what the data will look like, so I can start thinking about storage. I also know generally what I want the user experience to be. I don't want people waiting 10 seconds for a recommendation, so I have some ideas around goals for things like latency. I also know that I have a budget for this personal project, so I can include those kinds of decisions right at the beginning.

There's also one question I think people skip way too often in AI projects: does AI even make sense for this?

Adding a language model does not automatically make a project better or more impressive. If normal software can solve the problem just as well, the model mostly gives you extra cost and unpredictability, and makes you look like a noob.

Working with AI the right way

Once I understand the problem that I'm solving, I stop. Most of the template is still blank, because I still haven't selected a model, designed some giant architecture, or decided I need agents or RAG or whatever. I don't have evidence that I need those things yet.

What I do know is enough to make an MVP, and doing that right is really important. Because, again, it's really tempting to just start using Claude Code to build the project for you and get that little dopamine hit.

Now, I'm not going to say that you should code this by hand, but I do think there is an appropriate way to use coding assistants when you're doing portfolio projects. Remember, the point of portfolio projects is to learn and demonstrate that learning. If you have outsourced your thinking to Claude Code, you did not learn, and when you're asked in an interview, you will not be able to defend the decisions you made, because you did not make them.

The simple solution to this is to use some kind of spec-driven development workflow. Here's a really simple version, but keep in mind that there are also tools that implement this for you:

1. I set up some directories where I can keep track of docs as I go, then download the initial scoping doc as a Markdown file and add it to the repo.
2. I set up some basic steering files for Claude so that it adheres to my typical coding standards and rules about how we work together.
3. I install some skills that I intend to use throughout the project. In particular, I use the Supabase agent skills a lot. (Supabase sponsored the video version of this guide, but I use Supabase on all of my projects.) They've been super helpful for making sure I set up my database the right way.

With that setup in place, with rules for Claude and directories for proposing and tracking changes, any time I want to add a new feature, the assistant drafts a little proposal before it implements anything.

So if I say, "I want to add photo uploads," I want Claude to tell me what it thinks needs to change, what assumptions it's making, and how it plans to do the work.

Then I review that first, with a very critical eye. If it makes a decision that I did not make, I stop and figure out whether that's the appropriate decision. If it suggests something that I don't understand, I ask why. Once I'm happy with the plan, then it can move quickly with the actual implementation.

And I keep these decisions in files rather than relying on chat history, so that I remember what we decided and why. Before getting started, I also do a quick sanity check to make sure Claude understands the workflow we're going to use.

Project setup

Once that workflow is in place, I can start setting up the boring technical foundations.

I'm not going to turn this into a complete software engineering tutorial, because the exact tools you choose will depend on what you're building. But one thing I can say with pretty high certainty is that you will be working with data, and you're going to need somewhere to store it.

For this project, I'm using Supabase for the data layer. I've used it for a bunch of my own projects because it gives me a Postgres database and file storage in one place with a really generous free tier, and that happens to match the needs I already identified during scoping.

• I set up a new project in Supabase, install the Supabase CLI, grab the API keys for the agent to work with, and initialize a new project.
• I make sure all of my API keys are in a .env file that's gitignored properly.
• I use a tool like uv to manage my Python virtual environment.

The MVP

So now I'm ready to start building, but not the app just yet.

I need to make sure that what I'm building is actually reasonable. There are two open questions that could completely kill this project before I even start.

The first is whether an affordable vision model can look at a normal phone picture of a bookshelf and read the titles well enough to be useful. Book spines are sideways, some are tiny, some are partly covered, the lighting can suck, and on a real shelf there can be dozens of titles packed into one image.

If off-the-shelf models are terrible at this, the project changes a lot. Maybe I need a custom vision system. Maybe I need a completely different input. Maybe this idea stops being worth building. I want to know that before I spend two weeks making a pretty UI.

The second question is whether a reasonably affordable language model can take the books that were found, combine them with fairly minimal information about somebody's taste, and produce recommendations that are actually useful.

Those are the biggest unknowns, so those are what I address first.

And this is how I think about an MVP. MVP stands for minimum viable product, but people sometimes interpret that as "build a crappy version of the whole app." That's not really what I want here. For me, the MVP is the smallest thing I can build that answers the biggest questions I currently have about the viability of this project as a real product.

So ShelfScanner's first MVP is just a little command line script. I give it a photo. One model tries to read the titles. Another model takes those titles plus my preferences and gives me five recommendations. That's it!

This is also the first real change I let the coding assistant work through using the process I just described. It drafted the proposal, I found a couple of assumptions I didn't agree with, and we fixed those before it started writing implementation code.

Treat it like a real experiment

I want to approach this like a real experiment. I'm not just going to try a bunch of stuff and then go by vibes, so I need to save the result of every model call. Each attempt gets logged along with the things I'll need to compare later:

• what model ran
• what prompt it used
• what it returned
• how long it took
• roughly what it cost
• whether anything failed

So one of the first things I set up is a basic schema in Supabase to log the important things I need now. I use the Supabase agent skills I installed earlier to make sure I set everything up properly and catch any mistakes before they become an issue.

When I'm doing my evaluation, I also keep the shelf-reading stage separate from the recommendation stage, because there is no reason to assume the model that is best at reading tiny text in an image is also the model that is best at recommending books. Keeping them separate means I can compare each problem independently instead of treating every provider like one giant package. That also gives me a much cleaner system later if I need to swap one model without touching everything else.

Start with a small labeled test set

Before I can compare anything, I need some actual test data. I start with five bookshelf photos I already have.

Five is obviously nowhere near enough to claim the system works. Right now I'm trying to answer a much smaller question: is this idea promising enough that I should keep going?

I make some labels and give it some information about my personal reading preferences. As much as possible, I try to follow a consistent labeling rubric. This sounds really boring, but it will bite you in the butt if you don't think about it. Your labeling rules don't need to be perfect, but you must have rules that you apply consistently. Otherwise, your grading is completely random and doesn't help you.

For the first comparison I'm using OpenRouter, mostly because it lets me call models from several providers through one interface, and it's fast and easy. It's not perfect, but it's enough to get started with experimentation before investing a lot of time into something I don't even know is going to work.

What the MVP taught me

I had Claude run the test images through several different models and build a little report showing the results. For each shelf, I can see which titles the model found, what it recommended, how long the calls took, and roughly what they cost.

I looked at vision and recommendation separately, which turned out to be a good call, because different models perform differently on these tasks.

Reading the shelf: Claude Sonnet and Gemini Flash both found essentially all of the readable books on this tiny test set without inventing anything, while some of the smaller models missed a lot more titles and hallucinated books that were not on the shelf at all.

Cost turned out to be much less interesting than I expected. The models that performed well are comfortably inside the budget I wrote down during scoping, so shaving fractions of a cent off this part of the system is clearly not where I should be spending my time.

Latency is a different story. Some of these calls were already taking ten or twelve seconds just to read the shelf, before I'd asked another model to recommend anything, which is a problem when my target for the entire experience is around fifteen seconds. So after one experiment, I already had a much better idea of what might end up deserving optimization once we have more data.

Recommendations: Then I ran the recommendation models separately. This time every model got the same list of books and the same information about my preferences, which let me compare the recommendation step without mixing in mistakes from the vision model.

And this is where I got another result I was not expecting. One of the cheaper models, GPT-5.4 mini, matched my own book choices better than the more expensive models in this test, while Sonnet, which gave me some of the nicest-sounding explanations, actually performed worse against the metric I decided on beforehand.

Some other interesting things came up during the MVP too. I saw some quirks in how the models make recommendations depending on how I give them information about the reader's preferences, and I found some efficiency issues based on model settings I hadn't remembered to adjust. I also noticed a privacy problem: my source images had GPS information in the file metadata that I don't want to store.

So this MVP taught me some really important things:

• This is a legit problem that we can solve with the models available, at reasonable cost and latency.
• We should separate the image recognition and recommendation stages.
• I learned how to write a more effective prompt, which settings to use, and some privacy stuff I wouldn't have thought to worry about.

Go back and update the scope

Now, instead of guessing what the real app should look like, I can go back to the scope and update it based on what I actually learned. I might not have answers to every question that will come up as we build, but I certainly know enough to fill in most of the rest of the scoping doc.

I'm also now aware of some things that I'm pretty sure won't be part of the system at all.

• There's no RAG here, because I don't have a retrieval problem that needs one. The books are already coming from the current shelf, and the extra information I need about them can come from a normal catalogue lookup.
• I'm not using agents either, because there isn't anything for an agent to decide. The workflow is known ahead of time: read the shelf, verify the books, recommend five. Letting a model decide what happens next would mostly make a system that's already slower than I want even less predictable.
• I'm not fine-tuning anything, because off-the-shelf models are good enough as-is.

I think this is worth calling out, because when you're trying to make an AI project look impressive, there can be a real temptation to shove every AI technique you've learned into the same repo. You don't get extra engineering points because your architecture diagram has RAG, three agents, a fine-tuned model and six databases in it. If the problem doesn't need those things, leaving them out is the better decision.

Same goes for lots of cool product features that I could add but are definitely out of scope right now. We're not going to have logins or accounts. I'm not going to add purchase links. There are lots of neat things that could be useful in the future, but at least for V1, I'm not bothering with all that.

Working in phases

Now that I'm pretty sure what I'm going to build, the next thing I need to decide is how to break all of that work into pieces that can be built and tested without turning Claude loose on the entire application at once.

So I break the project into phases, and I try to make each phase leave me with something that still works end to end, just with one more new addition at a time.

Here are the kinds of components from the scoping doc I might add in each phase:

• Prompt engineering. I've done some of that already, but I want the prompts to become actual versioned parts of the system instead of random strings throughout my code. And as I improve them, I want a repeatable way to check whether those changes actually make the results better (more on that in the evaluation section below).
• Model selection. We already did a bit of this during the MVP, but we might do more as we develop the product.
• RAG. Depending on what you're building, you might have a RAG system. That means thinking about things like what information the model needs access to, how you split and retrieve it, and whether retrieval is actually giving the model the right context.
• Agents. Same thing with agents. If your system actually needs a model to choose between tools or decide what to do next, you'll need to design that workflow, handle failures, and make sure the agent can't get itself into some dumb loop.
• The product around the AI. At some point you'll probably need to turn the core AI pipeline into an actual product. That could mean an API, a user interface, authentication, file uploads, a database, or whatever else is necessary for somebody to actually use it.
• Reliability. Logging what happened, tracking errors, cost and latency, and collecting enough information that when something goes wrong, you can figure out why (more on that in the monitoring section below).
• More advanced things, like fine-tuning. But just like RAG or agents, that should solve a problem you actually found your system had. It's not a box you need to check to make the project look impressive.

The point is not that every project needs every one of these things. It's just that these are the kinds of components I can pull from as the project grows.

The exact phases are obviously going to look different for every project, but the thing I care about is the order. I want to deal with the risky parts early enough that they can still change the design. There isn't much point spending a week making a beautiful recommendation screen if I haven't figured out whether the recommendations are any good, and I definitely don't need a fancy monitoring setup for a workflow I may end up replacing.

For ShelfScanner:

1. Phase 1 was the model router and the automated quality check. This allowed me to confidently replace OpenRouter and easily switch out models from different providers.
2. Phase 2 was the smallest useful version of the actual app, where you just take a photo and see the titles it read. No recommendations yet, and still just local on my laptop.
3. Phase 3 was preferences, because obviously I need to know what people like in order to make recommendations.

And so on! I also put rough deadlines around each phase to help me avoid scope creep, where I keep adding random stuff for fun and never finish anything.

Building something like this, especially if you want to really deeply understand each component, takes time. I think this rebuild took me at least 100 commits, maybe more.

Now, obviously, we could build something like this really fast with AI, but I think this is a great opportunity to slow down and deeply learn about the decisions you're making before you let the AI actually make a spec for you.

For my build cohort, I'm estimating 10 to 15 hours a week over the course of 8 weeks, with about 6 weeks of actual building. But I expect the actual code and implementation time to be less than half of that. I expect people to spend the rest of that time essentially studying. Let's say I'm adding a RAG pipeline. I could just tell the agent, "Make my RAG system," or I could deliberately set that time aside to understand different embedding models, different chunking strategies and how retrieval works, and then ask the model to stress test the various options I'm considering.

As I go, I'm still using spec-driven development, so each change has a spec and a task list, and each commit is connected to a specific task within that spec. That makes it really easy to check that things are actually being completed as I intended, see where our progress is, look back at decisions that were made, and revert if needed.

And that brings me to the checks I use before I consider a change finished.

Evaluation

There are really two different kinds of correctness I care about in this project: the normal software around the AI needs to behave correctly, and the AI itself needs to keep producing answers that are good enough. These need different kinds of evaluation.

If I change the code that strips location data out of uploaded photos, that is normal deterministic software, and I can write a normal test for exactly what should happen.

If I change a prompt and then the vision model starts missing half the books on a tricky shelf, a normal unit test probably isn't going to tell me that.

That's what the evaluation set is for. So every meaningful change goes through both kinds of checks when they're relevant.

Normal software tests

On the normal software side, I test the pieces where I know what the correct behavior should be. Can somebody upload a valid image and get through the flow? Does the app reject things it shouldn't accept? Do saved books stay attached to the right person? And so on. As the project grows, I also test the full user flow in a browser, because individual pieces can all work perfectly while the product as a whole is broken.

I set up basic CI/CD using GitHub Actions, so every time I push a commit or open a pull request, all of my tests run. GitHub Actions are really simple YAML files where you define what should happen when a particular action happens in GitHub. Then, every time that action happens, the pipeline runs automatically: installing the relevant packages, then running my tests.

The AI evaluation set

Then I have a separate evaluation setup for the AI behavior. Every time I make a change that could affect the models, prompts, preprocessing, or anything else in that pipeline, I can rerun the evaluation and compare it against the version I currently trust.

To make this more reliable, I gathered more labeled photos. I took the original shelf photos and made deliberately worse versions with blur, glare, rotation and lower resolution. Then I added a bunch of completely different bookshelf photos, so I'm testing on a wider range of possible uses.

So if I make a change, I can run this evaluation set and see whether there's been a regression. For example, when I changed some things in the config, the evaluation flagged a failure, just like it should.

I can also use the evaluation set to automatically compare different versions of a prompt. As long as I have good labels and scoring rubrics, I'm able to test prompt changes systematically. This is only possible because I've versioned my prompts, which is one of the most important things that many beginners overlook. Even something as simple as prompt engineering should be approached like an experiment.

AI systems are weird and kinda annoying, because the code can be perfectly valid while the behavior gets worse. Nothing crashes when a new prompt causes the model to miss books or hallucinate.

So for this project I need both layers: normal software tests for the pieces where the correct behavior is deterministic, and the evaluation set for the parts where "correct" depends on what the models actually do.

One other thing I add is a step where the coding assistant reviews the change against the spec before it gets accepted, mostly as another set of eyes looking for something I missed.

So by the time a change makes it back into the main project:

• I know what it was supposed to do.
• The normal software checks pass.
• The AI evaluation still looks good, if the change can affect model behavior.
• The documentation still describes the system I actually have.

That sounds like a lot when I list everything out, but in practice most of it becomes fairly boring once it's automated, which is kinda the point.

Monitoring

Once I can catch a lot of these problems before I ship them, the next question is what happens when the app is running somewhere I'm not personally watching it, and the inputs stop coming from my nice labeled evaluation set.

Because production is going to find different problems. Sometimes people upload weird photos, or a provider gets slower or rate-limits me. I want something in place to tell me if a change I've made will break something in production, but I also need to know when unexpected things that I didn't do break production.

Which brings me to monitoring. We don't need some giant enterprise observability setup for my little bookshelf app. I mostly need enough information coming back from the system that if something changes, I can tell what changed and decide whether I care.

Luckily, I already did a lot of the annoying work earlier when I decided to save every model call. Those same records I used for the model comparison keep telling me which model handled a request, how long it took, what it cost, which prompt version it used, and whether the call failed.

With those logs, I made a very simple dashboard so I can look at what the system is doing. If I wanted to be fancy, I could set up automatic alerts when something unexpected happens.

Speaking of being fancy, I can even use failures to improve the system over time. If real users keep uploading a kind of shelf that fails in the same way, that probably belongs in the next version of my evaluation set. Once I can reproduce that failure in a controlled way, I can fix it and keep checking that it doesn't come back.

That's also why I collect simple feedback signals in the product, rather than trying to make every evaluation happen before launch. If somebody saves a recommendation, that is useful information. If they tell me a book is not for them, that is useful too.

That helps me make better recommendations and a better product over time. I can also get information about product performance and use it to figure out what to prioritize next. For example, if I notice that latency is higher than I'd like, that might make me prioritize adding caching when I otherwise wouldn't have.

The idea behind caching is pretty simple: if I've already looked up a book once, I save that result and reuse it the next time the same book shows up, instead of calling the external catalogue again.

The main takeaway here, though, is that I'm not adding every optimization right at the beginning for no reason. If I'm asked in an interview, "Why did you add a cache?" I don't want to say something like, "I don't know, production apps have caches." I want to be able to point to something I learned, or some theoretical justification I had for why it was actually important to prioritize at that point in the project.

Deployment

Once I feel pretty good that I have everything from my spec implemented in a way that passes all of my evaluations, and I have monitoring in place for the real app, it's finally time to get ShelfScanner off my laptop and somewhere other people can use it.

I use Vercel for all of my personal projects. You just connect your GitHub repo, and it literally takes a couple of clicks until you have a deployed project. Then, any time you make a change in GitHub, it deploys to Vercel automatically (assuming all the pre-deployment checks pass).

I already had a cute domain, shelfscanner.io, so I moved it over from the previous version to the rebuild and retired the old app.

And now I can finally use ShelfScanner the way I originally imagined it: walk up to a shelf, take a picture on my phone, let it figure out what books are there, and get recommendations based on what I actually like.

Now, clearly, this isn't a real mobile app. It's just a web app, and it's not perfect. In the interest of getting this out on time, I still used AI a lot more than I would have if I were approaching this as a real learning and portfolio project. Not everything is as optimized as I'd like, but it's good enough to show the process. There's a lot more I'd learn and optimize once I started using it and digging into all the little things that take it up to 100.

Presenting it as a portfolio project

But the thing is, when you just look at the app, you can't tell that I did all of this thinking and engineering work. That's all super hidden. If I sent just the link as a portfolio project, I'd miss out on a lot of the benefits I'd get from a proper presentation.

The default AI-generated README that was created as I went definitely doesn't do this well. So I spent some time writing a new README that does a better job explaining what I built, including the trade-offs, the functionality, and even little architecture diagrams.

At this point, it looks like a strong portfolio project, but what can make it even more impactful for your career is getting real users. So far, everything I've learned has come from my own testing, but I guarantee that when you check out ShelfScanner, you're going to find a ton of issues, and that's exactly the point of getting users. Even just a couple dozen people using your app will teach you so much and give you great stories for interviews.

And hey, if you get more than a couple dozen users, maybe a couple hundred or a couple thousand, that's starting to be pretty close to professional experience. I would argue it is professional experience at some point, which is great, because if you build a real app that has paying users and a legit customer base, I think the project can live in the experience section of your resume, not just the projects section.

This is how a portfolio project can help you overcome the catch-22 where you need experience to get experience. If you create a legitimate product, you can put it in your experience section and get that first job without anyone having to take a chance on you. You can make your own opportunities.

Not that you have to do that for an AI portfolio project to be useful. Like I said at the beginning, it is totally okay for a project to exist just so you can learn something.

But in order to learn something, you need to make your own project. So this is just a reminder: don't copy ShelfScanner. Copy the process.

And if you're trying to figure out what kind of project makes sense for where you are right now, watch my video on the five levels of AI engineering projects next.