The first version of my image generation pipeline was mostly what you would expect.

Give the model a source image, a logo, some text, brand context, and instructions describing the final composition. Let it generate the image.

It worked surprisingly well.

Until I needed it to work repeatedly.

That difference became much more obvious once I started using the pipeline behind KeepPost, where the goal isn't to generate one impressive image. The same pipeline has to produce content for different brands, different source photos, different formats, and different campaigns.

At that point, a visually good result is not enough.

The logo has to be where it is supposed to be. The product in the source image shouldn't suddenly change. Required text needs to survive generation. Brand rules need to stay consistent across posts.

And if I run a similar request tomorrow, I don't want the entire composition strategy to change because the model found another interpretation equally reasonable.

I initially tried solving most of this through prompting.

Eventually I stopped.

The problem wasn't that the prompts weren't detailed enough. I was asking a probabilistic model to make decisions that didn't need to be probabilistic in the first place.

The model is not my layout engine

This became the most useful distinction in the pipeline.

There are decisions where I want the model to be creative.

Lighting is a good example. So is adapting a restaurant photo into a more polished scene, extending the background, integrating an object naturally, or deciding how the generated environment should feel.

Those are exactly the things I want from an image model.

But there are other decisions where creativity adds almost no value.

If the brand logo belongs at a specific position and size, I don't need the model to reinterpret that requirement on every generation.

The same applies to things like output dimensions, protected areas, mandatory assets, exact text, and some composition constraints.

So the pipeline gradually moved away from:

(Assets + Prompt) -> Image Model -> Final Image

toward something closer to:

Now the position isn't something the prompt has to persuade the model to remember.

It's a constraint.

The same idea applies to other parts of the system:

DecisionTypeOutput aspect ratioDeterministicRequired assetsDeterministicLogo placementDeterministicProtected regionsDeterministicMandatory textDeterministic or validatedScene adaptationGenerativeLightingGenerativeBackgroundGenerativeObject integrationGenerativeCreative directionGenerative

This looks obvious when written down.

It wasn't as obvious when I started building it.

When image models become better at following instructions, the natural reaction is to give them more responsibility. In some parts of the pipeline I ended up doing the opposite.

Better generation made the creative parts better.

It didn't make deterministic requirements any less deterministic.

Some decisions sit in the middle

The interesting cases aren't always purely deterministic or purely generative.

Composition is one of them.

A model can be useful for understanding that the main subject occupies the right side of an image and that placing text there would cover the product.

That's a semantic decision. Hard-coded coordinates alone aren't particularly good at it.

But I also don't necessarily want the model to have unrestricted control over what happens next.

One approach is to let the model propose a composition plan as structured data:

{
  "subject_region": {
    "x": 0.36,
    "y": 0.16,
    "width": 0.52,
    "height": 0.70
  },
  "preferred_text_region": "upper_left",
  "background_extension": true,
  "creative_direction": "fine_dining"
}

That output can be validated before it reaches the generation stage.

This gives me a useful third category: the model proposes, code decides whether the proposal is valid.

I've found this pattern useful beyond image generation as well.

It is often a better boundary than either extreme: encoding every decision manually or giving an agent/model complete control.

Prompting wasn't the right fix for every failure

One of the traps with generative systems is treating every bad output as a prompt problem.

Logo moved? Improve the prompt.

Product changed? Add stronger instructions.

Text came back wrong? Make the requirement more explicit.

Composition changed too much? Add another paragraph describing the layout.

Sometimes that works.

But the prompt also gets longer, different constraints start interacting, and you're still relying on the model to reproduce behavior that your application could enforce directly.

I started asking a different question when something failed:

Does the model actually need to decide this?

If the answer is no, I would rather move that responsibility out of the prompt.

That turned out to be a much more useful debugging question than continuously rewriting instructions.

The source image is another boundary

For KeepPost, the source image often represents something real: a dish, product, venue, bottle, or other customer asset.

That changes how I think about generation.

The source isn't just inspiration.

It's input data that needs to preserve some truth.

If a restaurant sends a photo of a particular dish, generating a more attractive image while changing the dish itself can produce a technically impressive result that is useless for the customer.

So I treat source preservation differently from creative direction.

The model can have significant freedom around the subject while having much less freedom over the subject itself.

This distinction matters because prompts such as "keep the product unchanged" sound precise to us, but they're still instructions interpreted by a generative model.

When preservation really matters, the architecture should reflect that instead of assuming a stronger sentence in the prompt will solve everything.

Brand context is not the same as layout

Another thing I separated over time was brand understanding from deterministic brand rules.

Brand context can describe things like:

• visual mood
• typography preferences
• lighting
• palette
• composition tendencies
• use of negative space
• photography style

Those are useful signals for generation.

But "the logo goes here" is not really brand context.

It's a rule.

Mixing both into one large prompt technically works, but it hides two very different types of information inside natural language.

I prefer to let retrieved brand context influence creative decisions while keeping strict requirements represented explicitly wherever possible.

This makes the system easier to reason about when something goes wrong.

If an image has the wrong mood, I look at the generative context.

If the logo is in the wrong place, I shouldn't need to debug the model's interpretation of the brand guide.

Validation matters more once generation becomes automated

When you're manually generating an image, validation is easy.

You look at it.

If it's bad, you regenerate it.

That doesn't scale particularly well once image generation sits inside a larger automated workflow.

Now the pipeline needs to care about things such as:

• Did generation succeed?
• Is the expected output format correct?
• Are required elements present?
• Were protected assets preserved?
• Does the result satisfy composition constraints?
• Should this output be accepted, repaired, or regenerated?

Not every one of these checks can be perfectly deterministic, especially when you're evaluating visual semantics.

But that doesn't mean there shouldn't be a validation layer.

Generation and acceptance are separate problems.

I think this distinction becomes increasingly important as AI pipelines move from interactive tools into background automation.

The model saying "done" is not the same thing as the application deciding that the result is acceptable.

This is really a control problem

I originally thought most of the engineering work would be about getting better images from the model.

Some of it is.

But once the quality became good enough, another problem became more important: controlling variance.

There are places where variance is valuable.

I want the model to find a better way to light a dish. I want it to integrate an object naturally. I want it to understand the scene rather than follow a fixed Photoshop template.

There are also places where variance is simply another failure mode.

I don't want a creative interpretation of a logo coordinate.

The architecture became easier to reason about when I stopped treating "AI image generation" as one operation and started treating it as a pipeline containing different kinds of decisions.

Some belong to the model.

Some belong to code.

Some can be proposed by the model and verified by code.

And a few still need human judgment.

The same pattern shows up outside image generation

This lesson has changed how I look at other LLM systems too.

An agent doesn't need to decide something simply because it can.

A RAG pipeline doesn't need an LLM for a transformation that deterministic code handles more reliably.

A structured output schema shouldn't be replaced by instructions asking the model to "please always return these fields."

And business rules shouldn't quietly migrate into prompts just because adding another sentence is faster than changing the application.

The useful question isn't:

How much can I make the model do?

It's:

Where does probabilistic reasoning actually improve the system?

That's a very different architecture question.

For my image pipeline, the answer wasn't "everywhere."

The more I worked on it, the more I ended up giving the model freedom exactly where its uncertainty was useful, and removing freedom where the application already knew the correct answer.

The resulting pipeline is less magical than the original one.

That's probably why I trust it more.