Add experimental Python codegen CLI scaffold by jonathan343 · Pull Request #744 · smithy-lang/smithy-python

jonathan343 · 2026-07-19T03:18:30Z

Overview

This PR lays the foundation for a Python-native Smithy code generator. It adds an experimental smithy-python package with the command-line interface that will eventually generate Python clients and standalone types packages.

The CLI currently validates its inputs and then exits with an explicit “generation is not implemented yet” error. The existing Java generator remains the primary one while the Python implementation is developed incrementally.

Background

The Python generator will consume Smithy JSON AST models and produce packages compatible with the existing Smithy Python runtime. It is distributed separately from those runtime packages and is only needed during code generation.

Smithy’s run plugin invokes the generator as an external process, allowing it to integrate with standard Smithy builds without being loaded into the Smithy CLI or implemented in Java. Read Smithy run plugin docs for more info.

Changes Summary

Created a new smithy-python package.
Add generate client and generate types commands.
Support direct invocation using --model and --output.
- Note: This list of params will grow as in subsequent PRs
Support Smithy run plugin invocation through standard input and the provided plugin environment.
Validate invocation inputs and distinguish argument errors from I/O and generation failures.
Add unit tests covering CLI behavior, environment parsing, and the error hierarchy.
Document the generator architecture, CLI contract, and migration strategy.

CLI smoke tests

$ smithy-python --version
smithy-python 0.0.0

$ smithy-python generate --help
usage: smithy-python generate [-h] {client,types} ...

positional arguments:
  {client,types}
    client        Generate a client package
    types         Generate a standalone types package

options:
  -h, --help      show this help message and exit

Both generation commands accept a model and reach the expected placeholder behavior:

$ echo '{}' | smithy-python generate client --output /tmp/smithy-python-client
smithy-python: error: client generation is not implemented yet

$ echo '{}' | smithy-python generate types --output /tmp/smithy-python-types
smithy-python: error: types generation is not implemented yet

Both generation commands exit with status 1, as expected for the current implementation.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

jonathan343 · 2026-07-20T05:28:03Z

@@ -0,0 +1 @@
+# Changelog


Note for reviewers: Our release infra will automatically handle version bumping and changelog management.

JordonPhillips · 2026-07-20T13:25:41Z

I haven't talked much about this publicly since I'm still working on implementations and a blog post, but check a look at shape closures. This is a way to drive generation that doesn't root everything in a service.

What it means for you is relatively little, other than that you should tolerate generating without a set service. Current type codegen generates a synthetic one, but you shouldn't operate that way.

Computing a closure is beyond your scope as a non-java plugin. I would suggest rather that you generate what's given to you and put the onus on customers to use smithy-build transforms to narrow it down to what they want. Maybe later you could implement everything needed to compute it and relax that restriction.

Nice, thanks for making us aware of this.

I agree with you. I think not requiring --service <service_id> is also an improved user experience. It's pretty easy to detect on our end. For the AWS services, we've automated our infra to do smithy select --selector service to find it for us. I think we can do that at the Python layer here. The only issue I see here is if someone defines two services (not sure if smithy allows it). But in that case, we'd ask them to specify.

Yeah, that's basically how it works on the java side right now. You can ask that there be exactly one service. It may be possible to generate for multiple services, but there's no expectation.

JordonPhillips · 2026-07-20T13:25:49Z

+
+## Commands
+
+Generation is organized by artifact type:


But what if you could mix artifacts. You don't now - but you could. Types+client or types+server is an obvious combo that's easy to do without much difficulty.

smithy-python generate --modes client,types

You could default that to just be client since most will want that.

I'm a bit hesitant to combine multiple modes into one command since that complicates mode-specific options. I'm wondering if we instead could do something like smithy-python generate client ... --with-types or similar.

But honestly, I'm not sure what you mean by client+types compared to what we have now. Can you elaborate a bit?

The idea is to support related types that aren't part of the service's operations. For example, say a service wants to define the events they send out through SNS. Those wouldn't be part of the service shape closure, so if you're filtering down to that then you exclude those shapes.

Got it, that makes sense. The gap is that the Java client generator only emits the service closure (directive.connectedShapes()), so shapes like your SNS events example just get dropped silently today.

My first thought was to have the Python generator skip the closure entirely and generate every data shape in the AST. Before committing to that I ran it over all 431 published AWS models. It broke healthlake out of the box (leaked, unconnected shapes in a second namespace colliding with real ones) and pulled in orphan Cfn* types for two others, while the service closure reproduced the existing SDK surface on all 431. So I'm keeping the service closure as the default, which also matches what Java, smithy-ruby, and smithy-php do. The CLI prints a note when unconnected shapes get left out so it's not silent anymore.

Everything else you called out is in: --service is optional (we auto-detect when there's one), we don't synthesize a service, and types works without a service by just generating every data shape.

That does mean the "types outside the closure" case is deferred for now. I left room in the design doc for an all-shapes option or reading shapeClosures metadata later. I'd like to get moving on the core implementation and we can re-evaluate the interface before we cut a major version. Lmk if you see this as a blocker though and we can hash it out now.

JordonPhillips · 2026-07-20T13:52:17Z

+@dataclass(frozen=True, slots=True)
+class _Invocation:
+    artifact: str
+    model_source: bytes


A readable would be better here I think. You don't want to have to buffer all of that only to pass it into json.reads or whatever else.

I don't think a readable would provide much more value here. json.load just does fp.read() internally, and we need the whole AST in memory anyway to resolve shape references. Using an open readable also means owning when to close it. We can probably do that cleanly with a context manager, but seems unnecessary. Additionally, the file and stdin cases close differently so we'd also have to keep track of that.

Lmk if I'm missing something and happy to re-evaluate. Since this is all under the hood, we can also always add if later if we find a good reason to.

json.load just does fp.read() internally

Why is python like this.

As long as you discard the string as soon as possible in favor of a loaded dict or whatever then it's fine.

Alan4506 · 2026-07-20T14:42:58Z

+}
+```
+
+Artifact-specific options will be appended to `command` after they are defined.


This is more for my own understanding: the existing Java generator receives service, module, and moduleVersion as plugin settings for each generated client. So in the future are we going to add options like the following?

{ "run::python-client": { "command": [ "smithy-python", "generate", "client", "--service", "com.amazonaws.bedrockruntime#AmazonBedrockFrontendService", "--module", "aws_sdk_bedrock_runtime", "--module-version", "0.8.0" ] } }

Yup. I need to look into Jordon's comment to see if --service will still be needed. I'll also consider using --package-name which we derive the module name from w/ an optional override (.e.g a aws-sdk-s3 package name generates a aws_sdk_s3 module name). I also think --module-version should become --package-version.

We may also grow this overtime to add new options like --http-client aiohttp | httpx | awscrt

Note that you can also put these into environment variables. The run plugin supports putting stuff there. So you could also accept them from the environment if you want.

jonathan343

Thanks @JordonPhillips!

I addressed a few of your comments. Others might need clarification or further discussion.

jonathan343 · 2026-07-22T04:06:50Z

+
+## Commands
+
+Generation is organized by artifact type:


I'm a bit hesitant to combine multiple modes into one command since that complicates mode-specific options. I'm wondering if we instead could do something like smithy-python generate client ... --with-types or similar.

But honestly, I'm not sure what you mean by client+types compared to what we have now. Can you elaborate a bit?

jonathan343 · 2026-07-22T04:28:38Z

+@dataclass(frozen=True, slots=True)
+class _Invocation:
+    artifact: str
+    model_source: bytes


I don't think a readable would provide much more value here. json.load just does fp.read() internally, and we need the whole AST in memory anyway to resolve shape references. Using an open readable also means owning when to close it. We can probably do that cleanly with a context manager, but seems unnecessary. Additionally, the file and stdin cases close differently so we'd also have to keep track of that.

Lmk if I'm missing something and happy to re-evaluate. Since this is all under the hood, we can also always add if later if we find a good reason to.

jonathan343 · 2026-07-22T04:32:30Z

Nice, thanks for making us aware of this.

I agree with you. I think not requiring --service <service_id> is also an improved user experience. It's pretty easy to detect on our end. For the AWS services, we've automated our infra to do smithy select --selector service to find it for us. I think we can do that at the Python layer here. The only issue I see here is if someone defines two services (not sure if smithy allows it). But in that case, we'd ask them to specify.

jonathan343 · 2026-07-22T04:43:47Z

I'm so used to reviewing other people's PRs I accidentally submitted my responses as a review. Sorry if it looks weird. I see duplicate responses now.

Mark generation command examples as schematic and clarify option validation behavior. Exercise python -m smithy_python in a subprocess.

Document that --service is optional when the model contains a single service, that both artifacts generate every data shape in the model rather than the service closure, and that case-insensitive name conflicts are a hard error. Note that run plugin env settings may back command-line options.

Generating every shape in the model diverged from every other Smithy generator and fails on a published AWS model whose leaked, unconnected shapes collide with real ones. Document the service closure as the default selection, with a note when shapes are left out, and keep the whole-model behavior for the types artifact when no service is present.

stobrien89

just one minor comment

A missing --model path was pre-checked and reported as an invocation error with exit 2, while a file that existed but could not be read raised OSError and exited 1. The design classifies I/O failures as 1, and tools that distinguish usage errors from runtime failures, including the Smithy CLI, treat a missing input file as the latter. Drop the pre-check so every unreadable model exits 1 with a message that names the path and the cause. Also state in the design that model failures return 1, which the implementation already did but the text left implicit.

arandito

Had a tiny nit but looks good to me.

arandito · 2026-09-15T19:18:14Z

+    try:
+        _resolve_invocation(
+            args,
+            environ=os.environ if environ is None else environ,


tiny nit: The current path resolves environ twice: here and when PluginEnvironment.from_environ is called in _resolve_invocation. Is there a reason _resolve_invocation needs this to be resolved beforehand? If not, can we consolidate it?

Addressed in c7e2f67. This was actually done in the stacked PR, pulled it in here: c7e2f67

main substituted os.environ for a missing environ argument before passing it on, and PluginEnvironment.from_environ applied the same default again. Pass the argument through untouched so the fallback lives only in from_environ, and drop the now unused os import.

This PR has received approval from another team member.

jonathan343 requested a review from a team as a code owner July 19, 2026 03:18

jonathan343 commented Jul 20, 2026

View reviewed changes

JordonPhillips previously requested changes Jul 20, 2026

View reviewed changes

Alan4506 reviewed Jul 20, 2026

View reviewed changes

jonathan343 commented Jul 22, 2026

View reviewed changes

jonathan343 requested a review from JordonPhillips July 24, 2026 06:39

jonathan343 requested a review from a team September 12, 2026 05:57

jonathan343 mentioned this pull request Sep 13, 2026

Load Smithy models and select the shapes to generate #795

Draft

jonathan343 added this pull request to stack #796 September 13, 2026 06:47

jonathan343 added 8 commits September 13, 2026 02:57

Document Python code generator design

7d6de63

Add experimental Python codegen CLI

bc7ff3d

Remove redundant exception hierarchy test

cbe477c

Clarify CLI examples and test module entry point

0f09ece

Mark generation command examples as schematic and clarify option validation behavior. Exercise python -m smithy_python in a subprocess.

Address PR feedback

ba91974

docs(codegen): Document mixin resolution during model loading

e6b3026

jonathan343 force-pushed the add-smithy-python branch from 536e2c6 to e6b3026 Compare September 13, 2026 06:58

stobrien89 approved these changes Sep 13, 2026

View reviewed changes

Comment thread packages/smithy-python/src/smithy_python/cli.py Outdated

jonathan343 mentioned this pull request Sep 14, 2026

Replace the Java code generator with a Python implementation #797

Open

arandito reviewed Sep 15, 2026

View reviewed changes

arandito approved these changes Sep 15, 2026

View reviewed changes

jonathan343 merged commit 6ecbab2 into develop Sep 16, 2026
10 checks passed

Uh oh!

Conversation

jonathan343 commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Background

Changes Summary

CLI smoke tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonathan343 Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonathan343 Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonathan343 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jonathan343 Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jonathan343 commented Jul 22, 2026

Uh oh!

stobrien89 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

arandito left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

jonathan343 commented Jul 19, 2026 •

edited

Loading

jonathan343 Jul 22, 2026 •

edited

Loading

jonathan343 Sep 12, 2026 •

edited

Loading

jonathan343 Jul 22, 2026 •

edited

Loading