Filter the stateless-install project copy using .gitignore by dmargol1 · Pull Request #2272 · dataform-co/dataform

dmargol1 · 2026-09-03T01:09:44Z

Per @kolina's comment on the issue ("Approach with .gitignore seems more effective"), this takes the .gitignore-aware route rather than a hardcoded exclude list. Following review, it also supports an optional .dataformignore.

The problem

When dataformCoreVersion is set in workflow_settings.yaml, compile() copies the whole project directory to a temp dir before running npm i there:

fs.copySync(resolvedProjectPath, temporaryProjectPath);

No filter, so the copy's cost scales with everything under the project root rather than with the Dataform project itself. In a mixed-tooling repo, a Python .venv sitting next to definitions/ gets copied byte-for-byte on every compile.

The change

Adds buildProjectCopyFilter(), an fs-extra copySync filter built from:

The project's root .gitignore, if present. dataform init already writes one, but nothing in the compile pipeline consulted it until now.
An optional root .dataformignore, in the same syntax, read after .gitignore into the same matcher. It can exclude further paths, or un-ignore gitignored ones with !pattern, e.g. !definitions/generated/ for definitions generated into a gitignored directory.
Two fixed rules that neither ignore file can override: .git and node_modules are always excluded, and the root workflow_settings.yaml is always copied, since compile() has already read it and compilation can't proceed without it.

Using the project's .gitignore avoids having to enumerate every ecosystem's junk directories (.venv, target/, __pycache__/, vendor/, coverage/, ...). A project's .gitignore already states what that project treats as disposable.

The stateless-install guarantee is unchanged: core is still freshly installed into the temp dir on every compile.

Behavior change

An ignored file is no longer copied, so it is also no longer compiled. A project that generates definitions into a gitignored path needs to un-ignore that path, which .dataformignore now allows without touching .gitignore. As in git, a file can't be re-included while its parent directory is still excluded, so the directory itself must be un-ignored.

Two things soften it:

Verbose mode prints which ignore files were applied, so a user debugging a missing file has a thread to pull.
A project with neither ignore file is unaffected beyond the .git/node_modules exclusion.

Notes on the implementation

Case sensitivity follows the filesystem, as git does with core.ignorecase. The filter probes whether the project directory is case-insensitive (looking up an existing entry under its case-swapped name, falling back to the platform default) and applies that to pattern matching, negations, and both fixed rules. ignore matches case-insensitively by default, which on Linux would have let definitions/staging/ silently drop definitions/Staging/*.sqlx.
lstatSync, not statSync, classifies directories for trailing-slash patterns like .venv/. statSync follows symlinks and throws ENOENT on a dangling one, which is a perfectly copyable input. lstat also matches what copySync itself does by default.
Only root ignore files are read. Nested .gitignore files, .git/info/exclude and the user's global excludes are not consulted, so this under-excludes relative to git status rather than over-excluding. A .gitignore/.dataformignore that isn't a regular file is skipped.
git's index is not consulted, so a file that git tracks despite matching a pattern (e.g. via git add -f) is still excluded. Documented in the code.
A top-level node_modules cannot actually be present on this path, since compile() rejects the project before copying if it finds one. That fixed exclusion therefore covers nested ones.
No lockfile change is needed. ignore@^5.2.0 already resolves to 5.3.2 in yarn.lock as an existing transitive dependency; a clean yarn install leaves the lockfile byte-identical. ignore is declared under dependencies and listed in the CLI bundle's externals.

Testing

19 unit tests in cli/api/commands/compile_copy_filter_test.ts, covering .gitignore and .dataformignore separately and together, un-ignoring (including the parent-directory limit), the fixed rules against negations, both case modes (tests pin the mode explicitly so they behave the same on every platform) plus the detection itself, dangling symlinks, in-project paths beginning with two dots, ignore-file names that are directories, and integration-style tests that run a real copySync.

bazel test //cli/api/...
Executed 12 out of 12 tests: 12 tests pass.

./scripts/lint is clean, including its yarn --frozen-lockfile step.

Case-insensitive detection has only been exercised on Linux; the case-insensitive code paths are covered by the forced-mode tests.

🤖 Generated with Claude Code

google-cla · 2026-09-03T01:10:08Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

rafal-hawrylak · 2026-09-23T17:19:53Z

Please rebase and and solve conflicts

rafal-hawrylak · 2026-09-23T17:21:18Z

    "fs-extra",
    "glob",
    "google-sql-syntax-ts",
+    "ignore",


yarn.lock update is missing

ignore@^5.2.0 already has an entry in yarn.lock (resolving to 5.3.2) because other packages depend on it. The range I declared matches that entry exactly, so yarn has nothing to add. I checked by running yarn install from scratch on the rebased branch: the lockfile came out byte-identical, and the yarn --frozen-lockfile step in ./scripts/lint passes. I also moved ignore into the new dependencies section added in #2338, and the generated @dataform/cli package.json now lists it. If you'd prefer a pinned version, like other recent dependency changes, I'm happy to switch.

rafal-hawrylak · 2026-09-23T17:24:15Z

+ */
+export function buildProjectCopyFilter(resolvedProjectPath: string): (src: string) => boolean {
+  const ig = ignore();
+  const gitignorePath = path.join(resolvedProjectPath, ".gitignore");


Supporting .gitignore by default is the right step for 99% of users. We could add a new support to a .dataformignore file (similar to .dockerignore / .eslintignore). If .dataformignore exists, it could either override or supplement .gitignore (allowing users to un-ignore paths specifically for Dataform).

Good idea, and it addresses the one real downside of this PR, so it's done here. .dataformignore supplements .gitignore: it's read second into the same matcher, so it can exclude more paths or un-ignore gitignored ones with !pattern. I chose supplement over override so users don't have to copy their whole .gitignore to change one path.

A few details, all documented and tested:

As in git, a file can't be re-included while its parent directory is still excluded, so users un-ignore the directory (!definitions/generated/) rather than a single file inside it.

Neither file can un-ignore .git or node_modules, and workflow_settings.yaml is always copied, since compilation can't run without it.

Pattern matching follows the filesystem's case sensitivity, as git does with core.ignorecase. The ignore library ignores case by default, which on Linux would have let definitions/staging/ silently drop definitions/Staging/*.sqlx.

Patterns are evaluated without consulting git's index, so a file that git tracks despite matching a pattern (e.g. via git add -f) is still excluded.

rafal-hawrylak · 2026-09-23T17:25:18Z

    if (compileConfig.verbose) {
      print(`Using isolated environment for @dataform/core@${workflowSettingsDataformCoreVersion}\n`);
      print(`Copying project to temporary directory: ${temporaryProjectPath}\n`);
+      print(`Excluding .git, node_modules, and paths matched by the project's .gitignore\n`);


If the project has no .gitignore, this log message still claims it is excluding paths from .gitignore. A minor conditional check (hasGitignore ? ... : ...) would be cleaner.

Fixed. The verbose log now names the ignore files it actually found, e.g. "paths matched by: .gitignore, .dataformignore". When there are none, it says "Excluding .git and node_modules (no .gitignore or .dataformignore in project root)".

Fixes dataform-co#2269. When dataformCoreVersion is set in workflow_settings.yaml, compile() copies the whole project directory to a temporary directory before running `npm i` there. That copy had no filter, so its cost scaled with everything under the project root -- most commonly a large .venv or build-output directory sitting next to definitions/ -- rather than with the Dataform project itself. Adds buildProjectCopyFilter(), an fs-extra copySync filter built from the project's own root .gitignore (already written by `dataform init`, but not previously consulted anywhere in the compile pipeline), plus a small always-ignored floor of .git and node_modules that a .gitignore negation pattern cannot override. Using the project's .gitignore rather than a hardcoded list of directory names avoids having to enumerate every ecosystem's junk directories (.venv, target/, __pycache__/, vendor/, coverage/, ...), since a project's .gitignore already states what that project treats as disposable. Behavior change worth calling out in review: a gitignored file is no longer copied, so it is also no longer compiled. A project that generates definitions into a gitignored path now has to unignore that path. Verbose mode prints which exclusions are applied. Projects with no .gitignore are unaffected beyond the .git/node_modules floor. No lockfile regeneration is needed: `ignore` is already resolved in yarn.lock at 5.3.2 as an existing transitive dependency, and the declared ^5.2.0 range matches that entry.

- Read an optional .dataformignore from the project root after the .gitignore, into the same ignore instance. It can exclude further paths, or un-ignore gitignored ones with `!pattern` (for example, definitions generated into a gitignored directory). The .git and node_modules floor still cannot be overridden by either file. - The verbose log now names the ignore files actually found, and no longer mentions a .gitignore when the project has none. - Reformat the new files with the repo's Prettier 3 config.

- ignore() matches case-insensitively by default, so on a case-sensitive filesystem a `definitions/staging/` pattern also dropped `definitions/Staging/table.sqlx`, which git keeps. Always match case-sensitively: at worst this copies a little extra on case-insensitive filesystems, and never drops a definition. - Document that, as in git, a file can't be re-included while an ancestor directory is still excluded, with a concrete example, and test the child-only negation case.

- Compare the always-ignored .git/node_modules names case-insensitively, so .GIT or NODE_MODULES on a case-insensitive filesystem is excluded. - Always copy the root workflow_settings.yaml. compile() has already read it from the original project, and compilation can't proceed without it, so a broad pattern like `*.yaml` must not drop it. - Skip a .gitignore/.dataformignore that isn't a regular file, instead of failing the compile with EISDIR. - Document that patterns are evaluated without consulting git's index, so force-added tracked files matching a pattern are still excluded.

A fixed case policy was wrong in both directions: always-sensitive matching let a mixed-case negation miss its file on a case-insensitive filesystem, and always-folding the .git/node_modules floor excluded a legitimate NODE_MODULES directory on Linux. Detect once whether the project directory is case-insensitive (probing an existing entry under its case-swapped name, falling back to the platform default), and apply that to the ignore matcher, the always-ignored floor and the workflow_settings.yaml allowlist, as git does with core.ignorecase. Tests pin each mode explicitly so they behave the same on every platform.

dmargol1 · 2026-09-25T00:02:40Z

Rebased onto main and conflicts resolved. The only conflict was package.json, where ignore now goes under the new dependencies section. I've also addressed the inline comments; replies are on each thread.

dmargol1 force-pushed the fix/2269-filter-stateless-install-copy branch from d7a6df2 to e5f9f52 Compare September 3, 2026 01:12

dmargol1 marked this pull request as ready for review September 4, 2026 22:41

dmargol1 requested a review from a team as a code owner September 4, 2026 22:41

dmargol1 requested review from rafal-hawrylak and removed request for a team September 4, 2026 22:41

dmargol1 mentioned this pull request Sep 4, 2026

Stateless install copies entire project directory unfiltered, scaling compile time with unrelated files #2269

Open

rafal-hawrylak requested changes Sep 23, 2026

View reviewed changes

dmargol1 added 2 commits September 24, 2026 18:16

dmargol1 force-pushed the fix/2269-filter-stateless-install-copy branch from e5f9f52 to e261524 Compare September 24, 2026 23:19

dmargol1 added 3 commits September 24, 2026 18:49

dmargol1 force-pushed the fix/2269-filter-stateless-install-copy branch from 3981f2b to 2a61411 Compare September 25, 2026 00:06

dmargol1 requested a review from rafal-hawrylak September 25, 2026 00:07

Navigation Menu

Filter the stateless-install project copy using .gitignore - #2272

Filter the stateless-install project copy using .gitignore#2272
dmargol1 wants to merge 5 commits into
dataform-co:mainfrom
dmargol1:fix/2269-filter-stateless-install-copy

dmargol1 commented Sep 3, 2026 •

edited

Loading

Uh oh!

google-cla Bot commented Sep 3, 2026

Uh oh!

rafal-hawrylak commented Sep 23, 2026

Uh oh!

rafal-hawrylak Sep 23, 2026

Uh oh!

dmargol1 Sep 25, 2026

Uh oh!

rafal-hawrylak Sep 23, 2026

Uh oh!

dmargol1 Sep 25, 2026

Uh oh!

rafal-hawrylak Sep 23, 2026

Uh oh!

dmargol1 Sep 25, 2026

Uh oh!

dmargol1 commented Sep 25, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dmargol1 commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The problem

The change

Behavior change

Notes on the implementation

Testing

Uh oh!

google-cla Bot commented Sep 3, 2026

Uh oh!

rafal-hawrylak commented Sep 23, 2026

Uh oh!

rafal-hawrylak Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

dmargol1 Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

rafal-hawrylak Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

dmargol1 Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

rafal-hawrylak Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

dmargol1 Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

dmargol1 commented Sep 25, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dmargol1 commented Sep 3, 2026 •

edited

Loading