Skip to content

lexer: Update to Unicode 18 - #165289

Open
clubby789 wants to merge 1 commit into
rust-lang:mainfrom
clubby789:unicode-18
Open

lexer: Update to Unicode 18#165289
clubby789 wants to merge 1 commit into
rust-lang:mainfrom
clubby789:unicode-18

Conversation

@clubby789

@clubby789 clubby789 commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Split out of #160261.
Reference update: rust-lang/reference#2399

This bumps:

  • unicode-ident: 1.0.24 -> 1.0.27
  • unicode-normalization: 0.1.25 -> 0.1.26
  • unicode-properties: 0.1.4 -> 0.1.5
  • unicode-script: 0.5.7 -> 0.5.9
  • unicode-security: 0.1.2 -> 0.1.3
  • unicode-width: 0.2.2 -> 0.2.3

Previous bump: #148321

@rustbot

rustbot commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator

If the Unicode version changes are intentional,
it should also be updated in the reference at
https://github.com/rust-lang/reference/blob/HEAD/src/identifiers.md.

cc @rust-lang/lang-docs

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

@rustbot rustbot added A-CI Area: Our Github Actions CI A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Nov 1, 2026
@rustbot

rustbot commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

  • Owners of files modified in this PR: @Mark-Simulacrum

@clubby789 clubby789 changed the title Update to Unicode 18 lexer: Update to Unicode 18 Nov 1, 2026
@clubby789

clubby789 commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

r? compiler

@rustbot rustbot assigned JohnTitor and unassigned Mark-Simulacrum Nov 1, 2026
@clubby789 clubby789 mentioned this pull request Nov 1, 2026
Open
@clubby789

clubby789 commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Just in case: @bors try @rust-timer queue

@rust-timer

This comment has been minimized.

Sign in to view
@rustbot rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Nov 1, 2026
rust-bors Bot pushed a commit that referenced this pull request Nov 1, 2026
@bors
Auto merge of #165289 - clubby789:unicode-18, r=<try>
c2c9c6a 
lexer: Update to Unicode 18
@rust-bors

This comment has been minimized.

Sign in to view
@clubby789 clubby789 added A-Unicode Area: Unicode T-lang Relevant to the language team and removed A-testsuite Area: The testsuite used to check the correctness of rustc T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. A-CI Area: Our Github Actions CI labels Nov 1, 2026
@fmease fmease added I-lang-nominated Nominated for discussion during a lang team meeting. I-lang-easy-decision Issue: The decision needed by the team is conjectured to be easy; this does not imply nomination needs-fcp This change is insta-stable, or significant enough to need a team FCP to proceed. labels Nov 1, 2026
@fmease

fmease commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

For T-lang:

If I may be so bold:

@rfcbot merge lang

@rust-rfcbot

rust-rfcbot commented Nov 1, 2026 •
edited by joshtriplett
Loading

Copy link
Copy Markdown
Collaborator

@fmease has proposed to merge this. The next step is review by the rest of the tagged team members:

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

@fmease fmease assigned fmease and unassigned JohnTitor Nov 1, 2026
@joshtriplett

joshtriplett commented Nov 1, 2026

Copy link
Copy Markdown
Member

Looks reasonable to me.

@rust-bors

rust-bors Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

☀️ Try build successful (CI)
Build commit: c2c9c6a (c2c9c6a1e56929525b02f8799e54fe88f3b6823b)
Base parent: 3b5aab2 (3b5aab2cea0711a2cc87a5a71338b5dba241b6a9)

@rust-timer

This comment has been minimized.

Sign in to view
@clubby789 clubby789 mentioned this pull request Nov 1, 2026
Open
@rust-timer

rust-timer commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator

Finished benchmarking commit (c2c9c6a): comparison URL.

Overall result: ❌✅ regressions and improvements - no action needed

Benchmarking means the PR may be perf-sensitive. Consider adding rollup=never if this change is not fit for rolling up.

@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

mean range count
Regressions ❌
(primary)		 - - 0
Regressions ❌
(secondary)		 0.2% [0.1%, 0.4%] 2
Improvements ✅
(primary)		 - - 0
Improvements ✅
(secondary)		 -0.3% [-0.3%, -0.3%] 1
All ❌✅ (primary) - - 0

Max RSS (memory usage)

Results (primary 2.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

mean range count
Regressions ❌
(primary)		 2.0% [2.0%, 2.0%] 1
Regressions ❌
(secondary)		 - - 0
Improvements ✅
(primary)		 - - 0
Improvements ✅
(secondary)		 - - 0
All ❌✅ (primary) 2.0% [2.0%, 2.0%] 1

Cycles

Results (secondary 0.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

mean range count
Regressions ❌
(primary)		 - - 0
Regressions ❌
(secondary)		 2.7% [2.7%, 2.7%] 1
Improvements ✅
(primary)		 - - 0
Improvements ✅
(secondary)		 -2.3% [-2.3%, -2.3%] 1
All ❌✅ (primary) - - 0

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 487.913s -> 486.726s (-0.24%)
Artifact size: 406.78 MiB -> 406.83 MiB (0.01%)

@rustbot rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Nov 1, 2026
@fmease fmease added the needs-reference-pr This language change needs an approved Reference PR to proceed. label Nov 2, 2026
@rustbot rustbot added A-CI Area: Our Github Actions CI A-testsuite Area: The testsuite used to check the correctness of rustc T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Nov 2, 2026
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@clubby789

clubby789 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

The unicode-security update has introduced some new diagnostics into existing tests, which look reasonable to me

@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rustbot rustbot added the T-rustdoc-frontend Relevant to the rustdoc-frontend team, which will review and decide on the web UI/UX output. label Nov 2, 2026
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rustbot

rustbot commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

clippy is developed in its own repository. If possible, consider making this change to rust-lang/rust-clippy instead.

cc @rust-lang/clippy

@rustbot rustbot added the T-clippy Relevant to the Clippy team. label Nov 2, 2026
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@clubby789 clubby789 mentioned this pull request Nov 2, 2026
Merged
@clubby789

clubby789 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Will require rust-lang/cargo#17676 to land in the r-l/r tree for CI to pass

ranger-ross pushed a commit to ranger-ross/cargo that referenced this pull request Nov 3, 2026
@weihanglo
Allow uncommon_codepoints in confusable test (rust-lang#17676)
2be5575 
rust-lang/rust#165289 updates the Unicode
version used by the lexer to 18, which includes a new set of characters
firing `uncommon_codepoints`. Allow this in a test to prevent failures
there.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@fmease fmease

Labels

A-CI Area: Our Github Actions CI A-testsuite Area: The testsuite used to check the correctness of rustc A-Unicode Area: Unicode disposition-merge This issue / PR is in PFCP or FCP with a disposition to merge it. I-lang-easy-decision Issue: The decision needed by the team is conjectured to be easy; this does not imply nomination I-lang-nominated Nominated for discussion during a lang team meeting. needs-reference-pr This language change needs an approved Reference PR to proceed. proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-clippy Relevant to the Clippy team. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. T-lang Relevant to the language team T-rustdoc-frontend Relevant to the rustdoc-frontend team, which will review and decide on the web UI/UX output.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

10 participants

@clubby789 @rustbot @rust-timer @fmease @rust-rfcbot @joshtriplett @rust-log-analyzer @Mark-Simulacrum @JohnTitor @jamie-osec