Skip to content

gh-144384: Lazily import _colorize#149318

Merged
hugovk merged 16 commits intopython:mainfrom
hugovk:3.15-lazy-_colorize
May 6, 2026
Merged

gh-144384: Lazily import _colorize#149318
hugovk merged 16 commits intopython:mainfrom
hugovk:3.15-lazy-_colorize

Conversation

@hugovk
Copy link
Copy Markdown
Member

@hugovk hugovk commented May 3, 2026

Lazily import _colorize to avoid slow construction/import when not needed.

The easily testable:

Before After
difflib: 10ms 2ms (80% faster)
image image
doctest: 48ms 40ms (17% faster)
image image
json.tool: 13ms 6ms (54% faster)
image image
pdb: 43ms 37ms (14% faster)
image image
traceback: 15ms 8ms (47% faster)
image image
unittest: 20ms 14ms (30% faster)
image image

Some indirect improvements:

Before After
asyncio: 47ms 27ms (43% faster)
image image
logging: 17ms 10ms (41% faster)
image image

traceback needs some extra handling: add a shutdown theme, so if it attempts to reify _colorize during shutdown when the import machinery is no longer around, it has a no-op fallback.

Also add ensure_lazy_imports tests.

@hugovk hugovk requested a review from berkerpeksag as a code owner May 3, 2026 14:32
@hugovk hugovk added the performance Performance or resource usage label May 3, 2026
@hugovk hugovk added the stdlib Standard Library Python modules in the Lib/ directory label May 3, 2026
@hugovk hugovk requested review from ambv and gaogaotiantian as code owners May 3, 2026 14:32
Copy link
Copy Markdown
Member

@picnixz picnixz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have a convention for lazy imports

Comment thread Lib/_pyrepl/fancycompleter.py Outdated
Comment thread Lib/traceback.py
Comment thread Lib/traceback.py
Copy link
Copy Markdown
Member Author

@hugovk hugovk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have a convention for lazy imports

Yes, given we don't have a convention, and the SC left it the details to linters and auto-formatters, let's generally follow the Ruff/isort convention:

import json
import os
import subprocess
from collections import defaultdict
from pathlib import Path
from typing import Final
lazy import ast
lazy import shutil
lazy from dataclasses import dataclass

Comment thread Lib/traceback.py
Comment thread Lib/traceback.py
Comment thread Lib/traceback.py
@StanFromIreland
Copy link
Copy Markdown
Member

Seeing #149321, I think this should wait till that is somewhat figured out.

@hugovk
Copy link
Copy Markdown
Member Author

hugovk commented May 4, 2026

Seeing #149321, I think this should wait till that is somewhat figured out.

#149338 has been merged, updating.

@hugovk
Copy link
Copy Markdown
Member Author

hugovk commented May 4, 2026

And both the commands from #149321 are working.


The CIFuzz failures are unrelated network errors:

413.7 E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/t/tcltk-defaults/tcl-dev_8.6.9+1_amd64.deb  Connection failed [IP: 185.125.190.81 80]

There's another outage: https://status.canonical.com/#/incident/KNms6QK9ewuzz-7xUsPsNylV20jEt5kyKsd8A-3ptQG1OfBTMZO_fBkVMmGqQC3NLBLamiI6KOEL7KpsobFJ4Q==

@hugovk
Copy link
Copy Markdown
Member Author

hugovk commented May 6, 2026

CIFuzz now also passing.

Copy link
Copy Markdown
Contributor

@sergey-miryanov sergey-miryanov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, but I'm not very familiar with lazy imports.

@JelleZijlstra
Copy link
Copy Markdown
Member

I'm not sure this makes sense for the pyrepl modules. Aren't those essentially always going to want _colorize?

@DavidCEllis
Copy link
Copy Markdown
Contributor

I'm not sure the sqlite.__main__ lazy import is worthwhile? While it's not imported at top level, main will immediately import it with ArgumentParser() or get_theme a little later.

@hugovk
Copy link
Copy Markdown
Member Author

hugovk commented May 6, 2026

I'm not sure this makes sense for the pyrepl modules. Aren't those essentially always going to want _colorize?

NO_COLOR=1 ./python.exe is a thing, and gives you a colour-free REPL.

But it still gets imported:

❯ NO_COLOR=1 ./python.exe
Python 3.15.0a8+ (heads/3.15-lazy-_colorize:2d346d5a1ea, May  6 2026, 16:40:10) [Clang 17.0.0 (clang-1700.6.4.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> "_colorize" in sys.modules
True

So I'll remove that change.

@hugovk
Copy link
Copy Markdown
Member Author

hugovk commented May 6, 2026

And reverted in sqlite3.__main__ as well.

@read-the-docs-community
Copy link
Copy Markdown

read-the-docs-community Bot commented May 6, 2026

@hugovk
Copy link
Copy Markdown
Member Author

hugovk commented May 6, 2026

Also reverted pdb, it imports it via pyrepl:

image

@JelleZijlstra
Copy link
Copy Markdown
Member

Yes, pdb also seemed questionable to me since if you use pdb you'll almost certainly want the prompt, which will give you colors.

Copy link
Copy Markdown
Member

@JelleZijlstra JelleZijlstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

json.tool is also unnecessary, it uses colorize in main() which is the only interface to that module.

I haven't looked much into profiling, are there plausible code paths where that is being used that don't go through colorize? Same for unittest and doctest.

difflib and traceback seem OK, they're broad libraries and there are plenty of functions in there that don't need color.

@bedevere-app
Copy link
Copy Markdown

bedevere-app Bot commented May 6, 2026

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

Copy link
Copy Markdown
Member

@JelleZijlstra JelleZijlstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple still seem questionable but the cost of an extra lazy import isn't that high, so I'm OK with this going in.

@hugovk hugovk enabled auto-merge (squash) May 6, 2026 16:00
@hugovk hugovk merged commit 7cea70e into python:main May 6, 2026
67 of 68 checks passed
@hugovk hugovk deleted the 3.15-lazy-_colorize branch May 6, 2026 16:07
@hugovk
Copy link
Copy Markdown
Member Author

hugovk commented May 6, 2026

Thanks all!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Performance or resource usage stdlib Standard Library Python modules in the Lib/ directory

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants