Skip to content

Modernize Python tooling (uv, pyproject.toml, src layout, semantic release) #563

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
salman2013 wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Modernize Python tooling (uv, pyproject.toml, src layout, semantic release) #563

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
269e9f5
build: migrate package metadata to pyproject.toml (PEP 621)
salman2013 Aug 6, 2026
e698ff7
build: switch dependency management from pip-tools to uv
salman2013 Aug 6, 2026
19174c3
build: add python-semantic-release and update release workflow
salman2013 Aug 6, 2026
f056a2d
refactor: migrate to src layout
salman2013 Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions .coveragerc
View file
Open in desktop

This file was deleted.

22 changes: 9 additions & 13 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ on:
pull_request:
branches:
- '**'

workflow_call:

jobs:
run_tests:
name: tests
name: ${{ matrix.toxenv }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
Expand All @@ -20,18 +20,14 @@ jobs:

steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0

@irfanuddinahmad irfanuddinahmad Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing fetch-depth: 0. Without it, setuptools-scm can't see git tags during test runs and falls back to fallback_version (see the pyproject.toml comment) on every CI run, not just release.yml. release.yml's own checkout correctly has this set.

- name: setup python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
- name: Setup uv
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
python-version: ${{ matrix.python-version }}

- name: Install pip
run: pip install -r requirements/pip.txt
enable-cache: true
python-version: "${{ matrix.python-version }}"

- name: Install Dependencies
run: pip install -r requirements/ci.txt
- name: Install CI dependencies
run: uv sync --group ci

- name: Run Tests
env:
TOXENV: ${{ matrix.toxenv }}
run: tox
run: uv run tox -e ${{ matrix.toxenv }}
31 changes: 0 additions & 31 deletions .github/workflows/pypi-publish.yml
View file
Open in desktop

This file was deleted.

56 changes: 56 additions & 0 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
name: Semantic Release

on:
push:
branches: [master]

jobs:
run_tests:
uses: ./.github/workflows/ci.yml
secrets: inherit

release:
needs: run_tests
if: github.ref_name == 'master'
runs-on: ubuntu-latest
concurrency:
group: ${{ github.workflow }}-release
cancel-in-progress: false
permissions:
contents: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
fetch-depth: 0
ref: ${{ github.ref_name }}
- run: git reset --hard ${{ github.sha }}
- name: Python Semantic Release
id: release
uses: python-semantic-release/python-semantic-release@350c48fcb3ffcdfd2e0a235206bc2ecea6b69df0 # v10.5.3
with:
git_committer_name: "github-actions"
git_committer_email: "github-actions@github.com"
changelog: "false"

@irfanuddinahmad irfanuddinahmad Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changelog: "false" disables changelog generation entirely -- this is the exact setting that caused real CHANGELOG.rst content loss in a sibling repo earlier in this effort. Recommend removing this line and configuring [tool.semantic_release.changelog] in pyproject.toml instead (mode = "update", insertion_flag matching a .. changelog-insertion-marker line at the top of CHANGELOG.rst).

- name: Upload dist
if: steps.release.outputs.released == 'true'
uses: actions/upload-artifact@601987b96cf95d5f9a5fe5e0ec0e34c5583d7fc6 # v4.4.3

@irfanuddinahmad irfanuddinahmad Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a fabricated commit SHA -- gh api repos/actions/upload-artifact/commits/601987b96cf95d5f9a5fe5e0ec0e34c5583d7fc6 returns 422 No commit found. The action can't even resolve; this job will fail to start. The real commit SHA for v4.4.3 (verified via git ls-remote --tags + confirmed as a real commit) is b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882:

uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3

with:
name: dist
path: dist/
outputs:
released: ${{ steps.release.outputs.released }}
version: ${{ steps.release.outputs.version }}

publish_to_pypi:
needs: release
if: needs.release.outputs.released == 'true'
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
steps:
- uses: actions/download-artifact@fa0a91b85d4f404e444306234e16dac3d6cce7b0 # v4.1.8

@irfanuddinahmad irfanuddinahmad Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same issue -- fabricated SHA, verified via gh api repos/actions/download-artifact/commits/fa0a91b85d4f404e444306234e16dac3d6cce7b0 returning 422. The real commit SHA for v4.1.8 is fa0a91b85d4f404e444e00e005971372dc801d16 (very close to what's here but not the same -- easy to have mistyped/mis-copied):

uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8

with:
name: dist
path: dist/
- uses: pypa/gh-action-pypi-publish@release/v1
2 changes: 0 additions & 2 deletions MANIFEST.in
View file
Open in desktop

This file was deleted.

30 changes: 6 additions & 24 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ help: ## display this help message
@grep '^[a-zA-Z]' $(MAKEFILE_LIST) | sort | awk -F ':.*?## ' 'NF==2 {printf "\033[36m %-25s\033[0m %s\n", $$1, $$2}'

test: ## run all the tests
tox
uv run --group test pytest

pylint: ## check our own code with pylint
tox -e pylint
uv run --group quality pylint src/edx_lint test

clean: ## remove all the unneeded artifacts
-rm -rf .tox
Expand All @@ -21,26 +21,8 @@ clean: ## remove all the unneeded artifacts
-rm -rf .coverage .coverage.* htmlcov

requirements: ## install the developer requirements
pip install -qr requirements/pip-tools.txt
pip install -e .
pip install -r requirements/dev.txt
uv sync --group dev

compile-requirements: export CUSTOM_COMPILE_COMMAND=make upgrade
compile-requirements: ## compile the requirements/*.txt files with the latest packages satisfying requirements/*.in
# Make sure to compile files after any other files they include!
pip-compile -v ${COMPILE_OPTS} --allow-unsafe --rebuild -o requirements/pip-tools.txt requirements/pip-tools.in
pip install -qr requirements/pip-tools.txt
pip-compile -v ${COMPILE_OPTS} -o requirements/base.txt requirements/base.in
pip-compile -v ${COMPILE_OPTS} -o requirements/dev.txt requirements/dev.in
pip-compile -v ${COMPILE_OPTS} -o requirements/test.txt requirements/test.in
pip-compile -v ${COMPILE_OPTS} -o requirements/ci.txt requirements/ci.in

upgrade: ## update the requirements/*.txt files with the latest packages satisfying requirements/*.in
pip install -qr requirements/pip-tools.txt
$(MAKE) compile-requirements COMPILE_OPTS="--upgrade"

# Delete django, drf pins from test.txt so that tox can control
# Django version.
sed -i.tmp '/^[dD]jango==/d' requirements/test.txt
sed -i.tmp '/^djangorestframework==/d' requirements/test.txt
rm requirements/test.txt.tmp
upgrade: ## update the lockfile with the latest packages satisfying constraints
uv run --with edx-lint edx_lint write_uv_constraints pyproject.toml
uv lock --upgrade
126 changes: 126 additions & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
[build-system]
requires = ["setuptools", "setuptools-scm>8.1"]
build-backend = "setuptools.build_meta"

[project]
name = "edx-lint"
dynamic = ["version"]
description = "edX-authored pylint checkers"
readme = "README.rst"
license = "Apache-2.0"
license-files = ["LICENSE.txt"]
requires-python = ">=3.12"
authors = [
{name = "edX", email = "oscm@edx.org"},
]
classifiers = [
"Environment :: Console",
"Intended Audience :: Developers",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3.12",
"Topic :: Software Development :: Quality Assurance",
]
dependencies = [
"code-annotations>=1.1.0",
"click>=8.0",
"click-log>=0.4.0",
"pylint>=4.0",
"pylint-django>=2.6",
"pylint-celery>=0.3",
"six>=1.16.0",
"tomlkit>=0.10.0",
]

[project.urls]
Homepage = "https://github.com/openedx/edx-lint"

[project.scripts]
edx_lint = "edx_lint.cmd.main:main"
lint-amnesty = "edx_lint.cmd.amnesty:pylint_amnesty"

[tool.setuptools_scm]
version_scheme = "only-version"
local_scheme = "no-local-version"
fallback_version = "0.0.0.dev0"

@irfanuddinahmad irfanuddinahmad Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fallback_version = "0.0.0.dev0" -- this exact value crashed 17 tests in a sibling repo (openedx-events) because runtime code parsed __version__ via tuple(map(int, __version__.split("."))) and choked on the non-numeric dev0 segment. I checked this repo's own __version__ consumers (src/edx_lint/write.py, src/edx_lint/cmd/main.py) and neither int-parses it, so it won't crash here today -- but it's still worth using a plain int-parseable value like "0.0.0" to avoid the same class of bug if a consumer is ever added.


[tool.setuptools.packages.find]
where = ["src"]
include = ["edx_lint*"]
exclude = ["test*"]

[tool.setuptools.package-data]
"edx_lint" = [
"files/*",
"files/.*",
]

[dependency-groups]
test-base = [
"coverage",
"pytest",
]
test = [
{include-group = "test-base"},
"Django>=5.2,<6.0",
]
django42 = [
{include-group = "test-base"},
"Django>=4.2,<4.3",
]
quality = [
{include-group = "test"},
"edx-lint",
"pylint",
"pycodestyle",
]
ci = [
"tox",
"tox-uv>=1",
]
dev = [
{include-group = "quality"},
{include-group = "ci"},
]

[tool.semantic_release]
# No version_toml — version is dynamic via setuptools-scm
build_command = "python -m pip install --upgrade build && SETUPTOOLS_SCM_PRETEND_VERSION=$NEW_VERSION python -m build"
allow_zero_version = true
major_on_zero = false

[tool.uv]

@irfanuddinahmad irfanuddinahmad Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[dependency-groups] above defines a group literally named dev, which collides with uv's implicit default group name. Every invocation in this repo (Makefile: uv run --group test ..., uv run --group quality ...; ci.yml: uv sync --group ci) already names an explicit --group, which is exactly the condition under which default-groups = [] is needed here -- without it, those calls also silently sync the full dev superset alongside the requested narrower group.

package = true
conflicts = [
[{group = "test"}, {group = "django42"}],
]

# DO NOT EDIT constraint-dependencies DIRECTLY.
# This list is managed by `edx_lint write_uv_constraints`
# and will be overwritten the next time `make upgrade` is run.
# - GLOBAL constraints: edit edx_lint/files/common_constraints.txt
# - REPO-SPECIFIC constraints: edit [tool.edx_lint].uv_constraints in this file
constraint-dependencies = [
"Django<6.0",
"elasticsearch<7.14.0",
"social-auth-app-django<6.0.0",
"social-auth-core<5.0.0",
]

[tool.coverage.run]
branch = true
source = ["edx_lint"]
omit = ["*/tests/*", "*/migrations/*", "*/__pycache__/*", "test/input/*"]

[tool.coverage.report]
show_missing = true
exclude_lines = [
"pragma: no cover",
"def __repr__",
"raise AssertionError",
"raise NotImplementedError",
"if __name__ == .__main__.:",
"if TYPE_CHECKING:",
]

[tool.coverage.html]
directory = "htmlcov"
10 changes: 0 additions & 10 deletions requirements/base.in
View file
Open in desktop

This file was deleted.

61 changes: 0 additions & 61 deletions requirements/base.txt
View file
Open in desktop

This file was deleted.

3 changes: 0 additions & 3 deletions requirements/ci.in
View file
Open in desktop

This file was deleted.

Loading
Loading