Issue 564: add size filtering and typed min access level#891
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds GitLab project filtering by minimum access level and by project storage size; exports an AccessLevel enum; threads minAccessLevel and statistics into project-listing options; implements getProjectSizeBytes and size-based exclusion logic; updates schemas/types; adds unit tests for size exclusion. Changes
Sequence Diagram(s)mermaid Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
No actionable comments were generated in the recent review. 🎉 🧹 Recent nitpick comments
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
WilliamZhang9
force-pushed
the
u/williamzhang/issue_564_add_repo_size_exclude_option_for_gitlab
branch
3 times, most recently
from
3d0627a to
40a7708
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/backend/src/gitlab.ts (1)
183-219:⚠️ Potential issue | 🟠 MajorSize filtering won't apply to individually specified projects.
The
config.projectspath usesapi.Projects.show(project)which does not includestatistics: truefromprojectListOptions. As a result,getProjectSizeByteswill returnundefinedfor these projects, and size-based exclusion will be silently skipped. If a user configures bothprojectsandexclude.size, they may not get the expected behavior.Consider either:
- Passing
statistics: truetoProjects.show()whenexclude.sizeis configured, or- Documenting that size filtering only applies to projects discovered via
all,groups, orusers.Proposed fix
const { durationMs, data } = await measure(async () => { - const fetchFn = () => api.Projects.show(project); + const fetchFn = () => api.Projects.show(project, { + ...(config.exclude?.size ? { statistics: true } : {}), + }); return fetchWithRetry(fetchFn, `project ${project}`, logger); });
🤖 Fix all issues with AI agents
In `@packages/schemas/src/v3/connection.type.ts`:
- Around line 138-141: Update the union type for minAccessLevel in the
Connection type: remove the invalid value 0 and add 15 (Planner) so the field
reads minAccessLevel?: 5 | 10 | 15 | 20 | 30 | 40 | 50; — locate the
minAccessLevel property in connection.type.ts and adjust the allowed numeric
literals accordingly to match GitLab's API.
🧹 Nitpick comments (4)
packages/schemas/src/v3/gitlab.schema.ts (1)
167-180: Consider enforcing non-negative size bounds.Adding a minimum of 0 for
min/maxprevents negative sizes in config. Since this file is auto-generated, update the schema source/generator accordingly.♻️ Suggested schema tweak
"min": { "type": "integer", + "minimum": 0, "description": "Minimum project size (in bytes) to sync (inclusive). Projects smaller than this will be excluded." }, "max": { "type": "integer", + "minimum": 0, "description": "Maximum project size (in bytes) to sync (inclusive). Projects larger than this will be excluded." }schemas/v3/gitlab.json (1)
139-153: Schema definition forexclude.sizelooks good.
additionalProperties: falseis correctly set. Consider addingminimum: 0to theminandmaxproperties to prevent nonsensical negative byte values at the schema validation level, though this is a minor hardening.packages/backend/src/gitlab.ts (2)
279-292: Silent skip when statistics are missing could be surprising.When
exclude.sizeis configured butgetProjectSizeBytesreturnsundefined(e.g., statistics not available), the project passes through without any size check or warning. Consider logging a debug/warning message when statistics are expected but missing, so operators can diagnose why size filtering isn't working for specific projects.Proposed enhancement
if (exclude?.size) { const projectSizeBytes = getProjectSizeBytes(project); if (projectSizeBytes !== undefined) { if (exclude.size.min !== undefined && projectSizeBytes < exclude.size.min) { reason = `project size (${projectSizeBytes}) is less than \`exclude.size.min\` (${exclude.size.min})`; return true; } if (exclude.size.max !== undefined && projectSizeBytes > exclude.size.max) { reason = `project size (${projectSizeBytes}) is greater than \`exclude.size.max\` (${exclude.size.max})`; return true; } + } else { + logger.debug(`No statistics available for project ${projectName}, skipping size-based exclusion check.`); } }
325-356:getProjectSizeBytessilently falls back fromstorage_sizetorepository_size— these metrics differ significantly.
storage_sizeincludes the entire project (repo, LFS, packages, artifacts, etc.), whilerepository_sizeis just the git repository. Falling through from one to the other without any indication could lead to inconsistent filtering behavior across projects. The field name priority should be documented in the schema description or in a code comment explaining the preference order.Also, the dual snake_case/camelCase handling is a good defensive measure against
@gitbeaker/restresponse serialization differences.
| /** | ||
| * Minimum GitLab access level required for projects to be returned. Uses GitLab role levels where 20=Reporter, 30=Developer, 40=Maintainer, and 50=Owner. Note: GitLab project listing APIs do not accept 60 (Admin) for this field. | ||
| */ | ||
| minAccessLevel?: 0 | 5 | 10 | 20 | 30 | 40 | 50; |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
GitLab project listing API min_access_level parameter allowed values documentation
💡 Result:
In the GitLab Projects API (“List all projects”, GET /projects), the min_access_level query parameter is an integer with these allowed values:
5= Minimal access10= Guest15= Planner20= Reporter30= Developer40= Maintainer50= Owner
This filters to projects where the current user has at least that access level. [1]
Source: GitLab Docs → Projects API. [1]
Remove 0 from the enum and add 15 (Planner) to match GitLab API specification.
The GitLab Projects API accepts: 5 (Minimal access), 10 (Guest), 15 (Planner), 20 (Reporter), 30 (Developer), 40 (Maintainer), 50 (Owner). Update the enum to:
minAccessLevel?: 5 | 10 | 15 | 20 | 30 | 40 | 50;
🤖 Prompt for AI Agents
In `@packages/schemas/src/v3/connection.type.ts` around lines 138 - 141, Update
the union type for minAccessLevel in the Connection type: remove the invalid
value 0 and add 15 (Planner) so the field reads minAccessLevel?: 5 | 10 | 15 |
20 | 30 | 40 | 50; — locate the minAccessLevel property in connection.type.ts
and adjust the allowed numeric literals accordingly to match GitLab's API.
✅ Confirmed as addressed by @WilliamZhang9
There was a problem hiding this comment.
Good catch. I’ve removed 0 (no access), but I didn’t add 15 for minAccessLevel to stay aligned with the client library gitbeaker. Let me know if I missed anything.
https://github.com/jdalrymple/gitbeaker/blob/main/packages/core/src/constants.ts
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Fetch the gitbeaker constants file to check AccessLevel enum values
curl -s https://raw.githubusercontent.com/jdalrymple/gitbeaker/main/packages/core/src/constants.ts
Length of output: 162
🌐 Web query:
GitLab API min_access_level parameter accepted values 0 15
💡 Result:
For GitLab’s REST API parameters named min_access_level (for example, GET /groups?min_access_level=...), the accepted values are the GitLab role access levels:
5= Minimal access10= Guest15= Planner20= Reporter30= Developer40= Maintainer50= Owner
So 15 is valid, but 0 is not listed as a valid min_access_level value (it’s “No access” in other contexts like some access_level parameters, but not for this filter). [1], [2]
@WilliamZhang9, great fix on removing 0—that's correct since the GitLab API's min_access_level parameter doesn't accept it.
However, I notice that gitbeaker's enum appears incomplete: it's missing PLANNER = 15, which the GitLab API does accept. By excluding 15, users won't be able to filter for Planner access level even though the underlying API supports it.
Options:
- Full API support: Include
5 | 10 | 15 | 20 | 30 | 40 | 50to match what GitLab actually accepts - Strict gitbeaker alignment: Keep
5 | 10 | 20 | 30 | 40 | 50(current approach)
If your priority is library consistency, the current approach is fine. If you want to expose the full API capability, consider adding 15. Let me know your preference!
✏️ Learnings added
Learnt from: WilliamZhang9
Repo: sourcebot-dev/sourcebot PR: 891
File: packages/schemas/src/v3/connection.type.ts:0-0
Timestamp: 2026-02-14T15:22:21.470Z
Learning: The minAccessLevel field in GitlabConnectionConfig should align with the gitbeaker library's AccessLevel enum values rather than including all GitLab API-supported values, to maintain consistency with the client library being used in packages/backend/src/gitlab.ts.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@packages/backend/src/gitlab.ts`:
- Around line 64-73: projectListOptions only sets statistics: true when
config.exclude?.size is present, but the Projects.show() call (used to fetch
individually listed projects) doesn't pass that option, so project.statistics
stays undefined and getProjectSizeBytes() can't filter by size; update the call
to Projects.show(...) to pass the same options object (or at least { statistics:
true } when config.exclude?.size) as the second argument so that
project.statistics is populated for size filtering—use the existing
projectListOptions or a conditional { statistics: true } when calling
Projects.show in the code path that loads individual projects.
🧹 Nitpick comments (2)
packages/backend/src/gitlab.ts (1)
279-292: Silent skip when statistics are unavailable — consider logging a warning.When
exclude.sizeis configured butgetProjectSizeBytesreturnsundefined(e.g., statistics not available for the project), the size filter is silently bypassed. A debug/warn log here would help users troubleshoot why certain projects aren't being filtered.Proposed improvement
if (exclude?.size) { const projectSizeBytes = getProjectSizeBytes(project); if (projectSizeBytes !== undefined) { if (exclude.size.min !== undefined && projectSizeBytes < exclude.size.min) { reason = `project size (${projectSizeBytes}) is less than \`exclude.size.min\` (${exclude.size.min})`; return true; } if (exclude.size.max !== undefined && projectSizeBytes > exclude.size.max) { reason = `project size (${projectSizeBytes}) is greater than \`exclude.size.max\` (${exclude.size.max})`; return true; } + } else { + logger.warn(`Size exclusion configured but no statistics available for project ${projectName}. Skipping size filter.`); } }schemas/v3/gitlab.json (1)
138-153: Consider clarifying which GitLab size field is used.The description says "GitLab statistics size fields" but doesn't specify which field. The backend implementation (
getProjectSizeBytes) preferentially usesstorage_size(which includes LFS, artifacts, etc.) and falls back torepository_size. Documenting this in the schema description would help users set appropriate thresholds.Suggested description improvement
- "description": "Exclude projects based on GitLab statistics size fields (in bytes).", + "description": "Exclude projects based on GitLab statistics size fields (in bytes). Uses `storage_size` when available (includes LFS, artifacts, etc.), falling back to `repository_size`.",
| const minAccessLevel: ProjectsAccessLevel | undefined = config.minAccessLevel; | ||
| const projectListOptions = { | ||
| perPage: 100, | ||
| ...(minAccessLevel !== undefined ? { | ||
| minAccessLevel, | ||
| } : {}), | ||
| ...(config.exclude?.size ? { | ||
| statistics: true, | ||
| } : {}), | ||
| }; |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# First, let's locate the file and understand the context around line 64-73 and line 188
head -100 packages/backend/src/gitlab.ts | tail -50Repository: sourcebot-dev/sourcebot
Length of output: 1916
🏁 Script executed:
#!/bin/bash
# Check for Projects.show usage patterns in the file
rg -n "Projects\.show" packages/backend/src/gitlab.ts -B 2 -A 2Repository: sourcebot-dev/sourcebot
Length of output: 412
🏁 Script executed:
#!/bin/bash
# Search for getProjectSizeBytes to understand how statistics is used
rg -n "getProjectSizeBytes\|statistics" packages/backend/src/gitlab.ts -B 2 -A 2Repository: sourcebot-dev/sourcebot
Length of output: 49
🏁 Script executed:
#!/bin/bash
# Check type definitions or imports for Projects API
rg -n "import.*Projects\|from.*gitbeaker" packages/backend/src/gitlab.tsRepository: sourcebot-dev/sourcebot
Length of output: 49
🏁 Script executed:
#!/bin/bash
# Look for the entire gitlab.ts file to understand the flow
wc -l packages/backend/src/gitlab.tsRepository: sourcebot-dev/sourcebot
Length of output: 101
🏁 Script executed:
#!/bin/bash
# Get the entire gitlab.ts file to see context
cat -n packages/backend/src/gitlab.tsRepository: sourcebot-dev/sourcebot
Length of output: 17926
🏁 Script executed:
#!/bin/bash
# Search for getProjectSizeBytes in the entire repo
rg -n "getProjectSizeBytes" --type=ts -B 2 -A 5Repository: sourcebot-dev/sourcebot
Length of output: 1428
🏁 Script executed:
#!/bin/bash
# Search for where size filtering/exclusion happens
rg -n "exclude.*size\|size.*exclude" --type=ts -B 3 -A 3Repository: sourcebot-dev/sourcebot
Length of output: 49
🏁 Script executed:
#!/bin/bash
# Search for Projects.show signature or usage patterns more broadly
rg -n "Projects\.show\(" --type=ts -B 2 -A 2Repository: sourcebot-dev/sourcebot
Length of output: 567
🏁 Script executed:
#!/bin/bash
# Check if there are any type definitions or documentation for Projects.show
fd -t f "package.json" | head -5Repository: sourcebot-dev/sourcebot
Length of output: 196
🏁 Script executed:
#!/bin/bash
# Check node_modules or look for `@gitbeaker` documentation
find . -name "*.d.ts" -type f 2>/dev/null | grep -i gitbeaker | head -10Repository: sourcebot-dev/sourcebot
Length of output: 49
🏁 Script executed:
#!/bin/bash
# Search for any examples of Projects.show with options in the codebase
rg -n "Projects\.show.*\{" --type=tsRepository: sourcebot-dev/sourcebot
Length of output: 49
🌐 Web query:
@gitbeaker/rest Projects.show statistics parameter API documentation
💡 Result:
In @gitbeaker/rest, Projects.show maps to GitLab’s “Retrieve a project” endpoint (GET /projects/:id). GitLab supports a query parameter:
statistics(boolean): “Include project statistics” (adds astatisticsobject to the returned project, e.g.,commit_count,storage_size, etc.; requires at least Reporter/Developer/Maintainer/Owner). [1]
So in GitBeaker you pass it via the optional options object (GitBeaker’s general rule: required args first, then an options object for optional API params) [2]:
const project = await api.Projects.show(projectId, { statistics: true });
// -> calls GET /projects/:id?statistics=trueNote: this is different from GitLab’s separate Project statistics API endpoint GET /projects/:id/statistics (clone/pull fetch stats for the last 30 days). [3]
Sources: GitLab Projects API (“Retrieve a project” params) [1]; GitBeaker usage conventions (optional params in an options object) [2]; GitLab Project statistics API (/projects/:id/statistics) [3].
Pass statistics option to Projects.show() when size filtering is enabled.
projectListOptions includes statistics: true only when config.exclude?.size is set, and this is correctly spread into Projects.all(), Groups.allProjects(), and Users.allProjects() (lines 83, 107, 148). However, the Projects.show() call on line 188 doesn't pass this option, leaving project.statistics undefined for individually listed projects. As a result, getProjectSizeBytes() returns undefined and size-based exclusion silently skips those projects.
Since @gitbeaker/rest's Projects.show() API supports the statistics parameter as a second argument, pass it when size filtering is enabled:
const fetchFn = () => api.Projects.show(project);
+const fetchFn = () => api.Projects.show(project, { statistics: config.exclude?.size ? true : undefined });🤖 Prompt for AI Agents
In `@packages/backend/src/gitlab.ts` around lines 64 - 73, projectListOptions only
sets statistics: true when config.exclude?.size is present, but the
Projects.show() call (used to fetch individually listed projects) doesn't pass
that option, so project.statistics stays undefined and getProjectSizeBytes()
can't filter by size; update the call to Projects.show(...) to pass the same
options object (or at least { statistics: true } when config.exclude?.size) as
the second argument so that project.statistics is populated for size
filtering—use the existing projectListOptions or a conditional { statistics:
true } when calling Projects.show in the code path that loads individual
projects.
✅ Addressed in commit 525570a
Support excluding GitLab projects by statistics-backed size bounds and pass through minAccessLevel for project listing with AccessLevel-aligned typing.
WilliamZhang9
force-pushed
the
u/williamzhang/issue_564_add_repo_size_exclude_option_for_gitlab
branch
from
525570a to
a65ffd8
Compare
|
It looks only users with the Reporter, Developer, Maintainer, or Owner roles can retrieve a GitLab project’s size. Do we need to use 
|
1 participant
Problem
#564
Solution
Support excluding GitLab projects by statistics-backed size bounds and pass through
minAccessLevelfor project, and align it with the GitLab access level.Summary by CodeRabbit
New Features
Tests