feat: add init and upload in kernel-builder cli by drbh · Pull Request #378 · huggingface/kernels

drbh · 2026-03-18T17:44:45Z

This PR is a WIP to add the init command to the kernel-builder rust cli. This branch mainly ports the init command from the python cli, with some improvements to the interface and updates to the template

This PR adds an the init and upload command to the rust kernel-builder cli similar to implementation in the current python cli. It also adds a build command that is a facade for the nix run so devs can interact with a single tool for the full creation lifecycle of a kernel.

Mainly it aims to have better defaults to make the commands easier to use

the init command only requires the name of the kernels (folder) and the owner is pulled via a whoami request if the user is already logged into hf
the init command creates a new dir if a name is specified, if called within a dir, the cli will use the dir name as the kernel name
the upload command does not require a repo id if the id can be read from the build
the upload command does not require a path to the build folder and will default to looking for the build dir if no explicit path is specified

example usage

init from inside of a existing dir

mkdir some-kernel
cd some-kernel
kernel-builder init
# Downloading template from kernels-community/template...
# Initialized `drbh/some-kernelt` at /home/drbh/Projects/kernels/kernel-builder/some-kernel

init from outside of a dir

kernel-builder init some-kernel
# Downloading template from kernels-community/template...
# Initialized `drbh/some-kernelt` at /home/drbh/Projects/kernels/kernel-builder/some-kernel

build

kernel-builder build
# ...
# some-kernel-torch-ext> no Makefile or custom installCheckPhase, doing nothing
# some-kernel-torch-ext> Checking of ABI compatibility
# some-kernel-torch-ext> 🐍 Checking for compatibility with manylinux_2_28 and Python ABI version 3.9
# some-kernel-torch-ext> ✅ No compatibility issues found
# some-kernel-torch-ext> Checking loading kernel with get_kernel
# some-kernel-torch-ext> Check whether the kernel can be loaded with get-kernel: some_kernel
# some-kernel-torch-ext> Running phase: removeBytecodeHook
# some-kernel-torch-ext> Removing Python bytecode

upload

kernel-builder upload
# Found 7 build variant(s) in /home/drbh/Projects/kernels/kernel-builder/some-kernel/build
# Using branch `v1` (new)
# Uploading 36 operations...
# Kernel upload successful. Find the kernel at: https://hf.co/drbh/some-kernel

sayakpaul · 2026-03-20T05:09:35Z

Some high-level thoughts first:

the init command only requires the name of the kernels (folder) and the owner is pulled via a whoami request if the user is already logged into hf

Should we somehow inform the users about it and take confirmation before proceeding?

# Kernel upload successful. Find the kernel at: https://hf.co/drbh/some-kernel

Should this be Find the kernel at: https://hf.co/drbh/some-kernel/tree/{version}? This way, users can directly use that link to inspect files, etc.

sayakpaul

Very minor comments. No RUST expertise yet.

sayakpaul · 2026-03-20T05:30:45Z

template/CARD.md

@@ -0,0 +1,38 @@
+---


Do we want to maintain two copies? We have one in https://github.com/huggingface/kernels/blob/main/kernels/src/kernels/cli/card_template.md

kernel-builder/src/build.rs

danieldk · 2026-03-20T12:47:07Z

kernel-builder/src/init.rs

+                let bare_name = path.file_name().and_then(OsStr::to_str).ok_or_else(|| {
+                    eyre::eyre!("Cannot determine directory name from `{path_str}`")
+                })?;
+                let owner = hf::whoami_username()?;


kernel-builder/src/init.rs

kernel-builder/src/main.rs

danieldk

Reviewed the upload subcommand now as well. Really excited to get batched uploads!

kernel-builder/src/upload.rs

danieldk · 2026-03-23T08:24:36Z

kernel-builder/src/upload.rs

+    let mut operations: Vec<CommitOperation> = Vec::new();
+
+    collect_benchmark_ops(&kernel_dir, &existing_files, is_new_branch, &mut operations)?;
+    collect_readme_ops(&kernel_dir, &mut operations);


README should go to the main branch. I think probably the most elegant way is to make operations a HashMap<String, Vec<CommitOperation>> with the branch name as the key. That way we can have a nested loop to handle everything.

For later: we may want to do the card generation and upload in one go as part of upload, since we cannot do the card generation in the Nix sandbox iff we need to fetch something from the main repo to do it (depends a bit on whether we still need to enumerate all the build variants or whether that information is handled by the Hub itself through the new kernel type, if the remaining information in the card is relatively static, we can handle it during build as well).

Not for this PR though.

Some relevant internal discussions: https://huggingface.slack.com/archives/C090JN2P8NB/p1774236408982349

@drbh I think the most sensible thing for now would be to keep the card contents limited to what we can fetch from the build (after the build is successful). We will render capabilities, frameworks etc. on the Hub side upon each commit, anyway. So, the card could contain an example snippet, available function interfaces, etc. This is all implemented in Python CLI.

Totally okay if we want to do the segregation in a separate PR.

And then there is this other thing of always uploading the card to main revision.

@drbh I think the most sensible thing for now would be to keep the card contents limited to what we can fetch from the build (after the build is successful).

oh we are actually writing the card in the build step now! for exactly the reasons you mention above. Basically the card just extracts the functions from all and shows snippets/any description the user adds

And then there is this other thing of always uploading the card to main revision.

and we also do now push the card to main

kernels/kernel-builder/src/upload.rs

Line 140 in 6e14158

// README goes to main branch, build artifacts go to version branch.

The Python CLI does have a template for the snippet and derives it without having to rely on any fragile re. Are we replicating that as well?

kernels/kernels/src/kernels/cli/kernel_card_utils.py

Lines 177 to 180 in 6e14158

if func_names:

vars["usage_example"] = EXAMPLE_CODE.format(

repo_id=repo_id, func_name=func_names[0]

)

yep thats is done in this file

kernels/kernel-builder/src/pyproject/card.rs

Line 52 in 6e14158

pub fn write_card(build: &Build, kernel_dir: &Path) -> Result<()> {

and we also now avoid re for extracting the functions from __all__ by parsing the python ast. so generally I think this approach should be solid

Perfect. I think that should be it.

danieldk · 2026-03-23T08:27:45Z

kernel-builder/src/upload.rs

+}
+
+// Collect benchmark file operations: add matching files, delete stale ones.
+fn collect_benchmark_ops(


Maybe we should call them commit_ops? I was initially a bit confused, because we already use the term ops for ops in the kernel itself. Adding the commit_ prefix may make it a bit clearer that we are referring to different ops here.

Suggested change

fn collect_benchmark_ops(

fn collect_benchmark_commit_ops(

kernel-builder/src/upload.rs

danieldk

Thanks! Added some more smaller comments.

danieldk · 2026-03-25T16:11:41Z

kernel-builder/src/pyproject/card.rs

+    let content = fs::read_to_string(&init_path).ok()?;
+
+    let re = Regex::new(r#"__all__\s*=\s*\[\s*([^\]]*)\s*\]"#).ok()?;
+    let list_content = re.captures(&content)?.get(1)?.as_str();


Regex parsing is really fragile. Not for this PR, but we should use a Python AST parser. Maybe tree-sitter-python or whatever ruff/ty uses (if it's available from crates.io). Maybe it's worthwhile creating a ticket?

good point, I opt'ed to use https://crates.io/crates/rustpython-parser which is what ruff forked, only because ruff's parser has git dependencies and our use case is relatively simple

kernel-builder/src/build.rs

kernel-builder/src/main.rs

kernel-builder/src/upload.rs

sayakpaul

I left some comments regarding upload and the card components. Additionally:

Can the users optionally specify revision and repo_id in upload? I guess we always upload to a revision during our uploads (performed after the build) to respect our versioning system.
I see we have init_e2e.rs tests. But I think we should also have tests for upload and the card generation bits, preferably mimicking how we do it in the current Python CLI?
Do we want to update the docs in a separate PR?
I think we should require users to have hf-xet installed to benefit from fast uploads and downloads if we're not doing in here already?

danieldk · 2026-03-26T14:00:41Z

I think we should require users to have hf-xet installed to benefit from fast uploads and downloads if we're not doing in here already?

If it would be an extra requirement, but I think faster transfers are already enabled because the xet feature of the huggingface-hub dependency is enabled.

danieldk · 2026-03-26T14:01:35Z

kernel-builder/src/pyproject/card.rs

+        .join("__init__.py");
+
+    let content = fs::read_to_string(&init_path).ok()?;
+    let stmts = ast::Suite::parse(&content, "<module>").ok()?;


danieldk

Everything good from my side! Really nice work 🚀 .

drbh · 2026-03-26T14:14:37Z

Can the users optionally specify revision and repo_id in upload? I guess we always upload to a revision during our uploads (performed after the build) to respect our versioning system.

yep, the upload command accepts --repo_id and --branch as arguments and simply defaults to using the repo id and version from the build toml. snippet here

kernels/kernel-builder/src/upload.rs

Lines 46 to 53 in 4b4b3fa

    
               /// Repository ID on the Hugging Face Hub (e.g. `user/my-kernel`). 
        
               /// Defaults to `general.hub.repo-id` from `build.toml`. 
        
               #[arg(long)] 
        
               pub repo_id: Option<String>, 
        
               /// Upload to a specific branch (defaults to `v{version}` from metadata). 
        
               #[arg(long)] 
        
               pub branch: Option<String>,

I see we have init_e2e.rs tests. But I think we should also have tests for upload and the card generation bits, preferably mimicking how we do it in the current Python CLI?

we do have some tests for cards and uploads, here

kernels/kernel-builder/src/pyproject/card.rs

Line 98 in 4b4b3fa

#[cfg(test)]

and here

kernels/kernel-builder/src/upload.rs

Line 432 in 4b4b3fa

#[cfg(test)]

the main difference between those tests and the ones in the e2e file are that the e2e uses the built binary to confirm the commands work

kernels/kernel-builder/tests/init_e2e.rs

Line 6 in 4b4b3fa

    
           let bin = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("target/debug/kernel-builder");

I agree that it would be nice have more e2e tests but maybe those can be followed up in another PR? also for good upload e2e tests we'd need to determine the best way to mock the network requests..

Do we want to update the docs in a separate PR?

yea I think thats best, this PR is already touching many files and it might be best to update the docs when we deprecate the python cli (maybe those can be separate, not sure yet)

I think we should require users to have hf-xet installed to benefit from fast uploads and downloads if we're not doing in here already?

yep we are using xet-core directly within the huggingface-hub dependency, in our case we have the xet featured enabled and it uses xet under the hood

kernels/kernel-builder/Cargo.toml

Line 18 in 4b4b3fa

    
           huggingface-hub = { git = "https://github.com/huggingface/huggingface_hub_rust.git", rev = "8cbc662035e04d4be8e829316272893e980f5926", package = "huggingface-hub", features = ["blocking", "xet"] }

drbh temporarily deployed to testpypi March 19, 2026 15:24 — with GitHub Actions Inactive

drbh changed the title ~~feat: support init in builder cli~~ feat: add init and upload in kernel-builder cli Mar 19, 2026

drbh force-pushed the kernel-builder-init-command branch from 534af0b to 368adad Compare March 19, 2026 15:25

drbh temporarily deployed to testpypi March 19, 2026 15:25 — with GitHub Actions Inactive

drbh temporarily deployed to testpypi March 19, 2026 15:38 — with GitHub Actions Inactive

drbh temporarily deployed to testpypi March 19, 2026 15:53 — with GitHub Actions Inactive

drbh temporarily deployed to testpypi March 19, 2026 16:32 — with GitHub Actions Inactive

drbh marked this pull request as ready for review March 19, 2026 16:42

drbh temporarily deployed to testpypi March 19, 2026 16:54 — with GitHub Actions Inactive

sayakpaul reviewed Mar 20, 2026

View reviewed changes

danieldk reviewed Mar 20, 2026

View reviewed changes

danieldk reviewed Mar 23, 2026

View reviewed changes

drbh temporarily deployed to testpypi March 23, 2026 21:13 — with GitHub Actions Inactive

drbh force-pushed the kernel-builder-init-command branch from 57523a2 to f88776a Compare March 23, 2026 21:19

drbh temporarily deployed to testpypi March 23, 2026 21:20 — with GitHub Actions Inactive

drbh temporarily deployed to testpypi March 23, 2026 21:46 — with GitHub Actions Inactive

drbh temporarily deployed to testpypi March 23, 2026 21:53 — with GitHub Actions Inactive

drbh temporarily deployed to testpypi March 23, 2026 22:03 — with GitHub Actions Inactive

drbh marked this pull request as draft March 23, 2026 22:53

drbh temporarily deployed to testpypi March 24, 2026 19:20 — with GitHub Actions Inactive

drbh force-pushed the kernel-builder-init-command branch from a50eff0 to f6b9889 Compare March 24, 2026 19:25

drbh temporarily deployed to testpypi March 24, 2026 19:25 — with GitHub Actions Inactive

drbh temporarily deployed to testpypi March 24, 2026 20:29 — with GitHub Actions Inactive

drbh marked this pull request as ready for review March 24, 2026 21:05

danieldk reviewed Mar 25, 2026

View reviewed changes

drbh added 3 commits March 25, 2026 13:51

feat: support init in builder cli

4ae6fc4

feat: add upload support

5d35efa

fix: use cwd when init called within a dir

136d607

drbh added 11 commits March 25, 2026 13:51

feat: prefer fileset api

06e62eb

feat: simplify build commands

cd513dc

fix: improve upload kernel path resolution

2c90666

feat: refactor and cleanup upload logic

ff8466c

fix: bump hub dep commit and pin it

e262bd5

fix: update nix for template files

24b3396

feat: write card when building

fa82c7d

fix: fix metadata upload struct bug

00aa116

feat: prefer sync upload api

f1df972

feat: prefer nix module in build command

f7fb291

feat: support uploads to kernel repos

63dbeb1

drbh force-pushed the kernel-builder-init-command branch from 3ed8a0f to 63dbeb1 Compare March 25, 2026 18:52

drbh temporarily deployed to testpypi March 25, 2026 18:52 — with GitHub Actions Inactive

feat: improve build commands, parse ast for card and upload tests

b08573a

drbh temporarily deployed to testpypi March 25, 2026 18:53 — with GitHub Actions Inactive

fix: format tests

4b4b3fa

drbh temporarily deployed to testpypi March 25, 2026 18:56 — with GitHub Actions Inactive

sayakpaul reviewed Mar 26, 2026

View reviewed changes

sayakpaul mentioned this pull request Mar 26, 2026

README utility for main branches on the build repos on Hub #315

Closed

danieldk reviewed Mar 26, 2026

View reviewed changes

danieldk approved these changes Mar 26, 2026

View reviewed changes

drbh merged commit 6e14158 into main Mar 26, 2026
38 checks passed

sayakpaul deleted the kernel-builder-init-command branch March 26, 2026 14:17

danieldk mentioned this pull request Mar 26, 2026

Move kernels init to kernel-builder init #348

Closed

	if func_names:
	vars["usage_example"] = EXAMPLE_CODE.format(
	repo_id=repo_id, func_name=func_names[0]
	)

Conversation

drbh commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danieldk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

drbh Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danieldk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

danieldk commented Mar 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danieldk left a comment

Choose a reason for hiding this comment

Uh oh!

drbh commented Mar 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

drbh commented Mar 18, 2026 •

edited

Loading

sayakpaul commented Mar 20, 2026 •

edited

Loading

drbh Mar 26, 2026 •

edited

Loading