Skip to main content

Publish a New Model

Model code and configuration are shared through GoodWoolf. Model weights are shared through the Training Ledger repo.

Sharing large model weights requires storing them in a repository that provides a public URL from which others can download them. The example below uses Hugging Face, but you can store the weights anywhere, as long as they are accessible through a public URL.

To publish a new model for collaborative training:

  1. Upload your initial model weights to a public URL (e.g., a Hugging Face model):
  • $GOODWOOLF_MODELS/<vendor>/<model>

For example to upload model weights to Hugging Face:

hf upload <hf_user>/<hf_model> $GOODWOOLF_MODELS/<vendor>/<model> <vendor>/<model> --exclude "*.pt"
  1. Create a PR for GoodWoolf with these folders:
  • <gw_repo>/src/goodwoolf/config/<vendor>/<model>
  • <gw_repo>/src/goodwoolf/models/<vendor>/<model>
  1. Create a PR for the Training Ledger repo with the ledger file:
  • <gw_ledger_repo>/<vendor>/<model>/training.ledger.json

First clone the ledger repo:

git clone https://user:[email protected]/datasets/goodwoolf/training-ledger

And set the path where the training ledger was cloned:

export GOODWOOLF_LEDGER=<gw_ledger_repo>

Add a new entry to the model ledger:

goodwoolf ledger <vendor>/<model> --create randomized

Update the training.ledger.json in (3) and set the new entry's url to the URL in (1) where the initial model weights are uploaded, then check the entry:

goodwoolf ledger <vendor>/<model> --check

Create a new PR ref as HF discussion:

hf discussions create goodwoolf/training-ledger \
  --repo-type dataset \
  --title "Model ledger update" \
  --body "Description of the ledger update" \
  --pull-request

List all PR and make a note of your PR_NUMBER:

hf discussions list goodwoolf/training-ledger --repo-type=dataset --kind pull_request

Then finally add, commit and push the ledger file to the new PR branch:

git add .
git commit -m "Training ledger"
git push origin HEAD:refs/pr/<PR_NUMBER>

When all PR changes are ready, publish the PR from HF portal.