Skip to content

Commit 27d8f6a

Browse files
scalibyscaliby
authored
Refactor README.md (#852)
* docs: refactor README.md * docs: fix links and reorganize index * docs: fix links * docs: refactor README.md to use tables * docs: add license headers * docs: fix formatting * docs: use xpk instead of python3 xpk.py * docs: fix formatting of code snippet * docs: add missing space * docs: reapply #824 on new structure * docs: reapply #796 on new structure * feat: add docs for tpu7x * docs: add ironwood badge
1 parent 81f0e3d commit 27d8f6a

21 files changed

+2445
-1602
lines changed

‎README.md‎

Lines changed: 52 additions & 1592 deletions
Large diffs are not rendered by default.

‎docs/code-of-conduct.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,19 @@
1+
<!--
2+
Copyright 2025 Google LLC
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
https://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
-->
16+
117
# Code of Conduct
218

319
## Our Pledge

‎docs/contributing.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,19 @@
1+
<!--
2+
Copyright 2025 Google LLC
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
https://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
-->
16+
117
# How to Contribute
218

319
We would love to accept your patches and contributions to this project.

‎docs/installation.md‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
<!--
2+
Copyright 2025 Google LLC
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
https://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
-->
16+
17+
# Installation
18+
19+
There are 2 ways to install XPK:
20+
21+
- via Python package installer (`pip`),
22+
- clone from git and build from source.
23+
24+
## Prerequisites
25+
26+
The following tools must be installed:
27+
28+
- python >= 3.10: download from [here](https://www.python.org/downloads/)
29+
- pip: [installation instructions](https://pip.pypa.io/en/stable/installation/)
30+
- python venv: [installation instructions](https://virtualenv.pypa.io/en/latest/installation.html)
31+
(all three of above can be installed at once from [here](https://packaging.python.org/en/latest/guides/installing-using-linux-tools/#installing-pip-setuptools-wheel-with-linux-package-managers))
32+
- gcloud: install from [here](https://cloud.google.com/sdk/gcloud#download_and_install_the) and then:
33+
- Run `gcloud init`
34+
- [Authenticate](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login) to Google Cloud
35+
- kubectl: install from [here](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#install_kubectl) and then:
36+
- Install `gke-gcloud-auth-plugin` from [here](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#install_plugin)
37+
- docker: [installation instructions](https://docs.docker.com/engine/install/) and then:
38+
- Configure sudoless docker: [guide](https://docs.docker.com/engine/install/linux-postinstall/)
39+
- Run `gcloud auth configure-docker` to ensure images can be uploaded to registry
40+
41+
### Additional prerequisites when installing from pip
42+
43+
- kueuectl: install from [here](https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/installation/)
44+
- kjob: installation instructions [here](https://github.com/kubernetes-sigs/kjob/blob/main/docs/installation.md)
45+
46+
### Additional prerequisites when installing from source
47+
48+
- git: [installation instructions](https://git-scm.com/downloads/linux)
49+
- make: install by running `apt-get -y install make` (`sudo` might be required)
50+
51+
### Additional prerequisites to enable bash completion
52+
53+
- Install [argcomplete](https://pypi.org/project/argcomplete/) globally on your machine.
54+
```shell
55+
pip install argcomplete
56+
activate-global-python-argcomplete
57+
```
58+
- Configure `argcomplete` for XPK.
59+
```shell
60+
eval "$(register-python-argcomplete xpk)"
61+
```
62+
63+
## Installation via pip
64+
65+
To install XPK using pip, first install required tools mentioned in [prerequisites](#prerequisites) and [additional prerequisites](#additional-prerequisites-when-installing-from-pip). Then you can install XPK simply by running:
66+
67+
```shell
68+
pip install xpk
69+
```
70+
71+
If you see an error saying: `This environment is externally managed`, please use a virtual environment. For example:
72+
73+
```shell
74+
# One time step of creating the virtual environment
75+
VENV_DIR=~/venvp3
76+
python3 -m venv $VENV_DIR
77+
78+
# Activate your virtual environment
79+
source $VENV_DIR/bin/activate
80+
81+
# Install XPK in virtual environment using pip
82+
pip install xpk
83+
```
84+
85+
## Installation from source
86+
87+
To install XPK from source, first install required tools mentioned in [prerequisites](#prerequisites) and [additional prerequisites](#additional-prerequisites-when-installing-from-source). Afterwards you can install XPK from source using `make`
88+
89+
```shell
90+
# Clone the XPK repository
91+
git clone https://github.com/google/xpk.git
92+
cd xpk
93+
94+
# Install required dependencies and build XPK with make
95+
make install && export PATH=$PATH:$PWD/bin
96+
```
97+
98+
If you want the dependecies to be available in your PATH please run: `echo $PWD/bin` and add its value to `PATH` in .bashrc or .zshrc file.
99+
100+
If you see an error saying: `This environment is externally managed`, please use a virtual environment. For example:
101+
102+
```shell
103+
# One time step of creating the virtual environment
104+
VENV_DIR=~/venvp3
105+
python3 -m venv $VENV_DIR
106+
107+
# Activate your virtual environment
108+
source $VENV_DIR/bin/activate
109+
110+
# Clone the XPK repository
111+
git clone https://github.com/google/xpk.git
112+
cd xpk
113+
114+
# Install required dependencies and build XPK with make
115+
make install && export PATH=$PATH:$PWD/bin
116+
```

‎docs/local_testing.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
<!--
2+
Copyright 2025 Google LLC
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
https://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
-->
16+
17+
# Local testing with Kind
18+
19+
To facilitate development and testing locally, we have integrated support for testing with `kind`. This enables you to simulate a Kubernetes environment on your local machine.
20+
21+
## Prerequisites
22+
23+
- Install kind on your local machine. Follow the official documentation here: [Kind Installation Guide.](https://kind.sigs.k8s.io/docs/user/quick-start#installation)
24+
25+
## Usage
26+
27+
xpk interfaces seamlessly with kind to manage Kubernetes clusters locally, facilitating the orchestration and management of workloads. Below are the commands for managing clusters:
28+
29+
### Cluster Create
30+
* Cluster create:
31+
32+
```shell
33+
xpk kind create \
34+
--cluster xpk-test
35+
```
36+
37+
### Cluster Delete
38+
* Cluster Delete:
39+
40+
```shell
41+
xpk kind delete \
42+
--cluster xpk-test
43+
```
44+
45+
### Cluster List
46+
* Cluster List:
47+
48+
```shell
49+
xpk kind list
50+
```
51+
52+
## Local Testing Basics
53+
54+
Local testing is available exclusively through the `batch` and `job` commands of xpk with the `--kind-cluster` flag. This allows you to simulate training jobs locally:
55+
56+
```shell
57+
python xpk.py batch [other-options] --kind-cluster script
58+
```
59+
60+
Please note that all other xpk subcommands are intended for use with cloud systems on Google Cloud Engine (GCE) and don't support local testing. This includes commands like cluster, info, inspector, etc.
61+

‎docs/permissions.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
<!--
2+
Copyright 2025 Google LLC
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
https://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
-->
16+
17+
# Permissions needed on Cloud Console:
18+
19+
* Artifact Registry Writer
20+
* Compute Admin
21+
* Kubernetes Engine Admin
22+
* Logging Admin
23+
* Monitoring Admin
24+
* Service Account User
25+
* Storage Admin
26+
* Vertex AI Administrator
27+
* Filestore Editor (This role is neccessary if you want to run `storage create` command with `--type=gcpfilestore`)

0 commit comments

Comments
 (0)