Merge pull request #178 from linebender/ci2

Shader compilation in GitHub Actions

.github/actions/shader/action.yml

@@ -0,0 +1,30 @@
+name: compile shaders
+
+runs:
+  using: 'composite'
+  steps:
+    - uses: seanmiddleditch/gha-setup-ninja@master
+
+    - name: setup SPIRV tools
+      # consider install-vulkan-sdk instead
+      uses: humbletim/setup-vulkan-sdk@v1.2.0
+      with:
+        vulkan-query-version: 1.3.204.0
+        vulkan-components: Glslang, SPIRV-Cross
+        vulkan-use-cache: true
+
+    - name: install DXC
+      uses: napokue/setup-dxc@v1.0.0
+
+    - name: 'run shader compilers: piet-gpu'
+      run: mkdir gen && ninja
+      shell: pwsh
+      working-directory: piet-gpu/shader
+    - name: 'run shader compilers: tests'
+      run: mkdir gen && ninja
+      shell: pwsh
+      working-directory: tests/shader
+    - name: 'run shader compilers: piet-gpu-hal/examples'
+      run: mkdir gen && ninja
+      shell: pwsh
+      working-directory: piet-gpu-hal/examples/shader

.github/workflows/ci.yml

@@ -0,0 +1,27 @@
+on:
+  push:
+    branches:
+      - main
+      - dev
+  pull_request:
+
+jobs:
+  rustfmt:
+    runs-on: ubuntu-latest
+    name: cargo fmt
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: install stable toolchain
+        uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+          profile: minimal
+          components: rustfmt
+          override: true
+
+      - name: cargo fmt
+        uses: actions-rs/cargo@v1
+        with:
+          command: fmt
+          args: --all -- --check

.github/workflows/push-shader.yml

@@ -0,0 +1,38 @@
+on:
+  push:
+    branches:
+      - dev
+
+jobs:
+  push-shaders:
+    runs-on: windows-latest
+    name: compile shaders and push to main
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          # need history to make the merge work
+          # possibly we can optimize this and set
+          # allow-unrelated-histories on merge
+          fetch-depth: 0
+      - name: prepare repo for compilation
+        run: |
+          git fetch origin main
+          git switch main
+          git config user.name "Commit by GitHub Action"
+          git config user.email "nobody@example.com"
+          git merge dev -m "merge from dev branch"
+          sed -i '' '/shader\/gen/d' .gitignore
+          git add .gitignore
+          git rm -r --ignore-unmatch piet-gpu/shader/gen
+          git rm -r --ignore-unmatch tests/shader/gen
+          git rm -r --ignore-unmatch piet-gpu-hal/examples/shader/gen
+      - uses: ./.github/actions/shader
+      - name: commit compiled shaders
+        continue-on-error: true
+        run: |
+          git add piet-gpu/shader/gen
+          git add tests/shader/gen
+          git add piet-gpu-hal/examples/shader/gen
+          git commit -m "commit compiled shaders"
+      - name: push
+        run: git push origin main

.github/workflows/shader.yml

@@ -0,0 +1,12 @@
+on:
+  pull_request:
+    branches-ignore:
+     - main
+
+jobs:
+  push-shaders:
+    runs-on: windows-latest
+    name: compile shaders
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ./.github/actions/shader

README.md

@@ -42,7 +42,7 @@ In addition, the shaders are provided under the terms of the [Unlicense]. The in
 
 The dx12 backend was adapted from piet-dx12 by Brian Merchant.
 
-Contributions are welcome by pull request. The [Rust code of conduct] applies.
+Contributions are welcome by pull request. The [Rust code of conduct] applies. Pull requests should be against the `dev` branch; see [shader_compilation.md] for explanation and details.
 
 [piet-metal]: https://github.com/linebender/piet-metal
 [Direct2D]: https://docs.microsoft.com/en-us/windows/win32/direct2d/direct2d-portal
@@ -53,3 +53,4 @@ Contributions are welcome by pull request. The [Rust code of conduct] applies.
 [Druid]: https://github.com/linebender/druid
 [Unlicense]: https://unlicense.org/
 [Rust code of conduct]: https://www.rust-lang.org/policies/code-of-conduct
+[shader_compilation.md]: ./doc/shader_compilation.md

doc/blogs.md

@@ -2,6 +2,7 @@
 
 Much of the research progress on piet-gpu is documented in blog entries. Here are the most relevant:
 
+* [piet-gpu progress: clipping](https://raphlinus.github.io/rust/graphics/gpu/2022/02/24/piet-gpu-clipping.html), Feb 24, 2022
 * [Fast 2D rendering on GPU](https://raphlinus.github.io/rust/graphics/gpu/2020/06/13/fast-2d-rendering.html), Jun 13, 2020
 * [A sort-middle architecture for 2D graphics](https://raphlinus.github.io/rust/graphics/gpu/2020/06/12/sort-middle.html), Jun 12, 2020
 * [piet-gpu progress report](https://raphlinus.github.io/rust/graphics/gpu/2020/06/01/piet-gpu-progress.html), Jun 1, 2020
@@ -9,6 +10,12 @@ Much of the research progress on piet-gpu is documented in blog entries. Here ar
 
 There are some posts more general to GPU compute programming that might be of some interest:
 
+* [The stack monoid revisited](https://raphlinus.github.io/gpu/2021/05/13/stack-monoid-revisited.html), May 13, 2021
+* [Prefix sum on portable compute shaders](https://raphlinus.github.io/gpu/2021/11/17/prefix-sum-portable.html), Nov 17, 2021
 * [The stack monoid](https://raphlinus.github.io/gpu/2020/09/05/stack-monoid.html), Sep 5, 2020
 * [Prefix sum on Vulkan](https://raphlinus.github.io/gpu/2020/04/30/prefix-sum.html), Apr 30, 2020
 * [GPU resources](https://raphlinus.github.io/gpu/2020/02/12/gpu-resources.html), Feb 12, 2020
+
+A paper, [Fast GPU bounding boxes on tree-structured scenes], describes the algorithm to compute bounding boxes for clipping and blending.
+
+[Fast GPU bounding boxes on tree-structured scenes]: https://arxiv.org/abs/2205.11659

doc/shader_compilation.md

@@ -0,0 +1,25 @@
+# How shader compilation works
+
+We use git branches to support shader compilation in the cloud. The `dev` branch contains only shader source files (in GLSL format), while the `main` branch contains generated shaders. On every push to the `dev` branch, a GitHub action runs which compiles the shaders and pushes those to `main`.
+
+Thus, you can run piet-gpu from the `main` branch without requiring any shader compilation tools. Also, the `dev` branch has a relatively clean history, and PRs can be made against it without having to worry about merge conflicts in the generated shader files.
+
+If you do want to make changes to the shaders, you'll need some tools installed:
+
+* [Ninja]
+* [Vulkan SDK] (mostly for glslangValidate, spirv-cross)
+* [DirectX Shader Compiler][DXC]
+
+The GitHub action runs on Windows so the DXC signing can succeed (note that [hassle-rs] may provide an alternate solution). We currently only compile to MSL on Metal, not AIR, due to tooling friction. The Metal shader compiler is available on Windows, but a barrier to running in CI is that downloading it appears to require an Apple account. Longer term we will want to figure out a solution to this, because the piet-gpu vision involves ahead-of-time compilation of shaders as much as possible.
+
+Right now the scripts for compiling shaders are done in hand-written ninja files. This is likely to change, as the number of permutations will increase, and we also may want access to metadata from the shader compilation process.
+
+Following a few general rules should hopefully keep things running smoothly:
+
+* Prepare all PRs against the `dev` branch, not `main`.
+* Don't commit generated shaders in the PR.
+* Don't commit directly to `main`, it will cause divergence.
+
+[Ninja]: https://ninja-build.org/
+[Vulkan SDK]: https://www.lunarg.com/vulkan-sdk/
+[DXC]: https://github.com/microsoft/DirectXShaderCompiler

piet-gpu/Cargo.toml

@@ -3,6 +3,7 @@ name = "piet-gpu"
 version = "0.1.0"
 authors = ["Raph Levien <raph.levien@gmail.com>"]
 description = "A compute-centric GPU 2D renderer."
+readme = "README.md"
 license = "MIT/Apache-2.0"
 edition = "2018"