Running Locally

In order to run Pufferfish you need to start pufferfish-client and pufferfish apps. Please refer to their README's for detailed instructions.

Pufferfish

Pufferfish is a demo game that showcases the capabilities of the Fishjam, Smelter and TypeGPU. It is designed to be entertaining when played and eye catching when passing by.

Objectives

1. Make the game fun to play
2. Showcase the capabilities of Fishjam, Smelter, and TypeGPU

Requirements

Functional Requirements

• The game should allow for players to join and play together.
• The game lifecycle should be created in a way that makes the game appealing when:
  • Nobody is around to play
  • There are only a few players (SWM employees)
  • There are many players, including those who are waiting to play

Non-Functional Requirements

• The latency should be low enough to make the game feel fair
• The game should be able to run on a variety of phones, including cheap Android phones

Game Design

The core gameplay involves players controlling their pufferfish character. Players can inflate their pufferfish by inflating their cheeks. When inflated the pufferfish can kill other pufferfish. The goal is to eliminate other players' pufferfish while avoiding being eliminated yourself.

Core mechanics

The core mechanics of the game revolve around the inflation and deflation of the pufferfish characters. Players must time their inflations carefully to eliminate opponents while avoiding being eliminated themselves.

Movement

The movement of pufferfish is dictated by two rules:

1. Players do not control the movement of their pufferfish directly. Pufferfish swim around in a predictable pattern following simple physics (aka DVD screensaver). When the pufferfish reaches the edge of the screen it will simply bounce off while preserving its momentum.

2. When the two pufferfish collide and none of them die, they will bounce off each other (like billiard balls). Their speed is determined by their inflation state, described below.

Collisions

The players can kill each other by inflating their pufferfish. Each collision is resolved in the following way:

1. If both pufferfish are not inflated, they will bounce off each other and slow down.
2. If one pufferfish is inflated and the other is not, the inflated pufferfish instantly kills the other pufferfish and slows down.
3. If both pufferfish are inflated, they will bounce off each other and both will gain additional speed.

Inflation

The pufferfish inflates when the player inflates their cheeks. This state is represented by a binary value, so the pufferfish is either inflated or not inflated. The pufferfish stays inflated as long as the player has their cheeks inflated or when the inflation timer runs out, after which it will deflate automatically.

There is an additional cooldown period after deflating, during which the pufferfish cannot inflate again. This cooldown is to prevent players from staying in the inflated state too long.

Scoring

There is no scoring system in the game as players don't control movement of their pufferfish. At the end of the game, players are ranked based on the order of their elimination. The last player standing is the winner.

Game Lifecycle

The game lifecycle consists of several stages:

1. Lobby: Players join the game and wait for others to join. In the lobby they are presented with a tutorial and can test out the inflation mechanic. The game starts when a sufficient number of players are present or the administrator starts the game.
2. Gameplay: Players control their pufferfish and compete against each other. This stage continues until only one player remains.
3. Post-Game: When the player dies or the game ends, players are shown their rank. Players can go back to the lobby and start a new game or leave the game.

Technical Design

Overview

The overall architecture of the game consists of several components:

• Player Web App: The player web app is responsible for rendering the game the scene from a WHEP stream, rendering other required UI elements and handling player input with an on-device AI model.
• Game server: The game server is responsible for handling the game logic. It controls the scene via HTTP requests to the Smelter server, tracks the players' states and is responsible for the whole game lifecycle.
• Smelter: Smelter is used to handle video composition of segmented faces and static assets and rendering of the game. It will also apply WGSL shaders, preferably written in TypeGPU.
• Fishjam broadcaster: Fishjam broadcaster instance is used to handle the multimedia distribution of the finished video stream.

Infrastructure

All of the above components are deployed on the same server, which is a powerful machine with a GPU. This ensures low latency when sending video streams between the components.

Multimedia Flow

The camera feed from the players' devices is the only source of video input for the game.

The camera feed takes the following path:

1. The camera feed is captured by the player's device.
   1. It's processed with Mediapipe to detect face landmarks.
   2. Puffing cheeks is detected based on the face landmarks.
   3. The face landmarks are sent to the game engine.
   4. The feed is sent to Smalter via WHIP.
2. Smelter composes the game view using following elements:
   1. The whole view is rendered based on the updates from the game server.
   2. Each player's pufferfish face is rendered with a shader that cuts out a face from the background using the video feed and face landmarks from the user.
   3. Composed video is sent to the Fishjam broadcaster.

flowchart TD
    A[Player camera] --> B[On-device AI Model]
    B --> S[Applying shaders on each pufferfish and rendering the game view]
    S -- WHEP --> P1[Players' devices]
    S -- WHEP --> MS[Main screen]

Loading

Control flow

flowchart TD
    G[Game server]
    A[Player web app]
    S[Smelter server]

    G -- Scene control via HTTP --> S
    A -- Join/Leave/Inflate/Deflate --> G

Loading

E2E Sequence diagram

sequenceDiagram
    Player web app ->>+Game server: Join game
    Game server ->> Player web app: WHIP details
    Player web app ->> Smelter server: Stream camera
    Player web app ->> Game engine: Send face landmarks
    Game server ->> Smelter server: Render the game
    Player web app ->> Game server: Cheeks puffed
    Game server ->> Smelter server: Inflate player pufferfish

Loading

Technical Concerns

Cheek puffness detection

The AI team has doubts regarding the feasibility of the cheek puffness detection. Specifically, it may happen that this is achievable only by training a custom model.

There are concerns that this may be expensive to implement, because there is a possibility that only a custom model can accurately detect cheek puffness. This concern is currently being investigated.

Note

We managed to obtain a decently working solution using Mediapipe thanks to help from the AI team.

Compatibility

We all agreed that there are no major compatibility issues, but some minor adjustments or additional pipeline steps may be needed.

Overall API Design

Overall, there are only a couple of APIs between components. Some of those are already standardized:

• WHIP/WHEP APIs
• HTTP API for controlling the Smelter instance

Other will be specified in this doc:

• HTTP API for the Game server

Game server API

The game server API is responsible for handling all player needs apart from the multimedia itself. It should expose HTTP API and a WebSocket for real-time communication.

HTTP API will be used for initial player connection including the retrieval of WHIP details used for the multimedia transport.

The WebSocket will be used mainly for player state updates (puffing cheeks) and game state updates sent by the game server (game start, end of game, lobby status, etc.).

Cheek puff detection

Initial research has shown that it will be much easier to perform the detection on the player side. The detection will be performed by a mediapipe model running in the player's browser. Demo of this solution can be found at Cheek Puff Detector.

This approach uses two indicators that return a value for pucker and shrug expression. Using the both indicators with a OR condition serves as a good heuristic for detecting cheek puffing.