# Ankor > Ankor runs competitive Counter-Strike 2 matches on your own Linux hosts. An agent on each host, > a game node, connects to a controller that decides what runs where. These docs cover running > nodes and the protocol a controller implements. The full contract, every message, field and endpoint, is the AsyncAPI document at [/asyncapi.yaml](/asyncapi.yaml), rendered as the reference at [/reference/](/reference/). The `@ankorcs/protocol` npm package carries the same zod schemas, with the same descriptions as JSDoc, and its `@ankorcs/protocol/session` runs the agent socket's requests from a Node.js controller. The guides link to reference entries rather than repeat them. --- # Ankor game nodes > Managed CS2 game servers on your own Linux hosts, driven by a controller they connect to. Ankor hosts Counter-Strike 2 servers on your own hardware. You install the Ankor agent on a Linux machine and connect it to a controller. From then on the machine is a **game node**. The agent installs CS2 and keeps it up to date. It starts a dedicated server whenever a game server is requested. Players join that server directly and play their map. The server shuts itself down when the match is over. Every node connects to a controller. The controller decides who plays, on which map and on which node, and keeps the results. You choose which controller your nodes connect to: ### Official Ankor *ankor.sh* Sign in with Steam, add your nodes, start lobbies. **Pros** - Nothing to build or host besides your nodes. - Lobbies, map picks, statistics and demo storage. - Share nodes and update them from the browser. **Cons** - [ankor.sh](https://ankor.sh)'s rules and a [Steam Web API key](/ankor-web.md#your-steam-web-api-key). - Nodes run what [ankor.sh](https://ankor.sh) sends. [Trust](/how-it-works.md#trust) lists what that allows. ### Self-hosted *your own controller* Build your own controller on the open [protocol](/protocol.md). **Pros** - Your own match flow, rules and game modes. - Everything runs on your infrastructure. - A minimal implementation needs only a few messages. **Cons** - You build, host and maintain the controller. - You track the agent's release lines, as [Versioning](/protocol/versioning.md) explains. A controller that someone else built on the protocol has the same [control over your nodes](/how-it-works.md#trust) as [ankor.sh](https://ankor.sh). ## A node, not a server With [MatchZy](https://github.com/shobhit-pathak/MatchZy), [PugSharp](https://github.com/Lan2Play/PugSharp) or [get5](https://github.com/splewis/get5) you own a server. You install it, add a plugin and your configs, and keep them working through every game update. An Ankor node has no such server to look after. Each match runs in a fresh container from the official `ankor-cs2` image. The image carries the plugin, SwiftlyS2 and the server configuration. It runs on top of [one CS2 installation](/how-it-works.md#one-installation-many-servers) that the node shares between its servers. A match on ankor.sh plays CS2's competitive defaults, like one on Premier or FACEIT. Its launch sets only the match rules, such as friendly fire, GOTV and overtime. To play custom games, you must build your own implementation on the public agent protocol. Ankor's plugin runs in every game server. It reports its state and gameplay events to the agent, and it runs the match. [Kinds of launch](/how-it-works.md#kinds-of-launch) lists what it does in each launch. The plugin cannot be left out or replaced, because nodes depend on it. ## Get started - To play on [ankor.sh](https://ankor.sh) with your own node, follow the [quick start](/quickstart.md). - To run nodes in depth, read the [requirements](/requirements.md), then [install a node](/install.md). - To build a controller, start with the [protocol overview](/protocol.md), or follow [a minimal controller](/protocol/minimal-controller.md) step by step. - To run your own game modes, such as [ankor.sh](https://ankor.sh)'s aim warmups, add [plugins and cfgs](/operations.md#add-swiftlys2-plugins) to your nodes and [launch servers without a match](/how-it-works.md#kinds-of-launch). **Working with an AI assistant?** Give it `https://docs.ankor.sh/llms.txt` instead of these pages. It lists every guide as plain Markdown and links the full protocol as one AsyncAPI document. [llms-full.txt](/llms-full.txt) holds all guides in a single file. --- # How it works > The parts of an Ankor setup, what a node stores, and a match from launch to finish. An Ankor setup has three sides: - a controller, which organises matches - game nodes, which run them - the players who join. ## The moving parts > Diagram: The controller at the top; a game node below it, holding the agent, the CS2 installation and two slots, each running a game server; players at the bottom joining the game servers. **Terminology** ### Controller Organises matches and decides what runs where. Nodes never start a server on their own. A controller is anything that serves the agent's socket and its HTTP endpoints. It is typically a web service, such as [ankor.sh](https://ankor.sh) or one you build against the [protocol](/protocol.md). The `controller` keys in `agent.yml` hold the controller's address. ### Game node A Linux host you control, with a **node token** that identifies it to the controller. ### Agent The one long-running service on a node. It runs in a container, `ankor-agent`, beside the game servers. It dials the controller and runs the node's game servers and CS2 installation through Docker. ### Slots and game servers A **slot** is room for one game server. It has an id, a game port, a GOTV port and a [login token](/requirements.md#game-server-login-tokens). A **game server**, or **server**, is the CS2 process that a [launch](#kinds-of-launch) starts in a slot. Each server runs in a container of its own. The controller picks an **allocation id** for each launch, and the node names the server by it. A **match** is a server launched with a match configuration. Ankor's plugin runs it for one map. ### Plugin Ankor's SwiftlyS2 plugin, in every game server. What it does depends on the [launch](#kinds-of-launch). It talks only to the agent on its node, on [port 4500](#host-networking). The node's [SwiftlyS2 plugins](/operations.md#add-swiftlys2-plugins) load beside it. ### CS2 installation One per node, shared by every slot. [One installation, many servers](#one-installation-many-servers) explains how. ## Kinds of launch A launch sets the game mode, the player count, the cfgs and the console settings. It may also carry a match. A match may carry a **match id**, the controller's identifier for it. The match id names the match's demos. Every match reports its phase and score to the controller over the agent's socket. A match that sends **statistics** also uploads its gameplay events. The match and its statistics decide what the server does: | Launch | Ankor's plugin | Phase and score | Gameplay events | |-----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------|-----------------------------| | Match that sends statistics | Runs the roster and warmup for one map, and the knife round and demo recording when the [match](/reference/types/#schema-MatchConfig) asks for them. | Reported on the agent's socket. | Uploaded to the controller. | | Match without statistics | Runs the same match flow as a match that sends statistics. | Reported on the agent's socket. | Not collected. | | No match | Stays idle. The node's plugins and cfgs set the game. | Not reported. | Not collected. | A match can play on from a live round's backup, on a new server. [Launch a game](/protocol.md#6-launch-a-game) explains how. A match plays a stock map, which every CS2 installation has, in the `casual`, `competitive` or `wingman` game mode. A launch without a match can play a map from the Steam Workshop, in any [game mode](/reference/types/#schema-GameMode). ## Series A match is one map. Its launch carries no series information, such as the number of maps, the map's place in the series or the score so far. A best-of-three is therefore three matches, with three match ids, launched one after another. The controller runs the veto, the side picks, the series score and the launch of each next map. get5 and MatchZy work differently. They play a whole series on one server, which changes maps and keeps the series score. Ankor leaves the series to the controller, because a game server ends with its map: **Pros** - **Any map, any node.** Each map goes to whichever node has a free slot. A node that fails between maps loses no series state, because it stores none. - **One kind of game server.** A map in a series runs like any other. The plugin has no map change, series state or end-of-series handling. - **Formats are the controller's.** Bo2, Bo7, a map pool or a per-map side rule needs no plugin or agent release. **Cons** - **More to build.** A controller that offers series implements the veto, the score and the hand-over between maps itself. - **Players reconnect between maps,** since the next map will start on another server. - **No series in the records a node sends.** Demos carry only their match id, and events only their server. The controller groups them into a series. ## From launch to finished match 1. **Launch.** The controller asks a node to start a server. The agent claims a free slot and starts CS2 with the plugin. If the node has no free slot, it refuses the launch, and the controller tries another node. 2. **Ready.** Once the server is logged on to Steam and its map is loaded, the plugin reports that players can join. The agent then reports the slot `running`. The controller gives players the server's address. 3. **Play.** In a match, the plugin reports gameplay events. The agent forwards them to the controller. 4. **Finish.** A match ends with its map. Any server also ends when it is stopped, or when it stays empty for the launch's `idle_shutdown_seconds`. The server shuts down once its last events are stored. The agent then frees the slot and uploads the demo, if one was recorded. The messages behind each step are in the [protocol overview](/protocol.md) and the [reference](/reference/). ## Control and history Two kinds of traffic run between the agent and the controller: - **Live control** (launches, commands, server status) is not stored. After a reconnect the agent reports its current state again. - **Gameplay history** is stored on the node until the controller confirms it. [Event uploads](/protocol/events.md) describes how. ## Host networking Game servers and the agent run in Docker's [`host` network mode](https://docs.docker.com/engine/network/drivers/host/). They use the node's network stack instead of a private Docker network behind NAT. Game traffic is a steady stream of small UDP packets. Host networking keeps Docker's address translation and port forwarding out of that path. Players therefore connect straight to the server on the node's address. Each plugin talks to the agent on loopback TCP port 4500. > Diagram: The agent opens a connection out to the controller on TCP 443; on the node, each plugin talks to the agent on loopback port 4500; players reach each slot's game server on its UDP game and GOTV ports. ## One installation, many servers Every game server on a node reads the same CS2 installation, so CS2 is downloaded once per node. The plugin and its configuration sit in a separate overlay, mounted on top. Each `ankor-cs2` image has its own overlay. A CS2 update therefore never overwrites the plugin, and the plugin never changes the game files. The installation cannot change while a server reads it. A CS2 update therefore waits until every slot on the node is free. [Update CS2](/operations.md#update-cs2) has the details. [Node size versus node count](/requirements.md#node-size-versus-node-count) weighs this for many small nodes against a few large ones. ## Updates and restarts | Change | When | Running servers | |---------------------------------------|-----------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------| | CS2 update | Only when every slot is [idle](#one-installation-many-servers). | None run during it. | | CS2 image (`ankor-cs2`) replaced | Any time. | Keep their image. New servers get the new one. | | Agent restarted or its image replaced | Any time. | Keep running. The new agent takes over the running servers and their stored events. A failed replacement rolls back. | | Game server crashes | Without warning. | Events that the agent has stored are kept. The rest are lost. A relaunch is a new server. | | Host restarts | Any time. | End as in a crash. The agent starts again with Docker. | | Connection to the controller lost | Any time. | Keep running. Their events wait on the node. | The node updates CS2 and its images when the controller asks. While [`auto_update`](/configuration.md#config-auto_update) is on, it also updates them by itself. [Images](/requirements.md#images) lists which versions go together. [Automatic updates](/operations.md#automatic-updates) describes what updates by itself and how to stop it. ## Trust - By default, a node runs only the official images that the [Ankor GitHub repository](https://github.com/ankorcs/ankor) publishes, as [`official_images_only`](/configuration.md#config-official_images_only) describes. [`update_agent`](/reference/socket/#message-update_agent) and [`update_cs2_image`](/reference/socket/#message-update_cs2_image) refuse any other image, and the setup script refuses to run one. Only the node's operator can turn this off, in `agent.yml`. - With it off, [`update_agent`](/reference/socket/#message-update_agent) runs whatever image the controller sends, and that image can use the Docker socket. Connect such a node only to a controller you would give root on that host. - While [`auto_update`](/configuration.md#config-auto_update) is on, the node runs each new release or rebuild of its [line](/requirements.md#images) from the repositories that `agent.yml` names. - A workshop server receives the Steam Web API key from its launch. Whoever runs the node can therefore read the key. - Game containers run with dropped capabilities and no access to the Docker socket. - The agent listens for plugins on loopback only and accepts only the servers it launched. --- # Quick start > From a fresh Linux host to a lobby on ankor.sh. This page takes you from a Linux machine to your first lobby on [ankor.sh](https://ankor.sh). ## Before you start - A host that meets the [requirements](/requirements.md). - A Steam account that can create [Game Server Login Tokens](/requirements.md#game-server-login-tokens), and its [Steam Web API key](https://steamcommunity.com/dev). ## 1. Set up your account 1. Sign in to [ankor.sh](https://ankor.sh) with Steam. 2. Under **Settings** → **Steam key**, save your key. [Your Steam Web API key](/ankor-web.md#your-steam-web-api-key) explains why [ankor.sh](https://ankor.sh) needs it. ## 2. Install the node 1. Install the packages the host's [distribution](/install.md#distributions) needs. 2. Under **Settings** → **Nodes** → **Add node**, add the node. Copy its setup command and node token. 3. Run the setup command on the host and paste the token. The node shows as online under **Settings** → **Nodes**. 4. Under node options → **Edit** → **Add server**, add a slot for each game server. [ankor.sh](https://ankor.sh) creates each slot's login token. Then open the slot's game port (UDP) to players. 5. Under node options → **Install CS2**, click **Start install**. [Set up the node](/install.md#set-up-the-node) has the details for each step. ## 3. Start a lobby 1. On the front page, click **Create a lobby**. A node of yours, or one shared with you, must be online. 2. Copy the **Invite link** and send it to the other players. 3. Once a second player has joined, start an **Aim warmup** server for anyone to practise on. It also tests your node. 4. Choose the match setup: how teams are picked, the map selection, sides, overtime and friendly fire. 5. Once the lobby is full, move it through **Teams**, **Maps** and **Sides**. The match starts on your node, and every player gets a **Connect** button. ## Next - [Share your nodes](/ankor-web.md#share-your-nodes) so friends can start lobbies on them. - [ankor.sh](/ankor-web.md) lists where each node action is. - [Operating a node](/operations.md) covers updates, restarts and demos. - [Troubleshooting](/troubleshooting.md) helps when something does not connect. --- # ankor.sh > Where ankor.sh puts each node action, and the rules it adds for nodes. [ankor.sh](https://ankor.sh) is Ankor's controller. The node guides describe each action as the request that a controller sends to the agent. This page shows where each action is in [ankor.sh](https://ankor.sh). It also lists the requirements that [ankor.sh](https://ankor.sh) sets for its nodes. The [quick start](/quickstart.md) walks through a first node. ## Node actions Manage nodes under **Settings** → **Nodes**. Each node has a menu, called node options below. | Action | In [ankor.sh](https://ankor.sh) | Guide | |------------------------|--------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------| | Add a node | **Add node**. The node token is shown once. | [Install a node](/install.md) | | Add slots | Node options → **Edit** → **Add server**. [ankor.sh](https://ankor.sh) creates each slot's [login token](#login-tokens). | [Set up the node](/install.md#set-up-the-node) | | Change settings | Node options → **Edit**. | [Changing settings](/configuration.md#changing-settings) | | Replace a slot's token | The slot's **Get a new token** link, or **Reset token** under **Settings** → **Login tokens**. | [Replace a slot's token](/operations.md#replace-a-slots-token) | | Install or update CS2 | Node options → **Install CS2**, or **Update CS2** once installed. | [Update CS2](/operations.md#update-cs2) | | Update the CS2 image | Node options → **Update CS2 image**. The node moves to the `latest` tag. | [Update the CS2 image](/operations.md#update-the-cs2-image) | | Update the agent | Node options → **Update agent**. The node moves to the `latest` tag. | [Update the agent](/operations.md#update-the-agent) | | Read logs | Node options → **Logs & console**. | [Logs](/operations.md#logs) | | Test a node | In a lobby with a second player, **Aim warmup** → **Start server**. **Pause** your other nodes to pick this one. | [Launch a test server](/operations.md#launch-a-test-server) | | Stop a server | The slot's stop icon → **Stop server**. | | | Stop new launches | **Pause**. Running servers keep going. | | | Remove a node | Delete it under **Settings** → **Nodes**. Only its owner can. | [Remove a node](/install.md#remove-a-node) | The node table also shows each node's undelivered events. If a node runs an agent major that [ankor.sh](https://ankor.sh) no longer supports, the node is marked **Unsupported agent**. It receives no launches until you update it. ## Demos When you add or edit a node, **Demo upload** sets its [`demo_upload`](/configuration.md#config-demo_upload) target: - **This service**, the default, uploads to [ankor.sh](https://ankor.sh). - **Elsewhere** uploads to a URL you enter. - **Off** keeps demos on the node. **Settings** → **Nodes** lists the node's demos, and you can delete them there. ## Your Steam Web API key [ankor.sh](https://ankor.sh) requires a [Steam Web API key](https://steamcommunity.com/dev) from every node owner. It sends the key with every launch on the owner's nodes, including lobbies of users they share the nodes with. Servers download workshop maps with the key, and [ankor.sh](https://ankor.sh) creates your [login tokens](#login-tokens) with it. - Save it under **Settings** → **Steam key**. Adding a node needs a saved key. - Without a key, including after **Remove key**, [ankor.sh](https://ankor.sh) starts no servers on your nodes. - **Change** swaps the key for the next launch. Running servers keep the old one. - [ankor.sh](https://ankor.sh) sends the key only to your nodes. [Trust](/how-it-works.md#trust) explains who can read it there. ## Login tokens For each server you add, [ankor.sh](https://ankor.sh) creates a [Game Server Login Token](/requirements.md#game-server-login-tokens) on your Steam account with your key. It deletes the token when you remove the server. - **Settings** → **Login tokens** lists every CS2 token on your account and the slot that uses it. The list is read from Steam. Reset, rename or delete tokens there. - A token Steam refuses is replaced automatically, at most once an hour per slot. - A slot's token is never reset while the slot runs a server, because Steam drops the old token immediately. A refused token is replaced when the server ends. - [ankor.sh](https://ankor.sh) keeps no tokens. It matches a slot to its token by the slot's [`gslt_hint`](/reference/types/#schema-ReportedSlot). - Memos are labels only. Changing them on Steam's page changes nothing in [ankor.sh](https://ankor.sh). ## Share your nodes A node belongs to the user who added it, and only that user manages it. To let other users start lobbies on your nodes, open **Settings** → **Sharing**, enter their SteamID64 and click **Share**. - A share covers all of your nodes, including ones you add later. - Your nodes appear in their node table, marked with your name. The table shows the nodes' status, resources, running servers and stored demos. - They cannot configure your nodes, stop their servers, delete demos, or read their logs or console. - A lobby uses its creator's nodes, which are checked again at every launch. The lobby keeps them when the lobby leader changes. It loses them when you stop sharing. --- # Requirements > What a game node needs before setup, and what it gets from the controller. A game node can be almost any x86_64 Linux machine: a spare PC, a VM or a container on a home server, or a rented server. It needs: - A supported [host](#host). - Enough [CPU, memory and disk](#capacity) for its game servers. - A [network](#network) where players can reach its game ports. - A Steam account that can hold [Game Server Login Tokens](#game-server-login-tokens). - Its [node token](/setup-script.md#the-node-token) from the controller. ## Host - **Architecture:** x86_64. CS2 has no support for the arm64 architecture. - **Operating system:** Linux with systemd or OpenRC as PID 1. Setup supports Debian, Ubuntu, Fedora, RHEL and its rebuilds, Arch, openSUSE and Alpine. [Distributions](/install.md#distributions) has the notes for each. - **Machine:** bare metal, a VM or an LXC container with nesting enabled ([Proxmox LXC](/install.md#proxmox-lxc)). Alpine needs a VM. - **Packages:** bash, curl and CA certificates. Setup installs Docker Engine and needs root or sudo. - **Docker:** Docker Engine 25 or newer (Engine API 1.44). An [existing one](/install.md#existing-docker) must run as the system daemon. ## Capacity - **CPU and memory:** at least 2 vCPUs, or equivalent, and 2 GB of RAM. Add about one core and 1 GB per game server. A 1v1 uses about 700 MB. [`memory_mb`](/configuration.md#config-memory_mb) caps each game container, and 1536 leaves a 5v5 room to spare. - **Disk:** 90 GB beyond the operating system. About 70 GB holds the CS2 installation. The rest is room for an [update](/operations.md#update-cs2) to stage changed files in. The installation is in a Docker volume. Leave it where Docker puts it. - **Servers per node:** the host's cores, memory and network set the limit. A large workstation can run dozens. Add slots once you have seen a node's load during real matches. ### Node size versus node count Each node has [one CS2 installation](/how-it-works.md#one-installation-many-servers) for all its slots. You choose how to split capacity: | Fewer, larger nodes | More, smaller nodes | |------------------------------------------|---------------------------------------------------------| | One CS2 download covers many servers. | Each node needs its own [disk](#capacity) and download. | | Fewer hosts to install and keep patched. | More hosts, but each is simpler to replace. | | An outage or update stops many servers. | An outage or update stops a few servers. | ## Network | Direction | Port | Purpose | |-----------|----------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------| | Outbound | TCP 443 (or the controller's port) | The agent's WebSocket and its event uploads. | | Outbound | The [`demo_upload`](/configuration.md#config-demo_upload) target's port | Demo uploads, wherever the target points. | | Outbound | TCP 443 to `ghcr.io` | Pulling the [agent and CS2 images](#images) set in `agent.yml`. | | Outbound | TCP 443 to `api.steamcmd.net` | Checking for new CS2 builds. | | Outbound | Steam's [client ports](https://help.steampowered.com/en/faqs/view/2EA8-4D75-DA21-31EB) | Downloading CS2 and logging the slots' tokens on to Steam. | | Outbound | TCP 443 to `ipinfo.io`, over IPv4 | Looking up the node's public address and country. | | Inbound | UDP, each slot's game port | Players joining. | | Inbound | UDP, each slot's GOTV port | GOTV viewers, when a launch makes GOTV public. | | Loopback | TCP 4500 | The plugin talking to the agent over [host networking](/how-it-works.md#host-networking). | - You set up firewalls, routing and port forwarding. Behind NAT, forward only the slots' UDP ports. The controller never connects to the node. - TCP on a slot's game port is RCON, which is disabled. Keep it closed. - Players connect to the node's public IPv4 address. The agent looks it up, or you can pin it. If the lookup fails, the node has no address for players until you pin one or the agent restarts. IPv6 is not supported. ## Game Server Login Tokens A login token lets a CS2 server log in to Steam. The server needs that for players to join over the internet, and for Steam authentication and team-locking to work. Every slot has its own token, which the controller sends to the node. Steam creates tokens only for an account that owns CS2 and has a qualifying phone number. The account must not be limited, community banned or locked. Create the tokens on [Steam's game server page](https://steamcommunity.com/dev/managegameservers) with app ID `730` and a memo naming the slot. Then give each one to the controller. A controller can also create them with the account's Steam Web API key, as [ankor.sh](/ankor-web.md#login-tokens) does. - Steam holds the account responsible for every server that logs in with its tokens. Steam's [terms](https://steamcommunity.com/dev/managegameservers) forbid giving tokens to third parties. Give your tokens, or a Steam Web API key that creates them, only to the controller that your nodes connect to. A key can read, create and revoke every token on the account. - Use a token on no other server. Two servers logged in with one token log each other off Steam. - The node keeps idle tokens from expiring and reports whether Steam accepted each one. A server that becomes ready shows that Steam accepted its token, because CS2 does not log on anonymously when Steam refuses one. The controller [replaces](/operations.md#replace-a-slots-token) a refused token. - Changing the Steam account's password regenerates all its tokens, so every slot reports its token refused until the controller has replaced it. - If Steam bans a token's server account, the owning account can be restricted from playing CS2. ## Images Nodes run the official `ghcr.io/ankorcs/ankor-agent` and `ghcr.io/ankorcs/ankor-cs2` images. The controller sets their tags in the node's `agent.yml`, such as `ghcr.io/ankorcs/ankor-agent:0.9`. Setup pulls the agent image, and the agent pulls the CS2 image. The node therefore needs access to `ghcr.io`. - Each image has a semantic version and is tagged `X.Y.Z`, `X.Y` and `X`. `ankor-agent:1` is the newest 1.x release, and `ankor-agent:1.2.0` is one release. The newest release of all is also tagged `latest`, which crosses majors. - A release line is a major, such as `ankor-agent:1`. For a `0.x` image it is a minor, such as `ankor-agent:0.9`, because [SemVer](https://semver.org/#spec-item-4) lets a `0.x` minor break. - Both images and the protocol share their release line. Only their patch versions differ, so `ankor-agent:0.9` and `ankor-cs2:0.9` work together. - Only the newest release of each image is maintained. A CS2 update can break the plugin or SwiftlyS2 at any time, and only the newest release receives the fix. Run the newest line, and move to a new line when it ships. - The newest release is rebuilt weekly on updated base images under the same tags. Older releases keep their last build. A node with [`auto_update`](/configuration.md#config-auto_update) on pulls each rebuild of the release it runs. - To keep a node on an older build, such as for a LAN event, turn [`auto_update`](/configuration.md#config-auto_update) off or name the images by digest, as [Keep a node on its versions](/operations.md#keep-a-node-on-its-versions) describes. - Only a release on a new line breaks something: - For the agent, the protocol. [Versioning](/protocol/versioning.md) lists the releases that support each protocol version. - For the CS2 image, the software it carries, such as a SwiftlyS2 major that node plugins must be rebuilt for. - The controller chooses the line and can move nodes between versions with [`update_agent`](/reference/socket/#message-update_agent) and [`update_cs2_image`](/reference/socket/#message-update_cs2_image). Updating a node on its line's tag brings the newest release of that line. - A node with [`auto_update`](/configuration.md#config-auto_update) on moves each image to the newest release of its line by itself, as [Follow image releases](/operations.md#follow-image-releases) describes. --- # Install a node > Set up a Linux host as a game node with the setup script. You install a node with one command on the host. [Setup script](/setup-script.md) describes what that command does. Each step below names the request that the controller sends. On [ankor.sh](https://ankor.sh), follow the [quick start](/quickstart.md) instead. ## Prerequisites - A host that meets the [requirements](/requirements.md). ## Set up the node 1. Prepare the host as its [distribution](#distributions) needs, usually by installing `curl` and CA certificates. 2. Add the node in the controller and copy its setup command and node token. 3. Run the setup command on the host and paste the node token when it asks. Setup logs `agent connected.` and prints a summary. If the agent has not connected within a minute, setup warns that the agent keeps retrying. It still prints the summary. Follow the agent with `docker logs -f ankor-agent`. [Setup script](/setup-script.md) lists the options and unattended installs. 4. Add the node's slots in the controller, each with a game port, a GOTV port and a login token. The controller sends them with [`set_config`](/reference/socket/#message-set_config). A node without slots connects and reports, but cannot run a server. 5. Open each slot's game port (UDP) in the firewall and router. Open its GOTV port as well if matches on the slot are to be watched. 6. Install CS2 from the controller, with [`update_cs2`](/reference/socket/#message-update_cs2). This is the large download that [Capacity](/requirements.md#capacity) sizes. From then on CS2 [updates itself](/operations.md#update-cs2). 7. [Launch a test server](/operations.md#launch-a-test-server). ## Distributions Setup installs Docker with your distribution's package manager. Each distribution therefore has its own packages and quirks. Before you run setup, install the packages listed for yours and read its notes. Setup checks the architecture, distribution, init system and LXC nesting before it changes anything. If the host is not ready, setup stops and explains why. ### Debian and Ubuntu ```sh apt-get update && apt-get install -y curl ca-certificates ``` - Docker comes from Docker's repository for the release's codename. Setup stops on a release for which Docker publishes no packages. Docker lists its [supported releases](https://docs.docker.com/engine/install/debian/). - Ubuntu derivatives such as Linux Mint and Pop!_OS use the Ubuntu release they track. Debian derivatives with their own codename, such as Kali, are not in Docker's repository. - Remove Docker from the snap store first, with `snap remove docker`. [Existing Docker](#existing-docker) explains why. ### Fedora, RHEL, Rocky, AlmaLinux, CentOS Stream ```sh dnf install -y curl ca-certificates ``` - Setup refuses Fedora's `moby-engine`, because it runs with [SELinux labelling](#existing-docker). - On RHEL and its rebuilds, setup first removes Podman, runc and any package that depends on them, as Docker's guide prescribes. Fedora keeps Podman and loses only `podman-docker`. ### Arch ```sh pacman -Syu --needed curl ca-certificates ``` - Setup runs a full `pacman -Syu` before it installs Docker, because Arch supports no partial upgrades. - Proxmox's Arch template needs its keyring first: `pacman-key --init && pacman-key --populate archlinux`. - In LXC, pacman's download sandbox fails with "Landlock is not supported by the kernel". Turn it off: `sed -i '/^\[options\]/a DisableSandbox' /etc/pacman.conf`. ### openSUSE ```sh zypper install -y curl ca-certificates ``` ### Alpine ```sh apk add bash curl ca-certificates ``` - Docker comes from the community repository, which setup enables. - Alpine needs a VM, because in LXC OpenRC cannot hand the cgroup controllers to Docker. ## Proxmox LXC - Enable nesting with `pct set --features nesting=1`. keyctl is not needed. - Unprivileged containers work. - The node reports the container's CPUs and memory, not the Proxmox host's. ## Existing Docker Setup keeps a Docker Engine that runs as the system daemon on `/var/run/docker.sock`. It also enables that engine at boot. On any setting below, setup stops and names the setting at fault. By then it has installed the base packages and written `agent.yml`. Fix the setting and run setup again. - Docker Engine older than 25, because the agent needs Engine API 1.44. - Rootless Docker, because the agent container mounts the system daemon's socket and the game servers share the host network. - `userns-remap` in `/etc/docker/daemon.json`, because it maps uid 1000 away from the owner of `/srv/ankor` and refuses host networking. - SELinux labelling (`selinux-enabled`), because SELinux then blocks the containers' access to the Docker socket and to `/srv/ankor`. - Docker from the snap store, because its confinement cannot bind `/srv/ankor`. Setup replaces Podman's `docker` shim with Docker Engine. ## Remove a node A node exists in two places: on its host and as an entry in the controller. Deleting the entry only deauthorises the node. The agent stays installed but can no longer connect. The node therefore receives no new launches. To remove the node from its host as well, stop its servers and run the commands below. They remove every Ankor container, image and volume. They also remove the node's files, including events that it has not delivered. Docker stays installed. The node stops reporting to the controller, and you can then delete its entry. ```sh docker rm -f $(docker ps -aq --filter label=ankor.role) docker rmi $(docker images -q --filter label=ankor.image) docker volume rm ankor_cs2_data rm -rf /srv/ankor ``` To move a node to another host, or to start a host over, remove the node and set the host up as a new one. [One host per node](/operations.md#one-host-per-node) explains why. --- # Setup script > What node-setup.sh does, its options and when to change them. `node-setup.sh` turns a Linux host into a game node. It fetches the node's `agent.yml` with the node token, installs Docker if needed, and starts the agent. With [`--controller`](#options), the same script sets up nodes for your own controller. ```sh curl -fsSL https://ankor.sh/node-setup.sh | bash ``` Run it as root or as a user with sudo. Running it again is safe. It skips the steps that are already done and always replaces the `ankor-agent` container. Running servers [keep running](/how-it-works.md#updates-and-restarts) under the new agent. ## What it sets up - Docker Engine, as each [distribution](/install.md#distributions) gets it. Setup keeps a Docker Engine already on the host if it [can run the agent's containers](/install.md#existing-docker). - `/srv/ankor`, with `agent.yml` and the game servers' files. The `ankor_cs2_data` volume holds the shared CS2 install. Both paths are fixed. - The `ankor-agent` container. CS2 itself is installed later through the agent, from the controller. Setup leaves the firewall alone. Open the game ports yourself. [Install a node](/install.md) has the step. ## The node token The script asks for the node token on the terminal, with the input hidden. For an unattended install, set it in `ANKOR_NODE_TOKEN` instead: ```sh curl -fsSL https://ankor.sh/node-setup.sh | ANKOR_NODE_TOKEN= bash ``` - With neither a terminal nor the variable, setup stops and tells you to set the variable. - A malformed token stops setup before it changes anything. - A host that already has `/srv/ankor/agent.yml` keeps it and asks for no token. ## Write `agent.yml` yourself If your controller has no [setup endpoint](/protocol/node-setup.md), write `/srv/ankor/agent.yml` yourself, then run the script. The script finds the file and skips the fetch. - Write the file as [Configuration](/configuration.md#agentyml) describes, token included. - Make the [agent's user](/operations.md#permissions) its owner, and set mode `0600`. - Use the `cs2_image` and `agent_image` that your controller gives you, as [Images](/requirements.md#images) explains. - [`slots`](/configuration.md#config-slots) may be empty when the controller adds the slots later. ## Options The script reads its options after `bash -s --`, because it arrives through a pipe: ```sh curl -fsSL https://ankor.sh/node-setup.sh | bash -s -- --dry-run ``` | Option | Default | Change it when | |-----------------------|--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `--controller ` | `https://ankor.sh` | The node belongs to another controller. Its address with `https://`, or `http://` for one on your LAN. | | `--setup-path ` | `/api/node/setup` | That controller answers setup at another path. Ignored for `https://ankor.sh`. | | `--update` | | You want the agent image set in `agent.yml` pulled and the agent re-created. Needs no token. A failed pull leaves the running agent in place. Fails on a host that was never set up. | | `--no-pull` | | The agent image is already loaded on the host, for example on a host without registry access. Setup or update fails if it is missing. | | `--dry-run` | | You want to see every command without running any. Asks for no token. | | `--help` | | You want the options in the terminal. | The controller gives you the command with the options its nodes need. --- # Configuration and files > The node's agent.yml, which settings the controller can change, and where a node keeps its files. A node's settings are in one file, `agent.yml`. The controller sets most of them. It writes the file during setup. Later it can change parts of the file over the socket, such as the images when it updates the agent or the plugin. You edit the rest, such as the memory cap, on the host. ## `agent.yml` The file is `/srv/ankor/agent.yml`. Its schema is `AgentConfigSchema` in [agent-config.ts](https://github.com/ankorcs/ankor/blob/master/packages/protocol/src/agent-config.ts). ```yaml token: 3f1c… controller: host: cs.example.com tls: true pinned_ip: 203.0.113.7 cs2_image: ghcr.io/ankorcs/ankor-cs2:0.9 agent_image: ghcr.io/ankorcs/ankor-agent:0.9 memory_mb: 1536 demo_upload: url: https://demos.example.com/upload header: name: Authorization value: Bearer 9b2e… limit_mbit: 40 slots: - id: s1 port: 27015 tv_port: 27020 gslt: 0123456789ABCDEF0123456789ABCDEF - id: s2 port: 27016 tv_port: 27021 gslt: FEDCBA9876543210FEDCBA9876543210 ``` | Key | Required | Value | |-----|----------|-------| | token | yes | The node token from the controller, 32 to 128 letters and digits. [Node tokens](/protocol/#node-tokens) explains how one is made. **Keep the file secret.** | | controller | yes | Where the agent dials the controller. | | ├─ host | yes | The controller's host with an optional port, without scheme or path, such as `cs.example.com`, `192.168.0.10:3000` or `[fd00::1]`. The agent may dial over IPv6, but players need IPv4. Rejected when it contains whitespace, a control character, `/`, `\`, `?` or `#`. It is also rejected unless `http://` followed by the value parses as a URL that has a host, an optional port and nothing else, such as a user name. | | ├─ tls | yes | Whether the agent dials `wss://` and `https://` rather than `ws://` and `http://`. | | ├─ socket_path | no | The agent socket's path on `host`, a `/` followed by URL path characters. At most 255 characters. Default `/ws/agent`. | | └─ events_path | no | The path on `host` that event batches are posted to, a `/` followed by URL path characters. At most 255 characters. Default `/api/node/events`. | | pinned_ip | no | The IPv4 address players connect to, in place of the one the agent looks up. Rejected unless it is four numbers from 0 to 255 joined by dots, each without leading zeros. | | cs2_image | yes | The `ankor-cs2` image future game servers run, as a Docker image reference. | | agent_image | yes | The agent's image, which `node-setup.sh --update` runs and the agent replaces itself from, as a Docker image reference. | | memory_mb | no | MiB of memory each game container may use. Without it, game containers are not capped. CS2 installs and updates are exempt. Rejected when it is not a whole number from 1024 to 65536. A whole number from 1024 to 65536. | | auto_update | no | Whether the agent keeps the node up to date on its own. It updates CS2 to the newest public build and refuses launches while it is behind that build. It also moves `cs2_image` and `agent_image` to the newest release or rebuild on their release line. An image tagged `latest` stays `latest` and follows every release, across lines. It reads their tags from the registry every 60 minutes, and leaves an image named by digest or by a custom tag alone. When false, only [`update_cs2`](/reference/socket/#message-update_cs2), [`update_cs2_image`](/reference/socket/#message-update_cs2_image) and [`update_agent`](/reference/socket/#message-update_agent) change the build and the images. Default `true`. | | official_images_only | no | Whether the node runs only the official images: `agent_image` from `ghcr.io/ankorcs/ankor-agent` and `cs2_image` from `ghcr.io/ankorcs/ankor-cs2`, with any tag or digest. While it is on, the agent refuses an [`update_agent`](/reference/socket/#message-update_agent) or [`update_cs2_image`](/reference/socket/#message-update_cs2_image) to any other image with `image_not_allowed`, and `node-setup.sh` refuses to run one. Only the node's operator can turn it off, by setting it to false in the file, to run images of their own. [`set_config`](/reference/socket/#message-set_config) cannot change it. If the `agent.yml` that `node-setup.sh` fetches from the controller turns it off, the script stops. Default `true`. | | slots[] | yes | The node's game server slots. Empty for a node that runs no servers yet. Such a node connects and reports, but accepts no launch. A list. Each entry has the keys below. At most 64 entries. | | ├─ id | yes | The slot's id, 1 to 32 letters, digits, `_` and `-`. It is unique on its node. | | ├─ port | yes | The game port, UDP. Every `port` and `tv_port` on a node is different, free on the host, and not 4500, the port the agent listens on for plugins. [Host networking](/how-it-works/#host-networking) explains why. A whole number from 1 to 65535. | | ├─ tv_port | yes | The GOTV port, UDP, under the same rules as `port`. A whole number from 1 to 65535. | | └─ gslt | yes | The slot's Game Server Login Token, which every server in the slot logs in with. It is 10 to 64 letters and digits. [Game Server Login Tokens](/requirements/#game-server-login-tokens) explains them. | | demo_upload | no | Where finished demos are uploaded. Without it, demos stay on the node. [Demo uploads](/protocol/demos/) describes the request. | | ├─ url | yes | The `http` or `https` URL finished demos are posted to. Rejected when it carries a user name or a password, which belong in `header`. At most 255 characters. | | ├─ header | no | One header sent with every upload. | | │ ├─ name | yes | The header's name, up to 64 letters, digits and `-`. Rejected when it is, in any case, one of `host`, `content-type`, `content-length`, `transfer-encoding`, `connection`, `keep-alive`, `upgrade`, `expect`, `te`, `trailer`, `ankor-protocol`, `ankor-file-name`, `ankor-match-id` and `ankor-allocation-id`, which the agent sets itself or which change how the request is framed. | | │ └─ value | yes | The header's value, sent verbatim, without line breaks. At most 1024 characters. | | └─ limit_mbit | no | Megabits a second an upload may send while a game server runs on the node. Without it, uploads run at full speed. A whole number from 1 to 10000. | - A nested key is required only when its parent is present. - The agent warns about keys it does not know and ignores them. - A file that fails validation stops the agent, and its log lists each invalid key. - A controller on a private address or `localhost`, including `127.0.0.1:` on the node's host, usually needs `tls: false` under `controller`. No certificate authority issues certificates for those addresses. ## Changing settings The controller can change `slots`, `pinned_ip`, `auto_update`, `controller`, `demo_upload` and `memory_mb` while the node is connected, with [`set_config`](/reference/socket/#message-set_config). - The agent refuses an edit that would disturb an occupied slot. It also refuses any change to the controller address while a slot is occupied. The [`AgentErrorCode`](/reference/types/#schema-AgentErrorCode) in its reply names the case. Stop new launches on the node in the controller, wait for the slots to go idle, then edit again. - Demo settings and changes to idle slots apply while other slots are occupied. - An offline node cannot receive a corrected address. Fix `controller.host` in the file and restart the agent. - A slot's login token can be changed while the slot runs a server. The server keeps the token it started with until it ends. Revoking that token at Steam drops the server, so [replace a token](/operations.md#replace-a-slots-token) once its slot is idle. Edit the other keys in the file on the host, then [restart the agent](/operations.md#restart-the-agent). ## File layout Everything a node keeps is under `/srv/ankor/` or in the Docker volume `ankor_cs2_data`. ### In the game container Node paths are relative to `/srv/ankor/`, and container paths to `/home/steam/cs2-dedicated/game/`. Each column is a tree. A row names its path relative to the row it is indented under. The rest of the install is the Docker volume `ankor_cs2_data`. Every server shares it read-only, and only a CS2 update writes to it. | On the node | From | In the game container | Access | |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------| | `overlay//` | CS2 image | `csgo/` | | | ├─ `addons/` | CS2 image | ├─ `addons/` | Read-only | | │ └─ `swiftlys2/` | | │ └─ `swiftlys2/` | | | │ ├─ `plugins/Ankor/` | CS2 image | │ ├─ `plugins/Ankor/` | Read-only | | │ └─ `configs/plugins/Ankor/` | CS2 image | │ └─ `configs/plugins/Ankor/` | Read-only | | └─ `cfg/ankor/` | CS2 image | └─ `cfg/ankor/` | Read-only | | `slots//game/csgo/` | Slot | `csgo/` | | | ├─ `addons/swiftlys2/` | | ├─ `addons/swiftlys2/` | | | │ ├─ `configs/plugins/Ankor/config.jsonc` | Launch | │ ├─ `configs/plugins/Ankor/config.jsonc` | Read-only | | │ ├─ `data/` | Slot | │ ├─ `data/` | Writable | | │ ├─ `logs/` | Slot | │ ├─ `logs/` | Writable | | │ └─ `dumps/` | Slot | │ └─ `dumps/` | Writable | | ├─ `cfg/` | | ├─ `cfg/` | | | │ └─ `ankor_launch.cfg` | Launch | │ └─ `ankor_launch.cfg` | Read-only | | └─ `ankor/` | Slot | └─ `ankor/` | Writable | | ├─ `match.json` | Launch | ├─ `match.json` | Read-only | | ├─ `restore.txt` | Launch | ├─ `restore.txt` | Read-only | | ├─ `backups/` | Launch | ├─ `backups/` | Writable | | └─ `demos//` | Launch | └─ `demos/` | Writable | | .">`cfg/` | [Node cfgs](/operations.md#add-cfgs-for-servers) | `csgo/cfg/node/` | Read-only | | `swiftlys2/` | [Node plugins](/operations.md#add-swiftlys2-plugins) | `csgo/addons/swiftlys2/` | | | ├─ `plugins//` | [Node plugins](/operations.md#add-swiftlys2-plugins) | ├─ `plugins//` | Read-only | | └─ `configs/plugins//` | [Node plugins](/operations.md#add-swiftlys2-plugins) | └─ `configs/plugins//` | Writable | | `shared/game/bin/linuxsteamrt64/steamapps/` | Every slot | `bin/linuxsteamrt64/steamapps/` | Writable | A launch runs your cfgs from `cfg/` by naming them in `exec`. The protocol guide lists [when they run](/protocol.md#when-the-cfgs-run), and their order among the game mode's cfg and Ankor's. [SwiftlyS2](https://swiftlys2.net/docs/development/configuration) gives a plugin its paths through these functions. They map to these directories on the node: | SwiftlyS2 | On the node | |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------| | [`Core.PluginPath`](https://swiftlys2.net/api-docs/stable/shared/iswiftlycore#swiftlys2-shared-iswiftlycore-pluginpath) | `/srv/ankor/swiftlys2/plugins//` | | [`Core.Configuration.GetConfigPath()`](https://swiftlys2.net/api-docs/stable/services/ipluginconfigurationservice#swiftlys2-shared-services-ipluginconfigurationservice-getconfigpath-system-string) | `/srv/ankor/swiftlys2/configs/plugins//` | | [`Core.PluginDataDirectory`](https://swiftlys2.net/api-docs/stable/shared/iswiftlycore#swiftlys2-shared-iswiftlycore-plugindatadirectory) | `/srv/ankor/slots//game/csgo/addons/swiftlys2/data//` | Data and logs stay across the slot's launches. ### Only on the node No game container mounts these. Only the agent reads and writes them. | On the node | Contents | |---------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `/srv/ankor/agent.yml` | The node's [configuration](#agentyml), including its node token. | | `/srv/ankor/agent-update-failed.json` | Why the last agent update failed. The agent reports it to the controller until an agent update succeeds or the setup script starts the agent. | | `/srv/ankor/events/` | The agent's SQLite database of gameplay events and ended servers that wait for the controller. Its layout is internal to the agent. An update upgrades it in place. | | `/srv/ankor/gslt-refused.json` | The SHA-256 of each login token Steam refused, so the agent never logs one on again. Deleting it and restarting the agent makes it check every token once more. | --- # Operating a node > Test servers, updates, restarts and demos on a running node. Most actions on an installed node are requests from the controller to the agent. [ankor.sh](/ankor-web.md#node-actions) shows where each action is in [ankor.sh](https://ankor.sh). ## Launch a test server Before a node runs real matches, have the controller launch a server without a match on it. This is an [`start_server`](/reference/socket/#message-start_server) with `match` set to `null`. Join the server to check that CS2 is installed, the ports are open and players can get in. The node needs an idle slot whose login token Steam has not refused. ## Automatic updates A node keeps itself up to date while [`auto_update`](/configuration.md#config-auto_update) is on, as it is by default. It follows the game build and both images together, since a CS2 update can break the plugin or SwiftlyS2. The break lasts until a fixed `ankor-cs2` release is out. | What | Source of new versions | When it updates | |--------------------|------------------------------------------------------|-----------------------------------------------------| | The CS2 game build | Steam's public build | Once every slot is idle | | The CS2 image | Tags of the `cs2_image` repository in its registry | At the next registry read. Running servers carry on | | The agent image | Tags of the `agent_image` repository in its registry | At the next registry read. Running servers carry on | [Update CS2](#update-cs2) describes the game update. [Follow image releases](#follow-image-releases) describes the image updates. [`update_cs2`](/reference/socket/#message-update_cs2), [`update_cs2_image`](/reference/socket/#message-update_cs2_image) and [`update_agent`](/reference/socket/#message-update_agent) work whether the setting is on or off. While it is on, a manual update to an older release of a line does not last. At the next registry read, the image moves back to the newest release of its [line](/requirements.md#images). That read follows the manual update immediately. To keep an older release, name it by digest in `agent.yml` or turn the setting off. ### Keep a node on its versions You can keep a node on its installed build and images. This suits a LAN event, or any event where the players' clients stay on one build. Your controller turns [`auto_update`](/configuration.md#config-auto_update) off with [`set_config`](/reference/socket/#message-set_config). Alternatively, set it to `false` in `agent.yml` and [restart the agent](#restart-the-agent). The node then updates nothing by itself, and it accepts launches while a newer public build is out. Players whose clients have updated cannot join its servers. Manual updates still run. When the setting is turned back on, a node behind the public build updates as soon as every slot is idle. The images update immediately. ## Update CS2 Once CS2 is installed, the node keeps it up to date by itself, unless you [keep it on its versions](#keep-a-node-on-its-versions). When Valve releases a new build, the node stops accepting launches, since players on the new client cannot join the old build. Running servers play on, and the node updates as soon as every slot is idle. It makes one attempt per build, and one more after an agent restart. After a failed attempt, the node accepts no launches until the next build or a manual update succeeds. A manual update, [`update_cs2`](/reference/socket/#message-update_cs2), also installs CS2 the first time. It too is refused while any slot is busy. It takes one of the [update modes](/reference/types/#schema-Cs2UpdateMode). `verify` repairs a damaged installation. `reinstall` starts over when updates keep failing. An update usually takes a few minutes and sometimes an hour. The time depends on the host's CPU, its connection and the size of the update. Each mode refuses to start without enough free space on the install volume: | Mode | Free space needed on the install volume | |--------------------|-----------------------------------------------------| | `update`, `verify` | 20 GB | | `reinstall` | 70 GB after the wipe | ## Follow image releases While [`auto_update`](/configuration.md#config-auto_update) is on, the agent reads the tags of the `cs2_image` and `agent_image` repositories from their registry. It waits an hour between reads. It also reads them when the agent starts, when the setting is turned on, and after a manual update that names other images. When the registry has a newer `X.Y.Z` release on the node's release line, the agent updates to it. A node on `latest` updates to any newer release. It updates as [`update_cs2_image`](/reference/socket/#message-update_cs2_image) and [`update_agent`](/reference/socket/#message-update_agent) would, the CS2 image first. Without a newer release, the agent compares the registry's digest for the tag with the image it runs. It updates to a [weekly rebuild](/requirements.md#images) of its release the same way. The node reports each update in [`node_status`](/reference/socket/#message-node_status), as it reports a manual one. - The agent reads the running version from the image's `org.opencontainers.image.version` label, which release images carry. It leaves an image without the label alone. - The new reference keeps the precision of the tag in `agent.yml`. `ghcr.io/ankorcs/ankor-cs2:1` stays as it is and is pulled again. `:1.2` becomes `:1.3`, and `:1.2.0` becomes `:1.3.0`. - `latest` stays `latest`, so the node moves to every new line by itself. ankor.sh issues `latest` because it deploys each new line as it ships. A controller that pins a line issues its tag, such as `:0.9`. - An image named by digest, or with a tag other than `X`, `X.Y`, `X.Y.Z` or `latest`, stays as it is. - If a pull fails, the agent tries again at the next read. If an update is refused while other maintenance runs, the agent tries again when that maintenance ends. - If the swap or the new agent fails after an agent update's pull, the node keeps the old agent. It reports the failed update in [`node_status`](/reference/socket/#message-node_status). The agent then updates only to a newer release or rebuild, not to the build that failed. - If the helper crashes or runs out of memory, its rerun finishes or rolls back the swap. After a rollback, the agent tries the same build again at the next read. - The agent reads any registry over HTTPS that allows anonymous reads, Docker Hub included. `ghcr.io` allows them for the official images. Auto-update follows an image only when `agent.yml` names it by a release tag: `X`, `X.Y`, `X.Y.Z` or `latest`. An image named by a digest, such as `ghcr.io/ankorcs/ankor-cs2@sha256:…`, or by any other tag stays on its build. The other image and CS2 itself still update. ## Update the CS2 image The plugin, SwiftlyS2 and the server configuration ship together in the `ankor-cs2` image. [`update_cs2_image`](/reference/socket/#message-update_cs2_image) switches the node to another version. Only servers launched afterwards use it. [Images](/requirements.md#images) lists the version tags. The node also [follows image releases](#follow-image-releases) by itself. ## Update the agent [`update_agent`](/reference/socket/#message-update_agent) replaces the agent's container with another version of the agent image. Running servers [carry on](/how-it-works.md#updates-and-restarts). The node also [follows image releases](#follow-image-releases) by itself. The agent cannot replace the container it runs in. Instead, it starts a helper container, `ankor-self-update-`, from the new image, and the helper makes the swap from outside. When the old agent creates the helper's container, it describes the swap in three environment variables. If one is missing, the helper exits with an error before it touches a container: | Variable | Holds | |------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------| | `ANKOR_SELF_UPDATE_PREVIOUS` | the id of the agent container that the helper replaces. | | `ANKOR_SELF_UPDATE_SPEC` | the Docker create request for the new agent's container as JSON, with the old container's environment, labels, mounts and restart policy. | | `ANKOR_SELF_UPDATE_FAILURE` | the path of `agent-update-failed.json`, in the data directory that the helper and both agents mount. | The new agent receives the old container's environment through `ANKOR_SELF_UPDATE_SPEC`. The variables that the old image set are left out, because the new image sets its own. The helper passes its result back through `agent-update-failed.json` to whichever agent runs after the swap. > Diagram: The old agent creates the helper with the three variables. The helper stops the old agent and renames it, then creates and starts the new agent under its name, which connects while the helper waits. If the new agent stays up, the helper removes the old one and the new agent reports the update as done. If not, the helper removes the new agent, renames the old one back and starts it, and records the failure, which the old agent reports. 1. The helper stops `ankor-agent` and renames it `ankor-agent.previous`, which frees the name. 2. The helper creates the new agent's container as `ankor-agent`, with the old container's settings and the new image, and starts it. The new agent connects to the controller while the helper waits, and reports the update as under way. 3. If the new agent stays up for 10 seconds without a restart, the helper removes `ankor-agent.previous`. It also deletes `agent-update-failed.json`, so an earlier failed update is no longer reported. The new agent reports the update as done. 4. If a step fails, or the new agent exits or restarts within those 10 seconds, the helper undoes the steps that ran. It removes the new container, renames the old one back to `ankor-agent` and starts it again. If the helper could not stop the old agent, nothing needs undoing. Either way, the helper writes the reason to `agent-update-failed.json` in the [data directory](/configuration.md#only-on-the-node). The old agent reports the failed update. 5. If the helper crashes or runs out of memory, Docker restarts it, up to 3 times. The rerun checks the old agent: - If the old agent is gone and the new agent runs, the swap held. The helper reports the update as done. - If the old agent is gone and no new agent runs, the helper records the failure. - If the old agent is stopped or renamed, the helper rolls the swap back as in step 4. - If Docker refuses to show the old agent, the helper exits with an error, and Docker runs it again. A helper stopped with `docker kill` stays stopped, because Docker counts it as a manual stop. Every agent reports updates in [`node_status`](/reference/socket/#message-node_status). After a rollback, the controller therefore sees the new version send [`hello`](/reference/socket/#message-hello) and disconnect, and then the old version connect again. The helper's container stays on the host after it exits, as the record of the swap. The next agent update removes it. Its log shows how the swap ended and what the helper rolled back: ```sh docker ps -a --filter name=ankor-self-update- docker logs ankor-self-update- ``` The agent removes the images that an update replaced once no container uses them. It checks after each update and when it starts. ## Update the agent from the host If the agent cannot connect, or the controller rejects its protocol version, it cannot receive [`update_agent`](/reference/socket/#message-update_agent). Update it on the host instead, with `--update` of the [setup script](/setup-script.md): ```sh curl -fsSL https://ankor.sh/node-setup.sh | bash -s -- --update ``` ## Restart the agent Restarting the agent is [safe while servers run](/how-it-works.md#updates-and-restarts). ```sh docker restart ankor-agent ``` The agent's log lists the servers it took over and its reconnection to the controller. [Logs](#logs) shows how to read it. A server whose slot has been removed from `agent.yml` is stopped at startup. Its demo still uploads. ## One host per node - [ankor.sh](https://ankor.sh) connects one agent per node token at a time. It closes the socket of a second agent with `node_connected`. A second host with the same `agent.yml` therefore keeps retrying, and connects only once the first disconnects. A controller of your own should do the same, as [Closing the socket](/reference/socket/#closing) describes. To move a node, [remove it](/install.md#remove-a-node) and set the new host up as a new node. - A host runs one agent. If an agent finds another agent container running on its host, it stops before it changes anything. Its log names both containers. Remove the extra one with `docker rm -f `. Re-running setup does not help, because setup replaces only the container named `ankor-agent`. ## Logs The agent keeps its log and the recent output of each server. The controller reads them with [`watch_logs`](/reference/socket/#message-watch_logs). On the host: ```sh docker logs --tail 200 ankor-agent docker ps -a --filter label=ankor.slot --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}' docker logs --tail 200 CONTAINER_ID ``` A game server's log is deleted with its container when the server exits. ## Server console [`console_command`](/reference/socket/#message-console_command) runs a console command on a running server. On [ankor.sh](https://ankor.sh) it is under Node options → **Logs & console**. Use it to inspect a server (`status`), not to steer a match. **Do not change the game during a match.** The events record every round as it is played. `mp_restartgame`, a map change, or cvars that change round rules, teams or money therefore leave statistics that do not match the rounds played. The damage is permanent, because the events themselves record the repeated rounds and changed rules. To play a match on from an earlier round, restore it as [Backups and restores](/backups.md#restore-a-running-match) describes. ## Replace a slot's token When Steam refuses a slot's [Game Server Login Token](/requirements.md#game-server-login-tokens), the slot receives no launches until it has a new login token. Steam refuses a token, for example, after Valve invalidated it. The controller replaces the token on an idle slot, because revoking the token of a running server drops that server. [ankor.sh](/ankor-web.md#login-tokens) replaces it automatically and on request. For a controller of your own, [Replace a slot's token](/protocol/minimal-controller.md#replace-a-slots-token) has the steps. The agent logs the new token on immediately and reports whether Steam accepted it. ## Demos When a match records a demo, the agent uploads the finished file to the node's [`demo_upload`](/configuration.md#config-demo_upload) target. It deletes the file once uploaded. Without a target, demos stay on the node and fill its disk over time. [Demo uploads](/protocol/demos.md) covers the request, the answers and the retries. An upload can fill the node's uplink and make running servers lag. [`demo_upload.limit_mbit`](/configuration.md#config-demo_upload.limit_mbit) caps the rate while any slot runs a server. Uploads return to full speed once the node is idle. To size the cap, reserve about 10 Mbit/s of the uplink for each full server that the node runs at the same time. Set the cap to the remainder. ## Add SwiftlyS2 plugins Drop the plugin's folder into `/srv/ankor/swiftlys2/plugins/`. Servers launched afterwards load it beside Ankor. Paths in the game container are relative to `/home/steam/cs2-dedicated/game/`. [File layout](/configuration.md#file-layout) shows the whole tree. | On the node | Written by | In the game container | Access | |-----------------------------------------------------|--------------------------------|---------------------------------------------------------------|------------------------------------------| | `/srv/ankor/swiftlys2/` | Setup | | | | ├─ `plugins/` | Setup | | | | │ └─ `MyPlugin/` | You | `csgo/addons/swiftlys2/plugins/MyPlugin/` | Read-only | | │ └─ `MyPlugin.dll` | You | `csgo/addons/swiftlys2/plugins/MyPlugin/MyPlugin.dll` | Read-only | | └─ `configs/plugins/` | Setup | | | | └─ `MyPlugin/` | The agent | `csgo/addons/swiftlys2/configs/plugins/MyPlugin/` | Writable | | └─ `config.jsonc` | The plugin, then you | `csgo/addons/swiftlys2/configs/plugins/MyPlugin/config.jsonc` | Writable | - The folder is the plugin as it ships, and its DLL has the folder's name. - The plugin writes its config at its first launch. Edit it there. - A server loads its plugins and their configs when it starts, so changes to either apply from the next launch. - A plugin must support the SwiftlyS2 major version of the node's CS2 image. - A folder named `Ankor` is ignored. - Plugins run in matches too, and their changes to the game appear in the match's events. - Other plugins are unsupported. They must not interfere with the parts of the game that Ankor's plugin controls, such as team assignment, the match flow and the match cvars. Report a conflict with Ankor's plugin as an [issue on GitHub](https://github.com/ankorcs/ankor/issues). ### Permissions The agent and the game servers run as uid 1000. [Setup](/setup-script.md) creates the folders above owned by that user. After copying a plugin in, make it readable for the game servers: ```sh chmod -R a+rX /srv/ankor/swiftlys2/plugins/MyPlugin ``` ## Add cfgs for servers Your cfgs go in `/srv/ankor/cfg/`, which is read-only in every game container. A server sees them in `csgo/cfg/node/`, and a launch runs one by listing `node/` in [`exec`](/reference/types/#schema-ServerLaunchConfig), without `.cfg`. [When the cfgs run](/protocol.md#when-the-cfgs-run) shows when a server runs each cfg, and in which order. Ankor's own cfgs sit apart from yours, in `csgo/cfg/ankor/`. They come from the CS2 image, and a file that you add on the node never lands there. A node cfg that a launch runs overrides `ankor/base`, but not warmup or the knife round. | On the node | Written by | In the game container | A launch uses it with | |--------------------------------------------------|--------------------------------|-----------------------------|--------------------------| | `/srv/ankor/cfg/` | Setup | `csgo/cfg/node/` | | | └─ `retakes.cfg` | You | `csgo/cfg/node/retakes.cfg` | `exec: ["node/retakes"]` | - A server reads a cfg when it runs it, so an edit applies from the next run or launch. - Make the cfgs readable to the game servers, as Permissions above shows. --- # Ready system > How a match's warmup ends, the players' ready commands, force ready and the ready deadline. A match starts in warmup. The players warm up while Ankor's plugin waits for both teams to be ready. Once they are, the plugin counts down and starts the knife round or live play. A player who leaves during the countdown does not stop it. ## Ready modes The match's [`ready_mode`](/reference/types/#schema-MatchConfig) decides when a team is ready. [`ReadyMode`](/reference/types/#schema-ReadyMode) describes each mode. In every mode, bots and players outside the roster never make a team ready. ### Auto A team is ready once all its rostered players are connected. Players have no ready commands, and the match does not wait for spectators. ### Manual Each rostered player also types `!ready` in chat, and `!unready` withdraws it. When the match sets [`spectators_ready_up`](/reference/types/#schema-MatchConfig), the match also waits for every rostered spectator to connect and type `!ready`. - A `!ready` lasts until the player leaves the server. - The plugin regularly names the players, and the spectators it waits for, who are not ready. - The scoreboard shows `READY` or `NOT READY` as each one's clan tag. ### External Players have no ready commands, and only the controller starts the match, with [`start_match`](/reference/socket/#message-start_match). ## Force ready A match allows `!forceready` only with [`allow_force_ready`](/reference/types/#schema-MatchConfig) on. Without it, the command does nothing, and the match waits for every rostered player's `!ready`. Under `manual`, a team's captain can type `!forceready`. The captain is the first player of the team in the roster. The command readies the captain's team with whichever of its players are connected. The team then stops waiting for the rest of its roster and for their `!ready`. The countdown starts once the other team is ready too, either the normal way or by its own captain's `!forceready`. When the match waits for spectators, any rostered spectator can also type `!forceready`. It readies the spectators and no team. A rostered spectator who does not show up then does not block the match, as long as another one is connected. When none of them connects, only the ready deadline or [`start_match`](/reference/socket/#message-start_match) ends the warmup. - A force-ready lasts for the rest of the warmup, and `!unready` cannot withdraw it. - A team with none of its players connected is never ready, force-readied or not. - The plugin sends each force-ready as a `warn` [`log`](/reference/uploads/#message-log) that names the team, or the spectators, and who typed it. ## Start from the controller [`start_match`](/reference/socket/#message-start_match) starts the countdown in every mode without waiting for anyone to be ready. An admin can therefore always start a match whose players are stuck in warmup. The plugin refuses it until someone from each team is connected. The reply comes when the countdown has started. ## Ready deadline [`ready_timeout_seconds`](/reference/types/#schema-MatchConfig) in the match sets a deadline for both teams to be ready. The warmup clock in game counts the deadline down, and the plugin warns in chat as it approaches. When the deadline passes, the plugin tells players that the match was canceled and ignores any later ready-up. It ends warmup, ends the match with the game's end screen and ends the map with [`map_end`](/reference/uploads/#message-map_end). Ten seconds later it kicks every player. The `reason` in [`map_end`](/reference/uploads/#message-map_end) is `ready_timeout`, and `teams` says how many players of each team were connected and whether the team was ready. A force-readied team counts as ready. The map names no winner, so your controller decides the outcome, such as a win for the ready team or a draw. - `ready_timeout_seconds` must be null under `external`, because only the controller starts the match there. - If nobody joins, the deadline still passes on the empty server. The agent's [`idle_shutdown_seconds`](/reference/types/#schema-ServerLaunchConfig) stops an empty server first when it is shorter, and the agent then reports [`server_exited`](/reference/socket/#message-server_exited) instead. --- # Backups and restores > How a match's rounds are backed up, and how a match plays on from one of them. A live match is backed up round by round. If a match's server dies, the match can play on from the start of a round it reached, on any node. A running match can also play a round again. ## Round backups As each live round starts, CS2 writes a backup of it into the slot. The backup holds the round, the scores, and the money and equipment of every player. Ankor's plugin reads the file and sends it as [`round_backup`](/reference/uploads/#message-round_backup). The event adds team 1's side, the match's rules and the roster, which the file does not hold. Warmup and the knife round have no backups. - The files are in the slot's `ankor/backups/` directory, one `ankor_roundNN.txt` per round. Each file is numbered by the rounds played before it. [Configuration and files](/configuration.md) shows the slot's layout. The agent replaces the files at the next launch in the slot. - The [`round_backup`](/reference/uploads/#message-round_backup) events are the copy that lasts. With the map's launch, the newest one is all that a restore on another node needs. - When a round's file is missing, empty or older than the round, the plugin sends no backup for that round. It sends an `error` [`log`](/reference/uploads/#message-log) instead. - CS2 writes no file for a restored round, and the plugin sends no [`round_backup`](/reference/uploads/#message-round_backup) for it. After an [`match_restored`](/reference/uploads/#message-match_restored), the match stands at the backup of the round it was restored to. The server sent that backup when the round first started. The match stays at it until a later round sends a new one. Keep every backup of the map so a controller can find it. ## Restore a running match [`restore_round`](/reference/socket/#message-restore_round) plays a running match on from one of its earlier rounds, on the same server. The plugin loads the file that CS2 wrote as that round started. The match must be live and not in the halftime break. The reference entry lists the refusals. 1. The reply comes once the plugin has asked CS2 to load the backup. 2. The restored round opens paused in its freeze time, and the plugin lifts the pause shortly after. 3. The plugin sends [`match_restored`](/reference/uploads/#message-match_restored) ahead of the restored round's [`round_start`](/reference/uploads/#message-round_start). From that round start on, the server's events replace everything it sent for that round and the rounds after it. The node keeps the backup of every round the server has played, including rounds after the one it was restored to. A restore can therefore also go forward, to play that the match has since left. The next section says when that is safe. ## The space-time continuum **Never move a match forward in time after you go back and play any part of the loaded round.** Once the restored round is played, the server's events replace everything it sent for that round and the rounds after it. The later backups on the node still describe the old play. Loading one of them creates two separate timelines in one match. Their scores, money and events no longer agree, and the problems that follow are beyond counting. A forward restore is safe only while no part of the restored round has been played. After that, a controller should refuse it. ## Restore on a new server When a match's server has died, launch the same [`match_id`](/reference/types/#schema-MatchConfig) again with [`start_server`](/reference/socket/#message-start_server). Use any node and a new `allocation_id`. Set [`match.restore`](/reference/types/#schema-MatchConfig) from the newest [`round_backup`](/reference/uploads/#message-round_backup) of the map. [`RestoreFile`](/reference/types/#schema-RestoreFile) lists its fields. A match without statistics sends each backup in [`backup`](/reference/socket/#message-backup) instead. That backup is lost if the plugin or the agent is disconnected when the round starts. 1. The agent writes the backup into the slot as `ankor/restore.txt`. 2. Warmup runs as usual, and the [ready system](/ready-system.md) decides when it ends. The plugin seats team 1 on the backup's side. 3. After the countdown, the server skips the knife round and the side choice. It starts round 1 and loads the backup from inside it. 4. The plugin sends [`match_restored`](/reference/uploads/#message-match_restored), and the match plays on under the restore's `rules`. Until the backup is loaded, the server reports the match as `pending_restore`. If the load fails, the plugin gives the reason in an [`log`](/reference/uploads/#message-log). The server stays `pending_restore` until a load is tried again or the controller stops the server. ## Loads from the console **Use [`restore_round`](/reference/socket/#message-restore_round) instead.** A load typed at the console bypasses the controller. The controller learns of it only from the events that follow. Nothing checks that the file belongs to this match or is the round you meant. A wrong file plays the match on from a state it never reached. The events record that play, so the statistics then do not match the match. If you must, load a backup from the [server console](/operations.md#server-console) with `mp_backup_restore_load_file` and the file's path relative to the game directory: ``` mp_backup_restore_load_file ankor/backups/ankor_round04.txt ``` A file is numbered by the rounds played before it, so the file of round _n_ is `ankor_round` followed by _n_ − 1 in at least two digits. The command above plays round 5 again. `ankor_round00.txt` restores round 1. The `"round"` key inside the file holds the same number. The plugin follows a console load the same way as its own, and sends [`match_restored`](/reference/uploads/#message-match_restored). It refuses to load a file that does not exist. It also refuses a load during the knife round, the side choice or the halftime break, and after the match has ended. It logs why on the server. --- # Troubleshooting > What to do for common node failures. Each row below starts from something that a node shows: a line in the agent's [log](/operations.md#logs), a status code or a refusal. Look up the exact text you see. ## Setup and connection | Symptom | Next step | |--------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Setup stops at `pulling …/ankor-agent` | Check that the host can reach the image registry, then rerun setup. | | Agent log: the controller at … refused this node's token | The agent has stopped dialling. Restart it once the controller knows the token. | | Agent log: the controller at … uses a protocol version incompatible with this agent's … | The agent has stopped dialling. [Update it from the host](/operations.md#update-the-agent-from-the-host). | | Agent log: the controller at … refused this node's hello | The agent has stopped dialling. Its hello failed the controller's schema ([`invalid_hello`](/reference/types/#schema-HelloRefusedReason)). [Update it from the host](/operations.md#update-the-agent-from-the-host). | | Node unavailable with code [`event_storage`](/reference/types/#schema-NodeUnavailableCode), `detail` says the event spool has not answered | The disk under `/srv/ankor` hangs. Check the disk, and leave the agent running. Deleting `/srv/ankor/events/` or restarting the agent loses the records that wait to be written. The agent takes launches again once the event spool answers. | | Node unavailable with code [`event_storage`](/reference/types/#schema-NodeUnavailableCode), any other `detail` | The agent cannot open its events database, for example one that a downgrade left newer than the agent. The agent log gives the cause. [Update the agent from the host](/operations.md#update-the-agent-from-the-host), or delete `/srv/ankor/events/` and restart the agent. Deleting it loses every undelivered event and server exit, including those of running servers. It also loses the record of allocations the node has launched. A running server's later events are stored again from where its plugin resumes. | | Node unavailable with code [`disk_space`](/reference/types/#schema-NodeUnavailableCode) | The disk under `/srv/ankor` has less than 1024 MiB free. Check it with `df -h /srv/ankor`, then free space. For example, remove old demos, or remove unused Docker images with `docker image prune`. The agent checks every 30 seconds and accepts launches again once there is room. | ## Game servers | Symptom | Next step | |------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------| | CS2 missing after setup | [Install it](/operations.md#update-cs2). The agent installs CS2 only when the controller requests it. | | Node refuses launches after a CS2 release | The node updates CS2 once its running servers end. After a failed automatic attempt, [update](/operations.md#update-cs2) by hand. | | Update refused for free space | Grow the disk. [Update CS2](/operations.md#update-cs2) lists the thresholds. | | Slot receives no launches, login token refused | Steam refused the slot's login token. [Replace it](/operations.md#replace-a-slots-token). | | Missing game files or a mountpoint | Wait until the node is idle, then update CS2. | | Stop is unconfirmed | Check Docker on the host. The slot stays occupied until Docker reports the container removed. | --- # Protocol overview > Everything a controller exchanges with Ankor agents, for anyone building their own. The Ankor protocol is how a controller runs game servers on Ankor nodes. Each node's agent connects to the controller over a WebSocket and keeps that connection open. Over the socket, the controller tells the node what to do, such as installing CS2 or starting a game server. The node reports what it is doing: which slots are busy, which CS2 build it runs and what its game servers print. Gameplay events and recorded demos are uploaded separately over HTTP. A node can sit behind NAT, because the agent opens every connection. The node needs no inbound ports apart from the game ports that players join on. Your controller accepts the connections and answers them. This page walks through your controller's tasks step by step. [A minimal controller](/protocol/minimal-controller.md) takes one node from token to running server. The [protocol reference](/reference/) has every message, field and example. In Node.js, [`@ankorcs/protocol`](https://www.npmjs.com/package/@ankorcs/protocol) does the work: ```sh npm install @ankorcs/protocol ``` - `@ankorcs/protocol/session` runs the request side for you. An agent session sends each request under an id and parses every report. It resolves each request with the typed `data` or `error` of its reply, or with a timeout or the socket's close. It works with any WebSocket server if you hand it each frame and a function that writes one. - `@ankorcs/protocol` has the zod schema of every message and the constants. A controller can also parse and build every message with these schemas directly. - `@ankorcs/protocol/helpers` holds the functions that both ends share. Among them are `bearerNodeToken` for the upgrade, `slotTakesLaunch` for ranking nodes and `withLogBatch` for applying a [`log_batch`](/reference/socket/#message-log_batch). Each function's JSDoc documents it. Check the [protocol version](/protocol/versioning.md) on the upgrade, then run one session per accepted socket: ```ts import { PROTOCOL_HEADER } from '@ankorcs/protocol'; import { isCompatibleProtocol, protocolVersionOf, upgradeRefusal } from '@ankorcs/protocol/helpers'; import { createAgentSession } from '@ankorcs/protocol/session'; server.on('upgrade', (request, socket, head) => { const protocol = request.headers[PROTOCOL_HEADER.toLowerCase()]; // Check the node token as well, as step 1 describes. if (!isCompatibleProtocol(protocol)) { socket.end(upgradeRefusal(426)); return; } // The node's version, to offer it only the features it has. const version = protocolVersionOf(protocol); // … accept the socket. }); const node = createAgentSession({ send: (text) => ws.send(text) }); ws.on('message', (raw) => { const frame = node.receive(raw); if (frame.kind === 'report') { handleReport(frame.message); } }); ws.on('close', () => node.closed()); const result = await node.request({ type: 'reserve_slot', slot_id: 's1' }); ``` - `request` adds the `id` and resolves with the reply's `data` or `error`, or with `timeout` or `closed` when no reply came. Without a reply the agent may still have done the work, and its reports say whether it did. - A request waits 30 seconds by default. A `stop_server` waits 10 seconds for the stop and 35 seconds more, because its reply comes once the container has stopped. Set `timeoutMs` on the session or on one call, or null for no deadline. A stop behind a launch of the same allocation waits for that launch's reply first, so give such a stop a longer wait. - `receive` reads a frame as it arrived, and `receiveParsed` one your server already parsed from JSON. Each returns a `report`, a `reply` with whether it answered an open request, `unsupported` for a message from a newer agent, which you skip, or `invalid` with zod's `issues`. - `notify` sends the frames that get no reply, such as [`welcome`](/reference/socket/#message-welcome) and the watches. The session checks what it sends against the contract, so a request the contract refuses never reaches the socket. ## What you need | Endpoint | Caller | Auth | Support | Reference | |---------------------------|---------------------------|-----------------------|---------------------------------------|---------------------------------------| | `WS ` | agent | bearer token | Required | [Steps 1–4](#1-accept-the-connection) | | `POST ` | agent | bearer token | Required for matches with statistics | [Event uploads](/protocol/events.md) | | `POST ` | agent | the configured header | Optional | [Demo uploads](/protocol/demos.md) | | `POST /api/node/setup` | Ankor's `node-setup.sh` | bearer token | Optional | [Setup endpoint](/protocol/node-setup.md) | The socket runs game servers. Ankor's setup script fetches a node's `agent.yml` from the setup endpoint. Without that endpoint, operators write the file themselves. The reference marks each message `required` or `optional`, and optional ones can be ignored. The agent builds its URLs from [`controller.host`](/configuration.md#config-controller.host), [`controller.tls`](/configuration.md#config-controller.tls), [`controller.socket_path`](/configuration.md#config-controller.socket_path) (default `/ws/agent`) and [`controller.events_path`](/configuration.md#config-controller.events_path) (default `/api/node/events`) in its [`agent.yml`](/configuration.md#agentyml). It sends two headers with the socket upgrade and every event upload: `Authorization: Bearer ` and `Ankor-Protocol`. ### Node tokens Your controller issues one node token per node. A token is 32 to 128 letters and digits. Generate it from a cryptographic random source, for example 32 random bytes as hex: ```sh openssl rand -hex 32 ``` ```js import { randomBytes } from 'node:crypto'; const token = randomBytes(32).toString('hex'); ``` - An agent whose [`token`](/configuration.md#config-token) has another shape refuses to start. - Store a hash of the token rather than the token. A SHA-256 is enough for a random one. ## Messages and replies - Every socket frame is one JSON object with a `type`. - Every field of what the agent sends is present. `| null` means the value may be null. - Edits you send, such as [`set_config`](/reference/socket/#message-set_config), may leave out the fields the reference marks optional. - Unknown keys are ignored on both sides. - Requests carry an `id` that you choose. The agent answers each request with one [`reply`](/reference/socket/#message-reply), in any order. The reply sets `reply_to` to that `id` and `request` to the request's `type`. In a failure, `request` is `null` when the agent did not know the `type`. - Keep `id`s unique while their requests run. The agent ignores a request whose `id` is still running on the connection, because a reply could not tell the two apart. It logs a warning. - A reply has `ok: true` and `data`, or `ok: false` and an `error`. Branch on `error.code`. - `request` says what `data` holds, so a success can be handled without its request. A failure is about the request's arguments, so look the request up by `reply_to`. - When a request has an unknown `type` or does not match its schema, the agent answers with an [`AgentErrorCode`](/reference/types/#schema-AgentErrorCode) saying which. - A request stays open until its work is done, so time requests out by kind. The reference entry of each request says what its reply waits for. A request that only changes something replies with the id of the object it changed: **[`reserve_slot`](/reference/socket/#message-reserve_slot)** (controller → agent) ```json { "type": "reserve_slot", "id": "r2", "slot_id": "s1" } ``` **[`reply`](/reference/socket/#message-reply)** (agent → controller) ```json { "type": "reply", "reply_to": "r2", "request": "reserve_slot", "ok": true, "data": { "slot_id": "s1" } } ``` A failed request: **[`stop_server`](/reference/socket/#message-stop_server)** (controller → agent) ```json { "type": "stop_server", "id": "r9", "allocation_id": "9c04e1d2-3a7b-…" } ``` **[`reply`](/reference/socket/#message-reply)** (agent → controller) ```json { "type": "reply", "reply_to": "r9", "request": "stop_server", "ok": false, "error": { "code": "unknown_allocation", "detail": "unknown_allocation" } } ``` ## Versions - `Ankor-Protocol` is the protocol's `major.minor`. A peer on an incompatible version is refused with `426`. [Versioning](/protocol/versioning.md) says which versions are compatible, and how a controller handles a message or key that a newer agent sends. - Pin each node to its [release line's](/requirements.md#images) tag, such as `ghcr.io/ankorcs/ankor-agent:0.9`, in its `agent.yml` and in updates. Never pin `latest`. Releases on a line only add messages and fields, which a controller can ignore until it uses them. - Read the agent's version from `agent.version` in [`hello`](/reference/socket/#message-hello). Still welcome a node on a line your controller does not support, but launch no servers on it. You can then still move it with [`update_agent`](/reference/socket/#message-update_agent). ## 1. Accept the connection The agent connects to [`controller.socket_path`](/configuration.md#config-controller.socket_path) with an ordinary WebSocket upgrade. The example uses the default. Check `Authorization` and [`Ankor-Protocol`](#versions) before you accept. **`GET /ws/agent`** (agent → controller) ```http GET /ws/agent HTTP/1.1 Host: cs.example.com Upgrade: websocket Connection: Upgrade Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ== Sec-WebSocket-Version: 13 Authorization: Bearer 3f1c… Ankor-Protocol: ``` **`101 Switching Protocols`** (controller → agent) ```http HTTP/1.1 101 Switching Protocols Upgrade: websocket Connection: Upgrade Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo= ``` Your WebSocket library writes the `101`. You only decide whether to accept. To refuse, answer with a plain status. Only the status matters, so the body can be empty. **`401 Unauthorized`** (controller → agent) ```http HTTP/1.1 401 Unauthorized Content-Length: 0 ``` **`426 Upgrade Required`** (controller → agent) ```http HTTP/1.1 426 Upgrade Required Content-Length: 0 ``` - `401` for an unknown token. The agent stops trying until it is restarted. - `426` for a missing or incompatible [protocol version](/protocol/versioning.md). The agent stops trying too. - `429` to make it wait, as for a node over its [limits](/protocol/limits.md). Never answer `401` for that, because the agent would stop dialling until it is restarted. - A `5xx` if something on your side is temporarily broken. The agent retries a `429` or `5xx` as [How the agent reconnects](#how-the-agent-reconnects) describes. Keep one socket per node. [ankor.sh](https://ankor.sh), for example, [closes](/reference/socket/#closing) a second socket with `node_connected` until the first closes or [times out](#4-keep-the-connection-alive). The agent keeps retrying. A host whose connection dropped therefore reconnects once you close its old socket. When you close a socket, fail the requests still open on it. A session from `@ankorcs/protocol/session` does that on `closed()`. ## 2. Welcome the node The agent's first message is [`hello`](/reference/socket/#message-hello). It carries the node's configuration, its slots and their servers, its CS2 build and its demos. Save what it reports and answer [`welcome`](/reference/socket/#message-welcome), also for a node on a [release line you do not support](#versions). Send no requests before that. A node with `unavailable` set stays connected but accepts no launches. The value of `unavailable` gives the reason. **[`hello`](/reference/socket/#message-hello)** (agent → controller) ```json { "type": "hello", "agent": { "version": "0.1.0", … }, "public_ip": "203.0.113.7", "country": "FR", "config": { "slots": [ … ], … }, "slot_status": [ { "slot_id": "s1", "state": "idle", … } ], "cs2": { "build_id": "20114612", … }, … } ``` **[`welcome`](/reference/socket/#message-welcome)** (controller → agent) ```json { "type": "welcome" } ``` If a hello fails its schema, answer [`hello_refused`](/reference/socket/#message-hello_refused) with [`invalid_hello`](/reference/types/#schema-HelloRefusedReason). Then [close the socket](/reference/socket/#closing) with `hello_refused`. The agent logs the `detail` and stops dialling until it is restarted. Accept every other hello, whatever its `agent.version`. An agent on a version that your controller does not support stays connected but receives no launches. You can still send it [`update_agent`](/reference/socket/#message-update_agent). If you refused it at hello, it would stop dialling, and its host would have to update it. ## 3. Node sends information After the [`welcome`](/reference/socket/#message-welcome), the agent reports every change by itself, so there is nothing to poll. [`slot_status`](/reference/socket/#message-slot_status) reports each slot change. [`node_status`](/reference/socket/#message-node_status) reports each change to the node's state. **[`slot_status`](/reference/socket/#message-slot_status)** (agent → controller) ```json { "type": "slot_status", "status": { "slot_id": "s1", "state": "running", "allocation_id": "5b1e2c7a-8d4f-…", … } } ``` **[`node_status`](/reference/socket/#message-node_status)** (agent → controller) ```json { "type": "node_status", "cs2": { "build_id": "20114612", "latest_build_id": "20114670", … }, … } ``` Some reports arrive only while you watch them. A [`watch_logs`](/reference/socket/#message-watch_logs) or [`watch_resources`](/reference/socket/#message-watch_resources) request stays in force for 2 minutes. Send it again before then to keep the reports coming. - [`resource_batch`](/reference/socket/#message-resource_batch), while a [`watch_resources`](/reference/socket/#message-watch_resources) request is in force. - [`log_batch`](/reference/socket/#message-log_batch) with the lines of one source, while a [`watch_logs`](/reference/socket/#message-watch_logs) request for it is in force. - [`log_index`](/reference/socket/#message-log_index), listing every source with its line count, while any source is watched. > Diagram: The agent upgrades to a WebSocket with its token, sends hello, gets welcome, catches up with its log, then reports slot and node changes. ## 4. Keep the connection alive The agent pings every 30 seconds. WebSocket libraries answer pings for you. Close the socket of an agent that has sent nothing for 65 seconds. A ping counts as something sent, so an idle node stays connected. ### How the agent reconnects - Each dial has 10 seconds to finish the upgrade. - After a lost or failed connection, the agent dials again after a second. The delay doubles with each attempt, up to 30 seconds. Each delay is randomised by ±25%. - The delay starts over once a [`welcome`](/reference/socket/#message-welcome) arrives. - The agent also reconnects when a ping is still unanswered as the next one is due. - When the agent stops dialling, it also stops [uploading events](/protocol/events.md) until it is restarted. Events keep collecting on the node. - The agent replies only on the socket that a request arrived on. When that socket closes, the agent drops any reply still due. [Disconnects and restarts](/protocol/minimal-controller.md#disconnects-and-restarts) shows how to find a launch whose reply was lost. [Closing the socket](/reference/socket/#closing) lists each close and the agent's response to it. After every [`welcome`](/reference/socket/#message-welcome), the agent sends what the [`hello`](/reference/socket/#message-hello) did not cover, in the order that [`welcome`](/reference/socket/#message-welcome) describes. ## 5. Install CS2 A new node has no CS2. Send [`update_cs2`](/reference/socket/#message-update_cs2) once to install it. - The reply comes immediately and carries the update's `update_id`. - `cs2.update` in [`node_status`](/reference/socket/#message-node_status) reports the same `id`. It gets its `outcome` when the install has ended. - Every [`cs2_update_progress`](/reference/socket/#message-cs2_update_progress) names its update by `update_id`. After the install, the agent [updates CS2 by itself](/operations.md#update-cs2). To make it check for a new build immediately, send [`check_cs2_build`](/reference/socket/#message-check_cs2_build). ## 6. Launch a game Send [`start_server`](/reference/socket/#message-start_server) with a request `id`, a new [allocation id](/reference/types/#schema-AllocationId) and the launch configuration, such as the map and the roster. The allocation id names the server and its events, so every launch needs a new one. The agent picks a free, unreserved slot whose login token Steam has not refused. It starts the server with that slot's token. Once the container has started, the agent replies with the allocation and the slot's id. A `steam_web_api_key` in a launch is readable by whoever runs the node. [Trust](/how-it-works.md#trust) covers the risk. **[`start_server`](/reference/socket/#message-start_server)** (controller → agent) ```json { "type": "start_server", "id": "r1", "allocation_id": "5b1e2c7a-8d4f-…", "launch": { "map_name": "de_mirage", "game_mode": "competitive", "max_players": 10, "exec": [], "commands": [], "match": { "roster": { … }, "rules": { "max_rounds": 24, … }, … }, … } } ``` **[`start_server_reply`](/reference/socket/#message-start_server_reply)** (agent → controller) ```json { "type": "reply", "reply_to": "r1", "request": "start_server", "ok": true, "data": { "allocation_id": "5b1e2c7a-8d4f-…", "slot_id": "s1" } } ``` A launch with `match` set to `null` leaves Ankor's plugin idle. The server plays the launch's game mode, with the cfgs in `exec`, the console lines in `commands` and the node's own plugins. Use such launches to run your own game modes on Ankor nodes. A [`StartRefusal`](/reference/types/#schema-StartRefusal) comes before the node reserves a slot, so you can try the launch on another node. Each refusal's entry in [`AgentErrorCode`](/reference/types/#schema-AgentErrorCode) says when the node takes launches again. When the reply arrives, CS2 is still starting. Wait for that slot's [`slot_status`](/reference/socket/#message-slot_status) to report `running`, then [hand out the address](/protocol/minimal-controller.md#8-hand-out-the-address). When the server has ended, the agent sends [`server_exited`](/reference/socket/#message-server_exited) with its `allocation_id`. Then it sends a [`slot_status`](/reference/socket/#message-slot_status) with the slot back to `idle` and free for another launch. `reason` in [`server_exited`](/reference/socket/#message-server_exited) gives the reason the server ended, as one of the [exit reasons](/reference/types/#schema-ServerExitReason). `exit_code` carries the container's exit code. To stop a server early, send [`stop_server`](/reference/socket/#message-stop_server). The agent reports the slot `stopping`, then `idle`, and replies once the container has stopped. A match whose server died can play on from a round it reached. Launch the same [`match_id`](/reference/types/#schema-MatchConfig) again, on any node and under a new `allocation_id`. Take its [`match.restore`](/reference/types/#schema-MatchConfig) from the newest [`round_backup`](/reference/uploads/#message-round_backup). To play a running match on from an earlier round, send [`restore_round`](/reference/socket/#message-restore_round). The plugin confirms either with [`match_restored`](/reference/uploads/#message-match_restored). [Backups and restores](/backups.md) describes both. Warmup ends as the match's [`ready_mode`](/reference/types/#schema-MatchConfig) says. [`start_match`](/reference/socket/#message-start_match) starts the match in every mode. With `external`, it is the only way to start the match, because the players cannot. The reply comes when the countdown has started. The [ready system](/ready-system.md) describes each mode and the ready deadline. ### Cfgs and commands [`start_server`](/reference/socket/#message-start_server) carries the launch's settings in its `launch` field, a [`ServerLaunchConfig`](/reference/types/#schema-ServerLaunchConfig). The agent writes them into one cfg, `cfg/ankor_launch.cfg`, so that they all run together. The plugin execs it on one console line with Ankor's cfgs, and CS2 runs that line on a single tick. No other command runs between the settings. For a match, the file looks like this: ```c // generated by @ankorcs/agent - overwritten on every launch, do not edit // 1. Only when the launch sets a hostname. hostname "Ankor | cs.example.com" // 2. Only with the launch's gotv: its delay and any password. Without gotv, a match that // records a demo gets tv_delay 0 and a random tv_password that the agent sends nowhere. tv_delay 90 tv_password "spectate" // 3. Only for a match: its rules, or the restore's rules when the launch has one. The two // friendly-fire lines appear only when friendly_fire is off. mp_maxrounds 24 mp_overtime_enable 1 mp_overtime_maxrounds 6 ff_damage_reduction_bullets 0 ff_damage_reduction_other 0 // 4. Only when the launch's exec lists cfgs: one exec for each, in order. For example: exec node/league // 5. Only when the launch has commands: each line, in order. For example: mp_team_timeout_max 2 ``` - The match's `rules` are a field of its [`MatchConfig`](/reference/types/#schema-MatchConfig). - A name in `exec` is a cfg path under `game/csgo/cfg/` without `.cfg`, such as `node/league` for one of the [node's cfgs](/operations.md#add-cfgs-for-servers). A later line wins over an earlier one. `commands` therefore override the cfgs in `exec`, and both override `rules`. Keep `commands` consistent with `rules`, because [`going_live`](/reference/uploads/#message-going_live) reports `rules` as the format of the match. **Leave the match flow to the plugin** In a match, keep `exec` and `commands` to settings. Leave out commands that move the match on, such as `mp_warmup_end`, `mp_restartgame`, `mp_swapteams`, `mp_pause_match` or a map change. They run at warmup, at the side choice and again when the match goes live. There they disrupt the plugin, which runs warmup, the knife round and going live itself. ### Ankor's cfgs and your own The CS2 image ships Ankor's cfgs under `cfg/ankor/`. Their [source](https://github.com/ankorcs/ankor/tree/master/packages/plugin/cfg) lists each setting. - `cfg/ankor/base.cfg` holds Ankor's settings for every match, whatever the game mode. It keeps an empty server running and hands team assignment and the end of warmup to the plugin. - `cfg/ankor/reset.cfg` puts what warmup and the knife round change back to CS2's defaults. It runs ahead of the game mode's cfg, so the mode's values then apply over it. - `cfg/ankor/warmup.cfg` sets a practice warmup: respawns, full money, buying anywhere and no round or time limit. - `cfg/ankor/knife.cfg` sets a knife-only round with no money, buying or respawns. Every cfg is read-only in the game container. CS2 ships the game mode's cfgs, the CS2 image ships Ankor's, and the agent writes `ankor_launch` for each launch. To change a setting, use one of these: - **A node cfg**, which the node's operator provides. The operator puts `.cfg` in `/srv/ankor/cfg/` on the node, and the server reads it as `cfg/node/.cfg`. A launch runs it with `exec: ["node/"]`. [Add cfgs for servers](/operations.md#add-cfgs-for-servers) has the operator's steps. - **A line in `commands`**, which the controller sends with the launch. It needs no file on the node. Both run inside `ankor_launch`, so they override the game mode's cfg and `ankor/base`. ### When the cfgs run The engine runs the game mode's cfg, such as `gamemode_competitive`, on every map load. Ankor's plugin runs `ankor_launch` over it, together with Ankor's cfgs. Each step lists its cfgs in the order that they run, as paths under `game/csgo/`: > Diagram: After a map loads, the engine runs the game mode's cfg. One second later, a server without a match runs ankor_launch. A match runs ankor/base, ankor_launch and ankor/warmup for its warmup, ankor/knife for an optional knife round, ankor/reset, the game mode's cfg, ankor/base, ankor_launch and ankor/warmup for the side choice after it, and ankor/reset, the game mode's cfg, ankor/base and ankor_launch when it goes live. `ankor/warmup` and `ankor/knife` run over the launch's settings, so warmup and the knife round play the same on every launch. Going live runs `ankor/reset` and then the game mode's cfg, which undo their changes. The match therefore plays the game mode's format with the launch's settings. - A line in `commands` runs at every point where `ankor_launch` runs. In a match, that is warmup, the side choice and going live. A command that acts instead of setting a value, such as `mp_restartgame`, acts at each point. - On a server without a match, the game mode's settings apply for the first second after a map loads, until `ankor_launch` runs. - [`restore_round`](/reference/socket/#message-restore_round) plays a round of the running match again and runs no cfgs. The match plays on under the settings it has. A launch's `restore` starts a new server instead, which goes live on the cfgs above. - Leave hibernation off in `exec`, `commands` and the node's cfgs and plugins. The CS2 image starts every server with `sv_hibernate_when_empty 0`, and `ankor/base` sets it again. A hibernating server runs no frames, so Ankor's plugin runs no timers and reports no players. On a server without a match, `ankor_launch` then does not run after a map loads. The agent also never sees the server empty, so [`idle_shutdown_seconds`](/reference/types/#schema-ServerLaunchConfig) never stops it. ## 7. Get the result Every match reports its phase in [`phase_change`](/reference/socket/#message-phase_change) and its score in [`score`](/reference/socket/#message-score) on the socket. The match is over when its phase is `post_live`. The last [`score`](/reference/socket/#message-score) is the final score. The agent sends each once. If one is lost while the node is disconnected, [`get_server_status`](/reference/socket/#message-get_server_status) reads the phase and score again. A match that sends statistics, as [`match.send_statistics`](/reference/types/#schema-MatchConfig) asks, also uploads its gameplay events to [the events path](/protocol/events.md). You build statistics from them there. The last event of a map is [`map_end`](/reference/uploads/#message-map_end). Its [`reason`](/reference/types/#schema-MapEndReason) says why the map ended, and only a map that was played out names a winner. --- # Versioning > Which versions of the protocol, the agent, the CS2 image and a controller work together, and what each part does with a newer peer. Ankor has four parts that release separately: the `@ankorcs/protocol` package, the agent image, the CS2 image and your controller. The protocol and both images share their `major.minor`, and only their patch versions differ, so releases with the same `major.minor` work together under [SemVer](https://semver.org/). The protocol's version decides whether two parts can talk. [Images](/requirements.md#images) explains the image tags and how a node moves between releases. ## The protocol version Every request between the parts carries the protocol's version in the `Ankor-Protocol` HTTP header. The value is the `major.minor` of `@ankorcs/protocol`, such as `Ankor-Protocol: 0.9`. These requests carry it: - the agent's [socket upgrade](/protocol.md#1-accept-the-connection), [event uploads](/protocol/events.md) and [demo uploads](/protocol/demos.md); - the [setup script's](/protocol/node-setup.md) request for `agent.yml`; - the plugin's [socket upgrade](/reference/plugin/#connecting) to its agent. The receiver accepts a peer whose version has the same major as its own, at any minor. It answers `426 Upgrade Required` to any other. This holds for `0.x` too, so a breaking change goes to the next major there as well. The receiver can keep the peer's version. A controller then knows which features each node has. ## What a release may change | Change | Release | |---------------------------------------------------------------|-----------| | A new optional key | Minor | | A new request type | Minor | | A new message or gameplay event that is not a request | Minor | | A new enum value | **Major** | | A new required key, or a key that is removed or renamed | **Major** | | A key or value whose meaning changes | **Major** | A new enum value is a major, because an older receiver rejects a value it does not know. ## Receiving from a newer peer Between compatible versions, a peer may be on a newer minor than the receiver. The controller, the agent and the plugin each handle what they do not know in the same way: - A key the receiver does not know is ignored. - A request whose `type` it does not know is answered with [`unsupported_request`](/reference/types/#schema-AgentErrorCode). - Any other message or gameplay event whose `type` it does not know is skipped, and the type is logged once. - A skipped gameplay event still takes its sequence number, so its server's stream has no gap. ## In a controller 1. Check `Ankor-Protocol` with `isCompatibleProtocol` from `@ankorcs/protocol/helpers`, and answer `426` when it fails. Keep the node's version from `protocolVersionOf`. 2. Read the agent's frames with `@ankorcs/protocol/session`. It returns `unsupported` for a message that this version does not have. Skip that frame instead of reporting an error. 3. In an [event upload](/protocol/events.md), test each event with `unknownMessageType(PluginToControllerMessage, frame.event)`. Count the `seq` of an unknown event as received, and store or quarantine nothing of it. 4. Read an `unsupported_request` reply as a node that does not have the feature yet. Offer a feature only on nodes whose protocol version has it. --- # A minimal controller > The shortest path from a new node token to a running CS2 server on a node of your own controller. These steps take one node from a new token to a running CS2 server on it. The [protocol overview](/protocol.md) explains each exchange, and this page puts them in order. The examples use `cs.example.com` for your controller. ## Talking to the agent The [protocol overview](/protocol.md) describes how to accept a node's connection and exchange messages with its agent. In Node.js, the `@ankorcs/protocol` package holds the message schemas. Its session, `@ankorcs/protocol/session`, runs the exchanges for you. Agents may run a newer protocol version than your controller. [Versioning](/protocol/versioning.md) lists what your controller must accept from them, and what it skips. ## Setup ### 1. Create the node and its token Generate a [node token](/protocol.md#node-tokens). Show the token to the operator once and store only its SHA-256. ### 2. Give the operator the setup command The [setup script](/setup-script.md) is served from ankor.sh for every controller. `--controller` points it at yours: ```sh curl -fsSL https://ankor.sh/node-setup.sh | bash -s -- --controller https://cs.example.com ``` The operator also opens the game ports of the slots you add in step 5. [Install a node](/install.md) has the step. ### 3. Answer the setup request The script posts the token to your [setup endpoint](/protocol/node-setup.md). Look the token up by its hash. Answer with the node's `agent.yml`, and put in it the token that the request carried, since you store only the hash. `slots` may be empty: ```yaml token: controller: host: cs.example.com tls: true cs2_image: ghcr.io/ankorcs/ankor-cs2:0.9 agent_image: ghcr.io/ankorcs/ankor-agent:0.9 slots: [] ``` In TypeScript, `renderAgentYml` from `@ankorcs/protocol/agent-yml` writes this file. Without the endpoint, the operator writes this file before running the script. [Write `agent.yml` yourself](/setup-script.md#write-agentyml-yourself) has the steps. ### 4. Accept the socket and welcome the node The agent dials [`controller.socket_path`](/configuration.md#config-controller.socket_path) with its token and its [protocol version](/protocol/versioning.md). [Accept the upgrade](/protocol.md#1-accept-the-connection) only when both check out, then answer its [`hello`](/reference/socket/#message-hello) with [`welcome`](/reference/socket/#message-welcome). With `@ankorcs/protocol` and the `ws` package: ```ts import { createHash } from 'node:crypto'; import { WebSocketServer } from 'ws'; import { PROTOCOL_HEADER } from '@ankorcs/protocol'; import { bearerNodeToken, isCompatibleProtocol, protocolVersionOf, upgradeRefusal } from '@ankorcs/protocol/helpers'; import { createAgentSession } from '@ankorcs/protocol/session'; const sockets = new WebSocketServer({ noServer: true }); server.on('upgrade', (request, socket, head) => { const token = bearerNodeToken(request.headers.authorization); const node = token === null ? null : findNodeByTokenHash(createHash('sha256').update(token).digest('hex')); if (node === null) { socket.end(upgradeRefusal(401)); return; } const protocol = request.headers[PROTOCOL_HEADER.toLowerCase()]; if (!isCompatibleProtocol(protocol)) { socket.end(upgradeRefusal(426)); return; } // The node's version, to offer it only the features it has. node.protocol = protocolVersionOf(protocol); sockets.handleUpgrade(request, socket, head, (ws) => run(node, ws)); }); function run(node, ws) { const session = createAgentSession({ send: (text) => ws.send(text) }); ws.on('message', (raw) => { const frame = session.receive(raw); if (frame.kind === 'report' && frame.message.type === 'hello') { saveHello(node, frame.message); session.notify({ type: 'welcome' }); } // Skip `unsupported`, a message from a newer agent. }); ws.on('close', () => session.closed()); } ``` ### 5. Add slots With no slots in `agent.yml`, send [`set_config`](/reference/socket/#message-set_config) with each slot's ports and its own [Game Server Login Token](/requirements.md#game-server-login-tokens). `slots` is the whole layout: ```json { "type": "set_config", "id": "r1", "config": { "slots": [ { "id": "s1", "port": 27015, "tv_port": 27020, "gslt": "" } ] } } ``` The reply carries the node's settings with the new slots. An [`slot_status`](/reference/socket/#message-slot_status) follows for each new slot. With the session: ```ts const result = await node.request({ type: 'set_config', config: { slots } }); if (result.status === 'ok') { saveSettings(result.data); } ``` A controller can create the tokens with a user's Steam Web API key. [`IGameServersService`](https://partner.steamgames.com/doc/webapi/IGameServersService) `CreateAccount`, with `appid` `730` and a `memo`, returns the `login_token`. `GetAccountList`, `ResetLoginToken` and `DeleteAccount` manage tokens later. [ankor.sh](/ankor-web.md#login-tokens) creates its tokens this way. ### 6. Install CS2 Send [`update_cs2`](/reference/socket/#message-update_cs2) with `mode` `update`, then wait until `cs2.update` in [`node_status`](/reference/socket/#message-node_status) has an `outcome`. The install can take tens of minutes. [Install CS2](/protocol.md#5-install-cs2) explains the reports. ### 7. Start a server Store a new allocation id, then send it in [`start_server`](/reference/socket/#message-start_server). [Keep track of every server](#keep-track-of-every-server) explains why. This launch leaves out every field that has a default. It therefore has no `match`, and runs a plain CS2 server on `de_dust2` in competitive mode. Ankor's plugin runs no match. `map_name`, `exec`, `commands` and the node's own plugins set the game. [`ServerLaunchConfig`](/reference/types/#schema-ServerLaunchConfig) lists the defaults. The agent replies with the allocation and the slot it picked. ```json { "type": "start_server", "id": "r3", "allocation_id": "e7a3d9c1-6b2f-4e8a-9c5d-1f0b3a7e2d64", "launch": { "hostname": "Ankor", "map_name": "de_dust2", "max_players": 10, "idle_shutdown_seconds": null, "exec": [], "commands": [] } } ``` ### 8. Hand out the address Wait for the slot's [`slot_status`](/reference/socket/#message-slot_status) with `state` `running`. Players connect to the node's address and the slot's `port`, both from the latest [`hello`](/reference/socket/#message-hello). The address is `config.pinned_ip`, or `public_ip` when none is pinned. A `public_ip` of null means the node found no public address, so set `pinned_ip` with [`set_config`](/reference/socket/#message-set_config) first. A player then connects with: ```text connect 203.0.113.7:27015 ``` [`stop_server`](/reference/socket/#message-stop_server) ends the server. To run matches with statistics, launch with a [`MatchConfig`](/reference/types/#schema-MatchConfig) whose `send_statistics` is `true`, and accept [event uploads](/protocol/events.md). ## Keep track of every server Launches and stops race with the node's reports: - A reply may never come. - A node may reconnect mid-launch. - A stop may arrive before the launch it ends. Store each server's state and update it only from the node's reports. A stop is done once the node confirms it. ### Launches - Pick a new `allocation_id` for every launch and store it before you send [`start_server`](/reference/socket/#message-start_server). - Rank nodes by their latest reports with `slotTakesLaunch` and `cs2Behind` from `@ankorcs/protocol/helpers`. The node still decides, and may refuse. - The server is running once the reply or a [`slot_status`](/reference/socket/#message-slot_status) names its slot, whichever comes first. - A [`StartRefusal`](/reference/types/#schema-StartRefusal) means no server started. Try the next node. - `allocation_in_use` answers a [`start_server`](/reference/socket/#message-start_server) sent again with the same `allocation_id`. It means the first launch is starting or running on that node. Wait for its [`slot_status`](/reference/socket/#message-slot_status) instead of retrying on a different node. - Any other error, or no reply at all, may leave a server running, because the node's reports can arrive after the reply would have. Stop it on that node before you launch it anywhere else. With the session, no reply is a `timeout` or `closed` result. ### Stops - A stop may be sent while the server's launch is still waiting for its reply. The node then ends the launch after its current step, without starting the server. The launch's reply is an error. If the stop arrives while Docker starts the container, the launch succeeds and the server is then stopped. - The reply to a stop comes within 10 seconds for the stop and 35 seconds more. With the session, the stop waits that long whatever wait the session or the call sets. - `unknown_allocation` means no server runs under the allocation. `failed` leaves the server running. Send the stop again. ### Disconnects and restarts - A disconnected node may still run its servers. Keep their records until the node reconnects. - Compare each [`hello`](/reference/socket/#message-hello) with your records. A server that the `hello` no longer lists has ended. A launch whose reply never came is running if the `hello` lists it. Stop a server that you have no record of. - `server_exits` in the `hello` gives the reason for each server that ended while the node was away. A server with no entry there, and no [`server_exited`](/reference/socket/#message-server_exited) after the [`welcome`](/reference/socket/#message-welcome), ended without the agent knowing why. - An exit after the agent sent the [`hello`](/reference/socket/#message-hello) arrives as [`server_exited`](/reference/socket/#message-server_exited) after the [`welcome`](/reference/socket/#message-welcome). It can repeat what the `hello` already told you. - Keep [one socket per node](/protocol.md#1-accept-the-connection). ### Logs - Send [`watch_logs`](/reference/socket/#message-watch_logs) for a source while someone is looking at it. Send it again before 2 minutes pass to keep the source streaming. Each source needs a separate request. - Ask for a replay with a `replay_id` when a viewer opens a source. Apply each [`log_batch`](/reference/socket/#message-log_batch) with `withLogBatch` from `@ankorcs/protocol/helpers`. A batch with `reset` replaces the lines you hold. - A line can carry ANSI colour escapes (SGR, such as `ESC[32m`), as the server printed them. Render them as colours, or strip each `ESC[` … `m` to show plain text. The agent has already removed every other escape and control character, so a line cannot move a cursor, open a link or write to a clipboard. A line cut at the length limit can end in part of an escape; drop that part. [`LogLine`](/reference/types/#schema-LogLine) has the details. - [`log_index`](/reference/socket/#message-log_index) lists every source the node keeps. A source missing from it no longer exists on the node. - A [`console_command`](/reference/socket/#message-console_command) is answered once the server's plugin has handed the command to the engine. That answer says nothing about whether the command worked. The command's output arrives as log lines of that server, so watch the server's log to show it. ### Browsers - Push every change to an allocation or a node to every connected browser over a WebSocket. Push it from your stored state. The reply to a launch or stop request says only that the request was accepted. The outcome comes later, from the node, a deadline, or another tab or player. - Send the whole state when a browser connects, since it missed every change in between. ### Events - A batch arrives again when its receipt is lost, and can arrive after its server has ended. [Handling a batch](/protocol/events.md#handling-a-batch) keeps statistics consistent through both. ## Replace a slot's token When Steam refuses a slot's login token, the slot receives no launches until you replace the token. Replace it once the slot is idle, because revoking the token of a running server drops that server. 1. Send [`reserve_slot`](/reference/socket/#message-reserve_slot). From the reply on, the agent starts no server in the slot, so none starts on a token Steam is about to drop. A slot running a server cannot be reserved. 2. Revoke the old token at Steam and obtain a new one. 3. Send the new token with [`set_config`](/reference/socket/#message-set_config), which ends the reservation. If no new token follows, the reservation runs out. The agent logs the new token on immediately and reports whether Steam accepted it. ## One instance or several Run your controller as one instance. Add instances only when one process can no longer serve your nodes and viewers, because several instances turn simple in-memory state into distributed state. Each node keeps one socket open to the controller. Much of what the protocol asks of you is state about that socket: - the requests waiting for a reply - the slots and servers that the node last reported - which socket owns the node. In one process, that state is plain memory. Apply each node's updates one at a time, in the order its messages arrive, so two updates to a node never interleave. A single reader per socket or a lock per node does that in any language. A restart costs little, because every agent redials and its [`hello`](/reference/socket/#message-hello) rebuilds what the instance knew. Several instances behind a load balancer each hold some of the sockets. A request for a node has to reach the instance that holds the node's socket. An example is a launch from a user whose browser reached another instance. That routing needs a message bus to coordinate the instances, such as NATS with JetStream, Redis or Valkey streams, Kafka, RabbitMQ or Postgres `LISTEN`/`NOTIFY`. Each piece of per-node state also needs one owner: - **Socket ownership.** A node that reconnects can land on another instance while its old socket is still open on the first. Record which instance holds each node, with an expiry. Close a second socket with `node_connected`, as the [protocol](/protocol.md#1-accept-the-connection) asks. - **Requests and replies.** A reply arrives on the instance that holds the socket, matched by `reply_to`. A caller on another instance waits for it over the bus. When the holding instance dies, the caller receives no reply. Treat that as a launch or stop [that may have run](#launches). - **Decisions about servers.** Choosing a node for a match and recording the launch has to happen once. Two instances that act on the same match can launch it twice, so route each match's decisions through one queue or lock. - **Event uploads.** Batches are plain HTTP and can reach any instance. Lock the per-server cursor from [Handling a batch](/protocol/events.md#handling-a-batch) inside its transaction, for example with `SELECT … FOR UPDATE`. Two batches of one server then cannot both advance it. - **Live views.** Slot statuses, resources and logs arrive on the instance that holds the socket. Fan them out to viewers on the other instances over the bus. **Pros** - **Rolling updates.** A deploy replaces one instance at a time. The controller stays up, and only that instance's nodes redial. - **Partial outages.** A crashed instance drops only its share of the sockets. - **More capacity.** The controller grows past what one process can hold. **Cons** - **A bus to run.** The controller depends on one more system, and on its delivery guarantees. - **Distributed state.** Every rule above has to hold across instances. A missed lock or a lost bus message shows up as a launch that ran twice or a reply that never came. Such failures are hard to reproduce. --- # Setup endpoint > The endpoint that lets node-setup.sh fetch a node's agent.yml with its token. **Optional.** When an operator points Ankor's [setup script](/setup-script.md) at your controller with `--controller`, the script fetches the node's `agent.yml` from this endpoint with the node token. It fetches the file from you because you own the node's configuration: its token, the address it dials, and the agent and CS2 images it runs. Without the endpoint, operators [write `agent.yml` themselves](/setup-script.md#write-agentyml-yourself). ## `POST /api/node/setup` The script sends `Authorization: Bearer ` and the `Ankor-Protocol` header, with no body. It posts to `/api/node/setup` by default. To serve the endpoint at another path, have operators pass that path with `--setup-path`. - Answer with any `2xx` status and the node's whole `agent.yml` as `text/yaml`. The script saves it unchanged. [Configuration](/configuration.md#agentyml) lists its keys. [`slots`](/configuration.md#config-slots) may be empty. - The token stays valid, so answer it again on every request. - Answer an unknown token with `401`. The script tells the operator to copy it again. - Answer `409` with `{ "error": "node_connected" }` while the node is connected. The script says the node is connected from another host. - Answer `409` with any other JSON body when you cannot read the node's stored configuration. The script tells the operator to ask the controller's operator. - Answer `426` for a missing or incompatible [`Ankor-Protocol`](/protocol/versioning.md). - Answer `429` when an address sends too many requests. The script asks the operator to wait. - Any other status stops the script with that status. --- # Event uploads > How gameplay events reach the controller, and how to handle each batch. Statistics are built from gameplay events: kills, damage, grenades, rounds and the final score. The plugin reports them to the agent, which uploads them to your controller in batches. You need this endpoint when you launch matches that send statistics ([`match.send_statistics`](/reference/types/#schema-MatchConfig)). A match's phase and score go over the socket instead, as [Get the result](/protocol.md#7-get-the-result) describes. A match without statistics sends its backups there too, as [`backup`](/reference/socket/#message-backup). The [protocol reference](/reference/uploads/) has the batch body, the answers and every event type. ## Event identity Every event has an identity of two parts: - **`allocation_id`**: the server that sent it, named by the allocation you gave it in [`start_server`](/reference/socket/#message-start_server). The allocation names one server, because a node refuses to launch an allocation it has launched before. A batch carries the events of one server and names it once for all of them. - **`seq`**: the event's position among that server's events, counting from 0 without gaps. > Diagram: A server numbers its events from 0, and batches carry them to the controller. After a lost receipt, the next batch starts again at the first event that no receipt covered. 1. Batch 1 arrives. The controller stores seq 0–3 and answers a receipt for 0–3. The agent deletes them. 2. Batch 2 arrives. The controller stores seq 4–7, but the receipt is lost on the way back. The agent keeps them, and seq 8–12 arrive on the node meanwhile. 3. After its backoff, the agent sends seq 4–12 in one batch. The controller already has seq 4–7 of that server, so it stores 8–12 and answers a receipt for 4–12. 4. Uploads continue from seq 13. Each attempt is rebuilt from the oldest event that no receipt covered. It carries as many of the server's events as fit, up to 128 events and 2 MiB. A retry therefore starts at the same `seq` as the attempt that failed, and it can carry more events. The events it repeats are the same as those you stored, because a server's events never change once the node has them. Events survive an agent restart or a lost connection. They stay on the node until a receipt covers them. The node keeps a server's events for 7 days after its last event or receipt, so that it recognises a replay of delivered events. Events that no controller took in that time are dropped, and the agent logs how many. The agent backs off every server when no answer comes, or when your controller answers that it is unavailable or refuses the node. When you refuse one server's batch, the agent backs off only that server. The node's other servers keep uploading. ## `POST ` The agent posts to [`controller.events_path`](/configuration.md#config-controller.events_path), `/api/node/events` by default. Each request carries the agent's [headers](/protocol.md#what-you-need) and a JSON body for one server. The body holds the server's `allocation_id` and 1–128 frames of its events, with consecutive `seq`. The frames come to at most 2 MiB together, so accept request bodies of that size plus the wrapper. Each `event` carries the fields of its gameplay event, cut short here, and the last batch of a map ends with [`map_end`](/reference/uploads/#message-map_end): **[`event_batch`](/reference/uploads/#message-event_batch)** (agent → controller) ```http POST /api/node/events HTTP/1.1 Authorization: Bearer 3f1c… Ankor-Protocol: Content-Type: application/json { "allocation_id": "5b1e2c7a-8d4f-…", "frames": [ { "seq": 397, "event": { "type": "player_death", … } }, … { "seq": 412, "event": { "type": "map_end", "winner_team": 1, "team1_score": 13, "team2_score": 9, … } } ] } ``` **[`event_batch_receipt`](/reference/uploads/#message-event_batch_receipt)** (controller → agent) ```http HTTP/1.1 200 OK Content-Type: application/json { "rejected": [] } ``` ## Answers Your controller answers each batch in the HTTP response. Uploads stay off the WebSocket, apart from [`event_backlog`](/reference/socket/#message-event_backlog). That message carries the node's whole backlog of stuck events, and the agent sends it when the backlog changes. Replace your stored backlog with it. [`hello`](/reference/socket/#message-hello) carries the same backlog in its `event_backlog`. > Diagram: The agent posts a batch; the controller answers a 2xx with a receipt and the agent deletes the batch, or anything else and the agent backs off and retries. | Status | Body | The agent | |---------------------------------------------------|-----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `2xx` | `EventBatchReceipt` | Deletes the whole batch. It never resends frames listed in `rejected`, because they would fail the same way. A body that is not a valid receipt counts as a refusal of that server's batch. | | `401`, `403`, `408`, `429`, `502`, `503` or `504` | ignored | Keeps the batch, pauses every server's uploads, backs off and retries. No answer within 15 seconds counts the same. | | any other status | ignored, except a `409`'s `detail`, which is logged | Keeps the batch and backs off only that server. The other servers keep uploading. | ### Receipt A receipt covers the whole batch. Normally you keep every frame, and `rejected` is empty. If you cannot use a frame, for example because its event names another match, quarantine it and list it in `rejected`. Quarantining sets a frame aside with a reason instead of counting it. For a batch where one event was set aside: ```json { "rejected": [{ "seq": 140, "reason": "unknown weapon" }] } ``` ### Refusing a batch A batch that cannot be trusted points to a bug or to damage in transit, because the agent checks every batch before sending it. To make the cause visible, refuse the batch with `409` and this body. Its `detail` shows up in the agent's log and in [`event_backlog`](/reference/socket/#message-event_backlog): ```json { "code": "event_integrity_error", "detail": "frames 3 and 4 are not consecutive" } ``` ## Handling a batch You decide what your controller does with the frames. If you keep statistics, the steps below keep them consistent however often a batch is retried. You can store, log or drop quarantined frames. For each request: 1. **Authenticate.** Answer `401` for an unknown bearer token. Answer `426` when `Ankor-Protocol` is missing or names an [incompatible version](/protocol/versioning.md). 2. **Check the batch.** [Refuse it](#refusing-a-batch) if the body does not parse with `EventBatch` from [`@ankorcs/protocol`](https://www.npmjs.com/package/@ankorcs/protocol). It fails to parse when it does not name a server or its frames are not consecutive. 3. **Find the server and its match.** The batch's `allocation_id` must name a server that your controller launched on this node. That launch's match is the batch's match. Parse each frame with [`EventBatchFrame`](/reference/types/#schema-EventBatchFrame). If the server or the match is not yours, quarantine every frame and still answer with a receipt, so that the node deletes them. A retry cannot fix that batch. 4. **Walk the frames.** Keep a cursor per server: the highest `seq` handled so far. - A `seq` at or below the cursor is a duplicate. Check that it matches the stored event, and store nothing new. - A `seq` above the cursor is new. Store the event if it parses, belongs to the batch's match and names only players you launched the match with. Otherwise quarantine it with a reason. Either way the cursor moves past it, so one bad event never delays the rest. - An event whose `type` your protocol version does not have comes from a newer plugin. Move the cursor past it, and store or quarantine nothing. [Versioning](/protocol/versioning.md) explains why. - [`player_connect`](/reference/uploads/#message-player_connect) with `rejected` set is the one event that names an outsider by design. It means the plugin kicked that player. - A jump past cursor + 1 is a gap. Log it and carry on. The agent skips numbers only when it lost them, as when an operator deleted its event store. 5. **Commit, then answer.** Store the events, the quarantined frames and the cursor in one transaction. Then answer with a `2xx` status and a receipt for the whole batch. If you cannot, for example because storage is down, answer with a `5xx`. The agent backs off and tries again. ## Reading the events - Events can arrive after their round or after the server has ended. Place them by their `tick`, and their `round` where they carry one, not by arrival. - [`round_complete`](/reference/uploads/#message-round_complete) follows every event of its round. - A grenade is spread over several events. [`grenade_throw`](/reference/uploads/#message-grenade_throw) gives it a `grenade_id` that is unique within the server. The events it causes later carry that id, up to its [`grenade_end`](/reference/uploads/#message-grenade_end). - A `null` measurement means the plugin could not read it. Keep it missing, not zero. - After a [`match_restored`](/reference/uploads/#message-match_restored), the events from the next [`round_start`](/reference/uploads/#message-round_start) on replace what the server sent for that round and the rounds after it. --- # Demo uploads > How nodes post finished demos to the configured target. When a match records a demo, the node can upload it to any HTTP endpoint that accepts a `POST`. The endpoint can be your controller, a storage proxy or anything else. The target is set per node, as [`demo_upload`](/configuration.md#config-demo_upload) in `agent.yml` or `config.demo_upload` in [`set_config`](/reference/socket/#message-set_config). Changing the target does not interrupt an upload in progress. Uploads are optional. Without a target, demos stay on the node until [`delete_demo`](/reference/socket/#message-delete_demo) removes them. Recording starts `tv_delay` seconds after the match goes live. GOTV shows play that delay late, so the demo opens on the first live round. With `gotv` null, the delay is 0 and recording starts as the match goes live. ## Request `POST ` with the raw `.dem` file as the body, `Content-Type: application/octet-stream` and `Content-Length`. | Header | Value | |-----------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `Ankor-Protocol` | The [protocol version](/protocol.md#versions). | | `Ankor-File-Name` | The file name, `__.dem`. The time is the recording's start in UTC. The match id is empty for a match launched without one. | | `Ankor-Match-Id` | The match id. Absent for a match launched without one. | | `Ankor-Allocation-Id` | The `allocation_id` of the server that recorded it. | | configured header | The target's `header`, if set. It can have any name that your target expects, such as `Authorization: Basic …`. It is the only credential, and the URL carries none. | The configured header cannot take a name that the agent sets itself or that frames the request. `RESERVED_DEMO_HEADERS` in `@ankorcs/protocol` lists those names. The agent's own headers are the ones sent. The node token is never sent to a demo target. The node reports the header's value only as `value_sha256`. When you send a header back in [`set_config`](/reference/socket/#message-set_config) without `value`, the node keeps the value it has. Send `value` only to change it. ## Answers | Status | The node | |---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `2xx` | Deletes the file. A file it cannot delete stays, with the error in the node log, and is not uploaded again until the target changes or the agent restarts. | | `401`, `403` | Keeps every waiting file and stops uploading until a new demo is recorded, the target changes or the agent restarts. | | `422` | Keeps the file and skips it until the target changes or the agent restarts. Continues with the next file. | | anything else | Keeps every waiting demo and retries oldest first after 30 seconds, doubling to 15 minutes. After a day of failures, keeps them until another demo is recorded, the target changes or the agent restarts. Treats a network failure, or an upload that sends nothing for 2 minutes, the same way. | The node ignores the response body. A demo that waits after a failed upload carries the reason in its `upload_error`. The target must accept bodies of a few hundred megabytes, the typical size of a full match's demo. If a storage service needs signed requests, such as S3 with presigned URLs, put a small proxy in front of it that adds the signatures. --- # Limits > The limits ankor.sh holds nodes to, and how a controller limits a node. [ankor.sh](https://ankor.sh) bounds what each node sends, because a node may be run by someone else. Your controller can apply the same limits or its own. It enforces either with the answers in [Limiting a node](#limiting-a-node). ## ankor.sh's limits The numbers are constants in the contract, most of them in [limits.ts](https://github.com/ankorcs/ankor/blob/master/packages/protocol/src/limits.ts). An agent within them is never limited. Ankor's agent stays within them even with 64 busy slots. - A node has at most 64 slots. Its [`hello`](/reference/socket/#message-hello) lists at most the 50 newest demos. Each demo's file name has at most 160 characters. - A socket message is at most 960 KiB. A socket carries one [`hello`](/reference/socket/#message-hello), which is due within 10 seconds of the upgrade. - Each kind of message draws on a budget per connection. The budget allows a burst and refills at a steady rate. Where the table says "a slot", add that amount for every slot the node has. The slot count comes from the node's last [`hello`](/reference/socket/#message-hello) or applied [`set_config`](/reference/socket/#message-set_config) reply. Before the first of these, count one slot. | Messages | Burst | Refill | Over budget | |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------|-------------------------------------------------------|-------------| | Every message, in bytes | 32 MiB + 1 MiB a slot | 512 KiB/s + 16 KiB/s a slot | close | | [`hello`](/reference/socket/#message-hello), [`slot_status`](/reference/socket/#message-slot_status), [`node_status`](/reference/socket/#message-node_status), [`server_exited`](/reference/socket/#message-server_exited), [`demo`](/reference/socket/#message-demo), [`demo_removed`](/reference/socket/#message-demo_removed), [`event_backlog`](/reference/socket/#message-event_backlog), [`shutdown`](/reference/socket/#message-shutdown), unreadable | 64 + 32 a slot | 2/s + 1/s a slot | close | | [`log_batch`](/reference/socket/#message-log_batch), [`log_index`](/reference/socket/#message-log_index) | 128, and 16 MiB | 32/s, and 256 KiB/s | dropped | | [`resource_batch`](/reference/socket/#message-resource_batch) | 10 | 1/s | dropped | | [`cs2_update_progress`](/reference/socket/#message-cs2_update_progress) | 50 | 10/s | dropped | | [`demo_progress`](/reference/socket/#message-demo_progress) | 10 | 2/s | dropped | | [`phase_change`](/reference/socket/#message-phase_change), [`score`](/reference/socket/#message-score), [`backup`](/reference/socket/#message-backup) | 0 + 16 a slot | 1 every Infinity s + 1/s a slot | dropped | | [`reply`](/reference/socket/#message-reply) to no open request | 32 | 1/s | close | | WebSocket pings | 5 | 1/s | close | - A log replay that would exceed the byte budget arrives without its oldest lines, so frequent replays do not close the socket. - [ankor.sh](https://ankor.sh) closes a socket that breaks these rules with the code and reason that [Closing the socket](/reference/socket/#closing) lists. After a close for a larger message or a budget, ankor.sh answers the node's upgrades with `429` for a minute. - More than 10 upgrades of one node within a minute are answered `429` until that window ends. - More than 20 missing or unknown tokens from one address within a minute are answered `429` until that window ends. An IPv6 address counts as its whole /64. - [Event uploads](/protocol/events.md) may carry 32 MiB + 4 MiB a slot in a minute. Each request counts 1 KiB more than its body. Past that budget, [ankor.sh](https://ankor.sh) answers `429`. - The agent handles at most one [`check_cs2_build`](/reference/socket/#message-check_cs2_build) in a minute and drops any that comes sooner. The agent's scheduled read of the public CS2 build, every 30 minutes, is not limited. - [ankor.sh](https://ankor.sh)'s [demo receiver](/protocol/demos.md) accepts one upload per node at a time. It answers `429` to a second upload. It answers `403` to an upload that would push the node's stored demos past the service's quota. ## Limiting a node Enforce limits with the answers below. The agent needs nothing else from you. It backs off, retries and sends a fresh [`hello`](/reference/socket/#message-hello) by itself. It keeps every event it stored. - **Socket over a budget:** [close it](/reference/socket/#closing) with code `1008` and `rate_limited`. Refuse the node's upgrades with `429` for as long as you want to keep the node out. The agent redials with its usual [backoff](/protocol.md#how-the-agent-reconnects). Its next [`hello`](/reference/socket/#message-hello) carries its state. - **A message you would rather drop:** ignore it and keep the socket open. Only the messages that the table marks `dropped` are safe to ignore. The others carry state that the agent sends again only after it reconnects. - **Event upload:** answer `429`. The agent keeps the batch and retries with backoff. - **Demo upload:** answer `429`. The agent keeps the file and retries. [Answers](/protocol/demos.md#answers) has the details. - The agent ignores `Retry-After` and the response body of a `429`.