The shipper
For: players who run their own server, and anyone debugging the shipper. At the end: you know when it runs, what it keeps on your computer, its files and flags.
shipper/wow_stats.py runs on the computer you play on. One file, Python 3.11+
standard library only, run by uv (uv run --script) or a plain python3
when uv isn’t there.
When it runs
Section titled “When it runs”- macOS: a launchd agent (
local.wow-stats) every 5 minutes, and each time the game saves its SavedVariables (logout,/reload). - Windows: a scheduled task (
wow-stats) at logon and every 5 minutes, without a console window.
Each run waits until no file in WTF changed for 5 seconds (the game writes
its files one after another; it gives up after 2 minutes and tries next
run), then:
- archive: copies
_forever_/WTF/into a git repo (~/WoWArchive/wtf) and commits. Polestar Recorder deletes the chunks its uploader acked at the next login, so the git history is the only place every version survives. Skipped when git isn’t installed orwtf_archive = false(below). - extract: turns what is new into NDJSON in
~/WoWArchive/spool/: the SavedVariables, the combat logs, and Polestar Client’ssent.jsonlwhen the client is installed (below).state.jsonremembers what was already sent, so nothing goes twice. - ship: POSTs each spool file to
[ship] url, in posts of at most 4 MB, then moves it to~/WoWArchive/sent/gzipped. Server down or no network: it logs that and the spool waits for the next run. The spool file shrinks after each post, so a run cut short resends nothing. - update: at most every 6 hours, asks the same server for a newer shipper and takes it (Auto-update).
Auto-update
Section titled “Auto-update”The shipper updates itself. After a run’s own work, at most once every 6
hours, it asks the server it ships to for the current version:
GET <server>/install/update/shipper, with the same upload password as
/ingest. The answer is the version of the wow_stats.py that server
carries (its VERSION line) and the sha256 of each file:
{"version": "0.5.1", "files": {"wow_stats.py": {"sha256": "9f2c...", "size": 121344}, "install-mac.sh": {"sha256": "51ab...", "size": 4310}}}When that version is higher than its own, it downloads
/install/update/shipper/wow_stats.py from the same server, checks the
sha256 and that the file compiles and names the version announced, writes
it next to itself and renames it into place in one step. The old copy
stays as wow_stats.py.prev. The new version runs from the next run on.
When the server’s installer for your system (install-mac.sh and
wow-stats.plist, or install-windows.ps1) differs from the copy the
installer left next to the script, the shipper also runs the new installer
once, in the background, to rewrite the launchd agent or the scheduled task.
What it never does:
- Downgrade. An equal or lower version is left alone.
- Talk to anyone else. Only the server in
[ship] url, over https (plain http only to this computer,127.0.0.1orlocalhost), and no redirect. - Break the shipping. No endpoint (a server older than 0.5.1), no network,
a wrong hash, a file that doesn’t compile: it keeps the running version and
logs one line (
auto-update: ...; keeping 0.5.0). - Touch your config or
state.json. They stay where they are; a version that changes their format reads the old one. - Rewrite a copy you run by hand. Only the installed script
(
~/.local/share/wow-stats/wow_stats.py,%LOCALAPPDATA%\wow-stats\wow_stats.py) updates itself, not a git checkout.
To pin the version you have, put this at the top of the config, next to
wow_dir:
auto_update = falseA shipper from before 0.5.1 can’t update itself: run the update line once
(below). /start says so while the server still sees one: its setup report
carries the shipper’s version from 0.5.1 on.
Stats only (no WTF archive)
Section titled “Stats only (no WTF archive)”By default, when git is installed, every run commits your whole WTF
folder to ~/WoWArchive/wtf: every version of every SavedVariables file,
other addons’ included. That history is what --backfill replays after an
update that reads something new, and the only copy of the Recorder chunks
an uploader already deleted. It grows with WTF: 8 MiB of unrelated
files in WTF once made about 16 MiB of archive.
To keep the stats only, put this at the top of the config, next to
wow_dir:
wtf_archive = falseThe shipper then reads the game files where they are and commits nothing.
What stays either way, all under archive_dir:
| What | Size | |
|---|---|---|
spool/ |
events not sent yet | empty once the server has them |
sent/ |
every event sent, gzipped: your own copy, for a new server | grows with your play |
state.json |
what was already sent, so nothing goes twice | small |
wtf/_polestar-client/sent.jsonl |
the Polestar Client chunks kept beyond its 8 days (only with Polestar Client) | grows with your play |
Without the archive, --backfill has nothing to replay: events a past
run missed can only come back while the game files still hold them. An
archive already made stays where it is; delete ~/WoWArchive/wtf by hand
if you don’t want it (keep _polestar-client/ inside it if you use
Polestar Client).
The first run sends your whole history
Section titled “The first run sends your whole history”There’s no import step. The first run after installing finds nothing in
state.json, so it extracts everything already on disk: every character’s
SavedVariables, the Recorder chunks still there, the last 8 days of chunks
Polestar Client keeps, and the combat logs in Logs/. That can be 100,000
events or more. The log then has one line per feed and per character
(extract: ...: 23514 events; per feed {...}; per character {...}) and, for
a big spool file, a progress line after each post
(ship: ...: 40000/112000 events sent). Until [ship] url is set, all of
it waits in the spool.
Retention. VictoriaLogs drops events older than its retention when they
arrive, and still answers 200. The compose kit keeps 100 years. Sending to a
store with a shorter one (the default is 7 days)? Set retention_days in
[ship]: older events are kept in ~/WoWArchive/too-old/ instead of being
posted into a store that drops them, and the log says how many.
Polestar Client’s sent.jsonl
Section titled “Polestar Client’s sent.jsonl”Polestar Recorder deletes its chunks from the SavedVariables once Polestar
Client has uploaded them. The client keeps a copy of every chunk its server
accepted in the last 8 days in sent.jsonl, so the shipper reads that file
on every run, as a second source of the same chunks: it fills in what the
WTF archive missed (the play before you installed the shipper, a run that
didn’t happen). A chunk gives the same events whichever file it came from
(same source_id, same durations and XP), and one already sent is never
sent again.
Each run also appends the chunks it hasn’t kept yet to its own copy,
~/WoWArchive/wtf/_polestar-client/sent.jsonl, which the archive step commits
and never prunes: as long as the shipper runs once every 8 days, no chunk is lost.
Folder (Tauri’s app data folder for gg.polestar.client) |
|
|---|---|
| macOS | ~/Library/Application Support/gg.polestar.client/ |
| Windows | %APPDATA%\gg.polestar.client\ |
| Linux | ~/.local/share/gg.polestar.client/ |
polestar_client_dir in the config points it elsewhere. The shipper only
reads sent.jsonl there, never the client’s tokens or settings, and never
writes to that folder. No Polestar Client, no file: nothing to read, not an
error.
Lines that aren’t valid events never leave the PC: they go to
~/WoWArchive/rejected/ and the log says how many.
| What | macOS | Windows |
|---|---|---|
| Script | ~/.local/share/wow-stats/wow_stats.py |
%LOCALAPPDATA%\wow-stats\wow_stats.py |
| Config | ~/.config/wow-stats/config.toml |
%APPDATA%\wow-stats\config.toml |
| Log | ~/Library/Logs/wow-stats.log |
%LOCALAPPDATA%\wow-stats\wow-stats.log |
| Archive | ~/WoWArchive |
%USERPROFILE%\WoWArchive |
| Polestar Client (read only) | ~/Library/Application Support/gg.polestar.client/sent.jsonl |
%APPDATA%\gg.polestar.client\sent.jsonl |
WOW_STATS_CONFIG points it at another config file. config.example.toml
lists every setting: wow_dir (the _forever_ folder, found by itself in the
usual install places), archive_dir, wtf_archive (default true),
log_file, polestar_client_dir, auto_update (default true), and
[ship] with url, user, password or password_file, and
retention_days.
--dry-run: extract to stdout, write nothing, send nothing. The quickest way to see what it would send.--no-ship: archive and extract, keep the spool.--backfill: re-extract every archived version of WTF. Not needed after installing (the first run sends what is there); use it after an update that reads something new (quest names, Auctionator). Duplicates are harmless: every event has asource_id.--preflight: the installers’ last step, a checklist with one line per check (the WoW folder, Polestar, Polestar Recorder and its switch, Questie or Altoholic, each other optional addon, saved game data, the server with your password, the background program): a check mark, or a cross with the one thing that fixes it. Exits 2 only when the install can’t work as it is (no WoW folder, the password refused, no background program); a missing addon is a cross, never a failure. The addon list isdocs/videos/steps.json’s, copied intoADDONS(a test keeps them equal).
Addons it reads
Section titled “Addons it reads”Polestar and Polestar Recorder for nearly everything, Auctionator for prices,
Syndicator (equipped items), CharacterExport-Forever (profession caps),
Altoholic (its DataStore_Characters: /played since level 1; its
DataStore_Quests: the quests done before Polestar), Questie (the same
quests, exact),
GearQuestForever and ForeverLevelingBiS (tracked upgrades, spec) for the
gear panel, and LibForeverData, Polestar_QuestDB or Polestar_ItemDB only to
look up quest and item names and item levels.
Missing addons just mean fewer events.
Something looks wrong
Section titled “Something looks wrong”- Nothing on the page. Read the log: the last lines say what each step
did (
archive: committed 165 changed files,extract: ...,ship: ... sent (5299 events)).unreachablemeans the url or the server;401means the password doesn’t matchINGEST_PASSWORD. - Old days missing. The server’s VictoriaLogs drops events older than
its retention when they arrive. The compose kit keeps 100 years; a
VictoriaLogs you run yourself defaults to 7 days: set
retention_daysand look in~/WoWArchive/too-old/. Recorder chunks from before the shipper was installed exist only while Polestar Client keeps them, 8 days. - A new character doesn’t show up. It needs one logout first, so the game writes its SavedVariables. On macOS rerun the installer after that, so launchd watches the new folder too.
- Quest names missing. Install LibForeverData (or Polestar_QuestDB), then
run with
--backfill.
Update and uninstall
Section titled “Update and uninstall”Update: it updates itself (above). By hand, or once for a shipper from
before 0.5.1: git pull and rerun the installer, or the update line from
your start page. Config, state and archive stay.
Backup, uninstall and deleting your data are three separate steps:
Back up and restore and Uninstall. In
short: uninstalling (macOS launchctl bootout gui/$(id -u)/local.wow-stats
and the plist; Windows schtasks /Delete /TN wow-stats /F, not
Unregister-ScheduledTask: under a Restricted execution policy PowerShell 7
can’t load its module) leaves ~/WoWArchive in place until you delete it.