Administrator guide

For the people who install, run, change or move the SPLA API.

What it is made of

 internet  --HTTPS 443, cloudiq-spla.cleverpoc.com--+
                                                   +--> nginx --unix socket--> spla-api (waitress) --HTTPS--> api.crayon.com
 internal  --HTTPS 8443/8444, IP allow list--------+          \--unix socket--> spla-api-demo --> fake CloudIQ (127.0.0.1:8092)
                                                     (/demo/v1/... and port 8444 go to the demo)

The service only talks to the official SoftwareOne Customer API (https://api.crayon.com/api/v1/, Swagger at https://api.crayon.com/swagger/v1/swagger.json). It does not use the CloudIQ web portal, its GraphQL interface or any screen scraping. The endpoints it uses:

Lists: CloudIQ answers {"Items": [...], "TotalHits": n} for all lists used here, and a page past the end has an empty Items (checked live on 27-10-2026 for organizations, agreements and product containers). The service pages with 100 per page, stops at the total, refuses a repeated page and stops with an error after 50 pages or 90 seconds.

A report is a CloudIQ product container with Category 2 (usage report) and Program.Id 24 (SPLA). Type 1 is a draft, Type 4 is submitted ("ORDER" in the portal). The service exposes no other containers, refuses to change anything that is not Type 1, and never sends a body with another Type. It has no submit function.

Current installation

Production starts read-only (SPLA_API_ALLOW_WRITES=false). The demo has writes on. Production and demo run as different users with their own log folders, so a fault in the demo cannot touch the production audit log.

Settings

All settings are environment variables in the .env file of the instance. Defaults are safe.

After a change: systemctl restart spla-api (this ends all open sessions; users log in again). install.sh keeps the production settings file (it only adds settings that are new); the demo settings file is always replaced.

The real client address (for rate limits and the audit log) comes from nginx: it sets X-Forwarded-For to the connecting address, and waitress trusts that header only from its unix socket (--trusted-proxy=localhost). The socket is only reachable for root, the service and nginx, so no other process on the server can pretend to be another address or go around the allow list. Without the trusted-proxy options waitress removes the header and every request would look like it came from the same place.

Install and update

From a checkout on an admin machine that can reach the server over SSH:

DEPLOY_SUDO_PASS=... bash deploy/deploy.sh      # host from DEPLOY_HOST, default cit-start

deploy.sh runs the tests, checks that everything that goes to the server is ASCII, packs the code and runs deploy/install.sh as root on the server. install.sh is idempotent:

1. takes a lock (one installation at a time), installs nginx python3-flask python3-waitress python3-markdown if missing and stops if a version is older than tested (Flask 3.0, Werkzeug 3.0, waitress 3.0, Markdown 3.10; tested with Flask 3.1, waitress 3.0, Markdown 3.10 and 3.11, Python 3.13 and 3.14);
2. creates users and groups, /etc/spla-api (settings, allow list, self-signed TLS) and the log folders, keeping existing files;
3. copies the code and its deploy files to a new folder under /opt/spla-api/releases/;
4. installs the units, logrotate and the nginx site of that release; nginx is tested first and the old site file is put back if the test fails, so other sites on the server are never affected;
5. points /opt/spla-api/demo to the new release and restarts the demo and the fake CloudIQ, then runs a smoke test through nginx (certificate checked): log in, create a draft, change, add and remove a line, CSV, log out;
6. only then points /opt/spla-api/current to it, restarts production and checks it through nginx (/healthz, /v1, an empty login gives 400). Production never runs a release that failed as the demo;
7. if a check fails, the previous release comes back, with its own units and nginx site, and the script exits with an error;
8. makes ufw allow exactly the addresses in access.conf on 8443/8444 (adds new ones, removes old ones), and 443 from anywhere for the public name.

The public site gets its own smoke test in step 6: production, /demo, the pages, and that another name on port 443 (the start pages) does not reach this service. Until the Let's Encrypt certificate exists, install.sh puts a 30-day self-signed placeholder in public.crt so that nginx can start.

Note: production and demo share the nginx site files and are installed by the same step 4, after nginx -t. The running production process keeps the old code until step 6, but a change in the nginx site itself is live for both at step 4. Since 1.1.0 the socket paths are fixed, so this matters only when the nginx site really changes.

Old releases stay in /opt/spla-api/releases/ (they are small); remove them by hand when you want.

Roll back by hand: ls -t /opt/spla-api/releases/, copy the units, logrotate-spla-api and nginx-spla-api.conf from <older>/deploy/ to their places (see install.sh, function apply_config), run nginx -t && systemctl daemon-reload, then ln -sfn /opt/spla-api/releases/<older> /opt/spla-api/current, the same for /opt/spla-api/demo, and systemctl restart spla-api-mock spla-api-demo spla-api && systemctl reload nginx. Easier: check out the older git version and deploy it.

Remove completely: systemctl disable --now spla-api spla-api-demo spla-api-mock, remove /etc/nginx/sites-enabled/spla-api and spla-api-public and reload nginx, delete the ufw rules with comment spla-api and spla-api-public (the start pages have their own 443 rules), and remove the DNS name. Keep /var/log/spla-api (audit trail) and move /opt/spla-api to an archive folder.

Certificate

Public name. cloudiq-spla.cleverpoc.com has a Let's Encrypt certificate. The key public.key is made on the server and never leaves it. The admin machine renews the certificate every day when it has less than 30 days left (job spla-api-cert): the server makes a signing request with its own key, the admin machine proves ownership of the name in DNS (DNS-01) and copies the certificate back, checks that it belongs to the key and reloads nginx.

Internal ports. 8443/8444 use a self-signed certificate (name spla-api.internal and the server IP). Clients must trust exactly this certificate, never switch checking off (curl -k, verify=False): with checking off, anyone on the network path could read the CloudIQ passwords and secrets sent at login.

# on the server: fingerprint to compare and the file to hand to an internal client
openssl x509 -in /etc/spla-api/tls.crt -noout -fingerprint -sha256
# on the client
curl --cacert spla-api.crt --resolve spla-api.internal:8443:<server address> https://spla-api.internal:8443/healthz

Switching changes on (production)

Changes in CloudIQ are real: the drafts are the provider's monthly report to Microsoft. Before switching on:

1. Agree with the organization(s) concerned who may change drafts through the API.
2. Test the exact calls on the demo, then on production with ?dry_run=true.
3. Set SPLA_API_ALLOW_WRITES=true in /etc/spla-api/spla-api.env and restart.
4. Watch the audit log (event write) for the first month.

Submitting stays a manual step in the CloudIQ portal; there is no setting for it.

Who can reach it

The public name https://cloudiq-spla.cleverpoc.com is reachable from anywhere: the router forwards port 443 of the public address to this server, where nginx sends that name to this service and every other name to the start pages (with their own allow list). Protection on the public side comes from the limits (see the security notes), not from an address list. To take it off the internet: disable the site spla-api-public and reload nginx.

/etc/spla-api/access.conf is an nginx allow list (allow <ip>; lines, ending with deny all;) for the internal ports 8443 and 8444; install.sh makes ufw allow exactly the same addresses there. To add or remove an address: change the file and run the deploy again (or nginx -s reload plus the ufw rule by hand).

Audit log

One JSON object per line, UTC times. Events:

All events have ts, request_id (always made by the service; a client's own X-Request-Id is logged as client_request_id), ip and, when logged in, session (first 12 characters of the SHA-256 of the session token, not the token). The log never contains passwords, client secrets, access tokens, session tokens or request bodies; field names that look like secrets are replaced by [removed] before writing.

Look at the last changes: grep '"event": "write' /var/log/spla-api/audit.log | tail. The same events are in the journal: journalctl -u spla-api | grep AUDIT. For long-term tamper-proof storage, forward the journal to the central log server.

If the audit log cannot be written (disk full, wrong rights), requests fail with an error instead of running without a trace.

Monitoring

• GET /healthz (no login): {"status": "ok", "version": ...}. It does not call CloudIQ, so it stays green when CloudIQ is down. On the server: curl --unix-socket /run/spla-api/app.sock http://localhost/healthz (as root).
• journalctl -u spla-api shows start-up problems and unexpected errors (with request_id).
• CloudIQ problems show as 503 upstream_unavailable in the audit log (status 503).
• Once a month: du -sh /opt/spla-api/releases (old releases are kept; remove old ones by hand when needed).

Testing

python3 -m venv .venv && .venv/bin/pip install -r requirements-dev.txt
.venv/bin/python -m pytest -q tests                  # all tests, against the fake CloudIQ
.venv/bin/python -m mock.mock_cloudiq --port 8092    # run the fake CloudIQ by hand

Read-only check against the real CloudIQ (needs a real API client and user in the environment; it blocks every non-GET call before it is sent):

SPLA_LIVE_CLIENT_ID=... SPLA_LIVE_CLIENT_SECRET=... SPLA_LIVE_USERNAME=... SPLA_LIVE_PASSWORD=... \
  .venv/bin/python tools/live_read_check.py

The writes (create, PATCH, PUT) have only been tested against the fake CloudIQ. The fake follows the SoftwareOne documentation and the shapes seen in real read calls. Before the first real change, run that change once on production with a person watching in the CloudIQ portal.

Moving or publishing

The service has no database and no state on disk except the audit log, so moving it is: install on the new host, copy /etc/spla-api, point the name to it.

• Another Linux host: deploy with DEPLOY_HOST=<host> and EXPECTED_HOST=<host name> for the check at the top of install.sh.
• Container: docker build -f deploy/Dockerfile -t spla-api . then docker run -p 127.0.0.1:8090:8090 spla-api, behind a TLS proxy that sets X-Forwarded-For (the image trusts that header from any sender, so never expose port 8090 directly).
• At SoftwareOne / inside CloudIQ: the API contract (openapi/openapi.yaml) does not depend on where it runs. SoftwareOne could run this code, or implement the same contract natively; partners then only change the address.
• Another public name: change server_name in deploy/nginx-spla-api-public.conf and PUBLIC_NAME in deploy/install.sh, point the name at the server, and renew the certificate for it.

Limits of the current design to keep in mind when scaling: sessions live in memory of one process. Restarting logs everyone out, and running more than one copy needs sticky sessions or a shared session store.