feat: add backend, admin console and live monitoring

Integrate tunasync management, SQLite authentication and content settings, real monitoring and per-repository pull speeds. Add Chinese-only announcements and mirror help, editable related links, admin layout updates, deployment tools and tests.
This commit is contained in:
2026-11-01 18:04:01 +08:00
parent addadd7d15
commit f8cef413d6
57 changed files with 4222 additions and 101 deletions
+100
View File
@@ -0,0 +1,100 @@
# MirrorOUC 部署
当前预览:`http://100.69.2.2:8080`,后台:`/admin`。这是单 Worker 适配版本,使用服务器现有的 tunasync Manager、Worker 和并行 rsync 配置,不重新安装或替换同步引擎。
## 运行结构
| 服务 | 位置与用途 |
| --- | --- |
| Nginx 预览 | 只监听 `100.69.2.2:8080`;原有 80 端口站点保持原配置 |
| mirrorouc.service | `mirrorouc` 普通用户,API 与静态文件监听 `127.0.0.1:8081` |
| mirrorouc-config.service | root 受限配置写入器,Unix socket 只允许 `mirrorouc` 组访问 |
| mirrorouc-metrics.timer | 每 10 秒采集 Linux CPU/内存、NFS `/data` I/O 和磁盘容量 |
| mirrorouc-counters.service | 添加独立 nftables 计数表,只计数,不修改已有规则、不丢弃数据包 |
| mirrorouc-speeds.service | 每约 2 秒读取 TCP 接收计数,按仓库汇总并行连接 |
| tunasync | 继续使用已有 Manager `127.0.0.1:12345` 与 Worker `srv-software-mirror` |
代码发布目录 `/opt/mirrorouc/releases/<release>`;`/opt/mirrorouc/current` 指向当前版本。业务数据库和上传图片保存在本地磁盘 `/var/lib/mirrorouc/`,不要放在 NFS 上。监控历史位于其 `metrics/history.json`,保留最近 24 小时;刚启动时没有完整 24 小时历史。
## 首次安装/更新
本地 Node.js 22.18+ 与 Go 1.27+:
```sh
npm ci
npm run check:i18n
npm run check:monitoring
npm run build
cd backend
go test ./...
go vet ./...
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -o ../mirrorouc .
```
把 `mirrorouc`、`dist/` 和 `deploy/` 放进新的 `/opt/mirrorouc/releases/<release>/`,确认二进制可执行。在 Debian 服务器上执行:
```sh
python3 /opt/mirrorouc/releases/<release>/deploy/test_config_helper.py
python3 /opt/mirrorouc/releases/<release>/deploy/test_job_speeds.py
sh /opt/mirrorouc/releases/<release>/deploy/install-metrics.sh
sh /opt/mirrorouc/releases/<release>/deploy/install-preview.sh /opt/mirrorouc/releases/<release>
systemctl status mirrorouc mirrorouc-config mirrorouc-metrics.timer
curl http://127.0.0.1:8081/healthz
nginx -t
```
安装脚本针对当前服务器路径。迁移到其他服务器时,应先调整 `nginx-preview.conf`、环境文件、配置写入器的固定路径与 Manager 地址。脚本不会清理旧发布目录或删除镜像文件。
首次安装会创建管理员 `admin`,随机密码写入 root 才能读取的 `/etc/mirrorouc/initial-admin-password`。本次部署的本地凭据文件为被 Git 忽略的 `.local/deployment/admin-credentials.json`。后台「账户」可修改当前密码,修改后所有旧会话失效。此密码文件不会跟随密码修改而更新。
环境配置文件 `/etc/mirrorouc/mirrorouc.env` 不随发布覆盖,权限 0600。正式启用 HTTPS 后将 `MIRROROUC_SECURE_COOKIES=true` 并重启服务;当前 Tailnet 预览通过 HTTP 访问,故为 false。登录 Cookie 为 HttpOnly、SameSite=Strict,有效期 12 小时;写入接口另需会话 CSRF Token 与同源请求。
## 配置发布和权限
Web 服务不能直接写 `/etc/tunasync`。配置写入器固定只接受 `/etc/tunasync/worker.conf` 和 `/etc/tunasync/mirrors.d/*.conf`:
- 全局只可修改 concurrent、interval、retry、timeout。
- 已有镜像只可修改 upstream、interval、retry、timeout。自定义 command、rsync_options、provider 等字段必须保留。
- 新增只支持 rsync/two-stage-rsync,文件名固定为 `mirrorouc-<name>.conf`;不接受任意 Shell 命令。
- 编辑前后均验证 TOML、数值、上游地址和文件内容哈希。产生冲突时拒绝覆盖。
- 先生成可阅读的完整文件前后预览,再由管理员确认应用;应用失败恢复原文件并尝试重新加载。
- SQLite 保留变更版本与审计记录;应用过的变更可以恢复至变更前,若文件已被其他修改覆盖则拒绝恢复。
- 写入器保留待完成事务日志 `/var/lib/mirrorouc-config/pending.json`;若进程在写入中退出,下次启动恢复原文件。若恢复失败则拒绝继续写入,需要运维检查该文件与 Worker 状态。
Manager 返回成功表示 reload 命令已提交,并不等于 Worker 已完成重载,需查看 Worker 服务和任务状态。已有仓库禁用后用「开始同步」重新启动;tunasync 不提供单独的 enable 命令。
## 监控口径
HTTPS/rsync 使用 TCP 443/873 的 IPv4、IPv6 收发字节计数,双向合计,排除回环接口。8080/80 端口的 HTTP 预览流量不计入 HTTPS,也不展示 HTTP 曲线。没有经过这些端口的流量不会被推算进对应协议。没有有效采样时显示等待采集。
`/status` 同步明细的「拉取速度」来自 Linux TCP_INFO 的 `bytes_received`,通过 `ss -tinpeH` 读取,关联 `tunasync-worker.service` 的进程与仓库目标路径,并追溯 git/ssh 等辅助进程的父进程。多个 rsync 子进程共享同一个 socket 时按 inode/socket cookie 去重,多个并行连接求和;约每 2 秒采样并刷新,单位 B/s、KB/s、MB/s、GB/s。只计算 TCP 接收,不把上传量、NFS I/O 或磁盘写入速度当作拉取速度。
这是采样窗口的接收速率,采样间建立并关闭的短连接可能漏记,不作为精确流量计费数据。首次采样不使用连接生存期累计值冒充瞬时速度。仅同步中的仓库显示数值;有效采样中的零表示当时没有收到数据,未同步/采样失败/数据过期时显示「—」。数据超过 15 秒即失效。相关原始字段见 [Linux TCP_INFO](https://github.com/torvalds/linux/blob/master/include/uapi/linux/tcp.h) 和 [iproute2 ss](https://github.com/iproute2/iproute2/blob/main/misc/ss.c)。
NFS I/O 来自 `/proc/self/mountstats` 的 `/data` 挂载,磁盘容量来自 statvfs;CPU、内存来自 `/proc`。采样历史仅覆盖采集器启用后的时间,不补造旧数据。图表最多展示 241 个采样点,均值、当前值、峰值从当前窗口全部原始有效采样计算。
镜像文件由 Nginx 直接提供,API 不读取整个仓库;镜像目录响应设置 CSP sandbox 和 nosniff,避免仓库中的 HTML/脚本获得管理页面同源权限。
## 备份与回退
日常备份包括 SQLite、`uploads/`、`/etc/mirrorouc/` 和 `/etc/tunasync/`。数据库运行时使用 SQLite backup API 或 `sqlite3 ... '.backup ...'`;直接复制数据库文件前先停止 `mirrorouc.service`,同时处理 WAL 文件。备份文件包含账户哈希、会话和可能带内部地址的配置,应限制读取权限。
应用版本回退:将 `/opt/mirrorouc/current` 指回旧发布目录,再 `systemctl restart mirrorouc`。本次新增数据库表使用 `CREATE TABLE IF NOT EXISTS`;未来若出现不兼容数据库迁移,需要同时恢复对应数据库备份。
只撤下预览时,删除 `/etc/nginx/sites-enabled/mirrorouc-preview`,通过 `nginx -t` 后 reload Nginx,并停止 `mirrorouc.service` 与 `mirrorouc-config.service`。这不会修改原有 80 端口镜像站点。停止新增采集器不会影响 tunasync;若需要删除计数表,只删除 `inet mirrorouc_monitor`,不要 flush 全局 nftables 规则。
## 验证范围
`backend/*_test.go` 使用临时数据库与假的 Manager,覆盖鉴权、角色、CSRF、草稿隔离、并发冲突、任务提交、配置应用/恢复、上游不可用缓存、图片校验及监控统计。`test_config_helper.py` 独立验证写入白名单、自定义命令保留、注入拒绝与失败恢复,不操作真实同步配置。
本次部署核验真实 API、镜像目录、日志、监控与后台草稿保存。没有为验证效果而停止、重启仓库,或应用修改后的生产同步参数。
## 统一中文内容升级
公告的 title/body 和镜像的 help 已改为单个字符串,所有界面语言共用中文内容。旧版本的语言映射在服务启动时以事务迁移:优先保留 zh-CN,其次 zh;没有中文键的历史记录保留原内容供管理员整理。原始 JSON 保存在 SQLite 的 shared_content_backups 表,迁移会递增记录 revision,避免旧编辑页面覆盖新内容;重复启动不会重复迁移。
此次内容格式升级需要在回退旧应用时同时恢复升级前的数据库备份。部署前使用 SQLite backup API 生成一致性快照,备份目录仅 root 可读。迁移不修改镜像介绍、站点文案或同步配置。
## 首页相关链接
`settings/site` 的 `relatedLinks` 独立控制首页侧栏,与页脚 `links` 分开。旧设置缺少该字段时,接口补充原有的三个入口,不改写数据库。显式保存空数组会隐藏该栏目;仍打开旧版表单的客户端提交时,若未携带该字段,后端保留当前已保存的相关链接。条目支持 `kind=help` 的站内使用帮助与 `kind=url` 的 HTTP(S)/站内绝对路径;不接受脚本、data URL、协议相对地址或反斜杠路径。名称支持多语言,数组顺序即显示顺序。
+124
View File
@@ -0,0 +1,124 @@
#!/usr/bin/python3
"""Sample per-socket TCP receive counters and attribute worker processes to mirrors."""
import json
import os
from pathlib import Path
import re
import subprocess
import tempfile
import time
PROC = Path('/proc')
MIRRORS = '/data/mirrors/'
OUTPUT = Path('/var/lib/mirrorouc/metrics/job-speeds.json')
NAME = re.compile(r'^[a-zA-Z0-9][a-zA-Z0-9_.+-]{0,127}$')
def mirror_name(value):
if not value.startswith(MIRRORS):
return None
name = value[len(MIRRORS):].split('/')[0]
return name if NAME.fullmatch(name) and name not in ('.', '..') else None
def process_jobs(proc=PROC):
jobs = {}
parents = {}
for entry in proc.iterdir():
if not entry.name.isdigit():
continue
try:
if '/tunasync-worker.service' not in (entry / 'cgroup').read_text():
continue
pid = int(entry.name)
status = (entry / 'stat').read_text().rsplit(') ', 1)[1].split()
parents[pid] = int(status[1])
arguments = (entry / 'cmdline').read_bytes().decode(errors='replace').split('\0')
candidates = {name for argument in arguments if (name := mirror_name(argument))}
if not candidates:
name = mirror_name(os.readlink(entry / 'cwd'))
if name:
candidates.add(name)
if len(candidates) == 1:
jobs[pid] = candidates.pop()
except (OSError, ValueError, IndexError):
continue
# git/ssh/curl helpers may inherit a job's parent without its destination argument.
for pid in parents:
ancestor = pid
for _ in range(16):
if ancestor in jobs:
jobs[pid] = jobs[ancestor]
break
ancestor = parents.get(ancestor, 0)
if not ancestor:
break
return jobs
def parse_sockets(output, jobs):
sockets = {}
header = None
for line in output.splitlines():
if line and not line[0].isspace():
header = line
continue
if header is None:
continue
identity = re.search(r'\bino:(\d+)\s+sk:([a-fA-F0-9]+)', header)
if not identity:
continue
pids = [int(value) for value in re.findall(r'\bpid=(\d+)', header)]
names = {jobs[pid] for pid in pids if pid in jobs}
received = re.search(r'\bbytes_received:(\d+)', line)
# The same inherited socket can list several PIDs: count its inode/cookie once.
sockets[':'.join(identity.groups())] = {'bytes': int(received.group(1)) if received else 0, 'job': names.pop() if len(names) == 1 else None}
header = None
return sockets
def rates(current, previous, active, elapsed):
result = {name: {'bytesPerSecond': 0.0, 'connections': 0} for name in active}
if previous is None or elapsed <= 0 or elapsed > 15:
return {}
for identity, socket in current.items():
name = socket['job']
if name is None:
continue
old = previous.get(identity)
delta = socket['bytes'] - old['bytes'] if old else socket['bytes']
if delta < 0: # A reset/reused socket must never produce a negative rate/spike.
continue
value = result.setdefault(name, {'bytesPerSecond': 0.0, 'connections': 0})
value['bytesPerSecond'] += delta / elapsed
value['connections'] += 1
return result
def write(value):
fd, temporary = tempfile.mkstemp(prefix='.speeds-', dir=OUTPUT.parent)
try:
with os.fdopen(fd, 'w') as stream:
json.dump(value, stream, allow_nan=False, separators=(',', ':'))
os.chmod(temporary, 0o640)
os.replace(temporary, OUTPUT)
finally:
if os.path.exists(temporary):
os.unlink(temporary)
def run():
previous = None
last = time.monotonic()
while True:
start = time.monotonic()
try:
jobs = process_jobs()
result = subprocess.run(['/usr/bin/ss', '-tinpeH', 'state', 'established'], capture_output=True, text=True, timeout=4, check=True, env={**os.environ, 'LC_ALL': 'C'})
current = parse_sockets(result.stdout, jobs)
sampled = time.monotonic()
elapsed = sampled - last
values = rates(current, previous, set(jobs.values()), elapsed)
write({'updatedAt': round(time.time() * 1000), 'interval': elapsed, 'available': previous is not None and elapsed <= 15, 'jobs': values})
previous, last = current, sampled
except (OSError, ValueError, subprocess.SubprocessError):
previous = None
write({'updatedAt': round(time.time() * 1000), 'interval': 2, 'available': False, 'jobs': {}})
time.sleep(max(.1, 2 - (time.monotonic() - start)))
if __name__ == '__main__':
run()
+84
View File
@@ -0,0 +1,84 @@
#!/usr/bin/python3
"""Read Linux/NFS counters. A separate passive nftables table supplies protocol bytes."""
import json
import os
from pathlib import Path
import subprocess
import tempfile
import time
ROOT = Path(os.environ.get('MIRROROUC_METRICS_ROOT', '/var/lib/mirrorouc/metrics'))
ROOT.mkdir(parents=True, exist_ok=True)
path = ROOT / 'history.json'
try:
history = json.loads(path.read_text())
except (OSError, ValueError):
history = {'samples': []}
now = time.time()
cpu = list(map(int, Path('/proc/stat').read_text().splitlines()[0].split()[1:9]))
mem = {line.split(':')[0]: int(line.split()[1]) * 1024 for line in Path('/proc/meminfo').read_text().splitlines() if len(line.split()) > 1}
nfs = {}
for block in Path('/proc/self/mountstats').read_text().split('device '):
if ' mounted on /data ' not in block:
continue
for line in block.splitlines():
values = line.strip().split()
if values and values[0] in ('READ:', 'WRITE:'):
nfs[values[0][:-1]] = list(map(int, values[1:]))
counters = {}
try:
result = subprocess.run(['/usr/sbin/nft', '-j', 'list', 'counters', 'table', 'inet', 'mirrorouc_monitor'], capture_output=True, timeout=4, check=True)
for item in json.loads(result.stdout).get('nftables', []):
counter = item.get('counter', {})
if counter.get('name') in ('https4', 'https6', 'rsync4', 'rsync6'):
counters[counter['name']] = counter['bytes']
except (OSError, ValueError, subprocess.SubprocessError):
pass
previous = history.get('previous')
sample = {'time': round(now * 1000)}
if previous:
elapsed = max(.1, now - previous['time'])
delta = [max(0, a - b) for a, b in zip(cpu, previous['cpu'])]
total = sum(delta)
if total:
sample.update(user=(delta[0] + delta[1]) / total * 100, system=(delta[2] + delta[5] + delta[6] + delta[7]) / total * 100, wait=delta[4] / total * 100, idle=delta[3] / total * 100)
for key, value in counters.items():
old = previous.get('counters', {}).get(key)
if old is not None and value >= old:
sample[key] = (value - old) * 8 / elapsed / 1e9
for operation, prefix in [('READ', 'read'), ('WRITE', 'write')]:
a, b = nfs.get(operation), previous.get('nfs', {}).get(operation)
if a and b and a[0] >= b[0]:
sample[prefix + 'Ops'] = (a[0] - b[0]) / elapsed / 1000
byte_index = 4 if operation == 'READ' else 3
sample[prefix + 'Bytes'] = max(0, a[byte_index] - b[byte_index]) / elapsed / 1e9
cached = max(0, mem.get('Cached', 0) + mem.get('SReclaimable', 0) - mem.get('Shmem', 0))
free, buffers, total_mem = mem['MemFree'], mem.get('Buffers', 0), mem['MemTotal']
used = max(0, total_mem - free - cached - buffers)
sample.update(used=used / 2**30, cached=cached / 2**30, buffers=buffers / 2**30, free=free / 2**30)
if 'user' in sample:
history['samples'].append(sample)
disks = []
for mount in ['/data', '/']:
try:
disk = os.statvfs(mount)
disks.append({'name': mount, 'total': disk.f_blocks * disk.f_frsize / 1e12, 'used': (disk.f_blocks - disk.f_bfree) * disk.f_frsize / 1e12})
except OSError:
pass
history.update(previous={'time': now, 'cpu': cpu, 'nfs': nfs, 'counters': counters}, updatedAt=round(now * 1000), memoryTotal=mem['MemTotal'] / 2**30, disks=disks)
history['samples'] = [point for point in history['samples'] if point['time'] >= (now - 86400) * 1000][-8640:]
fd, temporary = tempfile.mkstemp(dir=ROOT, prefix='.metrics-')
try:
with os.fdopen(fd, 'w') as output:
json.dump(history, output, allow_nan=False, separators=(',', ':'))
output.flush()
os.fsync(output.fileno())
os.chmod(temporary, 0o640)
os.replace(temporary, path)
finally:
if os.path.exists(temporary):
os.unlink(temporary)
+178
View File
@@ -0,0 +1,178 @@
#!/usr/bin/python3
"""Root-only configuration writer, reachable only through a group-restricted Unix socket.
Independently verifies allowed TOML changes; never executes submitted commands.
"""
import copy
import glob
import grp
import hashlib
import json
import os
import re
import socketserver
import tempfile
import tomllib
import urllib.parse
import urllib.request
ROOT = '/etc/tunasync'
MAIN = ROOT + '/worker.conf'
INCLUDES = ROOT + '/mirrors.d'
SOCKET = '/run/mirrorouc-config/control.sock'
JOURNAL = '/var/lib/mirrorouc-config/pending.json'
IDENTIFIER = re.compile(r'^[a-zA-Z0-9][a-zA-Z0-9_.+-]{0,127}$')
def number(value, lower, upper):
return type(value) is int and lower <= value <= upper
def upstream(value):
if not isinstance(value, str) or len(value) >= 2048 or re.search(r'[\s`;$|<>\\\"\']', value):
return False
parsed = urllib.parse.urlsplit(value)
return parsed.scheme in ('rsync', 'https', 'http') and bool(parsed.hostname) and parsed.username is None
def sync_numbers(table):
return all(key not in table or number(table[key], low, high) for key, low, high in [('interval', 1, 525600), ('retry', 0, 20), ('timeout', 30, 1209600)])
def stripped_mirrors(doc):
result = copy.deepcopy(doc)
mirrors = result.get('mirrors', [])
assert isinstance(mirrors, list)
for mirror in mirrors:
for field in ('upstream', 'interval', 'retry', 'timeout'):
mirror.pop(field, None)
return result
def validate_new(doc, path):
assert set(doc) == {'mirrors'} and len(doc['mirrors']) == 1
mirror = doc['mirrors'][0]
assert set(mirror) <= {'name', 'provider', 'upstream', 'interval', 'retry', 'timeout', 'stage1_profile'}
assert IDENTIFIER.fullmatch(mirror['name']) and mirror['name'] not in ('.', '..')
assert path == INCLUDES + '/mirrorouc-' + mirror['name'] + '.conf'
assert mirror['provider'] in ('rsync', 'two-stage-rsync')
assert upstream(mirror['upstream']) and sync_numbers(mirror)
assert 'stage1_profile' not in mirror or mirror['stage1_profile'] == 'debian'
def validate(path, before, after):
assert path == MAIN or (os.path.dirname(path) == INCLUDES and os.path.basename(path).endswith('.conf'))
assert not os.path.islink(path)
old = tomllib.loads(before) if before is not None else None
new = tomllib.loads(after) if after is not None else None
assert old is not None or new is not None
if path == MAIN:
assert old is not None and new is not None
assert number(new['global'].get('concurrent'), 1, 64) and sync_numbers(new['global'])
for doc in (old, new):
for field in ('concurrent', 'interval', 'retry', 'timeout'):
doc['global'].pop(field, None)
assert old == new
elif old is None or new is None:
validate_new(new if new is not None else old, path)
else:
assert stripped_mirrors(old) == stripped_mirrors(new)
for previous, current in zip(old.get('mirrors', []), new.get('mirrors', [])):
if previous != current:
assert upstream(current['upstream']) and sync_numbers(current)
def read(path):
try:
with open(path, encoding='utf-8', newline='') as stream:
return stream.read()
except FileNotFoundError:
return None
def atomic(path, value):
if value is None:
os.unlink(path)
return
mode = os.stat(path).st_mode & 0o777 if os.path.exists(path) else 0o644
fd, temporary = tempfile.mkstemp(prefix='.mirrorouc-', dir=os.path.dirname(path))
try:
with os.fdopen(fd, 'w', encoding='utf-8', newline='') as stream:
stream.write(value)
stream.flush()
os.fsync(stream.fileno())
os.chmod(temporary, mode)
os.replace(temporary, path)
finally:
if os.path.exists(temporary):
os.unlink(temporary)
def reload_worker():
worker = tomllib.loads(read(MAIN))['global']['name']
body = json.dumps({'worker_id': worker, 'mirror_id': '', 'cmd': 'reload', 'args': [], 'options': {}}).encode()
request = urllib.request.Request('http://127.0.0.1:12345/cmd', data=body, headers={'Content-Type': 'application/json'})
with urllib.request.urlopen(request, timeout=10) as response:
assert response.status == 200
def signature():
pattern = tomllib.loads(read(MAIN)).get('include', {}).get('include_mirrors', '')
assert pattern == INCLUDES + '/*.conf'
names = sorted(set([MAIN] + glob.glob(pattern)))
digest = hashlib.sha256()
for path in names:
assert not os.path.islink(path)
raw = read(path).encode()
digest.update(str(len(path.encode())).encode() + b':' + path.encode() + str(len(raw)).encode() + b':' + raw)
return digest.hexdigest()
def restore(changes):
for path, change in changes.items():
if change['before'] is None and not os.path.exists(path):
continue
atomic(path, change['before'])
reload_worker()
def apply(request):
changes = request['files']
assert isinstance(changes, dict) and 0 < len(changes) <= 20
for path, change in changes.items():
validate(path, change['before'], change['after'])
if read(path) != change['before']:
return 'CONFLICT'
if signature() != request['baseHash']:
return 'CONFLICT'
# Write-ahead journal survives a helper/process restart during a multi-file update.
atomic(JOURNAL, json.dumps(changes))
try:
for path, change in changes.items():
atomic(path, change['after'])
reload_worker()
except Exception:
try:
restore(changes)
except Exception:
return 'CONFIG_ROLLBACK_FAILED'
os.unlink(JOURNAL)
return 'CONFIG_APPLY_FAILED'
os.unlink(JOURNAL)
return ''
class Handler(socketserver.StreamRequestHandler):
def handle(self):
self.connection.settimeout(5)
try:
raw = self.rfile.readline(2 * 1024 * 1024 + 1)
assert len(raw) <= 2 * 1024 * 1024 and raw.endswith(b'\n')
if os.path.exists(JOURNAL):
code = 'CONFIG_ROLLBACK_FAILED'
else:
code = apply(json.loads(raw))
except (AssertionError, KeyError, TypeError, ValueError):
code = 'INVALID_INPUT'
except Exception:
code = 'CONFIG_APPLY_FAILED'
self.wfile.write(json.dumps({'code': code}).encode() + b'\n')
if __name__ == '__main__':
if os.path.exists(JOURNAL):
with open(JOURNAL) as stream:
restore(json.load(stream))
os.unlink(JOURNAL)
if os.path.exists(SOCKET):
os.unlink(SOCKET)
with socketserver.UnixStreamServer(SOCKET, Handler) as server:
os.chown(SOCKET, 0, grp.getgrnam('mirrorouc').gr_gid)
os.chmod(SOCKET, 0o660)
server.serve_forever()
+5
View File
@@ -0,0 +1,5 @@
#!/bin/sh
set -eu
if ! /usr/sbin/nft list table inet mirrorouc_monitor >/dev/null 2>&1; then
/usr/sbin/nft -f /etc/mirrorouc/protocol-counters.nft
fi
+23
View File
@@ -0,0 +1,23 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")"
if ! id mirrorouc >/dev/null 2>&1; then
useradd --system --home-dir /var/lib/mirrorouc --shell /usr/sbin/nologin mirrorouc
fi
install -d -o mirrorouc -g mirrorouc -m 0700 /var/lib/mirrorouc
install -d -o root -g mirrorouc -m 0750 /var/lib/mirrorouc/metrics
install -d -o root -g root -m 0755 /usr/local/lib/mirrorouc
install -d -o root -g root -m 0755 /etc/mirrorouc
install -m 0644 collect_metrics.py /usr/local/lib/mirrorouc/collect_metrics.py
install -m 0644 collect_job_speeds.py /usr/local/lib/mirrorouc/collect_job_speeds.py
install -m 0755 ensure-counters.sh /usr/local/lib/mirrorouc/ensure-counters.sh
install -m 0644 protocol-counters.nft /etc/mirrorouc/protocol-counters.nft
install -m 0644 mirrorouc-counters.service mirrorouc-metrics.service mirrorouc-metrics.timer mirrorouc-speeds.service /etc/systemd/system/
if ! /usr/sbin/nft list table inet mirrorouc_monitor >/dev/null 2>&1; then
/usr/sbin/nft -c -f /etc/mirrorouc/protocol-counters.nft
fi
systemctl daemon-reload
systemctl enable --now mirrorouc-counters.service mirrorouc-metrics.timer
systemctl start mirrorouc-metrics.service
systemctl enable --now mirrorouc-speeds.service
systemctl restart mirrorouc-speeds.service
+46
View File
@@ -0,0 +1,46 @@
#!/bin/sh
# Run as root: sh deploy/install-preview.sh /opt/mirrorouc/releases/<release>
set -eu
release=$(readlink -f "$1")
case "$release" in /opt/mirrorouc/releases/*) ;; *) echo 'Invalid release path' >&2; exit 1;; esac
test -x "$release/mirrorouc"
test -f "$release/dist/index.html"
deploy="$release/deploy"
id mirrorouc >/dev/null 2>&1 || useradd --system --home-dir /var/lib/mirrorouc --shell /usr/sbin/nologin mirrorouc
install -d -m 0700 -o mirrorouc -g mirrorouc /var/lib/mirrorouc
install -d -m 0755 /etc/mirrorouc /usr/local/lib/mirrorouc
install -d -m 0700 /var/lib/mirrorouc-config
install -m 0755 "$deploy/config-helper.py" /usr/local/lib/mirrorouc/config-helper.py
install -m 0644 "$deploy/mirrorouc.service" "$deploy/mirrorouc-config.service" /etc/systemd/system/
if [ ! -f /etc/mirrorouc/mirrorouc.env ]; then
umask 077
cat > /etc/mirrorouc/mirrorouc.env <<'EOF'
MIRROROUC_LISTEN=127.0.0.1:8081
MIRROROUC_DATA=/var/lib/mirrorouc
MIRROROUC_STATIC=/opt/mirrorouc/current/dist
MIRROROUC_MANAGER=http://127.0.0.1:12345
MIRROROUC_WORKER=srv-software-mirror
MIRROROUC_WORKER_CONFIG=/etc/tunasync/worker.conf
MIRROROUC_LOGS=/var/log/tunasync
MIRROROUC_METRICS=/var/lib/mirrorouc/metrics/history.json
MIRROROUC_CONFIG_SOCKET=/run/mirrorouc-config/control.sock
MIRROROUC_SECURE_COOKIES=false
EOF
fi
# Leave current tunasync configurations, ownership and permissions intact.
if [ ! -s /var/lib/mirrorouc/mirrorouc.db ]; then
umask 077
runuser -u mirrorouc -- "$release/mirrorouc" -data /var/lib/mirrorouc -init-admin admin > /etc/mirrorouc/initial-admin-password
fi
ln -sfn "$release" /opt/mirrorouc/current.new
mv -Tf /opt/mirrorouc/current.new /opt/mirrorouc/current
install -m 0644 "$deploy/nginx-preview.conf" /etc/nginx/sites-available/mirrorouc-preview
ln -sfn /etc/nginx/sites-available/mirrorouc-preview /etc/nginx/sites-enabled/mirrorouc-preview
nginx -t
systemctl daemon-reload
systemctl enable --now mirrorouc-config.service
systemctl restart mirrorouc-config.service
systemctl enable mirrorouc.service
systemctl restart mirrorouc.service
systemctl reload nginx
echo 'MirrorOUC preview installed at http://100.69.2.2:8080'
+25
View File
@@ -0,0 +1,25 @@
[Unit]
Description=MirrorOUC restricted tunasync configuration writer
After=tunasync-manager.service
[Service]
ExecStart=/usr/bin/python3 /usr/local/lib/mirrorouc/config-helper.py
Restart=on-failure
RestartSec=5
RuntimeDirectory=mirrorouc-config
RuntimeDirectoryMode=0755
StateDirectory=mirrorouc-config
StateDirectoryMode=0700
UMask=0077
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/etc/tunasync /var/lib/mirrorouc-config /run/mirrorouc-config
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true
RestrictAddressFamilies=AF_UNIX AF_INET
[Install]
WantedBy=multi-user.target
+9
View File
@@ -0,0 +1,9 @@
[Unit]
Description=MirrorOUC passive protocol byte counters
After=nftables.service
[Service]
Type=oneshot
ExecStart=/usr/local/lib/mirrorouc/ensure-counters.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
+13
View File
@@ -0,0 +1,13 @@
[Unit]
Description=MirrorOUC read-only host metrics collection
[Service]
Type=oneshot
User=root
Group=mirrorouc
ExecStart=/usr/bin/python3 /usr/local/lib/mirrorouc/collect_metrics.py
UMask=0027
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/lib/mirrorouc/metrics
ProtectHome=true
PrivateTmp=true
+8
View File
@@ -0,0 +1,8 @@
[Unit]
Description=Collect MirrorOUC metrics every 10 seconds
[Timer]
OnBootSec=10
OnUnitActiveSec=10
AccuracySec=1
[Install]
WantedBy=timers.target
+22
View File
@@ -0,0 +1,22 @@
[Unit]
Description=MirrorOUC per-mirror upstream TCP receive rates
After=tunasync-worker.service
[Service]
User=root
Group=mirrorouc
ExecStart=/usr/bin/python3 /usr/local/lib/mirrorouc/collect_job_speeds.py
Restart=on-failure
RestartSec=3
UMask=0027
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/mirrorouc/metrics
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6
[Install]
WantedBy=multi-user.target
+26
View File
@@ -0,0 +1,26 @@
[Unit]
Description=MirrorOUC web and administration API
After=network.target tunasync-manager.service mirrorouc-config.service
Wants=mirrorouc-config.service
[Service]
User=mirrorouc
Group=mirrorouc
WorkingDirectory=/opt/mirrorouc/current
EnvironmentFile=/etc/mirrorouc/mirrorouc.env
ExecStart=/opt/mirrorouc/current/mirrorouc
Restart=on-failure
RestartSec=5
UMask=0077
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/mirrorouc
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
[Install]
WantedBy=multi-user.target
+22
View File
@@ -0,0 +1,22 @@
server {
listen 100.69.2.2:8080;
server_name 100.69.2.2;
client_max_body_size 5m;
location /mirrors/ {
alias /data/mirrors/;
autoindex on;
autoindex_exact_size off;
autoindex_localtime on;
add_header X-Content-Type-Options nosniff always;
add_header Content-Security-Policy "sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src 'self' data:" always;
}
location / {
proxy_pass http://127.0.0.1:8081;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 35s;
}
}
+25
View File
@@ -0,0 +1,25 @@
# Passive accounting only; no existing rules or tables are flushed.
table inet mirrorouc_monitor {
counter https4 { }
counter https6 { }
counter rsync4 { }
counter rsync6 { }
chain account {
meta nfproto ipv4 tcp sport 443 counter name https4
meta nfproto ipv4 tcp dport 443 tcp sport != 443 counter name https4
meta nfproto ipv6 tcp sport 443 counter name https6
meta nfproto ipv6 tcp dport 443 tcp sport != 443 counter name https6
meta nfproto ipv4 tcp sport 873 counter name rsync4
meta nfproto ipv4 tcp dport 873 tcp sport != 873 counter name rsync4
meta nfproto ipv6 tcp sport 873 counter name rsync6
meta nfproto ipv6 tcp dport 873 tcp sport != 873 counter name rsync6
}
chain input {
type filter hook input priority 300; policy accept;
iifname != "lo" jump account
}
chain output {
type filter hook output priority 300; policy accept;
oifname != "lo" jump account
}
}
+47
View File
@@ -0,0 +1,47 @@
import copy
import importlib.util
import json
import os
import tempfile
import unittest
from unittest.mock import patch
spec = importlib.util.spec_from_file_location('helper', os.path.join(os.path.dirname(__file__), 'config-helper.py'))
helper = importlib.util.module_from_spec(spec)
spec.loader.exec_module(helper)
class ValidationTests(unittest.TestCase):
def test_preserves_custom_commands(self):
before = '[[mirrors]]\nname="debian"\nprovider="rsync"\nupstream="rsync://old/debian"\ncommand="/usr/local/bin/parallel-rsync"\n'
after = before.replace('old/debian', 'new/debian') + 'interval=60\nretry=2\ntimeout=120\n'
helper.validate(helper.INCLUDES + '/debian.conf', before, after)
with self.assertRaises(AssertionError):
helper.validate(helper.INCLUDES + '/debian.conf', before, after.replace('parallel-rsync', 'other-command'))
def test_blocks_other_global_fields_and_paths(self):
before = '[global]\nname="worker"\nconcurrent=12\ninterval=60\nretry=2\ntimeout=120\n'
helper.validate(helper.MAIN, before, before.replace('concurrent=12', 'concurrent=8'))
for path, after in [('/etc/shadow', before), (helper.MAIN, before.replace('worker', 'other')), (helper.MAIN, before.replace('concurrent=12', 'concurrent=0'))]:
with self.assertRaises(AssertionError):
helper.validate(path, before, after)
def test_new_mirrors_and_injection(self):
value = '[[mirrors]]\nname="test"\nprovider="rsync"\nupstream="rsync://mirror/test"\ninterval=60\nretry=0\ntimeout=120\n'
helper.validate_new(helper.tomllib.loads(value), helper.INCLUDES + '/mirrorouc-test.conf')
for changed in [value + 'command="evil"\n', value.replace('rsync://mirror/test', 'rsync://mirror/`cmd`'), value.replace('retry=0', 'retry=-1')]:
with self.assertRaises(AssertionError):
helper.validate_new(helper.tomllib.loads(changed), helper.INCLUDES + '/mirrorouc-test.conf')
def test_reload_failure_restores_files(self):
with tempfile.TemporaryDirectory() as root:
path = root + '/worker.conf'
before = '[global]\nname="worker"\nconcurrent=12\ninterval=60\nretry=2\ntimeout=120\n'
helper.atomic(path, before)
changes = {path: {'before': before, 'after': before.replace('concurrent=12', 'concurrent=8')}}
with patch.object(helper, 'MAIN', path), patch.object(helper, 'JOURNAL', root + '/pending.json'), patch.object(helper, 'signature', return_value='base'), patch.object(helper, 'reload_worker', side_effect=[OSError('unavailable'), None]):
self.assertEqual(helper.apply({'files': changes, 'baseHash': 'base'}), 'CONFIG_APPLY_FAILED')
self.assertEqual(helper.read(path), before)
self.assertFalse(os.path.exists(root + '/pending.json'))
if __name__ == '__main__':
unittest.main()
+40
View File
@@ -0,0 +1,40 @@
import importlib.util
import os
from pathlib import Path
import tempfile
import unittest
spec = importlib.util.spec_from_file_location('speeds', os.path.join(os.path.dirname(__file__), 'collect_job_speeds.py'))
speeds = importlib.util.module_from_spec(spec)
spec.loader.exec_module(speeds)
class SpeedTests(unittest.TestCase):
def test_sockets_deduplicate_shared_pids_and_separate_repos(self):
data = '0 0 10.0.0.1:123 10.0.0.2:873 users:(("rsync",pid=1,fd=3),("rsync",pid=2,fd=3)) uid:9 ino:20 sk:abc <->\n\t cubic bytes_sent:4 bytes_received:1000\n0 0 10.0.0.1:124 10.0.0.2:873 users:(("rsync",pid=3,fd=3)) uid:9 ino:21 sk:abd <->\n\t cubic bytes_received:2000\n'
current = speeds.parse_sockets(data, {1: 'debian', 2: 'debian', 3: 'ubuntu'})
result = speeds.rates(current, {'20:abc': {'bytes': 400}, '21:abd': {'bytes': 1000}}, {'debian', 'ubuntu'}, 2)
self.assertEqual(result['debian'], {'bytesPerSecond': 300, 'connections': 1})
self.assertEqual(result['ubuntu']['bytesPerSecond'], 500)
def test_parallel_and_new_connections_add_without_startup_spike(self):
current = {'1:a': {'bytes': 1000, 'job': 'debian'}, '2:b': {'bytes': 500, 'job': 'debian'}, '3:c': {'bytes': 9000, 'job': None}}
self.assertEqual(speeds.rates(current, None, {'debian'}, 2), {})
result = speeds.rates(current, {'1:a': {'bytes': 800}}, {'debian', 'ubuntu'}, 2)
self.assertEqual(result['debian']['bytesPerSecond'], 350)
self.assertEqual(result['ubuntu']['bytesPerSecond'], 0)
self.assertEqual(speeds.rates(current, {}, {'debian'}, 20), {})
def test_process_ancestry_and_path_boundaries(self):
with tempfile.TemporaryDirectory() as root:
for pid, parent, command in [(1, 0, ['/usr/bin/python', '/worker']), (2, 1, ['rsync', '/data/mirrors/debian/']), (3, 2, ['ssh', 'host']), (4, 0, ['rsync', '/data/mirrors/ubuntu/'])]:
path = Path(root) / str(pid); path.mkdir()
(path/'stat').write_text(f'{pid} (process) S {parent} 0 0')
(path/'cgroup').write_text('0::/system.slice/tunasync-worker.service' if pid != 4 else '0::/system.slice/other.service')
(path/'cmdline').write_bytes('\0'.join(command).encode())
jobs = speeds.process_jobs(Path(root))
self.assertEqual(jobs, {2: 'debian', 3: 'debian'})
self.assertIsNone(speeds.mirror_name('/data/mirrors-other/secret'))
self.assertIsNone(speeds.mirror_name('/data/mirrors/../secret'))
if __name__ == '__main__':
unittest.main()