diff --git a/servers/rpi.md b/servers/rpi.md index 670f7e5..cc68c27 100644 --- a/servers/rpi.md +++ b/servers/rpi.md @@ -290,3 +290,10 @@ Docker CE 29.9 + Compose v5.6 из репо download.docker.com (в Debian ст - 2026-10-14 владелец опознал Onyx Boox: IP при обнаружении 192.168.10.146, MAC 00:0a:0a:fd:89:17, hostname android-dhcp-11. Ранее неизвестное устройство; через `router known add` записано имя «Onyx Boox Руслана» в known.json. Это только отметка реестра, DHCP и сеть не менялись. - 2026-10-15 исправлено распознавание локальных вложений с пробелами вокруг Markdown-пути: прежняя ссылка `[PDF]( /tmp/...)` не вызывала отправку. astra-bot `dd79caf`, attachments.py установлен в /opt/astra-bot, прежний сохранён .bak-before-spaces; 133 теста прошли (включая пробелы и угловые скобки). PDF morphemes-20261015 заново доставлен владельцу штатным send-preview, Telegram подтвердил успех. Перезапуск с проверкой готовности запланирован через astra-attachment-space-restart на 60 секунд; остальные ограничения путей сохранены. + +## Telegram: выгрузка второго аккаунта владельца (с 2026-10-17) + +- Проект `tg_export` (Telethon) перенесён с Mac: код, юниты и установка — [`stacks/rpi-tg-export/`](../stacks/rpi-tg-export/README.md). На малине — **второй аккаунт владельца**, докачка **только текста** раз в сутки (`tg-export.timer`, 04:30, `run --no-media --archived`), выгрузка в `/mnt/backup/tg/` (tgexport:astra 2750/640 — Astra читает). Старая выгрузка первого аккаунта (~14 ГБ) — на Mac, не переносилась (решение владельца). +- Отдельный системный пользователь `tgexport` (без шелла): сессия `/var/lib/tg-export/tg.session` (600) = полный доступ к аккаунту — Astra её не видит; ключи приложения `/etc/tg-export/env` (root:tgexport 640, те же `api_id`/`api_hash`, что на Mac). Вход — владелец: `ssh -t rpi sudo -u tgexport tg-export login`. +- Мониторинг — `backup-metrics backup_tg_export` → `BackupFailed`/`BackupStale` (>30 ч). ⚠️ Первый прогон может идти дольше суток из-за FloodWait. +- Доступ: тексты — только для владельца; Оле и Поле закрыто (правило в `codex/AGENTS.md` и скилл `tg`, astra-bot `27bf8e6`), как почта. diff --git a/stacks/rpi-tg-export/README.md b/stacks/rpi-tg-export/README.md new file mode 100644 index 0000000..a3060fb --- /dev/null +++ b/stacks/rpi-tg-export/README.md @@ -0,0 +1,33 @@ +# Выгрузка Telegram — второй аккаунт владельца (rpi) + +Проект `tg_export` (Telethon, MTProto) перенесён с Mac 2026-10-17: на малине — **второй аккаунт владельца**, +регулярная докачка **только текста** (медиа — позже, если понадобится). Старая выгрузка первого аккаунта +(~14 ГБ) осталась на Mac в `~/pets/tg_export/out`, сюда не переносилась; анализ (`stats`, `trips`, `presence`…) — там же. + +| Что | Где | +|---|---| +| Код | `/opt/tg-export/` (`export.py`, `fetch_public.py`, `channel_comments.py`), venv `/opt/tg-export/venv` | +| Обёртка | `/usr/local/bin/tg-export` — подставляет `--session` и `--out`, ключи из env | +| Ключи приложения (`TG_API_ID`/`TG_API_HASH`, my.telegram.org) | `/etc/tg-export/env` — root:tgexport 640, в git нет | +| Сессия (= **полный доступ к аккаунту**) | `/var/lib/tg-export/tg.session` — tgexport 600 | +| Выгрузка | `/mnt/backup/tg/` — `dialogs.json`, `chats/_<название>/{messages.jsonl,messages.txt,meta.json,_state.json}`; tgexport:astra, каталоги 2750, файлы 640 | +| Таймер | `tg-export.timer` — 04:30 ±10 мин, `tg-export.service` (oneshot, `run --no-media --archived`) | +| Мониторинг | `ExecStopPost=+backup-metrics backup_tg_export` → алерты `BackupFailed` / `BackupStale` (>30 ч) | + +Пользователь `tgexport` — системный, без шелла: Astra (группа astra) читает тексты, но **не видит сессию и ключи**. + +## Установка и вход + +```sh +sudo sh install.sh # код, venv, пользователь, каталоги, юниты +sudo install -o root -g tgexport -m 640 env /etc/tg-export/env # ключи приложения (как .env.example) +ssh -t rpi sudo -u tgexport tg-export login # номер, код из Telegram, 2FA — делает владелец +sudo systemctl enable --now tg-export.timer +``` + +Вручную: `sudo -u tgexport tg-export list` (диалоги), `sudo -u tgexport tg-export run --no-media --only @чат`. +Повторный запуск догружает только новое (`_state.json` в каждом чате), `--full` — перезалить чат. + +⚠️ Первый прогон долгий: Telegram троттлит массовое чтение (FloodWait, скрипт ждёт до 24 ч) — `BackupStale` +может сработать, пока он идёт; следующий запуск таймера не стартует поверх идущего. +⚠️ Медиа выключены (`--no-media`); включить — убрать флаг в юните (лимит по умолчанию 50 МБ на файл). diff --git a/stacks/rpi-tg-export/app/.env.example b/stacks/rpi-tg-export/app/.env.example new file mode 100644 index 0000000..dd3ea29 --- /dev/null +++ b/stacks/rpi-tg-export/app/.env.example @@ -0,0 +1,3 @@ +# https://my.telegram.org → API development tools → создать приложение +TG_API_ID=1234567 +TG_API_HASH=0123456789abcdef0123456789abcdef diff --git a/stacks/rpi-tg-export/app/channel_comments.py b/stacks/rpi-tg-export/app/channel_comments.py new file mode 100644 index 0000000..d469029 --- /dev/null +++ b/stacks/rpi-tg-export/app/channel_comments.py @@ -0,0 +1,200 @@ +#!/usr/bin/env python3 +"""Выгрузка канала вместе с комментариями из привязанной группы обсуждений. + +Обычный `export.py run` берёт только посты канала: комментарии физически +лежат в отдельном linked chat и через iter_dialogs не приходят. Здесь для +каждого поста дополнительно вытягивается его ветка (iter_messages(reply_to=...)), +плюс, при --with-linked, вся история группы обсуждений целиком — там бывают +сообщения, не привязанные ни к одному посту. + + uv run channel_comments.py @cpython_notes + uv run channel_comments.py @cpython_notes --with-linked --session copy.session + +Формат вывода совпадает с export.py: сырой to_dict() в *.jsonl, плоское +представление в *_pretty.jsonl, читаемая расшифровка в *.txt. +""" + +from __future__ import annotations + +import argparse +import asyncio +import datetime as dt +import json +import sys +from pathlib import Path + +from telethon import TelegramClient, errors, functions + +import export as ex + +ROOT = Path(__file__).resolve().parent + + +async def dump_stream(client, entity, it, jsonl, pretty, txt, cache, label, tag=None): + """Прогоняет итератор сообщений в три файла. Возвращает счётчик.""" + n = 0 + async for msg in it: + if ex.STOP: + break + record = msg.to_dict() + if tag: + record.update(tag) + sender = await ex.sender_label(client, msg, cache) + jsonl.write(json.dumps(record, ensure_ascii=False, cls=ex.Encoder) + "\n") + if pretty: + flat = ex.simplify_message(msg, sender, record) + if tag: + flat.update(tag) + pretty.write(json.dumps(flat, ensure_ascii=False, cls=ex.Encoder) + "\n") + if txt: + txt.write(ex.render_line(msg, sender)) + n += 1 + if n % 500 == 0: + print(f" … {label}: {n}", flush=True) + return n + + +async def main_async(args) -> None: + api_id, api_hash = ex.require_credentials() + client = TelegramClient(args.session, api_id, api_hash) + client.flood_sleep_threshold = 24 * 60 * 60 + await client.start() + try: + chan = await client.get_entity(args.channel) + full = await client(functions.channels.GetFullChannelRequest(channel=chan)) + linked_id = getattr(full.full_chat, "linked_chat_id", None) + linked = None + if linked_id: + linked = next((c for c in full.chats if c.id == linked_id), None) + + dialog_id = int(f"-100{chan.id}") + folder = Path(args.out) / "chats" / f"{dialog_id}_{ex.slugify(getattr(chan, 'title', args.channel))}" + folder.mkdir(parents=True, exist_ok=True) + + header = { + "id": dialog_id, + "name": getattr(chan, "title", None), + "kind": ex.dialog_kind(chan), + "username": getattr(chan, "username", None), + "participants": getattr(full.full_chat, "participants_count", None), + "linked_chat_id": linked_id, + "linked_chat_title": getattr(linked, "title", None) if linked else None, + "exported_at": dt.datetime.now(dt.timezone.utc).isoformat(), + } + (folder / "meta.json").write_text( + json.dumps({**header, "entity": chan.to_dict()}, + ensure_ascii=False, indent=2, cls=ex.Encoder), + encoding="utf-8") + + print(f"Канал: {header['name']} (@{header['username']}) · " + f"подписчиков {header['participants']}") + print(f"Группа обсуждений: {header['linked_chat_title']} (id={linked_id})\n" + if linked_id else "Группа обсуждений не привязана — комментариев нет\n") + + cache: dict = {} + mode = "w" if args.full else "a" + + # ---- посты канала ---- + pj = (folder / "messages.jsonl").open(mode, encoding="utf-8") + pp = (folder / "_pretty.jsonl").open(mode, encoding="utf-8") + pt = (folder / "messages.txt").open(mode, encoding="utf-8") + posts = [] + async for m in client.iter_messages(chan, reverse=True): + posts.append(m) + for m in posts: + record = m.to_dict() + sender = await ex.sender_label(client, m, cache) + pj.write(json.dumps(record, ensure_ascii=False, cls=ex.Encoder) + "\n") + pp.write(json.dumps(ex.simplify_message(m, sender, record), + ensure_ascii=False, cls=ex.Encoder) + "\n") + pt.write(ex.render_line(m, sender)) + for f in (pj, pp, pt): + f.close() + print(f"постов: {len(posts)}") + + if not linked_id: + return + + # ---- комментарии: по ветке на каждый пост ---- + cj = (folder / "comments.jsonl").open(mode, encoding="utf-8") + cp = (folder / "comments_pretty.jsonl").open(mode, encoding="utf-8") + thread = (folder / "threads.txt").open(mode, encoding="utf-8") + total_c = 0 + with_c = 0 + for i, post in enumerate(posts, 1): + if ex.STOP: + break + try: + it = client.iter_messages(chan, reply_to=post.id, reverse=True) + n = 0 + head_written = False + async for cm in it: + if not head_written: + head = (post.message or "").split("\n")[0][:90] + thread.write(f"\n{'='*78}\nПОСТ #{post.id} " + f"[{post.date:%Y-%m-%d %H:%M}] {head}\n{'-'*78}\n") + head_written = True + record = cm.to_dict() + record["_post_id"] = post.id + sender = await ex.sender_label(client, cm, cache) + cj.write(json.dumps(record, ensure_ascii=False, cls=ex.Encoder) + "\n") + flat = ex.simplify_message(cm, sender, record) + flat["_post_id"] = post.id + cp.write(json.dumps(flat, ensure_ascii=False, cls=ex.Encoder) + "\n") + thread.write(" " + ex.render_line(cm, sender).replace("\n", "\n ").rstrip() + "\n") + n += 1 + except (errors.MsgIdInvalidError, errors.ChannelPrivateError) as exc: + # у поста отключены комментарии либо ветка недоступна + print(f" пост #{post.id}: {type(exc).__name__}", file=sys.stderr) + continue + except errors.FloodWaitError as exc: + print(f" FloodWait {exc.seconds}s на посте #{post.id} — жду", + file=sys.stderr) + await asyncio.sleep(exc.seconds + 1) + continue + total_c += n + if n: + with_c += 1 + print(f" [{i}/{len(posts)}] пост #{post.id}: {n} комментариев", flush=True) + for f in (cj, cp, thread): + f.close() + print(f"\nкомментариев: {total_c} в {with_c} ветках " + f"(из {len(posts)} постов)") + + # ---- вся группа обсуждений целиком ---- + if args.with_linked: + lj = (folder / "linked_chat.jsonl").open(mode, encoding="utf-8") + lp = (folder / "linked_chat_pretty.jsonl").open(mode, encoding="utf-8") + lt = (folder / "linked_chat.txt").open(mode, encoding="utf-8") + n = await dump_stream( + client, linked, client.iter_messages(linked, reverse=True), + lj, lp, lt, cache, "linked") + for f in (lj, lp, lt): + f.close() + print(f"вся группа обсуждений: {n} сообщений") + + print(f"\nКаталог: {folder}") + finally: + await client.disconnect() + + +def main() -> None: + ex.load_dotenv(ROOT / ".env") + p = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("channel", help="@username, id или часть названия канала") + p.add_argument("--session", default=str(ROOT / "tg_export.session")) + p.add_argument("--out", default=str(ROOT / "out")) + p.add_argument("--with-linked", action="store_true", + help="дополнительно выгрузить всю группу обсуждений целиком") + p.add_argument("--full", action="store_true", help="перезаписать файлы с нуля") + args = p.parse_args() + ex.install_signal_handlers() + try: + asyncio.run(main_async(args)) + except KeyboardInterrupt: + sys.exit(130) + + +if __name__ == "__main__": + main() diff --git a/stacks/rpi-tg-export/app/export.py b/stacks/rpi-tg-export/app/export.py new file mode 100644 index 0000000..ea8d10f --- /dev/null +++ b/stacks/rpi-tg-export/app/export.py @@ -0,0 +1,901 @@ +#!/usr/bin/env python3 +"""Полная выгрузка истории личного Telegram-аккаунта через Telethon. + +Возможности: + * логин по номеру телефона (MTProto, свой api_id/api_hash); + * индекс всех диалогов (личные, боты, группы, каналы); + * выгрузка всех сообщений в JSONL (сырой `to_dict()` — ничего не теряется) + плюс опциональный человекочитаемый .txt; + * скачивание медиа с лимитом на размер; + * возобновление: повторный запуск догружает только новое; + * аккуратная обработка FloodWait и приватных/недоступных чатов. + +Команды: + python export.py login — только авторизация (создаёт .session) + python export.py list — вывести список диалогов + python export.py run — выгрузить историю +""" + +from __future__ import annotations + +import argparse +import asyncio +import base64 +import datetime as dt +import json +import os +import re +import signal +import sys +import unicodedata +from pathlib import Path + +from telethon import TelegramClient, errors, utils +from telethon.tl import types + +# --------------------------------------------------------------------------- # +# конфиг +# --------------------------------------------------------------------------- # + +ROOT = Path(__file__).resolve().parent + + +def load_dotenv(path: Path) -> None: + """Минимальный .env-лоадер, чтобы не тащить python-dotenv.""" + if not path.exists(): + return + for raw in path.read_text(encoding="utf-8").splitlines(): + line = raw.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, _, val = line.partition("=") + key, val = key.strip(), val.strip().strip("'\"") + os.environ.setdefault(key, val) + + +def require_credentials() -> tuple[int, str]: + api_id = os.environ.get("TG_API_ID") + api_hash = os.environ.get("TG_API_HASH") + if not api_id or not api_hash: + sys.exit( + "Нет TG_API_ID / TG_API_HASH.\n" + "Получи их на https://my.telegram.org → API development tools,\n" + f"затем скопируй {ROOT / '.env.example'} в {ROOT / '.env'} и заполни." + ) + try: + return int(api_id), api_hash + except ValueError: + sys.exit(f"TG_API_ID должен быть числом, получено: {api_id!r}") + + +# --------------------------------------------------------------------------- # +# утилиты +# --------------------------------------------------------------------------- # + +class Encoder(json.JSONEncoder): + """Сериализует всё, что встречается в Telethon-объектах.""" + + def default(self, o): # noqa: D102 + if isinstance(o, (dt.datetime, dt.date, dt.time)): + return o.isoformat() + if isinstance(o, dt.timedelta): + return o.total_seconds() + if isinstance(o, (bytes, bytearray, memoryview)): + return base64.b64encode(bytes(o)).decode("ascii") + if isinstance(o, set): + return sorted(o, key=str) + return str(o) + + +_SLUG_BAD = re.compile(r"[^\w\-. ]+", re.UNICODE) + + +def slugify(name: str, limit: int = 48) -> str: + name = unicodedata.normalize("NFKC", name or "").strip() + name = _SLUG_BAD.sub("", name) + name = re.sub(r"\s+", "_", name).strip("._-") + return (name[:limit] or "unnamed") + + +def dialog_kind(entity) -> str: + if isinstance(entity, types.User): + return "bot" if getattr(entity, "bot", False) else "user" + if isinstance(entity, types.Chat) or isinstance(entity, types.ChatForbidden): + return "group" + if isinstance(entity, types.Channel): + return "supergroup" if getattr(entity, "megagroup", False) else "channel" + return "unknown" + + +def human_size(n: int | None) -> str: + if not n: + return "0B" + for unit in ("B", "K", "M", "G", "T"): + if n < 1024 or unit == "T": + return f"{n:.0f}{unit}" if unit == "B" else f"{n:.1f}{unit}" + n /= 1024 + return f"{n:.1f}T" + + +def parse_date(value: str | None) -> dt.datetime | None: + if not value: + return None + try: + parsed = dt.datetime.fromisoformat(value) + except ValueError: + sys.exit(f"Не смог разобрать дату {value!r}. Формат: YYYY-MM-DD или YYYY-MM-DDTHH:MM") + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=dt.timezone.utc) + return parsed + + +STOP = False + + +def install_signal_handlers() -> None: + def handler(signum, frame): # noqa: ARG001 + global STOP + if STOP: + sys.exit(130) + STOP = True + print("\n[!] Останавливаюсь после текущего сообщения; прогресс сохранён. Ctrl-C ещё раз — выход.", + file=sys.stderr) + + signal.signal(signal.SIGINT, handler) + signal.signal(signal.SIGTERM, handler) + + +# --------------------------------------------------------------------------- # +# состояние (для возобновления) +# --------------------------------------------------------------------------- # + +class State: + def __init__(self, path: Path): + self.path = path + self.data = {"last_id": 0, "count": 0, "media": 0, "updated_at": None} + if path.exists(): + try: + self.data.update(json.loads(path.read_text(encoding="utf-8"))) + except (ValueError, OSError): + pass + + @property + def last_id(self) -> int: + return int(self.data.get("last_id") or 0) + + def bump(self, msg_id: int, media: int = 0) -> None: + self.data["last_id"] = max(self.last_id, int(msg_id)) + self.data["count"] = int(self.data.get("count") or 0) + 1 + self.data["media"] = int(self.data.get("media") or 0) + media + + def save(self) -> None: + self.data["updated_at"] = dt.datetime.now(dt.timezone.utc).isoformat() + tmp = self.path.with_suffix(".tmp") + tmp.write_text(json.dumps(self.data, ensure_ascii=False, indent=2), encoding="utf-8") + tmp.replace(self.path) + + +# --------------------------------------------------------------------------- # +# текстовое представление +# --------------------------------------------------------------------------- # + +def render_line(msg, sender_name: str) -> str: + when = msg.date.astimezone().strftime("%Y-%m-%d %H:%M:%S") if msg.date else "?" + body = (msg.message or "").replace("\n", "\n ") + tags = [] + if msg.media is not None and not isinstance(msg.media, types.MessageMediaWebPage): + tags.append(type(msg.media).__name__.replace("MessageMedia", "").lower()) + if getattr(msg, "reply_to_msg_id", None): + tags.append(f"reply→{msg.reply_to_msg_id}") + if getattr(msg, "fwd_from", None): + tags.append("forwarded") + if getattr(msg, "action", None): + tags.append(type(msg.action).__name__.replace("MessageAction", "action:")) + suffix = f" [{', '.join(tags)}]" if tags else "" + return f"[{when}] #{msg.id} {sender_name}:{suffix}\n {body}\n" if body else \ + f"[{when}] #{msg.id} {sender_name}:{suffix}\n" + + +async def sender_label(client, msg, cache: dict) -> str: + sid = msg.sender_id + if sid is None: + return "—" + if sid in cache: + return cache[sid] + label = str(sid) + try: + sender = await msg.get_sender() + if sender is not None: + if isinstance(sender, types.User): + parts = [sender.first_name or "", sender.last_name or ""] + label = " ".join(p for p in parts if p) or (sender.username or str(sid)) + else: + label = getattr(sender, "title", None) or str(sid) + except (errors.RPCError, ValueError, TypeError): + pass + cache[sid] = label + return label + + +# --------------------------------------------------------------------------- # +# читаемое представление (messages.json) +# --------------------------------------------------------------------------- # + +def media_info(msg, record: dict) -> dict | None: + media = getattr(msg, "media", None) + if media is None: + return None + + if isinstance(media, types.MessageMediaWebPage): + url = getattr(getattr(media, "webpage", None), "url", None) + return {"type": "link", "url": url} if url else None + + info: dict = {"type": type(media).__name__.replace("MessageMedia", "").lower()} + + handle = getattr(msg, "file", None) + for key in ("name", "mime_type", "size", "duration"): + val = getattr(handle, key, None) + if val: + info[key] = round(val, 1) if key == "duration" else val + if info.get("size"): + info["size_human"] = human_size(info["size"]) + + for attr in getattr(getattr(media, "document", None), "attributes", None) or []: + if isinstance(attr, types.DocumentAttributeSticker): + info["type"] = "sticker" + info["emoji"] = attr.alt + elif isinstance(attr, types.DocumentAttributeAudio) and attr.voice: + info["type"] = "voice" + + if isinstance(media, types.MessageMediaPoll): + poll = media.poll + info["question"] = getattr(poll.question, "text", poll.question) + info["answers"] = [getattr(a.text, "text", a.text) for a in poll.answers] + elif isinstance(media, (types.MessageMediaGeo, types.MessageMediaGeoLive)): + geo = media.geo + info["lat"], info["long"] = getattr(geo, "lat", None), getattr(geo, "long", None) + elif isinstance(media, types.MessageMediaContact): + info["contact"] = " ".join( + p for p in (media.first_name, media.last_name, media.phone_number) if p + ) + + for key in ("_media_file", "_media_skipped", "_media_error"): + if key in record: + info[key.lstrip("_").removeprefix("media_")] = record[key] + return info + + +def forward_info(msg) -> dict | None: + fwd = getattr(msg, "fwd_from", None) + if fwd is None: + return None + info: dict = {} + if fwd.from_name: + info["from_name"] = fwd.from_name + if fwd.from_id is not None: + try: + info["from_id"] = utils.get_peer_id(fwd.from_id) + except (TypeError, ValueError): + info["from_id"] = str(fwd.from_id) + if fwd.date: + info["original_date"] = fwd.date.astimezone().isoformat() + if fwd.post_author: + info["post_author"] = fwd.post_author + return info or None + + +def reactions_info(msg) -> list | None: + results = getattr(getattr(msg, "reactions", None), "results", None) or [] + out = [] + for res in results: + emoji = getattr(res.reaction, "emoticon", None) \ + or getattr(res.reaction, "document_id", None) + out.append({"reaction": str(emoji), "count": res.count}) + return out or None + + +def simplify_message(msg, sender: str, record: dict) -> dict: + """Плоское, читаемое представление сообщения для messages.json.""" + out: dict = { + "id": msg.id, + "date": msg.date.astimezone().isoformat() if msg.date else None, + "from": sender, + "from_id": msg.sender_id, + "outgoing": bool(getattr(msg, "out", False)), + "text": getattr(msg, "message", "") or "", + } + if getattr(msg, "reply_to_msg_id", None): + out["reply_to"] = msg.reply_to_msg_id + if getattr(msg, "edit_date", None): + out["edited"] = msg.edit_date.astimezone().isoformat() + if fwd := forward_info(msg): + out["forwarded"] = fwd + if media := media_info(msg, record): + out["media"] = media + if reactions := reactions_info(msg): + out["reactions"] = reactions + + action = getattr(msg, "action", None) + if action is not None: + out["service_action"] = type(action).__name__.replace("MessageAction", "") + if duration := getattr(action, "duration", None): + out["service_action_duration_sec"] = duration + if getattr(msg, "views", None): + out["views"] = msg.views + if getattr(msg, "via_bot_id", None): + out["via_bot_id"] = msg.via_bot_id + if getattr(msg, "pinned", False): + out["pinned"] = True + if getattr(msg, "grouped_id", None): + out["album_id"] = str(msg.grouped_id) + return out + + +# медиа без файла: качать нечего, для них download_media вернёт None. +# MessageMediaContact сюда НЕ входит — Telethon сохраняет контакт как .vcard. +NO_FILE_MEDIA = frozenset({ + "MessageMediaWebPage", "MessageMediaGeo", "MessageMediaGeoLive", + "MessageMediaPoll", "MessageMediaDice", "MessageMediaVenue", + "MessageMediaGame", "MessageMediaUnsupported", "MessageMediaStory", + "MessageMediaInvoice", "MessageMediaGiveaway", + "MessageMediaGiveawayResults", "MessageMediaEmpty", +}) + + +def raw_media_tags(media: dict) -> set[str]: + """Метки медиа из сырого to_dict(): тип, mime, расширение, вид документа. + + Позволяет отсеивать по `--skip-media webp` / `sticker` / `video` ещё до + запроса к API — по данным, которые уже лежат в messages.jsonl. + """ + if not media: + return set() + kind = media.get("_", "") + tags = {kind.replace("MessageMedia", "").lower()} + if kind == "MessageMediaPhoto": + tags |= {"photo", "jpg", "jpeg", "image/jpeg"} + + doc = media.get("document") or {} + mime = (doc.get("mime_type") or "").lower() + if mime: + tags.add(mime) + tags.add(mime.rsplit("/", 1)[-1]) + + for attr in doc.get("attributes") or []: + name = attr.get("_", "") + if name == "DocumentAttributeSticker": + tags.add("sticker") + elif name == "DocumentAttributeAudio": + tags.add("voice" if attr.get("voice") else "audio") + elif name == "DocumentAttributeVideo": + tags.add("round" if attr.get("round_message") else "video") + elif name == "DocumentAttributeAnimated": + tags.add("gif") + elif name == "DocumentAttributeFilename": + filename = attr.get("file_name") or "" + if "." in filename: + tags.add(filename.rsplit(".", 1)[-1].lower()) + return tags + + +def has_downloadable_media(msg) -> bool: + media = getattr(msg, "media", None) + return media is not None and type(media).__name__ not in NO_FILE_MEDIA + + +def load_media_index(folder: Path) -> dict[str, dict]: + path = folder / "media_index.json" + if not path.exists(): + return {} + try: + return json.loads(path.read_text(encoding="utf-8")) + except (ValueError, OSError): + return {} + + +def assemble_pretty(folder: Path, header: dict) -> int: + """Собирает messages.json из накопленного _pretty.jsonl (устойчиво к докачке).""" + src = folder / "_pretty.jsonl" + if not src.exists(): + return 0 + by_id: dict[int, dict] = {} + with src.open(encoding="utf-8") as handle: + for line in handle: + line = line.strip() + if not line: + continue + try: + rec = json.loads(line) + except ValueError: + continue + by_id[rec["id"]] = rec + + # подмешиваем результаты догрузки медиа, если она уже прошла + for msg_id, entry in load_media_index(folder).items(): + rec = by_id.get(int(msg_id)) + if rec is None or "media" not in rec: + continue + # не ссылаемся на файлы, которых уже нет на диске (удалили вручную и т.п.) + if "file" in entry and not (folder / entry["file"]).exists(): + continue + rec["media"].update({k: v for k, v in entry.items() if v is not None}) + + messages = [by_id[key] for key in sorted(by_id)] + payload = { + "chat": header, + "messages_count": len(messages), + "first_message_date": messages[0]["date"] if messages else None, + "last_message_date": messages[-1]["date"] if messages else None, + "messages": messages, + } + (folder / "messages.json").write_text( + json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8" + ) + return len(messages) + + +# --------------------------------------------------------------------------- # +# выгрузка одного диалога +# --------------------------------------------------------------------------- # + +async def export_dialog(client, dialog, out_root: Path, args) -> dict: + entity = dialog.entity + kind = dialog_kind(entity) + folder = out_root / "chats" / f"{dialog.id}_{slugify(dialog.name)}" + media_dir = folder / "media" + folder.mkdir(parents=True, exist_ok=True) + + state = State(folder / "_state.json") + start_from = 0 if args.full else state.last_id + + header = { + "id": dialog.id, + "name": dialog.name, + "kind": kind, + "username": getattr(entity, "username", None), + "phone": getattr(entity, "phone", None), + "is_archived": bool(getattr(dialog, "archived", False)), + "exported_at": dt.datetime.now(dt.timezone.utc).isoformat(), + } + (folder / "meta.json").write_text( + json.dumps({**header, "entity": entity.to_dict()}, + ensure_ascii=False, indent=2, cls=Encoder), + encoding="utf-8", + ) + + excluded_types = {t.lower().lstrip(".") for t in args.skip_media} + mode = "w" if args.full else "a" + jsonl = (folder / "messages.jsonl").open(mode, encoding="utf-8") + txt = (folder / "messages.txt").open(mode, encoding="utf-8") if args.txt else None + pretty = (folder / "_pretty.jsonl").open(mode, encoding="utf-8") if args.pretty else None + cache: dict = {} + new_msgs = 0 + new_media = 0 + media_bytes = 0 + errors_count = 0 + + try: + async for msg in client.iter_messages( + entity, + reverse=True, # от старых к новым — так работает докачка + offset_id=start_from, + offset_date=args._since, + limit=args.limit, + wait_time=args.wait or None, + ): + if STOP: + break + if args._until and msg.date and msg.date > args._until: + break + + record = msg.to_dict() + + if args.media and has_downloadable_media(msg): + size = getattr(getattr(msg, "file", None), "size", None) or 0 + hit = raw_media_tags(record.get("media") or {}) & excluded_types + if hit: + record["_media_skipped"] = f"type_excluded:{','.join(sorted(hit))}" + elif args.max_media_mb and size > args.max_media_mb * 1024 * 1024: + record["_media_skipped"] = f"too_large:{human_size(size)}" + else: + media_dir.mkdir(parents=True, exist_ok=True) + try: + path = await client.download_media(msg, file=str(media_dir / f"{msg.id}")) + if path: + record["_media_file"] = str(Path(path).relative_to(folder)) + new_media += 1 + media_bytes += size or Path(path).stat().st_size + except errors.FloodWaitError as exc: + print(f" FloodWait {exc.seconds}s на медиа #{msg.id} — жду", file=sys.stderr) + await asyncio.sleep(exc.seconds + 1) + except (errors.RPCError, OSError, ValueError) as exc: + record["_media_error"] = f"{type(exc).__name__}: {exc}" + errors_count += 1 + + jsonl.write(json.dumps(record, ensure_ascii=False, cls=Encoder) + "\n") + + label = await sender_label(client, msg, cache) if (txt or pretty) else "" + if txt is not None: + txt.write(render_line(msg, label)) + if pretty is not None: + pretty.write(json.dumps(simplify_message(msg, label, record), + ensure_ascii=False, cls=Encoder) + "\n") + + state.bump(msg.id, media=1 if "_media_file" in record else 0) + new_msgs += 1 + if new_msgs % 500 == 0: + jsonl.flush() + for stream in (txt, pretty): + if stream is not None: + stream.flush() + state.save() + print(f" … {new_msgs} новых (всего {state.data['count']})", flush=True) + finally: + jsonl.close() + for stream in (txt, pretty): + if stream is not None: + stream.close() + state.save() + if args.pretty: + assemble_pretty(folder, header) + + return { + "id": dialog.id, + "name": dialog.name, + "kind": kind, + "new_messages": new_msgs, + "total_messages": state.data["count"], + "new_media": new_media, + "media_bytes": media_bytes, + "media_errors": errors_count, + "folder": str(folder.relative_to(out_root)), + } + + +# --------------------------------------------------------------------------- # +# отбор диалогов +# --------------------------------------------------------------------------- # + +def dialog_matches(dialog, args) -> bool: + kind = dialog_kind(dialog.entity) + if args.kinds and kind not in args.kinds: + return False + if not args.archived and getattr(dialog, "archived", False): + return False + + if args.skip: + for needle in args.skip: + if _needle_hits(dialog, needle): + return False + if args.only: + return any(_needle_hits(dialog, needle) for needle in args.only) + return True + + +def _needle_hits(dialog, needle: str) -> bool: + needle = needle.strip() + if not needle: + return False + if needle.lstrip("-").isdigit() and int(needle) == dialog.id: + return True + username = getattr(dialog.entity, "username", None) or "" + if needle.lstrip("@").lower() == username.lower() and username: + return True + return needle.lower() in (dialog.name or "").lower() + + +# --------------------------------------------------------------------------- # +# команды +# --------------------------------------------------------------------------- # + +async def cmd_login(client, args) -> None: # noqa: ARG001 + me = await client.get_me() + print(f"OK: {me.first_name or ''} {me.last_name or ''} " + f"(@{me.username or '—'}, id={me.id}). Сессия: {args.session}") + + +async def cmd_list(client, args) -> None: + rows = [] + async for dialog in client.iter_dialogs(): + if not dialog_matches(dialog, args): + continue + rows.append((dialog.id, dialog_kind(dialog.entity), dialog.name or "", + getattr(dialog.entity, "username", None) or "", + getattr(dialog, "unread_count", 0))) + width = max((len(r[2]) for r in rows), default=10) + print(f"{'ID':>15} {'ТИП':<10} {'НАЗВАНИЕ':<{min(width, 50)}} USERNAME") + for did, kind, name, username, _ in rows: + print(f"{did:>15} {kind:<10} {name[:50]:<{min(width, 50)}} " + f"{'@' + username if username else ''}") + print(f"\nвсего: {len(rows)}") + + +async def cmd_media(client, args) -> None: + """Догружает медиа к уже выгруженным сообщениям. + + Идёт не по API-истории, а по локальному messages.jsonl, поэтому: + * не зависит от прогресса `run` и не мешает ему; + * инкрементальна — уже скачанное фиксируется в media_index.json; + * может гоняться повторно, пока `run` дописывает историю. + """ + out_root = Path(args.out).expanduser().resolve() + folders = sorted(p for p in (out_root / "chats").glob("*") if p.is_dir()) + if not folders: + sys.exit(f"В {out_root / 'chats'} нет выгруженных чатов — сначала запусти `run`.") + + limit_bytes = int(args.max_media_mb * 1024 * 1024) if args.max_media_mb else 0 + grand_new = grand_bytes = grand_skip = grand_err = 0 + + for folder in folders: + try: + header = json.loads((folder / "meta.json").read_text(encoding="utf-8")) + except (ValueError, OSError): + print(f"[!] {folder.name}: нет читаемого meta.json — пропуск", file=sys.stderr) + continue + + needles = args.only or [] + if needles and not any( + n.lstrip("@").lower() in ((header.get("username") or "").lower(), + str(header.get("id"))) + or n.lower() in (header.get("name") or "").lower() + for n in needles + ): + continue + + raw = folder / "messages.jsonl" + if not raw.exists(): + continue + + index = load_media_index(folder) + excluded = {t.lower().lstrip(".") for t in args.skip_media} + todo: list[int] = [] + by_type_excluded = 0 + with raw.open(encoding="utf-8") as handle: + for line in handle: + line = line.strip() + if not line: + continue + try: + rec = json.loads(line) + except ValueError: + continue # хвост может быть недописан, если `run` ещё идёт + media = rec.get("media") + if not media or media.get("_") in NO_FILE_MEDIA: + index.pop(str(rec["id"]), None) # чистим ложные nothing_to_download + continue + if excluded and (raw_media_tags(media) & excluded): + index.pop(str(rec["id"]), None) # забываем то, что решили не хранить + by_type_excluded += 1 + continue + if rec.get("_media_file"): + continue + # готово только то, у чего реально есть файл: записи skipped/error + # переспрашиваются — иначе поднятый --max-media-mb ничего не даст + if "file" in index.get(str(rec["id"]), {}): + continue + todo.append(rec["id"]) + + print(f"\n{folder.name}: к догрузке {len(todo)} медиа " + f"(уже есть {len(index)}" + + (f", исключено по типу {by_type_excluded}" if by_type_excluded else "") + + ")", flush=True) + if not todo: + continue + + entity = await client.get_entity(int(header["id"])) + media_dir = folder / "media" + media_dir.mkdir(parents=True, exist_ok=True) + sem = asyncio.Semaphore(max(1, args.jobs)) + done = new = skipped = failed = 0 + got_bytes = 0 + + async def fetch(msg) -> None: + nonlocal done, new, skipped, failed, got_bytes + async with sem: + if msg is None or STOP: + return + size = getattr(getattr(msg, "file", None), "size", None) or 0 + if limit_bytes and size > limit_bytes: + index[str(msg.id)] = {"skipped": f"too_large:{human_size(size)}"} + skipped += 1 + else: + try: + path = await client.download_media( + msg, file=str(media_dir / f"{msg.id}")) + if path: + index[str(msg.id)] = { + "file": str(Path(path).relative_to(folder))} + new += 1 + got_bytes += size or Path(path).stat().st_size + else: + index[str(msg.id)] = {"error": "nothing_to_download"} + failed += 1 + except errors.FloodWaitError as exc: + print(f" FloodWait {exc.seconds}s — жду", file=sys.stderr, flush=True) + await asyncio.sleep(exc.seconds + 1) + return + except (errors.RPCError, OSError, ValueError) as exc: + index[str(msg.id)] = {"error": f"{type(exc).__name__}: {exc}"} + failed += 1 + done += 1 + if done % 200 == 0: + (folder / "media_index.json").write_text( + json.dumps(index, ensure_ascii=False, indent=1), encoding="utf-8") + print(f" … {done}/{len(todo)} +{new} файлов " + f"({human_size(got_bytes)}), крупных пропущено {skipped}, " + f"ошибок {failed}", flush=True) + + for start in range(0, len(todo), 100): + if STOP: + break + chunk = todo[start:start + 100] + try: + msgs = await client.get_messages(entity, ids=chunk) + except errors.FloodWaitError as exc: + print(f" FloodWait {exc.seconds}s на выборке — жду", file=sys.stderr, flush=True) + await asyncio.sleep(exc.seconds + 1) + msgs = await client.get_messages(entity, ids=chunk) + await asyncio.gather(*(fetch(m) for m in msgs)) + + (folder / "media_index.json").write_text( + json.dumps(index, ensure_ascii=False, indent=1), encoding="utf-8") + if args.pretty: + assemble_pretty(folder, header) + print(f" итого: +{new} файлов ({human_size(got_bytes)}), " + f"крупных пропущено {skipped}, ошибок {failed}", flush=True) + grand_new += new + grand_bytes += got_bytes + grand_skip += skipped + grand_err += failed + + print(f"\nМедиа{' (прервано)' if STOP else ''}: +{grand_new} файлов " + f"({human_size(grand_bytes)}), пропущено по размеру {grand_skip}, " + f"ошибок {grand_err}.") + + +async def cmd_run(client, args) -> None: + out_root = Path(args.out).expanduser().resolve() + (out_root / "chats").mkdir(parents=True, exist_ok=True) + + dialogs = [] + async for dialog in client.iter_dialogs(): + if dialog_matches(dialog, args): + dialogs.append(dialog) + + me = await client.get_me() + print(f"Аккаунт: @{me.username or me.id} · диалогов к выгрузке: {len(dialogs)} · вывод: {out_root}\n") + + index, summary = [], [] + for n, dialog in enumerate(dialogs, 1): + if STOP: + break + kind = dialog_kind(dialog.entity) + print(f"[{n}/{len(dialogs)}] {kind:<10} {dialog.name} (id={dialog.id})", flush=True) + try: + result = await export_dialog(client, dialog, out_root, args) + except errors.FloodWaitError as exc: + print(f" FloodWait {exc.seconds}s — жду и повторяю", file=sys.stderr) + await asyncio.sleep(exc.seconds + 1) + result = await export_dialog(client, dialog, out_root, args) + except (errors.ChannelPrivateError, errors.ChatAdminRequiredError, + errors.UserBannedInChannelError) as exc: + print(f" пропуск: {type(exc).__name__}", file=sys.stderr) + result = {"id": dialog.id, "name": dialog.name, "kind": kind, + "error": type(exc).__name__} + except errors.RPCError as exc: + print(f" ошибка: {type(exc).__name__}: {exc}", file=sys.stderr) + result = {"id": dialog.id, "name": dialog.name, "kind": kind, + "error": f"{type(exc).__name__}: {exc}"} + + index.append(result) + summary.append(result) + if "error" not in result: + print(f" +{result['new_messages']} сообщений, +{result['new_media']} медиа " + f"({human_size(result['media_bytes'])}), всего {result['total_messages']}", + flush=True) + + (out_root / "dialogs.json").write_text( + json.dumps(index, ensure_ascii=False, indent=2), encoding="utf-8") + + total_msgs = sum(r.get("new_messages", 0) for r in summary) + total_media = sum(r.get("new_media", 0) for r in summary) + total_bytes = sum(r.get("media_bytes", 0) for r in summary) + failed = [r for r in summary if "error" in r] + print(f"\nГотово{' (прервано)' if STOP else ''}: " + f"{len(summary)} диалогов, +{total_msgs} сообщений, " + f"+{total_media} медиа ({human_size(total_bytes)}).") + if failed: + print(f"Недоступны ({len(failed)}): " + + ", ".join(f"{r['name']} [{r['error']}]" for r in failed[:10]) + + (" …" if len(failed) > 10 else "")) + print(f"Каталог: {out_root}") + + +# --------------------------------------------------------------------------- # +# CLI +# --------------------------------------------------------------------------- # + +KINDS = ("user", "bot", "group", "supergroup", "channel") + + +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser( + description="Выгрузка истории личного Telegram через Telethon", + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + ) + p.add_argument("command", choices=("login", "list", "run", "media")) + p.add_argument("--session", default=str(ROOT / "tg_export.session"), + help="файл сессии Telethon") + p.add_argument("--out", default=str(ROOT / "out"), help="каталог для выгрузки") + + sel = p.add_argument_group("отбор диалогов") + sel.add_argument("--kinds", nargs="*", choices=KINDS, default=list(KINDS), + help="какие типы чатов выгружать") + sel.add_argument("--only", nargs="*", default=[], metavar="NEEDLE", + help="только эти чаты: id, @username или часть названия") + sel.add_argument("--skip", nargs="*", default=[], metavar="NEEDLE", + help="исключить чаты (id / @username / часть названия)") + sel.add_argument("--archived", action="store_true", help="включить архив") + + rng = p.add_argument_group("границы истории") + rng.add_argument("--since", metavar="YYYY-MM-DD", help="только сообщения не раньше даты") + rng.add_argument("--until", metavar="YYYY-MM-DD", help="только сообщения не позже даты") + rng.add_argument("--limit", type=int, default=None, + help="максимум сообщений на чат (для пробного прогона)") + rng.add_argument("--full", action="store_true", + help="игнорировать сохранённый прогресс и перезаписать всё с нуля") + + med = p.add_argument_group("медиа и вывод") + med.add_argument("--media", action=argparse.BooleanOptionalAction, default=True, + help="скачивать медиафайлы") + med.add_argument("--max-media-mb", type=float, default=50, + help="не качать файлы крупнее (0 — без лимита)") + med.add_argument("--skip-media", nargs="*", default=[], metavar="TAG", + help="не качать медиа с такими метками: расширение (webp, mp4), " + "mime (image/webp) или вид (sticker, video, voice, round, gif)") + med.add_argument("--pretty", action=argparse.BooleanOptionalAction, default=True, + help="писать читаемый messages.json (плоские поля, отступы)") + med.add_argument("--txt", action=argparse.BooleanOptionalAction, default=True, + help="дополнительно писать читаемый messages.txt") + med.add_argument("--wait", type=float, default=0.0, + help="пауза между запросами истории, сек (мягче к лимитам)") + med.add_argument("--jobs", type=int, default=4, + help="параллельных загрузок в команде media") + return p + + +async def main_async(args) -> None: + api_id, api_hash = require_credentials() + client = TelegramClient(args.session, api_id, api_hash) + client.flood_sleep_threshold = 24 * 60 * 60 # сам ждёт длинные FloodWait + + await client.start( + phone=lambda: input("Номер телефона (в формате +7...): ").strip(), + password=lambda: input("Пароль двухфакторной аутентификации: ").strip(), + ) + try: + handler = {"login": cmd_login, "list": cmd_list, + "run": cmd_run, "media": cmd_media}[args.command] + await handler(client, args) + finally: + await client.disconnect() + + +def main() -> None: + load_dotenv(ROOT / ".env") + args = build_parser().parse_args() + args._since = parse_date(args.since) + args._until = parse_date(args.until) + if args.max_media_mb <= 0: + args.max_media_mb = 0 + install_signal_handlers() + try: + asyncio.run(main_async(args)) + except (errors.PhoneNumberInvalidError, errors.SessionPasswordNeededError) as exc: + sys.exit(f"Ошибка авторизации: {type(exc).__name__}: {exc}") + except KeyboardInterrupt: + sys.exit(130) + + +if __name__ == "__main__": + main() diff --git a/stacks/rpi-tg-export/app/fetch_public.py b/stacks/rpi-tg-export/app/fetch_public.py new file mode 100644 index 0000000..919f95d --- /dev/null +++ b/stacks/rpi-tg-export/app/fetch_public.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python3 +"""Выгрузка публичной группы/канала, в которой мы не состоим. + +`export.py run` ходит по iter_dialogs и потому видит только свои чаты. +Здесь сущность резолвится по @username и история читается напрямую. + + uv run fetch_public.py @cyschools --since 2026-03-04 +""" +from __future__ import annotations + +import argparse +import asyncio +import datetime as dt +import json +from pathlib import Path + +from telethon import TelegramClient + +import export as ex + +ex.load_dotenv(ex.ROOT / ".env") + + +async def main_async(args) -> None: + api_id, api_hash = ex.require_credentials() + client = TelegramClient(args.session, api_id, api_hash) + client.flood_sleep_threshold = 24 * 60 * 60 + await client.start() + try: + entity = await client.get_entity(args.chat) + dialog_id = int(f"-100{entity.id}") + folder = Path(args.out) / "chats" / f"{dialog_id}_{ex.slugify(getattr(entity, 'title', args.chat))}" + folder.mkdir(parents=True, exist_ok=True) + + since = dt.datetime.fromisoformat(args.since).replace(tzinfo=dt.timezone.utc) + (folder / "meta.json").write_text(json.dumps({ + "id": dialog_id, + "name": getattr(entity, "title", None), + "kind": ex.dialog_kind(entity), + "username": getattr(entity, "username", None), + "fetched_since": args.since, + }, ensure_ascii=False, indent=2), encoding="utf-8") + + cache: dict = {} + n = 0 + with (folder / "messages.jsonl").open("w", encoding="utf-8") as jsonl, \ + (folder / "messages_pretty.jsonl").open("w", encoding="utf-8") as pretty: + async for msg in client.iter_messages(entity, offset_date=since, reverse=True): + record = msg.to_dict() + sender = await ex.sender_label(client, msg, cache) + jsonl.write(json.dumps(record, ensure_ascii=False, cls=ex.Encoder) + "\n") + pretty.write(json.dumps(ex.simplify_message(msg, sender, record), + ensure_ascii=False, cls=ex.Encoder) + "\n") + n += 1 + if n % 1000 == 0: + print(f" … {n} ({msg.date:%Y-%m-%d})", flush=True) + print(f"{folder.name}: {n} сообщений") + finally: + await client.disconnect() + + +def main() -> None: + p = argparse.ArgumentParser() + p.add_argument("chat") + p.add_argument("--since", required=True) + p.add_argument("--out", default=str(ex.ROOT / "out")) + p.add_argument("--session", default=str(ex.ROOT / "tg_export.session")) + asyncio.run(main_async(p.parse_args())) + + +if __name__ == "__main__": + main() diff --git a/stacks/rpi-tg-export/app/pyproject.toml b/stacks/rpi-tg-export/app/pyproject.toml new file mode 100644 index 0000000..7b8298e --- /dev/null +++ b/stacks/rpi-tg-export/app/pyproject.toml @@ -0,0 +1,12 @@ +[project] +name = "tg-export" +version = "0.1.0" +requires-python = ">=3.13" +description = "Полная выгрузка истории личного Telegram через Telethon" +dependencies = [ + "cryptg>=0.6.0", + "telethon>=1.44.0", +] + +[tool.uv] +package = false diff --git a/stacks/rpi-tg-export/app/uv.lock b/stacks/rpi-tg-export/app/uv.lock new file mode 100644 index 0000000..20825b4 --- /dev/null +++ b/stacks/rpi-tg-export/app/uv.lock @@ -0,0 +1,86 @@ +version = 1 +revision = 3 +requires-python = ">=3.13" + +[[package]] +name = "cryptg" +version = "0.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9a/cc/3cc53619c07ab1def2f0ce7a958c10528c947e973821234714cfc61d7dd7/cryptg-0.6.0.tar.gz", hash = "sha256:f8fd59fea56398ab812f218c044b42fbbaeee8a9ae32771d3108a3cf210781aa", size = 15019, upload-time = "2026-04-12T13:31:09.978Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c8/77/b56510a48a5c8f3cd38a0a2f3b93a6eee9e1e27ca3b1b8a3ad4a323f8311/cryptg-0.6.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:0ad5ccdd23d95f857c90e4109a7a8bd21cd998b342d8447d69881125f72b337e", size = 208770, upload-time = "2026-04-12T18:33:51.402Z" }, + { url = "https://files.pythonhosted.org/packages/80/92/2780c8f178396c64e1a8bf350d68e66a08d801cd0ba355e81f8d10e58bee/cryptg-0.6.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:2d5a99c8c3897930974f47bbfe75ab3304606a1d98f66b0d35d9dafdd98e0512", size = 238481, upload-time = "2026-04-12T18:35:15.194Z" }, + { url = "https://files.pythonhosted.org/packages/aa/db/84a3375d757c55f8c0e6d471828560a0f5bd6e8b2ecdf94cee44c1657c3d/cryptg-0.6.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:28497a18178b09f39b3d5faeaa34235ac1762b74017fbeac7bc34724c8efe0a0", size = 250130, upload-time = "2026-04-12T18:35:34.83Z" }, + { url = "https://files.pythonhosted.org/packages/52/1a/28f81903ac4c0a1db0f17b1774e67bcc64733d89188f21c10d3b19aef396/cryptg-0.6.0-cp313-cp313-win32.whl", hash = "sha256:c1bc4edee45eda7dd5963420d578bbf86f0a7e2cbbf0d25d5a812b4293a223f0", size = 110091, upload-time = "2026-04-12T18:36:01.976Z" }, + { url = "https://files.pythonhosted.org/packages/24/c3/742794894911bac8d83971b4a9ce742a0ca3b64d13ee2a64d2bb3cf5f01e/cryptg-0.6.0-cp313-cp313-win_amd64.whl", hash = "sha256:2b11d6ac297a63ba413cb79c97030cfb93763f041fd85755a488e0db13699336", size = 109113, upload-time = "2026-04-12T18:36:03.613Z" }, + { url = "https://files.pythonhosted.org/packages/a9/88/51f40b7991d77fab8d307fbe2b7e28dbef6d499141f953b0ade73faa2ab7/cryptg-0.6.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e726a6a40967d91ad1eaa0f6245cc79035b528c059a22de34aca18bfb61efd2e", size = 208094, upload-time = "2026-04-12T18:33:52.905Z" }, + { url = "https://files.pythonhosted.org/packages/17/71/16f4b747d9be7c363be16717d4c2094d710f09acce15efea39e21aaff9a7/cryptg-0.6.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:3ccaa95f9ed6e6457b80ca003c99ee035c072dc421636b2482d069fc575ab5ba", size = 236844, upload-time = "2026-04-12T18:35:17.103Z" }, + { url = "https://files.pythonhosted.org/packages/25/ae/8af9ebfa5aaca1da3d2ab8520b886b41194be1019e9cb414c5c4c492afaa/cryptg-0.6.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:c1931178451aa59da98294973bf6916115572a412514960d8fd8afb7919fb7fd", size = 250346, upload-time = "2026-04-12T18:35:36.593Z" }, + { url = "https://files.pythonhosted.org/packages/d8/d9/0214b6bc2b153abf91359de1a6035e59832e7377152de8749b378e6eebb5/cryptg-0.6.0-cp313-cp313t-win32.whl", hash = "sha256:0c312d2c0cc592cbf7bb08c1374f3ab27bcb01f46739b3ba23a4108f4f48661d", size = 109723, upload-time = "2026-04-12T18:35:58.77Z" }, + { url = "https://files.pythonhosted.org/packages/b0/89/133592fdb0859c639b5e62f5dfa24de4474b2e4e223668210e063592ddc8/cryptg-0.6.0-cp313-cp313t-win_amd64.whl", hash = "sha256:8748fe0ba8cce4268b2f3570f9fc8cd5e7d1730598174e271f525a93d5f64c90", size = 108149, upload-time = "2026-04-12T18:36:00.377Z" }, + { url = "https://files.pythonhosted.org/packages/68/ca/740ba5201f47d50cd24809719ddff38b8a5d950f9921e426a0d2990de084/cryptg-0.6.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:7eb4e335ac3df00333c202cf98e65edbecd8983c9a639c11d93f1f369ae64e05", size = 208586, upload-time = "2026-04-12T18:33:54.23Z" }, + { url = "https://files.pythonhosted.org/packages/ab/d0/cfc3664d4b8c57f2058278472de7052719d2e37aece5591512d07e14777e/cryptg-0.6.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:28f6ca127a907d00ad075252e8d970bbac55e7f3df0f66a277f52b986fab0a57", size = 238396, upload-time = "2026-04-12T18:35:19.201Z" }, + { url = "https://files.pythonhosted.org/packages/16/59/abc03cb099cec7b5e28866a8401040b40e99c719f3bef49fa7a0b302e09a/cryptg-0.6.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:4127534527ff741026b981359083da660010ddd556eb5e0f22523b9ce0fe80a2", size = 250040, upload-time = "2026-04-12T18:35:38.802Z" }, + { url = "https://files.pythonhosted.org/packages/8a/6e/707de2c52842f093ab05811d002214e284c50f18057160352d7baa4ef1fa/cryptg-0.6.0-cp314-cp314-win32.whl", hash = "sha256:9c823319f6ff00c3c04be00cad78bd6555ccfc832fffc6b709eb983bbe0182b0", size = 112774, upload-time = "2026-04-12T18:36:08.654Z" }, + { url = "https://files.pythonhosted.org/packages/d2/49/6d8c6313789c3649f089a5f277369013072a5f8d96ba93d8c83def9c4a75/cryptg-0.6.0-cp314-cp314-win_amd64.whl", hash = "sha256:0776b0256cd79ff202dd4c187872eb309bbee7ea63b9be722f9063b232b42ecd", size = 112208, upload-time = "2026-04-12T18:36:10.289Z" }, + { url = "https://files.pythonhosted.org/packages/de/31/3df7c7bfa022b8e49fba5bd511226eb8ab02b335832441dc9d5c07a154e7/cryptg-0.6.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:877e872b0967f03d008e2cb16a1aa44c8011b2f200bdf14771e6f783228e8bce", size = 207943, upload-time = "2026-04-12T18:33:55.483Z" }, + { url = "https://files.pythonhosted.org/packages/06/81/dea9beefd7317c353484a5736ce3de69ef9c1e84a93ea46cbe47f04a7f18/cryptg-0.6.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:b8ba3ed799086af279c1efb7bfa72b226dc9427f85034ab0b05ff23bccb97b45", size = 236856, upload-time = "2026-04-12T18:35:21.006Z" }, + { url = "https://files.pythonhosted.org/packages/e9/f1/fcc52170486f35698229b573c0b6bc844f0504745f43fc47fde0cb61888e/cryptg-0.6.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:ae96b26dfb53968fe4986f70222f2419c82651936f2ef3b2c2dc47bc74e77d96", size = 250297, upload-time = "2026-04-12T18:35:40.577Z" }, + { url = "https://files.pythonhosted.org/packages/c3/d5/5041385f3a55d6eaa8f86c79e9fbb799d09680977619d506e35c09a9a3f9/cryptg-0.6.0-cp314-cp314t-win32.whl", hash = "sha256:e4988fd1657aa86d91bc98d8262b78b141dec8ef51a843f1ef78d69e8a81fdf0", size = 111989, upload-time = "2026-04-12T18:36:05.193Z" }, + { url = "https://files.pythonhosted.org/packages/88/08/86db1d558e836c75724ef893c024d3f087886d1f8b173f2f64164505e256/cryptg-0.6.0-cp314-cp314t-win_amd64.whl", hash = "sha256:43f195810c642c6f6c552d291a216a5b5cb36b06aabe2832873db9dde3e65bb1", size = 111602, upload-time = "2026-04-12T18:36:06.81Z" }, +] + +[[package]] +name = "pyaes" +version = "1.6.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/44/66/2c17bae31c906613795711fc78045c285048168919ace2220daa372c7d72/pyaes-1.6.1.tar.gz", hash = "sha256:02c1b1405c38d3c370b085fb952dd8bea3fadcee6411ad99f312cc129c536d8f", size = 28536, upload-time = "2017-09-20T21:17:54.23Z" } + +[[package]] +name = "pyasn1" +version = "0.6.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a4/9a/23310166d960def5897e91fe20e5b724601b02a22e84ba1f94232c0b7f67/pyasn1-0.6.4.tar.gz", hash = "sha256:9c447d8431c947fe4c8febc4ed9e760bc29011a5b01e5c74b67025bd9fb8ce81", size = 151262, upload-time = "2026-07-09T01:12:33.988Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9a/3b/6163796d69c3977d1e4287bea4a6979161cbbdd170ebb430511e8e1999ce/pyasn1-0.6.4-py3-none-any.whl", hash = "sha256:deda9277cfd454080ec40b207fb6df82206a3a2688735233cdcd8d3d565f088b", size = 84410, upload-time = "2026-07-09T01:12:32.92Z" }, +] + +[[package]] +name = "rsa" +version = "4.9.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pyasn1" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/da/8a/22b7beea3ee0d44b1916c0c1cb0ee3af23b700b6da9f04991899d0c555d4/rsa-4.9.1.tar.gz", hash = "sha256:e7bdbfdb5497da4c07dfd35530e1a902659db6ff241e39d9953cad06ebd0ae75", size = 29034, upload-time = "2025-04-16T09:51:18.218Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/64/8d/0133e4eb4beed9e425d9a98ed6e081a55d195481b7632472be1af08d2f6b/rsa-4.9.1-py3-none-any.whl", hash = "sha256:68635866661c6836b8d39430f97a996acbd61bfa49406748ea243539fe239762", size = 34696, upload-time = "2025-04-16T09:51:17.142Z" }, +] + +[[package]] +name = "telethon" +version = "1.44.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pyaes" }, + { name = "rsa" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/49/e2/d4137e598db0ba50e4cf9bf32f2e26d016b2c4bb0e9763e3d5a07007d780/telethon-1.44.0.tar.gz", hash = "sha256:7de1473e9e412e20bd57b102d8f50d48372712462be9ff6c003bb81c50ad7e98", size = 701463, upload-time = "2026-06-15T15:47:39.44Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/82/fd/4ad621d7a4b8655dfc964ee1c1267407496ec6fd10c91901a64ea29c16c8/telethon-1.44.0-py3-none-any.whl", hash = "sha256:52fc49efb67a4916c2aedcb295ad286f4afa2aba9bf15d83ed2acdc64af0c718", size = 789786, upload-time = "2026-06-15T15:47:41.534Z" }, +] + +[[package]] +name = "tg-export" +version = "0.1.0" +source = { virtual = "." } +dependencies = [ + { name = "cryptg" }, + { name = "telethon" }, +] + +[package.metadata] +requires-dist = [ + { name = "cryptg", specifier = ">=0.6.0" }, + { name = "telethon", specifier = ">=1.44.0" }, +] diff --git a/stacks/rpi-tg-export/bin/tg-export b/stacks/rpi-tg-export/bin/tg-export new file mode 100755 index 0000000..481d2dd --- /dev/null +++ b/stacks/rpi-tg-export/bin/tg-export @@ -0,0 +1,9 @@ +#!/bin/sh +# tg-export КОМАНДА [опции export.py] — выгрузка второго Telegram-аккаунта владельца (infra stacks/rpi-tg-export). +# Запускать от tgexport: sudo -u tgexport tg-export list | login | run … +# Ключи приложения — /etc/tg-export/env (root:tgexport 640), сессия — /var/lib/tg-export/tg.session (tgexport 600), +# выгрузка — /mnt/backup/tg (группа astra, только чтение). +set -eu +set -a; . /etc/tg-export/env; set +a +exec /opt/tg-export/venv/bin/python /opt/tg-export/export.py "$@" \ + --session /var/lib/tg-export/tg.session --out /mnt/backup/tg diff --git a/stacks/rpi-tg-export/install.sh b/stacks/rpi-tg-export/install.sh new file mode 100755 index 0000000..5397785 --- /dev/null +++ b/stacks/rpi-tg-export/install.sh @@ -0,0 +1,17 @@ +#!/bin/sh +# Установка/обновление выгрузки Telegram на rpi: sudo sh install.sh (из копии каталога stacks/rpi-tg-export) +# Ключи (/etc/tg-export/env) и вход (sudo -u tgexport tg-export login) — отдельно, см. README. +set -eu +cd "$(dirname "$0")" +id tgexport >/dev/null 2>&1 || useradd --system --home-dir /var/lib/tg-export --shell /usr/sbin/nologin tgexport +install -d -m 755 /opt/tg-export +install -m 644 app/export.py app/fetch_public.py app/channel_comments.py app/pyproject.toml /opt/tg-export/ +[ -x /opt/tg-export/venv/bin/python ] || python3 -m venv /opt/tg-export/venv +/opt/tg-export/venv/bin/pip install -q --disable-pip-version-check 'telethon>=1.44.0' 'cryptg>=0.6.0' +install -m 755 bin/tg-export /usr/local/bin/tg-export +install -d -o root -g tgexport -m 750 /etc/tg-export +install -d -o tgexport -g tgexport -m 700 /var/lib/tg-export +install -d -o tgexport -g astra -m 2750 /mnt/backup/tg +install -m 644 systemd/tg-export.service systemd/tg-export.timer /etc/systemd/system/ +systemctl daemon-reload +echo "ok: код в /opt/tg-export; таймер включается после входа: systemctl enable --now tg-export.timer" diff --git a/stacks/rpi-tg-export/systemd/tg-export.service b/stacks/rpi-tg-export/systemd/tg-export.service new file mode 100644 index 0000000..6519359 --- /dev/null +++ b/stacks/rpi-tg-export/systemd/tg-export.service @@ -0,0 +1,17 @@ +[Unit] +Description=Telegram: докачка истории второго аккаунта владельца (только текст) +Wants=network-online.target +After=network-online.target +RequiresMountsFor=/mnt/backup + +[Service] +Type=oneshot +User=tgexport +Group=tgexport +# файлы выгрузки 640, каталог setgid astra — Astra читает, остальные пользователи малины нет +UMask=0027 +Nice=10 +# первый прогон долгий: Telegram троттлит чтение истории (FloodWait), скрипт ждёт сам +TimeoutStartSec=infinity +ExecStart=/usr/local/bin/tg-export run --no-media --archived +ExecStopPost=+/usr/local/bin/backup-metrics backup_tg_export diff --git a/stacks/rpi-tg-export/systemd/tg-export.timer b/stacks/rpi-tg-export/systemd/tg-export.timer new file mode 100644 index 0000000..b24a1cb --- /dev/null +++ b/stacks/rpi-tg-export/systemd/tg-export.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Telegram: докачка истории второго аккаунта раз в сутки + +[Timer] +OnCalendar=*-*-* 04:30 +RandomizedDelaySec=10m +Persistent=true + +[Install] +WantedBy=timers.target