rpi: выгрузка Telegram второго аккаунта (tg_export с Mac) — stacks/rpi-tg-export
Пользователь tgexport, сессия /var/lib/tg-export (600), тексты в /mnt/backup/tg (группа astra), докачка раз в сутки без медиа, итог в backup-metrics. Вход делает владелец. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
gilfanovr
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
8373a87826
commit
8a06244ef4
@@ -290,3 +290,10 @@ Docker CE 29.9 + Compose v5.6 из репо download.docker.com (в Debian ст
|
||||
- 2026-10-14 владелец опознал Onyx Boox: IP при обнаружении 192.168.10.146, MAC 00:0a:0a:fd:89:17, hostname android-dhcp-11. Ранее неизвестное устройство; через `router known add` записано имя «Onyx Boox Руслана» в known.json. Это только отметка реестра, DHCP и сеть не менялись.
|
||||
|
||||
- 2026-10-15 исправлено распознавание локальных вложений с пробелами вокруг Markdown-пути: прежняя ссылка `[PDF]( /tmp/...)` не вызывала отправку. astra-bot `dd79caf`, attachments.py установлен в /opt/astra-bot, прежний сохранён .bak-before-spaces; 133 теста прошли (включая пробелы и угловые скобки). PDF morphemes-20261015 заново доставлен владельцу штатным send-preview, Telegram подтвердил успех. Перезапуск с проверкой готовности запланирован через astra-attachment-space-restart на 60 секунд; остальные ограничения путей сохранены.
|
||||
|
||||
## Telegram: выгрузка второго аккаунта владельца (с 2026-10-17)
|
||||
|
||||
- Проект `tg_export` (Telethon) перенесён с Mac: код, юниты и установка — [`stacks/rpi-tg-export/`](../stacks/rpi-tg-export/README.md). На малине — **второй аккаунт владельца**, докачка **только текста** раз в сутки (`tg-export.timer`, 04:30, `run --no-media --archived`), выгрузка в `/mnt/backup/tg/` (tgexport:astra 2750/640 — Astra читает). Старая выгрузка первого аккаунта (~14 ГБ) — на Mac, не переносилась (решение владельца).
|
||||
- Отдельный системный пользователь `tgexport` (без шелла): сессия `/var/lib/tg-export/tg.session` (600) = полный доступ к аккаунту — Astra её не видит; ключи приложения `/etc/tg-export/env` (root:tgexport 640, те же `api_id`/`api_hash`, что на Mac). Вход — владелец: `ssh -t rpi sudo -u tgexport tg-export login`.
|
||||
- Мониторинг — `backup-metrics backup_tg_export` → `BackupFailed`/`BackupStale` (>30 ч). ⚠️ Первый прогон может идти дольше суток из-за FloodWait.
|
||||
- Доступ: тексты — только для владельца; Оле и Поле закрыто (правило в `codex/AGENTS.md` и скилл `tg`, astra-bot `27bf8e6`), как почта.
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
# Выгрузка Telegram — второй аккаунт владельца (rpi)
|
||||
|
||||
Проект `tg_export` (Telethon, MTProto) перенесён с Mac 2026-10-17: на малине — **второй аккаунт владельца**,
|
||||
регулярная докачка **только текста** (медиа — позже, если понадобится). Старая выгрузка первого аккаунта
|
||||
(~14 ГБ) осталась на Mac в `~/pets/tg_export/out`, сюда не переносилась; анализ (`stats`, `trips`, `presence`…) — там же.
|
||||
|
||||
| Что | Где |
|
||||
|---|---|
|
||||
| Код | `/opt/tg-export/` (`export.py`, `fetch_public.py`, `channel_comments.py`), venv `/opt/tg-export/venv` |
|
||||
| Обёртка | `/usr/local/bin/tg-export` — подставляет `--session` и `--out`, ключи из env |
|
||||
| Ключи приложения (`TG_API_ID`/`TG_API_HASH`, my.telegram.org) | `/etc/tg-export/env` — root:tgexport 640, в git нет |
|
||||
| Сессия (= **полный доступ к аккаунту**) | `/var/lib/tg-export/tg.session` — tgexport 600 |
|
||||
| Выгрузка | `/mnt/backup/tg/` — `dialogs.json`, `chats/<id>_<название>/{messages.jsonl,messages.txt,meta.json,_state.json}`; tgexport:astra, каталоги 2750, файлы 640 |
|
||||
| Таймер | `tg-export.timer` — 04:30 ±10 мин, `tg-export.service` (oneshot, `run --no-media --archived`) |
|
||||
| Мониторинг | `ExecStopPost=+backup-metrics backup_tg_export` → алерты `BackupFailed` / `BackupStale` (>30 ч) |
|
||||
|
||||
Пользователь `tgexport` — системный, без шелла: Astra (группа astra) читает тексты, но **не видит сессию и ключи**.
|
||||
|
||||
## Установка и вход
|
||||
|
||||
```sh
|
||||
sudo sh install.sh # код, venv, пользователь, каталоги, юниты
|
||||
sudo install -o root -g tgexport -m 640 env /etc/tg-export/env # ключи приложения (как .env.example)
|
||||
ssh -t rpi sudo -u tgexport tg-export login # номер, код из Telegram, 2FA — делает владелец
|
||||
sudo systemctl enable --now tg-export.timer
|
||||
```
|
||||
|
||||
Вручную: `sudo -u tgexport tg-export list` (диалоги), `sudo -u tgexport tg-export run --no-media --only @чат`.
|
||||
Повторный запуск догружает только новое (`_state.json` в каждом чате), `--full` — перезалить чат.
|
||||
|
||||
⚠️ Первый прогон долгий: Telegram троттлит массовое чтение (FloodWait, скрипт ждёт до 24 ч) — `BackupStale`
|
||||
может сработать, пока он идёт; следующий запуск таймера не стартует поверх идущего.
|
||||
⚠️ Медиа выключены (`--no-media`); включить — убрать флаг в юните (лимит по умолчанию 50 МБ на файл).
|
||||
@@ -0,0 +1,3 @@
|
||||
# https://my.telegram.org → API development tools → создать приложение
|
||||
TG_API_ID=1234567
|
||||
TG_API_HASH=0123456789abcdef0123456789abcdef
|
||||
@@ -0,0 +1,200 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Выгрузка канала вместе с комментариями из привязанной группы обсуждений.
|
||||
|
||||
Обычный `export.py run` берёт только посты канала: комментарии физически
|
||||
лежат в отдельном linked chat и через iter_dialogs не приходят. Здесь для
|
||||
каждого поста дополнительно вытягивается его ветка (iter_messages(reply_to=...)),
|
||||
плюс, при --with-linked, вся история группы обсуждений целиком — там бывают
|
||||
сообщения, не привязанные ни к одному посту.
|
||||
|
||||
uv run channel_comments.py @cpython_notes
|
||||
uv run channel_comments.py @cpython_notes --with-linked --session copy.session
|
||||
|
||||
Формат вывода совпадает с export.py: сырой to_dict() в *.jsonl, плоское
|
||||
представление в *_pretty.jsonl, читаемая расшифровка в *.txt.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import datetime as dt
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from telethon import TelegramClient, errors, functions
|
||||
|
||||
import export as ex
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
async def dump_stream(client, entity, it, jsonl, pretty, txt, cache, label, tag=None):
|
||||
"""Прогоняет итератор сообщений в три файла. Возвращает счётчик."""
|
||||
n = 0
|
||||
async for msg in it:
|
||||
if ex.STOP:
|
||||
break
|
||||
record = msg.to_dict()
|
||||
if tag:
|
||||
record.update(tag)
|
||||
sender = await ex.sender_label(client, msg, cache)
|
||||
jsonl.write(json.dumps(record, ensure_ascii=False, cls=ex.Encoder) + "\n")
|
||||
if pretty:
|
||||
flat = ex.simplify_message(msg, sender, record)
|
||||
if tag:
|
||||
flat.update(tag)
|
||||
pretty.write(json.dumps(flat, ensure_ascii=False, cls=ex.Encoder) + "\n")
|
||||
if txt:
|
||||
txt.write(ex.render_line(msg, sender))
|
||||
n += 1
|
||||
if n % 500 == 0:
|
||||
print(f" … {label}: {n}", flush=True)
|
||||
return n
|
||||
|
||||
|
||||
async def main_async(args) -> None:
|
||||
api_id, api_hash = ex.require_credentials()
|
||||
client = TelegramClient(args.session, api_id, api_hash)
|
||||
client.flood_sleep_threshold = 24 * 60 * 60
|
||||
await client.start()
|
||||
try:
|
||||
chan = await client.get_entity(args.channel)
|
||||
full = await client(functions.channels.GetFullChannelRequest(channel=chan))
|
||||
linked_id = getattr(full.full_chat, "linked_chat_id", None)
|
||||
linked = None
|
||||
if linked_id:
|
||||
linked = next((c for c in full.chats if c.id == linked_id), None)
|
||||
|
||||
dialog_id = int(f"-100{chan.id}")
|
||||
folder = Path(args.out) / "chats" / f"{dialog_id}_{ex.slugify(getattr(chan, 'title', args.channel))}"
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
header = {
|
||||
"id": dialog_id,
|
||||
"name": getattr(chan, "title", None),
|
||||
"kind": ex.dialog_kind(chan),
|
||||
"username": getattr(chan, "username", None),
|
||||
"participants": getattr(full.full_chat, "participants_count", None),
|
||||
"linked_chat_id": linked_id,
|
||||
"linked_chat_title": getattr(linked, "title", None) if linked else None,
|
||||
"exported_at": dt.datetime.now(dt.timezone.utc).isoformat(),
|
||||
}
|
||||
(folder / "meta.json").write_text(
|
||||
json.dumps({**header, "entity": chan.to_dict()},
|
||||
ensure_ascii=False, indent=2, cls=ex.Encoder),
|
||||
encoding="utf-8")
|
||||
|
||||
print(f"Канал: {header['name']} (@{header['username']}) · "
|
||||
f"подписчиков {header['participants']}")
|
||||
print(f"Группа обсуждений: {header['linked_chat_title']} (id={linked_id})\n"
|
||||
if linked_id else "Группа обсуждений не привязана — комментариев нет\n")
|
||||
|
||||
cache: dict = {}
|
||||
mode = "w" if args.full else "a"
|
||||
|
||||
# ---- посты канала ----
|
||||
pj = (folder / "messages.jsonl").open(mode, encoding="utf-8")
|
||||
pp = (folder / "_pretty.jsonl").open(mode, encoding="utf-8")
|
||||
pt = (folder / "messages.txt").open(mode, encoding="utf-8")
|
||||
posts = []
|
||||
async for m in client.iter_messages(chan, reverse=True):
|
||||
posts.append(m)
|
||||
for m in posts:
|
||||
record = m.to_dict()
|
||||
sender = await ex.sender_label(client, m, cache)
|
||||
pj.write(json.dumps(record, ensure_ascii=False, cls=ex.Encoder) + "\n")
|
||||
pp.write(json.dumps(ex.simplify_message(m, sender, record),
|
||||
ensure_ascii=False, cls=ex.Encoder) + "\n")
|
||||
pt.write(ex.render_line(m, sender))
|
||||
for f in (pj, pp, pt):
|
||||
f.close()
|
||||
print(f"постов: {len(posts)}")
|
||||
|
||||
if not linked_id:
|
||||
return
|
||||
|
||||
# ---- комментарии: по ветке на каждый пост ----
|
||||
cj = (folder / "comments.jsonl").open(mode, encoding="utf-8")
|
||||
cp = (folder / "comments_pretty.jsonl").open(mode, encoding="utf-8")
|
||||
thread = (folder / "threads.txt").open(mode, encoding="utf-8")
|
||||
total_c = 0
|
||||
with_c = 0
|
||||
for i, post in enumerate(posts, 1):
|
||||
if ex.STOP:
|
||||
break
|
||||
try:
|
||||
it = client.iter_messages(chan, reply_to=post.id, reverse=True)
|
||||
n = 0
|
||||
head_written = False
|
||||
async for cm in it:
|
||||
if not head_written:
|
||||
head = (post.message or "").split("\n")[0][:90]
|
||||
thread.write(f"\n{'='*78}\nПОСТ #{post.id} "
|
||||
f"[{post.date:%Y-%m-%d %H:%M}] {head}\n{'-'*78}\n")
|
||||
head_written = True
|
||||
record = cm.to_dict()
|
||||
record["_post_id"] = post.id
|
||||
sender = await ex.sender_label(client, cm, cache)
|
||||
cj.write(json.dumps(record, ensure_ascii=False, cls=ex.Encoder) + "\n")
|
||||
flat = ex.simplify_message(cm, sender, record)
|
||||
flat["_post_id"] = post.id
|
||||
cp.write(json.dumps(flat, ensure_ascii=False, cls=ex.Encoder) + "\n")
|
||||
thread.write(" " + ex.render_line(cm, sender).replace("\n", "\n ").rstrip() + "\n")
|
||||
n += 1
|
||||
except (errors.MsgIdInvalidError, errors.ChannelPrivateError) as exc:
|
||||
# у поста отключены комментарии либо ветка недоступна
|
||||
print(f" пост #{post.id}: {type(exc).__name__}", file=sys.stderr)
|
||||
continue
|
||||
except errors.FloodWaitError as exc:
|
||||
print(f" FloodWait {exc.seconds}s на посте #{post.id} — жду",
|
||||
file=sys.stderr)
|
||||
await asyncio.sleep(exc.seconds + 1)
|
||||
continue
|
||||
total_c += n
|
||||
if n:
|
||||
with_c += 1
|
||||
print(f" [{i}/{len(posts)}] пост #{post.id}: {n} комментариев", flush=True)
|
||||
for f in (cj, cp, thread):
|
||||
f.close()
|
||||
print(f"\nкомментариев: {total_c} в {with_c} ветках "
|
||||
f"(из {len(posts)} постов)")
|
||||
|
||||
# ---- вся группа обсуждений целиком ----
|
||||
if args.with_linked:
|
||||
lj = (folder / "linked_chat.jsonl").open(mode, encoding="utf-8")
|
||||
lp = (folder / "linked_chat_pretty.jsonl").open(mode, encoding="utf-8")
|
||||
lt = (folder / "linked_chat.txt").open(mode, encoding="utf-8")
|
||||
n = await dump_stream(
|
||||
client, linked, client.iter_messages(linked, reverse=True),
|
||||
lj, lp, lt, cache, "linked")
|
||||
for f in (lj, lp, lt):
|
||||
f.close()
|
||||
print(f"вся группа обсуждений: {n} сообщений")
|
||||
|
||||
print(f"\nКаталог: {folder}")
|
||||
finally:
|
||||
await client.disconnect()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ex.load_dotenv(ROOT / ".env")
|
||||
p = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("channel", help="@username, id или часть названия канала")
|
||||
p.add_argument("--session", default=str(ROOT / "tg_export.session"))
|
||||
p.add_argument("--out", default=str(ROOT / "out"))
|
||||
p.add_argument("--with-linked", action="store_true",
|
||||
help="дополнительно выгрузить всю группу обсуждений целиком")
|
||||
p.add_argument("--full", action="store_true", help="перезаписать файлы с нуля")
|
||||
args = p.parse_args()
|
||||
ex.install_signal_handlers()
|
||||
try:
|
||||
asyncio.run(main_async(args))
|
||||
except KeyboardInterrupt:
|
||||
sys.exit(130)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,901 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Полная выгрузка истории личного Telegram-аккаунта через Telethon.
|
||||
|
||||
Возможности:
|
||||
* логин по номеру телефона (MTProto, свой api_id/api_hash);
|
||||
* индекс всех диалогов (личные, боты, группы, каналы);
|
||||
* выгрузка всех сообщений в JSONL (сырой `to_dict()` — ничего не теряется)
|
||||
плюс опциональный человекочитаемый .txt;
|
||||
* скачивание медиа с лимитом на размер;
|
||||
* возобновление: повторный запуск догружает только новое;
|
||||
* аккуратная обработка FloodWait и приватных/недоступных чатов.
|
||||
|
||||
Команды:
|
||||
python export.py login — только авторизация (создаёт .session)
|
||||
python export.py list — вывести список диалогов
|
||||
python export.py run — выгрузить историю
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import base64
|
||||
import datetime as dt
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import signal
|
||||
import sys
|
||||
import unicodedata
|
||||
from pathlib import Path
|
||||
|
||||
from telethon import TelegramClient, errors, utils
|
||||
from telethon.tl import types
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# конфиг
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def load_dotenv(path: Path) -> None:
|
||||
"""Минимальный .env-лоадер, чтобы не тащить python-dotenv."""
|
||||
if not path.exists():
|
||||
return
|
||||
for raw in path.read_text(encoding="utf-8").splitlines():
|
||||
line = raw.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, _, val = line.partition("=")
|
||||
key, val = key.strip(), val.strip().strip("'\"")
|
||||
os.environ.setdefault(key, val)
|
||||
|
||||
|
||||
def require_credentials() -> tuple[int, str]:
|
||||
api_id = os.environ.get("TG_API_ID")
|
||||
api_hash = os.environ.get("TG_API_HASH")
|
||||
if not api_id or not api_hash:
|
||||
sys.exit(
|
||||
"Нет TG_API_ID / TG_API_HASH.\n"
|
||||
"Получи их на https://my.telegram.org → API development tools,\n"
|
||||
f"затем скопируй {ROOT / '.env.example'} в {ROOT / '.env'} и заполни."
|
||||
)
|
||||
try:
|
||||
return int(api_id), api_hash
|
||||
except ValueError:
|
||||
sys.exit(f"TG_API_ID должен быть числом, получено: {api_id!r}")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# утилиты
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
class Encoder(json.JSONEncoder):
|
||||
"""Сериализует всё, что встречается в Telethon-объектах."""
|
||||
|
||||
def default(self, o): # noqa: D102
|
||||
if isinstance(o, (dt.datetime, dt.date, dt.time)):
|
||||
return o.isoformat()
|
||||
if isinstance(o, dt.timedelta):
|
||||
return o.total_seconds()
|
||||
if isinstance(o, (bytes, bytearray, memoryview)):
|
||||
return base64.b64encode(bytes(o)).decode("ascii")
|
||||
if isinstance(o, set):
|
||||
return sorted(o, key=str)
|
||||
return str(o)
|
||||
|
||||
|
||||
_SLUG_BAD = re.compile(r"[^\w\-. ]+", re.UNICODE)
|
||||
|
||||
|
||||
def slugify(name: str, limit: int = 48) -> str:
|
||||
name = unicodedata.normalize("NFKC", name or "").strip()
|
||||
name = _SLUG_BAD.sub("", name)
|
||||
name = re.sub(r"\s+", "_", name).strip("._-")
|
||||
return (name[:limit] or "unnamed")
|
||||
|
||||
|
||||
def dialog_kind(entity) -> str:
|
||||
if isinstance(entity, types.User):
|
||||
return "bot" if getattr(entity, "bot", False) else "user"
|
||||
if isinstance(entity, types.Chat) or isinstance(entity, types.ChatForbidden):
|
||||
return "group"
|
||||
if isinstance(entity, types.Channel):
|
||||
return "supergroup" if getattr(entity, "megagroup", False) else "channel"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def human_size(n: int | None) -> str:
|
||||
if not n:
|
||||
return "0B"
|
||||
for unit in ("B", "K", "M", "G", "T"):
|
||||
if n < 1024 or unit == "T":
|
||||
return f"{n:.0f}{unit}" if unit == "B" else f"{n:.1f}{unit}"
|
||||
n /= 1024
|
||||
return f"{n:.1f}T"
|
||||
|
||||
|
||||
def parse_date(value: str | None) -> dt.datetime | None:
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
parsed = dt.datetime.fromisoformat(value)
|
||||
except ValueError:
|
||||
sys.exit(f"Не смог разобрать дату {value!r}. Формат: YYYY-MM-DD или YYYY-MM-DDTHH:MM")
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=dt.timezone.utc)
|
||||
return parsed
|
||||
|
||||
|
||||
STOP = False
|
||||
|
||||
|
||||
def install_signal_handlers() -> None:
|
||||
def handler(signum, frame): # noqa: ARG001
|
||||
global STOP
|
||||
if STOP:
|
||||
sys.exit(130)
|
||||
STOP = True
|
||||
print("\n[!] Останавливаюсь после текущего сообщения; прогресс сохранён. Ctrl-C ещё раз — выход.",
|
||||
file=sys.stderr)
|
||||
|
||||
signal.signal(signal.SIGINT, handler)
|
||||
signal.signal(signal.SIGTERM, handler)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# состояние (для возобновления)
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
class State:
|
||||
def __init__(self, path: Path):
|
||||
self.path = path
|
||||
self.data = {"last_id": 0, "count": 0, "media": 0, "updated_at": None}
|
||||
if path.exists():
|
||||
try:
|
||||
self.data.update(json.loads(path.read_text(encoding="utf-8")))
|
||||
except (ValueError, OSError):
|
||||
pass
|
||||
|
||||
@property
|
||||
def last_id(self) -> int:
|
||||
return int(self.data.get("last_id") or 0)
|
||||
|
||||
def bump(self, msg_id: int, media: int = 0) -> None:
|
||||
self.data["last_id"] = max(self.last_id, int(msg_id))
|
||||
self.data["count"] = int(self.data.get("count") or 0) + 1
|
||||
self.data["media"] = int(self.data.get("media") or 0) + media
|
||||
|
||||
def save(self) -> None:
|
||||
self.data["updated_at"] = dt.datetime.now(dt.timezone.utc).isoformat()
|
||||
tmp = self.path.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps(self.data, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
tmp.replace(self.path)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# текстовое представление
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
def render_line(msg, sender_name: str) -> str:
|
||||
when = msg.date.astimezone().strftime("%Y-%m-%d %H:%M:%S") if msg.date else "?"
|
||||
body = (msg.message or "").replace("\n", "\n ")
|
||||
tags = []
|
||||
if msg.media is not None and not isinstance(msg.media, types.MessageMediaWebPage):
|
||||
tags.append(type(msg.media).__name__.replace("MessageMedia", "").lower())
|
||||
if getattr(msg, "reply_to_msg_id", None):
|
||||
tags.append(f"reply→{msg.reply_to_msg_id}")
|
||||
if getattr(msg, "fwd_from", None):
|
||||
tags.append("forwarded")
|
||||
if getattr(msg, "action", None):
|
||||
tags.append(type(msg.action).__name__.replace("MessageAction", "action:"))
|
||||
suffix = f" [{', '.join(tags)}]" if tags else ""
|
||||
return f"[{when}] #{msg.id} {sender_name}:{suffix}\n {body}\n" if body else \
|
||||
f"[{when}] #{msg.id} {sender_name}:{suffix}\n"
|
||||
|
||||
|
||||
async def sender_label(client, msg, cache: dict) -> str:
|
||||
sid = msg.sender_id
|
||||
if sid is None:
|
||||
return "—"
|
||||
if sid in cache:
|
||||
return cache[sid]
|
||||
label = str(sid)
|
||||
try:
|
||||
sender = await msg.get_sender()
|
||||
if sender is not None:
|
||||
if isinstance(sender, types.User):
|
||||
parts = [sender.first_name or "", sender.last_name or ""]
|
||||
label = " ".join(p for p in parts if p) or (sender.username or str(sid))
|
||||
else:
|
||||
label = getattr(sender, "title", None) or str(sid)
|
||||
except (errors.RPCError, ValueError, TypeError):
|
||||
pass
|
||||
cache[sid] = label
|
||||
return label
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# читаемое представление (messages.json)
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
def media_info(msg, record: dict) -> dict | None:
|
||||
media = getattr(msg, "media", None)
|
||||
if media is None:
|
||||
return None
|
||||
|
||||
if isinstance(media, types.MessageMediaWebPage):
|
||||
url = getattr(getattr(media, "webpage", None), "url", None)
|
||||
return {"type": "link", "url": url} if url else None
|
||||
|
||||
info: dict = {"type": type(media).__name__.replace("MessageMedia", "").lower()}
|
||||
|
||||
handle = getattr(msg, "file", None)
|
||||
for key in ("name", "mime_type", "size", "duration"):
|
||||
val = getattr(handle, key, None)
|
||||
if val:
|
||||
info[key] = round(val, 1) if key == "duration" else val
|
||||
if info.get("size"):
|
||||
info["size_human"] = human_size(info["size"])
|
||||
|
||||
for attr in getattr(getattr(media, "document", None), "attributes", None) or []:
|
||||
if isinstance(attr, types.DocumentAttributeSticker):
|
||||
info["type"] = "sticker"
|
||||
info["emoji"] = attr.alt
|
||||
elif isinstance(attr, types.DocumentAttributeAudio) and attr.voice:
|
||||
info["type"] = "voice"
|
||||
|
||||
if isinstance(media, types.MessageMediaPoll):
|
||||
poll = media.poll
|
||||
info["question"] = getattr(poll.question, "text", poll.question)
|
||||
info["answers"] = [getattr(a.text, "text", a.text) for a in poll.answers]
|
||||
elif isinstance(media, (types.MessageMediaGeo, types.MessageMediaGeoLive)):
|
||||
geo = media.geo
|
||||
info["lat"], info["long"] = getattr(geo, "lat", None), getattr(geo, "long", None)
|
||||
elif isinstance(media, types.MessageMediaContact):
|
||||
info["contact"] = " ".join(
|
||||
p for p in (media.first_name, media.last_name, media.phone_number) if p
|
||||
)
|
||||
|
||||
for key in ("_media_file", "_media_skipped", "_media_error"):
|
||||
if key in record:
|
||||
info[key.lstrip("_").removeprefix("media_")] = record[key]
|
||||
return info
|
||||
|
||||
|
||||
def forward_info(msg) -> dict | None:
|
||||
fwd = getattr(msg, "fwd_from", None)
|
||||
if fwd is None:
|
||||
return None
|
||||
info: dict = {}
|
||||
if fwd.from_name:
|
||||
info["from_name"] = fwd.from_name
|
||||
if fwd.from_id is not None:
|
||||
try:
|
||||
info["from_id"] = utils.get_peer_id(fwd.from_id)
|
||||
except (TypeError, ValueError):
|
||||
info["from_id"] = str(fwd.from_id)
|
||||
if fwd.date:
|
||||
info["original_date"] = fwd.date.astimezone().isoformat()
|
||||
if fwd.post_author:
|
||||
info["post_author"] = fwd.post_author
|
||||
return info or None
|
||||
|
||||
|
||||
def reactions_info(msg) -> list | None:
|
||||
results = getattr(getattr(msg, "reactions", None), "results", None) or []
|
||||
out = []
|
||||
for res in results:
|
||||
emoji = getattr(res.reaction, "emoticon", None) \
|
||||
or getattr(res.reaction, "document_id", None)
|
||||
out.append({"reaction": str(emoji), "count": res.count})
|
||||
return out or None
|
||||
|
||||
|
||||
def simplify_message(msg, sender: str, record: dict) -> dict:
|
||||
"""Плоское, читаемое представление сообщения для messages.json."""
|
||||
out: dict = {
|
||||
"id": msg.id,
|
||||
"date": msg.date.astimezone().isoformat() if msg.date else None,
|
||||
"from": sender,
|
||||
"from_id": msg.sender_id,
|
||||
"outgoing": bool(getattr(msg, "out", False)),
|
||||
"text": getattr(msg, "message", "") or "",
|
||||
}
|
||||
if getattr(msg, "reply_to_msg_id", None):
|
||||
out["reply_to"] = msg.reply_to_msg_id
|
||||
if getattr(msg, "edit_date", None):
|
||||
out["edited"] = msg.edit_date.astimezone().isoformat()
|
||||
if fwd := forward_info(msg):
|
||||
out["forwarded"] = fwd
|
||||
if media := media_info(msg, record):
|
||||
out["media"] = media
|
||||
if reactions := reactions_info(msg):
|
||||
out["reactions"] = reactions
|
||||
|
||||
action = getattr(msg, "action", None)
|
||||
if action is not None:
|
||||
out["service_action"] = type(action).__name__.replace("MessageAction", "")
|
||||
if duration := getattr(action, "duration", None):
|
||||
out["service_action_duration_sec"] = duration
|
||||
if getattr(msg, "views", None):
|
||||
out["views"] = msg.views
|
||||
if getattr(msg, "via_bot_id", None):
|
||||
out["via_bot_id"] = msg.via_bot_id
|
||||
if getattr(msg, "pinned", False):
|
||||
out["pinned"] = True
|
||||
if getattr(msg, "grouped_id", None):
|
||||
out["album_id"] = str(msg.grouped_id)
|
||||
return out
|
||||
|
||||
|
||||
# медиа без файла: качать нечего, для них download_media вернёт None.
|
||||
# MessageMediaContact сюда НЕ входит — Telethon сохраняет контакт как .vcard.
|
||||
NO_FILE_MEDIA = frozenset({
|
||||
"MessageMediaWebPage", "MessageMediaGeo", "MessageMediaGeoLive",
|
||||
"MessageMediaPoll", "MessageMediaDice", "MessageMediaVenue",
|
||||
"MessageMediaGame", "MessageMediaUnsupported", "MessageMediaStory",
|
||||
"MessageMediaInvoice", "MessageMediaGiveaway",
|
||||
"MessageMediaGiveawayResults", "MessageMediaEmpty",
|
||||
})
|
||||
|
||||
|
||||
def raw_media_tags(media: dict) -> set[str]:
|
||||
"""Метки медиа из сырого to_dict(): тип, mime, расширение, вид документа.
|
||||
|
||||
Позволяет отсеивать по `--skip-media webp` / `sticker` / `video` ещё до
|
||||
запроса к API — по данным, которые уже лежат в messages.jsonl.
|
||||
"""
|
||||
if not media:
|
||||
return set()
|
||||
kind = media.get("_", "")
|
||||
tags = {kind.replace("MessageMedia", "").lower()}
|
||||
if kind == "MessageMediaPhoto":
|
||||
tags |= {"photo", "jpg", "jpeg", "image/jpeg"}
|
||||
|
||||
doc = media.get("document") or {}
|
||||
mime = (doc.get("mime_type") or "").lower()
|
||||
if mime:
|
||||
tags.add(mime)
|
||||
tags.add(mime.rsplit("/", 1)[-1])
|
||||
|
||||
for attr in doc.get("attributes") or []:
|
||||
name = attr.get("_", "")
|
||||
if name == "DocumentAttributeSticker":
|
||||
tags.add("sticker")
|
||||
elif name == "DocumentAttributeAudio":
|
||||
tags.add("voice" if attr.get("voice") else "audio")
|
||||
elif name == "DocumentAttributeVideo":
|
||||
tags.add("round" if attr.get("round_message") else "video")
|
||||
elif name == "DocumentAttributeAnimated":
|
||||
tags.add("gif")
|
||||
elif name == "DocumentAttributeFilename":
|
||||
filename = attr.get("file_name") or ""
|
||||
if "." in filename:
|
||||
tags.add(filename.rsplit(".", 1)[-1].lower())
|
||||
return tags
|
||||
|
||||
|
||||
def has_downloadable_media(msg) -> bool:
|
||||
media = getattr(msg, "media", None)
|
||||
return media is not None and type(media).__name__ not in NO_FILE_MEDIA
|
||||
|
||||
|
||||
def load_media_index(folder: Path) -> dict[str, dict]:
|
||||
path = folder / "media_index.json"
|
||||
if not path.exists():
|
||||
return {}
|
||||
try:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
except (ValueError, OSError):
|
||||
return {}
|
||||
|
||||
|
||||
def assemble_pretty(folder: Path, header: dict) -> int:
|
||||
"""Собирает messages.json из накопленного _pretty.jsonl (устойчиво к докачке)."""
|
||||
src = folder / "_pretty.jsonl"
|
||||
if not src.exists():
|
||||
return 0
|
||||
by_id: dict[int, dict] = {}
|
||||
with src.open(encoding="utf-8") as handle:
|
||||
for line in handle:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
by_id[rec["id"]] = rec
|
||||
|
||||
# подмешиваем результаты догрузки медиа, если она уже прошла
|
||||
for msg_id, entry in load_media_index(folder).items():
|
||||
rec = by_id.get(int(msg_id))
|
||||
if rec is None or "media" not in rec:
|
||||
continue
|
||||
# не ссылаемся на файлы, которых уже нет на диске (удалили вручную и т.п.)
|
||||
if "file" in entry and not (folder / entry["file"]).exists():
|
||||
continue
|
||||
rec["media"].update({k: v for k, v in entry.items() if v is not None})
|
||||
|
||||
messages = [by_id[key] for key in sorted(by_id)]
|
||||
payload = {
|
||||
"chat": header,
|
||||
"messages_count": len(messages),
|
||||
"first_message_date": messages[0]["date"] if messages else None,
|
||||
"last_message_date": messages[-1]["date"] if messages else None,
|
||||
"messages": messages,
|
||||
}
|
||||
(folder / "messages.json").write_text(
|
||||
json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
|
||||
)
|
||||
return len(messages)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# выгрузка одного диалога
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
async def export_dialog(client, dialog, out_root: Path, args) -> dict:
|
||||
entity = dialog.entity
|
||||
kind = dialog_kind(entity)
|
||||
folder = out_root / "chats" / f"{dialog.id}_{slugify(dialog.name)}"
|
||||
media_dir = folder / "media"
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
state = State(folder / "_state.json")
|
||||
start_from = 0 if args.full else state.last_id
|
||||
|
||||
header = {
|
||||
"id": dialog.id,
|
||||
"name": dialog.name,
|
||||
"kind": kind,
|
||||
"username": getattr(entity, "username", None),
|
||||
"phone": getattr(entity, "phone", None),
|
||||
"is_archived": bool(getattr(dialog, "archived", False)),
|
||||
"exported_at": dt.datetime.now(dt.timezone.utc).isoformat(),
|
||||
}
|
||||
(folder / "meta.json").write_text(
|
||||
json.dumps({**header, "entity": entity.to_dict()},
|
||||
ensure_ascii=False, indent=2, cls=Encoder),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
excluded_types = {t.lower().lstrip(".") for t in args.skip_media}
|
||||
mode = "w" if args.full else "a"
|
||||
jsonl = (folder / "messages.jsonl").open(mode, encoding="utf-8")
|
||||
txt = (folder / "messages.txt").open(mode, encoding="utf-8") if args.txt else None
|
||||
pretty = (folder / "_pretty.jsonl").open(mode, encoding="utf-8") if args.pretty else None
|
||||
cache: dict = {}
|
||||
new_msgs = 0
|
||||
new_media = 0
|
||||
media_bytes = 0
|
||||
errors_count = 0
|
||||
|
||||
try:
|
||||
async for msg in client.iter_messages(
|
||||
entity,
|
||||
reverse=True, # от старых к новым — так работает докачка
|
||||
offset_id=start_from,
|
||||
offset_date=args._since,
|
||||
limit=args.limit,
|
||||
wait_time=args.wait or None,
|
||||
):
|
||||
if STOP:
|
||||
break
|
||||
if args._until and msg.date and msg.date > args._until:
|
||||
break
|
||||
|
||||
record = msg.to_dict()
|
||||
|
||||
if args.media and has_downloadable_media(msg):
|
||||
size = getattr(getattr(msg, "file", None), "size", None) or 0
|
||||
hit = raw_media_tags(record.get("media") or {}) & excluded_types
|
||||
if hit:
|
||||
record["_media_skipped"] = f"type_excluded:{','.join(sorted(hit))}"
|
||||
elif args.max_media_mb and size > args.max_media_mb * 1024 * 1024:
|
||||
record["_media_skipped"] = f"too_large:{human_size(size)}"
|
||||
else:
|
||||
media_dir.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
path = await client.download_media(msg, file=str(media_dir / f"{msg.id}"))
|
||||
if path:
|
||||
record["_media_file"] = str(Path(path).relative_to(folder))
|
||||
new_media += 1
|
||||
media_bytes += size or Path(path).stat().st_size
|
||||
except errors.FloodWaitError as exc:
|
||||
print(f" FloodWait {exc.seconds}s на медиа #{msg.id} — жду", file=sys.stderr)
|
||||
await asyncio.sleep(exc.seconds + 1)
|
||||
except (errors.RPCError, OSError, ValueError) as exc:
|
||||
record["_media_error"] = f"{type(exc).__name__}: {exc}"
|
||||
errors_count += 1
|
||||
|
||||
jsonl.write(json.dumps(record, ensure_ascii=False, cls=Encoder) + "\n")
|
||||
|
||||
label = await sender_label(client, msg, cache) if (txt or pretty) else ""
|
||||
if txt is not None:
|
||||
txt.write(render_line(msg, label))
|
||||
if pretty is not None:
|
||||
pretty.write(json.dumps(simplify_message(msg, label, record),
|
||||
ensure_ascii=False, cls=Encoder) + "\n")
|
||||
|
||||
state.bump(msg.id, media=1 if "_media_file" in record else 0)
|
||||
new_msgs += 1
|
||||
if new_msgs % 500 == 0:
|
||||
jsonl.flush()
|
||||
for stream in (txt, pretty):
|
||||
if stream is not None:
|
||||
stream.flush()
|
||||
state.save()
|
||||
print(f" … {new_msgs} новых (всего {state.data['count']})", flush=True)
|
||||
finally:
|
||||
jsonl.close()
|
||||
for stream in (txt, pretty):
|
||||
if stream is not None:
|
||||
stream.close()
|
||||
state.save()
|
||||
if args.pretty:
|
||||
assemble_pretty(folder, header)
|
||||
|
||||
return {
|
||||
"id": dialog.id,
|
||||
"name": dialog.name,
|
||||
"kind": kind,
|
||||
"new_messages": new_msgs,
|
||||
"total_messages": state.data["count"],
|
||||
"new_media": new_media,
|
||||
"media_bytes": media_bytes,
|
||||
"media_errors": errors_count,
|
||||
"folder": str(folder.relative_to(out_root)),
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# отбор диалогов
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
def dialog_matches(dialog, args) -> bool:
|
||||
kind = dialog_kind(dialog.entity)
|
||||
if args.kinds and kind not in args.kinds:
|
||||
return False
|
||||
if not args.archived and getattr(dialog, "archived", False):
|
||||
return False
|
||||
|
||||
if args.skip:
|
||||
for needle in args.skip:
|
||||
if _needle_hits(dialog, needle):
|
||||
return False
|
||||
if args.only:
|
||||
return any(_needle_hits(dialog, needle) for needle in args.only)
|
||||
return True
|
||||
|
||||
|
||||
def _needle_hits(dialog, needle: str) -> bool:
|
||||
needle = needle.strip()
|
||||
if not needle:
|
||||
return False
|
||||
if needle.lstrip("-").isdigit() and int(needle) == dialog.id:
|
||||
return True
|
||||
username = getattr(dialog.entity, "username", None) or ""
|
||||
if needle.lstrip("@").lower() == username.lower() and username:
|
||||
return True
|
||||
return needle.lower() in (dialog.name or "").lower()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# команды
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
async def cmd_login(client, args) -> None: # noqa: ARG001
|
||||
me = await client.get_me()
|
||||
print(f"OK: {me.first_name or ''} {me.last_name or ''} "
|
||||
f"(@{me.username or '—'}, id={me.id}). Сессия: {args.session}")
|
||||
|
||||
|
||||
async def cmd_list(client, args) -> None:
|
||||
rows = []
|
||||
async for dialog in client.iter_dialogs():
|
||||
if not dialog_matches(dialog, args):
|
||||
continue
|
||||
rows.append((dialog.id, dialog_kind(dialog.entity), dialog.name or "",
|
||||
getattr(dialog.entity, "username", None) or "",
|
||||
getattr(dialog, "unread_count", 0)))
|
||||
width = max((len(r[2]) for r in rows), default=10)
|
||||
print(f"{'ID':>15} {'ТИП':<10} {'НАЗВАНИЕ':<{min(width, 50)}} USERNAME")
|
||||
for did, kind, name, username, _ in rows:
|
||||
print(f"{did:>15} {kind:<10} {name[:50]:<{min(width, 50)}} "
|
||||
f"{'@' + username if username else ''}")
|
||||
print(f"\nвсего: {len(rows)}")
|
||||
|
||||
|
||||
async def cmd_media(client, args) -> None:
|
||||
"""Догружает медиа к уже выгруженным сообщениям.
|
||||
|
||||
Идёт не по API-истории, а по локальному messages.jsonl, поэтому:
|
||||
* не зависит от прогресса `run` и не мешает ему;
|
||||
* инкрементальна — уже скачанное фиксируется в media_index.json;
|
||||
* может гоняться повторно, пока `run` дописывает историю.
|
||||
"""
|
||||
out_root = Path(args.out).expanduser().resolve()
|
||||
folders = sorted(p for p in (out_root / "chats").glob("*") if p.is_dir())
|
||||
if not folders:
|
||||
sys.exit(f"В {out_root / 'chats'} нет выгруженных чатов — сначала запусти `run`.")
|
||||
|
||||
limit_bytes = int(args.max_media_mb * 1024 * 1024) if args.max_media_mb else 0
|
||||
grand_new = grand_bytes = grand_skip = grand_err = 0
|
||||
|
||||
for folder in folders:
|
||||
try:
|
||||
header = json.loads((folder / "meta.json").read_text(encoding="utf-8"))
|
||||
except (ValueError, OSError):
|
||||
print(f"[!] {folder.name}: нет читаемого meta.json — пропуск", file=sys.stderr)
|
||||
continue
|
||||
|
||||
needles = args.only or []
|
||||
if needles and not any(
|
||||
n.lstrip("@").lower() in ((header.get("username") or "").lower(),
|
||||
str(header.get("id")))
|
||||
or n.lower() in (header.get("name") or "").lower()
|
||||
for n in needles
|
||||
):
|
||||
continue
|
||||
|
||||
raw = folder / "messages.jsonl"
|
||||
if not raw.exists():
|
||||
continue
|
||||
|
||||
index = load_media_index(folder)
|
||||
excluded = {t.lower().lstrip(".") for t in args.skip_media}
|
||||
todo: list[int] = []
|
||||
by_type_excluded = 0
|
||||
with raw.open(encoding="utf-8") as handle:
|
||||
for line in handle:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except ValueError:
|
||||
continue # хвост может быть недописан, если `run` ещё идёт
|
||||
media = rec.get("media")
|
||||
if not media or media.get("_") in NO_FILE_MEDIA:
|
||||
index.pop(str(rec["id"]), None) # чистим ложные nothing_to_download
|
||||
continue
|
||||
if excluded and (raw_media_tags(media) & excluded):
|
||||
index.pop(str(rec["id"]), None) # забываем то, что решили не хранить
|
||||
by_type_excluded += 1
|
||||
continue
|
||||
if rec.get("_media_file"):
|
||||
continue
|
||||
# готово только то, у чего реально есть файл: записи skipped/error
|
||||
# переспрашиваются — иначе поднятый --max-media-mb ничего не даст
|
||||
if "file" in index.get(str(rec["id"]), {}):
|
||||
continue
|
||||
todo.append(rec["id"])
|
||||
|
||||
print(f"\n{folder.name}: к догрузке {len(todo)} медиа "
|
||||
f"(уже есть {len(index)}"
|
||||
+ (f", исключено по типу {by_type_excluded}" if by_type_excluded else "")
|
||||
+ ")", flush=True)
|
||||
if not todo:
|
||||
continue
|
||||
|
||||
entity = await client.get_entity(int(header["id"]))
|
||||
media_dir = folder / "media"
|
||||
media_dir.mkdir(parents=True, exist_ok=True)
|
||||
sem = asyncio.Semaphore(max(1, args.jobs))
|
||||
done = new = skipped = failed = 0
|
||||
got_bytes = 0
|
||||
|
||||
async def fetch(msg) -> None:
|
||||
nonlocal done, new, skipped, failed, got_bytes
|
||||
async with sem:
|
||||
if msg is None or STOP:
|
||||
return
|
||||
size = getattr(getattr(msg, "file", None), "size", None) or 0
|
||||
if limit_bytes and size > limit_bytes:
|
||||
index[str(msg.id)] = {"skipped": f"too_large:{human_size(size)}"}
|
||||
skipped += 1
|
||||
else:
|
||||
try:
|
||||
path = await client.download_media(
|
||||
msg, file=str(media_dir / f"{msg.id}"))
|
||||
if path:
|
||||
index[str(msg.id)] = {
|
||||
"file": str(Path(path).relative_to(folder))}
|
||||
new += 1
|
||||
got_bytes += size or Path(path).stat().st_size
|
||||
else:
|
||||
index[str(msg.id)] = {"error": "nothing_to_download"}
|
||||
failed += 1
|
||||
except errors.FloodWaitError as exc:
|
||||
print(f" FloodWait {exc.seconds}s — жду", file=sys.stderr, flush=True)
|
||||
await asyncio.sleep(exc.seconds + 1)
|
||||
return
|
||||
except (errors.RPCError, OSError, ValueError) as exc:
|
||||
index[str(msg.id)] = {"error": f"{type(exc).__name__}: {exc}"}
|
||||
failed += 1
|
||||
done += 1
|
||||
if done % 200 == 0:
|
||||
(folder / "media_index.json").write_text(
|
||||
json.dumps(index, ensure_ascii=False, indent=1), encoding="utf-8")
|
||||
print(f" … {done}/{len(todo)} +{new} файлов "
|
||||
f"({human_size(got_bytes)}), крупных пропущено {skipped}, "
|
||||
f"ошибок {failed}", flush=True)
|
||||
|
||||
for start in range(0, len(todo), 100):
|
||||
if STOP:
|
||||
break
|
||||
chunk = todo[start:start + 100]
|
||||
try:
|
||||
msgs = await client.get_messages(entity, ids=chunk)
|
||||
except errors.FloodWaitError as exc:
|
||||
print(f" FloodWait {exc.seconds}s на выборке — жду", file=sys.stderr, flush=True)
|
||||
await asyncio.sleep(exc.seconds + 1)
|
||||
msgs = await client.get_messages(entity, ids=chunk)
|
||||
await asyncio.gather(*(fetch(m) for m in msgs))
|
||||
|
||||
(folder / "media_index.json").write_text(
|
||||
json.dumps(index, ensure_ascii=False, indent=1), encoding="utf-8")
|
||||
if args.pretty:
|
||||
assemble_pretty(folder, header)
|
||||
print(f" итого: +{new} файлов ({human_size(got_bytes)}), "
|
||||
f"крупных пропущено {skipped}, ошибок {failed}", flush=True)
|
||||
grand_new += new
|
||||
grand_bytes += got_bytes
|
||||
grand_skip += skipped
|
||||
grand_err += failed
|
||||
|
||||
print(f"\nМедиа{' (прервано)' if STOP else ''}: +{grand_new} файлов "
|
||||
f"({human_size(grand_bytes)}), пропущено по размеру {grand_skip}, "
|
||||
f"ошибок {grand_err}.")
|
||||
|
||||
|
||||
async def cmd_run(client, args) -> None:
|
||||
out_root = Path(args.out).expanduser().resolve()
|
||||
(out_root / "chats").mkdir(parents=True, exist_ok=True)
|
||||
|
||||
dialogs = []
|
||||
async for dialog in client.iter_dialogs():
|
||||
if dialog_matches(dialog, args):
|
||||
dialogs.append(dialog)
|
||||
|
||||
me = await client.get_me()
|
||||
print(f"Аккаунт: @{me.username or me.id} · диалогов к выгрузке: {len(dialogs)} · вывод: {out_root}\n")
|
||||
|
||||
index, summary = [], []
|
||||
for n, dialog in enumerate(dialogs, 1):
|
||||
if STOP:
|
||||
break
|
||||
kind = dialog_kind(dialog.entity)
|
||||
print(f"[{n}/{len(dialogs)}] {kind:<10} {dialog.name} (id={dialog.id})", flush=True)
|
||||
try:
|
||||
result = await export_dialog(client, dialog, out_root, args)
|
||||
except errors.FloodWaitError as exc:
|
||||
print(f" FloodWait {exc.seconds}s — жду и повторяю", file=sys.stderr)
|
||||
await asyncio.sleep(exc.seconds + 1)
|
||||
result = await export_dialog(client, dialog, out_root, args)
|
||||
except (errors.ChannelPrivateError, errors.ChatAdminRequiredError,
|
||||
errors.UserBannedInChannelError) as exc:
|
||||
print(f" пропуск: {type(exc).__name__}", file=sys.stderr)
|
||||
result = {"id": dialog.id, "name": dialog.name, "kind": kind,
|
||||
"error": type(exc).__name__}
|
||||
except errors.RPCError as exc:
|
||||
print(f" ошибка: {type(exc).__name__}: {exc}", file=sys.stderr)
|
||||
result = {"id": dialog.id, "name": dialog.name, "kind": kind,
|
||||
"error": f"{type(exc).__name__}: {exc}"}
|
||||
|
||||
index.append(result)
|
||||
summary.append(result)
|
||||
if "error" not in result:
|
||||
print(f" +{result['new_messages']} сообщений, +{result['new_media']} медиа "
|
||||
f"({human_size(result['media_bytes'])}), всего {result['total_messages']}",
|
||||
flush=True)
|
||||
|
||||
(out_root / "dialogs.json").write_text(
|
||||
json.dumps(index, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
total_msgs = sum(r.get("new_messages", 0) for r in summary)
|
||||
total_media = sum(r.get("new_media", 0) for r in summary)
|
||||
total_bytes = sum(r.get("media_bytes", 0) for r in summary)
|
||||
failed = [r for r in summary if "error" in r]
|
||||
print(f"\nГотово{' (прервано)' if STOP else ''}: "
|
||||
f"{len(summary)} диалогов, +{total_msgs} сообщений, "
|
||||
f"+{total_media} медиа ({human_size(total_bytes)}).")
|
||||
if failed:
|
||||
print(f"Недоступны ({len(failed)}): " +
|
||||
", ".join(f"{r['name']} [{r['error']}]" for r in failed[:10]) +
|
||||
(" …" if len(failed) > 10 else ""))
|
||||
print(f"Каталог: {out_root}")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# CLI
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
KINDS = ("user", "bot", "group", "supergroup", "channel")
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(
|
||||
description="Выгрузка истории личного Telegram через Telethon",
|
||||
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
|
||||
)
|
||||
p.add_argument("command", choices=("login", "list", "run", "media"))
|
||||
p.add_argument("--session", default=str(ROOT / "tg_export.session"),
|
||||
help="файл сессии Telethon")
|
||||
p.add_argument("--out", default=str(ROOT / "out"), help="каталог для выгрузки")
|
||||
|
||||
sel = p.add_argument_group("отбор диалогов")
|
||||
sel.add_argument("--kinds", nargs="*", choices=KINDS, default=list(KINDS),
|
||||
help="какие типы чатов выгружать")
|
||||
sel.add_argument("--only", nargs="*", default=[], metavar="NEEDLE",
|
||||
help="только эти чаты: id, @username или часть названия")
|
||||
sel.add_argument("--skip", nargs="*", default=[], metavar="NEEDLE",
|
||||
help="исключить чаты (id / @username / часть названия)")
|
||||
sel.add_argument("--archived", action="store_true", help="включить архив")
|
||||
|
||||
rng = p.add_argument_group("границы истории")
|
||||
rng.add_argument("--since", metavar="YYYY-MM-DD", help="только сообщения не раньше даты")
|
||||
rng.add_argument("--until", metavar="YYYY-MM-DD", help="только сообщения не позже даты")
|
||||
rng.add_argument("--limit", type=int, default=None,
|
||||
help="максимум сообщений на чат (для пробного прогона)")
|
||||
rng.add_argument("--full", action="store_true",
|
||||
help="игнорировать сохранённый прогресс и перезаписать всё с нуля")
|
||||
|
||||
med = p.add_argument_group("медиа и вывод")
|
||||
med.add_argument("--media", action=argparse.BooleanOptionalAction, default=True,
|
||||
help="скачивать медиафайлы")
|
||||
med.add_argument("--max-media-mb", type=float, default=50,
|
||||
help="не качать файлы крупнее (0 — без лимита)")
|
||||
med.add_argument("--skip-media", nargs="*", default=[], metavar="TAG",
|
||||
help="не качать медиа с такими метками: расширение (webp, mp4), "
|
||||
"mime (image/webp) или вид (sticker, video, voice, round, gif)")
|
||||
med.add_argument("--pretty", action=argparse.BooleanOptionalAction, default=True,
|
||||
help="писать читаемый messages.json (плоские поля, отступы)")
|
||||
med.add_argument("--txt", action=argparse.BooleanOptionalAction, default=True,
|
||||
help="дополнительно писать читаемый messages.txt")
|
||||
med.add_argument("--wait", type=float, default=0.0,
|
||||
help="пауза между запросами истории, сек (мягче к лимитам)")
|
||||
med.add_argument("--jobs", type=int, default=4,
|
||||
help="параллельных загрузок в команде media")
|
||||
return p
|
||||
|
||||
|
||||
async def main_async(args) -> None:
|
||||
api_id, api_hash = require_credentials()
|
||||
client = TelegramClient(args.session, api_id, api_hash)
|
||||
client.flood_sleep_threshold = 24 * 60 * 60 # сам ждёт длинные FloodWait
|
||||
|
||||
await client.start(
|
||||
phone=lambda: input("Номер телефона (в формате +7...): ").strip(),
|
||||
password=lambda: input("Пароль двухфакторной аутентификации: ").strip(),
|
||||
)
|
||||
try:
|
||||
handler = {"login": cmd_login, "list": cmd_list,
|
||||
"run": cmd_run, "media": cmd_media}[args.command]
|
||||
await handler(client, args)
|
||||
finally:
|
||||
await client.disconnect()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
load_dotenv(ROOT / ".env")
|
||||
args = build_parser().parse_args()
|
||||
args._since = parse_date(args.since)
|
||||
args._until = parse_date(args.until)
|
||||
if args.max_media_mb <= 0:
|
||||
args.max_media_mb = 0
|
||||
install_signal_handlers()
|
||||
try:
|
||||
asyncio.run(main_async(args))
|
||||
except (errors.PhoneNumberInvalidError, errors.SessionPasswordNeededError) as exc:
|
||||
sys.exit(f"Ошибка авторизации: {type(exc).__name__}: {exc}")
|
||||
except KeyboardInterrupt:
|
||||
sys.exit(130)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Выгрузка публичной группы/канала, в которой мы не состоим.
|
||||
|
||||
`export.py run` ходит по iter_dialogs и потому видит только свои чаты.
|
||||
Здесь сущность резолвится по @username и история читается напрямую.
|
||||
|
||||
uv run fetch_public.py @cyschools --since 2026-03-04
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import datetime as dt
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from telethon import TelegramClient
|
||||
|
||||
import export as ex
|
||||
|
||||
ex.load_dotenv(ex.ROOT / ".env")
|
||||
|
||||
|
||||
async def main_async(args) -> None:
|
||||
api_id, api_hash = ex.require_credentials()
|
||||
client = TelegramClient(args.session, api_id, api_hash)
|
||||
client.flood_sleep_threshold = 24 * 60 * 60
|
||||
await client.start()
|
||||
try:
|
||||
entity = await client.get_entity(args.chat)
|
||||
dialog_id = int(f"-100{entity.id}")
|
||||
folder = Path(args.out) / "chats" / f"{dialog_id}_{ex.slugify(getattr(entity, 'title', args.chat))}"
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
since = dt.datetime.fromisoformat(args.since).replace(tzinfo=dt.timezone.utc)
|
||||
(folder / "meta.json").write_text(json.dumps({
|
||||
"id": dialog_id,
|
||||
"name": getattr(entity, "title", None),
|
||||
"kind": ex.dialog_kind(entity),
|
||||
"username": getattr(entity, "username", None),
|
||||
"fetched_since": args.since,
|
||||
}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
cache: dict = {}
|
||||
n = 0
|
||||
with (folder / "messages.jsonl").open("w", encoding="utf-8") as jsonl, \
|
||||
(folder / "messages_pretty.jsonl").open("w", encoding="utf-8") as pretty:
|
||||
async for msg in client.iter_messages(entity, offset_date=since, reverse=True):
|
||||
record = msg.to_dict()
|
||||
sender = await ex.sender_label(client, msg, cache)
|
||||
jsonl.write(json.dumps(record, ensure_ascii=False, cls=ex.Encoder) + "\n")
|
||||
pretty.write(json.dumps(ex.simplify_message(msg, sender, record),
|
||||
ensure_ascii=False, cls=ex.Encoder) + "\n")
|
||||
n += 1
|
||||
if n % 1000 == 0:
|
||||
print(f" … {n} ({msg.date:%Y-%m-%d})", flush=True)
|
||||
print(f"{folder.name}: {n} сообщений")
|
||||
finally:
|
||||
await client.disconnect()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("chat")
|
||||
p.add_argument("--since", required=True)
|
||||
p.add_argument("--out", default=str(ex.ROOT / "out"))
|
||||
p.add_argument("--session", default=str(ex.ROOT / "tg_export.session"))
|
||||
asyncio.run(main_async(p.parse_args()))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,12 @@
|
||||
[project]
|
||||
name = "tg-export"
|
||||
version = "0.1.0"
|
||||
requires-python = ">=3.13"
|
||||
description = "Полная выгрузка истории личного Telegram через Telethon"
|
||||
dependencies = [
|
||||
"cryptg>=0.6.0",
|
||||
"telethon>=1.44.0",
|
||||
]
|
||||
|
||||
[tool.uv]
|
||||
package = false
|
||||
Generated
+86
@@ -0,0 +1,86 @@
|
||||
version = 1
|
||||
revision = 3
|
||||
requires-python = ">=3.13"
|
||||
|
||||
[[package]]
|
||||
name = "cryptg"
|
||||
version = "0.6.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/9a/cc/3cc53619c07ab1def2f0ce7a958c10528c947e973821234714cfc61d7dd7/cryptg-0.6.0.tar.gz", hash = "sha256:f8fd59fea56398ab812f218c044b42fbbaeee8a9ae32771d3108a3cf210781aa", size = 15019, upload-time = "2026-04-12T13:31:09.978Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/c8/77/b56510a48a5c8f3cd38a0a2f3b93a6eee9e1e27ca3b1b8a3ad4a323f8311/cryptg-0.6.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:0ad5ccdd23d95f857c90e4109a7a8bd21cd998b342d8447d69881125f72b337e", size = 208770, upload-time = "2026-04-12T18:33:51.402Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/80/92/2780c8f178396c64e1a8bf350d68e66a08d801cd0ba355e81f8d10e58bee/cryptg-0.6.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:2d5a99c8c3897930974f47bbfe75ab3304606a1d98f66b0d35d9dafdd98e0512", size = 238481, upload-time = "2026-04-12T18:35:15.194Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/aa/db/84a3375d757c55f8c0e6d471828560a0f5bd6e8b2ecdf94cee44c1657c3d/cryptg-0.6.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:28497a18178b09f39b3d5faeaa34235ac1762b74017fbeac7bc34724c8efe0a0", size = 250130, upload-time = "2026-04-12T18:35:34.83Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/52/1a/28f81903ac4c0a1db0f17b1774e67bcc64733d89188f21c10d3b19aef396/cryptg-0.6.0-cp313-cp313-win32.whl", hash = "sha256:c1bc4edee45eda7dd5963420d578bbf86f0a7e2cbbf0d25d5a812b4293a223f0", size = 110091, upload-time = "2026-04-12T18:36:01.976Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/24/c3/742794894911bac8d83971b4a9ce742a0ca3b64d13ee2a64d2bb3cf5f01e/cryptg-0.6.0-cp313-cp313-win_amd64.whl", hash = "sha256:2b11d6ac297a63ba413cb79c97030cfb93763f041fd85755a488e0db13699336", size = 109113, upload-time = "2026-04-12T18:36:03.613Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a9/88/51f40b7991d77fab8d307fbe2b7e28dbef6d499141f953b0ade73faa2ab7/cryptg-0.6.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e726a6a40967d91ad1eaa0f6245cc79035b528c059a22de34aca18bfb61efd2e", size = 208094, upload-time = "2026-04-12T18:33:52.905Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/17/71/16f4b747d9be7c363be16717d4c2094d710f09acce15efea39e21aaff9a7/cryptg-0.6.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:3ccaa95f9ed6e6457b80ca003c99ee035c072dc421636b2482d069fc575ab5ba", size = 236844, upload-time = "2026-04-12T18:35:17.103Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/25/ae/8af9ebfa5aaca1da3d2ab8520b886b41194be1019e9cb414c5c4c492afaa/cryptg-0.6.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:c1931178451aa59da98294973bf6916115572a412514960d8fd8afb7919fb7fd", size = 250346, upload-time = "2026-04-12T18:35:36.593Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d8/d9/0214b6bc2b153abf91359de1a6035e59832e7377152de8749b378e6eebb5/cryptg-0.6.0-cp313-cp313t-win32.whl", hash = "sha256:0c312d2c0cc592cbf7bb08c1374f3ab27bcb01f46739b3ba23a4108f4f48661d", size = 109723, upload-time = "2026-04-12T18:35:58.77Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b0/89/133592fdb0859c639b5e62f5dfa24de4474b2e4e223668210e063592ddc8/cryptg-0.6.0-cp313-cp313t-win_amd64.whl", hash = "sha256:8748fe0ba8cce4268b2f3570f9fc8cd5e7d1730598174e271f525a93d5f64c90", size = 108149, upload-time = "2026-04-12T18:36:00.377Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/68/ca/740ba5201f47d50cd24809719ddff38b8a5d950f9921e426a0d2990de084/cryptg-0.6.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:7eb4e335ac3df00333c202cf98e65edbecd8983c9a639c11d93f1f369ae64e05", size = 208586, upload-time = "2026-04-12T18:33:54.23Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ab/d0/cfc3664d4b8c57f2058278472de7052719d2e37aece5591512d07e14777e/cryptg-0.6.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:28f6ca127a907d00ad075252e8d970bbac55e7f3df0f66a277f52b986fab0a57", size = 238396, upload-time = "2026-04-12T18:35:19.201Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/16/59/abc03cb099cec7b5e28866a8401040b40e99c719f3bef49fa7a0b302e09a/cryptg-0.6.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:4127534527ff741026b981359083da660010ddd556eb5e0f22523b9ce0fe80a2", size = 250040, upload-time = "2026-04-12T18:35:38.802Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/8a/6e/707de2c52842f093ab05811d002214e284c50f18057160352d7baa4ef1fa/cryptg-0.6.0-cp314-cp314-win32.whl", hash = "sha256:9c823319f6ff00c3c04be00cad78bd6555ccfc832fffc6b709eb983bbe0182b0", size = 112774, upload-time = "2026-04-12T18:36:08.654Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d2/49/6d8c6313789c3649f089a5f277369013072a5f8d96ba93d8c83def9c4a75/cryptg-0.6.0-cp314-cp314-win_amd64.whl", hash = "sha256:0776b0256cd79ff202dd4c187872eb309bbee7ea63b9be722f9063b232b42ecd", size = 112208, upload-time = "2026-04-12T18:36:10.289Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/de/31/3df7c7bfa022b8e49fba5bd511226eb8ab02b335832441dc9d5c07a154e7/cryptg-0.6.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:877e872b0967f03d008e2cb16a1aa44c8011b2f200bdf14771e6f783228e8bce", size = 207943, upload-time = "2026-04-12T18:33:55.483Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/06/81/dea9beefd7317c353484a5736ce3de69ef9c1e84a93ea46cbe47f04a7f18/cryptg-0.6.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:b8ba3ed799086af279c1efb7bfa72b226dc9427f85034ab0b05ff23bccb97b45", size = 236856, upload-time = "2026-04-12T18:35:21.006Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e9/f1/fcc52170486f35698229b573c0b6bc844f0504745f43fc47fde0cb61888e/cryptg-0.6.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:ae96b26dfb53968fe4986f70222f2419c82651936f2ef3b2c2dc47bc74e77d96", size = 250297, upload-time = "2026-04-12T18:35:40.577Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c3/d5/5041385f3a55d6eaa8f86c79e9fbb799d09680977619d506e35c09a9a3f9/cryptg-0.6.0-cp314-cp314t-win32.whl", hash = "sha256:e4988fd1657aa86d91bc98d8262b78b141dec8ef51a843f1ef78d69e8a81fdf0", size = 111989, upload-time = "2026-04-12T18:36:05.193Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/88/08/86db1d558e836c75724ef893c024d3f087886d1f8b173f2f64164505e256/cryptg-0.6.0-cp314-cp314t-win_amd64.whl", hash = "sha256:43f195810c642c6f6c552d291a216a5b5cb36b06aabe2832873db9dde3e65bb1", size = 111602, upload-time = "2026-04-12T18:36:06.81Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pyaes"
|
||||
version = "1.6.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/44/66/2c17bae31c906613795711fc78045c285048168919ace2220daa372c7d72/pyaes-1.6.1.tar.gz", hash = "sha256:02c1b1405c38d3c370b085fb952dd8bea3fadcee6411ad99f312cc129c536d8f", size = 28536, upload-time = "2017-09-20T21:17:54.23Z" }
|
||||
|
||||
[[package]]
|
||||
name = "pyasn1"
|
||||
version = "0.6.4"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/a4/9a/23310166d960def5897e91fe20e5b724601b02a22e84ba1f94232c0b7f67/pyasn1-0.6.4.tar.gz", hash = "sha256:9c447d8431c947fe4c8febc4ed9e760bc29011a5b01e5c74b67025bd9fb8ce81", size = 151262, upload-time = "2026-07-09T01:12:33.988Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/9a/3b/6163796d69c3977d1e4287bea4a6979161cbbdd170ebb430511e8e1999ce/pyasn1-0.6.4-py3-none-any.whl", hash = "sha256:deda9277cfd454080ec40b207fb6df82206a3a2688735233cdcd8d3d565f088b", size = 84410, upload-time = "2026-07-09T01:12:32.92Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rsa"
|
||||
version = "4.9.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "pyasn1" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/da/8a/22b7beea3ee0d44b1916c0c1cb0ee3af23b700b6da9f04991899d0c555d4/rsa-4.9.1.tar.gz", hash = "sha256:e7bdbfdb5497da4c07dfd35530e1a902659db6ff241e39d9953cad06ebd0ae75", size = 29034, upload-time = "2025-04-16T09:51:18.218Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/64/8d/0133e4eb4beed9e425d9a98ed6e081a55d195481b7632472be1af08d2f6b/rsa-4.9.1-py3-none-any.whl", hash = "sha256:68635866661c6836b8d39430f97a996acbd61bfa49406748ea243539fe239762", size = 34696, upload-time = "2025-04-16T09:51:17.142Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "telethon"
|
||||
version = "1.44.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "pyaes" },
|
||||
{ name = "rsa" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/49/e2/d4137e598db0ba50e4cf9bf32f2e26d016b2c4bb0e9763e3d5a07007d780/telethon-1.44.0.tar.gz", hash = "sha256:7de1473e9e412e20bd57b102d8f50d48372712462be9ff6c003bb81c50ad7e98", size = 701463, upload-time = "2026-06-15T15:47:39.44Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/82/fd/4ad621d7a4b8655dfc964ee1c1267407496ec6fd10c91901a64ea29c16c8/telethon-1.44.0-py3-none-any.whl", hash = "sha256:52fc49efb67a4916c2aedcb295ad286f4afa2aba9bf15d83ed2acdc64af0c718", size = 789786, upload-time = "2026-06-15T15:47:41.534Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tg-export"
|
||||
version = "0.1.0"
|
||||
source = { virtual = "." }
|
||||
dependencies = [
|
||||
{ name = "cryptg" },
|
||||
{ name = "telethon" },
|
||||
]
|
||||
|
||||
[package.metadata]
|
||||
requires-dist = [
|
||||
{ name = "cryptg", specifier = ">=0.6.0" },
|
||||
{ name = "telethon", specifier = ">=1.44.0" },
|
||||
]
|
||||
Executable
+9
@@ -0,0 +1,9 @@
|
||||
#!/bin/sh
|
||||
# tg-export КОМАНДА [опции export.py] — выгрузка второго Telegram-аккаунта владельца (infra stacks/rpi-tg-export).
|
||||
# Запускать от tgexport: sudo -u tgexport tg-export list | login | run …
|
||||
# Ключи приложения — /etc/tg-export/env (root:tgexport 640), сессия — /var/lib/tg-export/tg.session (tgexport 600),
|
||||
# выгрузка — /mnt/backup/tg (группа astra, только чтение).
|
||||
set -eu
|
||||
set -a; . /etc/tg-export/env; set +a
|
||||
exec /opt/tg-export/venv/bin/python /opt/tg-export/export.py "$@" \
|
||||
--session /var/lib/tg-export/tg.session --out /mnt/backup/tg
|
||||
Executable
+17
@@ -0,0 +1,17 @@
|
||||
#!/bin/sh
|
||||
# Установка/обновление выгрузки Telegram на rpi: sudo sh install.sh (из копии каталога stacks/rpi-tg-export)
|
||||
# Ключи (/etc/tg-export/env) и вход (sudo -u tgexport tg-export login) — отдельно, см. README.
|
||||
set -eu
|
||||
cd "$(dirname "$0")"
|
||||
id tgexport >/dev/null 2>&1 || useradd --system --home-dir /var/lib/tg-export --shell /usr/sbin/nologin tgexport
|
||||
install -d -m 755 /opt/tg-export
|
||||
install -m 644 app/export.py app/fetch_public.py app/channel_comments.py app/pyproject.toml /opt/tg-export/
|
||||
[ -x /opt/tg-export/venv/bin/python ] || python3 -m venv /opt/tg-export/venv
|
||||
/opt/tg-export/venv/bin/pip install -q --disable-pip-version-check 'telethon>=1.44.0' 'cryptg>=0.6.0'
|
||||
install -m 755 bin/tg-export /usr/local/bin/tg-export
|
||||
install -d -o root -g tgexport -m 750 /etc/tg-export
|
||||
install -d -o tgexport -g tgexport -m 700 /var/lib/tg-export
|
||||
install -d -o tgexport -g astra -m 2750 /mnt/backup/tg
|
||||
install -m 644 systemd/tg-export.service systemd/tg-export.timer /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
echo "ok: код в /opt/tg-export; таймер включается после входа: systemctl enable --now tg-export.timer"
|
||||
@@ -0,0 +1,17 @@
|
||||
[Unit]
|
||||
Description=Telegram: докачка истории второго аккаунта владельца (только текст)
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
RequiresMountsFor=/mnt/backup
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=tgexport
|
||||
Group=tgexport
|
||||
# файлы выгрузки 640, каталог setgid astra — Astra читает, остальные пользователи малины нет
|
||||
UMask=0027
|
||||
Nice=10
|
||||
# первый прогон долгий: Telegram троттлит чтение истории (FloodWait), скрипт ждёт сам
|
||||
TimeoutStartSec=infinity
|
||||
ExecStart=/usr/local/bin/tg-export run --no-media --archived
|
||||
ExecStopPost=+/usr/local/bin/backup-metrics backup_tg_export
|
||||
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=Telegram: докачка истории второго аккаунта раз в сутки
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 04:30
|
||||
RandomizedDelaySec=10m
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Reference in New Issue
Block a user