Microsoft hat App-Passwords für consumer-Outlook im September 2024 abgeschaltet. Diese Welle bringt OAuth2/XOAUTH2-Support als zweiten AuthMethod-Pfad — Gmail/ iCloud/GMX/Yahoo bleiben unangetastet auf App-Password. Backend (rebreak-backend): - POST /api/mail/oauth/microsoft/init: PKCE-Flow-Start, generiert code_verifier + Authorization-URL, persistiert pending state mit TTL - POST /api/mail/oauth/microsoft/callback: Token-Exchange (PKCE, kein client_secret weil Public Client), id_token-Decode für Email, MailConnection upsert mit auth_method='oauth2_microsoft' + encrypted Tokens - Token-Refresh-Util backend/server/utils/ms-oauth.ts + DB-Function refreshAndSaveTokens(connectionId, clientId) mit optimistic-concurrency- Race-Condition-Schutz (UPDATE WHERE oauth_token_expiry = <gelesener-wert>, bei affected_rows=0 → frischen Wert lesen statt nochmal refreshen sonst invalid_grant via Token-Rotation) - Neue Tabelle oauth_pending_states (TTL via createdAt + Cleanup-Job-TODO) - [id].delete.ts: echter OAuth-Disconnect — DB-Token-Löschung + Audit-Log (MS hat keinen Drittanbieter-Revoke-Endpoint, daher User-Information-Pflicht per Frontend-Modal, siehe DSB-Memo Section 5.1) - Consent-Gate auch in scan.post.ts + scan-internal.post.ts (Cron-Trigger war ohne Consent-Check = DSGVO-Lücke, jetzt geschlossen mit skippedNoConsent-Field in Response) IDLE-Daemon (backend/imap-idle/index.mjs, mo): - XOAUTH2-Auth-Branch via getCredentialsForConnection() — wenn auth_method='oauth2_microsoft', Token-Expiry-Check (<5min remaining → proaktiver Refresh), sonst decrypted accessToken zu ImapFlow - AUTHENTICATIONFAILED-Recovery: bis 3× reaktiv refresh + reconnect, danach last_connect_error='auth_revoked' (kein Endlos-Loop) - IDLE_RENEW_INTERVAL_MS = 10min — passt für MS 29min-Timeout (gleich wie Gmail/iCloud) - Consent-Pause: Connections mit consent_at=null laufen IDLE weiter (für exists-Event-Wiederaufnahme), aber triggerScan() ist deaktiviert bis consent erteilt - start-idle-staging.sh: MS_OAUTH_CLIENT_ID explizit weiterleiten in den inneren bash -c-Block (war Infisical-Var, ging aber durch strict-mode verloren) Frontend (rebreak-native-ui): - Outlook-Tile re-aktiviert (war disabled mit "Kommt bald" seit Sept-2024- Awareness), authMethod-Discriminator löst statt Email+Pw-Form den OAuth-Flow aus - ConnectMailSheet: neuer view-State 'oauth_warning' (Outing-Effekt-Hinweis per Hans-Müller-Memo Section 6.1) + 'oauth_pending' (Browser-Step-Spinner) - Deep-Link-Handler app/auth/mail-oauth-callback.tsx — auto-registriert durch expo-router-File-Routing, kein Native-Rebuild (scheme 'rebreak' schon im app.config.ts) - mailConnectDraft-Store: pendingOAuthConnectionId für Title-Edit-Sheet direkt nach Connect - MailAccountCard: Password-Row hidden für OAuth-Connections, Post-Disconnect- Modal mit MS-Account-Anleitung (DSB-konform — kompensiert fehlenden Drittanbieter-Revoke-Endpoint mit User-Information) Hans-Müller-DSB-Memo (mail-outlook-oauth-dsgvo-review.md): - Section 4.1 Datenschutzerklärung-Textbaustein: "Wir widerrufen den Token aktiv bei Microsoft"-Satz raus (war faktisch falsch — MS hat keinen Drittanbieter-Revoke). Neuer Wortlaut: DB-Löschung + User-Anleitung account.microsoft.com → Sicherheit → App-Berechtigungen - Section 4.1: User.Read-Scope offen dokumentiert mit Datenminimierungs- Klausel (Scope breiter, wir nutzen NUR Display-Name + Email-Claim) - Section 5.1: ehrliche Doku dass MS keinen RFC-7009-Revoke hat - Section 9 Anwalts-Themen: neue Frage 5 zur Art. 17-Erfüllung trotz fehlendem MS-Revoke Architektur-Eigenschaften: - Generisches AuthMethod-Framework — Gmail/iCloud/Yahoo können später als reine Config-Erweiterung OAuth bekommen, kein Refactor nötig - Token-Encryption via bestehendes crypto.ts (AES-256-GCM, Key aus Infisical) - Consent-Gate konsistent: ConnectMailSheet-Consent-Step VOR Provider- Auswahl (Frontend), backend-Endpoint 412 wenn consent fehlt, Daemon + Scan-Endpoints pausieren bei consent_at=null Open follow-ups: - oauth_pending_states-Cleanup-Cron für abgelaufene Entries (TODO im Backend-Code dokumentiert) - Anwalts-Klärung Hans-Müller Section 9 (DPA-Anspruch ohne MS-Lizenz + Art. 17 mit User-Information statt Revoke-Endpoint) - TIA (Transfer Impact Assessment) für MS-Sub-AV — Hans-Müller-Draft-Aufgabe - Outlook-Tile-Wieder-Aktivierung ist live, aber Phase-1-Production-Test steht aus (User Test auf iPhone nach Pipeline-Deploy) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
731 lines
27 KiB
JavaScript
731 lines
27 KiB
JavaScript
/**
|
|
* rebreak-imap-idle — IMAP IDLE Daemon
|
|
*
|
|
* Hält pro aktivem MailConnection-Eintrag eine persistente IMAP-IDLE-Session.
|
|
* Wenn der Server "EXISTS" meldet (neue Mail), feuert der Daemon sofort
|
|
* POST /api/mail/scan-internal gegen das lokale Backend — ohne 30min-Warte.
|
|
*
|
|
* Auth-Methoden:
|
|
* app_password — Gmail, iCloud, GMX, etc. (App-Password / IMAP-Passwort)
|
|
* oauth2_microsoft — Outlook / Hotmail / O365 via XOAUTH2 (ImapFlow-nativ)
|
|
*
|
|
* Env-Vars (via Infisical-Wrapper):
|
|
* DATABASE_URL — Postgres-Connection-String
|
|
* ADMIN_SECRET — Header-Secret für /api/mail/scan-internal
|
|
* ENCRYPTION_KEY — AES-256 Key (gleicher wie im Backend, 32+ Zeichen)
|
|
* BACKEND_URL — z.B. http://127.0.0.1:3016 (default: 3016)
|
|
* MS_OAUTH_CLIENT_ID — Azure App Registration Client ID
|
|
* NODE_ENV — production / staging
|
|
*
|
|
* Starten:
|
|
* node index.mjs
|
|
*
|
|
* Prozess-Signale:
|
|
* SIGTERM / SIGINT → graceful shutdown (alle IMAP-Sessions schließen)
|
|
*/
|
|
|
|
import { ImapFlow } from "imapflow";
|
|
import pg from "pg";
|
|
import { createCipheriv, createDecipheriv, randomBytes } from "crypto";
|
|
|
|
// ─── Config ──────────────────────────────────────────────────────────────────
|
|
|
|
const BACKEND_URL =
|
|
process.env.BACKEND_URL ||
|
|
(process.env.NODE_ENV === "production"
|
|
? "http://127.0.0.1:3015"
|
|
: "http://127.0.0.1:3016");
|
|
|
|
const ADMIN_SECRET =
|
|
process.env.NUXT_ADMIN_SECRET || process.env.ADMIN_SECRET || "";
|
|
|
|
const MS_OAUTH_CLIENT_ID =
|
|
process.env.MS_OAUTH_CLIENT_ID || "";
|
|
|
|
const DB_REFRESH_INTERVAL_MS = 5 * 60 * 1000; // 5 min — neue Connections entdecken
|
|
|
|
// IDLE_RENEW von 25min → 10min: GMX dropped IDLE-connections silent vor 25min
|
|
// → exists-events kommen nie an + ImapFlow.idle() hängt ohne reject. 10min
|
|
// deckt alle bekannten Provider-Timeouts ab:
|
|
// GMX ~10-15min (aggressivster Provider)
|
|
// Gmail ~29min
|
|
// iCloud ~29min
|
|
// outlook.office365.com ~29min (Microsoft dokumentiert 29min IDLE-Timeout)
|
|
// Trade-off: alle 10min full reconnect-cycle. Vertretbar.
|
|
const IDLE_RENEW_INTERVAL_MS = 10 * 60 * 1000; // 10 min
|
|
|
|
// NOOP-heartbeat alle 2min während IDLE: detect silent-drops (GMX-pattern).
|
|
// Wenn NOOP fehlschlägt → close → loop iteriert → reconnect.
|
|
const IDLE_NOOP_INTERVAL_MS = 2 * 60 * 1000; // 2 min
|
|
|
|
// Token-Refresh-Schwelle: wenn Access-Token in weniger als 5min abläuft, vor
|
|
// dem IMAP-Connect refreshen. Verhindert Mid-Session-Expiry.
|
|
const TOKEN_EXPIRY_THRESHOLD_MS = 5 * 60 * 1000; // 5 min
|
|
|
|
// Bei AUTHENTICATIONFAILED: max. 3 Refresh-Versuche bevor Connection als
|
|
// auth_broken markiert und der Daemon sie aufgibt.
|
|
const MAX_AUTH_RETRIES = 3;
|
|
|
|
const RECONNECT_DELAYS_MS = [1000, 5000, 30_000]; // exponential backoff
|
|
const RECONNECT_LOOP_DELAY_MS = 60 * 1000;
|
|
|
|
// ─── DB-Pool (direktes pg, kein Prisma — Daemon ist kein Nitro-Kontext) ─────
|
|
|
|
const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
|
|
|
|
async function updateConnectionError(connId, errorText) {
|
|
await pool.query(
|
|
`UPDATE rebreak.mail_connections
|
|
SET last_connect_error = $1, last_connect_error_at = NOW()
|
|
WHERE id = $2`,
|
|
[errorText, connId],
|
|
);
|
|
}
|
|
|
|
async function clearConnectionError(connId) {
|
|
await pool.query(
|
|
`UPDATE rebreak.mail_connections
|
|
SET last_connect_error = NULL, last_connect_error_at = NULL
|
|
WHERE id = $1`,
|
|
[connId],
|
|
);
|
|
}
|
|
|
|
async function updateIdleHeartbeat(connId) {
|
|
await pool.query(
|
|
`UPDATE rebreak.mail_connections
|
|
SET last_idle_heartbeat_at = NOW()
|
|
WHERE id = $1`,
|
|
[connId],
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Lädt alle aktiven MailConnections inkl. OAuth-Felder und consent_at.
|
|
* Consent-Gate: connections ohne consent_at werden geladen aber vom
|
|
* IDLE-Loop nicht gescannt (nur gehalten — kein Delete).
|
|
*/
|
|
async function loadActiveConnections() {
|
|
const { rows } = await pool.query(
|
|
`SELECT id,
|
|
user_id AS "userId",
|
|
email,
|
|
imap_host AS "imapHost",
|
|
imap_port AS "imapPort",
|
|
password_encrypted AS "passwordEncrypted",
|
|
reject_unauthorized AS "rejectUnauthorized",
|
|
use_starttls AS "useStarttls",
|
|
auth_method AS "authMethod",
|
|
oauth_access_token AS "oauthAccessToken",
|
|
oauth_refresh_token AS "oauthRefreshToken",
|
|
oauth_token_expiry AS "oauthTokenExpiry",
|
|
consent_at AS "consentAt"
|
|
FROM rebreak.mail_connections
|
|
WHERE is_active = true`,
|
|
);
|
|
return rows;
|
|
}
|
|
|
|
// ─── Crypto (analog zu server/utils/crypto.ts) ───────────────────────────────
|
|
|
|
const AES_ALGO = "aes-256-gcm";
|
|
const KEY_LENGTH = 32;
|
|
|
|
function getKey() {
|
|
const raw =
|
|
process.env.NUXT_ENCRYPTION_KEY || process.env.ENCRYPTION_KEY || "";
|
|
if (!raw || raw.length < KEY_LENGTH) {
|
|
return Buffer.alloc(
|
|
KEY_LENGTH,
|
|
raw.padEnd(KEY_LENGTH, "0").slice(0, KEY_LENGTH),
|
|
);
|
|
}
|
|
return Buffer.from(raw.slice(0, KEY_LENGTH), "utf8");
|
|
}
|
|
|
|
function decrypt(stored) {
|
|
const parts = stored.split(":");
|
|
if (parts.length !== 3) throw new Error("Invalid encrypted format");
|
|
const [ivHex, tagHex, dataHex] = parts;
|
|
const decipher = createDecipheriv(
|
|
AES_ALGO,
|
|
getKey(),
|
|
Buffer.from(ivHex, "hex"),
|
|
);
|
|
decipher.setAuthTag(Buffer.from(tagHex, "hex"));
|
|
return decipher.update(Buffer.from(dataHex, "hex")) + decipher.final("utf8");
|
|
}
|
|
|
|
function encrypt(plaintext) {
|
|
const key = getKey();
|
|
const iv = randomBytes(12); // 96-bit IV für AES-256-GCM
|
|
const cipher = createCipheriv(AES_ALGO, key, iv);
|
|
const encrypted = Buffer.concat([
|
|
cipher.update(plaintext, "utf8"),
|
|
cipher.final(),
|
|
]);
|
|
const tag = cipher.getAuthTag();
|
|
return `${iv.toString("hex")}:${tag.toString("hex")}:${encrypted.toString("hex")}`;
|
|
}
|
|
|
|
// ─── Microsoft Token Refresh (inline, kein Prisma-Kontext) ───────────────────
|
|
|
|
const MS_TOKEN_ENDPOINT =
|
|
"https://login.microsoftonline.com/common/oauth2/v2.0/token";
|
|
|
|
const MS_OAUTH_SCOPES = [
|
|
"https://outlook.office.com/IMAP.AccessAsUser.All",
|
|
"offline_access",
|
|
"openid",
|
|
"User.Read",
|
|
].join(" ");
|
|
|
|
/**
|
|
* Refresht Microsoft Access+Refresh-Token direkt via HTTP.
|
|
* Race-Condition-Strategie (Optimistic Concurrency):
|
|
* 1. Lese aktuelle oauth_token_expiry aus DB als "Fingerprint" des aktuellen Stands.
|
|
* 2. POST an MS Token-Endpoint.
|
|
* 3. UPDATE ... WHERE oauth_token_expiry = <gelesener Wert>
|
|
* → affected_rows = 0: anderer Prozess hat parallel refresht.
|
|
* Dann: frischen token lesen + zurückgeben (kein doppelter Refresh!).
|
|
* Doppelter Refresh würde MS-Refresh-Token-Rotation verletzen →
|
|
* AADSTS70043 beim nächsten Versuch.
|
|
* 4. Gibt plaintext Access-Token zurück, sofort für IMAP nutzbar.
|
|
*
|
|
* Wirft wenn:
|
|
* - Connection nicht gefunden / kein Refresh-Token
|
|
* - MS Token-Endpoint antwortet mit Fehler (revoked/expired Refresh-Token)
|
|
*/
|
|
async function refreshAndSaveTokensDaemon(connectionId, clientId) {
|
|
// Step 1: Aktuellen Token-Stand lesen
|
|
const { rows } = await pool.query(
|
|
`SELECT oauth_refresh_token, oauth_access_token, oauth_token_expiry
|
|
FROM rebreak.mail_connections
|
|
WHERE id = $1 AND auth_method = 'oauth2_microsoft'`,
|
|
[connectionId],
|
|
);
|
|
|
|
const conn = rows[0];
|
|
if (!conn?.oauth_refresh_token) {
|
|
throw new Error(
|
|
`Connection ${connectionId} has no oauth_refresh_token — cannot refresh`,
|
|
);
|
|
}
|
|
|
|
const currentExpiry = conn.oauth_token_expiry;
|
|
const decryptedRefreshToken = decrypt(conn.oauth_refresh_token);
|
|
|
|
// Step 2: MS Token-Endpoint
|
|
const body = new URLSearchParams({
|
|
grant_type: "refresh_token",
|
|
client_id: clientId,
|
|
refresh_token: decryptedRefreshToken,
|
|
scope: MS_OAUTH_SCOPES,
|
|
});
|
|
|
|
const res = await fetch(MS_TOKEN_ENDPOINT, {
|
|
method: "POST",
|
|
headers: { "Content-Type": "application/x-www-form-urlencoded" },
|
|
body: body.toString(),
|
|
});
|
|
|
|
if (!res.ok) {
|
|
const errText = await res.text().catch(() => "unknown");
|
|
throw new Error(`MS token refresh failed (${res.status}): ${errText}`);
|
|
}
|
|
|
|
const data = await res.json();
|
|
if (!data.access_token) {
|
|
throw new Error("MS refresh response missing access_token");
|
|
}
|
|
|
|
const newExpiry = new Date(Date.now() + data.expires_in * 1000);
|
|
// MS rotiert Refresh-Tokens — wenn kein neuer kommt, den alten behalten
|
|
const newRefreshToken = data.refresh_token ?? decryptedRefreshToken;
|
|
|
|
const encryptedAccess = encrypt(data.access_token);
|
|
const encryptedRefresh = encrypt(newRefreshToken);
|
|
|
|
// Step 3: Optimistic update — nur wenn oauth_token_expiry noch gleich wie gelesen
|
|
// IS NOT DISTINCT FROM deckt NULL=NULL korrekt ab (vs. = das NULL!=NULL behandelt)
|
|
const result = await pool.query(
|
|
`UPDATE rebreak.mail_connections
|
|
SET oauth_access_token = $1,
|
|
oauth_refresh_token = $2,
|
|
oauth_token_expiry = $3
|
|
WHERE id = $4
|
|
AND oauth_token_expiry IS NOT DISTINCT FROM $5`,
|
|
[encryptedAccess, encryptedRefresh, newExpiry, connectionId, currentExpiry],
|
|
);
|
|
|
|
if (result.rowCount === 0) {
|
|
// Step 4: Anderer Prozess hat parallel refresht — lese deren frischen Token
|
|
const { rows: fresh } = await pool.query(
|
|
`SELECT oauth_access_token FROM rebreak.mail_connections WHERE id = $1`,
|
|
[connectionId],
|
|
);
|
|
if (!fresh[0]?.oauth_access_token) {
|
|
throw new Error(
|
|
`Concurrent refresh for ${connectionId} detected but no token found`,
|
|
);
|
|
}
|
|
return decrypt(fresh[0].oauth_access_token);
|
|
}
|
|
|
|
return data.access_token;
|
|
}
|
|
|
|
/**
|
|
* Markiert eine Connection als "auth_broken" — kein weiterer Retry im Daemon.
|
|
* User sieht den Error im Frontend (last_connect_error = 'auth_revoked').
|
|
*/
|
|
async function markConnectionAuthBroken(connectionId) {
|
|
await pool.query(
|
|
`UPDATE rebreak.mail_connections
|
|
SET last_connect_error = 'auth_revoked',
|
|
last_connect_error_at = NOW()
|
|
WHERE id = $1`,
|
|
[connectionId],
|
|
);
|
|
}
|
|
|
|
// ─── Credentials-Resolution ──────────────────────────────────────────────────
|
|
|
|
/**
|
|
* Gibt die IMAP-Credentials für eine Connection zurück.
|
|
* auth_method-aware:
|
|
*
|
|
* app_password → { type: 'password', user, pass }
|
|
* oauth2_microsoft → { type: 'xoauth2', user, accessToken }
|
|
*
|
|
* Für OAuth: wenn Token in <5min abläuft, wird vor Connect refresht.
|
|
* Das ist der "proaktive" Refresh-Path (Hot-Path).
|
|
*
|
|
* Der "reaktive" Refresh-Path (Cold-Path) liegt in runSession():
|
|
* AUTHENTICATIONFAILED während laufender Session → refreshAndSaveTokensDaemon()
|
|
* → neuer ImapFlow-Connect mit frischem Token.
|
|
*
|
|
* FORCE_REFRESH_FOR_TEST: wenn env IDLE_FORCE_TOKEN_REFRESH=1 gesetzt ist,
|
|
* wird IMMER refresht unabhängig von Expiry. Nur für manuelle Tests.
|
|
*/
|
|
async function getCredentialsForConnection(conn) {
|
|
if (conn.authMethod === "oauth2_microsoft") {
|
|
const forceRefresh = process.env.IDLE_FORCE_TOKEN_REFRESH === "1";
|
|
const fiveMinFromNow = Date.now() + TOKEN_EXPIRY_THRESHOLD_MS;
|
|
const isExpiring =
|
|
!conn.oauthTokenExpiry ||
|
|
new Date(conn.oauthTokenExpiry).getTime() < fiveMinFromNow;
|
|
|
|
if (forceRefresh || isExpiring) {
|
|
const reason = forceRefresh ? "IDLE_FORCE_TOKEN_REFRESH=1" : "token expiring <5min";
|
|
// NIEMALS Token-Werte loggen — nur Fakt dass refresht wird
|
|
console.log(`[idle/${conn.email}] proactive token refresh (${reason})`);
|
|
const accessToken = await refreshAndSaveTokensDaemon(conn.id, MS_OAUTH_CLIENT_ID);
|
|
return { type: "xoauth2", user: conn.email, accessToken };
|
|
}
|
|
|
|
// Token noch gültig — direkt entschlüsseln
|
|
if (!conn.oauthAccessToken) {
|
|
throw new Error(`Connection ${conn.id} has no oauth_access_token`);
|
|
}
|
|
const accessToken = decrypt(conn.oauthAccessToken);
|
|
return { type: "xoauth2", user: conn.email, accessToken };
|
|
}
|
|
|
|
// Bestand: Gmail / iCloud / GMX / Custom-IMAP → App-Password
|
|
if (!conn.passwordEncrypted) {
|
|
throw new Error(`Connection ${conn.id} has no password_encrypted`);
|
|
}
|
|
const pass = decrypt(conn.passwordEncrypted);
|
|
return { type: "password", user: conn.email, pass };
|
|
}
|
|
|
|
// ─── Logging ─────────────────────────────────────────────────────────────────
|
|
|
|
function log(email, msg) {
|
|
// NIEMALS password/accessToken/credentials loggen. email ist safe.
|
|
console.log(`[idle/${email}] ${msg}`);
|
|
}
|
|
|
|
function logError(email, msg, err) {
|
|
// responseText = IMAP-Serverantwort (z.B. "NO [AUTHENTICATIONFAILED] Invalid credentials")
|
|
// Credentials tauchen NICHT in err.responseText auf — ImapFlow legt sie nicht rein.
|
|
const errMsg = err?.responseText || err?.message || String(err);
|
|
console.error(`[idle/${email}] ${msg}: ${errMsg}`);
|
|
}
|
|
|
|
// ─── Auth-Fehler Detection ───────────────────────────────────────────────────
|
|
|
|
function isAuthError(err) {
|
|
const text = (err?.responseText || err?.message || "").toUpperCase();
|
|
return (
|
|
text.includes("AUTHENTICATIONFAILED") ||
|
|
text.includes("AUTHENTICATE") ||
|
|
text.includes("INVALID CREDENTIALS") ||
|
|
text.includes("AUTHENTICATION FAILED") ||
|
|
// MS-spezifische Fehlercodes: AADSTS = Azure AD Token Service Error
|
|
text.includes("AADSTS")
|
|
);
|
|
}
|
|
|
|
// ─── Session-Registry ────────────────────────────────────────────────────────
|
|
// Map<connectionId, SessionHandle>
|
|
|
|
const sessions = new Map();
|
|
|
|
let shuttingDown = false;
|
|
|
|
// ─── Scan-Trigger ─────────────────────────────────────────────────────────────
|
|
|
|
async function triggerScan(conn) {
|
|
try {
|
|
const res = await fetch(`${BACKEND_URL}/api/mail/scan-internal`, {
|
|
method: "POST",
|
|
headers: {
|
|
"Content-Type": "application/json",
|
|
"x-admin-secret": ADMIN_SECRET,
|
|
},
|
|
body: JSON.stringify({ userId: conn.userId }),
|
|
});
|
|
if (!res.ok) {
|
|
log(conn.email, `scan-trigger HTTP ${res.status}`);
|
|
} else {
|
|
const data = await res.json().catch(() => ({}));
|
|
log(
|
|
conn.email,
|
|
`scan-triggered → scanned=${data.scanned ?? "?"} blocked=${data.blocked ?? "?"}`,
|
|
);
|
|
}
|
|
} catch (err) {
|
|
logError(conn.email, "scan-trigger failed", err);
|
|
}
|
|
}
|
|
|
|
// ─── IDLE-Session ─────────────────────────────────────────────────────────────
|
|
|
|
/**
|
|
* Startet eine einzelne IDLE-Session für eine MailConnection.
|
|
* Reconnect-Loop läuft intern — diese Funktion returned nie (bis shutdown).
|
|
*
|
|
* Auth-Retry-Loop (OAuth-spezifisch):
|
|
* Bei AUTHENTICATIONFAILED → Token refreshen → neu verbinden.
|
|
* Max MAX_AUTH_RETRIES (3) mal. Dann: markConnectionAuthBroken + exit.
|
|
* authRetries wird nach jedem erfolgreichen Connect resettet.
|
|
*
|
|
* Consent-Gate:
|
|
* conn.consentAt === null → IDLE-Verbindung wird gehalten (keep-alive),
|
|
* aber triggerScan() wird NICHT aufgerufen. exists-Events werden still ignoriert.
|
|
* Sobald der User consent erteilt (DB-Refresh-Cycle nach max. 5min),
|
|
* wird die Connection neu gestartet mit consentAt gesetzt.
|
|
*/
|
|
async function runSession(conn) {
|
|
let attempt = 0;
|
|
let authRetries = 0;
|
|
|
|
while (!shuttingDown) {
|
|
// Credentials holen (proaktiver Token-Refresh wenn nötig)
|
|
let creds;
|
|
try {
|
|
creds = await getCredentialsForConnection(conn);
|
|
} catch (err) {
|
|
logError(conn.email, "credential resolution failed — session aborted", err);
|
|
// Kein retry: kaputte Credentials oder Token-Refresh fehlgeschlagen
|
|
// (z.B. Refresh-Token revoked). Als auth_broken markieren wenn OAuth.
|
|
if (conn.authMethod === "oauth2_microsoft") {
|
|
await markConnectionAuthBroken(conn.id).catch(() => {});
|
|
} else {
|
|
await updateConnectionError(conn.id, err?.message || String(err)).catch(() => {});
|
|
}
|
|
return;
|
|
}
|
|
|
|
const useImplicitTls = !conn.useStarttls;
|
|
const imap = new ImapFlow({
|
|
host: conn.imapHost,
|
|
port: conn.imapPort,
|
|
secure: useImplicitTls,
|
|
...(conn.useStarttls ? { requireTLS: true } : {}),
|
|
// ImapFlow 1.2.18 unterstützt XOAUTH2 nativ:
|
|
// { user, accessToken } → AUTHENTICATE XOAUTH2 <base64-token>
|
|
// { user, pass } → LOGIN oder PLAIN (je nach Server-Capability)
|
|
auth: creds.type === "xoauth2"
|
|
? { user: creds.user, accessToken: creds.accessToken }
|
|
: { user: creds.user, pass: creds.pass },
|
|
logger: false,
|
|
tls: { rejectUnauthorized: conn.rejectUnauthorized ?? true },
|
|
// outlook.office365.com: disableCompression verhindert edge-cases bei
|
|
// partial reads nach Reconnect. Gilt für oauth2_microsoft Connections.
|
|
disableCompression: conn.imapHost.includes("office365"),
|
|
});
|
|
|
|
// Referenz ablegen damit shutdown() darauf zugreifen kann
|
|
const handle = sessions.get(conn.id);
|
|
if (handle) handle.imap = imap;
|
|
|
|
try {
|
|
await imap.connect();
|
|
log(conn.email, `connected (${conn.imapHost}:${conn.imapPort}, auth=${creds.type})`);
|
|
attempt = 0; // Reset nach erfolgreicher Verbindung
|
|
authRetries = 0; // Auth-Retry-Counter ebenfalls reset
|
|
await clearConnectionError(conn.id).catch(() => {});
|
|
|
|
await imap.getMailboxLock("INBOX");
|
|
|
|
// Consent-Gate-Log: einmalig beim Connect — nur wenn consentAt fehlt
|
|
if (!conn.consentAt) {
|
|
log(
|
|
conn.email,
|
|
"consent_at=NULL — IDLE session held but scan/delete suspended. " +
|
|
"Re-consent required via app.",
|
|
);
|
|
}
|
|
|
|
// IDLE-Loop: alle IDLE_RENEW_INTERVAL_MS erneuern
|
|
while (!shuttingDown && sessions.has(conn.id)) {
|
|
const idlePromise = new Promise((resolve, reject) => {
|
|
imap.idle().then(resolve).catch(reject);
|
|
});
|
|
|
|
// exists-event → sofort scannen (nur wenn consent erteilt)
|
|
const onExists = () => {
|
|
if (!conn.consentAt) {
|
|
// Consent fehlt: exists-event ignorieren, Mail bleibt in Inbox.
|
|
// UI zeigt Re-Consent-Modal (via /api/mail/pending-consent Endpoint).
|
|
log(conn.email, "exists-event received — skipped (no consent_at)");
|
|
return;
|
|
}
|
|
log(conn.email, "exists-event received (new mail)");
|
|
triggerScan(conn); // fire-and-forget
|
|
};
|
|
imap.on("exists", onExists);
|
|
|
|
// IDLE nach 10min erneuern
|
|
// Gilt für: GMX (~10-15min), Gmail (~29min), iCloud (~29min), outlook.office365.com (~29min)
|
|
const renewTimer = setTimeout(() => {
|
|
log(conn.email, "idle renewing (10min threshold)");
|
|
imap.close();
|
|
}, IDLE_RENEW_INTERVAL_MS);
|
|
|
|
// NOOP-heartbeat alle 2min: silent-drop early-detection (GMX-pattern).
|
|
const noopTimer = setInterval(async () => {
|
|
try {
|
|
await imap.noop();
|
|
await updateIdleHeartbeat(conn.id).catch(() => {});
|
|
} catch (err) {
|
|
logError(conn.email, "noop failed — connection dead, force reconnect", err);
|
|
imap.close();
|
|
}
|
|
}, IDLE_NOOP_INTERVAL_MS);
|
|
|
|
try {
|
|
await idlePromise;
|
|
} finally {
|
|
clearInterval(noopTimer);
|
|
clearTimeout(renewTimer);
|
|
imap.removeListener("exists", onExists);
|
|
}
|
|
|
|
if (shuttingDown || !sessions.has(conn.id)) break;
|
|
|
|
await sleep(500);
|
|
}
|
|
|
|
try { await imap.logout(); } catch { /* ignore */ }
|
|
return; // Sauberer Exit (shutdown oder Connection entfernt)
|
|
|
|
} catch (err) {
|
|
logError(conn.email, "connection error", err);
|
|
try { imap.close(); } catch { /* ignore */ }
|
|
|
|
// ── AUTHENTICATIONFAILED-Recovery (OAuth-spezifisch) ──────────────────
|
|
// Cold-Path: Token zwischen zwei IDLE-Renewals abgelaufen (>1h Session).
|
|
// Oder: proaktiver Refresh ist fehlgeschlagen und wir landen hier.
|
|
if (conn.authMethod === "oauth2_microsoft" && isAuthError(err)) {
|
|
authRetries++;
|
|
log(
|
|
conn.email,
|
|
`AUTHENTICATIONFAILED detected — refresh attempt ${authRetries}/${MAX_AUTH_RETRIES}`,
|
|
);
|
|
|
|
if (authRetries > MAX_AUTH_RETRIES) {
|
|
// Auth dauerhaft revoked (User hat App-Permission entzogen o.ä.)
|
|
// Connection als auth_broken markieren — User muss re-connecten.
|
|
log(conn.email, `auth_broken after ${MAX_AUTH_RETRIES} retries — session stopped`);
|
|
await markConnectionAuthBroken(conn.id).catch(() => {});
|
|
sessions.delete(conn.id);
|
|
return;
|
|
}
|
|
|
|
// Token refreshen — direkt hier im catch-Block.
|
|
// Wenn der Refresh selbst fehlschlägt (revoked Refresh-Token),
|
|
// wirft refreshAndSaveTokensDaemon — wir landen im äußeren catch
|
|
// und der normale Reconnect-Backoff greift (attempt++).
|
|
// Beim nächsten Attempt ruft getCredentialsForConnection() wieder refresh auf.
|
|
try {
|
|
const freshToken = await refreshAndSaveTokensDaemon(conn.id, MS_OAUTH_CLIENT_ID);
|
|
// conn ist das ursprüngliche Objekt aus loadActiveConnections.
|
|
// Wir patchen oauthAccessToken + oauthTokenExpiry inline damit
|
|
// getCredentialsForConnection() beim nächsten Loop-Durchlauf
|
|
// den frischen Token nutzt ohne erneuten DB-Read.
|
|
conn.oauthAccessToken = encrypt(freshToken);
|
|
conn.oauthTokenExpiry = new Date(Date.now() + 55 * 60 * 1000); // ~55min buffer
|
|
log(conn.email, "token refreshed — reconnecting immediately");
|
|
// Kein normaler Backoff nach Auth-Refresh — sofort neu verbinden.
|
|
// attempt bleibt unverändert (auth-error != network-error).
|
|
continue;
|
|
} catch (refreshErr) {
|
|
logError(conn.email, "token refresh failed after AUTHENTICATIONFAILED", refreshErr);
|
|
// Refresh selbst gescheitert → normaler Backoff (Netz, Serverproblem o.ä.)
|
|
// authRetries bleibt erhöht — beim nächsten Auth-Fehler zählt es weiter.
|
|
await updateConnectionError(conn.id, refreshErr?.message || String(refreshErr)).catch(() => {});
|
|
}
|
|
} else {
|
|
// Nicht-Auth-Fehler (Netz, TLS, etc.) — normal in DB schreiben
|
|
const errText = err?.responseText || err?.message || String(err);
|
|
await updateConnectionError(conn.id, errText).catch(() => {});
|
|
}
|
|
}
|
|
|
|
if (shuttingDown || !sessions.has(conn.id)) return;
|
|
|
|
// Exponential backoff
|
|
const delay =
|
|
attempt < RECONNECT_DELAYS_MS.length
|
|
? RECONNECT_DELAYS_MS[attempt]
|
|
: RECONNECT_LOOP_DELAY_MS;
|
|
attempt++;
|
|
log(conn.email, `reconnecting in ${delay / 1000}s (attempt ${attempt})`);
|
|
await sleep(delay);
|
|
}
|
|
}
|
|
|
|
// ─── Session-Management ───────────────────────────────────────────────────────
|
|
|
|
function startSession(conn) {
|
|
if (sessions.has(conn.id)) return; // bereits aktiv
|
|
log(conn.email, "session starting");
|
|
const handle = { conn, imap: null, promise: null };
|
|
sessions.set(conn.id, handle);
|
|
handle.promise = runSession(conn).catch((err) => {
|
|
logError(conn.email, "session crashed (unhandled)", err);
|
|
sessions.delete(conn.id);
|
|
});
|
|
}
|
|
|
|
async function stopSession(connectionId, email) {
|
|
const handle = sessions.get(connectionId);
|
|
if (!handle) return;
|
|
log(email ?? connectionId, "session stopping");
|
|
sessions.delete(connectionId);
|
|
if (handle.imap) {
|
|
try { handle.imap.close(); } catch { /* ignore */ }
|
|
}
|
|
await handle.promise.catch(() => {});
|
|
}
|
|
|
|
// ─── DB-Refresh-Loop ──────────────────────────────────────────────────────────
|
|
|
|
async function refreshConnections() {
|
|
let rows;
|
|
try {
|
|
rows = await loadActiveConnections();
|
|
} catch (err) {
|
|
console.error("[idle/db] loadActiveConnections failed:", err.message);
|
|
return;
|
|
}
|
|
|
|
const activeIds = new Set(rows.map((r) => r.id));
|
|
|
|
// Neue Connections starten
|
|
for (const row of rows) {
|
|
if (!sessions.has(row.id)) {
|
|
startSession(row);
|
|
}
|
|
}
|
|
|
|
// Entfernte Connections stoppen
|
|
for (const [id, handle] of sessions.entries()) {
|
|
if (!activeIds.has(id)) {
|
|
await stopSession(id, handle.conn?.email);
|
|
}
|
|
}
|
|
|
|
// Consent-Status für laufende Sessions aktualisieren.
|
|
// Wenn ein User consent erteilt hat seit dem letzten DB-Refresh:
|
|
// Die laufende Session bekommt conn.consentAt gesetzt — next exists-event
|
|
// löst dann sofort triggerScan() aus. Kein Restart nötig.
|
|
for (const row of rows) {
|
|
const handle = sessions.get(row.id);
|
|
if (handle && handle.conn.consentAt !== row.consentAt) {
|
|
if (!handle.conn.consentAt && row.consentAt) {
|
|
log(row.email, "consent_at received — scan/delete now active");
|
|
}
|
|
handle.conn.consentAt = row.consentAt;
|
|
}
|
|
}
|
|
|
|
const consentPending = rows.filter((r) => !r.consentAt).length;
|
|
console.log(
|
|
`[idle/db] refreshed — ${activeIds.size} active connections, ${sessions.size} sessions` +
|
|
(consentPending > 0 ? `, ${consentPending} pending consent` : ""),
|
|
);
|
|
}
|
|
|
|
// ─── Graceful Shutdown ────────────────────────────────────────────────────────
|
|
|
|
async function shutdown(signal) {
|
|
console.log(`[idle] received ${signal} — shutting down gracefully`);
|
|
shuttingDown = true;
|
|
|
|
const stopPromises = [];
|
|
for (const [id, handle] of sessions.entries()) {
|
|
stopPromises.push(stopSession(id, handle.conn?.email));
|
|
}
|
|
await Promise.allSettled(stopPromises);
|
|
|
|
await pool.end().catch(() => {});
|
|
console.log("[idle] shutdown complete");
|
|
process.exit(0);
|
|
}
|
|
|
|
process.on("SIGTERM", () => shutdown("SIGTERM"));
|
|
process.on("SIGINT", () => shutdown("SIGINT"));
|
|
|
|
// ─── Startup ──────────────────────────────────────────────────────────────────
|
|
|
|
function sleep(ms) {
|
|
return new Promise((resolve) => setTimeout(resolve, ms));
|
|
}
|
|
|
|
function assertEnv() {
|
|
const missing = [];
|
|
if (!process.env.DATABASE_URL) missing.push("DATABASE_URL");
|
|
if (!ADMIN_SECRET) missing.push("ADMIN_SECRET / NUXT_ADMIN_SECRET");
|
|
if (!MS_OAUTH_CLIENT_ID) missing.push("MS_OAUTH_CLIENT_ID");
|
|
if (missing.length > 0) {
|
|
console.error(
|
|
`[idle] FEHLER: fehlende Env-Vars: ${missing.join(", ")}. Daemon startet nicht.`,
|
|
);
|
|
process.exit(1);
|
|
}
|
|
}
|
|
|
|
async function main() {
|
|
assertEnv();
|
|
|
|
console.log(
|
|
`[idle] starting — backend=${BACKEND_URL} env=${process.env.NODE_ENV ?? "unknown"}`,
|
|
);
|
|
|
|
await refreshConnections();
|
|
|
|
setInterval(() => {
|
|
if (!shuttingDown) refreshConnections();
|
|
}, DB_REFRESH_INTERVAL_MS);
|
|
}
|
|
|
|
main().catch((err) => {
|
|
console.error("[idle] fatal startup error:", err);
|
|
process.exit(1);
|
|
});
|