LexiShift TODOs

Status: Active backlog Role: Planning / WIP Last updated: 2026-09-02 Last verified: 2026-08-27 beta integration, Python/build-install health tooling, and full typecheck-gate restoration; older backlog content not globally re-audited Purpose: consolidated product and architecture backlog retained after root README cleanup Source-of-truth: backlog planning only; current implementation truth lives in source code, tests, and docs/developer/feature_state_matrix.md.

This file is the consolidated TODO source of truth. It replaces the large TODO and plan sections that previously lived in the root README.md.

Related decision context:

• docs/architecture/chrome_web_store_review_working_doc.md
• docs/architecture/design_diagram_workplan.md

Architecture Diagram Rollout

Use docs/architecture/design_diagram_workplan.md and docs/architecture/diagrams/README.md as the execution tracker.

• DG-01: Data ownership + storage layout ([AS-IS] first).
• DG-02: Settings propagation (Options -> runtime mirrors -> content runtime).
• DG-03: Rule resolution + DOM replacement pipeline.
• DG-04: Feedback + eventual sync queue/retry path.
• DG-05: SRS initialize/refresh control flow.
• DG-06: Helper availability/degraded-mode state model.

Detailed Product Improvements (From 2026-02-18 Discussion)

1) Helper dependency UX: strong warning plus graceful degraded mode

Product intent:

• SRS value requires helper connectivity (installed dictionaries + SRS updates).
• If helper is missing, the extension should make this obvious immediately.
• Handcrafted JSON rules should still work without helper.

Implementation TODO:

• Add a high-visibility warning state in extension options and SRS surfaces when helper is unavailable.
• Add blocking guidance for helper-required SRS actions (init, refresh, rulegen-triggering actions).
• Keep non-SRS manual rules runtime available and clearly labeled as the degraded mode.
• Add explicit install/recovery guidance for both install orders (desktop-first and extension-first).

Open questions:

• Final warning copy/severity level and exact placement.
• Whether to provide a one-click “Retry helper connection” action everywhere.

Acceptance criteria:

• Missing helper is visible to users without opening deep diagnostics.
• SRS helper-required actions fail fast with a clear reason and recovery path.
• Handcrafted local JSON rules continue functioning when helper is unavailable.

2) Sensitive sentence history hardening (planned default OFF; current default ON)

Current-state note (verified 2026-09-02): the encounter-history popup module is enabled by default and may retain a short latest sentence excerpt for a practiced word. Those encounter records do not retain the page URL. Separate local replacement-exposure and feedback records may retain page URLs and are disclosed in the public privacy policy.

Product intent:

• Sentence history can include sensitive text and must be treated as high-risk data.
• Default behavior should minimize stored sensitive data.

Implementation TODO:

• Keep sentence-history capture disabled by default.
• Remove URL persistence from sentence/history records (no URL field stored).
• Keep exposure telemetry URL retention (srsExposureLog) explicitly documented as “retained for now” until policy changes.
• Keep processing/storage local-only and document this in user-facing privacy text.
• Add clear opt-in language for any sentence-history capture features.
• Add or confirm user controls for clearing sentence-history data.

Open questions:

• Final retention window and minimization policy (for opted-in users).
• Whether sentence excerpt storage remains enabled behind opt-in after additional research.

Completed for the 0.1.1 beta:

• Privacy and reviewer-facing wording now disclose the current local URL and sentence-excerpt behavior.

Acceptance criteria:

• Fresh install has sentence-history capture OFF.
• Stored sentence-history records do not include URL values.
• Users can understand and control sentence-history behavior from settings.

3) Helper-unavailable reliability policy (fail-fast SRS actions plus eventual feedback retry)

Product intent:

• Manual rules should remain usable offline from helper.
• SRS control operations should not silently degrade.
• Feedback/sync paths should eventually converge when helper returns.

Implementation TODO:

• For helper-dependent SRS controls, return immediate structured errors when helper is unavailable.
• Queue feedback events on the extension side so they can be retried after helper recovery.
• Persist queue state across MV3 service worker idle/restart boundaries.
• Add bounded queue limits and diagnostics visibility (queue depth, last retry, last error).
• Retry queued feedback with explicit backoff policy until success or retention limit.

Open questions:

• Queue size and retention limits.
• Exact diagnostics detail shown to end users vs advanced diagnostics only.
• Final retry/backoff timing policy.

Acceptance criteria:

• Interactive SRS controls fail fast with actionable error text while helper is down.
• Feedback generated during helper downtime is retained and eventually synced after recovery.
• Service worker wake/idle cycles do not lose queued feedback.

Post-Current-Workstream Beta Infrastructure And Dictionary Follow-Ups

Captured on 2026-08-26 after implementing local Yomitan dictionary import, per-language-pair ordered lookup stacks, acquisition guidance, and packaged GUI validation. The dictionary branch was integrated with the SRS beta checkpoint on 2026-08-27; the remaining tasks can now resume from the combined beta branch.

Highest-priority infrastructure

INFRA-01: Make the Python development and hook environment reproducible — completed 2026-08-27

Problem:

• Repository launchers currently fall through to the first available python3 when no repository virtual environment is present.
• A verified dictionary change hit a false-negative pre-push failure because Homebrew Python 3.14 did not have the project’s fsrs dependency, while the focused suite and packaged build passed under the supported Python 3.10 environment.

Completed work:

• Declared Python 3.10.16 as the preferred repository patch and Python 3.10.x as the supported workflow line, matching hosted CI.
• Added one-command development/build .venv setup and read-only checks with exact direct dependency pins.
• Made Python-backed npm workflows and the feature-state hook use the same version-gated launcher instead of silently selecting an unrelated system interpreter.
• Verified a real clean build-environment bootstrap and full packaged build.

Acceptance criteria:

• A fresh contributor setup creates the supported environment with one documented command.
• npm --prefix scripts run check and the pre-push hook select the same Python environment.
• Interpreter drift no longer requires --no-verify; the repository typecheck gate is green under the same environment.

INFRA-02: Finish packaged GUI startup diagnosis and remove blocking work — completed 2026-08-27

Owning plan:

• docs/developer/packaged_gui_startup_performance_plan.md

Resolved problem:

• Recent packaged resource-settings launches recorded approximately 224 and 487 seconds inside MainWindow construction. Fine-grained checkpoints and a process stack sample isolated the current-machine hang to synchronous QPixmap file opening for an active custom-theme image under ~/Downloads; installed dictionary/resource inventory construction was not the blocker.

Completed work:

• Added fine-grained startup checkpoints around MainWindow construction and resource-panel initialization, plus structured checkpoint extraction in the packaged measurement artifact.
• Moved custom-theme image file reads and decoding to a shared daemon loader so protected, unavailable, or slow external image paths cannot block the GUI thread. Requests are deduplicated, stale results are ignored, and source payloads are bounded at 64 MiB.
• Reused and focused an already-visible Settings dialog for repeated resource-settings activation instead of constructing nested modal dialogs.
• Added exact app-PID cleanup, p95 reporting, and optional median/p95 budget enforcement to repeatable packaged measurements.
• Rebuilt and installed both app bundles without replacing Application Support. Three installed cold launches to settings_dialog.shown passed at 1573.5 ms median / 2313.8 ms measured p95; three already-running activations passed at 254.4 ms median / 743.3 ms measured p95.

Acceptance criteria:

• The responsible blocking operation is identified with checkpoint evidence.
• Resource Settings meets the targets in the owning startup-performance plan, or a current-machine exception is explicitly documented.
• Large installed dictionaries do not make the application appear dead.

Verification evidence:

• docs/test_outputs/dev_workflow/gui_startup_performance_open_latest.json
• docs/test_outputs/dev_workflow/gui_startup_performance_activation_latest.json
• Installed-bundle build/validation and 66 focused GUI/startup tests passed.

INFRA-03: Add one-command macOS build, install, verify, and relaunch — completed 2026-08-27

Problem:

• The current manual PyInstaller, quit, ditto, comparison, and relaunch flow is error-prone; macOS can retain the previous application process during a bundle replacement.

Implemented work:

• Added supported package commands that build and validate both bundles, stop only executables running from the selected install directory, wait for clean exit, stage replacements, validate the installed copies, and optionally relaunch the main app.
• Kept app-bundle replacement separate from Application Support so user data is not part of the install operation.
• Added focused process-targeting, replacement, user-data preservation, and workflow-contract tests; the pinned-environment packaged build passes.

Acceptance criteria:

• One documented command produces and launches the same bundle that passed validation.
• The command detects a stale running process and reports each lifecycle step.
• Verification does not rely on recursive directory comparison that follows framework symlink loops.

Completed verification:

• Ran the install workflow repeatedly against /Applications; installed bundle validation passed, the live app opened with the existing custom theme and dictionary data, and the user confirmed the preserved-data smoke.

INFRA-04: Restore a fully green repository typecheck gate — completed 2026-08-27

Problem:

• The pinned, supported environment removed interpreter/tool-version drift and exposed 8 mypy errors at NumPy .tolist() boundaries in two en-es compiled-scoring files.

Completed work:

• Added explicit, runtime-neutral narrowing for arrays whose one-dimensional shapes and scalar dtypes were already established by construction.
• Ran the required en-es benchmark, quality gate, triage extraction, and focused scoring tests. Benchmark metrics remained at the known baseline; the separately tracked rulegen quality-floor policy remains red.
• Verified the full repository safety gate, including all 838 tests, mypy over 188 source files, strict style, Windows parity, docs, and LP conformance.

Acceptance criteria:

• npm --prefix scripts run check completes normally in the bootstrapped environment.
• Ordinary pre-push operation no longer needs a type-debt exception.

Dictionary resilience and maintainability

DICT-01: Add a redistributable large-dictionary performance fixture — completed 2026-08-27

Implementation TODO:

• Generate a synthetic Yomitan format-3 archive large enough to exercise multi-bank import, indexing, repeat import, lookup, and cancellation costs.
• Add bounded import/lookup performance reporting without committing commercial dictionary data or using local Daijirin files as CI inputs.

Completed work:

• Added a deterministic Yomitan format-3 generator and local quality command covering multi-bank import/indexing, exact repeat import, lookup latency, cancellation, and partial-install cleanup.
• Kept correctness failures unconditional while making machine-sensitive timing budgets explicit opt-ins for stable local or CI runners.
• Documented the command and retained-artifact option in docs/developer/yomitan_dictionary_performance.md.

DICT-02: Add dictionary health, recovery, and source visibility — completed 2026-08-27

Implementation TODO:

• Detect missing, incompatible, or corrupt installed dictionary artifacts and offer a clear reimport/repair path.
• Preserve the compact dictionary title already shown with displayed definitions, and add clearer source/health details where recovery requires them without exposing local filesystem paths.
• Consider exporting/importing dictionary-stack assignments while explicitly excluding dictionary contents.

Completed work:

• Added bounded metadata, manifest, path, schema, and SQLite readability probes. Settings starts them only after the panel renders, on a daemon worker; normal popup lookup and app startup do not hash archives, walk dictionary trees, or run database integrity scans.
• Added learner-facing Healthy, Needs repair, and Incompatible states without showing managed filesystem paths. Broken configured dictionaries are removed from the usable stack after the check and continue to fall through safely.
• Added a confirmed Reimport flow that requires the same original ZIP, rebuilds and validates a staged local copy, atomically replaces managed files, keeps all language-pair assignments, and never changes the user’s source ZIP.
• Kept compact popup source-title behavior unchanged. Export/import of stack assignments remains a separate backup/portability UX decision; dictionary contents would remain excluded if that workflow is added.

Completed integration checkpoint

INTEGRATION-01: Merge the dictionary work into the beta line while fresh — completed 2026-08-27

Completed work:

• Merged the coherent dictionary history into codex/beta-integration with the SRS beta checkpoint as the other parent.
• Passed the combined focused dictionary/word-info/extension suite, changed-file gate, feature-state audit, SRS quality harness, and validated packaged build.
• Preserved prior real-extension testing of import, per-pair ordering, first-match fallback, and displayed source identity; final release-candidate smoke remains part of the CWS upload checklist.

Recorded outcome:

• Dictionary, popup, sentence-density, and beta-release work coexist on one tested integration branch.
• The single feature-state merge conflict was resolved from current product intent, and the missing cited SRS evidence artifact was regenerated and committed.

Deliberately deferred dictionary expansion

Do not treat the following as beta blockers unless testing reveals a concrete need:

• merging definitions from multiple dictionaries instead of first-match fallback;
• Yomitan image/media import;
• per-profile dictionary stacks instead of the current global-per-pair model;
• a full curated in-app dictionary catalogue or automatic commercial-data download flow.

Backlog Migrated From README

GUI app UI overhaul

• Do a full UX pass across all major GUI screens (navigation, hierarchy, spacing, and affordances).
• Normalize layout density and visual consistency across locales, themes, and window sizes.
• Simplify settings information architecture so language packs, profiles, and SRS controls are easier to find.
• Define and implement keyboard shortcuts for profile/ruleset workflows after the Manage Profiles UX architecture is finalized.
• Fix squished UI elements in settings of core app.
• Fix ghost ruleset populated in ruleset selection UI in core app when no ruleset exists.

Guide completion

• Complete the manual end-to-end (all sections finalized, screenshots added, and setup steps verified against current UI labels).
• Keep guide steps synchronized with extension/core app releases when labels or flow change.

Ruleset display/highlight settings

• Move manual rules display/highlight controls into per-ruleset settings (not one shared/manual-only setting).
• Per-ruleset settings:
  • Display
  • Highlight replaced words (click to toggle original)
  • Highlight color

CJK / no-space languages

• Detect whether input is likely a no-space language (CJK) using lightweight heuristics.
• If CJK, choose between:
  • character/n-gram tokenization with a trie that matches sequences, or
  • exact substring replacement without token boundaries.
• Keep exact substring mode as a user-selectable fallback for mixed-language text.

Replacement pacing/sensitivity controls

• Page replacement density settings now apply to the merged SRS/manual runtime output and use explicit standard defaults.
• Consider limiting replacements per sentence.
• Add settings to adjust sensitivity/strictness for the rules above.

S-set visibility/review UX

• Add a list view of all the words currently in S.
• Consider a dedicated study/review view for S.
• Move Run sampled rulegen (5)... guidance into an Advanced diagnostics-only category in setup docs/UI (not baseline setup flow).

Japanese script quality

• Check and improve the accuracy of generated romaji for Japanese words.

Rule generation quality

• Execution-order reference for the current rulegen workstream:
  • docs/developer/rulegen_workstream_execution_order.md
• Improve rulegen quality by making generation/scoring shallower and higher precision.
• Improve SRS rulegen quality (helper-published rules) for better pedagogical precision and fewer broad/ambiguous outputs.
• en-ja now uses strict JMdict reading match (surface + reading from word_package); targets with no reading-matched entry currently stay in S but emit no rules.
• Evaluate a disposal/pruning policy for those unmatched S targets (for example, remove or quarantine after repeated misses).
• Add reverse-check scoring when reverse dictionaries are available:
  • score/penalize candidate rules by source->target consistency in reverse lookup.
  • implementation spec: docs/rulegen/reverse_check_scoring_phase1.md
  • follow-up: tune exact-hit ambiguity penalty against reverse_check_total; the first bounded en-es experiment did not beat existing miss/far penalty settings, but the signal is now implemented and harness-exposed.
• Add the next low-hanging additive en-es rulegen signals before or around the broad sweep:
  • next provenance / competition signals beyond the first implemented late_sense_clean_earlier_competition_penalty:
    • richer uses of target_provenance
    • richer uses of gloss_provenance
    • richer uses of sense_provenance
    • richer uses of kaikki_policy_shadow
  • per-family Kaikki demotion strengths rather than one coarse family block
  • gloss-decay shape exposure rather than only gloss-decay weight
  • narrow short-phrase admission for lexical verb phrases instead of global multiword admission
  • execution-order reference: docs/developer/rulegen_workstream_execution_order.md
• Add sense-risk penalties for ambiguous/specialized senses:
  • use lexical cues/metadata to demote risky polysemic or niche senses.
  • Kaikki/Wiktionary follow-up for en-es: use topics/categories/tags to demote domain-specific lexical senses after structural candidate recovery is fixed first (for example presentar -> table, plaza -> bullring).
• Add a formal Kaikki/Wiktionary gloss-format investigation and robust normalization/splitting pass for en-es:
  • classify real raw gloss structures before adding more pair-specific heuristics,
  • preserve sense order and metadata while recovering broad early lexical candidates such as to happen, to occur and part; section; portion; share.
• Add runtime apply-time polysemy safeguards (pair-aware):
  • abstain from replacement for high-risk ambiguous matches when confidence/sense margin is weak,
  • optional local-context heuristics around the matched source token,
  • structured runtime diagnostics for skipped replacements and reason codes.
• Add multi-source agreement bonus:
  • increase confidence when a candidate mapping is supported by multiple independent resources.
• Add true lexical-frequency signals for emitted source candidates:
  • move en-es beyond gloss-order-as-frequency,
  • evaluate source-side English lexical frequency,
  • later evaluate source-target frequency-gap features carefully rather than assuming “common beats specific.”
• Add trait-conditioned rulegen profiles driven by runtime-computable features rather than human tags:
  • emit per-case trait vectors in benchmark artifacts,
  • define a small bank of named profiles,
  • analyze which profiles win by feature region,
  • later route runtime rulegen through an interpretable profile selector,
  • planning spec: docs/rulegen/trait_conditioned_rulegen_profiles.md.
• Generalize the benchmark/resource contract beyond the current en-es reference lane:
  • clean up generic resource naming so non-FreeDict providers do not leak through generic fields,
  • define a normalized translation-pack contract for benchmark/helper/adapter code,
  • move installed translation packs to a manifest-backed compiled-artifact model with SQLite as the preferred canonical runtime format,
  • treat provider-native raw downloads/extraction trees as build inputs rather than runtime contracts, and delete them after successful build by default,
  • treat app-managed GUI/helper/native-host/tooling surfaces as free to rename/remove now rather than preserving legacy freedict_* or TEI-first naming for compatibility,
  • apply the same manifest-backed compiled-artifact model to existing frequency and embedding pack flows, not only translation packs,
  • require any new data-source onboarding to use pack-id roots + manifest + canonical compiled artifact + raw cleanup by default,
  • generalize the compiled pair-context boundary,
  • use de-en as the first additional translation LP once the contract is clean enough,
  • use German frequency workflow polish as the first missing data-source lane that unlocks multiple LPs,
  • planning specs:
    • docs/developer/data_source_normalization_architecture.md
    • docs/developer/data_source_normalization_execution_order.md
    • docs/developer/language_pair_generalization_roadmap.md
    • docs/language_pairs/de_en_workstream_roadmap.md
• Improve benchmark portability before the large broad sweep:
  • optionally add single-file archive/import ergonomics on top of the existing portable bundle export/replay flow if directory transfer becomes annoying.

Pair-specific morphology expansion

• Current paired morphology expansion is intentionally narrow (en-es noun plural source -> target surface mapping).
• Add explicit morphology resolvers per LP (en-de, es-en, and future pairs) and extend beyond plural nouns.

Embeddings scoring integration

• Hook embeddings into rule-generation scoring (downloads and one-time conversion are wired; scoring integration is pending).
• Keep embeddings as a secondary signal only:
  • require a real active embedding_provider,
  • validate pair-specific impact before exposing as a meaningful sweep dimension for en-es.

Planned Milestones (Migrated From README)

1. Persist all GUI knowledge inside profiles/rulesets:
   • Store per-profile dictionary selection (mono vs cross-lingual) and language choices.
   • Store synonym settings (thresholds, embeddings) per profile or ruleset where appropriate (currently global).
2. Sync profiles/rulesets into clients:
   • Export active profile + ruleset list + language pack selection to Chrome/BD.
   • Add profile/ruleset switcher in extension/plugin settings.
   • Allow enabling/disabling multiple manual rulesets under a selected profile (not only one active manual ruleset).
   • Persist per-ruleset display settings:
     • Display
     • Highlight replaced words (click to toggle original)
     • Highlight color
3. Finish language pack UX polish:
   • Pack-specific validators for edge layouts.
   • Clear handling for external/manual paths vs. app-managed files.
   • Decide whether to phase out broad manual file-path selection entirely; current expectation is probably yes unless a concrete user need survives.
   • Re-enable Wiktionary when we are ready to handle large downloads.
4. Add language selection controls tied to profiles/rulesets:
   • Monolingual vs cross-lingual toggle per profile or per ruleset.
   • Persist target/source language choices for bulk generation.
5. Scale large pack handling:
   • Background indexing for large packs (progress + cancel).
   • Optional cached indexes for fast reloads.
6. Add per-rule exception patterns or context gates if needed.
7. Add streaming/liveness adapter for live text replacement.
8. Localize the BetterDiscord plugin for multiple languages.
9. Consider larger Sigma symbol spaces for Share Code to shorten codes.