Queuei Handbook

Queuei turns a stream of long-form video into a queryable knowledge base. Every video in a monitored playlist flows through a pipeline that fetches a transcript, runs a configurable Gemini prompt, generates a 3072-dimension embedding, and writes one document to MongoDB. The dashboard reads those documents back as browsable Stacks, powers semantic Intel Chat, and renders relationship graphs.

Beyond video, the same machinery is generalised: define a prompt, attach a schedule, and Queuei runs it every morning and paints the result as its own dashboard tab. That is how the built-in NASDAQ stock analysis and any custom Dashboard Feed work.

Two Django apps sit on top of two databases.

Two apps

• records — the user-facing frontend: auth, dashboard, Intel Chat, Nexus graph, Command Center, Settings, Cron manager, and this documentation portal.
• tasks — the ingestion & scheduling backend: the YouTube pipeline, stock & feed pipelines, the APScheduler runtime, and utility endpoints.

Two databases

MongoDB connections are raw pymongo. get_db_collection() in records/views.py opens a fresh connection per request; the pipeline shares a module-level client from tasks/config.py. There is deliberately no ORM over Mongo.

Every video runs through YouTubeLLMPipeline (tasks/script_custom/youtube_llm_pipeline.py). One document out per video in.

1. 1

   Discover

   yt-dlp lists recent videos from every playlist in YOUTUBE_PLAYLIST_IDS, up to PLAYLIST_FETCH_LIMIT each. Already-processed IDs are skipped.

2. 2

   Transcript

   Tries the local cache, then the source chain (see Transcript Sources). Successful pulls are written to TranscriptCache so retries never re-fetch.

3. 3

   LLM action

   The task's prompt_template is formatted with {transcript} and sent to Gemini. The response is the intelligence body.

4. 4

   Embed

   The LLM output is embedded with gemini-embedding-001 into a 3072-dim vector.

5. 5

   Persist

   A document — metadata + LLM result + embedding + any EXTRA_DOCUMENT_ARGS — is inserted into the task's target collection in MongoDB.

Concurrency is bounded by EXECUTOR_WORKERS with an INBETWEEN_TASK_SLEEP pause between batches — the throttle that keeps Gemini rate limits happy. Trigger a run with POST /tasks/queuei/<task_key>/.

Transcripts are fetched through a fallback chain — the first source that returns text wins, and the result is cached.

Runtime behaviour lives in PostgreSQL, editable from the UI — no redeploy to change how the pipeline runs.

A single APScheduler BackgroundScheduler (UTC) reads CronJob rows and fires them. Two job types:

• pipeline — invokes a Python pipeline class directly (e.g. the YouTube or stock pipeline).
• endpoint — dispatches an internal request to a relative URL with the task secret header (used by feeds).

MongoDB documents are schemaless; a record typically carries video metadata, the LLM result, entity annotations, and a 3072-dim embedding.

Secrets and infrastructure config live in .env, never in the database.

Pipeline-behaviour vars (playlist IDs, fetch limit, AI model, workers, sleep, extra args) are only seeds — once the Settings page runs, the DB value takes over.

New signups are is_active=False until a supervisor approves them. Authorisation is by Django group plus the superuser flag.

Unauthorised authenticated users see access_denied.html rather than a redirect.

Local development:

# run the app
python manage.py migrate
python manage.py runserver

# after model changes
python manage.py makemigrations

# build & run the container (connects to production DBs)
./build.sh

The container entrypoint runs migrate then gunicorn on port 8000 with --env-file .env. The scheduler boots inside the web process and hot-loads cron jobs from the DB.