Dgenerate

Fix unquoted `$HF_TOKEN` value in Stable Diffusion 3 and DeepFloyd console UI recipe templates, causing issues
with the `\setp` directive used in the template. Needed to have quotes around `$HF_TOKEN` in the template due
to changes in how and when the template renderer decides to automatically shell quote a form value.

Fix syntax highlighter failing to immediately re-highlight after Undo/Redo, Cut/Paste, Delete operations
preformed from the UI menus without using hotkeys. Syntax highlighting would not resume until the next
keypress, temporarily displaying unhighlighted text.

Documentation typo fix for `letterbox` image processor.

Handle out of memory conditions for pipelines / image processors / and prompt weighters more thoroughly.

Preform memory cleanup as best as possible so that dgenerate can continue to run in REPL mode without
VRAM memory leaks after an OOM condition occurs.

And a few documentation additions and corrections.

Update DeepFloyd insert recipe form in the Console UI to use the newer widgets for the model entry fields, forgot this one.

Catch potential image processor OOM which might result in ungraceful exit if the GPU runs out of memory while using an argument such as `--post-processors`, `--seed-image-processors`, `--control-image-processors`, or `--mask-image-processors`

1.) Prompt weighting

Stable Diffusion Web UI / CivitAI / ComfyUI / InvokeAI style prompt weighting syntax is now implemented
through the argument `--prompt-weighter` and a new plugin class `PromptWeighter`.

This argument allows selecting a specific prompt weighter plugin to preform prompt parsing, token weighting, and prompt embed generation.

Prompt weighters circumvent restrictions on prompt length, which is typically `77` tokens.

Currently there are two prompt weighters implemented, `compel` and `sd-embed`.

Their documentation can be seen with: `dgenerate --prompt-weighter-help compel sd-embed`

The `compel` prompt weighter plugin uses the [compel library](https://github.com/damian0815/compel) to support [InvokeAI](https://github.com/invoke-ai/InvokeAI) style prompt weighting syntax, and also supports automatically translating your prompt from Stable Diffusion Web UI syntax into `compel` syntax via the plugin argument `syntax=sdwui`.

The `compel` prompt weighter supports Stable Diffusion 1/2, and Stable Diffusion XL model types:

* `--model-type torch`
* `--model-type torch-pix2pix`
* `--model-type torch-upscaler-x4`
* `--model-type torch-sdxl`
* `--model-type torch-sdxl-pix2pix`

The `sd-embed` prompt weighter uses the [sd_embed library](https://github.com/xhinker/sd_embed) to support Stable Diffusion Web UI / CivitAI / ComfyUI prompt weighting syntax. This prompt weighter is currently using code from a fork I have VRAM optimized, [found here](https://github.com/Teriks/sd_embed)

The `sd-embed` prompt weighter supports Stable Diffusion 1/2, Stable Diffusion XL, and Stable Diffusion 3 model types:

* `--model-type torch`
* `--model-type torch-pix2pix`
* `--model-type torch-upscaler-x4`
* `--model-type torch-sdxl`
* `--model-type torch-sdxl-pix2pix`
* `--model-type torch-sd3`

For more information about prompt weighting with dgenerate, see: [Prompt Weighting and Enhancement](https://github.com/Teriks/dgenerate/tree/master?tab=readme-ov-file#prompt-weighting-and-enhancement) in the readme.

2.) Prompt length warnings

Warn the user about prompt truncation when not using a `--prompt-weighter` implementation, this will warn you when your prompt is too long for the models tokenizer max sequence length. If you get this warning you should select a `--prompt-weighter` implementation, or reduce the length of your prompt.

3.) New image processors & image processor fixes

Implemented new image processors:

* `anyline` (MistoLine Control Any Line preprocessor, see: https://huggingface.co/TheMistoAI/MistoLine)
* `lineart-standard` (Basic non model based variant of lineart / lineart-anime)
* `teed` (TEED tiny efficient edge detector)
* `zoe` (ZoeDepth depth detector)

Fixed an issue with the `detect-resolution` argument of various image processors not returning the image to its original size
after detection was preformed.

4.) Improved extensibility

All plugin types implemented by dgenerate can now be loaded through `--plugin-modules`, and `\import_plugins` inside config scripts.

Previously only image processors, config directives, and config template functions supported external extensibility, in a hard coded manner internally. Now sub-commands and prompt weighters can be written as external plugins as well.

An architecture change has been made so that going forward any plugin loader class can automatically discover classes through `--plugin-modules` and `\import_plugins` without any extra work having to be done internally to support it.

New plugin code examples for sub-commands and prompt weighters can be found in the [writing_plugins examples folder](https://github.com/Teriks/dgenerate/tree/master/examples/writing_plugins)

5.) Improved argument validation

Better argument validation when using plugin help arguments such as `--image-processor-help` etc.. this corrects the problem
of the CLI allowing you to mistype something such as `dgenerate --plugin-paths plugin.py --image-processor-help my-processor`, resulting in your plugin not being listed because the correct command is infact:

`dgenerate --plugin-modules plugin.py --image-processor-help my-processor`

Previously the only validated argument was the plugin help argument, with all unknown arguments being ignored. Now when one of these arguments is encountered, the other arguments are validated, meaning it will raise an error on `--plugin-paths` for example as that is not a valid argument name.

6.) New template functions for image dimension manipulation

Added config template functions `align_size`, `pow2_size`, `size_is_aligned`, and `size_is_pow2` for checking and adjusting the alignment of image dimension values inside of config scripts.

See: `dgenerate --functions-help align_size pow2_size size_is_aligned size_is_pow2` for their documentation.

Or: `\functions_help align_size pow2_size size_is_aligned size_is_pow2` if you are working with a config file or the Console UI.

7.) Console UI, insert recipe form overhaul

The insert recipe form now scrolls to support more form fields, and now supports selecting image processor implementations by name where applicable.

Image processor selection includes dynamic form controls for specifying image processor argument values, and a help button that you can click to show the documentation for the selected image processor.

The insert recipe form can now be used to select `dgenerate` argument values for `--post-processors`, `--seed-image-processors`, `--control-image-processors`, and `--mask-image-processors` using this new feature.

Two new recipes named `Generic Image Process` and `Generic Image Process (to directory)` have been added which make use of this feature to build an `\image_process` directive that can be inserted into your config.

There have been *many* other changes to the insert recipe form, including additional fields / access to more options, toggles, drastically improved input validation, and the ability to select a file OR directory for applicable model arguments.

8.) Console UI, new insert / build URI options

Two new URI insertion features have been added to the Console UI.

Using `Edit -> Insert Image Seed URI` will allow you to use a form with file select dialogs to build an `--image-seeds` URI, you can specify each component of the URI (seed image, inpaint mask, control image) and a URI with proper syntax will be formatted for you and inserted into the config. You may also specify the per image-seed `resize` & and `aspect` correct resize argument values (the default value for aspect correct resizing is `True`)

Using `Edit -> Insert Image Processor URI` will allow you to insert an image processor URI in a similar fashion to inserting a Karras Scheduler URI (a previously existing feature).

A form is generated based on the selected image processors arguments.

Inserting with this form will insert an Image Processor URI formatted with all the provided arguments.

9.) Console UI, version info / latest version check

The Console UI now shows the current software version next to the `Help -> Homepage` menu item.

If you are connected to the internet upon launch and github is reachable, a check for the latest release will occur,
if there is a newer release, a link to it will be added to the bottom section of the `Help` menu.

It will be in the format `Newer Release Available! (vX.X.X)`, and clicking it will open your web browser to the page
for the latest release on github.

This way you can check if there is a software update available from the UI.


10.) Image generation improvements.

Stable Cascade can now produce images with output dimensions aligned to `128` instead of only power of 2 images.

Stable Diffusion 3 can now preform VAE tiling `--vae-tiling` on images with output dimensions aligned to `16` instead of only power of 2 images.

11.) Bugfix model loading from directories

Fix bug in determining diffusers model loading method, which caused loading diffusers models out of directories to stop working.

12.) Dependency updates

Cleaner shutdown when ``ctrl-c`` is used from the terminal to exit the Console UI tkinter app, especially on Linux.

Fix stuck forever behavior when ``dgenerate --console file.dgen`` was used from the terminal and ``file.dgen`` does not exist. Caused
by ``sys.exit`` call on the main thread without the tkinter main loop thread exiting, leaving it waiting forever for events and preventing
the process from shutting down.

1.) Support for Stable Diffusion 3 ``--model-type torch-sd3``, supports txt2img, img2img, multiple Control Nets, multiple LoRAs (scale not supported yet by diffusers). Inpainting is not supported currently, neither are textual inversions. Enabling ``--vae-tiling`` restricts output to power of 2 sizes. All output / input needs to be aligned by 16 pixels normally. See: [Examples](https://github.com/Teriks/dgenerate/tree/master/examples/stablediffusion3)

New SD3 Arguments:

* ``--sd3-max-sequence-length`` (max accepted tokens by the T5 encoder, text encoder number 3)
* ``--sd3-second-prompts`` (text encoder number 2 prompts)
* ``--sd3-third-prompts`` (text encoder number 3 prompts)

2.) Support for specifying the text encoders involved in a diffusion pipeline using typical dgenerate URI syntax. All text encoders of any model can be specified, including individual encoders leaving others as defaults, using the new arguments ``--text-encoders`` for the main pipeline and ``--text-encoders2`` for the SDXL Refiner or Stable Cascade Decoder. See: [Specifying Text Encoders](https://github.com/Teriks/dgenerate?tab=readme-ov-file#specifying-text-encoders) for syntax. This is to better support single file loads when using Stable Diffusion 3. Single file loads for text encoders are not supported and they must be loaded from a huggingface repository or folder on disk containing the model + configuration.

3.) New config directive for clearing the Text Encoder in memory model cache manually: ``\clear_text_encoder_cache``

4.) New config function / template function for converting the enums that exist in the template variables ``last_model`` and ``last_dtype``
to a value that is usable for a dgenerate argument. New functions are: ``format_model_type(last_model)`` and ``format_dtype(last_dtype)``,
they accept the enum values as arguments and return a string.

5.) Improved syntax highlighting in the Console UI for directives which accept a variable name as their first argument such as ``\set``, directive can now be immediately followed by a line continuation before the variable name, and specifying the variable name with a template now highlights and behaves correctly.

6.) Added Stable Diffusion 3 recipe to Console UI recipes menu. Improved ``auth_token`` recipe code for Stable Diffusion 3 and Deep Floyd.

7.) Bug fix for the enter key not working in the Console UI find/replace dialog and getting the dialog into a bad state, enter action is now (find next).

8.) Bug fix to (in memory) cache accounting of user specified sub models such as UNets, VAEs, Text Encoders, and Control Nets. Previously
the memory used by these models was not being tracked properly as they entered / exited GPU memory, leading to non optimal garbage collection according to memory constraint settings for these models.

9.) New ``letterbox`` image processor for letterboxing input / output images, see: [Image Processors](https://github.com/Teriks/dgenerate?tab=readme-ov-file#image-processors), and ``dgenerate --image-processor-help letterbox``