Langroid

* Adds top-level imports for LanceDocChatAgent.
* Moves lance_tools.py out of langroid.agent.special.lance_rag to avoid a dependency cycle.
* Updates Ruff config (lint prefix removed).
* ChatAgent token stats, now correctly use agent.llm.config instead of agent.config.llm
(this makes a difference only when llm is overriden globally via --m option for example; previously the LLM model was being shown based
on the Agent's original config, ignoring this global override)

UrlLoader: recognize content from header when URL does has pdf, doc or docx but does not end with these suffix/extensions

LanceDB-related bug fixes:
- Use limit=None in various queries to ensure we get full collections list and get all documents
- Fix add_documents() bug where ids were not being indexed correctly

Chainlit, LiteLLM version bumps

Make more dependencies optional:
* `chromadb` (has some strange sqlite requirements that mess with dockerization in some envs)
* `mkdocs` (only need this for doc generation of course)
* `huggingface-hub`
See `pyproject.toml` for how to install these as extra dependencies using, e.g. `pip install "langroid[chromadb,transformers]"`

DocChatAgent:
* set default embedding config based on whether sentence_transformer is available
* set default `cross_encoder_reranking_model` to "cross-encoder/ms-marco-MiniLM-L-6-v2" if sentence_transformer available, else ""