Scalellm

What's Changed
* build: fix build error with gcc-13 by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/264
* kernel: upgrade cutlass to 3.5.0 + cuda 12.4 for sm89 fp8 support by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/265
* cmake: define header only library instead of symbol link for cutlass and flashinfer by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/266
* feat: added range to support Range-for loops by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/267
* kernel: added attention cpu implementation for testing by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/268
* build: added nvbench as submodule by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/269
* build: upgrade cmake required version from 3.18 to 3.26 by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/270
* ci: build and test in devel docker image by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/272
* ci: use manylinux image to build wheel and run pytest by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/271
* attention: added tile logic using cute::local_tile into cpu attention by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/273
* kernel: added playground for learning and experimenting cute. by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/274
* feat: added rope scaling support for llama3.1 by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/277
* update docs for llama3.1 support and bump up version by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/278


**Full Changelog**: https://github.com/vectorch-ai/ScaleLLM/compare/v0.1.6...v0.1.7

What's Changed
* alllow deploy docs when triggered on demand by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/253
* [model] support vision language model llava. by liutongxuan in https://github.com/vectorch-ai/ScaleLLM/pull/178
* dev: fix issues in run_in_docker script by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/254
* dev: added cuda 12.4 build support by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/255
* build: fix multiple definition issue by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/256
* fix: check against num_tokens instead of num_prompt_tokens for shared blocks by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/257
* bugfix: fix invalid max_cache_size when device is cpu. by liutongxuan in https://github.com/vectorch-ai/ScaleLLM/pull/259
* ci: fail test if not all tests were passed successfully by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/263
* Revert "[model] support vision language model llava. (178)" by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/262


**Full Changelog**: https://github.com/vectorch-ai/ScaleLLM/compare/v0.1.5...v0.1.6

Major changes
* added stream options to include usage info in response
* fix multiple gpu cuda graph capture issue

What's Changed
* feat: added include_usage into stream options for stream scenarios by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/243
* feat: added unittests for openai server by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/244
* [minor] use available memory to caculate cache_size by default. by liutongxuan in https://github.com/vectorch-ai/ScaleLLM/pull/245
* refactor: only do sampling in driver worker (rank=0) by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/247
* fix multiple devices cuda graph capture issue by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/248
* revert torch.cuda.empty_cache change by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/249
* ci: added release workflow by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/250
* fix workflow by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/251
* fix: pass in secrets for workflow calls. by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/252


**Full Changelog**: https://github.com/vectorch-ai/ScaleLLM/compare/v0.1.4...v0.1.5

Major changes
* Added logprobs for completion and chat apis
* Added best_of for completion and chate apis
What's Changed
* feat: added openai compatible logprobs support by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/232
* feat: added logprobs support for legacy completion api by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/233
* feat: added logprobs for grpc server by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/234
* feat: added best_of functionality for completion apis by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/236
* feat: added token_ids into sequence output for better debuggability. by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/237
* feat: added id_to_token for tokenizer to handle unfinished byte sequence, ending with "�" by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/238
* refactor: split pybind11 binding definitions into seperate files by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/239
* feat: added logprobs support for speculative decoding by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/240
* feat: added synchronization for batch inference by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/241
* feat: added '__repr__' function for scalellm package by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/242


**Full Changelog**: https://github.com/vectorch-ai/ScaleLLM/compare/v0.1.3...v0.1.4

Major changes
* Model arg hotfix for llama3
* Added more help functions
What's Changed
* fix: load vocab_size first then use it to decide model type for model sharing between llama3, llama2 and Yi. by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/230
* feat: added with statement support to release memory and exposed help function for tokenizer by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/231


**Full Changelog**: https://github.com/vectorch-ai/ScaleLLM/compare/v0.1.2...v0.1.3

Major changes
* set up github pages for docs https://docs.vectorch.com/
* set up whl repository to host published whls: https://whl.vectorch.com/
* support pip install with different versions: for example: `pip install scalellm -i https://whl.vectorch.com/cu121/torch2.3/`
* added latency and system metrics
* added initial monitoring dashboard.
* bug fix for decoder, rejection sampler, and default value for llama2

What's Changed
* ci: added workflow to publish docs to GitHub Pages by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/206
* docs: added docs skeleton by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/207
* docs: fixed source directory and added announcement by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/208
* feat: added monitoring docker compose for prometheus and grafana by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/209
* feat: Added prometheus metrics by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/210
* feat: added token related latency metrics by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/211
* fix: fix weight load issue for fused qkv and added more unittests for weight loading by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/213
* fix: use a consistent version for whl by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/214
* refactor: move setup.py to top level by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/217
* feat: carry over prompt to output for feature parity by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/218
* added missing changes for carrying over prompt by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/219
* fix: set correct default value of rope_theta for llama2 by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/223
* feat: convert pickle to safetensors for fast loading by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/224
* docs: add livehtml for docs development by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/225
* fix: use error instead of CHECK when prompt input is empty by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/226
* fix: avoid tensor convertion for converted ones. by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/228
* feat: added time_to_first_token and inter_token metrics for both stream and non-stream requests by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/227
* fix: decode ending tokens one by one to handle unfinished tokens by guocuimi in https://github.com/vectorch-ai/ScaleLLM/pull/229


**Full Changelog**: https://github.com/vectorch-ai/ScaleLLM/compare/v0.1.1...v0.1.2