Firecrawl

Get structured web data with /extract

We’re excited to announce the [release of **/extract**](https://www.firecrawl.dev/extract) - get data from any website with just a prompt. With /extract, you can retrieve any information from anywhere on a website without being limited by scraping roadblocks or the typical context constraints of LLMs.


No more manual copy-pasting, broken scraping scripts, or debugging LLM calls. - it’s never been easier to **enrich your data**, **create datasets**, or **power AI applications** with clean, structured data from any website.

Companies are already using extract to:
- Enrich CRM data
- Streamline KYB processes
- Monitor competitors
- Supercharge onboarding experiences
- Build targeted prospecting lists

Instead of spending hours manually researching, fixing broken scrapers, or piecing together data from multiple sources, simply specify what information you need and the target website, and let the Firecrawl handle the entire retrieval process.

Specifically, you can:
- Extract structured data from entire websites using URL wildcards (https://example.com/*)
- Define custom schemas to capture exactly what you need—from simple product details to complex organizational structures
- Guide the extraction with custom prompts to ensure the LLM focuses on your target information
- Deploy anywhere with comprehensive support for Python, Node, cURL, and other popular tools. For no-code workflows, just connect via Zapier or use our API to set up integrations with other tools.

This versatility translates into a wide range of real-world applications—enabling you to enrich web data for just about any use case.

Limitations - (and the road ahead)
- Let's be honest - while /extract is pretty awesome at grabbing web data, it's not perfect yet. Here's what we're still working on:
- Big sites are tricky - It can't (yet!) grab every single product on Amazon in one go
- Complex searches need work - Things like "find all posts from 2025" aren't quite there
- Sometimes, it's a bit quirky - Results can vary between runs, though it usually gets what you need
- **But here's the exciting part: we're seeing the future of web scraping take shape**

Try it out
Curious to try /extract out for yourself?
Visit our [playground](https://www.firecrawl.dev/playground?mode=extract) to try out /extract - you get **500,000 tokens for free**
Dive into our [Extract Beta documentation](https://docs.firecrawl.dev/features/extract-beta) for detailed technical guidance and API reference
Want a no-code solution? Connect /extract to thousands of applications through our enhanced [Zapier integration](https://zapier.com/apps/firecrawl/integrations)


That's all for now! Happy Extracting from the whole Firecrawl team 🔥


**Full Changelog**: https://github.com/mendableai/firecrawl/compare/v.1.3.0...v1.4.0

v.1.3.0
What's Changed
* feat: new snips test framework (FIR-414) by mogery in https://github.com/mendableai/firecrawl/pull/1033
* (feat/extract) New re-ranker + multi entity extraction by nickscamara in https://github.com/mendableai/firecrawl/pull/1061
* __experimental_streamSteps by nickscamara in https://github.com/mendableai/firecrawl/pull/1063


**Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.2.1...v.1.3.0

What's Changed
* Indexes, Caching for /extract, Improvements by nickscamara in https://github.com/mendableai/firecrawl/pull/1037
* [SDK] fixed none and undefined on response by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1034
* feat: use new random user agent instead of the old one by 1101-1 in https://github.com/mendableai/firecrawl/pull/1038
* (feat/extract) Move extract to a queue system by nickscamara in https://github.com/mendableai/firecrawl/pull/1044

/extract (beta) changes
* We have updated the /extract endpoint to now be asynchronous. When you make a request to /extract, it will return an ID that you can use to check the status of your extract job. If you are using our SDKs, there are no changes required to your code, but please make sure to update the SDKs to the latest versions as soon as possible.

* For those using the API directly, we have made it backwards compatible. However, you have 10 days to update your implementation to the new asynchronous model.

* For more details about the parameters, refer to the docs sent to you.

New Contributors
* 1101-1 made their first contribution in https://github.com/mendableai/firecrawl/pull/1038

**Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.2.0...v1.2.1

Changelog: https://www.firecrawl.dev/changelog#/extract-changes

/v1/search

The search endpoint combines web search with Firecrawl’s scraping capabilities to return full page content for any query.

Include `scrapeOptions` with `formats: ["markdown"]` to get complete markdown content for each search result otherwise it defaults to getting SERP results (url, title, description).

More info here [/v1/search docs](https://docs.firecrawl.dev/api-reference/endpoint/search)

What's Changed
* /extract URL trace by nickscamara in https://github.com/mendableai/firecrawl/pull/1014
* (feat/v1) Search by nickscamara in https://github.com/mendableai/firecrawl/pull/1032

**Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.1.1...v1.2.0

What's Changed
* feat(python-sdk): Make API key optional for self-hosted instances by RutamBhagat in https://github.com/mendableai/firecrawl/pull/990
* Sitemap fixes by mogery in https://github.com/mendableai/firecrawl/pull/1010
* fixed optional+default bug on llm schema by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/955
* [FIR-37] feat: extract and return favicon URL during scraping by ftonato in https://github.com/mendableai/firecrawl/pull/1018
* fix: merge mock success data by yujunhui in https://github.com/mendableai/firecrawl/pull/1013
* feat(rust-sdk): Make API key optional for self-hosted instances by RutamBhagat in https://github.com/mendableai/firecrawl/pull/991
* feat(scrapeURL/pdf): switch to MU (FIR-356) by mogery in https://github.com/mendableai/firecrawl/pull/1016

New Contributors
* ftonato made their first contribution in https://github.com/mendableai/firecrawl/pull/1018
* yujunhui made their first contribution in https://github.com/mendableai/firecrawl/pull/1013

**Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.1.0...v1.1.1

Changelog Highlights

Feature Enhancements
- **New Features**:
- Geolocation, mobile scraping, 4x faster parsing, better webhooks,
- Credit packs, auto-recharges and batch scraping support.
- Iframe support and query parameter differentiation for URLs.
- Similar URL deduplication.
- Enhanced map ranking and sitemap fetching.

Performance Improvements
- Faster crawl status filtering and improved map ranking algorithm.
- Optimized Kubernetes setup and simplified build processes.
- Sitemap discoverability and performance improved

Bug Fixes
- Resolved issues:
- Badly formatted JSON, scrolling actions, and encoding errors.
- Crawl limits, relative URLs, and missing error handlers.
- Fixed self-hosted crawling inconsistencies and schema errors.

SDK Updates
- Added dynamic WebSocket imports with fallback support.
- Optional API keys for self-hosted instances.
- Improved error handling across SDKs.

Documentation Updates
- Improved API docs and examples.
- Updated self-hosting URLs and added Kubernetes optimizations.
- Added articles: mastering `/scrape` and `/crawl`.

Miscellaneous
- Added new Firecrawl examples
- Enhanced metadata handling for webhooks and improved sitemap fetching.
- Updated blocklist and streamlined error messages.


What's Changed
* Add docs to api spec example by ericciarla in https://github.com/mendableai/firecrawl/pull/637
* [Docs] upgraded the path of the self-hosted documentation URL to `/v1`. by shige in https://github.com/mendableai/firecrawl/pull/635
* Removal of generic classnames/ids from onlyMainContent cleaning by nickscamara in https://github.com/mendableai/firecrawl/pull/638
* Improved team credits check and billing notifications by nickscamara in https://github.com/mendableai/firecrawl/pull/640
* Fixed 500 errors when JSON is badly formatted by nickscamara in https://github.com/mendableai/firecrawl/pull/648
* Better engine for wait + other params by nickscamara in https://github.com/mendableai/firecrawl/pull/649
* fix(py-sdk): removed asyncio package by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/654
* perf(js-sdk): move `dotenv` and `uuid` to `devDependencies`, fix `zod` import by MonsterDeveloper in https://github.com/mendableai/firecrawl/pull/614
* build(js-sdk): simplify build process by MonsterDeveloper in https://github.com/mendableai/firecrawl/pull/611
* fix(v0/crawl-status): don't crash on big crawls when requesting jobs from supa by mogery in https://github.com/mendableai/firecrawl/pull/653
* Manual Rate Limiter for select team ids by nickscamara in https://github.com/mendableai/firecrawl/pull/664
* O1 crawler example by ericciarla in https://github.com/mendableai/firecrawl/pull/676
* [Bug] Fixed screenshot typo and added test for fullpage screenshot by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/677
* v1/map improvements + higher limits by nickscamara in https://github.com/mendableai/firecrawl/pull/674
* Remove print statement in map by anjor in https://github.com/mendableai/firecrawl/pull/612
* fix wrong link to self host documentation by itasli in https://github.com/mendableai/firecrawl/pull/623
* feat: kubernetes example optimization by yekkhan in https://github.com/mendableai/firecrawl/pull/639
* Rust SDK 1.0.0 by mogery in https://github.com/mendableai/firecrawl/pull/689
* feat: Actions by mogery in https://github.com/mendableai/firecrawl/pull/682
* Fix the error message when trying search in v0 by nickscamara in https://github.com/mendableai/firecrawl/pull/690
* remove space in the examples/o1_web_crawler folder name by h4r5h4 in https://github.com/mendableai/firecrawl/pull/679
* o1 job recommender example by ericciarla in https://github.com/mendableai/firecrawl/pull/707
* Move auth and check credits operations into an RPC by mogery in https://github.com/mendableai/firecrawl/pull/704
* bugfix: using onlyIncludeTags and removeTags together by skeptrunedev in https://github.com/mendableai/firecrawl/pull/685
* Concurrency limits by mogery in https://github.com/mendableai/firecrawl/pull/721
* Docs: Remove wait_until_done from python-sdk example by bytrangle in https://github.com/mendableai/firecrawl/pull/728
* Improves error handler in Node SDK to return the status code by nickscamara in https://github.com/mendableai/firecrawl/pull/727
* Fixes crawl failed and webhooks not working properly by nickscamara in https://github.com/mendableai/firecrawl/pull/731
* [BUG] Fixed URLs with params by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/732
* Fixed the self host issues where methods don't work by nickscamara in https://github.com/mendableai/firecrawl/pull/733
* Make sure the entrypoint script has the correct line endings by busaud in https://github.com/mendableai/firecrawl/pull/753
* Rm cluster mode + rm fly deployments by nickscamara in https://github.com/mendableai/firecrawl/pull/754
* Fixed Issue 734 by Harsh0707005 in https://github.com/mendableai/firecrawl/pull/747
* bugfix: self-host crawling doesnt respect limit by busaud in https://github.com/mendableai/firecrawl/pull/755
* [BUG] Fixed missing error handling in JS-SDK by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/759
* [SKD] Cancel Crawl by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/760
* fixed developer.notion special case by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/762
* Spelling Corrections in README by fadkeabhi in https://github.com/mendableai/firecrawl/pull/763
* [RPC] Improvements to credit_usage rpc by nickscamara in https://github.com/mendableai/firecrawl/pull/767
* [BUG] filters failed and unknown jobs now by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/761
* [Doc] Better explained how includePaths and excludePaths work by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/766
* Update README.md by busaud in https://github.com/mendableai/firecrawl/pull/757
* ADDED : Contributors and Back to top by Ruhi14 in https://github.com/mendableai/firecrawl/pull/768
* Retries for ACUC RPC + Price credits fallback by nickscamara in https://github.com/mendableai/firecrawl/pull/773
* [BUG] added check files on crawl by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/779
* [Feat] Performance improvements crawl status filters by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/780
* Admin alerts for high usage by nickscamara in https://github.com/mendableai/firecrawl/pull/783
* Geolocation support for Firecrawl by nickscamara in https://github.com/mendableai/firecrawl/pull/784
* Return all the website metadata by nickscamara in https://github.com/mendableai/firecrawl/pull/785
* Extractor options logging v1 fix by nickscamara in https://github.com/mendableai/firecrawl/pull/788
* Update requirements.txt by rishi-raj-jain in https://github.com/mendableai/firecrawl/pull/790
* Improved /map ranking algorithm for search queries by nickscamara in https://github.com/mendableai/firecrawl/pull/798
* Fix Typos and Grammar in `SELF_HOST.md` by Mefisto04 in https://github.com/mendableai/firecrawl/pull/799
* [Bug] encoding error for special token by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/793
* [BUG-SDK] missing error in response by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/796
* examples: sales web crawler by rishi-raj-jain in https://github.com/mendableai/firecrawl/pull/797
* feat: clear ACUC cache endpoint based on team ID by mogery in https://github.com/mendableai/firecrawl/pull/807
* feat: skipTlsVerification by tomkosm in https://github.com/mendableai/firecrawl/pull/808
* feat: Batch Scrape by mogery in https://github.com/mendableai/firecrawl/pull/789
* feat: Auto Recharge Credits + Credit Packs by nickscamara in https://github.com/mendableai/firecrawl/pull/809
* Remove ph logs for single_urls by nickscamara in https://github.com/mendableai/firecrawl/pull/829
* Bump to gemini-1.5-pro-002 website_qa_with_gemini_caching.ipynb and add flash example by s-smits in https://github.com/mendableai/firecrawl/pull/739
* Add SearchApi as a Web Search Tool by SebastjanPrachovskij in https://github.com/mendableai/firecrawl/pull/628
* RM wait before interacting by nickscamara in https://github.com/mendableai/firecrawl/pull/838
* chore(README.md): use `satisfies` instead of `as` for ts example by twlite in https://github.com/mendableai/firecrawl/pull/831
* Geo-location rename to location by nickscamara in https://github.com/mendableai/firecrawl/pull/830
* concurrency limit fix by mogery in https://github.com/mendableai/firecrawl/pull/824
* [feat] Iframe support by tomkosm in https://github.com/mendableai/firecrawl/pull/855
* Fix go parser by tomkosm in https://github.com/mendableai/firecrawl/pull/856
* Support for the 2 new actions by nickscamara in https://github.com/mendableai/firecrawl/pull/858
* Adds support for mobile web scraping + mobile screenshot by nickscamara in https://github.com/mendableai/firecrawl/pull/847
* [Feat] Added remove base64 images options (true by default) by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/867
* [Fix] Prevent Python Firecrawl logger from interfering with loggers in client applications by reasonmethis in https://github.com/mendableai/firecrawl/pull/613
* [BUG] Added trycatch and removed redundancy by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/869
* Update CONTRIBUTING.md by swyxio in https://github.com/mendableai/firecrawl/pull/849
* `WebScraper` refactor into `scrapeURL` by mogery in https://github.com/mendableai/firecrawl/pull/714
* Exec js - actions by nickscamara in https://github.com/mendableai/firecrawl/pull/872
* feat(crawl): Similar URL deduplication by mogery in https://github.com/mendableai/firecrawl/pull/878
* [SDK] Added next handler for python sdk (js is ok) by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/880
* [BUG] fixes scroll action by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/881
* feat(crawl): add parameter to treat differing query parameters as different URLs by mogery in https://github.com/mendableai/firecrawl/pull/892
* fix(crawler): relative URL handling on non-start pages by mogery in https://github.com/mendableai/firecrawl/pull/893
* Redlock for sending email notifications by nickscamara in https://github.com/mendableai/firecrawl/pull/895
* feat(v1/webhook): complex webhook object w/ headers by mogery in https://github.com/mendableai/firecrawl/pull/899
* fix(sitemap): scrape with tlsclient by mogery in https://github.com/mendableai/firecrawl/pull/891
* Allows `/map` to only return links present in the sitemap by nickscamara in https://github.com/mendableai/firecrawl/pull/901
* Node SDK : Add Mobile Scraping by ad-angelo in https://github.com/mendableai/firecrawl/pull/914
* Add notebook and markdown files for two articles: mastering /scrape and mastering /crawl by BexTuychiev in https://github.com/mendableai/firecrawl/pull/918
* Extract (beta) by nickscamara in https://github.com/mendableai/firecrawl/pull/915
* Add a new project to examples that shows how to scrape Hacker News website by BexTuychiev in https://github.com/mendableai/firecrawl/pull/928
* Fix/while next loop by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/939
* Crawl not respecting the limit when ignoreSitemap is false by nickscamara in https://github.com/mendableai/firecrawl/pull/940
* Crawl fixes: fixed the n-1 bug by nickscamara in https://github.com/mendableai/firecrawl/pull/941
* fixed keyerror for data on sdk by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/943
* Submitting two articles to the blog by BexTuychiev in https://github.com/mendableai/firecrawl/pull/944
* removed microsoft from blocklist by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/958
* Add assets for the Automated Amazon Price Tracking article by BexTuychiev in https://github.com/mendableai/firecrawl/pull/946
* Fixed Prettier by nickscamara in https://github.com/mendableai/firecrawl/pull/965
* Fixes schema base model extract by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/954
* Metadata for webhooks by nickscamara in https://github.com/mendableai/firecrawl/pull/970
* Remove Block List by ericciarla in https://github.com/mendableai/firecrawl/pull/971
* feat(runWebScraper): retry a scrape max 3 times in a crawl if the status code is failure (FIR-293) by mogery in https://github.com/mendableai/firecrawl/pull/975
* feat: add scrapeOptions.fastMode (FIR-288) by mogery in https://github.com/mendableai/firecrawl/pull/973
* fix: adjust Playwright service response to match API schema expectations by AsyncArtisan in https://github.com/mendableai/firecrawl/pull/977
* Timeout fixes on user defined timeouts by nickscamara in https://github.com/mendableai/firecrawl/pull/978
* Default to pdf2md, if under 500 chars (indicating failure) use LlamaParse by ericciarla in https://github.com/mendableai/firecrawl/pull/984
* Revert to pdf parse by ericciarla in https://github.com/mendableai/firecrawl/pull/987
* [BUG] fixed title extra info by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/996
* feat-SDK/added crawl id to ws by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/994
* [SDK] Added try catch to ws message handler by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/998
* Credit usage endpoint by nickscamara in https://github.com/mendableai/firecrawl/pull/999
* Refactored Blocklist Error Messages by nickscamara in https://github.com/mendableai/firecrawl/pull/1001
* feat(js-sdk): Make API key optional for self-hosted instances by RutamBhagat in https://github.com/mendableai/firecrawl/pull/989
* docs(CONTRIBUTING.md): Add Docker Compose setup instructions by RutamBhagat in https://github.com/mendableai/firecrawl/pull/1005
* feat: Support environments without ws by dynamically importing WebSocket module with error handling by tomkosm in https://github.com/mendableai/firecrawl/pull/997
* [bug/JS-SDK]Added check for object and trycatch as workaround for 502s by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/974
* docs(credit-usage-api): add new endpoint documentation for credit usage by RutamBhagat in https://github.com/mendableai/firecrawl/pull/1003
* Improves sitemap fetching by nickscamara in https://github.com/mendableai/firecrawl/pull/1015

New Contributors
* shige made their first contribution in https://github.com/mendableai/firecrawl/pull/635
* MonsterDeveloper made their first contribution in https://github.com/mendableai/firecrawl/pull/614
* anjor made their first contribution in https://github.com/mendableai/firecrawl/pull/612
* itasli made their first contribution in https://github.com/mendableai/firecrawl/pull/623
* yekkhan made their first contribution in https://github.com/mendableai/firecrawl/pull/639
* h4r5h4 made their first contribution in https://github.com/mendableai/firecrawl/pull/679
* skeptrunedev made their first contribution in https://github.com/mendableai/firecrawl/pull/685
* bytrangle made their first contribution in https://github.com/mendableai/firecrawl/pull/728
* busaud made their first contribution in https://github.com/mendableai/firecrawl/pull/753
* Harsh0707005 made their first contribution in https://github.com/mendableai/firecrawl/pull/747
* fadkeabhi made their first contribution in https://github.com/mendableai/firecrawl/pull/763
* Ruhi14 made their first contribution in https://github.com/mendableai/firecrawl/pull/768
* rishi-raj-jain made their first contribution in https://github.com/mendableai/firecrawl/pull/790
* Mefisto04 made their first contribution in https://github.com/mendableai/firecrawl/pull/799
* s-smits made their first contribution in https://github.com/mendableai/firecrawl/pull/739
* SebastjanPrachovskij made their first contribution in https://github.com/mendableai/firecrawl/pull/628
* twlite made their first contribution in https://github.com/mendableai/firecrawl/pull/831
* reasonmethis made their first contribution in https://github.com/mendableai/firecrawl/pull/613
* swyxio made their first contribution in https://github.com/mendableai/firecrawl/pull/849
* ad-angelo made their first contribution in https://github.com/mendableai/firecrawl/pull/914
* BexTuychiev made their first contribution in https://github.com/mendableai/firecrawl/pull/918
* AsyncArtisan made their first contribution in https://github.com/mendableai/firecrawl/pull/977
* RutamBhagat made their first contribution in https://github.com/mendableai/firecrawl/pull/989

**Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.0.0...v1.1.0

We're gonna be updating the release every week now and will be updating it on firecrawl.dev/changelog too!

**Firecrawl V1 is here! With that we introduce a more reliable and developer friendly API.**

_August 29th, 2024_

Here is what’s new:

- Output Formats for /scrape. Choose what formats you want your output in.
- New [/map endpoint](https://docs.firecrawl.dev/features/map) for getting most of the URLs of a webpage.
- Developer friendly API for /crawl/{id} status.
- 2x Rate Limits for all plans.
- [Go SDK](https://docs.firecrawl.dev/sdks/go) and [Rust SDK](https://docs.firecrawl.dev/sdks/rust)
- Teams support
- API Key Management in the dashboard.
- onlyMainContent is now default to true.
- /crawl webhooks and websocket support.

Learn more about it [here](https://docs.firecrawl.dev/v1-welcome)

Start using v1 right away at [https://firecrawl.dev](https://firecrawl.dev)

What's Changed (including v0 + v1)
* Delete .DS_Store by szepeviktor in https://github.com/mendableai/firecrawl/pull/8
* [Bugfix] added normalized apikey to craw/status route by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/12
* [Feat] improving reative paths by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/4
* Fix typos by szepeviktor in https://github.com/mendableai/firecrawl/pull/9
* [Feat] Added html to markdown table parser by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/11
* Option to extract only the main content, excluding headers, navs, footers etc. by nickscamara in https://github.com/mendableai/firecrawl/pull/14
* [Feat] Adding pdf parser by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/17
* adding ci-cd workflow by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/20
* adding workflow by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/21
* adding env secrets by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/22
* [Feat] Added TSDocs and types for js-sdk by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/28
* Added option to replace all relative paths with absolute paths by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/25
* [Bugfix] Fixed scrape preview test by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/30
* Caleb: fixing some documentation and rebuilding the server by calebpeffer in https://github.com/mendableai/firecrawl/pull/32
* Rate limit fixes for crawl status by nickscamara in https://github.com/mendableai/firecrawl/pull/36
* Better logging by nickscamara in https://github.com/mendableai/firecrawl/pull/35
* [Feat] Added type declarations by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/31
* Refactor api routes by nickscamara in https://github.com/mendableai/firecrawl/pull/37
* Logging by nickscamara in https://github.com/mendableai/firecrawl/pull/38
* Cjp/making db auth optional <> Running project locally by calebpeffer in https://github.com/mendableai/firecrawl/pull/40
* chore: add context.close by mattzcarey in https://github.com/mendableai/firecrawl/pull/46
* Fixes table parsing for websites such as news.ycombinator.com (HN) by nickscamara in https://github.com/mendableai/firecrawl/pull/52
* [Feat] Server health check + slack message by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/53
* [Feat] Added blocklist for social media urls by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/55
* [Feat:mvp] Search Endpoint => serp api + firecrawl => 🔥 🔍 by nickscamara in https://github.com/mendableai/firecrawl/pull/56
* [Feat] Added anthropic vision api by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/5
* [Bugfix] Trim and Lowercase all urls by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/13
* Implements the ability for the crawler to output all the links it found, without scraping by nickscamara in https://github.com/mendableai/firecrawl/pull/34
* Serper params by nickscamara in https://github.com/mendableai/firecrawl/pull/62
* Support for tbs, filter, lang, country and location with Serper search. by rogerserper in https://github.com/mendableai/firecrawl/pull/61
* [Feat] Added allowed urls by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/64
* /search support in node sdk by nickscamara in https://github.com/mendableai/firecrawl/pull/72
* Free credits increase by nickscamara in https://github.com/mendableai/firecrawl/pull/75
* [Bugfix] JS-SDK: Remove dotenv and add tests by mdp in https://github.com/mendableai/firecrawl/pull/68
* [Feat] Coupon system by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/66
* Specific website params support by nickscamara in https://github.com/mendableai/firecrawl/pull/83
* Greenpay fixes by nickscamara in https://github.com/mendableai/firecrawl/pull/84
* [Feat] Implemented retry attempts to handle 502 errors by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/67
* feat: LLM Extraction (mvp) by nickscamara in https://github.com/mendableai/firecrawl/pull/90
* Update README.md by bllchmbrs in https://github.com/mendableai/firecrawl/pull/110
* Add Posthog Logging by ericciarla in https://github.com/mendableai/firecrawl/pull/109
* Refactor of main web scraper + Partial data streaming by nickscamara in https://github.com/mendableai/firecrawl/pull/120
* [Feat] Added includeHTML option by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/126
* Cancel Job Route by nickscamara in https://github.com/mendableai/firecrawl/pull/129
* [Feat] Added max depth option by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/130
* Add keyAuth endpoint by ericciarla in https://github.com/mendableai/firecrawl/pull/131
* [Test] Added integration tests suite by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/118
* Adds Zod Integration for LLM Extraction in the Firecrawl JS SDK by nickscamara in https://github.com/mendableai/firecrawl/pull/135
* [Docs] Updated examples by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/137
* Switching to AGPL - We Need Your Consent! by calebpeffer in https://github.com/mendableai/firecrawl/pull/134
* Nsc/refactor scraping order by nickscamara in https://github.com/mendableai/firecrawl/pull/139
* Update models.ts by ericciarla in https://github.com/mendableai/firecrawl/pull/144
* Timeout on /scrape by nickscamara in https://github.com/mendableai/firecrawl/pull/145
* [Doc] Added default value for crawlOptions.limit by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/142
* feat: 4x-5x faster crawler (fast mode) by nickscamara in https://github.com/mendableai/firecrawl/pull/149
* Add Docker Compose for easy self hosting by chand1012 in https://github.com/mendableai/firecrawl/pull/119
* refactor: fix typo in WebScraper/index.ts by eltociear in https://github.com/mendableai/firecrawl/pull/27
* [Tests] Added crawl test suite -> crawl improvements by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/153
* feat: Docx Support by nickscamara in https://github.com/mendableai/firecrawl/pull/158
* Fixes pdfs not found if .pdf is not present by nickscamara in https://github.com/mendableai/firecrawl/pull/29
* Update README.md: Typo fix by elimisteve in https://github.com/mendableai/firecrawl/pull/160
* [Feat] Added rate limits by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/151
* Allow override of API URL by mattjoyce in https://github.com/mendableai/firecrawl/pull/166
* feat: HyperDX Integration by nickscamara in https://github.com/mendableai/firecrawl/pull/167
* beta: Fire-Engine fallback by nickscamara in https://github.com/mendableai/firecrawl/pull/174
* Add additional file extensions to crawler.ts by tractorjuice in https://github.com/mendableai/firecrawl/pull/77
* [Bug] Fixing /crawl limit by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/143
* Update issue templates by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/180
* [Feat] Added proxy and media blocking support for Playwright by JakobStadlhuber in https://github.com/mendableai/firecrawl/pull/181
* update: wait until body attached in playwright-service by qyou in https://github.com/mendableai/firecrawl/pull/170
* feat: Allow privacy/legal/ other pages in social media websites by nickscamara in https://github.com/mendableai/firecrawl/pull/168
* [Bug] Added data check for python SDK by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/176
* Fix FIRECRAWL_API_URL bug, also various PyLint fixes by mattjoyce in https://github.com/mendableai/firecrawl/pull/178
* [Feat] Added idempotency key to crawl route by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/132
* Feat: Provide more details for 429 error msg by simonha9 in https://github.com/mendableai/firecrawl/pull/190
* Limit on /search is not deterministic by Keredu in https://github.com/mendableai/firecrawl/pull/186
* Various PyPi Metadata by mattjoyce in https://github.com/mendableai/firecrawl/pull/191
* [Test] Added sdk e2e tests by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/183
* Allow users to manually set the waitFor param on /scrape by nickscamara in https://github.com/mendableai/firecrawl/pull/200
* [Feat] Added custom scraping conditions for readme docs by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/204
* Feat/screenshot support by ericciarla in https://github.com/mendableai/firecrawl/pull/207
* feat: New pricing/limits changes by nickscamara in https://github.com/mendableai/firecrawl/pull/216
* [sdk] Fixes waiting status not being present on check status by nickscamara in https://github.com/mendableai/firecrawl/pull/218
* Fixed fire-engine content bug by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/228
* Use instead of for default BULL_AUTH_KEY. Hash mark is reserved for URI fragments. by rombru in https://github.com/mendableai/firecrawl/pull/229
* feat: Ability to forward headers to reliable providers for auth etc... by nickscamara in https://github.com/mendableai/firecrawl/pull/221
* Improvements to the blocklist regex by nickscamara in https://github.com/mendableai/firecrawl/pull/185
* Playwright service bugs 222 179 197 by mattjoyce in https://github.com/mendableai/firecrawl/pull/224
* [Fix] Changed timeout parameter name on js sdk by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/211
* [Bug] Improved js response and test for getting partial_data by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/212
* Better fallbacks for initial crawl start by nickscamara in https://github.com/mendableai/firecrawl/pull/148
* [Feat] Added custom scraping for google-drive pdf usecase by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/235
* Script to check local vs published versions of sdk packages by mattjoyce in https://github.com/mendableai/firecrawl/pull/213
* Update README.md by calebpeffer in https://github.com/mendableai/firecrawl/pull/140
* Refactor custom scraping and added security center vanta by nickscamara in https://github.com/mendableai/firecrawl/pull/237
* [Feat] Added scroll xpaths on fire-engine for handling readme docs by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/239
* Add Kubernetes configuration for Firecrawl deployment by JakobStadlhuber in https://github.com/mendableai/firecrawl/pull/236
* [Feat] Improved the scrape for gdrive pdfs by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/238
* Add support for Self-Hosted Webhook URL Usage and added project_id into the webhook payload by JakobStadlhuber in https://github.com/mendableai/firecrawl/pull/234
* Partial Data Sliding window of 50 by nickscamara in https://github.com/mendableai/firecrawl/pull/242
* feat: Transactional emails for rate limits and credit limits by nickscamara in https://github.com/mendableai/firecrawl/pull/244
* Python-SDK transitional build setup for pyproject.toml by mattjoyce in https://github.com/mendableai/firecrawl/pull/196
* [Feat] CI/CD for publishing js and python SDKs by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/246
* 194 sdk ci pipeline for publishing pythonnode sdk by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/248
* fixing minor problems on workflow by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/250
* Fix 208 py sdk interval poll name by mattjoyce in https://github.com/mendableai/firecrawl/pull/252
* Moving from fetch to axios and preventing deadlocks by setting timeouts on fallbacks by nickscamara in https://github.com/mendableai/firecrawl/pull/260
* ignoreSitemap feature, pageOptions now respected in the initial crawl as well by nickscamara in https://github.com/mendableai/firecrawl/pull/263
* Fixed bugs associated with absolute path replacements by nickscamara in https://github.com/mendableai/firecrawl/pull/268
* Only fetch webhook from db if self host webhook not set and using db auth by nickscamara in https://github.com/mendableai/firecrawl/pull/262
* [Feat] Added route to clean completed jobs and a github action cron that triggers every 24h by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/265
* [Feat] Added allowBackwardCrawling option by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/269
* [Feat] Added jobId to webhook data by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/272
* Clusters support to scale our API to of CPUs running by nickscamara in https://github.com/mendableai/firecrawl/pull/274
* Added pageOptions.removeTags by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/275
* Added logging to python sdk FIRECRAWL_LOGGING_LEVEL by mattjoyce in https://github.com/mendableai/firecrawl/pull/256
* Python sdk improve response handling by mattjoyce in https://github.com/mendableai/firecrawl/pull/253
* [Feat] Added parsePDF option to pageOptions by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/271
* [Feat] Added metadata.pageStatusCode and metadata.pageError properties by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/276
* Make maxDepth relative to the entered url by ericciarla in https://github.com/mendableai/firecrawl/pull/277
* Test/load testing by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/175
* Fixes crawler getting confused with base paths that contain www. by nickscamara in https://github.com/mendableai/firecrawl/pull/283
* Feat/max depth relative by ericciarla in https://github.com/mendableai/firecrawl/pull/285
* [Feat] Improvements on response document types by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/298
* test: Rate Limit Unit Tests by nickscamara in https://github.com/mendableai/firecrawl/pull/282
* Added local host support for the javascript SDK by NeevJewalkar in https://github.com/mendableai/firecrawl/pull/296
* [Test] Transcribed from e2e to unit tests for many cases by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/294
* [Feat] Added support for RegEx in removeTags by AndyMik90 in https://github.com/mendableai/firecrawl/pull/297
* [Bug] Fixed includeHTML to use cleanedHtml as response by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/301
* Cjp/email to posthog logging by calebpeffer in https://github.com/mendableai/firecrawl/pull/302
* Fix Broken Link by Lakr233 in https://github.com/mendableai/firecrawl/pull/303
* docs: Fix pydanti to pydantic by 100gle in https://github.com/mendableai/firecrawl/pull/308
* [Bug] Fixed axios bug that were making jobs stuck on active queue by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/312
* fix multi-word search term issue: /search (w/o Serp) by snippet in https://github.com/mendableai/firecrawl/pull/313
* [Bug] Fixed clean jobs by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/317
* [Bug] Fixed the regex test for google drive pdf files by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/318
* add some types by snippet in https://github.com/mendableai/firecrawl/pull/316
* [Bug] Added default values and fixed pdf bug by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/321
* [Test] Added E2E tests for checking metadata values by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/322
* pageOptions.onlyIncludeTags param by nickscamara in https://github.com/mendableai/firecrawl/pull/328
* Update CONTRIBUTING.md by george-zakharov in https://github.com/mendableai/firecrawl/pull/329
* Fix headers | pageOptions not being passed correctly to fire engine by nickscamara in https://github.com/mendableai/firecrawl/pull/331
* [Tests] Added crawl limit unit test by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/323
* Adds pageOptions.includeRawHtml and new extraction mode "llm-extraction-from-raw-html" by ericciarla in https://github.com/mendableai/firecrawl/pull/332
* [Proposal] new feature allowExternalContentLinks by snippet in https://github.com/mendableai/firecrawl/pull/336
* [PROPOSAL] (deps): making sure all deps are always up to date by Sanix-Darker in https://github.com/mendableai/firecrawl/pull/338
* [PROPOSAL] (docker-compose) regroup envs vars between services by Sanix-Darker in https://github.com/mendableai/firecrawl/pull/337
* new playwright service by snippet in https://github.com/mendableai/firecrawl/pull/325
* apps/playwright-service(deps): bump the prod-deps group in /apps/playwright-service with 3 updates by dependabot in https://github.com/mendableai/firecrawl/pull/346
* Logging for all scraper methods by nickscamara in https://github.com/mendableai/firecrawl/pull/363
* setting up docker to ts playwright service by snippet in https://github.com/mendableai/firecrawl/pull/361
* (Docs) Self Host added new ts playwright service instructions by snippet in https://github.com/mendableai/firecrawl/pull/362
* apps/test-suite(deps): bump the prod-deps group in /apps/test-suite with 6 updates by dependabot in https://github.com/mendableai/firecrawl/pull/347
* apps/test-suite(deps-dev): bump typescript from 5.4.5 to 5.5.3 in /apps/test-suite in the dev-deps group by dependabot in https://github.com/mendableai/firecrawl/pull/348
* apps/api(deps): bump the prod-deps group in /apps/api with 28 updates by dependabot in https://github.com/mendableai/firecrawl/pull/349
* [Feat] Added veeva to special case params by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/371
* Only check Supabase if configured to when cancelling job by StefanTerdell in https://github.com/mendableai/firecrawl/pull/374
* dependabot for security checks, fixed crawl test by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/370
* [Feat] Added implementation for saving docs on supabase by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/326
* Self-hosting quality of life fixes by StefanTerdell in https://github.com/mendableai/firecrawl/pull/375
* [Feat] Added fire-engine fallback for getting sitemaps by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/386
* fix(bull): requeue jobs after restart by mogery in https://github.com/mendableai/firecrawl/pull/393
* Fix USE_DB_AUTHENTICATION checks for self-host by kun432 in https://github.com/mendableai/firecrawl/pull/395
* [BUG] Fixed docker self hosting issue by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/396
* Slack Alerts for when queue is over capacity by nickscamara in https://github.com/mendableai/firecrawl/pull/397
* CI for alerts instead by nickscamara in https://github.com/mendableai/firecrawl/pull/398
* Log Fire-engine page errors by nickscamara in https://github.com/mendableai/firecrawl/pull/394
* Sitemap fallback fixes w/ fire-engine by nickscamara in https://github.com/mendableai/firecrawl/pull/399
* Separate Rate Limiter from Main Redis by nickscamara in https://github.com/mendableai/firecrawl/pull/417
* [Feat] Pass along current, total, current_step, and current_url in js sdk by jhoseph88 in https://github.com/mendableai/firecrawl/pull/391
* Redis Health Checks by nickscamara in https://github.com/mendableai/firecrawl/pull/424
* Small fix for empty pageOptions by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/420
* Caleb: Return a list of links on a page by default by calebpeffer in https://github.com/mendableai/firecrawl/pull/423
* [Docs] Updating docs by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/427
* Fix queue stuck bug via lock settings changes by mogery in https://github.com/mendableai/firecrawl/pull/429
* fix(js-sdk): transform tests with ts-jest and configure node by mogery in https://github.com/mendableai/firecrawl/pull/433
* fix(WebScraper): infinite regex leading to fly.io instance hangs by mogery in https://github.com/mendableai/firecrawl/pull/436
* Support chrome-cdp and restructure sitemap fire-engine support. by tomkosm in https://github.com/mendableai/firecrawl/pull/410
* /scrape should now be 600ms-900ms faster by nickscamara in https://github.com/mendableai/firecrawl/pull/447
* Notion Website Fixes by nickscamara in https://github.com/mendableai/firecrawl/pull/453
* Firecrawl UI template by ericciarla in https://github.com/mendableai/firecrawl/pull/458
* Added regex for links in sitemap by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/449
* Added logger by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/450
* Add scrape monitoring by mogery in https://github.com/mendableai/firecrawl/pull/455
* Readiness liveness probes by JakobStadlhuber in https://github.com/mendableai/firecrawl/pull/457
* Client side error handling by nickscamara in https://github.com/mendableai/firecrawl/pull/461
* Admin router + Improve redis notifications by nickscamara in https://github.com/mendableai/firecrawl/pull/460
* support custom models by NiuBlibing in https://github.com/mendableai/firecrawl/pull/414
* DNS Cacheable Lookup to avoid DNS blocking ops by nickscamara in https://github.com/mendableai/firecrawl/pull/478
* Fire-engine to be the default for our cloud-service by nickscamara in https://github.com/mendableai/firecrawl/pull/479
* [Bug] Rate limiter bug on switch for some cases by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/482
* [Feat] Better self hosting guide by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/483
* Fix parallelization issues with caching by nickscamara in https://github.com/mendableai/firecrawl/pull/484
* [Bug] Issue with crawl going beyond Limit by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/485
* [Bug] pdfs and logging pdf events, also added trycatchs for docx by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/475
* [Bug] Nested sitemaps by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/491
* Fix AMZN | Removal of redis alerts by nickscamara in https://github.com/mendableai/firecrawl/pull/492
* Custom engine params fix by nickscamara in https://github.com/mendableai/firecrawl/pull/494
* [Bug] Fixed the empty excludes.filter is undefined bug by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/503
* [Feat] Added fullpagescreenshot capabilities by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/504
* fix(js-sdk): build both CommonJS and ESM versions by mogery in https://github.com/mendableai/firecrawl/pull/432
* fix(js-sdk): build both CJS and EJS versions by mogery in https://github.com/mendableai/firecrawl/pull/508
* Improve logs by tak-s in https://github.com/mendableai/firecrawl/pull/496
* Digikey support by nickscamara in https://github.com/mendableai/firecrawl/pull/512
* [Feat] Add Go SDK implementation by KentHsu in https://github.com/mendableai/firecrawl/pull/497
* Redlock cache in auth by nickscamara in https://github.com/mendableai/firecrawl/pull/529
* Update redis urls in example .env by wahpiangle in https://github.com/mendableai/firecrawl/pull/515
* Self-host fix: Moving comments of .env.example values from end-of-line to above-line. by kevinswiber in https://github.com/mendableai/firecrawl/pull/531
* fix: go-sdk module name by KentHsu in https://github.com/mendableai/firecrawl/pull/534
* Fixed e2e tests by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/536
* Removed obsoleted declaration by matsubo in https://github.com/mendableai/firecrawl/pull/532
* [Feat] Added go-sdk as submodule by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/537
* feat: Move scraper to queue by mogery in https://github.com/mendableai/firecrawl/pull/459
* Reduce metrics ingestion w/ HyperDX v0.8.1 by nickscamara in https://github.com/mendableai/firecrawl/pull/480
* Check team credits based on the crawl limit by nickscamara in https://github.com/mendableai/firecrawl/pull/554
* added variables to beta customers by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/555
* Set the crawl limit to the remaining credits by nickscamara in https://github.com/mendableai/firecrawl/pull/559
* [Feat] Added check job and cancel to fire-engine requests by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/560
* [Bug] Added a way for dealing with DNS without IP resolution by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/561
* Added Sentry Monitoring by nickscamara in https://github.com/mendableai/firecrawl/pull/562
* Internal Concurrency Limits <> Job Priorities by nickscamara in https://github.com/mendableai/firecrawl/pull/566
* [V1] Release by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/527
* [Feat]: Add RUST SDK client for firecrawl API by Sanix-Darker in https://github.com/mendableai/firecrawl/pull/373
* Update README.md by ericciarla in https://github.com/mendableai/firecrawl/pull/590
* [Bug] Moved delete rawHtml to end of controller by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/582
* Fixed RPC cloudflare issues with job ids by nickscamara in https://github.com/mendableai/firecrawl/pull/593
* [v1] Websockets SDKs by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/591
* [v1] LLM Extract by nickscamara in https://github.com/mendableai/firecrawl/pull/586
* [v1] Webhooks by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/594
* fix(v1): maxDepth by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/607
* Ensuring USE_DB_AUTHENTICATION is true in single URL scraper. by kevinswiber in https://github.com/mendableai/firecrawl/pull/516
* Bill team async by nickscamara in https://github.com/mendableai/firecrawl/pull/619
* Optimize check credits func w/ caching by nickscamara in https://github.com/mendableai/firecrawl/pull/624
* Feat: Added go html to md parser by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/608
* Feat: parser singleton by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/629

New Contributors
* szepeviktor made their first contribution in https://github.com/mendableai/firecrawl/pull/8
* rafaelsideguide made their first contribution in https://github.com/mendableai/firecrawl/pull/12
* nickscamara made their first contribution in https://github.com/mendableai/firecrawl/pull/14
* calebpeffer made their first contribution in https://github.com/mendableai/firecrawl/pull/32
* mattzcarey made their first contribution in https://github.com/mendableai/firecrawl/pull/46
* rogerserper made their first contribution in https://github.com/mendableai/firecrawl/pull/61
* mdp made their first contribution in https://github.com/mendableai/firecrawl/pull/68
* bllchmbrs made their first contribution in https://github.com/mendableai/firecrawl/pull/110
* ericciarla made their first contribution in https://github.com/mendableai/firecrawl/pull/109
* chand1012 made their first contribution in https://github.com/mendableai/firecrawl/pull/119
* eltociear made their first contribution in https://github.com/mendableai/firecrawl/pull/27
* elimisteve made their first contribution in https://github.com/mendableai/firecrawl/pull/160
* mattjoyce made their first contribution in https://github.com/mendableai/firecrawl/pull/166
* tractorjuice made their first contribution in https://github.com/mendableai/firecrawl/pull/77
* JakobStadlhuber made their first contribution in https://github.com/mendableai/firecrawl/pull/181
* qyou made their first contribution in https://github.com/mendableai/firecrawl/pull/170
* simonha9 made their first contribution in https://github.com/mendableai/firecrawl/pull/190
* Keredu made their first contribution in https://github.com/mendableai/firecrawl/pull/186
* rombru made their first contribution in https://github.com/mendableai/firecrawl/pull/229
* NeevJewalkar made their first contribution in https://github.com/mendableai/firecrawl/pull/296
* AndyMik90 made their first contribution in https://github.com/mendableai/firecrawl/pull/297
* Lakr233 made their first contribution in https://github.com/mendableai/firecrawl/pull/303
* 100gle made their first contribution in https://github.com/mendableai/firecrawl/pull/308
* snippet made their first contribution in https://github.com/mendableai/firecrawl/pull/313
* george-zakharov made their first contribution in https://github.com/mendableai/firecrawl/pull/329
* Sanix-Darker made their first contribution in https://github.com/mendableai/firecrawl/pull/338
* dependabot made their first contribution in https://github.com/mendableai/firecrawl/pull/346
* StefanTerdell made their first contribution in https://github.com/mendableai/firecrawl/pull/374
* mogery made their first contribution in https://github.com/mendableai/firecrawl/pull/393
* kun432 made their first contribution in https://github.com/mendableai/firecrawl/pull/395
* jhoseph88 made their first contribution in https://github.com/mendableai/firecrawl/pull/391
* tomkosm made their first contribution in https://github.com/mendableai/firecrawl/pull/410
* NiuBlibing made their first contribution in https://github.com/mendableai/firecrawl/pull/414
* tak-s made their first contribution in https://github.com/mendableai/firecrawl/pull/496
* KentHsu made their first contribution in https://github.com/mendableai/firecrawl/pull/497
* wahpiangle made their first contribution in https://github.com/mendableai/firecrawl/pull/515
* kevinswiber made their first contribution in https://github.com/mendableai/firecrawl/pull/531
* matsubo made their first contribution in https://github.com/mendableai/firecrawl/pull/532

**Full Changelog**: https://github.com/mendableai/firecrawl/commits/v1.0.0