Firecrawl-py

Summary of changes
- Open Deep Research: An open source version of OpenAI Deep Research. See [here](https://github.com/nickscamara/open-deep-research)
- R1 Web Extractor Feature: New extraction capability added.
- O3-Mini Web Crawler: Introduces a lightweight crawler for specific use cases.
- Updated Model Parameters: Enhancements to o3-mini_company_researcher.
- URL Deduplication: Fixes handling of URLs ending with /, index.html, index.php, etc.
- Improved URL Blocking: Uses tldts parsing for better blocklist management.
- Valid JSON via rawHtml in Scrape: Ensures valid JSON extraction.
- Product Reviews Summarizer: Implements summarization using o3-mini.
- Scrape Options for Extract: Adds more configuration options for extracting data.
- O3-Mini Job Resource Extractor: Extracts job-related resources using o3-mini.
- Cached Scrapes for Extract evals: Improves performance by using cached data for extractions evals.

What's Changed
* You forgot an 'e' by sami0596 in https://github.com/mendableai/firecrawl/pull/1118
* added cached scrapes to extract by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1107
* Added R1 web extractor feature by aparupganguly in https://github.com/mendableai/firecrawl/pull/1115
* Feature o3-mini web crawler by aparupganguly in https://github.com/mendableai/firecrawl/pull/1120
* Updated Model Parameters (o3-mini_company_researcher) by aparupganguly in https://github.com/mendableai/firecrawl/pull/1130
* Fix corepack and self hosting setup by rothnic in https://github.com/mendableai/firecrawl/pull/1131
* fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends (FIR-827) by mogery in https://github.com/mendableai/firecrawl/pull/1134
* feat(blocklist): Improve URL blocking with tldts parsing by ftonato in https://github.com/mendableai/firecrawl/pull/1117
* fix(scrape): allow getting valid JSON via rawHtml (FIR-852) by mogery in https://github.com/mendableai/firecrawl/pull/1138
* Implemented prodcut reviews summarizer using o3 mini by aparupganguly in https://github.com/mendableai/firecrawl/pull/1139
* [Feat] Added scrapeOptions to extract by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1133
* Feature/o3 mini job resource extractor by aparupganguly in https://github.com/mendableai/firecrawl/pull/1144

New Contributors
* sami0596 made their first contribution in https://github.com/mendableai/firecrawl/pull/1118
* aparupganguly made their first contribution in https://github.com/mendableai/firecrawl/pull/1115
* rothnic made their first contribution in https://github.com/mendableai/firecrawl/pull/1131

**Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.4.2...v1.4.3

We're excited to announce several new features and improvements:

New Features
- Added web search capabilities to the extract endpoint via the `enableWebSearch` parameter
- Introduced source tracking with `__experimental_showSources` parameter
- Added configurable webhook events for crawl and batch operations
- New `timeout` parameter for map endpoint
- Optional ad blocking with `blockAds` parameter (enabled by default)

Infrastructure & UI
- Enhanced proxy selection and infrastructure reliability
- Added domain checker tool to cloud platform
- Redesigned LLMs.txt generator interface for better usability


What's Changed
* (feat/extract) Refactor and Reranker improvements by nickscamara in https://github.com/mendableai/firecrawl/pull/1100
* Fix bad WebSocket URL in CrawlWatcher by ProfHercules in https://github.com/mendableai/firecrawl/pull/1053
* (feat/extract) Add sources to the extraction by nickscamara in https://github.com/mendableai/firecrawl/pull/1101
* feat(v1/map): Timeout parameter (FIR-393) by mogery in https://github.com/mendableai/firecrawl/pull/1105
* fix(scrapeURL/fire-engine): default to separate US-generic proxy list if no location is specified (FIR-728) by mogery in https://github.com/mendableai/firecrawl/pull/1104
* feat(scrapeUrl/fire-engine): add blockAds flag (FIR-692) by mogery in https://github.com/mendableai/firecrawl/pull/1106
* (feat/extract) Logs analyzeSchemaAndPrompt output did not match the schema by nickscamara in https://github.com/mendableai/firecrawl/pull/1108
* (feat/extract) Improved completions to use model's limits by nickscamara in https://github.com/mendableai/firecrawl/pull/1109
* feat(v0): store v0 users (team ID) in Redis for collection (FIR-698) by mogery in https://github.com/mendableai/firecrawl/pull/1111
* feat(github/ci): connect to tailscale (FIR-748) by mogery in https://github.com/mendableai/firecrawl/pull/1112
* (feat/conc) Move fully to a concurrency limit system by nickscamara in https://github.com/mendableai/firecrawl/pull/1045
* Added instructions for empty string to extract prompts by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1114

New Contributors
* ProfHercules made their first contribution in https://github.com/mendableai/firecrawl/pull/1053

**Full Changelog**: https://github.com/mendableai/firecrawl/compare/1.4.1...v1.4.2
**Firecrawl website changelog**: https://firecrawl.dev/changelog

We've significantly enhanced our data extraction capabilities with several key updates:

- Extract now returns a lot more data due to a new re-ranker system
- Improved infrastructure reliability
- Migrated from Cheerio to a high-performance Rust-based parser for faster and more memory-efficient parsing
- Enhanced crawl cancellation functionality for better control over running jobs

What's Changed
* Added "today" to extract prompts by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1084
* docs: update cancel crawl response by ftonato in https://github.com/mendableai/firecrawl/pull/1087
* port most of cheerio stuff to rust by mogery in https://github.com/mendableai/firecrawl/pull/1089
* Re-ranker changes by nickscamara in https://github.com/mendableai/firecrawl/pull/1090
* Rerank with lower threshold + back to map if length = 0 by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1086


**Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.4.0...1.4.1

Get structured web data with /extract

We’re excited to announce the [release of **/extract**](https://www.firecrawl.dev/extract) - get data from any website with just a prompt. With /extract, you can retrieve any information from anywhere on a website without being limited by scraping roadblocks or the typical context constraints of LLMs.


No more manual copy-pasting, broken scraping scripts, or debugging LLM calls. - it’s never been easier to **enrich your data**, **create datasets**, or **power AI applications** with clean, structured data from any website.

Companies are already using extract to:
- Enrich CRM data
- Streamline KYB processes
- Monitor competitors
- Supercharge onboarding experiences
- Build targeted prospecting lists

Instead of spending hours manually researching, fixing broken scrapers, or piecing together data from multiple sources, simply specify what information you need and the target website, and let the Firecrawl handle the entire retrieval process.

Specifically, you can:
- Extract structured data from entire websites using URL wildcards (https://example.com/*)
- Define custom schemas to capture exactly what you need—from simple product details to complex organizational structures
- Guide the extraction with custom prompts to ensure the LLM focuses on your target information
- Deploy anywhere with comprehensive support for Python, Node, cURL, and other popular tools. For no-code workflows, just connect via Zapier or use our API to set up integrations with other tools.

This versatility translates into a wide range of real-world applications—enabling you to enrich web data for just about any use case.

Limitations - (and the road ahead)
- Let's be honest - while /extract is pretty awesome at grabbing web data, it's not perfect yet. Here's what we're still working on:
- Big sites are tricky - It can't (yet!) grab every single product on Amazon in one go
- Complex searches need work - Things like "find all posts from 2025" aren't quite there
- Sometimes, it's a bit quirky - Results can vary between runs, though it usually gets what you need
- **But here's the exciting part: we're seeing the future of web scraping take shape**

Try it out
Curious to try /extract out for yourself?
Visit our [playground](https://www.firecrawl.dev/playground?mode=extract) to try out /extract - you get **500,000 tokens for free**
Dive into our [Extract Beta documentation](https://docs.firecrawl.dev/features/extract-beta) for detailed technical guidance and API reference
Want a no-code solution? Connect /extract to thousands of applications through our enhanced [Zapier integration](https://zapier.com/apps/firecrawl/integrations)


That's all for now! Happy Extracting from the whole Firecrawl team 🔥


**Full Changelog**: https://github.com/mendableai/firecrawl/compare/v.1.3.0...v1.4.0

v.1.3.0
What's Changed
* feat: new snips test framework (FIR-414) by mogery in https://github.com/mendableai/firecrawl/pull/1033
* (feat/extract) New re-ranker + multi entity extraction by nickscamara in https://github.com/mendableai/firecrawl/pull/1061
* __experimental_streamSteps by nickscamara in https://github.com/mendableai/firecrawl/pull/1063


**Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.2.1...v.1.3.0

What's Changed
* Indexes, Caching for /extract, Improvements by nickscamara in https://github.com/mendableai/firecrawl/pull/1037
* [SDK] fixed none and undefined on response by rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1034
* feat: use new random user agent instead of the old one by 1101-1 in https://github.com/mendableai/firecrawl/pull/1038
* (feat/extract) Move extract to a queue system by nickscamara in https://github.com/mendableai/firecrawl/pull/1044

/extract (beta) changes
* We have updated the /extract endpoint to now be asynchronous. When you make a request to /extract, it will return an ID that you can use to check the status of your extract job. If you are using our SDKs, there are no changes required to your code, but please make sure to update the SDKs to the latest versions as soon as possible.

* For those using the API directly, we have made it backwards compatible. However, you have 10 days to update your implementation to the new asynchronous model.

* For more details about the parameters, refer to the docs sent to you.

New Contributors
* 1101-1 made their first contribution in https://github.com/mendableai/firecrawl/pull/1038

**Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.2.0...v1.2.1

Changelog: https://www.firecrawl.dev/changelog#/extract-changes

/v1/search

The search endpoint combines web search with Firecrawl’s scraping capabilities to return full page content for any query.

Include `scrapeOptions` with `formats: ["markdown"]` to get complete markdown content for each search result otherwise it defaults to getting SERP results (url, title, description).

More info here [/v1/search docs](https://docs.firecrawl.dev/api-reference/endpoint/search)

What's Changed
* /extract URL trace by nickscamara in https://github.com/mendableai/firecrawl/pull/1014
* (feat/v1) Search by nickscamara in https://github.com/mendableai/firecrawl/pull/1032

**Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.1.1...v1.2.0