Improvements
Memory Management
- Added robust memory monitoring and management system
- Implemented memory usage checks throughout XBRL processing pipeline
- Added configurable memory thresholds (default 16GB) to prevent OOM errors
- Improved cleanup of large objects after processing
XBRL Processing
- Enhanced fact extraction with better statement classification
- Improved handling of multi-statement concept appearances
- Added support for tracking concept order in financial statements
- Better handling of segment and dimension information
Performance
- Optimized taxonomy presentation processing
- Added force_reload option for fact DataFrame generation
- Reduced memory footprint during filing processing
Error Handling
- Better error reporting for memory-related issues
- Improved logging throughout the processing pipeline
- Added graceful handling of large integer values
Bug Fixes
- Resolved issues with concept tracking across multiple statements
- Fixed handling of large integer values in filing processing
- Improved cleanup of temporary objects
Breaking Changes
None
Usage Notes
Memory thresholds can be configured when calling processing functions
alpha-three-a
taxpres.py - Taxonomy Presentation Processing Module
This module provides functionality for processing XBRL taxonomy presentation linkbases and
extracting structured concept information. It helps organize concepts into statements and
validates segment/dimension information.
Key Classes:
-----------
TaxonomyPresentation
Main class that processes taxonomy presentation networks and organizes concepts into
primary statements and disclosures.
Key Functions:
-------------
get_presentation_networks(taxonomy)
Extracts presentation networks from a taxonomy by examining linkbases and base sets.
get_network_details(tax, network, reporter)
Processes a presentation network to extract concept details and relationships.
get_child_concepts(reporter, network, concept, taxonomy, visited=None)
Recursively extracts child concepts from a presentation network hierarchy.
process_children(reporter, network, parent, concepts, grandparent_qname)
Helper function to process child concepts in a presentation network.
ins_facts(xid, tax)
Extracts facts from an XBRL instance document and organizes them based on the
presentation structure.
Example Usage:
-------------
Create a TaxonomyPresentation instance
t_pres = TaxonomyPresentation(taxonomy, reporter)
Get facts from an instance document
fact_df = ins_facts(xbrl_instance, taxonomy)
Access statement information
print(t_pres.statement_concepts) Concepts in primary statements
print(t_pres.disclosure_concepts) Concepts in disclosures
print(t_pres.statement_dimensions) Allowed dimensions per statement
Classes:
--------
TaxonomyPresentation:
Attributes:
tax: The taxonomy object being processed
reporter: TaxonomyReporter instance for label handling
concept_df: DataFrame containing all concepts
allowed_segments_by_statement: Dict mapping statements to allowed segments
concept_dict: Dict containing all concepts
statement_concepts: Dict containing primary statement concepts
disclosure_concepts: Dict containing disclosure concepts
statement_dimensions: Dict containing allowed dimensions per statement
so_name: Name of Statement of Operations
fp_name: Name of Financial Position statement
cf_name: Name of Cash Flow statement
Methods:
populate_concept_df(): Creates DataFrame from concept dictionaries
_is_primary_statement(role_name): Determines if a role represents a primary statement
_process_network_dimensions(network, statement_name): Processes dimensions in a network
_validate_segment(segment_data, statement_name): Validates segment data against statement
_process_taxonomy(): Main method to process taxonomy and build concept dictionaries
is_valid_concept(concept_qname): Checks if a concept exists in presentation
get_concept_info(concept_qname): Gets detailed information about a concept
is_valid_segment(concept_qname, segment_data, statement_name): Validates segment data
Notes:
------
- The module assumes a standard XBRL taxonomy structure with presentation linkbases
- Primary statements are identified using keyword matching in role names
- Segment validation supports both axis/member and dimension/member terminology
- Period types and other attributes are obtained from the concept definitions
alpha-twohalf
It can now print out an income statement exactly the same as reported. At least for Tesla 2019. https://github.com/reeyarn/openesef/blob/master/examples/ScreenshotTSLA.png
alpha-three
fact_id label concept_name value_mln value fact_included
0 F_000123 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 19952.0 19952000000 True
1 F_000126 Operating Leases Income Statement Lease Revenue OperatingLeasesIncomeStatementLeaseRevenue 869.0 869000000 True
2 F_000129 Sales Revenue Automotive SalesRevenueAutomotive 20821.0 20821000000 True
3 F_000132 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 1531.0 1531000000 True
4 F_000135 Sales Revenue Services And Other Net SalesRevenueServicesAndOtherNet 2226.0 2226000000 True
5 F_000138 Revenues Revenues 24578.0 24578000000 True
6 F_000141 Cost Of Goods And Services Sold CostOfGoodsAndServicesSold 15939.0 15939000000 True
7 F_000144 Cost Of Automotive Leasing CostOfAutomotiveLeasing 459.0 459000000 True
8 F_000147 Cost Of Revenues Automotive CostOfRevenuesAutomotive 16398.0 16398000000 True
9 F_000150 Cost Of Goods And Services Sold CostOfGoodsAndServicesSold 1341.0 1341000000 True
10 F_000153 Cost Of Services And Other CostOfServicesAndOther 2770.0 2770000000 True
11 F_000156 Cost Of Revenue CostOfRevenue 20509.0 20509000000 True
12 F_000159 Gross Profit GrossProfit 4069.0 4069000000 True
13 F_000162 Research And Development Expense ResearchAndDevelopmentExpense 1343.0 1343000000 True
14 F_000165 Selling General And Administrative Expense SellingGeneralAndAdministrativeExpense 2646.0 2646000000 True
15 F_000168 Restructuring And Other Expenses RestructuringAndOtherExpenses 149.0 149000000 True
16 F_000170 Operating Expenses OperatingExpenses 4138.0 4138000000 True
17 F_000173 Operating Income Loss OperatingIncomeLoss -69.0 -69000000 True
18 F_000176 Investment Income Interest InvestmentIncomeInterest 44.0 44000000 True
19 F_000179 Interest Expense InterestExpense 685.0 685000000 True
20 F_000182 Other Nonoperating Income Expense OtherNonoperatingIncomeExpense 45.0 45000000 True
21 F_000185 Income Loss From Continuing Operations Before ... IncomeLossFromContinuingOperationsBeforeIncome... -665.0 -665000000 True
22 F_000188 Income Tax Expense Benefit IncomeTaxExpenseBenefit 110.0 110000000 True
23 F_000194 Net Income Loss Attributable To Noncontrolling... NetIncomeLossAttributableToNoncontrollingInterest 87.0 87000000 True
24 F_000197 Net Income Loss NetIncomeLoss -862.0 -862000000 True
25 F_000200 Earnings Per Share Basic EarningsPerShareBasic NaN -4.92 True
26 F_000203 Earnings Per Share Diluted EarningsPerShareDiluted NaN -4.92 True
27 F_000206 Weighted Average Number Of Shares Outstanding ... WeightedAverageNumberOfSharesOutstandingBasic NaN 177000000 True
28 F_000209 Weighted Average Number Of Diluted Shares Outs... WeightedAverageNumberOfDilutedSharesOutstanding NaN 177000000 True
29 F_000645 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 594.0 594000000 False
30 F_000655 Operating Leases Income Statement Lease Revenue OperatingLeasesIncomeStatementLeaseRevenue 218.0 218000000 False
31 F_000664 Operating Leases Income Statement Lease Revenue OperatingLeasesIncomeStatementLeaseRevenue 186.0 186000000 False
32 F_000685 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 19212.0 19212000000 False
33 F_000687 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 146.0 146000000 False
34 F_000689 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 594.0 594000000 False
35 F_000691 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 1000.0 1000000000 False
36 F_000693 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 2226.0 2226000000 False
37 F_000695 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 23178.0 23178000000 False
38 F_000697 Operating Leases Income Statement Lease Revenue OperatingLeasesIncomeStatementLeaseRevenue 869.0 869000000 False
39 F_000699 Operating Leases Income Statement Lease Revenue OperatingLeasesIncomeStatementLeaseRevenue 531.0 531000000 False
40 F_000701 Revenues Revenues 24578.0 24578000000 False
41 F_000733 Cost Of Goods And Services Sold CostOfGoodsAndServicesSold 27.0 27000000 False
42 F_001752 Income Tax Expense Benefit IncomeTaxExpenseBenefit 110.0 110000000 False
43 F_001764 Income Loss From Continuing Operations Before ... IncomeLossFromContinuingOperationsBeforeIncome... -665.0 -665000000 False
44 F_001785 Income Tax Expense Benefit IncomeTaxExpenseBenefit 110.0 110000000 False
45 F_001858 Income Tax Expense Benefit IncomeTaxExpenseBenefit 110.0 110000000 False
46 F_001891 Cost Of Goods And Services Sold CostOfGoodsAndServicesSold 350.0 350000000 False
47 F_001989 Revenues Revenues 23047.0 23047000000 False
48 F_001992 Gross Profit GrossProfit 3879.0 3879000000 False
49 F_001995 Revenues Revenues 1531.0 1531000000 False
50 F_001998 Gross Profit GrossProfit 190.0 190000000 False
51 F_002001 Revenues Revenues 12653.0 12653000000 False
52 F_002004 Revenues Revenues 2979.0 2979000000 False
53 F_002007 Revenues Revenues 1590.0 1590000000 False
54 F_002010 Revenues Revenues 1201.0 1201000000 False
55 F_002013 Revenues Revenues 6155.0 6155000000 False
56 F_002016 Revenues Revenues 24578.0 24578000000 False
alpha-two
- Integrated the code from `farhadab/sec-edgar-financials`;
- Using `memfs` to load XBRL files from inside EDGAR's full-text file without writing to tempdir
alpha
testing