- [compare](https://github.com/slow-but-steady/yt-videos-list/compare/v0.4.7...v0.5.0) changes to previous version
- major change includes changing output file name to include `_reverse_chronological` or `_chronological`
- e.g.
- `MyChannel_reverse_chronological_videos_list.txt`
- `MyChannel_chronological_videos_list.txt`
- compare with previous naming convention of
- `MyChannel_videos_list.txt` - REGARDLESS of whether the file was in reverse chronological order or chronological order
<details>
<summary><strong>addresses issues:</strong></summary>
- 'gbk' codec can't decode byte 0x80
- https://github.com/slow-but-steady/yt_videos_list/issues/3
- PR https://github.com/slow-but-steady/yt_videos_list/pull/5
- commit 1fcf32a5f433ab8a82b9c56e90be6894eecf12e1: Explicitly use utf-8 for encoding and decoding all files
- commit 4c4cc5a0c6d7f8ee0422e44cdfecd4d152d4adf5: Specify name,operation,encoding params for all file IO
- Can lc.create_list_for() return the csv file name?
- https://github.com/slow-but-steady/yt_videos_list/issues/4
- PR https://github.com/slow-but-steady/yt_videos_list/pull/6
- commit a30be66e9ded08faa2f7bd13247cf1105078d62a: Return file name after lc.create_list_for() finishes
- Temporal filename "yt_videos_list_temp.txt"
- https://github.com/slow-but-steady/yt_videos_list/issues/7
- https://github.com/slow-but-steady/yt_videos_list/pull/8
- commit 5d7bfada4bb14e407e01b3d8dfcef2fba7b64417: Name temp files using channel (for file creation)
- commit c781a2f70455d3f7f7e28bd66f9234173b100aa2: Name temp files using channel (for file updates)
- commit 3772d57465ef72567cf6d85f0e83fee72ca50f6e: Indicate type of temp file being written
- append time to end of temp file name
- initially appended UNIX time
- commit d0f76e2e4205766d441209771a2cab1ee7c5231b: Append UNIX timestamp to temporary file name
- commit 110912d39c8044621d168bcfa1c5c2c83b7164af: Replace the dot in timestamp with a dash (should have been included in commit above)
- then changed appended time from UNIX time to ISO 8601 datetime format to increase readability
- commit a349c5b4d7eb0325a655aeae598da6cb0834dfcc: Append ISO 8601 datetime instead of UNIX time to temp file name
</details>
<details>
<summary><strong>inserts "_reverse_chronological" or "_chronological" to file name:</strong></summary>
- commit 8cf2e1555e0b7312d7a9e3eb8966c205482852c2: Append (reverse_)?chronological to file name
- commit 3285d930684afbfba82b9eec30ff4ce9448dfcc3: Update location for testing file paths
- commit dc38e2da3b27f41ae30320821a7ecfac23051cfa: Modify output file naming
- commit 90212e88e2e6285cd7efca44e7c1123117cf10d8: Simplify file suffix creation
</details>
<details>
<summary><strong>significantly improves logging:</strong></summary>
- vertically align similar messages to facilitate quick comparisons between related messages
- commit c87792755004fd0898a74bd7843172180f4cfeee: Vertically align logging output
- commit e7458d036eb5527afdf4eec50223c93ba7ada55a: Make logging messages more visible
- commit 34878884012ed5c0e220369c8a5946ea6e2d83e5: Right pad all testing log messages with ">"
- commit 60cbee3916e226a6881d4ac1ba2d33e3a15669b1: Log thread being created (during testing)
- commit 18146c49ca0360165906773711bea96d95a1a240: Rejustify program logging messages
- commit 296fc4d7fc2a143f26e8220cfedf732ab5233b9c: log program info with custom logger helper module (↑ DRY)
- commit ab188409af1480690596a341ff3675fef1c28349: Log "PROGRAM COMPLETE" instead of "PROGRAM COMPLETED"
- log datetime for every event
- commit b00b0885439c2570057997a4288cb5c962704cd8: Print datetime during testing
- commit e6101b89a9b1cd58281b05068a6369acb4d8f000: Log datetime while running program
- **LOG ALL information to corresponding LOG FILE for channel**
- log file naming
- commit 257e9e9fab06e541d2ec7fed9c165488d6c24bed: Name log file using ISO 8601 datetime
- commit f53e6af8624b2c4d67f9a36ad67e1d47a529cbab: Name log file using output file name
- general logging
- commit 2b6bb4f952833f15cd8bb2991e3e6bca64787582: Enable optional logging to user specified log_file
- commit b1e784a5292f1deec5df07ea33c68a9e15bcdc1e: Log test output to "{suffix}.log" (testing)
- commit 2b63e9c42f72e2727149f7b09a1075430d6ab36f: Enable INFO level logging by default
- commit 82a0129d82ea67475af902cf4a8a07c016d853b4: Simplify logging via custom context manager text writer (EXTREMELY detailed!)
- commit 3b6e3fcec63e7d76e9facfaf961d9b6a67dec52b: Pass logging_output_location to txt_writer()
- commit 6513697c6a5dbf212ca16e1f7e2ef42d552738fe: Log program start & end messages instead of printing to console
- commit add1f35fad70463dfc56547e686afb21e73c5df6: Log name of driver during testing
- commit 21e6bde45e7966d5cebdb3ab94f35d7462007d9e: Add testing info to log files during tests
- commit 9a41424151c89836928e75e2e0407c844e447781: Enable logging to multiple files during testing
- commit 8bb10084efe6ce320ad07beb9faf1652287cf9b9: Simplify testing info logging
- commit 9a41424151c89836928e75e2e0407c844e447781: Enable logging to multiple files during testing
- commit 99d7be0896aa422bfe8138b7a882bd446e9ecc4b: Add "*" 200x when test starts to clearly divide log file
- commit ca7f4c91f1c8294e85ad8f34e5222d057036de0f: Log thread name when new thread created (during testing)
- commit 97ccc33716da2dc281c84989092f76b8b9741c3d: Log ">>>STARTING PROGRAM<<<"
- commit 2e20d8a7f53a0a6af0d0b84d54e316558c055b48: Log ">>>PROGRAM COMPLETED<<<"
- commit 8a3e4f290f35fb04d91e1b93da0c10571d3c4694: Log write & file renaming successes separately
- commit dab5ecf27e26a5eec6878c83e90c67ee5487acf8: Move create_file.py & update_file.py decorator code → log_extraction_information()
- commit fb8311869e89179dcf2bbf2849edcd5f97b216a1: Always log to log file but allow console logging muting
- commit 56bc309e7229d3ad4eee6da3e42c1e7171c7e7b9: Log "video" if 1 new video found, otherwise log "videos"
</details>
<details>
<summary><strong>interesting logging (python standard library package) bug and workaround:</strong></summary>
- commit 82a0129d82ea67475af902cf4a8a07c016d853b4: Simplify logging via custom context manager text writer (also mentioned above, EXTREMELY detailed!)
</details>
<details>
<summary><strong>multi-threading bug (very detailed explanations) and workaround (just avoid using global variables):</strong></summary>
- only occurs when
- scraping the same channel on 2 threads with reverse_chronological set to `True` on one thread and `False` on the other thread
- and starting both threads WITHIN a few tenths of a second of each other
- WITH pre-existing files for both reverse_chronological file and chronological files but DIFFERENT number of videos in the files for reverse_chronological and the chronological files
- commit 97d928f8a9b73a7d7d4c6f02e49ff4f44fa37db6: Test pre-existing csv, txt, md files first
- commit 7bf88c10e7ebbb4f02be9fc2301026bdb4deb1c7: Modify partial chronological files (catch bug more frequently)
- commit e787c3a76352602b50d7db2a00b50a1962830107: Return visited videos sets instead of creating global variables
</details>
<details>
<summary><strong>other multi-threading bugs/challenges/changes:</strong></summary>
- commit 3b78b0a28cadff93cc7dc602c4c9b1df360acb67: Delete only relevant files before testing
- commit e2e1ae9f20c42de0feefffc80fc1961a8a00dd8e: Avoid starting new thread after last test case
- commit 7be64c3e177547734a1a918200362367b645e139: Explicitly check which thread ends first
- commit 5aafa78dcdef410d0ae3347876b980277cca6da2: Simplify threading logic for tests
- commit 930b59c19ed80f0ee3613e661c926abc633fe18b: Avoid multi-threading for safaridriver
- commit 4da6186f4a856b4097dc866c2dd81029e7333a84: Remove debugging print statements ("previous commit" refers to the commit above)
- commit fcd744da234a93937fddd3204b3a72c15e1f1177: Verify variable exists before printing message
- commit 9c2529d2981e2dab6685f8712cbebad6b4802dbf: Ensure threads finish before proceeding
- commit 27cc6a9fde087715c7179d6745b139daf3bb731e: Make thread checks more robust
</details>
<details>
<summary><strong>removes deprecated create_list_for() arguments:</strong></summary>
- commit 6bbac4944c93ec22605cd23bad003a4135b50b83: Remove deprecated create_list_for() arguments
</details>
<details>
<summary><strong>**creates custom threading.Thread subclass to store result of thread during testing**:</strong></summary>
- commit 8fc62703047b9f8de287306239885cd5138a8d7e: Add custom class to store thread result
- commit f1d58f6deeb2becf9038a94c3fb964bccc5321d3: Make ThreadWithResult attribute names more descriptive
- commit b10480b6979f96443ab9e2e62e515c4da30eccdb: Add ThreadWithResult class docstring (test_shared.py)
</details>
<details>
<summary><strong>points future drivers to newest available driver:</strong></summary>
- commit fd8ad485685768b3bdf918859e7ea514c1db2963: Point future drivers to newest available driver ("next commit" refers to commit below)
- commit fd878f3d339732f776da841ed53d22d77ec96f73: Indicate failed update may be due to new driver version
</details>
<details>
<summary><strong>creates json file with all download commands:</strong></summary>
- commit e0569f2aced090f7555bedbfaf254d39454738f6: Create json file for download commands
- previously the project only provided pseudo json in the yt_videos_list/docs/dependencies_pseudo_json.txt file
</details>
<details>
<summary><strong>fixes inability to update package due to testing module dependency on package submodule:</strong></summary>
- started with
- commit 6fa0deb3d8b6c73612d79ea927f81f62e1141c29: Run "pip" on Windows and "pip3" on Unix
- following commit 8c73de6295edf461b64b594fef0f8a01470056bb: Make PATH_SLASH a global variable
- addressed with
- commit 9550ca5b5c330238e94be8d4874fe0f4953731cb: Update local package without yt_videos_list submodule function
- commit 878fb67585debc00200ce831ae2106ddd4c7acea: Remove duplicate import (test_cross_platform_drivers.py) (since function now imported from tests.determine module)
- commit 0204dd2201cbb3b68ca3cac333312216a7d65041: Run pip install directly from test script
- commit 829a1ae681215ca9b06e74dad435372aef108c91: Update local package if python test module called directly
<details>
<summary><b>Benchmarking</b></summary>
without yt_videos_list submodule function
for i in {1..10}; do (time (for i in {1..100}; do python3 minifier.py; done)); done
real 0m8.261s
user 0m5.433s
sys 0m2.259s
real 0m8.288s
user 0m5.429s
sys 0m2.247s
real 0m8.022s
user 0m5.272s
sys 0m2.164s
real 0m7.989s
user 0m5.266s
sys 0m2.165s
real 0m7.984s
user 0m5.253s
sys 0m2.163s
real 0m8.009s
user 0m5.268s
sys 0m2.164s
real 0m8.047s
user 0m5.269s
sys 0m2.175s
real 0m8.068s
user 0m5.242s
sys 0m2.182s
real 0m8.030s
user 0m5.289s
sys 0m2.164s
real 0m8.046s
user 0m5.284s
sys 0m2.176s
with yt_videos_list submodule function
for i in {1..10}; do (time (for i in {1..100}; do python3 minifier.py; done)); done
real 1m28.987s
user 0m42.470s
sys 0m41.508s
real 1m28.921s
user 0m42.508s
sys 0m41.411s
real 1m28.753s
user 0m42.436s
sys 0m41.378s
real 1m29.467s
user 0m42.700s
sys 0m41.732s
real 1m28.672s
user 0m42.286s
sys 0m41.406s
real 1m28.415s
user 0m42.297s
sys 0m41.202s
real 1m28.629s
user 0m42.360s
sys 0m41.244s
real 1m29.088s
user 0m42.587s
sys 0m41.527s
real 1m29.392s
user 0m42.644s
sys 0m41.637s
real 1m29.345s
user 0m42.657s
sys 0m41.643s
without yt_videos_list submodule function again
for i in {1..10}; do (time (for i in {1..100}; do python3 minifier.py; done)); done
real 0m8.488s
user 0m5.585s
sys 0m2.308s
real 0m8.293s
user 0m5.497s
sys 0m2.251s
real 0m8.115s
user 0m5.396s
sys 0m2.188s
real 0m8.116s
user 0m5.396s
sys 0m2.179s
real 0m8.145s
user 0m5.395s
sys 0m2.198s
real 0m8.066s
user 0m5.367s
sys 0m2.170s
real 0m8.042s
user 0m5.340s
sys 0m2.162s
real 0m8.029s
user 0m5.329s
sys 0m2.159s
real 0m8.170s
user 0m5.420s
sys 0m2.195s
real 0m8.154s
user 0m5.426s
sys 0m2.190s
</details>
</details>
<details>
<summary><strong>other interesting bugs:</strong></summary>
- commit 1cdd8f5d203c895c01be4e7a8400afefd7c18c30: Revert "Make command consistent with other unix commands"
- commit 6783c40e0dd97d16bd569370145be5f8c992bfd8: Make command consistent with other unix commands
- following commit 76c066f2751acf9376e39575bd74bd779894b934: Move repeated commands into helper functions
- addressed in
- commit d762b00acee4557f0de92bd17d5ea28f7b4ecb60: Remove `rm /usr/local/bin/sha512_sum` command (bravedriver)
- commit e32f69ff1bf22f7c7da49d7b2bad55a06269eca8: Remove sha512 removal command for Windows bravedriver too
</details>
<details>
<summary><strong>not a bug, but best practice:</strong></summary>
- commit 30e9701ed984122d61a28abc5b2598adcea60e93: Make global varaibles local
</details>