Dawgie

[issue 55: test and (59)](https://github.com/al-niessner/DAWGIE/commit/fab77cad4152048b04e3177c596ae39be1be8336)
*Author*: al-niessner
*Date*: Wed, 6 Nov 2019 14:42:56 -0800
* move some code around

Part of the problem with run ids is that they are controlled from two locations. The simplest control is in dawgie.pl.farm and the other part is on dawgie.pl.scheduler. The first step is to pull the run ID bits from dawgie.pl.farm.dispatch() into their own functions and then test them to make sure they work as expected.

* update testing

test_06.Schedule.test_update() now checks the result of the update and makes sure everyone has the correct run ID. The test, however, is failing because not all of the children are being scheduled. This would explain why tasks have the wrong run IDs. The more complete verification is now that all children are scheduled and that they have the correct run ID.

* actual fixes

Updated dawgie.pl.schedule._priors(). Seems it was checking for dawgie.Task when it should have been dawgie.Algorithm. Fixed this and the test workds just fine.

[issue 54: clean out docker (58)](https://github.com/al-niessner/DAWGIE/commit/9ca0e88a63dbe9d2382b0940659ff22f96023fa0)
*Author*: al-niessner
*Date*: Mon, 28 Oct 2019 12:47:03 -0700
* remove unused docker

Some docker images are needed for the CI tests (CIT). Left the exercise script as well, but will need to build up a docker image for it at some point.

[issue 53: always use the installed version (57)](https://github.com/al-niessner/DAWGIE/commit/5f7c04e29bc8a61ccd194462ea06fe352c29802a)
*Author*: al-niessner
*Date*: Mon, 28 Oct 2019 12:23:25 -0700

Update dawgie.de.html() to insert the appropriate bokeh dependencies based on the installed version of bokeh. It will keep the generated dawgie.StateVector.view() HTML compatible with the installed version of bokeh.

[issue 51: patch post2shelve (56)](https://github.com/al-niessner/DAWGIE/commit/8ed9c78156bb91a1c90e978d531a9391037bcc7a)
*Author*: al-niessner
*Date*: Wed, 23 Oct 2019 14:37:48 -0700
* keep run 0 when already 0

[Issue 44: fixing up scheduler (50)](https://github.com/al-niessner/DAWGIE/commit/e80f3ba8073055063414cf37cde5da04674b6d00)
*Author*: al-niessner
*Date*: Wed, 16 Oct 2019 11:53:43 -0700
* add capabilities

dawgie.MOMENT added "boot" which will trigger something when the pipeline boots and only boots (not reload from git). This is an odd request but since there is a need, added it.

dawgie.schedule() was updated to properly handle the new "boot" in dawgie.MOMENT. The check was modified to be more adaptable when dawgie.MOMENT changes again in the future.

dawgie.tools.compliant was updated to check that events() returns valid dawgie.MOMENTS since nothing requires the user use dawgie.schedule().

* added unit testing

Single unit test for dawgie.pl.schedule._delay() that it computes 0 delay and then throws the exception when the same unit comes around a second time.

* include more checks on building an EVENT

[Issue 45 (49)](https://github.com/al-niessner/DAWGIE/commit/e0f7bd8c6fbb0d42edca4ebf926a2e6d2b15f8e1)
*Author*: al-niessner
*Date*: Mon, 14 Oct 2019 18:32:19 -0700
* have a test that highlights the error

* fix bad assumption

dawgie.pl.schedule.update() assumed everything was a dawgie.Algorithm and called previous(). The correction was to use an _priors() function that called traits(), variables(), or previous() depending if it as Analyzer(), Regression(), Task() respectively.

[Issue 47: backlog (48)](https://github.com/al-niessner/DAWGIE/commit/86b0acd917d9c589604a35ea23fb96fe2442769c)
*Author*: al-niessner
*Date*: Wed, 9 Oct 2019 08:24:23 -0700
* add configurable TCP/IP backlog

As the worker size scales up, the backlog needs to increase with it. Since it should be directly related to the number of workers, called it worker_backlog.

* update all worker interfaces

There are many connections between workers and the pipeline server. Update them all to use a big enough backlog to support the scale of workers being used.

[issue 31: missing log page (41)](https://github.com/al-niessner/DAWGIE/commit/863540932f20eef6b2eaf2a5d149254ebb27f83f)
*Author*: al-niessner
*Date*: Wed, 21 Aug 2019 12:24:43 -0700
* update logging

move dawgie.pl.logger.fe from building a complex JSON object using logging.Formatter to building a string using ;\n; as the delimiter between elements. Tests in a notebook show this should work but no test exists for it currently. Will have to test it in a live pipeline at a later date.

updated dawgie.fe.app to convert the now [{}] returned by dawgie.pl.logger.fe.remembered() to a JSON object using json.dumps(). This should prevent further problems with the object being invalid in the future.

[Verifiying the display order matches the order from the json. I was reverse ordering the logs, but no longer. (40)](https://github.com/al-niessner/DAWGIE/commit/bd95f4fd97264eb2df0735ff53e3ac40cf4d1d1a)
*Author*: Jacob Llamas
*Date*: Tue, 20 Aug 2019 16:30:14 -0700
Updated the logs function to show new entries at the top instead of the bottom.
[Issue 36 QoL update (39)](https://github.com/al-niessner/DAWGIE/commit/925ea1d11b2ef6168f89ff46cc23b3b2df0fd64d)
*Author*: Jacob Llamas
*Date*: Tue, 20 Aug 2019 14:55:15 -0700
* Fixing a bug where filtering was not working for State Vector searches.

* Hiding header rows on SV searches when filtering.

Updating site config to not flash {{tag_names}} before replacing them with content.

* Fixing a js bug in branding, when an element does not exist.

[Fixing a bug where filtering was not working for State Vector searches. (38)](https://github.com/al-niessner/DAWGIE/commit/60a67b7c138484874423f7d5577f59740c4caa07)
*Author*: Jacob Llamas
*Date*: Tue, 20 Aug 2019 08:47:49 -0700

[Issue 29: filter defaults (34)](https://github.com/al-niessner/DAWGIE/commit/adcf02c7814585547bf127a6621487c4439c7e33)
*Author*: al-niessner
*Date*: Wed, 7 Aug 2019 16:11:58 -0700
* non-sequitur changes

Added some logging to help understand why navel gazing is taking so long. They were needed to debug the private pipeline to the point were this issue could be tested.

* Improper handling of object or string

dawgie.fe.app.search_filter_{a,d,u}() were not uniformly handling the type of the object. Some parts treated it as a dictionary others a string. Made it a dictionary consistently until it is turned to JSON at the return.

[Issue 20: correct substring (33)](https://github.com/al-niessner/DAWGIE/commit/256ed2ec90f5e882bb3f173cfab8781ce2763621)
*Author*: al-niessner
*Date*: Tue, 6 Aug 2019 17:13:27 -0700
* new requirements becuase actual deploy is done via PyPI now but local tools like .ci/exercise_01.sh still require the docker image

* since the targets are sorted, ignore the __all__

[Issue 27: expand compliant checks (28)](https://github.com/al-niessner/DAWGIE/commit/99787da02ab222853b8b4f28a2fbf6a92e79cce9)
*Author*: al-niessner
*Date*: Fri, 2 Aug 2019 12:56:37 -0700
* add rule 9: no state vectors

First, add the rule_09() to tools/compliant.py. It simply runs through the factories and makes sure that all units return an non-zero length array.

Second, add a negative test to Test/test_08.py along with a bad AE (bae). Since there is a positive test already to verify Test/ae will work, the negative test will verify that rule 9 is coded correctly.

Lastly, using the test with the bae correct the code and test until it all works.

* non sequitour change

Updated dawgie.tools.dag to have a verbose switch. That is when it was discovered that state_vectors() must not be zero length to get added to the dag. Still, these changes would be nice to expand upon in the future so keeping them here.

niessnerelysium:~/Projects/DAWGIE$ cat /tmp/t.t
[issue 25: PyPI script (26)](https://github.com/al-niessner/DAWGIE/commit/1b47f46bad58af7f2b99b1f7954c60f7e5a04312)
*Author*: al-niessner
*Date*: Sun, 28 Jul 2019 16:19:21 -0700
* .ci/pypi.sh

While not part of continuous integration, it is a helper script for me to take a [release](https://github.com/al-niessner/DAWGIE/releases), build it, then install it at PyPI for those using DAWGIE.


[Issue 23: fix up setup.py (24)](https://github.com/al-niessner/DAWGIE/commit/4d2808b95b04725f97e2c15cc65940cf2bfc7a12)
*Author*: al-niessner
*Date*: Sun, 28 Jul 2019 15:57:52 -0700
* add a little more stuff to delivery and try using a patched version of pydot3.

Added README.md and LICENSE to the egg data directory just because. Seems better to have it all around than to not have it at all. Might be misguided.

Built a patched version of [pydot3](https://github.com/log0/pydot3) in my [fork of it](https://github.com/al-niessner/pydot3/releases/tag/1.0.10) then pushed it to PyPI as [dawgie-pydot3](https://pypi.org/project/dawgie-pydot3/1.0.10/). Once built and deployed, then changed this setup.py to depend upon it.

* fix vulnerability

[Issue 21: start PyPI (22)](https://github.com/al-niessner/DAWGIE/commit/137cd7933be87ce13780c07ead0263e9da29ec8e)
*Author*: al-niessner
*Date*: Sat, 27 Jul 2019 13:44:57 -0700
* update setup.py

Following the PyPI documentation, try building the package for distribution on PyPI. Will need a parent script to do more work than this.

* start of deploying via PyPI

An instiall script to build the a deployment. Right now the script builds this directory rather than collecting a released version from github.com. It will come with time, but here the mechanics of how to build the artifact for PyPI and deliver it is complete. The script is left in safe mode where it will deploy to test.pypi instead of pypi.

Updated the setup.py to work correctly with PyPI. It does a description (poorly) but does one. It has a license and describes it nicely on the website. All inall, it looks like the setup.py is sufficient.

[update utility for making release notes](https://github.com/al-niessner/DAWGIE/commit/c9d39931e47e16cb4a37c5e47495b75f3093d226)
*Author*: Al Niessner
*Date*: Tue, 23 Jul 2019 14:02:29 -0700

[issue 7: FE updates (10)](https://github.com/al-niessner/DAWGIE/commit/de767694289f7d01350fd24a99393b152e59a6d6)
*Author*: al-niessner
*Date*: Tue, 23 Jul 2019 13:40:15 -0700
* Updating FE with Build v19

Changed typeahead engine to use fuzzy matches on the search page.

Made State Vector search use filtering the same as Target Name search.

* Used the `.ci/check_04.sh 1` script to update the LICENSE information in the html files.

* change default filters

The Admin filter is working in tests so should be sufficient. The Dev and User are set to be the same by default. Still testing with those so they may update again in the future.

* Re-arragning filter buttons.

Fixing a bug in filters for exact match.

* reset user and dev default values now that filters are set correctly

* Adding a cookie for selected filter state

Adding a way to brand the frontend with javascripts/siteconfig.js

* Removing un-used images

[fill in some docs](https://github.com/al-niessner/DAWGIE/commit/c6e8aa12e4f82bce7b632a3b0444f4ee70be2f3f)
*Author*: Al Niessner
*Date*: Fri, 19 Jul 2019 16:44:17 -0700

[Issue 18: ref verification (19)](https://github.com/al-niessner/DAWGIE/commit/79bc9f60627f25d13f6232d55670e5c52c89b542)
*Author*: al-niessner
*Date*: Sun, 14 Jul 2019 17:05:15 -0700
* the feat should always be a string

While the documentation stated object, dawgie expects it to be a string because dawgie.StateVector uses a string (feature name) for the key. Secondary, the dawgie.StateVector key is used to form a canonical name with the feat being the last part.

* unit test

dawgie.tool.compliant is the unit test for this problem. Seems that not all SV_REF and V_REF are created equal and this will check a system that they all comply with the expectation. Given that the tests have a toy AE in them, this deployment check also becomes a unit test.

* demonstration of fault

A unit test (Test/test_08.py) that uses the toy test/ae to demonstrate the detection of wrong types in *_REF. All of the ALG_REF work without problem. It is the newer and much less used SV_REF and V_REF. Currently Test/test_08.py is finding the errors in test/ae as expected.

* update toy ae in Test

ae.feedback.bot updated so that all SV_REF and V_REF follow the dawgie documentation (doctext in dawgie/__init__.py). They now set all items to dawgie.StateVector instead of strings. The result was to cause deeper parts of dawgie to fail then be corrected.

* clean up dawgie

dawgie.pl.dag -- all ref.item are now treated as a dawgie.StateVector
dawgie.util -- all ref.item and vref.item are treated as dawgie.StateVector

* final validation

With all of the changes above completed and tested via .ci/check*.sh passing, ran the scripts .ci/deploy_0[1-6].sh inclusive. Fixed a trivial poor use of git checkout to git reset in .ci/deploy_01.sh that is non sequitur. Proceeded to run .ci/exercise_01.sh after all of the deploy scripts completed. The exercise ran to completion verifying that dawgie.db is sufficient as is and that all other changes work in concert correctly.
[issue 13: adapt to gnupg version (14)](https://github.com/al-niessner/DAWGIE/commit/e202cbe8062d9c337e6c86dc3c2d767d0acc0551)
*Author*: al-niessner
*Date*: Tue, 9 Jul 2019 07:56:39 -0700
* adapt to version of gnupg

[Issue 16: add new targets when detected (17)](https://github.com/al-niessner/DAWGIE/commit/080ddbf07a4d1e9d42c3ff4d90913915dc5b06fe)
*Author*: al-niessner
*Date*: Sun, 30 Jun 2019 15:35:49 -0700
* test to illustrate problem

The first step in fixing the problem is to write a unit test to demonstrate it. This test fixture is constructed to have an __all__ target in it making the unit test rather simple -- test_issue_16().

The complexity is really in the setUpClass(). Here it overrides critical parts of dawgie.db.shelf that normally speak over the network via twisted. The unit test does not require the communication but does require the standard processing. Hence, setattr() is used to do the overrides:

mock_acquire and mock_release override dawgie.db.shelf.Connector versions because there can only be one user of the database -- pytest. Therefore, the locks can simply be ignored.

mock_do() overrides dawgie.db.shelf.Connector.__do() skipping the pickling sending and receiving. Instead it calls dawgie.db.shelf.Worker.do() directly and the response is passed back via a test global variable.

mock_send() overrides dawgie.db.shelf.Worker._send() to skip pickling and sending. Instead it writes the respose to a test global variable.

* add target insertion

Updated dawgie.db.shelf.Interface to add a target if it is not known. The unit test now passes showing the fix works.

[log message (12)](https://github.com/al-niessner/DAWGIE/commit/7da900fa35879b094dc81855315e5cc265bae096)
*Author*: al-niessner
*Date*: Tue, 25 Jun 2019 08:36:34 -0700

[change log backup count (9)](https://github.com/al-niessner/DAWGIE/commit/c015af489cb61418d3d99a93fff54a12b8c3ab36)
*Author*: al-niessner
*Date*: Mon, 17 Jun 2019 14:41:15 -0700
dawgie.context - add log_backup and make the arguments.

dawgie.pl.logger - use dawgie.context.log_backup
[fix](https://github.com/al-niessner/DAWGIE/commit/771644e34f169951ae98fd691b1204196f1a9d3c)
*Author*: al-niessner
*Date*: Mon, 17 Jun 2019 11:50:58 -0700

[Use latest v2 request (6)](https://github.com/al-niessner/DAWGIE/commit/194c65bcc8e9cc539d8d771c1dbf5aee45d3d3dd)
*Author*: al-niessner
*Date*: Mon, 17 Jun 2019 11:48:38 -0700

[work in progress (2)](https://github.com/al-niessner/DAWGIE/commit/fc95d3c298c66c2263727d76bc09a232b21b60ee)
*Author*: al-niessner
*Date*: Mon, 17 Jun 2019 11:34:55 -0700
* updated build v18

* updated the build process

Jenkins is different enough from Travis-CI that had to reorder the steps and fix up the steps themselves.

* fix a bug

Seems dawgie/pl/logger.py crept back in. Removed it.
[initial commit](https://github.com/al-niessner/DAWGIE/commit/f37f6c09e103a24d5c038a4505c74129f6a02750)
*Author*: Al Niessner
*Date*: Tue, 11 Jun 2019 09:24:49 -0700