Redi

* Summary: Merge rx2ra.py and rediJS.py in to redi.py

* removed comments from setup.json (Radha Kandula)
* added function get_emr_data() (Radha Kandula)
* added documentation for plan of the log levels in configure_logging() function (Radha Kandula)
* added new variables for ftp server settings (Radha Kandula)
* updated the old variables in setup.json to python naming standards (Radha Kandula)
* added prototype to convert_clinical_data_to_xml function (Radha Kandula)
* Add spreadsheet showing the frequency of non-numeric values in lab data relevant to HCV Target (Philip Chase)
* Add the file units_found_in_ufshands_clinical_lab_values.csv which includes just that with frequency data (Philip Chase)
* Roll raw.txt back to the orignal format as the new format is not ready for primetime (Philip Chase)
* Add component mapping hints for hep.rnaHcv.rnaHeptimax and LOINC details for possible LOINC mapping (Philip Chase)
* pylint changes to logger (Radha Kandula)
* pylint changes on the new functions added (Radha Kandula)
* added params validation in the POSTed string (Radha Kandula)
* added buffer value to main to supress warnings from TestSuite (Radha Kandula)
* added comments to functions (Radha Kandula)
* added the mock listener function for response comparision (Radha Kandula)
* updated to fix test results (Nicholas Rejack)
* added logging to parse_raw_xml function (Nicholas Rejack)
* added logging to parse_translation_table function (Nicholas Rejack)
* added logging to parse_form_events function (Nicholas Rejack)
* added the send_data_to_redcap to testsuite (Radha Kandula)
* sending data to redcap added in test (Radha Kandula)
* created test for send_data_to_redcap (Radha Kandula)
* new flow updated (Radha Kandula)
* Add redi-dataFlow-forSecurityEval.graffle and its pdf (Philip Chase)
* Renamed REDI-JS to REDI in redi-js-dataFlow.* (Philip Chase)
* Add a specification for the clinical data file that embraces EPIC Clarity defaults (Philip Chase)
* changes done to email and pwd (Radha Kandula)
* report mail initial commit (Radha Kandula)
* no more rx2ra.py.. rebranded to redi.py (Radha Kandula)
* changes to test files for rx2ra to redi changes (Radha Kandula)
* merged redi JS with rx2ra (Radha Kandula)
* bin/rx2ra.py (Radha Kandula)
* integrated rediJS and rx2ra into a single flow (Radha Kandula)
* added test for generate redcap eav (Radha Kandula)
* added test for generate redcap eav (Radha Kandula)
* Adapt getEmrData.sh and subjectMap.csv to EMR processes (Philip Chase)
* updated function name changes for pylint: configure_logging (Radha Kandula)
* more pylint changes & logger usage changes (Radha Kandula)
* PylintReport updated (Radha Kandula)
* pylint changes & TestSuite updates (Radha Kandula)
* made TestWriteToFile class independent of file IO (Radha Kandula)
* added logger configuration in testwritefile (Radha Kandula)
* added null check for form_name in update_event_name function (Radha Kandula)
* added the None checks before joining the strings in UpdateEventName function (Radha Kandula)
* logger function called inappropriately. so, updated it (Radha Kandula)
* logger function accessed without configuring logging. so corrected it (Radha Kandula)

* Summary: Add rx2ra.py, getEmrData.sh

* Add documentation for getEmrData.sh (Philip Chase)
* adding logic puzzle image (Nicholas Rejack)
* updated powerpoint and report (Nicholas Rejack)
* Update SIPOC of data flow from emr to redcap (Philip Chase)
* added screenshots of draft report and agenda (Radha Kandula)
* added screenshots of draft report and agenda (Radha Kandula)
* Add presentation of data flow (Philip Chase)
* Add sample report to send to data abstractors (Philip Chase)
* Add counting of distinct values. Also removed HH:MM from timestamp (Philip Chase)
* Add balance of chemistry form fields to translationTable.xml (Philip Chase)
* Fix formDateField for chemistry form in formEvents.xml (Philip Chase)
* cast integer as string in log message in update_event_name (Philip Chase)
* Add and improve log output in generate_redcap_eav and update_event_name (Philip Chase)
* Move data tree sort before event name update (Philip Chase)
* Replace dummy subjectMap.csv with legit data (Philip Chase)
* Change call to sort_element_tree in main and exclude the output file created (Philip Chase)
* Fix sort_element_tree and its test (Philip Chase)
* Add eventName output in generate_redcap_eav (Philip Chase)
* Add timestamp_group concept in update_event_name to fix application of eventNames (Philip Chase)
* Revise input and output data in TestUpdateEventName class (Philip Chase)
* removed testing print statment in rx2ra (Nicholas Rejack)
* updated sort function. (Nicholas Rejack)
* Replace logException references with logger.level references (Philip Chase)
* Add event_name output in generate_redcap_eav (Philip Chase)
* Add call to update_event_name in rx2ra.py main (Philip Chase)
* Add event_list length error handling, logging, and comments in update_event_name (Philip Chase)
* Remove unused code from test/TestUpdateEventName.py (Philip Chase)
* updated docstring for sorting (Nicholas Rejack)
* updated sorted function (Nicholas Rejack)
* updated sorting function (Nicholas Rejack)
* added sorting helper function to rx2ra (Nicholas Rejack)
* Add update_event_name, its test and references in TestSuite class (Philip Chase)
* Remove redcap.eav output file from source control (Philip Chase)
* Add output for redcap eav event date form complete records (Philip Chase)
* Add output for redcap eav datum record and units record (Philip Chase)
* Add calls to update_formcompletedfieldname and update_redcap_field_name_value_and_units in rx2ra.py main (Philip Chase)
* Change variable and class references in TestSuite to 'rx2ra' (Philip Chase)
* Add update_redcap_field_name_value_and_units, its test and references to test in TestSuite (Philip Chase)
* Fix element references in function prototype comments of generate_redcap_eav (Philip Chase)
* Add new elements redcapFieldNameValue and redcapFieldNameUnits to add_elements_to_tree and its test (Philip Chase)
* Add TestAddElementsToTree class and references to it in TestSuite (Philip Chase)
* Add TestUpdateDataFromLookup class and references to it in TestSuite (Philip Chase)
* Fix indentation levels in TestSortElementTree (Philip Chase)
* Revert "fixed path in import for TestLog.py" (Philip Chase)
* Revise generate_redcap_eav to have right structure, but only dummy output records (Philip Chase)
* incremental update on TestSortElementTree, still not working. (Nicholas Rejack)
* incremental update on TestSortElementTree, still not working. (Nicholas Rejack)
* add TestSortElementTree to TestSuite (Nicholas Rejack)
* added test data to TestSortElementTree.py (Nicholas Rejack)
* fixed path in import for TestLog.py (Nicholas Rejack)
* added additional fields to sorting function (Nicholas Rejack)
* redcap.eav file (Radha Kandula)
* added generate_redcap_eav function (Radha Kandula)
* resolved bug in TestSuite (Radha Kandula)
* function names changed as per pylint standards (Radha Kandula)
* removed trailing whitespace (Nicholas Rejack)
* Exclude log and scratch files in .gitignore (Philip Chase)
* Delete scratch log file (Philip Chase)
* added docstring to add_elements_to_tree in rx2ra.py (Nicholas Rejack)
* Replace hcvtarget name for log file with rx2ra in test/TestLog.py (Philip Chase)
* Fix parameter names in test/TestReadConfig.py (Philip Chase)
* Replace hcvtarget name in log file with rx2ra. Prep for reading log file name parameter. (Philip Chase)
* Fix update_redcap_form (Philip Chase)
* Add generic function update_form_field and revise update_formcompletedfieldname to use it (Philip Chase)
* Add commented out code to call update_formcompletedfieldname (Philip Chase)
* Add update_formcompletedfieldname function, tests, and improved formEvents.xml (Philip Chase)
* added docstring for my parse_form_events function. (Nicholas Rejack)
* fixed indentation in my docstring (Nicholas Rejack)
* added docstring for my parse_raw_xml function. (Nicholas Rejack)
* fixed write method (Nicholas Rejack)
* changed redcapFormName to redcap_form_name in rx2ra.py (Nicholas Rejack)
* changed rawXmlFile in config/setup.json to raw_xml_file (Nicholas Rejack)
* changed form_events_file to formEventsFile in config/setup.json (Nicholas Rejack)
* changed translationTableFile to translation_table_file in config/setup.json (Nicholas Rejack)
* changed formName to form_name, component_id_text in rx2ra.py (Nicholas Rejack)
* changed formName to form_name, component_id_text in rx2ra.py (Nicholas Rejack)
* changed translationTable to translation_table, undefinedForm to undefined_form, formName to form_name in rx2ra.py (Nicholas Rejack)
* changed dtString and dateTime to dt_string and date_time in rx2ra.py (Nicholas Rejack)
* changed fileName to file_name in rx2ra.py (Nicholas Rejack)
* changed elementTree argument to element_tree (Nicholas Rejack)
* changed variable names translationTableRoot, translationTableDict, dataRoot, componentID to valid python variable names in rx2ra.py (Nicholas Rejack)
* changed parse_raw_XML to parse_raw_xml in rx2ra.py (Nicholas Rejack)
* changed formEventsTree to form_events_tree in rx2ra.py (Nicholas Rejack)
* changed collectionTime to collection_time in rx2ra.py (Nicholas Rejack)
* changed rawXmlFile to raw_xml_file in rx2ra.py (Nicholas Rejack)
* changed collectionDate to collection_date in rx2ra.py (Nicholas Rejack)
* removed 'file' (a python builtin reserved name) and replaced it with 'item' (Nicholas Rejack)
* reverted previous docstring change. (Nicholas Rejack)
* moved docstring inside module in rx2ra.py (Nicholas Rejack)
* wrapped line 91 in rx2ra.py (over 80 character limit) (Nicholas Rejack)
* changed translationTableFile to translation_table_file in rx2ra.py (Nicholas Rejack)
* changed translationalTableTree to translational_table_tree in rx2ra.py (Nicholas Rejack)
* changed requiredParameters to required_parameters in rx2ra.py (Nicholas Rejack)
* changed formEventsFile to form_events_file in rx2ra.py (Nicholas Rejack)
* changed setupJson to setup_json in rx2ra.py (Nicholas Rejack)
* changes to comply with pylint (Radha Kandula)
* added update formDatefield test function (Radha Kandula)
* updated important usage notes at the top of the file (Radha Kandula)
* changed function name update_formdatefield to comply with standards (Radha Kandula)
* added documentation of how to add the imports for adding testcases (Radha Kandula)
* added default value to failed lookup for update formDateField (Radha Kandula)
* Fix case in formEvents so that formname lookups will work (Philip Chase)
* created update formDatefield functions (Radha Kandula)
* changes made in test files to reflect the changed method names (Radha Kandula)
* Created a test suite (Radha Kandula)
* Add bin/utils/getEmrData.sh, a basic script to moved data to and from the SFTP host. (Philip Chase)
* updated a few file names in rx2ra.py: added test for logging: updated TestUpdateTimestamp indentation changes (Radha Kandula)
* changed updateRedcapForm to update_redcap_form in rx2ra.py (Nicholas Rejack)
* changed updateTimeStamp to update_time_stamp in rx2ra.py (Nicholas Rejack)
* changed writeElementTreeToFile to write_element_tree_to_file in rx2ra.py (Nicholas Rejack)
* changed addElementsToTree to add_elements_to_tree in rx2ra.py (Nicholas Rejack)
* changed parseTranslationTable to parse_translation_table in rx2ra.py (Nicholas Rejack)
* changed parseFormEvents to parse_form_events in rx2ra.py (Nicholas Rejack)
* changed parseRawXml to parse_raw_XML in rx2ra.py (Nicholas Rejack)
* changed readConfig to read_config in rx2ra.py (Nicholas Rejack)
* added logger functionality and put some test logs in functions (Radha Kandula)
* assigned 1 function to NR (Nicholas Rejack)
* Pylint report created and added (Radha Kandula)
* replaced tabs with 4 spaces (Nicholas Rejack)
* PYLINT: updated the code for coding standards & warnings (Radha Kandula)
* updated the correct class name (Radha Kandula)
* cleaning up timestamp function (Nicholas Rejack)
* updated timestamp format for correct separators (Nicholas Rejack)
* modified working updateTimeStamp function (Nicholas Rejack)
* commented out unneeded lines in timeStamp (Nicholas Rejack)
* modified rx2ra.py to output REDCap-style timestamp (Nicholas Rejack)
* added TestSortElementTree.py. incomplete. (Nicholas Rejack)
* Add test/TestUpdateRedcapForm.py (Philip Chase)
* created TestSortElementTree.py (Nicholas Rejack)
* rename makeHcvtargetTranslationTableXML.sh (Philip Chase)
* rename input and output files in makeHcvtargetTranslationTableXML.sh (Philip Chase)
* Add updateRedcapForm function (Philip Chase)
* unit test for the function updateTimeStamp (Radha Kandula)
* changed the access of data from global to local in func timestamp (Radha Kandula)
* update the func updatetimestamp (Radha Kandula)
* updated sorting on study_ID (Nicholas Rejack)
* Remove initialization of data and pointless copy of data (Philip Chase)
* Make data a non-global variable (Philip Chase)
* commented func call to updatetimestamp() (Radha Kandula)
* added updatetimestamp function (Radha Kandula)
* added updatetimestamp function (Radha Kandula)
* Change class name in test/TestReadConfig.py (Philip Chase)
* updated whitespace and comments for legibility (Nicholas Rejack)
* updated whitespace and comments for legibility (Nicholas Rejack)
* adds blank elements to data tree (Nicholas Rejack)
* Add line wrap in test/TestReadConfig.py (Philip Chase)
* deleting the temporarily created test file and added description (Radha Kandula)
* Make all .py files executable (Philip Chase)
* Add file creation and deletion so file error chekcing in readConfig will not raise an exception in testing (Philip Chase)
* Revise readConfig to raise logException on error (Philip Chase)
* Replace each tab with 4 spaces in testwrite.py (Philip Chase)
* Replace each tab with 4 spaces in rx2ra.py (Philip Chase)
* Exclude serialized test data files in .gitignore (Philip Chase)
* deleted aux function (Radha Kandula)
* global variable change (Radha Kandula)
* global variable change (Radha Kandula)
* updated add func call (Nicholas Rejack)
* etree changes (Radha Kandula)
* Add escaping of & in raw.txt input (Philip Chase)
* changed indentation (Radha Kandula)
* added missing tabs in addElementsToTree (Nicholas Rejack)
* Escape < and > symbols in raw.txt input (Philip Chase)
* implemented main,write,auxilaryhandler functions (Radha Kandula)
* Add gitignore lines for .pyc and .pyo files (Philip Chase)
* Add function readConfig, test, and associated config changes (Philip Chase)
* added addElementsToTree function (Nicholas Rejack)
* added XML file reading functions (Nicholas Rejack)
* Fix comments in setup.json (Philip Chase)
* Rename files referenced in rx2ra-dataFlow.graffle and pdf (Philip Chase)
* updated assignments (Radha Kandula)
* updated prototypes (Radha Kandula)
* Rename formElements to formEvents (Philip Chase)
* Add input parameter rawXmlFile in setup.json (Philip Chase)
* added function prototypes (Radha Kandula)
* Add sample input data files. (Philip Chase)
* Consolidate json configuraton data into one file. (Philip Chase)

* Created new project from redi-js and redi-js-config repos

2013-11-22 ( REDI-JS 70d9eaade3e1ca1a0d8b19fef3c92ec32d4f341d no release )

* added prototyping.txt with control flow and functions for program. (Nicholas Rejack)
* prototype new functions (Nicholas Rejack)
* Update rx2ra-dataFlow graffle and pdf files (Philip Chase)
* initial commit of rx2ra.py (Philip Chase)
* Add study_id to dts-event flow. Add study_id to Result_Value flow. (Philip Chase)
* Add study_id/Record to rx2ra.py flowchart (Philip Chase)
* Add data flowchart for rx2ra.py (Philip Chase)
* Add major update of ./src/redi/config and minor update to rediJS.py (Philip Chase)
* Move README.md to doc/README-install.md and add doc/README-development.md (Philip Chase)
* Add a proposed process flow chart (Philip Chase)

* Relocate work folder and ignore and remove log folder (Philip Chase)
* Create separate logger for each jobname, make python3 rediMain.py work from command line (Yang Li)
* add work folder, only clean up data file after importing done (Yang Li)
* Loop jobnames and process single job, need to complete single log for each job (Yang Li)
* Restore the logging code, add some try except blocks in code (Yang Li)
* Add flow charts of redi-js workflows and data flows to redi-js docs (Philip Chase)
* Move extra README files to a doc subdirectory (Philip Chase)
* Improve spelling, grammar, and flow in README-newJob.md (Philip Chase)
* Fleshed out REDCap Form Design Concerns section of REDI-JS README-newJob.md (Philip Chase)
* Fleshed out Human Resources section of REDI-JS README-newJob.md (Philip Chase)
* Fleshed out Technical Requirements section of REDI-JS README-newJob.md (Philip Chase)
* Add Testing sections to redi-js README-newJob.md (Philip Chase)
* Fleshed out Process section of redi-js README-newJob.md (Philip Chase)
* Fix bad wording in config.example/README.md concerning sampleData (Philip Chase)
* Added document outline in README-newJob.md (Philip Chase)
* Create python logging (Yang Li)
* Add REDI-JS documentation (Philip Chase)
* Complete README.md in config.example (Philip Chase)
* Add example job myProject_myForm to config.example (Philip Chase)
* Use configuration files Philip provided, re-coding the python module (Yang Li)
* Start config.example configuration (Philip Chase)
* Add Git repo setup instructions in README.md (Philip Chase)
* Add basic redi-js README.md (Philip Chase)
* Add functions and unit tests for ExportRecord and ImportRecord, also modify code to use config.json (Yang Li)
* Commit code before update submodule (Yang Li)
* Add red-js-config as a submodule at src/redi/config (Philip Chase)
* add .gitignore (Yang Li)
* spelling correction and comment update (Yang Li)
* Create python project with download-API calls and download meta data unittest (Yang Li)
* Initial commit to redi-js (Philip Chase)

2013-11-21 ( redi-js-config 522afd8dbd39f4c86b7dbd92e09ce6cb33ad619b no release )

* Move rx2ra.py to redi-js repo (Philip Chase)
* Add sql to delete sample data from REDCap (Philip Chase)
* Add sample data for separate weeks (Philip Chase)
* Fix out-of-range values in CBC data in hcvtarget.redcapAPISampleData.csv (Philip Chase)
* Fix out of range data. (Philip Chase)
* updated 5 events for 5 subjects -m (Nicholas Rejack)
* Change subjects 101 and 103 to 100 to get multiple events on subject 100 (Philip Chase)
* updated sham data in redcapAPISampleData (Nicholas Rejack)
* convert cbc_lbdtc from date to DTS. Add cbc_complete with value of completed (Philip Chase)
* updated rx2ra.py (Nicholas Rejack)
* added main start (Nicholas Rejack)
* initial rx2ra.py commit (Nicholas Rejack)
* Round out cbc form with hemoglobin and lymphocyte data, translation, and component reports (Philip Chase)
* Add componentReporting tool, source dta and output data files (Philip Chase)
* Restore files in job-config to make rediMain.py happy (Philip Chase)
* Fix event names in hcvtargetFormEvents.xml (Philip Chase)
* Add hcvtargetFormEvents.xml (Philip Chase)
* Add hcvtarget.raw.xml, a sample input file for development of rx2ra.py (Philip Chase)
* Add rawTxt2rawXml.sh to convert a raw xml file coming from the UF IDR into a raw XML file for HCV Target (Philip Chase)
* in csv2xml.py, change incorrectly named 'declaration' parameter to 'xml-declaration' (Philip Chase)
* updated event name to match target data (Nicholas Rejack)
* Relocate old style job configuration files to deprecatedConfigFiles (Philip Chase)
* Relocate makeHcvtargetTranslationTableXML.sh to the tools directory (Philip Chase)
* Add tools/csv2xml.py (Philip Chase)
* update hcvtarget.translationTable.xml that was out of sync with hcvtarget.translationTable.csv (Philip Chase)
* Add hcvtarget.raw.txt, a tab delimited data export from the IDR delivered 2013-11-13 (Philip Chase)
* Add sample data and translation table for the cbc form for one study subject (Philip Chase)
* Add hcvtarget.translationTable.xml derived from hcvtarget.translationTable.csv (Philip Chase)
* Add hcvtarget.translationTable.csv with data for PLATELET COUNT, BILIRUBIN TOTAL, and BILIRUBIN DIRECT (Philip Chase)
* Add .gitignore to ignore work/ (Philip Chase)
* remove comma from the last entry of config.json (Yang Li)
* remove host and path from this file (Yang Li)
* Add gitMode parameter to config.json (Philip Chase)
* Fix bad parameter name in config.json (Philip Chase)
* Set parameter names and test values in config.json (Philip Chase)
* Add screenshots of REDCap forms for hcvtarget* (Philip Chase)
* Add subjects and and additional records to hcvtarget*.sampleData files (Philip Chase)
* Remove formName_complete fields from hcvtarget* sample files (Philip Chase)
* Update sampleData files for hcvtarget jobs based on new excludedFieldPattern files (Philip Chase)
* Removed calculated fields and added a second subject to hcvtarget_chemistry_coags_log.sampleData (Philip Chase)
* Add redcapDataImport.py and enumerateCalculatedFields.sh (Philip Chase)
* Added path to perl to cleanupREDCapXML.pl (Philip Chase)
* Add execute permission on ./tools/cleanupREDCapXML.pl (Philip Chase)
* Add hcvtarget_cbc_log job config files (Philip Chase)
* filled out all field_names in hcvtarget_chemistry_coags_log.json (Philip Chase)
* Update hcvtarget_chemistry_coags_log files: metadata file becomes the dictionary (Philip Chase)
* Add getFormMetadata.sh, a tool for acquiring, cleaning and formatting REDCap form metadata (Philip Chase)
* Add job files for hcvtarget_chemistry_coags_log (Philip Chase)
* Add redi-js-deploy keys (Philip Chase)
* Initial commit of redi-js-config (Philip Chase)