Longbow

-------------

1. Bug fix - OMP environment variable added to all schedulers to fix a specific
set of user reported issues (issue 114).

2. Bug fix - Further PBS and NAMD SMP issues relating to under subscription,
users had to do a hacky way by dropping the corespernode parameter to under
subscribe which resulted in errors from the scheduler. Now users wishing to
do this should set mpiprocs in their job or host conf files to do this (issue 105).

3. Bug fix - The memory parameter only worked with the PBS scheduler, this has
now been extended to work in all schedulers (issue 98).

4. Bug fix - The --maxtime parameter went missing from the --help output (issue 96).

5. Doc fix - The documentation for the recovery mode was incorrect (issue 102).

6. New feature - An update mode has been added so that users doing
disconnectable sessions can simply run an update to get the current
simulation status and download a snapshot of the data (issue 61).

7. New feature - Move documentation to be under version control, using sphinx
and readthedocs for auto documentation assembly. Documentation can then
become part of the CI cycle and thus be enforced on code contribution (issue 90).

8. Enhancement - users can now explicitly set the filenames of stderr and
stdout from the scheduler script using the parameters "stdout = filename"
and "stderr = filename" in their host of job conf files (issue 108).

9. Enhancement - Users can now make use of existing job scripts, by providing
the name of the script to the parameter "subfile" in their host or job conf
files. This mode is mainly aimed at advanced users that understand the short
falls of doing this and the problems that could occur (issue 77).

10. Enhancement - Users can now set the naming scheme of the replicate
directories. Instead of having to provide directories of the form
rep1, rep2, ...., repx. Users can now set the name of the "rep" part by
setting the "replicate-naming" parameter. So "replicate-naming = foo"
would need directories named foo1, foo2, ...., foox (issue 92).

11. Enhancement - Documentation for the examples have been cleaned up and added
to the new sphinx docs (issue 10).

12. Enhancement - Refactor the exception code in the top level API methods to
remove duplication (issue 44).

13. Removed support for Python versions 2.6, 3.2 and 3.3 due to these versions
being old unsupported versions and various python packages such as
pip/ci-tools withdrawing support. Longbow may still work for these versions
but this is no longer guaranteed (issue 113).

-------------

1. Bug fix - a number of parameters that are only used in specific scheduler or
application plugins have been renamed to include the plugin name prefix (issue 54).

2. Bug fix - executables expressed as absolute paths for supported plugins
would cause a crash due to searching for a module with that key (issue 63).

3. Bug fix - wrong error message was displayed when an executable didn't exist
on HPC machine (issue 65).

4. Bug fix - fix for crash when using NAMD SMP builds, the commandline
parameters beginning with "+" would trigger the crash (issue 66).

5. Bug fix - fixed misleading error messages about missing files and flags (issue 67).

6. Bug fix - fixed problem where the newly added bash autocomplete did not
allow filenames on disk to autocomplete (issue 68).

7. Bug fix - fixed a problem when a large number of short jobs that trigger job
subqueuing would cause a crash (issue 72).

8. Bug fix - fixed user reported bug with strange looking error messages
concealing a further absolute path bug (issue 75).

9. New feature - Added support for the slurm gres flag so users can do
something like this "slurm-gres = gpu:1" in their host or job conf files (issue 76).

10. Bug fix - Restored the ability to issue --maxtime on the commandline (issue 78).

11. New feature - support added for the upcoming release of python chemshell (issue 80).

12. Bug fix - fix for problem where parameters in configuration files
containing the "=" sign would cause the input file parser to misread
them (issue 81).

13. Bug fix - fix for problem with LAMMPS jobs where files provided with
"include" or as part of other parameters would not be transferred, thanks
to Anders Johansson for suggesting some ideas and solutions (issue 86).

14. The structure of the Longbow API has been simplified, the source files no
longer reside in a subdirectory called "core" within the installation
directory. This has made importing much simpler and imports shorter.

15. Examples have been restructured and the how to run instructions updated.
The actual run files remain the same, there are still incompatibilities
with these and newer versions of MD codes but this will be addressed in
the next version.

-------------

1. Change of license from GPLv2 to the BSD 3 clause license. This will resolve
the copy-left issues a number of projects are having (Issue 30).

2. Fix for problem where environment is not properly loaded on some machines,
this is a non-login shell related problem (Issue 4).

3. Removal of capitalisation from around the whole project for convenience
(Issues 26 28 46 47).

4. Longbow will now clean out recovery files upon successful completion of the
session (Issue 24).

5. Bash autocomplete creation added to the setup.py (Issue 23 49).

6. Fix for bad exit from recovery mode when user issues a keyboard interrupt
(ctrl-c) (Issue 25).

7. Fix for monitoring phase causing CPU to be spun up to 100% (Issue 35).

8. Fix for bug when examples are downloaded in app, the original zip archive
is not removed after extraction (Issue 34).

9. Fixed a bug in setup script with the python version check code (Issue 32).

10. Python 3.6 added to supported versions (Issue 33).

11. All public methods in the Longbow API can now be accessed from a single
import, (ie "import longbow" then "longbow.somemethod()") (Issue 27).

-------------

This release is a rather large release with a lot of changes to the code and
also to the way things are being done. This project is moving towards using CI
tools more and more, and in this release and all future releases tools for
automated unit testing and code quality checks will be used.

In this release the following changes have been made in this regard:

1. Travis-CI is now used to automate testing for all versions of python 2.6
through 3.5 https://travis-ci.org/HECBioSim/Longbow.

2. Unit testing added to repository and increased to 100% code coverage. Badges
have also been implemented so users can monitor success rate. Code coverage
is picked up from Travis and compiled at
https://coveralls.io/github/HECBioSim/Longbow.

3. Automated running of code quality checks is now triggered via github pushes
and is done at https://landscape.io/github/HECBioSim/Longbow

These tools will eventually be used to test upon merge into the main two
branches of the repo (master and development) and will have tolerances set for
auto-rejection.

The actual changes to the code are:

1. Added in some extra commonly found naming for GROMACS executables
(gmx_mpi and gmx_mpi_d).

2. Bugs detected in shellwrappers.py during unit test writing have now been
fixed.

3. Fixed formatting issue with logging in staging.py - stage_downstream().

4. Removed import statements for relative imports, this eliminated the need for
the try/except imports at the top of each code module.

5. The parameter "frequency" has been split into two parameters
"polling-frequency" and "staging-frequency". This enables the user to have
Longbow poll jobs without staging all the time.

6. The timing mechanism for timing between polling events has now been changed
from a disruptive wait() to a timestamp comparison. This stops the process
being "blocked" by wait() and in future will allow the addition of other
features that can happen on different timelines to polling.

7. There are now two steps to job completion, instead of marking a job as
finished when it is finished on the remote host and results downloaded,
Longbow will now mark at as finished as it is finished on the remote host
and then once staging has happened it will then be marked as complete.

8. Fixed glitch in applications.py processjobs() where the wrong comparitor was
used on the if statement to construct the upload include list.

9. Huge refactoring of code to break down larger methods into smaller easier to
test methods. All new methods are private methods (starting with "_") these
should not be used by people making use of Longbow in their own code, unless
they know what they are doing!

10. Moved all code for Longbow entrypoints out of the executable and into
corelibs/entrypoints.py, this allows more options for integration and also
simplifies it somewhat since the library top level can now be imported from
the library (otherwise hooking against the executable or copying the code
into a project was the only way).

11. The plugin framework no longer has a complicated path such as
plugins/apps/gromacs.py, now two directories "apps" and "schedulers" sit at
the same level as corelibs in the library. This removed most of the
complexity in the import system, now it is a very simple and elegant way to
provide plugins for these two categories.

12. Fixed the problem of returning information upon job submission error, this
was a typo in each of the scheduler plugins.

13. Fix for strange job status glitch when jobid appeard in say a timestamp or
some other parameter in the output of qstat etc, this has been fixed across
all schedulers.

14. Fixed spacing problem in some generated job submit files.

15. Modification into the way command-lines are parsed, this is so that the
detection of executables and their commands as well as Longbow commands is
much more robust and can now handle arbitrary executables.

16. Addition of --nochecks command-line flag, this will disable testing whether
the application is available on the remote host. In some circumstances it
is very difficult to get Longbow to recognise an application is installed.
This is aimed at advanced users only.

17. All parameters in the main entry point that were previous passed into
longbow have now all been assimilated into one dictionary "parameters" this
then allowed refactoring all the switch cases for command-line parameters
into a much neater single method.

18. Fix for problem detected during unit test writing for configurations.py
saveconfigs() where if used incorrectly would blow up.

19. Fixed problem in applications.py causing failed run under python 3.2.

20. Fix for filenaming glitch when using global files in replicates.

21. Fixed problem where if required files were not found this was ignored but
should have been flagged up.

22. Fixed a number of bad initialisers that would cause filenames to go
missing.

23. legacy code in status method in all schedulers removed.

24. Fixed freezing glitch when all jobs failed to submit.

25. Fixed bad parameter in substitutions.

-------------

1. Fix for new gromacs packaging where the gmx mdrun CLI would not be
recognised.

2. Fix for case in gromacs where if input file -s and -deffnm would be provided
together and a file was a global file in a replicate job that all files
would be set to output global.

3. Can now use -deffnm with gromacs on replicate jobs with global files without
having to also set -s.

4. Fix for missing space in sge replicate script generator.

5. Added the ability for emails to be sent to the user, these are invoked by
providing the email address in a submit file by
'email-address = blahblah.com' and also to set the email flags, these flags
should be the same ones and same format you use in your submit script for
example 'email-flags = -M' or 'email-flags = ib'.

6. New common executable naming schemes added for things like cuda.

7. Fixed some missing newline characters from the job submit script generator.

8. Recovery system broken by renaming the method to same as a variable, this
has now been resolved.

9. Rewrites to the applications.py module, it was too unwield to add to. This
has now been split down to allow future expansion in a much easier fashion.

-------------

1. For machines running SGE a new parameter to control the flag used on the -pe
directive has been implemented. This parameter is "sge-peflag" and has a
default value of "mpi", if your cluster requires something different then
use this parameter in a configuration file to set it.

2. Some cases on SGE clusters, it can be set that even if a job needs only 1
core that the -pe mpi cores must be set. A new flag called "sge-peoverride"
will make this happen, just provide "sge-peoverride = true" in a
configuration file to enable.

3. Fixed a problem where Longbow would exit when no jobs are running, this was
fixed in PBS but not in other schedulers. It is now fixed in all supported
schedulers.

4. The import statements that looked pretty nasty across the library have been
changed for nicer more pythonic ones.

5. Some refactoring to get rid of pylint warnings about not using lazy logging.

6. Ability to add script calls into the job submission script that is created
by Longbow. The parameter "scripts" should be used, for multiple script
references then a comma separated list should be provided. Just add
something like this to your configuration file "scripts = source /some/file"
or "scripts = source /first/file, source /second/file".

7. Fix a problem when using job configuration files and not referencing a
resource, what should have happened is that Longbow should choose the
default (top one in hosts.conf). But it crashed with a KeyError exception,
this is now fixed.