Pyftpdlib

=================================

**Bug fixes**

- 236: MultiprocessFTPServer and ThreadedFTPServer hanging in case of failed
authentication.

=================================

**Enhancements**

- 76: python 3.x porting.
- 198: full unicode support (RFC-2640).
- 203: asyncore IO loop has been rewritten from scratch and now supports
epoll() on Linux and kqueue() on OSX/BSD.
Also select() (Windows) and poll() pollers have been rewritten
resulting in pyftpdlib being an order of magnitude faster and more
scalable than ever.
- 204: a new FilesystemError exception class is available in order send
custom error strings to client from an AbstracteFS subclass.
- 207: added on_connect() and on_disconnect() callback methods to FTPHandler
class.
- 212: provided two new classes:
Logging_managementpyftpdlib.servers.ThreadedFTPServer and
pyftpdlib.servers.MultiprocessFTPServer (POSIX only).
They can be used to change the base async-based concurrecy model and
use a multiple threads / processes based approach instead.
Your FTPHandler subclasses will finally be free to block! ;)
- 219: it is not possible to instantiate different FPTS classes using
different SSL certificates.
- 213: DummyAuthorizer.validate_authentication() has changed in that it
no longer returns a bool but instead raises AuthenticationFailed()
exception to signal a failed authentication.
This has been done in order allow customized error messages on failed
auth. Also it now expects a third 'handler' argument which is passed in
order to allow IP-based authentication logic. Existing code overriding
validate_authentication() must be changed in accordance.
- 223: ftpserver.py has been split in submodules.
- 225: logging module is now used for logging. ftpserver.py's log(), logline()
and logerror() functions are deprecated.
- 231: FTPHandler.ftp_* methods implementing filesystem-related commands
now return a meaningful value on success (tipically the path name).
- 234: FTPHandler and DTPHandler class provide a nice __repr__.
- 235: FTPServer.serve_forever() has a new handle_exit parameter which
can be set to False in order to avoid handling SIGTERM/SIGINT signals
and logging server start and stop.
- 236: big logging refactoring; by default only useful messages are logged
(as opposed to *all* commands and responses exchanged by client and
server). Also, FTPHandler has a new 'log_prefix' attribute which can
be used to format every line logged.

**Bug fixes**

- 131: IPv6 dual-stack support was broken.
- 206: can't change directory (CWD) when using UnixAuthorizer and process
cwd is == "/root".
- 211: pyftpdlib doesn't work if deprecated py-sendfile 1.2.4 module is
installed.
- 215: usage of FTPHandler.sleeping attribute could lead to 100% CPU usage.
FTPHandler.sleeping is now removed. self.add_channel() /
self.del_channel() should be used instead.
- 222: an unhandled exception in handle_error() or close() can cause server
to crash.
- 229: backslashes on UNIX are not handled properly.
- 232: hybrid IPv4/IPv6 support is broken. (patch by Claus Klein)

**New modules**

All the code contained in pyftpdlib/ftpserver.py and pyftpdlib/contrib
namespaces has been moved here:

- pyftpdlib.authorizers
- pyftpdlib.filesystems
- pyftpdlib.servers
- pyftpdlib.handlers
- pyftpdlib.log

**New APIs**

- pyftpdlib.authorizers.AuthenticationFailed
- pyftpdlib.filesystems.FilesystemError
- pyftpdlib.servers.ThreadedFTPServer
- pyftpdlib.servers.MultiprocessFTPServer
- pyftpdlib.handlers.FTPHandler's on_connect() and on_disconnect() callbacks.
- pyftpdlib.handlers.FTPHandler.ftp_* methods return a meaningful value on
success.
- FTPServer, FTPHandler, DTPHandler new ioloop attribute.
- pyftpdlib.lib.ioloop.IOLoop class (not supposed to be used directly)
- pyftpdlib.handlers.FTPHandler.log_prefix

**Deprecated name spaces**

- pyftpdlib.ftpserver
- pyftpdlib.contrib.*

**Incompatible API changes**

- All the main classes have been extracted from ftpserver.py and split into sub
modules.

+-------------------------------------+---------------------------------------+
| Before | After |
+=====================================+=======================================+
| pyftpdlib.ftpserver.FTPServer | pyftpdlib.servers.FTPServer |
+-------------------------------------+---------------------------------------+
| pyftpdlib.ftpserver.FTPHandler | pyftpdlib.handlers.FTPHandler |
+-------------------------------------+---------------------------------------+
| pyftpdlib.ftpserver.DTPHandler | pyftpdlib.handlers.DTPHandler |
+-------------------------------------+---------------------------------------+
| pyftpdlib.ftpserver.DummyAuthorizer | pyftpdlib.authorizers.DummyAuthorizer |
+-------------------------------------+---------------------------------------+
| pyftpdlib.ftpserver.AbstractedFS | pyftpdlib.filesystems.AbstractedFS |
+-------------------------------------+---------------------------------------+

Same for pyftpflib.contribs namespace which is deprecated.

+-------------------------------------------------+-----------------------------------------+
| Before | After |
+=================================================+=========================================+
| pyftpdlib.contrib.handlers.TLS_FTPHandler | pyftpdlib.handlers.TLS_FTPHandler |
+-------------------------------------------------+-----------------------------------------+
| pyftpdlib.contrib.authorizers.UnixAuthorizer | pyftpdlib.authorizers.UnixAuthorizer |
+-------------------------------------------------+-----------------------------------------+
| pyftpdlib.contrib.authorizers.WindowsAuthorizer | pyftpdlib.authorizers.WindowsAuthorizer |
+-------------------------------------------------+-----------------------------------------+
| pyftpdlib.contrib.filesystems.UnixFilesystem | pyftpdlib.filesystems.UnixFilesystem |
+-------------------------------------------------+-----------------------------------------+

Both imports from pyftpdlib.ftpserver and pyftpdlib.contrib.* will still work
though and will raise a DeprecationWarning exception.

**Other incompatible API changes**

- DummyAuthorizer.validate_authentication() signature has changed. A third
'handler' argument is now expected.
- DummyAuthorizer.validate_authentication() is no longer expected to return a
bool. Instead it is supposed to raise AuthenticationFailed(msg) in case of
failed authentication and return None otherwise.
(see issue 213)
- ftpserver.py's log(), logline() and logerror() functions are deprecated.
logging module is now used instead. See:
https://pyftpdlib.readthedocs.io/en/latest/tutorial.html#logging-management
- Unicode is now used instead of bytes pretty much everywhere.
- FTPHandler.__init__() and TLS_FTPHandler.__init__() signatures have changed:
from __init__(conn, server)
to __init__(conn, server, ioloop=None)
- FTPServer.server_forever() signature has changed:
from serve_forever(timeout=1.0, use_poll=False, count=None)
to serve_forever(timeout=1.0, blocking=True, handle_exit=True)
- FTPServer.close_all() signature has changed:
from close_all(ignore_all=False)
to close_all()
- FTPServer.serve_forever() and FTPServer.close_all() are no longer class
methods.
- asyncore.dispatcher and asynchat.async_chat classes has been replaced by:
pyftpdlib.ioloop.Acceptor
pyftpdlib.ioloop.Connector
pyftpdlib.ioloop.AsyncChat
Any customization relying on asyncore (e.g. use of asyncore.socket_map to
figure out the number of connected clients) will no longer work.
- pyftpdlib.ftpserver.CallLater and pyftpdlib.ftpserver.CallEvery are
deprecated. Instead, use self.ioloop.call_later() and self.ioloop.call_every()
from within the FTPHandler. Also delay() method of the returned object has
been removed.
- FTPHandler.sleeping attribute is removed. self.add_channel() and
self.del_channel() should be used to pause and restart the handler.

**Minor incompatible API changes**

- FTPHandler.respond(resp) -> FTPHandler.respond(resp, logfun=logger.debug)
- FTPHandler.log(resp) -> FTPHandler.log(resp, logfun=logger.info)
- FTPHandler.logline(resp) -> FTPHandler.logline(resp, logfun=logger.debug)

=================================

**Enhancements**

- 152: uploads (from server to client) on UNIX are now from 2x (Linux) to 3x
(OSX) faster because of sendfile(2) system call usage.
- 155: AbstractedFS "root" and "cwd" are no longer read-only properties but
can be set via setattr().
- 168: added FTPHandler.logerror() method. It can be overridden to provide
more information (e.g. username) when logging exception tracebacks.
- 174: added support for SITE CHMOD command (change file mode).
- 177: setuptools is now used in setup.py
- 178: added anti flood script in demo directory.
- 181: added CallEvery class to call a function every x seconds.
- 185: pass Debian licenscheck tool.
- 189: the internal scheduler has been rewritten from scratch and it is an
order of magnitude faster, especially for operations like cancel()
which are involved when clients are disconnected (hence invoked very
often). Some benchmarks:
schedule: +0.5x,
reschedule: +1.7x,
cancel: +477x (with 1 million scheduled functions),
run: +8x
Also, a single scheduled function now consumes 1/3 of the memory thanks
to ``__slots__`` usage.
- 195: enhanced unix_daemon.py script which (now uses python-daemon library).
- 196: added callback for failed login attempt.
- 200: FTPServer.server_forever() is now a class method.
- 202: added benchmark script.

**Bug fixes**

- 156: data connection must be closed before sending 226/426 reply. This was
against RFC-959 and was causing problems with older FTP clients.
- 161: MLSD 'unique' fact can provide the same value for files having a
similar device/inode but that in fact are different.
(patch by Andrew Scheller)
- 162: (FTPS) SSL shutdown() is not invoked for the control connection.
- 163: FEAT erroneously reports MLSD. (patch by Andrew Scheller)
- 166: (FTPS) an exception on send() can cause server to crash (DoS).
- 167: fix some typos returned on HELP.
- 170: PBSZ and PROT commands are now allowed before authentication fixing
problems with non-compliant FTPS clients.
- 171: (FTPS) an exception when shutting down the SSL layer can cause server
to crash (DoS).
- 173: file last modification time shown in LIST response might be in a
language different than English causing problems with some clients.
- 175: FEAT response now omits to show those commands which are removed from
proto_cmds map.
- 176: SO_REUSEADDR option is now used for passive data sockets to prevent
server running out of free ports when using passive_ports directive.
- 187: match proftpd LIST format for files having last modification time
> 6 months.
- 188: fix maximum recursion depth exceeded exception occurring if client
quickly connects and disconnects data channel.
- 191: (FTPS) during SSL shutdown() operation the server can end up in an
infinite loop hogging CPU resources.
- 199: UnixAuthorizer with require_valid_shell option is broken.

=================================

**Enhancements**

- 68: added full FTPS (FTP over SSL/TLS) support provided by new
TLS_FTPHandler class defined in pyftpdlib.contrib.handlers module.
- 86: pyftpdlib now reports all ls and MDTM timestamps as GMT times, as
recommended in RFC-3659. A FTPHandler.use_gmt_times attributed has
been added and can be set to False in case local times are desired
instead.
- 124: pyftpdlib now accepts command line options to configure a stand alone
anonymous FTP server when running pyftpdlib with python's -m option.
- 125: logs are now provided in a standardized format parsable by log
analyzers. FTPHandler class provides two new methods to standardize
both commands and transfers logging: log_cmd() and log_transfer().
- 127: added FTPHandler.masquerade_address_map option which allows you to
define multiple 1 to 1 mappings in case you run a FTP server with
multiple private IP addresses behind a NAT firewall with multiple
public IP addresses.
- 128: files and directories owner and group names and os.readlink are now
resolved via AbstractedFS methods instead of in format_list().
- 129, 139: added 4 new callbacks to FTPHandler class:
on_incomplete_file_sent(), on_incomplete_file_received(), on_login()
and on_logout().
- 130: added UnixAuthorizer and WindowsAuthorizer classes defined in the new
pyftpdlib.contrib.authorizers module.
- 131: pyftpdlib is now able to serve both IPv4 and IPv6 at the same time by
using a single socket.
- 133: AbstractedFS constructor now accepts two argumets: root and cmd_channel
breaking compatibility with previous version. Also, root and and cwd
attributes became properties. The previous bug consisting in resetting
the root from the ftp handler after user login has been fixed to ease
the development of subclasses.
- 134: enabled TCP_NODELAY socket option for the FTP command channels
resulting in pyftpdlib being twice faster.
- 135: Python 2.3 support has been dropped.
- 137: added new pyftpdlib.contrib.filesystems module within
UnixFilesystem class which permits the client to escape its home
directory and navigate the real filesystem.
- 138: added DTPHandler.get_elapsed_time() method which returns the transfer
elapsed time in seconds.
- 144: a "username" parameter is now passed to authorizer's
terminate_impersonation() method.
- 149: ftpserver.proto_cmds dictionary refactoring and get rid of
_CommandProperty class.

**Bug fixes**

- 120: an ActiveDTP() instance is not garbage collected in case a client
issuing PORT disconnects before establishing the data connection.
- 122: a wrong variable name was used in AbstractedFS.validpath method.
- 123: PORT command doesn't bind to correct address in case an alias is
created for the local network interface.
- 140: pathnames returned in PWD response should have double-quotes '"'
escaped.
- 143: EINVAL not properly handled causes server crash on OSX.
- 146: SIZE and MDTM commands are now rejected unless the "l" permission has
been specified for the user.
- 150: path traversal bug: it is possible to move/rename a file outside of the
user home directory.

=================================

**Enhancements**

- 103: added unix_daemon.py script.
- 108: a new ThrottledDTPHandler class has been added for limiting the speed
of downloads and uploads.

**Bug fixes**

- 100: fixed a race condition in FTPHandler constructor which could throw an
exception in case of connection bashing (DoS). (thanks Bram Neijt)
- 102: FTPServer.close_all() now removes any unfired delayed call left behind
to prevent potential memory leaks.
- 104: fixed a bug in FTPServer.handle_accept() where socket.accept() could
return None instead of a valid address causing the server to crash.
(OS X only, reported by Wentao Han)
- 104: an unhandled EPIPE exception might be thrown by asyncore.recv() when
dealing with ill-behaved clients on OS X . (reported by Wentao Han)
- 105: ECONNABORTED might be thrown by socket.accept() on FreeBSD causing the
server to crash.
- 109: an unhandled EBADF exception might be thrown when using poll() on OSX
and FreeBSD.
- 111: the license used was not MIT as stated in source files.
- 112: fixed a MDTM related test case failure occurring on 64 bit OSes.
- 113: fixed unix_ftp.py which was treating anonymous as a normal user.
- 114: MLST is now denied unless the "l" permission has been specified for the
user.
- 115: asyncore.dispatcher.close() is now called before doing any other
cleanup operation when client disconnects. This way we avoid an endless
loop which hangs the server in case an exception is raised in close()
method. (thanks Arkadiusz Wahlig)
- 116: extra carriage returns were added to files transferred in ASCII mode.
- 118: CDUP always changes to "/".
- 119: QUIT sent during a transfer caused a memory leak.

=================================

**Enhancements**

- 79: added two new callback methods to FTPHandler class to handle
"on_file_sent" and "on_file_received" events.
- 82: added table of contents in documentation.
- 92: ASCII transfers are now 200% faster on those systems using "\r\n" as
line separator (typically Windows).
- 94: a bigger buffer size for send() and recv() has been set resulting in a
considerable speedup (about 40% faster) for both incoming and outgoing
data transfers.
- 98: added preliminary support for SITE command.
- 99: a new script implementing FTPS (FTP over TLS/SSL) has been added to the
demo directory. See:
https://code.google.com/p/pyftpdlib/source/browse/trunk/demo/tls_ftpd.py

**Bug fixes**

- 78: the idle timeout of passive data connections gets stopped in case of
rejected "site-to-site" connections.
- 80: demo/md5_ftpd.py should use hashlib module instead of the deprecated md5
module.
- 81: fixed some tests which were failing on SunOS.
- 84: fixed a very rare unhandled exception which could occur when retrieving
the first bytes of a corrupted file.
- 85: a positive MKD response is supposed to include the name of the new
directory.
- 87: SIZE should be rejected when the current TYPE is ASCII.
- 88: REST should be rejected when the current TYPE is ASCII.
- 89: "TYPE AN" was erroneously treated as synonym for "TYPE A" when "TYPE L7"
should have been used instead.
- 90: an unhandled exception can occur when using MDTM against a file modified
before year 1900.
- 91: an unhandled exception can occur in case accept() returns None instead
of a socket (it happens sometimes).
- 95: anonymous is now treated as any other case-sensitive user.