Ffgo

[FEATURE] New apt.dat parser; new apt digest file format with an index for
very fast access to all the data present in apt.dat.gz.

* Replace the old apt.dat parser with a new parser. This new parser
should be compliant with the APT1000 specification[1], provide
better error reporting and fix a few small bugs (see below). It
also extracts more information about airports and runways, unifies
recently added extraction of parking metadata from apt.dat.gz with
this airport metadata, and generates an “apt digest file”[2] in a
new format bringing the following improvements:

- more robust (proper header including magic number, encoding
declaration and format version; fields separated by the NUL
char ('\0') instead of '=');

- contains only the information FFGo must read on startup in
order to make this startup as fast as possible; in
particular, runway names for *all* airports are not read
anymore on startup and held into memory, they are loaded on
demand when needed for the selected airport;

- since runway and parking metadata are loaded on demand from
apt.dat (after trying the groundnet file in the case of
parkings), the new format includes an index for each airport
allowing to quickly find its data in apt.dat.gz. Of course,
this requires keeping the apt digest file in sync with
apt.dat.gz, which should not be much of a problem since the
default for “Airport database update” is “Automatic” since
version 1.3.0.

[1] See <http://data.x-plane.com/file_specs/>.

[2] Also known as “airport database”: ~/.ffgo/apt everywhere
except on Windows, where it is %APPDATA%/FFGo/apt.

* The creation of the apt digest file is a bit slower than before
(about 25 seconds on my system, as opposed to 10 or 11 before),
but this should only be necessary when apt.dat.gz changes---which
does not happen very often, the current version in FG 3.7 dating
from 2013---or if I have to again change the format of the apt
digest file.

Note: the slowdown is essentially caused by gzip.GzipFile.tell()
being slow, however this is necessary to create an efficient
index. So, it's a bit slower once in a while when rebuilding
the airport database, but pays off the rest of the time when
using FFGo and looking at the airport, runway or parking
tooltips, or simply choosing among the newly-available
startup locations read (since version 1.6.0) from
apt.dat.gz.

* Startup and config reloading should now be significantly faster
when the airport list is “filtered” (the configuration option that
causes FFGo to only show the airports you already have scenery for).
(Florent Rougon)
[USABILITY] Add a colored, disabled (i.e., not clickable) and empty header to
the runway popup menu. This makes it more visible, visually similar
to the parking popup, and avoids it disappearing in a flash with the
first entry being accidentally selected if the user just clicked
without holding the button down. (Florent Rougon)
[FEATURE] Add tooltips to the airport list giving basic information about the
airport under the mouse pointer (type of airport [land airport,
seaplane base or heliport], latitude, longitude, elevation and, if
applicable, the lists of land runways, water runways and helipads).

The magnetic variation (also called magnetic declination) at the
airport is also displayed in the airport tooltip if GeographicLib's
MagneticField program is available. If this program is not in your
PATH, you can indicate its location using the Preferences dialog.

Note: MagneticField requires a particular dataset to be installed in
order to work properly. In Debian, it is part of the
geographiclib-tools package, which also provides a script
called 'geographiclib-get-magnetic'. Running a command such as
'/usr/sbin/geographiclib-get-magnetic minimal' with root
privileges will automatically download and install a minimal
dataset under /usr/share/GeographicLib/magnetic, where it can
be found by MagneticField (not registering the added files
with dpkg, which is a bit ugly, cf.
<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=687253>).
(Florent Rougon)
[BUGFIX] Fix slightly incorrect calculations of v810 airport coordinates (two
opposite runway ends counted as one). This bugfix comes with the new
apt.dat parser. (Florent Rougon)
[BUGFIX] Fix computation of runway names from v810 data (“6” instead of “06”
for the “other runway” opposed to “24”). This also comes with the
new apt.dat parser.

Note: these fixes are untested, as there is no single runway in v810
format in my apt.dat 2013.10, build 20131335. (Florent Rougon)
[FEATURE] Display magnetic heading in parking tooltips when 'MagneticField' is
available (see above for details concerning 'MagneticField').
(Florent Rougon)
[FEATURE] Tooltip for each runway in the runway popup menu, giving: latitude,
longitude (of the runway end), heading (true and magnetic if
'MagneticField' is available, otherwise only the true heading),
length, width, surface type, shoulder surface type, type of
markings, smoothness.

Helipads also have a line specifying whether they have edge
lighting. Water runways have less fields because things such as
shoulder surface type and smoothness are obviously not applicable.
They have all the data that is present for them in apt.dat.gz,
except for the “perimeter buoys” flag. The code to handle this flag
is ready, but the way it is used in apt.dat.gz is completely bogus
and in contradiction with the APT1000 spec. I'll activate the code
when this is fixed.

Since they are not directly available in apt.dat.gz, the length and
heading of runways are computed from the latitude and longitude of
opposite runway ends. FFGo ships with an implementation of
Vincenty's formula to do this with better than 1 mm precision
(assuming the coordinates of the runway ends are that accurate in
apt.dat.gz!), as well as a few more things to deal with the cases
that Vincenty's formula doesn't handle well.

If GeographicLib's Python implementation is installed, it will be
used in preference to these methods, because it is even more
accurate (about 5 nanometers according to its website), was written
by an expert in geodetic calculations (Charles F. F. Karney), and
apparently handles all cases of the geodetic inverse problem,
contrary to Vincenty's formula. (Florent Rougon)
[BUGFIX] Correct calculation to find the nearest METAR station. The previous
code didn't behave properly for points located on either side of the
±180° meridian, or close to one of the poles. Use a fast method
(GeodCalc.modifiedFccDistance()) that gives reasonably correct
results to find the best 15 candidates for the title of “closest
station”, then find the closest among these using GeodCalc.inverse()
(which is an FFGo abstraction layer for the algorithms mentioned
above: Karney's algorithm from GeographicLib if available, otherwise
Vincenty's formula plus a few others for the cases that Vincenty's
formula doesn't handle well). (Florent Rougon)
[BUGFIX] Always write the timestamp file after rebuilding the apt digest file
(i.e., after “rebuilding the airport database”). With the previous
code, if the airport database was manually rebuilt and afterwards
the “Airport database update” setting was changed from Manual to
Automatic, then the database would be unnecessarily rebuilt. This
should be fixed now. (Florent Rougon)
[USABILITY] Improve some tooltips of the Preferences dialog. (Florent Rougon)
[FEATURE] Update the German translation. (chris_blues)

[BUGFIX] Apparently, the HOME environment variable is not always defined on
Windows. This prevents FFGo from starting in such a case
(<https://sourceforge.net/p/flightgear/mailman/message/34584497/>).
Remove FG_HOME and DEFAULT_TERRASYNC_DIR from ffgo/constants.py as
well as the default setting to DEFAULT_TERRASYNC_DIR for
Config.FG_scenery. The default value for Config.FG_scenery is now
empty, as it used to be some time ago... (Florent Rougon)
[MAINT] Small changes to the AptDat and ToolTip classes. (Florent Rougon)
[FEATURE] Update the German translation. (chris_blues)

[FEATURE] New parser for apt.dat/apt.dat.gz. Should be compliant with the
v1000 spec. Currently, it is only used to lookup fallback parking
positions for a given airport as described below. (Florent Rougon)
[FEATURE] Read parking positions from apt.dat.gz as a fallback. If the
standard lookup from the groundnet files gives no result, look up
the parking data in apt.dat.gz. This is slow, but the last 50
results are cached in memory to mitigate the inconvenience.

With the apt.dat.gz file distributed with current FlightGear 3.7
(which is identical to the latest file distributed on
<http://data.x-plane.com/get_data.html> [and very outdated with
respect to contributions by FlightGear users]), the number of
airports having at least one startup location declared in apt.dat.gz
is 3437. This means that FFGo can offer parking positions/startup
locations for at least this number of airports (the parking data
from FlightGear groundnet files is used preferentially to this data
from apt.dat.gz, but it is maybe possible that some airports have
parking positions declared in a groundnet file but not in
apt.dat.gz). (Florent Rougon; thanks to pommesschranke/d-laser for
the suggestion)
[USABILITY] Small color changes. (Florent Rougon)
[MAINT] Cleanup of code related to the METAR widget (most notably, threading
code). Apart from possible more or less random crashes which should
not happen anymore, the most user-visible result of these changes is
that the widget contents is automatically updated whenever the state
of the “Decoded” checkbox is changed (and every such update is the
result of a new HTTP request). (Florent Rougon)
[MAINT] Various InfoWindow improvements. (Florent Rougon)
[FEATURE] Update the German translation. (chris_blues)

[FEATURE] Add support for context-sensitive translations. (Florent Rougon)
[MAINT] Clean up handling of airport, parking, runway and carrier settings.
(Florent Rougon)
[MAINT] Cleaner handling of runway and parking changes. Most notably,
replace old code checking for runway and parking changes every
250 ms by event handlers (observers in Tkinter-speak). This
increases the interface responsiveness and reduces FFGo's CPU usage.
(Florent Rougon)
[MAINT] Simplify and accelerate the airport list build process. Rewrite the
airport and aircraft search code using Tkinter observers to make the
interface more reactive and reduce CPU usage at the same time.
(Florent Rougon)
[USABILITY] When clicking on the Clear button for the aircraft or airport
search, set focus to the corresponding entry widget to allow
immediate typing of the search text. (Florent Rougon)
[FEATURE] New function, under the Tools menu, to copy a POSIX shell command
equivalent to the current FG command to the clipboard. Due to
limitations in Tk/Tkinter as it seems, this new function can only
copy FG commands that can be encoded to ISO 8859-1 (which should be
sufficient in most cases). An error dialog is displayed if this is
not possible. On X11, the Tkinter clipboard functions used here seem
to use the PRIMARY selection. (Florent Rougon)
[USABILITY] Use a lighter background color for the FlightGear Command Window.
[MAINT] Overhaul of the tooltip infrastructure. New tooltip class:
MenuToolTip. (Florent Rougon)
[BUGFIX] Tooltips could be displayed outside their master widget due to the
fact that (at least with Tk 8.6 and Python 3.4.3) Tkinter sometimes
sends Motion events even though the mouse pointer is largely out of
the master widget. Work around this problem. (Florent Rougon)
[FEATURE] Add airport parking tooltips in the parking popup. The newly-added
tooltips contain most of the metadata available for parking
positions (i.e., everything pilots are likely to be concerned about,
but not FlightGear implementation details such as the parking
'index' or its 'pushBackRoute'). The parking name/number and its
associated flight type are not repeated in the tooltip since they
are already displayed in the parking popup. (Florent Rougon)
[MAINT] Remove polling for:
- comments highlighting in the Option Window;
- the code that updates various elements when a different aircraft
is selected in the aircraft list;
- the code that updates various elements when a different airport
is selected in the airport list.

Again, replace this mechanism by Tkinter observers that are called
immediately after the relevant change without causing unjustified
overhead. Together with the previous optimizations, this causes the
FFGo CPU load when idle to drop from about 2-3 % in version 1.4.0 to
0.0 % in this version on my computer.

Note: in previous versions, the FFGo CPU load was already very low
while FlightGear was running because certain functions were
disabled in this case to save CPU time for FlightGear.
(Florent Rougon)
[BUGFIX] Thanks to the previously-mentioned changes, changing the aircraft or
airport in FFGo, or comment out/decommenting text in the Options
Window while FlightGear is running should work perfectly now,
without leaving related fields unchanged (such as the aircraft
thumbnail when selecting a different aircraft). All this should now
work exactly the same as when FlightGear is not running.
(Florent Rougon)

[BUGFIX] Don't hardcode ':' as path list separator for FG_AIRCRAFT and
FG_SCENERY. Use os.pathsep instead, which should work better on
Windows a least (normally uses ';' since ':' appears in things suchs
as 'C:\path\to\foobar'). Having received no single report from users
using FFGo to run FlightGear under a non-Linux OS, the only way I
can find such portability issues is via code analysis.
(Florent Rougon)
[BUGFIX] Support use of the FG_AIRCRAFT environment variable. Mix its
contents with that of the 'Additional aircraft path(s)' setting from
the Preferences dialog, which is mapped to --fg-aircraft. This
better reflects what FlightGear does. (Florent Rougon)
[FEATURE] Detect, log and display in Help → About the FlightGear version that
FFGo is going to run based on the FG_bin setting from the
Preferences dialog. Nicer display of the Python version too.
(Florent Rougon)
[BUGFIX] Hide tooltips when the mouse pointer is moved. (Florent Rougon)
[FEATURE] Tooltip refactoring. New ListBoxToolTip allowing to have a tooltip
that depends on the ListBox item under the mouse pointer, using a
function of the item index. (Florent Rougon)
[FEATURE] Distinguish between identically-named aircrafts; aircraft tooltips.
FFGo stops using only the aircraft name to identify an aircraft.
From now on, it uses the aircraft name in combination with the
directory containing its -set.xml file. This directory is stored in
a new config file parameter (AIRCRAFT_DIR) to allow reliably using
the same aircraft between sessions even if there are other aircrafts
with the same name in the aircraft paths. The aircraft list now has
tooltips showing the aircraft directory, i.e. the directory
containing the -set.xml file for the aircraft under the mouse
pointer.

Pass the --aircraft-dir option to fgfs (FlightGear) according to the
selected aircraft. If the FlightGear version is lower than 3.8, use
os.path.realpath() on its value (workaround for the FlightGear bug
fixed in commit 7198dec355144fbb0eaccb39f0c241dd07ebaee0, dated
October 6, 2015).

New 'aircraftDir' external variable that can be used in the
configuration processed by CondConfigParser. This variable contains
the selected aircraft's directory. (Florent Rougon)
[BUGFIX] Slightly more efficient aircraft search. Make sure there is always a
selected item in the aircraft list except when it is empty, in
particular after doing a search (an aircraft could previously be
selected for FFGo but not highlighted in the aircraft list).
(Florent Rougon)

[FEATURE] Improve gathering and handling of parking data. All of the parking
metadata is available internally through a Parking class. Errors in
groundnet files generate warnings on the terminal and a popup window
if particularly serious. Refactoring of this part of FFGo.
(Florent Rougon)
[FEATURE] Better sorting of parking positions in the parking popup menu (by
flight type first, then by name using a better algorithm than before,
so that A9 comes before A10, which itself comes before A10a, A10b,
A11, etc.). Here, "flight type" corresponds to the 'type' attribute
value of Parking elements in groundnet files: ga, cargo, gate,
mil-fighter, mil-cargo, vtol. (Florent Rougon)
[BUGFIX] Change the default for “Airport data source” to “Scenery”.
This reflects FlightGear's current behavior when processing
--parkpos and should provide much better parking data for people
using TerraSync. The old default didn't make use of the TerraSync
data, which was a pity.

As indicated in ffgo/data/help/help_en, it is necessary to have
downloaded data for the desired airport in one of the FG_SCENERY
components before parking positions can be shown for this airport
with the new default setting.

Actually, “Airport data source” is a misnomer. What it really means
is “Parking data source”. But since it corresponds to the
APT_DATA_SOURCE parameter in the config file, and has been so for
years (in FGo!), I'll leave it as is for now and just try to
describe the effects as clearly as possible in the associated
tooltip. (Florent Rougon)
[BUGFIX] When “Airport data source” is set to “Scenery”, stop at the first
groundnet file found in FG_SCENERY for the selected airport, even if
it doesn't declare any parking position. This is what FlightGear
(3.7) does, therefore doing otherwise and offer, in case of
redundant FG_SCENERY components, parking positions that FlightGear
won't find is not a good idea IMHO. (Florent Rougon)
[BUGFIX] For the same reason, don't attempt to read parking data from
FG_SCENERY/Airports/I/C/A/ICAO.parking.xml. This was used when
"Airport data source" was set to Scenery, but FlightGear doesn't
read these files. (Florent Rougon)
[USABILITY] Change the default for “Airport database update” to “Automatic”. I
think this is more likely to improve user experience than to degrade
it. (Florent Rougon)
[FEATURE] Interactive dialogs proposing to change the “Airport data source”
and “Airport database update” settings to their new defaults. Each
question is asked only once (if applicable). Give an empty value to
ALREADY_PROPOSED_CHANGES in the config file if you want to see the
questions again. (Florent Rougon)
[BUGFIX] Don't special-case FILTER_APT_LIST in Config.update(). It seems
counter-intuitive to me that the “Reload config” button reloads
everything from the config file, except the FILTER_APT_LIST setting
(which corresponds to the “Show installed airports only” entry of
the Settings menu). Remove this special case. (Florent Rougon)
[USABILITY] Enabled/disabled state for the “Update list of installed airports”
menu entry. Since this menu entry is only applicable when “Show
installed airports only” is selected, enable or disable it as
appropriate. (Florent Rougon)
[BUGFIX] Fix button size (“Reload config” & Co). In non-English translations,
some buttons were too narrow for the text to fit. This should be
better now. (Florent Rougon)
[FEATURE] Print the Python and CondConfigParser versions in the About dialog
box. This should make it easier to help users or even to help them
help themselves. :-) (Florent Rougon)
[FEATURE] Update the French translation of the interface. (Florent Rougon)