Nemo

Bug fixes

- fixed calendar/tool control bug when using the search bar and it takes 2 searches to show the correct feed
- fixed capitalization in tool customization page
- fixed order of after tool logoff popups

New features
* Added support for pre-usage questions (thanks `Stanford SNF` for the contribution!).
* Added support for multiple areas in area access tablet screen, i.e. a door can provide entry to more than one area (thanks `UPenn Singh Center` for the contribution!).
* Added new `Formula` type of Post usage question, allowing more flexibility when charging for consumables or for converting units based on other fields. Formula type questions are hidden from the user and evaluated after the user answers post usage questions. They can be nested and call another formula results, and they can also be used on a group question in which case the variable becomes a list of all the values (allowing to use `sum(gold_used)` for example, if the user is asked how much gold they used for each of their processes).
* Added User and Staff knowledge base pages, where facilities can list documents, links, and other information for users and staff to browse (for example procedures for staff, training for users etc.).

Improvements
* Made the email preview button in the broadcast email feature scroll to the preview section.
* Now showing access expiration date in users' page.
* Tool superusers can now see the list of qualified users in tool control and there is a new customization option to display that list to everyone. 201
* Added an option (Customization -> Tools) to allow everyone to see the list of qualified users in tool control
* Added options (Customization -> Calendar) to get a confirmation dialog before creating a reservation. It can be set globally and overridden individually by users in their preferences (thanks abuckles-uci, crprice and Alyssa Sheffler from `UC Irvine` for the contribution!).
* Improved menu layout with smaller sizes. The menu will collapse into the hamburger menu when the menu items break the layout (for long names for example).
* Added staff charge info on collapsed menu bar.
* Updated most related fields in detailed admin (user, project, etc.) to use data lists instead of basic drop-downs.
* Now displaying the user's main email address in their preferences page.
* Checking for duplicate emails in email broadcast feature so it's not sent twice to the same address.
* Added support for consumable_id in post usage questions to identify consumable uniquely (since consumable name is not unique).
* Create a new impersonate permission so it can be given to specific users without the need for them to be super admins.
* Added a label on the calendar (top left corner) to show which tool/area/selection the user is looking at. 204
* Added option to only show tool documents to qualified users (thanks `Polytechnique Montreal` for the contribution!).
* Added option to allow everyone to email users with an upcoming reservation (thanks `Stanford SNF` for the contribution!).
* 0 min training records are now allowed.
* Sensor chart time x-axis formats can now be customized in `Customization -> Sensor data`.
* Added support for `read_coils` in sensors' formula (thanks `UPenn Singh Center` for the contribution!).
* Adjustment notifications are now automatically deleted if an adjustment is deleted.
* Added date filtering by creation time for alerts in detailed admin.
* Added training customizations to limit training to just one type, and to allow trainers to set the training date.
* Projects not allowing consumable withdrawals will not be shown in the checkout page anymore.
* Self checkout of consumables can now be limited to only certain users (thanks `Stanford SNF` for the contribution!).
* Badge number will not be shown anymore when badging in and out of area and kiosk. Previous behavior can be restored in Customization -> Application
* Added an option to offer the user to logout of the area they are logged in when disengaging a tool (thanks `UPenn Singh Center` for the contribution!).
* Projects have a new option to not allow staff charges.
* Optimized performance and load time on all pages by refactoring the queries for notifications and customizations.
* Tests will now be published to PyPi as part of the NEMO package. This will allow other plugin developers to easily run NEMO's tests.
* Now limiting the number of workers for gunicorn to a maximum of 9 to avoid issues with resources, database connections and overall performance.

API
* Fixed error with Configuration & Resource API endpoints when it would break if filtering due to many to many relationship not being supported in filters
* Tool status endpoint now requires the `can view tool` permission
* Updated extra fields in API (mostly related fields like users in projects, permissions etc) to not be required and allow null

Bug fixes
* Fixed a bug when reloading the qualified user list in tool control on mobile devices
* Fixed issues with some models when users are given read only permissions in detailed admin
* Fixed a major bug removing the user office role when updating their profile information in `Administration -> Users`
* Fixed a Kiosk issue with links in tooltip that would display NEMO's main page within the kiosk page (thanks r-xyz for the contribution!). 202
* Fixed unique constraint on interlock Card/Channel preventing users from creating interlocks for different unit_id/bank.
* Fixed potential CSRF vulnerability when deleting a buddy request.
* Fixed multi email field issue when using the API and failing validation with empty string as value.
* Fixed inactive users not being able to be used in the usage data history filter.

Libraries
* cryptography 41.0.7 -> 42.0.5
* Django 3.2.23 -> 3.2.25
* python-dateutil 2.8.2 -> 2.9.0
* pytz 2023.3 -> 2024.1

New features
* Users logged in outside their allowed area access (for example users staying after hours in an area when they only have business hours access) will now be shown in red in the status dashboard and other pages showing occupancy like the Jumbotron (thanks `Princeton Micro/NanoFabrication Center` for the contribution!).
* Added customization setting in `Customization -> Calendar` to show configuration settings the user picked directly in the reservation feed, similar to the Configuration agenda feed (thanks `UPenn Singh Center` for the contribution!).
* Added customization setting in `Customization -> Status dashboard` to hide projects the user is working on in the area occupancy tab (thanks `Cornell NanoScale Facility` for the contribution!).
* Added new broadcast email option to send an email to all Principal investigators. It can be filtered to only be sent to PIs of active projects, and can also be filtered down to a subset of selected projects (thanks `Cornell NanoScale Facility` for the contribution!).


Improvements
* Updated jumbotron, kiosk and area access templates to use minified versions of js and css files
* Added customization setting in `Customization -> Tools -> Tool control` for showing task details by default in tool control (thanks abuckles-uci for the contribution!). 185
* Updated configuration history to include slot number in item name when using multiple settings in the same configuration
* Added users to Permission and Group in Detailed administration
* Updated reservation flow on mobile and kiosk to have the user pick a date and time before showing configuration and reservation questions, to be consistent with the desktop calendar flow (thanks `Stanford SNF` for the contribution!).
* Added direct link to the tool control page on the tool information tooltip displayed in the status dashboard. 191
* Optimized access request queries to improve load time on the access requests page
* Added favicon link in all base templates
* In accounts and projects page, the search results will now have [INACTIVE] before the project or account name and inactive ones will be at the bottom of the list


API
* Added project application `iexact` filter to allow case insensitive search for a project by its application identifier.
* Added `users` to authentication permissions and groups endpoints. This allows viewing and setting the list of users through the API.
* Added `users` and `principal_investigators` to projects endpoint. This allows viewing and setting them through the API.
* Reservation's `question_data` field can now be null (thanks NickiShaw for the contribution!). 194
* Added InterlockCardCategories, InterlockCards and Interlocks endpoints.
* Added `/tool_status/` endpoint to return useful current tool information (thanks `Cornell NanoScale Facility` for the contribution!):
* id (integer)
* name (string)
* category (string)
* in_use (boolean)
* visible (boolean)
* operational (boolean)
* problematic (boolean)
* problem descriptions (string: list of problems)
* customer id (integer)
* customer name (string: first name last name)
* customer username (string)
* operator id (integer)
* operator name (string: first name last name)
* operator username (string)
* current usage id (integer)
* current usage start (datetime)
* outages (string: list of current outages)
* partial outages (string: list of current partial outages)
* required resources unavailable (string: list of unavailable resources the tool fully depends on)
* optional resources unavailable (string: list of unavailable resources the tool partially depends on)


Bug fixes
* Fixed bug when searching user preferences in Detailed administration
* Fixed empty string being recorded as a valid answer for required post usage questions
* Fixed tool documents order not being respected in tool control page (reverse order of date uploaded)
* Fixed error in training when entering a wrong tool name or group name would throw an exception

Libraries
* Django 3.2.22 -> 3.2.23
* django-filter 23.4 -> 23.5
* Pillow 10.1.0 -> 10.2.0

Upgrade notes
* Be aware that due to the reservation/configuration model changes, depending on the number of reservations and configurations in your instance, the update to NEMO 5.1.0 can take up to 10-15 minutes.

New features
* Added long awaited color-coded configuration agenda feed in Calendar! Colors are set in `Detailed administration -> Configurations -> Calendar colors` (thanks `UPenn Singh Center` and `Stanford SNF` for the contribution!) 165
* Added transfer of charges from one project to another. The option needs to be enabled in `Customization -> Projects and accounts` (thanks `Stanford SNF` for the contribution!)
* Added new parameters to display tool usage and/or alerts in Kiosk main page (thanks r-xyz for the contribution!) 178
* Added Customization options to rename main menu items: Calendar, Tool Control, Safety, Requests (Thanks abuckles-uci for the contribution!) 181

Improvements
* Allowing to change outage dates when looking at outage details in the calendar. 171
* Consumable categories are now collapsible in supplies page. They can be set to either expanded or collapsed in Customization and are expanded by default (thanks `Stanford SNF` for the contribution!)
* Added customization to send tool problems and updates to superusers. 173
* Now allowing 0 as a valid integer in customization integer lists

API
* Added related object names as valid filters (i.e. `tool` and `tool_id` are both valid for filtering, not only `tool_id`)

Libraries
* cryptography 41.0.4 -> 41.0.7
* django-filter 23.3 -> 23.4
* Pillow 10.0.1 -> 10.1.0

* This new version of NEMO does not contain any new functionality, but the codebase has been entirely reformatted using [black](https://github.com/psf/black).
This means that from now on NEMO's code will be using spaces instead of tabs, and all code and contributions should be formatted using black with the option `--line-length=120`

* Both NEMO and Splash Pad containers are now running `Python 3.11`

Upgrade notes (Forks only)
If you have a fork of NEMO, the easiest way to update is the following:
1. Merge all changes up to NEMO 4.7.1
2. Create a new branch in your fork
3. In PyCharm, change tabs to spaces in Settings -> Editor -> Code Style -> Python -> uncheck "use tab character" and install and set up black in Settings -> Tools -> Black with the `--line-length=120` parameter
4. Run black on the entire *.py project files
5. Merge NEMO 5.0.0
6. Resolve any potential conflicts by accepting your version over NEMO

Bug fixes
- Fixed calendar issue when looking at qualified tools only