Inductiva

List of changes in the v0.10 release

Table of contents
- [Resources costs](costs)
- [Tasks cost](taskcosts)
- [Storage cost](storagecosts)
- [Improve visibility of the Task’s life cycle](taskslife)
- [Terminate an ongoing Task from the Console](killtask)
- [Input and output files: Download to local computer](download)
- [Get a URL to download input and output files](url)
- [Download the Task’s Logs from Console](logs)
- [c4 and n4 machines available](machines)
- [New simulator Quantum Espresso](simulator)
- [More rich content](content)

Resources costs <a name="costs"></a>
Users were already aware of their [credits](https://docs.inductiva.ai/en/latest/api_reference/tiers_and_quotas.html), and now they also gain visibility over the costs of the resources they use and create, namely tasks and storage.

Tasks cost <a name="taskcosts"></a>
[396](https://github.com/inductiva/tasks/issues/396), [#438](https://github.com/inductiva/tasks/issues/438), [#161](https://github.com/inductiva/console-web/issues/161), [#186](https://github.com/inductiva/console-web/issues/186), [#454](https://github.com/inductiva/tasks/issues/454)
In order for the user to realize the estimated cost of each task, it’s now displayed together with other task’s details, namely:
- In [CLI](https://docs.inductiva.ai/en/latest/cli/cli-overview.html), by running the command inductiva task info <task_id>;
- In the prints resulting from task info or task get_info() in python scripts;
- In the [Console](https://console.inductiva.ai/), in the Tasks list, as well as each task’s detail.

Storage cost <a name="storagecosts"></a>
[447](https://github.com/inductiva/tasks/issues/447), [#194](https://github.com/inductiva/console-web/issues/194)
After the simulations finish, the outputs are stored in the user’s remote storage bucket, at a cost. Here’s how to check the total storage size used and its Cost per month:
In [CLI](https://docs.inductiva.ai/en/latest/cli/cli-overview.html), by running the command inductiva storage size or inductiva storage list;
In the [Console](https://console.inductiva.ai/), on top of the Storage area.

Improve visibility of the Task’s life cycle <a name="taskslife"></a>
[413](https://github.com/inductiva/tasks/issues/413), [#412](https://github.com/inductiva/tasks/issues/412), [#175](https://github.com/inductiva/console-web/issues/175), [#402](https://github.com/inductiva/tasks/issues/402)
A new feature was created to enable the user to [track their task](https://docs.inductiva.ai/en/latest/cli/tracking-tasks.html) in real time and better perceive the stages it goes through, why it got there and realize the recommendations to overcome or avoid an unwanted status.
This feature coordinates several resources, so that the user takes advantage of the range of assets provided by Inductiva, while being supplied with different degrees of detail - succinct and to-the-point descriptions for immediate understanding and action, and more detailed descriptions for deeper comprehension of the tasks’ life cycle.
Here’s a quick overview of the ensemble:
- If the user chooses to run task wait() in their script, the [CLI](https://docs.inductiva.ai/en/latest/cli/cli-overview.html) will print regular short messages that help the user keep track of what’s happening.
- A link to the Console is also printed, so that the user can seamlessly navigate to the ongoing task’s detail screen, where a bit more descriptive information is provided.
- In the task’s detail screen in the Console, a timeline can be found with the statuses that the task has consecutively reached and their respective timestamps. By clicking these statuses an expanded area becomes visible, with a brief description of the context behind it.
- For the users who’d like to go deeper into the tasks’ life cycle, a link to the [Tutorials](https://tutorials.inductiva.ai/intro_to_api/tasks.html#task-lifecycle) is displayed next to the timeline, directing the user thorough documentation that includes a flowchart and detailed descriptions of What happened, Why, and Recommendations to overcome.

[390](https://github.com/inductiva/tasks/issues/390) To facilitate even more the task tracking in real time through the Console, the Tasks list and Task’s detail screens can be refreshed manually by clicking the button on the top right corner, or automatically, every 15 seconds, by turning on the “Auto refresh” toggle.

Terminate an ongoing Task from the Console <a name="killtask"></a>
[158](https://github.com/inductiva/console-web/issues/158)
The [Console](https://console.inductiva.ai/) becomes another interface for the user to expedite daily actions, namely terminate a running Task from its detail screen.
While the task is executing, a button becomes visible on the top right corner, and the user has the ability to terminate that task merely by clicking it. Naturally, the user is prompted to confirm the action, after which the following actions are automatically performed: the task is killed; the termination button is no longer visible; the screen is refreshed to show the task's updated status and other data.

Input and output files: Download to local computer <a name="download"></a>
[349](https://github.com/inductiva/tasks/issues/349), [#151](https://github.com/inductiva/console-web/issues/151), [#1606](https://github.com/inductiva/inductiva/pull/1606), [#392](https://github.com/inductiva/tasks/issues/392)
Several improvements were implemented to provide quick and easy downloading of a Task’s input and output files from the user’s remote storage bucket to their local computer.
Inputs are only available to be downloaded after the task has started. Outputs are only available when the task is finished.

From the Console, the task’s detail screen displays the “Input” and “Output” sections, both now include buttons to download the files - to the ‘Downloads’ default folder configured in the web browser being used to access the Console.

The user has always been able to [download the outputs](https://docs.inductiva.ai/en/latest/how_to/manage_and_retrieve_results.html#downloading-outputs) from the python client / CLI, and they are now able to also download the inputs.
By default, all files are downloaded to a directory named after the task ID, within a parent directory named “inductiva_output”.
To better organize this content,
the default location for the input files is inductiva_output\<task_id>\inputs,
and for the output it’s inductiva_output\<task_id>\outputs,
but the user can select any other location, by setting the parent directory name using the inductiva.set_output_dir function.

Here’s a few examples in CLI:

download all output files (default)
inductiva tasks download <task ID>
download all input files
inductiva tasks download <task ID> --input
download all output files
inductiva tasks download <task ID> --output
download all input and output files
inductiva tasks download <task ID> --input --output
download selected input and output files
inductiva tasks download <task ID> --input --output --filenames input.json stdout.txt


Example in Python:

import inductiva

task = inductiva.tasks.Task("<task ID>")

download all inputs
task.download_inputs()

select some input files to download
task.download_inputs(
input_dir='json-file-and-control-txt',
filenames=['input.json', 'control.txt'],
)


Get a URL to download input and output files <a name="url"></a>
[150](https://github.com/inductiva/console-web/issues/150), [#151](https://github.com/inductiva/console-web/issues/151)
This URL enables users to download the files via a web browser, or can be shared so that others can do the same.
Inputs’ URL is only available after the task has started. Outputs’ URL is only available when the task is finished.

The URLs can be retrieved:
- From the Console, in the “Input” and “Output” sections in the task’s detail screen
- In python scripts: task.get_input_url() and task.get_output_url()


Download the Task’s Logs from Console <a name="logs"></a>
[168](https://github.com/inductiva/console-web/pull/168)
Besides [streaming the logs of a running task in real time](https://docs.inductiva.ai/en/latest/cli/streaming-logs.html) and additional [sub-methods to access logs](https://github.com/inductiva/inductiva/releases/tag/v0.7.2), the user was already capable of downloading the log files (stdout.txt and stderr.txt) by using download_outputs().
Even if the task fails, the user is still be able to download the logs.

The Console also provides easy ways to access these files:
- Download logs - button in the “Output” section of the task’s detail screen;
- Display Logs - button “Display task logs” right next to “Input” and “Output” sections.

c4 and n4 machines available <a name="machines"></a>
[358](https://github.com/inductiva/tasks/issues/358), [#414](https://github.com/inductiva/tasks/issues/414)
These [powerful machine series](https://cloud.google.com/blog/products/compute/a-closer-look-at-compute-engine-c4-and-n4-machine-series) are ready for use in Inductiva’s cloud.
Check out other [available computational resources](https://tutorials.inductiva.ai/intro_to_api/computational-infrastructure.html#available-computational-resources) with function inductiva resources available.

New simulator Quantum Espresso <a name="simulator"></a>
[367](https://github.com/inductiva/tasks/issues/367)
[Quantum ESPRESSO](https://inductiva.ai/simulators/quantum-espresso) is an open-source software suite widely used for electronic structure calculations and materials modeling at the nanoscale.
The [documentation](https://tutorials.inductiva.ai/simulators/QuantumEspresso.html#) details: how to set it up for use with Inductiva API; A code example to get started; The list of allowed commands.

The complete list of integrated simulators is available on the [website](https://inductiva.ai/simulators) or using the Python method inductiva simulators list.

More rich content <a name="content"></a>
[361](https://github.com/inductiva/tasks/issues/361), [#362](https://github.com/inductiva/tasks/issues/362)
Inductiva provides a myriad of rich content and resources to the advantage of the community that follows our work. These are posted publicly on the [tutorials](https://tutorials.inductiva.ai/), [blog](https://inductiva.ai/blog), [website](https://inductiva.ai/simulators) and also other well-known repositories, such as [Hugging face](https://huggingface.co/inductiva) or [Docker Hub](https://hub.docker.com/r/inductiva/kutu).
Take a look at some of the latest published works:
- [OpenFoam advanced Example](https://tutorials.inductiva.ai/simulators/OpenFOAM.html#advanced-example-running-mb9-micro-benchmark-by-exafoam): Running MB9 Micro-benchmark by ExaFOAM
- [Wind Tunnel dataset](https://huggingface.co/datasets/inductiva/windtunnel-20k)

List of changes in the v0.9 release

**Removal of Freemium tier**
[328](https://github.com/inductiva/tasks/issues/328), [#329](https://github.com/inductiva/tasks/issues/329)
Inductiva recently introduced a set of incremental [tiers](https://docs.inductiva.ai/en/latest/api_reference/tiers_and_quotas.html) that started with the entry-level Freemium tier that provided free access to a very limited set of capabilities and resources - Members of this tier could only submit tasks to the shared queue.
By removing this tier, new users are instead granted access to the Standard tier and 5$US to spend at their will, with no time restrictions.
This way, a user can still have initial free access to try out Inductiva’s API, but taking advantage of much more powerful [capabilities](https://docs.inductiva.ai/en/latest/api_reference/tiers_and_quotas.html#capabilities), thus providing a more accurate perspective of the API’s potential.


**The registered Freemium users are bumped up to the Standard tier and offered 5$US in credits with no expiration date**, and can now start experiencing the set of more powerful features, such as launching and managing [dedicated computational resources](https://tutorials.inductiva.ai/intro_to_api/shared_dedicated_resources.html#dedicated-resources).
<br/>

**Deletion of the Shared Queue**
[330](https://github.com/inductiva/tasks/issues/330), [#331](https://github.com/inductiva/tasks/issues/331), [#332](https://github.com/inductiva/tasks/issues/332), [#333](https://github.com/inductiva/tasks/issues/333), [#334](https://github.com/inductiva/tasks/issues/334), [#335](https://github.com/inductiva/tasks/issues/335), [#336](https://github.com/inductiva/tasks/issues/336)
This subset of allocated computational resources was created to be used in a shared fashion among all Inductiva’s users, and was the only resource available to Freemium tier users.
With the removal of the Freemium tier, and all users becoming able to launch and manage dedicated machines, this very limited resource loses its main purpose, so it’s retired.

Therefore, **from now on, tasks can only be run in a [dedicated machine group](https://tutorials.inductiva.ai/intro_to_api/shared_dedicated_resources.html#dedicated-resources)** that has been created and started by the user.
<br/>

**VM instances with two disks**
[1132](https://github.com/inductiva/inductiva-web-api/pull/1132)
Instead of a single disk, the VMs are set up with two disks:
- boot disk - based on the disk image, fixed size;
- data disk - the "data_disk_gb" parameter specified by the user in the python client sets the size of the disk for user data (in GB).
<br/>

**Automatic increase of the data disk size during task execution**
[276](https://github.com/inductiva/tasks/issues/276), [#1143](https://github.com/inductiva/inductiva-web-api/pull/1143), [#352](https://github.com/inductiva/tasks/issues/352), [#1587](https://github.com/inductiva/inductiva/pull/1587/files), [#1587](https://github.com/inductiva/inductiva/pull/1587)
Mechanism to prevent the tasks from failing due to not enough local storage space.
The "out of space" errors can occur when a task generates a quantity of output files that exceeds the size of the local storage, so the user is now able to allow the hard disk of the cloud VM to increase dynamically during the task execution, when the free space falls below a certain threshold.


However, this automatic resize is limited, for good reason: increasing the disk size during a task execution increases the cost of local storage associated with the VM.
As a safeguard, the user must set an upper limit to the disk size, i.e. the maximum size in GB that the hard disk of the cloud VM can reach, to prevent uncontrolled costs:
auto_resize_disk_max_gb: The maximum size that the hard disk of the cloud VM can reach (in GB).

machine_group = inductiva.resources.MachineGroup(
machine_type="c2-standard-16",
data_disk_gb=20,
auto_resize_disk_max_gb=100,
)

Once that limit is reached, the disk is no longer automatically resized, and if the task continues to output files, it will fail.
<br/>

**Threads per core**
[297](https://github.com/inductiva/tasks/issues/297)
Simultaneous multithreading (SMT), which is known on Intel processors as Hyper-Threading Technology (HTT), lets a CPU core run as two hardware multithreads. On Compute Engine, each virtual CPU (vCPU) is implemented as a single hardware multithread, and two vCPUs share each physical CPU core by default - check more information [here](https://cloud.google.com/compute/docs/instances/set-threads-per-core).


However, reducing the number of threads that run on each physical CPU core can help improve the performance of workloads that are highly-parallel or that perform floating point calculations.
Therefore, when starting a machine, the user is able to define the number of threads per core, such as:

machine_group = inductiva.resources.MachineGroup(
machine_type="n2d-highcpu-64",
spot=True,
threads_per_core=1,
data_disk_gb=20)



Threads_per_core can only be 1 or 2. The default value is 2.
SMT is disabled if threads_per_core is set to 1.
<br/>

**Link to download output**
[354](https://github.com/inductiva/tasks/issues/354)
New method provides an URL for the user to download the zip file with all output files:

task.get_output_url()

This way, users can download the output file even if the internet connection is flaky, taking advantage of the efficient caching mechanism of the web browsers.
Also, this URL can be easily shared with colleagues for them to download the file themselves.
<br/>

**Input files included in the simulation’s output**
[164](https://github.com/inductiva/tasks/issues/164), [#1219](https://github.com/inductiva/inductiva-web-api/pull/1219), [#10](https://github.com/inductiva/task-runner/pull/10)
All input files are included in the output.zip file, which is especially relevant for the input files to be included in the task’s output in the cases where the simulator updates the input files as part of its execution.
This causes the bucket to have duplicated files and doubled duplicated storage, but allows the user to download both the original and the updated input files.

<br/>

**Wait to start a machine group when quotas are reached**
[351](https://github.com/inductiva/tasks/issues/351)
When starting a machine group, the system could throw an exception if the [quotas](https://docs.inductiva.ai/en/latest/api_reference/tiers_and_quotas.html#quotas) were exceeded. The new flag available in resource.start() defines the machine group's behavior when quota limits are reached:
wait_on_pending_quota=False - the process will crash immediately when quotas are reached.
wait_on_pending_quota=True - the process will wait until the required quotas become available.
This way, the user can guarantee that the machine group will be started at least when there’s enough quotas, thus avoiding repeating the instruction if quotas were exceeded.
The flag’s default value is False.

While the process is waiting for quotas, the user is alerted with a message printed to CLI, so that they can take action and maye actively manage their resources to free up quotas. Yet, this flag makes resource.start() a blocking call until the quotas become available, so the user can only terminate another machine group with a new process.

List of changes in the v0.8.6 release

**[300](https://github.com/inductiva/tasks/issues/300) New simulator FVCOM**
FVCOM (Finite Volume Community Ocean Model) is a three-dimensional hydrodynamic model designed for simulating coastal and ocean dynamics.
The [documentation](https://docs.staging.inductiva.ai/simulators/FVCOM.html#) details which of this simulator’s available flags were used in its compilation and provides an example of how to run it in Inductiva’s API.
Compiling this simulator can be challenging, so Inductiva posted a summarized guide for [configuring and compiling FVCOM](https://inductiva.ai/blog/article/compiling-fvcom-a-handy-toolbox-for-solving-common-issues).
The complete list of integrated simulators is available in the [website](https://inductiva.ai/simulators) or using the Python method inductiva simulators list.

<br/>

**New features in the user’s [Console](https://console.inductiva.ai/):**

- **[94](https://github.com/inductiva/console-web/issues/94) Access a Task’s detail**
By clicking a line in the Tasks list, the user accesses a screen with that task’s detail. This new screen provides a clear and thorough overview of the selected task (namely time breakdown, input and output files).
Similarly to the insight provided by the task.print_summary() python method introduced in [v0.7.2](https://github.com/inductiva/inductiva/releases/tag/v0.7.2), this feature brings visibility to the user’s work using the API.

- **[341](https://github.com/inductiva/tasks/issues/341) Minor improvements in the Tasks list**
The Tasks list was also improved by adding the Projects column, and making the list sortable by any of the columns, facilitating the organizations of the data and searching through it.

- **Account data tab**
The user account data was moved into its own tab, providing more visibility on the user’s current credits, [tier and quotas](https://docs.inductiva.ai/en/latest/api_reference/tiers_and_quotas.html).

<br/>

**[135](https://github.com/inductiva/tasks/issues/135) Allow to run DynamicCode capability in OpenFOAM**
[OpenFOAM](https://docs.inductiva.ai/en/latest/simulators/OpenFOAM.html) allows user-provided code to be compiled and loaded dynamically during the course of the simulation to dynamically manipulate how the simulation evolves. An example is to dynamically change the mesh(es) in runtime. Due to how the code was running on our infrastructure, users were unable to use this feature. With the changes introduced in this release, dynamic code execution is no possible with no changes on how the simulation is deployed.

List of changes in the v0.8.5 release

**[226](https://github.com/inductiva/tasks/issues/226) New stats in Console Dashboard**
[Inductiva’s Console](https://console.inductiva.ai/) is growing to become a powerful asset for users to take advantage of a myriad of data encompassed in a user-friendly UI.
In this dashboard, the user has access to graphs and data that provides insight into their usage of Inductiva’s API - tasks status, machine types, activity over time, etc.

**[962](https://github.com/inductiva/inductiva-web-api/issues/962) , [#1144](https://github.com/inductiva/inductiva-web-api/issues/1144) - Simultaneous compression and upload of output**
Once a simulation is finished, its output is now compressed and uploaded to the bucket simultaneously. Before this change, these two actions would be done consecutively which would use more disk space (the zip file would temporarily be stored together with the output files).

**[160](https://github.com/inductiva/tasks/issues/160) Improved issue in MPI jobs with a large number of vCPUs in environments using Apptainer**
In environments utilizing [Apptainer](https://apptainer.org/), a container platform, MPI (Message Passing Interface) jobs could fail on systems with a large number of vCPUs, particularly when high -np values were specified.
The issues stemmed from limitations in Apptainer's configuration (specifically the maximum number of loop devices) and system-defined limits (such as the number of open file descriptors). After changing the Apptainer configuration, the system can now accommodate the demands of large-scale MPI jobs by providing adequate loop devices and file descriptors.

**Improved messaging in CLI, to enhance the user experience:**
[185](https://github.com/inductiva/tasks/issues/185) inductiva storage list - Message after the list of folders informing the number of results displayed, how they are ordered, and instructions on how to change the number of results listed.
[177](https://github.com/inductiva/tasks/issues/177) When the task fails, the last few lines of stderr (outlined in red) and stdout (outlined in blue) are printed to the CLI.
The user continues to be able to inspect the full stderr file: inductiva_output/[copy_paste_my_task_id]/stderr.txt, but this automatic print provides an immediate insight into the task.
[279](https://github.com/inductiva/tasks/issues/279) After a task is finished, the message printed to CLI informing that stdout and stderr files are being downloaded now include the respective download paths.

List of changes in the v0.8.3 release 🚀🚀

**Run custom Docker images in Inductiva’s platform [265](https://github.com/inductiva/tasks/issues/265), [#273](https://github.com/inductiva/tasks/issues/273)**
Adding to the Docker images available in [Kutu](https://inductiva.ai/kutu), the user is able to run custom images on Inductiva’s platform, by using the `CustomImage` simulator. This feature is only available for Enterprise users.
Besides the parameters through which the user selects how the task will run, such as use spot instances, this simulator accepts a list of commands to run inside the container.
Here’s an example of how to use it:

import inductiva


input_dir = inductiva.utils.download_from_url(
"https://storage.googleapis.com/inductiva-api-demo-files/fds-input-example.zip", unzip=True)


custom_simulator = inductiva.simulators.CustomImage(container_image="docker://inductiva/kutu:fds_v6.8")


task = custom_simulator.run(input_dir=input_dir, commands=["fds mccaffrey.fds"])
task.wait()

In this example, `commands` is a list of strings to be run inside the container.
For more advanced usage, such as running a command with a specific MPI version, we introduce the `Command` and `MPIConfig` classes, which enable the user to run commands on [MPI clusters](https://docs.inductiva.ai/en/latest/how_to/set-up-mpi-cluster.html) supporting multiple MPI versions.
A list of `commands` can include `Command` instances, each potentially having an `MPIConfig`.
See an example of how to run more than one instance of a custom image using MPI version 4.1.6, in our [community documentation](https://tutorials.inductiva.ai/intro_to_api/custom_docker_images.html).


<ins>Requirement:</ins>
The MPI version in the custom container must match the one specified in `MPIConfig`. Currently, we support OpenMPI versions 1.10.7 and 4.1.6. If you need a different version, please contact us.
<br/>

**New version of OpenFOAM simulator [237](https://github.com/inductiva/tasks/issues/237)**
Shortly after [ESI Group](https://www.openfoam.com/) released [v2406 of OpenFOAM](https://www.openfoam.com/news/main-news/openfoam-v2406), we make it available to our users in [Kutu](https://inductiva.ai/kutu), Inductiva’s free repository of Docker images for simulation software.
Check out our documentation [here](https://docs.inductiva.ai/en/latest/simulators/OpenFOAM.html) and the Docker image [here](https://hub.docker.com/r/inductiva/kutu/tags?page=&page_size=&ordering=&name=openfoam-esi).
<br/>

**Time and storage saving**
The simulation output is compressed while it is uploaded to the bucket, instead of being compressed locally and uploaded afterwards.
This brings several advantages:
Saves time - compressing and uploading are done simultaneously;
Saves storage space - the compressed file would take up more space in the disk;
Reduces the risk of the task not being finished due to lack of disk space.
<br/>

**Manage remote storage + Metrics Dashboard in Console [258](https://github.com/inductiva/tasks/issues/258) [#226](https://github.com/inductiva/tasks/issues/226)**
The [Inductiva API Console](https://console.genesis.inductiva.ai/) is growing to provide functionalities similar to those available in the CLI with a user-friendly graphic interface.
Besides the Tasks information made available in a previous release, a few new features are:

**Dashboard**
This area of the Console will grow massively to empower the user in realizing how they are using Inductiva’s platform. In this version, we start by making available the following usage statistics, which are calculated within the timeframe selected on top of the screen:
- Number of Tasks: Total number of tasks run within the selected timeframe.
- Average Task Duration: Average duration of the tasks run within the selected timeframe (including files upload and download times).
- Task Success Rate: Percentage of successful tasks out of the tasks that reached a final state (i.e. excluding tasks terminated by the user, namely ‘killed’, ‘zombie’, ‘executer_terminated_by_user’, ‘executer_failed’ - learn about the task lifecycle and all states it may reach in our [community documentation](https://tutorials.inductiva.ai/intro_to_api/tasks.html#)).
- Preempted Tasks: Number of tasks interrupted due to machine preemption within the selected timeframe.

**Storage**
This screen shows the total storage space used and a list of the directories within the user’s remote storage, characterizing each with: directory name, size, and creation date / time.
The methods to [manage remote storage](https://docs.inductiva.ai/en/latest/how_to/manage-remote-storage.html#) in the CLI or programmatically continue to be available.

**Machine Groups**
Here, the user will find a list of their active machine groups and respective attributes: Name, Machine Type, Elastic (Yes/No), Type, Number of machines, Disk Size, Spot (Yes/No), Start Date and Time, Idle Time, Cost per hour
This list only includes the dedicated machine groups that were allocated by the user, not
The methods to [manage computational resources](https://docs.inductiva.ai/en/latest/how_to/manage_computational_resources.html#) in the CLI or programmatically continue to be available.

List of changes in the v0.8.2 release 🚀🚀

Bug fix in the Back-end, with no visibility or impact to end users.