Simgrid

OVERALL CHANGES:
* Port to windows.
We still experience issues on this platform, but we believe that at
least MSG is usable.

GRAS API BREAKAGE (for simplification purpose, sorry):
* the gras_msgtype_by_name is not used anymore. Instead of
gras_msg_send(toserver, gras_msgtype_by_name("request"), &request);
you can write (and must)
gras_msg_send(toserver, "request", &request);
- If you still want to pass a gras_msgtype_t to the function (to cache
the type and avoid the lookup time), use the gras_msg_send_() variant.
- Impacted functions:
gras_cb_register, gras_cb_unregister, gras_msg_send, gras_msg_wait,
gras_msg_rpccall, gras_msg_rpc_async_call, gras_msg_wait_ext
* The callbacks are now expected to return 0 when everything went well
(just like the main() function)

GRAS new features and improvements:
* New module mechanism where user code can use per process globals [Mt]
This is similar to gras_userdata_*() functions, but for libraries. It
factorize some code developped over and over in the examples and AMOK.
It has still to be documented and used (only amok/peermanagement is
converted for now).
* Fix a vicious bug in the TCP buffering mechanism which leaded to message
loss when they were small enough to fit into the buffer and sent quickly
enough so that they can all get received in one shoot.
* gras_datadesc_by_name and gras_msgtype_by_name: now raise an exception
if not found. Use the *_or_null() variant for the old semantic.
* In gras_msg_handle, do not discard messages without callback.
They are probably messages to be explicitly awaited later (ie, proofs of
mis-synchronization in userland since they are sent before being awaited)
No big deal usually.
* gras_socket_meas_send/recv: semantic changed!
The numerical arguments used to be (1) the total amount of data to send
and (2) msg_size. This was changed to (1) msg_size and (2) amount of
messages. This was need for the fool willing to send more than MAXINT
bytes on quite fat pipes.

AMOK:
* Do really rename the hostmanagement module to peermanagement. [Mt]
Ie, rename functions from amok_hm_* to amok_pm_*. This breaks the API,
but this is rather new and this was documented in the module
documentation (poor excuses, I admit)
* Bandwidth measurement semantic changed! This follows the changes to
gras_socket_meas_send/recv explained above.

SIMDAG:
* A sequential mode has been added to the workstations. When a workstation
is in sequential mode, it can execute only one task, and the other tasks
are waiting in a FIFO. [Christophe Thiery]

SURF:
* The KCCFLN05 workstation model now handles parallel tasks. It is the
model for SIMDAG. [Christophe Thiery]
* Bug fix in the maxmin solver: Some values were close to 0 instead of
equal to 0, which caused some bad behaviors in
saturated_constraint_set_update. I now use a threshold mechanism like in
surf. [AL]

XBT:
* When running manually src/testall, you select specific units [Mt]
testall is the result of our cunit mechanism, and should replace all
the scripty thingy around since bash don't run easily on billware.

* A mallocator system has been added. [Christophe Thiery]
Mallocators allow you to recycle your unused objects instead of freeing them
and allocating new ones.

Documentation update:
* FAQ reworking + New FAQs:
- "Valgrind spits tons of errors!" [Mt]
- "How to repport bugs" [Mt]
- "Cross-compiling a Windows DLL of SimGrid from Linux" [Mt]
- "What is the difference between MSG, SimDag, and GRAS?" [Mt]
- Communication time measurement within MSG [AL]
- I experience weird communication times when I change the latency [AL]
* GRAS tutorial [Mt]
It contains:
- an introduction to the framework and to the used communication model
- an initiatic tour introducing the most proheminent features:
o Part 1: Bases
. Lesson 0: Installing GRAS
. Lesson 1: Setting up your own project
o Part 2: Message passing
. Lesson 2: Exchanging simple messages
. Lesson 3: Passing arguments to the processes (in SG)
. Lesson 4: Attaching callbacks to messages
. Lesson 5: Using globals in processes
. Lesson 6: Logging information properly
. Lesson 7: Using internal timers
. Lesson 8: Handling errors through exceptions
. Lesson 9: Exchanging simple data
. Lesson 10: Remote Procedure Calling (RPC)
. Lesson 11: Explicitly waiting for messages
. Recapping of message passing features in GRAS
- A HOWTO section containing:
o HOWTO design a GRAS application
More are due, of course. They will come latter. In the meanwhile, you can
check the examples which are still here.

-- Da SimGrid team <simgrid-devellists.gforge.inria.fr> Fri Mar 16 21:11:46 CET 2007

General:
* Port to gcc 4.x
There was a stack corruption somewhere, visible only when optimizing
with these versions. [Vince]

SIMDAG:
* This is a NEW module! SimDAG (SD for short) is a revival of the old SG
module that enabled to play with Directed Acyclic Graphs. It is built
directly on top of SURF and provides an API rather close to the old
SG. Some old codes using SG are currently under rewrite to check that
all needful functions are provided. [Christophe Thiery]

SURF:
* Complete rewrite of the KCCFLN05 workstation model. It is now an
extension of the classical CLM03 model that gracefully handles
failures. This is now the default model for MSG and GRAS. It doesn't
handle parallel tasks yet though. [AL]
* Bug fix: Weights were not correctly set in the network part.
WARNING: This may have resulted in incorrect results with simulations
where there are more than one flow on a given link. [AL]

SURF, MSG, GRAS:
* After a (long ?) discussion on simgrid-devel, we have decided that the
convention we had on units was stupid. That is why it has been decided
to move from (MBits, MFlops, seconds) to (Bits, Flops, seconds).
WARNING: This means that all previous platform files will not work as
such with this version! A warning is issued to ask users to update
their files. [AL]
A conversion script can be found in the contrib module of the CVS, under
the name contrib/platform_generation/surfxml_update.pl [MQ]

MSG,GRAS:
* Bug fix: Processes were started in reverse order, wrt deployment file.
WARNING: if your code relies on this bug, please fix it. [AL]
* Bug fix: Add a test in MSG_task_execute to stop whenever a task is
being executed on two different locations. [AL]
* Bug fix: Failures are now better supported thanks to Derrick's tests
(there was many failure situations I hadn't thought of and that weren't
correctly handled). [AL]
* New function: MSG_host_is_avail indicates you whether a given m_host_t
is up or down. [AL]

GRAS:
* New! a real RPC mechanism, as it ought to be since too long. [MQ]
Exception occurring on server-side are propagated back to client (!).

API CHANGE: the callback changed their prototype. Change:
int my_handler(gras_socket_t expeditor, void *payload_data) {
to:
int my_handler(gras_msg_cb_ctx_t ctx , void *payload_data) {
gras_socket_t expeditor=gras_msg_cb_ctx_from(ctx);
and you're set.
* New! function: gras_msg_handleall to deal with all messages arriving
within a given period.
* New! function: gras_socket_server_range to get a server socket in a
range of port numbers (ease to avoid port number conflicts) [MQ]
* New! gras processes display their backtrace when they get a SIGUSR1
or when Ctrl-C is pressed. Use Ctrl-C Ctrl-C to exit.
Sweet to debug RL processes [MQ]

AMOK:
* Bandwidth module:
- Do not force experiment sizes to be expressed in kb, or it becomes
impossible to measure the latency this way (needs one byte-long tests)
WARNING: this changes the amok_bw_* function semantic. [MQ]
- Implements the link saturation stuff. [MQ]
* Peer management module:
New! module factorizing code that we wrote over and over [MQ].

XBT:
* New module: cunit (my jUnit implementation in ansi C) [MQ]
- Test units are placed directly into the library code, they get extracted
automatically and placed into the src/testall binary.
- Convert most of the XBT tests to this system.
* New functions: xbt_dynar_getfirst_as() and xbt_dynar_getlast_as() [MQ]
* XML parsing: rewrote parts of flexml to enable multiple xml parsers to
live in the same C code. This required to change a little bit the API
of surfxml parsing but shouldn't be an issue for end-users. [AL]
* New module: sparse graph structure with basic algorithms (this is work
in progress and the API is not considered to be frozen yet). [AL]
* Display more information on backtraces: source line & function names are
now displayed just like valgrind does (rely on addr2line tool) [MQ]
* New function: xbt_backtrace_display(). Sweet while debugging [MQ]
* Reworked a little bit some include statements to load only required
headers. Some user code that relied on SimGrid to include stdlib or
stdio may need to include it by themselves. [AL]
* Fixed xbt/log.h. A missing SG_BEGIN_DECL prevented compilation with
g++. [AL]
* Renamed xbt_host_t into xbt_peer_t since it betterly describes what I
meant. This breaks the API of AMOK and of xbt/config. Sorry about this,
but I guess that almost nobody used those parts. [MQ]

-- Da SimGrid team <simgrid-devellists.gforge.inria.fr> Fri, 14 Jul 2006 01:32:27 +0200

XBT:
* Unfortunately, I had missed 5 misnamed functions:
xbt_fifo_item_t xbt_fifo_newitem(void);
void xbt_fifo_freeitem(xbt_fifo_item_t);
xbt_fifo_item_t xbt_fifo_getFirstItem(xbt_fifo_t l);
xbt_fifo_item_t xbt_fifo_getNextItem(xbt_fifo_item_t i);
xbt_fifo_item_t xbt_fifo_getPrevItem(xbt_fifo_item_t i);
They're now deprecated. Please use their new versions:
xbt_fifo_item_t xbt_fifo_new_item(void);
void xbt_fifo_free_item(xbt_fifo_item_t);
xbt_fifo_item_t xbt_fifo_get_first_item(xbt_fifo_t l);
xbt_fifo_item_t xbt_fifo_get_next_item(xbt_fifo_item_t i);
xbt_fifo_item_t xbt_fifo_get_prev_item(xbt_fifo_item_t i);
[AL]
* Bugfix: really disconnect fifo items which are remove_item()ed [AL]
* Documentation: xbt_log module unmercifully reworked [MQ]
* Bugfix: there was a problem with the ending of contexts with
the pthread backend. It caused some weird deadlock or behavior
depending on the pthread implementation. [AL]
* Bugfix: get the exceptions raised in the simulator repport where
and why they come from when they are not caught in time [AL, MQ]

SURF:
* Bugfix: Do repport the error when two non-connected hosts try to
exchange data (Thanks to Flavien for stumbling into this one) [AL]

SURF:
* Add additionnal checkings on communications. Assert that two
communicating hosts are connected by a set of links... [AL]

MSG:
* Add additionnal checkings on channel values in communication [AL]
* New: MSG_task_get_source to see on which host a task was generated [HC]
* New: int MSG_task_probe_from_host(int channel, m_host_t host): returns
the number of tasks waiting to be received on channel and sent
by host. [AL]
* New: MSG_error_t MSG_task_get_from_host(m_task_t * task, int channel, m_host_t host);
waits for the first task coming from a given host.. [AL]

GRAS new functionnalities: [MQ]
* Enhance the parsing macro to allow the size of multidimentional objects
to be given thru annotations.
* New example (and documentation): Matrix Multiplication a la RPC
(as when I was young!) and fix a bunch of bugs found on the way.

GRAS performance improvements: [MQ]
[DataDesc]
* Reduce the amount of cbps creation/destruction by making it static to
datadesc_send/recv() and using a (newly created) cbps_reset (based on
dynar_reset ())
[Virtu]
* Change libdata to a set so that we can search for stuff by ID (and thus
reduce the insane amount of dict lookups)

[Transport]
* Actually implement gras_datadesc_copy() so that we don't have to mimick
RL communication on top of SG since it's so uneffective.
It may also be used for inter-thread communication in RL, one day.
* Use gras_datadesc_copy() to exchange messages on top of SG
Allows to:
- improve message exchange performance on top of SG
- deprecate transport_plugin_sg.c:gras_trp_sg_chunk_send() & recv()
* Don't exchange on the network the size of the used part of buffer,
instead, specify the possible buffer size to read().
Advantages:
- reduces the amount of read/write calls (one pair per exchange)
- reduces the amount of exchanged data (the size)
- allows to retrieve all arrived data on receiver side, if we don't need
it right now (subsequent read will peek the buffer)
- allows the receiver to proceed with the beginning of the stream before
everything is arrived
- make it possible to build an iov transport (using readv/writev)
Extra difficulty:
- take care of the data with non-stable storage (like stacked data),
and bufferize them.
* If possible, TCP send uses vector I/O (when writev() is here)
- Don't use it for receive since we send data sizes and data on the
same stream, so we wouldn't be able to chain large amount of chunks
before having to flush the stuff to read the size.
* Rework the transport plugin mechanism to simplify it and reduce the
amount of pointer dereferencement when searching for the right function
to use.

* I guess that now, we do almost as few system calls as possible while
doing as few data copy as possible.

To improve it further, we could try to send all the sizes first and then
all the data (to use iov on receiving size), but it's only a partial
solution: when you have 2 dimensional data, the sizes of the second
dimension is data of the first dimension, so you need 3 streams.

I'm not sure the potential performance gains justify the coding burden.

-- Da SimGrid team <simgrid-devellists.gforge.inria.fr> Fri, 21 Oct 2005 14:42:20 +0200

SURF:
* New! Give the possibility to hijack the surf parser and thus bypass
MSG_create_environment and MSG_launch_application. Have a look at
examples/msg/msg_test_surfxml_bypassed.c to see how it can be done.

-- Arnaud Legrand <simgrid-devellists.gforge.inria.fr> Sat, 20 Aug 2005 23:25:25 -0700

AKA SimGrid 3 rc 2.

XBT:
* New! Exception handling with setjmp or such (code from OSSP ex) [MQ]
This deprecates the xbt_error_t mechanisms.
It modifies (simplifies) all XBT and GRAS API.
MSG API keeps unchanged (exceptions raised by XBT are caught from
within MSG and masked with existing error handling facilities)

SURF:
* New! Add a FATPIPE model. [AL]
* New! Add a parallel task model. [AL]
* New! Add automatically a loopback interface (in the default
network model) if none was precised.

MSG
* Bugfix: MSG_process_resume now works with the current running process.
[AL]
* New! Add MSG_parallel_task_create and MSG_parallel_task_execute. [AL]
* Modification of MSG_task_get_compute_duration. Once a task has been
processed, the value returned by this function is now equal to 0. [AL]
* New! Add double MSG_task_get_remaining_computation(m_task_t task) and
MSG_error_t MSG_task_cancel(m_task_t task). Add a state
(MSG_TASK_CANCELLED) to MSG_error_t corresponding to the cancelation
of a m_task. For now, MSG_task_cancel only works with computation
tasks. [AL]
* New! Add double MSG_get_host_speed(m_host_t h) that returns the speed
of the processor (in Mflop/s) regardless of the current load on the
machine. [AL]
* API Change: use proper naming convention for MSG_getClock and
MSG_process_isSuspended: MSG_get_clock and MSG_process_is_suspended.
[AL]
* New! Add void MSG_task_set_priority(m_task_t task, double priority).
This function changes the priority of a computation task. This priority
doesn't affect the transfer rate. A priority of 2 will make a task
receive two times more cpu power than the other ones. This function
has been added to suit the needs of Nguyen The Loc and hasn't been that
much tested yet. So if it fails, please report it and send me your code.
[AL]
* API Change: removed all functions and types that were marked "deprecated"
since many months. Renamed MSG_global_init_args to MSG_global_init.

-- Da SimGrid team <simgrid-devellists.gforge.inria.fr> Mon, 8 Aug 2005 17:58:47 -0700

XBT
* Steal some nice code to GNU pth to fix context detection and usage [AL]
* Cleanup in the xbt_config API; add configuration callbacks. [MQ]
* Cleanup in the initialization API: the unused "defaultlog" is dead. [MQ]

SURF
* Bugfix: Allow absolute paths for platform description files [MQ]
* Bugfix: do free the variables after use. Leads to drastic performance
improvement [AL]
* Implement max_duration (ie, timeouts) on resources [AL]

MSG
* Implement MSG_config to configure MSG at runtime. xbt_cfg test on a real
case ;) [MQ]
* Implement MSG_channel_select_from() to help GRAS now that SURF provide
the needed support (timeouts) [AL]

GRAS (new features)
* Implement measurement sockets. You can now get the bandwidth between two
hosts thanks to AMOK (see below). [MQ]
* gras_datadesc_dynar() builds a dynar type descriptor, allowing to send
dynar over the network (yeah) [MQ]
* Real (even if simplistic) implementation of gras_os_myname() on RL [MQ]
* simple/static token-ring example. [Alexandre Colucci and MQ]
* Use MSG_channel_select_from() and remove the *slow* hack we had to put
in place before [MQ]

GRAS (bug fixes)
* Differentiate the types "char[22]" and "unsigned char[22]" in automatic
type parsing. "short" and "long" modifiers were also ignored; other
modifier (such as reference level) are still ignored. [MQ]
* Embeed the buffer size within the buffer itself on SG. [MQ]
That way, send() are atomic and cannot get intermixed anymore (at least
the ones which are less than 100k; bigger messages still have the issue)
* Array size pushed by the field, not by the field type (or each
and every long int will push stuff to the cbps) [MQ]
* use select() to sleep since it allows to portably sleep less than one
second. [MQ]

GRAS (minor cleanups)
* <project>.Makefile.local (generated from gras_stub_generator) |MQ]:
- Do clean .o files
- Compile with -g
* Type Callbacks now receive the gras_datadesc_type_t they work on as argument.
* type category 'ignored' killed as it was never used and were difficult
to transmit.
* whether a type can cycle or not is now a flag, leaving room for more
flags (as "ignored", if we feel the need one day ;)
* Rename raw sockets to measurement sockets since "raw" has another
meaning in networking community.

AMOK
* Advanced Metacomputing Overlay Kit introduction. It is based over GRAS
and offers features not belonging to GRAS but that most applications
need. One day, it may be a set of plugins loadable at runtime.
* New module: bandwidth
bandwidth measurement between arbitrary nodes running this module. [MQ]

-- Da SimGrid team <simgrid-devellists.gforge.inria.fr> Thu, 30 Jun 2005 16:29:20 -0700