FLCOMMS

What is it?

• FLCOMMS is a lightweight, shared memory-based interprocess communications facility
• Purpose: to transfer data between a FLIGHTLAB model and external applications
• Shared memory layer uses SYSV IPC
• Network layer uses broadcast UDP

FLCOMMS data model

• Designed for real-time applications
• Collection of named data blocks
• Data blocks defined by name, size, type, and ``magic number''
• All processes accessing a block must agree on data block structure
• Data blocks typically contain several fields; contents are (mostly) transparent to FLCOMMS library
• Data blocks created when first process attaches
• Data blocks (normally) removed when last process detaches
• Single-writer, multiple-reader discipline enforced
• ``Best effort delivery''

FLCOMMS data model - figure

Scope interface - attaching variable lists

• ATTACH(@outputs, "w")
  Attaches variable list in write mode: data copied from Scope model variables to FLCOMMS data block
• ATTACH(@inputs, "r")
  Attaches variable list in read mode: data copied from FLCOMMS data block to Scope model variables
• ATTACH([])
  Disconnects from FLCOMMS block
• Read-mode variable lists automatically read before invoking an operation
• Write-mode variable lists automatically written after invoking an operation
• APCTL("sync");
  Manually synchronizes attached variable lists with FLCOMMS

Scope API - Example

// Load a model:
exec("$FL_DIR/flme/models/articulated/arti-rgd-3iv-qs.def",1);

// Set up variable lists for the data we want to share:
pushg(world_model_airframe_cpg_xaout);
    varlist @STATES "Aircraft rigid body states" = ..
            posxi, posyi, poszi, ..
            phi, theta, psi;
popg;
pushg(world_model_control_cpg_in);
    varlist @PILOTIN "Pilot inputs" = ..
       xa, xb, xc, xp;
popg

// Publish the shared variables:
attach(@STATES,  "w");
attach(@PILOTIN, "r");

// Synchronize data:
apctl("sync");

flcomms Command-line utility

% flcomms -l
Data blocks (serial=13)
   #: hflg flags name   size   addr #rw owner   serial    magic
---------------------------------------------------------------
 124: 0104 r   D PILOTIN  32 0x1100   1     0 s=000000 m=0xB1F7
 125: 0204  w  D STATES   48 0x1000   1 29392 s=000006 m=0x2D8C

• Columns 1, 2, 7: internal data, used for debugging
• Mode flags: r = attached readable, w = attached writable,
• Type indicator: I(nteger), L(ong), F(loat), D(ouble) or M(ixed)
• Data block name and size
• Number of active handles
• Process ID of owner (0 if write lock available)
• s=serial number -- incremented each time block is written
• m=magic number -- used for extra consistency check

flcomms Command-line utility (cont)

• flcomms -c: Clean up after crashed writers, release write locks
• flcomms -t: Show status of system IPC resources (shared memory segment and semaphore array)
• flcomms -r: Remove system IPC resources
• flcomms [-h]: Show command usage information
• ipcs: Standard Unix utility to display IPC resource information
• ipcrm: Standard Unix utility to remove IPC resources ipcs and ipcrm usage varies depending on system

FLCOMMS: C API

• #include "flcomms.h" in all C code
• Link with -lflcomms ($FL_DIR/lib$FL_MACHTYPE/libflcomms.a)
• Call flc_attach(NULL,0); at program startup
• Data blocks accessed through handles
• Call flc_open(...) to acquire data block handles
• Use flc_read(...) and flc_write(...) to transfer data
• Call flc_close(...) to release a handle (optional)
• Call flc_detach(); before exiting program (not optional)

Connecting to data blocks

FLC_HANDLE h = flc_open("name", size, magic, mode_flag | type_flag);

• name: data block name
• size: size in bytes of data block
• magic: magic number
• flags: specify mode and type:
  • FLC_MODE_READ : open data block for reading
  • FLC_MODE_WRITE : open for writing
  • FLC_TYPE_DOUBLE, FLC_TYPE_INT, etc: specify data type
  • FLC_TYPE_MIXED: anything else
  • Flags combined with bitwise-OR operator (|)
• Scope always uses FLC_TYPE_DOUBLE
• name is the variable list name without leading @ sign, all uppercase

Reading and writing

• flc_read(handle, dst, nbytes);
• flc_write(handle, src, nbytes);
• handle: FLC_HANDLE returned from flc_open(...)
• src, dst: may be an array or pointer to structure
• nbytes: size of data block in bytes
• flc_ready(read_handle): returns 1 if data block has been written to since the last time the handle was read

C API - Example

#include "flcomms.h"
struct STATES {
    double posxi, posyi, poszi;  /* position */
    double phi, theta, psi;      /* orientation */
} states;
double pilotin[4];
...
flc_attach(NULL, 0);
FLC_HANDLE states_handle = flc_open(
    "STATES", sizeof(struct STATES),
    0, FLC_TYPE_DOUBLE|FLC_MODE_READ);
FLC_HANDLE pilotin_handle = flc_open(
    "PILOTIN", 4*sizeof(double),
    0, FLC_TYPE_DOUBLE|FLC_MODE_WRITE);
...
flc_write(pilotin_handle, pilotin, 4*sizeof(double));
flc_read(states_handle, &states, sizeof(states));

Error reporting

• flc_attach() returns 1 if OK, 0 on error
• flc_open() returns NULL on error
• flc_read() may return 0 in case of concurrent write access
• Reason for failure recorded in standard C errno variable:
  • EBUSY ("Device or resource busy"): write lock unavailable
  • EINVAL ("Invalid argument"): size, type, or magic number mismatch
  • Others, see libflcomms(3), shmget(2) and shmat(2) manual pages for full list
• Can use perror("...") to report libflcomms-related errors

What to do when things go wrong

• Symptom: "Invalid argument" (EINVAL)
• Probable cause: data structures have changed
• Solution: all attached processes must release data blocks before reattaching
• Symptom: "Device or resource busy" (EBUSY)
• Probable cause: process has a write lock, possibly crashed
• Solution: run flcomms -c to release write locks
• Symptom: flc_read() returns 0, errno=ETIME ("Timeout")
• Probable cause: concurrent write access
• Solution: Probably safe to ignore; use previous value
• If ETIME persists, writer has probably been killed or suspended
• E2BIG, ENOSPC -- FLCOMMs size limitation exceeded (data block too large, too many data blocks)

Limitations

• Data blocks no larger than 2048 bytes (= 256 doubles)
• At most 126 separate data blocks
• 64Kb total size of FLCOMMS master block
• Data block names < 16 characters
• Actual limits are slightly less in practice due to memory allocation policies

FLCOMMS over a network - figure

NETFLC

• Usage: netflc &
• Uses broadcast UDP over the local area network
• Synchronizes FLCOMMS blocks on all participating hosts
• Periodically broadcasts current values of all data blocks being written to on this host
• Listens for updates from other hosts
• Handles floating point and integer format conversions between different architectures

Other netflc options

• netflc -k: broadcasts a ``shutdown'' message, tells all netflc processes on the LAN to quit
• netflc -i interval: specifies frame rate in millisec (default 20 millisec)
• netflc -v: verbose output
• netflc -q: quiet mode -- omit diagnostics other than errors
• netflc -r: ``receive mode'' -- don't broadcast updates
• netflc -s: ``send mode'' -- ignore updates from other hosts

Magic Numbers

• Provide an extra layer of consistency checking
• Scope computes magic number from a hash code of variable list field names
• Changing the variable list yields a different magic number (with low probability of collision)
• Use flcomms -l to determine magic number...
• ...or compute it by hand (see FLC_HASH_INITIAL and FLC_HASH_MULTIPLIER in flcomms.h header file)
• ...or use 0x0 as magic number to bypass check

Thread-safety issues

• flc_attach() is not thread-safe --
  should be called in main thread before starting any subthreads
• Concurrent access to an FLC_HANDLE is not thread-safe --
  acquire separate handle for each thread
• Concurrent access to data blocks is safe if each thread accesses it through a different handle
• flc_open() is thread-safe

Conclusion

Questions?