


  mcxio(5)                  FILE FORMATS                   mcxio(5)


  NAME
      mcxio  -  the format specifications for input and output in the mcl fam-
      ily.

  DESCRIPTION
      This document describes

      native matrix format
        The format that can be read in by  any  mcl  application  expecting  a
        matrix  argument.  The  native  format closely resembles the layout of
        matrices as residing in computer memory. There are two distinct encod-
        ings,  respectively interchange and binary.  Their relative merits are
        described further below.

      concatenated native matrix format
        This always pertains to matrices in native format  concatenated  in  a
        single  file,  refered  to  as  a  cat file. It is used for example to
        encode hierarchical clusterings as generated  by  mclcm.  A  cat  file
        either  consists  of  matrices in interchange format or of matrices in
        binary format.

      raw intermediate format
        This is read by mcxassemble(1).

      tab format
        Used by applications such as mcl(1) and mcxdump(1) to convert  between
        meaningful  labels describing the input data and the numerical identi-
        fiers used internally.

      label format
        The  format  used  when  streaming  labels  directly  into  mcl(1)  or
        mcxload(1).

      transformation syntax
        The  syntax  accepted  by mcl(1), mcxload(1), mcxsubs(1) and mcxassem-
        ble(1) to transform values in an input stream or in  input  or  output
        matrices.

      The  interchange  format  is  a  portable format that can be transmitted
      across computers and over networks and will work with any version of mcl
      or  its  sibling  programs.  It  is  documented  (here) and very stable.
      Applications can easily create matrices in this format.  The drawback of
      interchange  format  is that for very large graphs matrix encodings grow
      very big and are slow to read.

      The binary format is not garantueed to be portable  across  machines  or
      different  versions  of mcl or differently compiled versions of mcl. Its
      distinct advantage is that for very large  graphs  the  speed  advantage
      over interchange format is dramatic.

      Conversion  between  the two formats is easily achieved with mcxconvert.
      Both mcl(1) and mcxload(1) can save a matrix in either format after con-
      structing it from label input.




  mcxio 1.008, 09-308         4 Nov 2009                          1





  mcxio(5)                  FILE FORMATS                   mcxio(5)


      The  concatenated format is generated e.g. by mclcm(1) and can be trans-
      formed by mcxdump(1) using the -imx-cat option. In cat  format  matrices
      are simply concatenated, so it is easily generated from the command line
      if needed.  For native binary format it is imperative that no additional
      bytes  are  inserted  inbetween  the matrix encodings. For native inter-
      change format the only requirement is that the last matrix  is  followed
      by nothing but white space.

      A  remark  on the sloppy naming conventions used for mcl and its sibling
      utilities may be in order here. The  prefix  mcx  is  used  for  generic
      matrix  functionality,  the prefix clm is used for generic cluster func-
      tionaliy. The utility mcx is a general purpose interpreter  for  manipu-
      lating  matrices (and grahps, sets, and clusterings). The set of all mcl
      siblings (cf. mclfamily(7)) is loosely refered to  as  the  mcl  family,
      which  makes  use  of the mcl libraries (rather than the mcx libraries).
      The full truth is even more horrible, as the mcl/mcx prefix  conventions
      used in the C source code follow still other rules.

      In  this document, 'MCL' means 'the mcl setting' or 'the mcl family'. An
      MCL program is one of the programs in the mcl family. The  remainder  of
      this document contains the following sections.

  Internal representation of matrices in MCL
      There  are  several aspects to the way in which MCL represents matrices.
      Internally, indices never act as an ofset in an array,  and  neither  do
      they  participate in ofset computations. This means that they purely act
      as identifiers. The upshot is that matrices can be handled in which  the
      index  domains  are  non-sequential (more below). Thus one can work with
      different graphs and matrices all using  subsets  of  the  same  set  of
      indices/identifiers.  This aids in combining data sets in different ways
      and easily comparing the respective  results  when  experimenting.  Sec-
      ondly, only nonzero values (and their corresponding indices) are stored.
      Thirdly, MCL stores a matrix as a listing of columns. Iterating  over  a
      column  is trivial; iterating over a row requires a costly transposition
      computation.  The last two points should matter little to  the  user  of
      MCL programs.

      In textbook expositions and in many matrix manipulation implementations,
      matrices are represented with sequentially  indexed  rows  and  columns,
      with the indices usually starting at either zero or one. In the MCL set-
      ting, the requirement of sequentiality is dropped, and it follows  natu-
      rally that no requirement is posed on the first index. The only require-
      ment MCL poses on the indices is that they be nonnegative,  and  can  be
      represented  by  the  integer  type  used  by MCL. On many machines, the
      largest allowable integer will be 2147483647.

      MCL associates two domains with a matrix M, the row  domain  and  column
      domain.  The matrix M can only have entries M[i,j] where i is in the row
      domain and j is in the column domain. This is vital  when  specifying  a
      matrix:  it  is illegal to specify an entry M[i,j] violating this condi-
      tion. However, it is not necessary to specify all entries M[i,j] for all
      possible  combinations  of i and j. One needs only specify those entries
      for which the value is nonzero, and only nonzero values will  be  stored
      internally.  In  the  MCL  matrix  format,  the  matrix  domains must be



  mcxio 1.008, 09-308         4 Nov 2009                          2





  mcxio(5)                  FILE FORMATS                   mcxio(5)


      specified explicitly if they are not canonical (more below).

      Strictly as an  aside,  the  domains  primarily  exist  to  ensure  data
      integrity. When combining matrices with addition or multiplication (e.g.
      using the mcx utility), MCL will happily combine matrices for which  the
      domains do not match, although it will usually issue a warning.  Concep-
      tually, matrices auto-expand to the dimensions required for  the  opera-
      tion.  Alternatively,  a  matrix  can be viewed as an infinite quadrant,
      with the domains delimiting the  parts  in  which  nonzero  entries  may
      exist.   In  the  future, facilities could be added to MCL (c.q. mcx) to
      allow for placing strict domain requirements on matrices when  submitted
      to binary operations such as addition and multiplication.

  Specifying matrices
      From here on, all statements about matrices and graphs are really state-
      ments about matrices and graphs in the MCL setting. The specification of
      a matrix quite closely matches the internal representation.

      A  matrix  M has two domains: the column domain and the row domain. Both
      simply take the form of a  set  (represented  as  an  ordered  list)  of
      indices. A canonical domain is a domain of some size K where the indices
      are simply the first K nonnegative integers 0,1..,K-1. The domains  dic-
      tate  which  nonzero  entries  are  allowed  to  occur in a matrix; only
      entries M[i,j] are allowed where i is in the row domain and j is in  the
      column domain.

      The  matrix  M  is  specified  in  three  parts, for which the second is
      optional.  The parts are:

      Header specification
        This specifies the dimensions K and L of the matrix, where  K  is  the
        size  of  the  row domain, and L is the size of the column domain.  It
        looks as follows:

        (mclheader
        mcltype matrix
        dimensions 9x14
        )

        This dictates that a matrix will be specified for which the row domain
        has dimension 9 and the column domain has dimension 14.

      Domain specification
        The  domain specification can have various forms: if nothing is speci-
        fied, the matrix will have canonical domains and a canonical represen-
        tation, similar to the representation encountered in textbooks. Alter-
        natively, the row and column domains can each be specified separately,
        and  it  is  also possible to specify only one of them; the other will
        simply be a canonical domain again. Finally, it is possible to declare
        the  two domains identical and specify them simultaneously. It is per-
        fectly legal in each case to explicitly specify a canonical domain. It
        is required in each case that the number of indices listed in a domain
        corresponds with the dimension given in the header.




  mcxio 1.008, 09-308         4 Nov 2009                          3





  mcxio(5)                  FILE FORMATS                   mcxio(5)


        An example where both a row domain and a column domain are specified:

        (mclrows
         100 200 300 400 500 600 700 800 900 $
        )
        (mclcols
         30 32 34 36 38 40 42 44 46 48 50 52 56 58 $
        )

        This example combines with the header given above, as  the  dimensions
        fit.   Had  the  row domain specification been omitted, the row domain
        would automatically be set to the integers  0,1,..8.  Had  the  column
        specification been omitted, it would be set to 0,1,..13.

        Suppose now that the header did specify the dimensions 10x10.  Because
        the dimensions are identical, this raises  the  possibility  that  the
        domains  be identical.  A valid way to specify the row domain and col-
        umn domain in one go is this.

        (mcldoms
         11 22 33 44 55 66 77 88 99 100 $
        )


      Matrix specification
        The matrix specification starts with the sequence

        (mclmatrix
        begin

        The 'begin' keyword in the '(mclmatrix' part is followed by a list  of
        listings,  where  the primary list ranges over all column indices in M
        (i.e.  indices in the column domain), and where each  secondary  lists
        encodes  all positive entries in the corresponding column. A secondary
        list (or matrix column) starts with the index c  of  the  column,  and
        then  contains  a  listing  of  all row entries in c (these are matrix
        entries M[r,c] for varying r). The entry M[r,c] is specified either as
        'r'  or  as  'r:f',  where  f is a float. In the first case, the entry
        M[r,c] defaults to 1.0, in the second case, it is set to f.  The  sec-
        ondary  list is closed with the `$' character. A full fledged examples
        thus looks as follows:

        (mclheader
        mcltype matrix
        dimensions 12x3
        )
        (mclrows
         11 22 33 44 55 66 77 88 99 123 456 2147483647 $
        )
        (mclcols
          0  1  2 $
        )
        (mclmatrix
        begin



  mcxio 1.008, 09-308         4 Nov 2009                          4





  mcxio(5)                  FILE FORMATS                   mcxio(5)


        0    44 88 99 456 2147483647 $
        1    11 66 77 123 $
        2    22 33 55 $
        )

        Note that the column domain is canonical; its specifiation could  have
        been omitted. In this example, no values were specified. See below for
        more.

  Specifying graphs
      A graph is simply a matrix where the row domain is the same as the  col-
      umn domain. Graphs should have positive entries only. Example:

      (mclheader
      mcltype matrix
      dimensions 12x12
      )
      (mcldoms
      11 22 33 44 55 66 77 88 99 123 456 2147483647 $
      )
      (mclmatrix
      begin
      11    22:2  66:3.4  77:3  123:8 $
      22    11:2  33:3.8  55:8.1 $
      33    22:3.8  44:7  55:6.2 $
      44    33:7  88:5.7  99:7.0 456:3 $
      55    22:8.1  33:6.2  77:2.9  88:3.0 $
      66    11:3.4  123:5.1 $
      77    11:3  55:2.9  123:1.5 $
      88    44:5.7  55:3.0  99:3.0 456:4.2 $
      99    44:7.0  88:3.0 456:1.8 2147483647:3.9 $
      123   11:8  66:5.1  77:1.5 $
      456   44:3  88:4.2  99:1.8 2147483647:6.3 $
      2147483647   99:3.9 456:6.3 $
      )

      Incidentally,  clustering this graph with mcl, using default parameters,
      yields a cluster that is represented by the 12x3 matrix shown earlier.

      The following example shows the same graph, now represented on a canoni-
      cal domain, and with all values implicitly set to 1.0:

      (mclheader
      mcltype matrix
      dimensions 12x12
      )
      (mclmatrix
      begin
      0    1  5  6  9 $
      1    0  2  4 $
      2    1  3  4 $
      3    2  7  8 10 $
      4    1  2  6  7 $
      5    0  9 $



  mcxio 1.008, 09-308         4 Nov 2009                          5





  mcxio(5)                  FILE FORMATS                   mcxio(5)


      6    0  4  9 $
      7    3  4  8 10 $
      8    3  7 10 11 $
      9    0  5  6 $
      10   3  7  8 11 $
      11   8 10 $
      )

      Additional notes
      There are few restrictions on the format that one might actually expect.
      Vectors and entries may occur in any  order  and  need  not  be  sorted.
      Repeated  entries  and  repeated vectors are allowed but are always dis-
      carded while an error message is emitted.

      If you want functionally interesting  behaviour  in  combining  repeated
      vectors  and  repeated  entries,  have a look at the next section and at
      mcxassemble(1).

      Within the vector listing, the '#' is a token that introduces a  comment
      until the end of line.

  Raw format
      A file in raw format is simply a listing of vectors without any section-
      ing structure. No header specification, no domain specification, and  no
      matrix  introduction syntax is used - these are supplied to the process-
      ing application by other means. The end-of-vector token '$'  must  still
      be  used,  and  the  comment  token  '#'  is still valid. mcxassemble(1)
      imports a file in raw format, creates a  native  matrix  from  the  data
      therein,  and  writes  the  matrix to (a different) file. It allows cus-
      tomizable behaviour in how to combine repeated entries and repeated vec-
      tors.  This  is  typically used in the following procedure. A) Do a one-
      pass-parse on some external cooccurrence file/format, generate raw  data
      during  the  parse and write it to file (without needing to build a huge
      data structure in memory). B) mcxassemble takes the raw data and  assem-
      bles it according to instruction into a native mcl matrix.

  Tab format / label information
      Several  mcl  programs  accept options such as -tab, -tabc, -tabr, -use-
      tab, -strict-tab, and -extend-tab.  The argument  to  these  options  is
      invariably the name of a so-called tab file.  Tab files are used to con-
      vert between labels (describing entities in the  data)  and  indices  as
      used  in  the  mcl matrix format.  In a tab file each line starts with a
      unique number which presumably corresponds to an index used in a  matrix
      file.   The  rest  of  the line contains a descriptive string associated
      with the number. It is required that each string is unique, although not
      all  mcl  programs  enforce this at the time of writing.  The string may
      contain spaces.  Lines starting with # are considered  comment  and  are
      disregarded.

      Tab domain
      The  ordered  set  of  indices  found  in the tab file is called the tab
      domain.

      Tab files are almost always employed in conjunction with an  mcl  matrix



  mcxio 1.008, 09-308         4 Nov 2009                          6





  mcxio(5)                  FILE FORMATS                   mcxio(5)


      file.   mcxdump(1)  and  clmformat(1)  require  by  default that the tab
      domain coincides with the matrix domain (either row or column  or  both)
      to which they will be applied. This can be relaxed for either by supply-
      ing the --lazy-tab option.

      mcl provides explicit modes for dealing with tab structures by means  of
      the  -extend-tab,  -restrict-tab  and  -strict-tab options. Refer to the
      mcl(1) documentation.

  Label input
      Label input is a line based input where two nodes and an optional  value
      are  specified on each line. The nodes should be specified by labels. If
      the labels contain spaces they should be  separated  by  tabs  (and  the
      value  if  present should be separated from the second label by a tab as
      well). The parse code will assume tab-separated labels if it sees a  tab
      character in the input, otherwise it will split the input on any kind of
      whitespace.  Any line where the first non-whitespace  character  is  the
      octothorp (#) is ignored. The following is an example of label input.

      ---8<------8<------8<------8<------8<---
      # the cat and the hat example
      cat hat  0.2
      hat bat  0.16
      bat cat  1.0
      bat bit  0.125
      bit fit  0.25
      fit hit  0.5
      hit bit  0.16
      --->8------>8------>8------>8------>8---

      mcl(1) can read in label input and cluster it when it is given the --abc
      option. It can optionally save the input graph in native format and save
      the  label  information in a tab file with the -save-graph and -save-tab
      options.

      Refer to the MCL getting started and MCL manual  examples  sections  for
      more information on how MCL deals with label input.

      mcxload(1)  is  a  general purpose program for reading in label data and
      other stream formats. It encodes them  in  native  mcl  format  and  tab
      files.  It allows intermediate transformations on the values.

  Transformation syntax
      mcl(1),  mcxload(1),  mcxsubs(1),  mcxassemble(1)  all  accept  the same
      transformation language in their respective tf-type options and mcxsub's
      val specification.

      A  statement  in this language is simply a comma-separated list of func-
      tions accepting a single numerical value.   The  syntax  of  a  function
      invocation  in general is func(arg).  The functions exp, log, neglog can
      also be given an empty parameter list, indicating that e is taken as the
      exponent  base. In this case, the invocation looks like func(). The fol-
      lowing functions are supported.




  mcxio 1.008, 09-308         4 Nov 2009                          7





  mcxio(5)                  FILE FORMATS                   mcxio(5)


      lt       Filter out values greater than or equal to arg.
      lq       Filter out values greater than arg.
      gq       Filter out values less than arg.
      gt       Filter out values less than or equal to arg.
      ceil     Set everything higher than arg to arg.
      floor    Set everything lower than arg to arg.
      mul      Multiply by arg.
      add      Add arg to it.
      power    Raise to power arg.
      exp      Raise arg (e if omitted) to value.
      log      Take log in base arg (e if omitted).
      neglog   Take minus log in base arg (e if omitted).

      NOTE
      mcl(1) accepts --abc-neg-log and --abc-neg-log10 to specify  log  trans-
      formations.  Similarly,  mcxload(1)  accepts --stream-log, --stream-neg-
      log, and --stream-neg-log10. The reason is that probabilities are  some-
      times  encoded  below  the precision dictated by the IEEE (32 bit) float
      specification. This poses a problem as the mcl applications encode  val-
      ues  by  default  as  floats,  and the transformation specifications are
      always applied to the mcl  encoding.  The  options  just  mentioned  are
      applied  after  a value has been read from an input stream and before it
      is converted to the native encoding.

  SEE ALSO
      mcxassemble(1), and mclfamily(7) for an overview of all  the  documenta-
      tion and the utilities in the mcl family.

  AUTHOR
      Stijn van Dongen.



























  mcxio 1.008, 09-308         4 Nov 2009                          8


