Repository Reorganization Planning

Last time I did a full checkout of ITAPS svn repo, it took forever and I got a slew of things I didn't really care about. And, I think the whole checkout is 1.3 Gibabytes!

Taxonomy of the kinds of things we seem to be storing in the repo...

  • Software
    • Managed via ITAPS repository...
      • This is software that has no life of its own apart from the ITAPS project.
        • Compliance tests
        • iMesh/iMeshP
        • Scripts to manage
      • Tools
        • eyeMesh
        • iMeshIO
      • Interface header files (aka API specifications)
        • iMesh.h, iBase.h, iRel.h, iGeom.h, iField.h
        • How ensure all implementations keep in sync. with these blessed versions in the repo?
        • Implementations only want to have to deal with changes in these at prescribed times of their own choosing.
      • Examples
        • Stuff created for various tutorials
        • Not currently regualarly maintained or tested and ought to be
    • Managed outside ITAPS repo...
      • Service/Implementation component Software
      • This is software that has a life of its own apart from ITAPS project but which the ITAPS project depends on.
      • Examples
        • MOAB, FMDB, GRUMMP tarballs
        • Zoltan, Mequite tarballs
      • This is software that the repo really only maintains a snapshot of, a copy from some other source (e.g. some contributor's repo).
      • We have copies of products that represent current (and past) releases of those products as well as copies that were known to integrate properly with each other.
        • We do not need to use the svn repo to maintain such copies. We just need some publicly available web-presence to store them in perpetuity.
  • Non-Software
    • Desgin documentation
      • Documentation of the data model and specification
      • Not sure what items this is but probably all the TSTT<whatever> docs
    • Other documentation
      • presentations (drafts and finals)
      • papers (drafts and finals)
    • www content
      • will new web site still require stuff in svn repo?

Zoltan/iZoltan and Mesquite as examples

Both Zoltan and Mesquite are ITAPS services.

Zoltan is developed/managed by Sandia National Labs as part of the Trilinos project. However, using Zoltan as an ITAPS service requires both Zoltan and the ITAPS interface to Zoltan, iZoltan. While Zoltan has a life of its own apart from ITAPS, iZoltan probably does not. The question arises, where is it appropriate to manage iZoltan? At SNL along side Zoltan or in the ITAPS repo?

Mesquite is similarly developed/managed by SNL. However, the ITAPS interface to Mesquite is not separated and managed apart from Mesquite itself. There is no technical or software engineering reason for this. It just so happens it is a matter of convenience in maintaining and managing Mesquite development. So, the question of where to manage the ITAPS interface to Mesquite does not present itself as strongly as it does for Zoltan.

When the question of where to house the ITAPS relevant portions of products such as these arise, the following options are available...
  • Manage/house it along with the product which it is a part of outside the ITAPS repo
    • This is currently what both Mesquite and Zoltan do
  • Manage/house it in the ITAPS repo
    • Because developers from outside SNL have expressed an interest in collaborating on iZoltan development, it may be appropriate in the future to move iZoltan into the ITAPS repo.
  • Manage/house it outside the ITAPS repo but maintain a vendor branch for it in the ITAPS repo.

Developers of contributing products such as these are free to choose which of the options above are most suitable for their needs.

User's Experience with Software from ITAPS Repository.

Users only care about software (and the documentation that relates to the use of said software). So, I am wondering about separating a lot of 'doc' type content from the 'software' type content so that no one has to download that to get the software. Also, I am wondering about organizing the software content more like the standard 'trunk, branches, tags' we ordinarily see in other software projects. In addition, move a lot of the current 'examples' to tests.

A standard svn usage is the trunk, branches, tags trio of dirs. Where this trio of dirs occurs in the hierarchy relative to the others is an important issue. A standard svn usage is to put under trunk, everything need to build the product(s). In the example, I even put distros under trunk. But, that is not necessary and maybe not even a good idea. Maybe distros should go above the trunk, branches, tags trio.

Some other issues to consider is use of upper case letters in dir names. Most projects tend to avoid that. We shoud definiately avoid things where names differ only in case as that winds up being a problem on Windows.

New proposed repository organization

Proposed new top-level directory structure

 older/ (for older TSTTX and SIDL stuff)
 docs/ (for all non-software, non-web content)
 www/ (web site related content)
     branches/ (have branches/tags/trunk trio only if deemed necessary)
     tags/
     trunk/
 software/ (sofware managed/released by ITAPS project)
     branches/
     tags/
     trunk/
         interface/
             iBase/ (see notes below regarding common subdirs under these)
                 spec/ lang/ impl/ config/ test/ mux/ examples/ data/
             iMesh/   " 
             iMeshP/  " 
             iGeom/   " 
             iRel/    " 
             iField/  " 
             .
             .
             .
         service/
             eyeMesh/
             iMeshIO/ (not really mainained by ITAPS)
             Swapping/
             visit_plugins/ (not really maintained by ITAPS)
             .
             .
             .
         tools/ (developer related tools)
         distros/ (all .tar.gz files of other products needed for releases)
             patches/ (any patches to original distributions)

Existing top-level directory structure

Below, existing top-level directories are listed along with their proposed disposition. Broadly speaking, one of the few following actions will be taken...
  • Move:
    • several existing top-level dirs will simply be moved to under the new docs dir
  • Delete:
    • a few directories are empty or are no longer needed.
  • Break Apart
    • For a directory with iBase,iMesh,iMeshP... children, its subdirs will be broken apart and moved under the respective child of the interface dir.
ComplianceTests (break apart)
Config (break apart)
Distributions (delete)
Documentation (move)
Examples (break apart)
Implementations (break apart)
Interface (move)
Papers (move)
Presentations (move)
SupportingTechnologies (delete)
Tools (rename to service)

Subdirs common to any child of interface dir

The immediate children of the interface dir are the ITAPS interfaces, iBase, iGeom, iMesh, iMeshP, iRel, iField, etc. For each interface, I propose we maintain a set of common subdirs; spec, lang, impl, config, test, mux, examples, data. We describe this structure here.

Remember, in theory, we aim to maintain C/C++/Fortran and even SIDL interface bindings for a given interface specification. For iBase and iMesh, we sort of cheated in that we adopted a C/C++/Fortran agnostic usage convention in declaring the functions such that the resulting header file works to define language bindings pretty much 1:1 with any of these three languages. But, we did give up something important in doing that and that is an interface that looks like what developers would expect to see programming in their native language. The interface doesn't look like what a C, C++ or Fortran programmer would expect to see. As an aside, I think that situation can be remedied by developing some simple tools (probably in perl) to massage existing interface specifcations into more native C, C++ and Fortran forms as well as generating some wrapper code to call a common implementation.

We have not given up on SIDL. But, SIDL interface bindings have definately lagged in development. In addition, the SIDL implementation of an interface specification is non-trivial. It is not just a single .h file but instead involves a whole directory-tree of files for server and client code as well as ability to interact with babel tools correctly.

I propose that each of the iBase, iMesh, iGeom, ... children of the interface directory will have the following sub-directories...
  • spec/
    • Interface specification. Currently the language-agnostic .h file. Eventually, I could see this being replaced with a .xml file and a simple perl script that generates the language specific bindings in the lang/ dir.
  • lang/
    • Room for possible language specific bindings for C, C++, Fortran.
    • SIDL bindings go here too, but they are a whole directory tree of stuff.
  • impl/
    • I think SimpleMesh is the only example of an ITAPS-managed implementation of any of the interfaces we support. So, 'impl' dir in most cases will be empty and maybe not necessary until it is actually non-empty. For iMesh, the impl dir would contain C and SIDL implementations of SimpleMesh.
  • config/
    • Interface specific stuff from current top-level Config dir goes here.
  • test/
    • Test code. We currently have iMesh and iMeshP test code.
  • mux/
    • Multiplexor code. We currently only have some code for iMesh.
  • examples/
    • Example code to demonstrate use of the interface. We mostly have C, C++ and Fortran examples for iMesh.
  • data/
    • Small, probably implementation-independent (e.g. VTK) data files used in testing, examples, whatever.

Services

Some services are wholly maintained by ITAPS and live in the ITAPS repo. An example is the eyeMesh browser. However, I think most services like VisIt plugins or Jim P's PyTAPS will come from somewhere else. Its important that ITAPS be aware, somehow, of these products. But, I am not sure how to go about codifying that awareness in the ITAPS repo. For certain, I see no value in placing tarballs here.

We could perhaps maintain vendor branches of such products (see vendor branches). A vendor branch is different from a "distro" in that a vendor_branch acts sort of like an svn mirror of a product that is actually managed out of another repo. A distro is just a distribution tarball. The contents of the tarball are obviously not under revision control. However, the contents of a vendor branch are under revision control. The primary reason for having a vendor_branch of, for example, RefImpl is to manage whatever minor changes we may need to maintain to RefImpl apart from the RefImpl true product life in another repository.

Referring back to the example of Zoltan and iZoltan, if we wanted to migrate iZoltan into the ITAPS repo, we would add it under the service directory here. Since iZoltan is not really the Zoltan service itself and is instead only an ITAPS compliant interface to that service, it may be appropriate to create an child directory of the service directory which maintains service interfaces apart from services themselves.

What is Swapping? Is that an ITAPS managed service?

Component tarballs and the distros directory.

In the distros directory, we can maintain only the tarballs of non-ITAPS managed products that are relevant to ITAPS releases. For example, we would put the FMDB-1.2 tarball there as well as MOAB-4.0 tarball, GRUMMP-0.6.1 etc. Aside from committing tarballs of these products as part of an ITAPS release, I propose we no longer commit tarballs of these products anywhere else or at any other time in the ITAPS repo. In other words, we need only commit tarballs of those contributor's products that are used in building released ITAPS code(s).

Handling contributor tarballs this way gives us (me) the opportunity to adjust these tarballs in any tiny way that might be necessary to complete a release without having to wait/demand that each collaborating organization fix one tiny thing or another which can often leads to delays in making a release that could otherwise be avoided. However, whenever such a tarball is perturbed from its original, prestine from as presented from the collaborating organization, we will
  • maintain patch files for them
  • keep the patched tarballs in the distros dir in the ITAPS repo

When we make an ITAPS release, the released code will use/obtain tarballs of these products from the distros dir in the ITAPS repo. However, as an option, we will also support building a release by using/obtaining any given collaborator product from their respective organization's download points.

For the preceding reasons, I propose to remove the following tarballs...

./Tools/iMeshIO/imeshio_test_data.tar.gz
./Distributions/LIIPBMod/LIIPBMod.tar.gz
./Distributions/cmake/cmake-2.4.6-Linux-i386.tar.gz
./Distributions/iGeom_Distributions/GMI/GMI-1.0.tar.gz
./Distributions/IPComMan/ipcomman-1.0.tar.gz
./Distributions/babel/babel-1.0.4-451.tar.gz
./Distributions/babel/babel-0.10.10.tar.gz
./Distributions/babel/babel-1.0.4.tar.gz
./Distributions/Zoltan/izoltanparallel-0.8.tar.gz
./Distributions/Zoltan/zoltan_distrib_v3.3.tar.gz
./Distributions/Zoltan/iZoltanTutorial.tar.gz
./Distributions/Zoltan/izoltanserial-0.9.tar.gz
./Distributions/autopack/autopack-1.3.2.tar.gz
./Distributions/iMesh_Distributions/GRUMMP/ITAPS-GRUMMP-1.0-3.tar.gz
./Distributions/iMesh_Distributions/FMDB/FMDB-1.2.tar.gz
./Distributions/iMesh_Distributions/FMDB/SCUtil.tar.gz
./Distributions/iMesh_Distributions/FMDB/GMI-1.0.tar.gz
./Distributions/iMesh_Distributions/FronTier/FronTier.Lite.06_28_07.tar.gz
./Distributions/iMesh_Distributions/FronTier/FronTier.tar.gz
./Distributions/iMesh_Distributions/RefImpl/RefImpl-1.1.0.tar.gz
./Distributions/iMesh_Distributions/iMesh-test.tar.gz.1
./Distributions/iMesh_Distributions/iMesh-test-1.1.tar.gz
./Distributions/Swapping/ITAPS-Swap-1.0-p6.tar.gz
./Distributions/Tutorial_Exercises/Hello_iMesh-1.0.tar.gz
./Distributions/Tutorial_Exercises/iZoltanTutorial.tar.gz
./Distributions/VisIt/ITAPS_C_for_VisIt_1.11.X.tar.gz
./Distributions/VisIt/ITAPS_C_for_VisIt_2.0.tar.gz
./Distributions/iBase_Distributions/RPI/Utilities.tar.gz
./www/software/examples/FindAdjacency.tar.gz
./Implementations/FronTier/FronTier_no_TSTT.05_03_07.tar.gz
./Implementations/FronTier/FronTier.gas.10_18_05.tar.gz
./Implementations/NWGrid/linux86-HPF-CC.tar
./Implementations/NWGrid/NWGrid_simple_example.tar.gz
./Implementations/MOAB/MOAB-1.01.tar.gz
./Implementations/MOAB/TSTTBM.tar.gz
./SupportingTechnologies/babel-0.9.8.tar.gz

Planning the actual SVN operations to perform the re-organzation

  1. Create a to_be_deleted directory and commit that
  2. Move whole Distributions and SupportingTecnologies dirs and other tarballs listed above to the 'to_be_deleted' directory, commit that
  3. Create to top-level doc directory, commit that
  4. Move top level Documentation, Presentation and Papers directories to the doc directory, commit that
  5. Create the following directory tree
     software/
         branches/
         tags/
         trunk/
             interface/
                 iBase/
                     spec/ lang/ impl/ config/ test/ mux/ examples/ data/
                 iMesh/ 
                     spec/ lang/ impl/ config/ test/ mux/ examples/ data/
                 iMeshP/
                     spec/ lang/ impl/ config/ test/ mux/ examples/ data/
                 iGeom/ 
                     spec/ lang/ impl/ config/ test/ mux/ examples/ data/
                 iRel/ 
                     spec/ lang/ impl/ config/ test/ mux/ examples/ data/
                 iField/
                     spec/ lang/ impl/ config/ test/ mux/ examples/ data/  
             service/
             tools/
             distros/
                 patches/
    
  6. commit the emty directory tree above.
  7. move following service software
    • eyeMesh
    • iMeshIO
    • Swapping
    • commit the above moves
  8. move interface header files to respective spec dirs
  9. move contents of top-level Config dir to respective config dirs
  10. move contents of ComplianceTest dir to respective test dirs
  11. add mux code for iMesh
  12. move SimpleMesh impl to iMesh impl dir
  13. move contents of Examples dir to respective examples dirs
  14. move various developer tools to tools dir
  15. commit above changes
  16. At this point, Config, Examples, ComplianceTest, Implementation and Tools dirs should be empty, remove them and commit that change
  17. After a couple weeks, delete the 'to_be_deleted' directory.

Checking out ITAPS software and other release management operations

Be sure to read the few pages of SVN manual starting here.

With the new directory organization, to check out the current (trunk) of ITAPS software, the command would look like...

svn co https://redmine.scorec.rpi.edu/svn/itaps/software/trunk

To check out a particular release, such as the 1.2 release, the command would look like...
svn co https://redmine.scorec.rpi.edu/svn/itaps/software/tags/1.2

However, be aware that we presently have not actually pulled together all the pieces for the 1.2 release and tagged that. We will do that hopefully by end of January just as an execrise. Then, you will actually see a '1.2' in the tags directory.

To create a branch, do some work on that branch and then merge the branch into the trunk, the commands would look like

# create a branch
svn copy https://redmine.scorec.rpi.edu/svn/itaps/software/trunk \
         https://redmine.scorec.rpi.edu/svn/itaps/software/branches/eyeMesh_qt4.5_fixes \
    -m 'making a branch for fixes to eyeMesh for Qt-4.5'
# check out the branch
svn co https://redmine.scorec.rpi.edu/svn/itaps/software/branches/eyeMesh_qt4.5_fixes
# do work on your working copy of the branch and commit changes to the branch
cd eyeMesh_qt4.5_fixes

svn commit -m 'did some work on the branch'

# keep your branch in sync with the trunk by merging trunk changes into your branch
svn merge https://redmine.scorec.rpi.edu/svn/itaps/software/trunk
# commit merged changes from trunk to your branch
svn commit -m 'syncing with trunk'

# Ok, you are done with the branch, now merge all the work on it to the trun
# to do that, you need a working (local) copy of the trunk. So, check it out
svn co https://redmine.scorec.rpi.edu/svn/itaps/software/trunk
cd trunk
svn merge --reintegrate https://redmine.scorec.rpi.edu/svn/itaps/software/branches/eyeMesh_qt4.5_fixes
svn commit -m 'merging my qt fixes branch to trunk'