Overview and Publications

Its features include the following (for a pictorial introduction, see the Tour):

Find synteny between two sequenced eukaryotic genomes with optional annotation. Draft sequence ordering by synteny, i.e. align a draft genome to a fully sequence (not draft-to-draft). For multiple selected synteny pairs, display using dot plot, circular, and side-by-side. Complete queries on annotation, collinear genes, cross-species gene families, etc.

Click an image to see the closeup.

Publications

        C. Soderlund, M. Bomhoff, and W. Nelson (2011)
        SyMAP: A turnkey synteny system with application to plant genomes.
        Nucleic Acids Research 39(10):e68.

        C. Soderlund,  W. Nelson, A. Shoemaker and A. Paterson (2006)
        SyMAP: A System for Discovering and Viewing Syntenic Regions of FPC maps
        Genome Research 16:1159-1168.

Steps for finding synteny

The released symap.jar file has been compiled with Java 17.0.10, which is upward compatible. If you need a version compile with Java 1.8, email symap@agcol.arizona.edu.

For performing large alignments (e.g. 1Gb genomes or more) it is essential to have multiple CPUs and at least 5Gb of RAM for each CPU that you intend to use. Note that you can set the number of CPUs for SyMAP to use. See MUMmer for information on insufficient memory.

For viewing alignments, CPU and memory needs are typically negligible, unless you are performing queries on more than 4-5 genomes at once.

SyMAP has been tested on the following:

Installation

     > tar -xf symap_5.tar.gz

   LICENSE   README   data/   ext/   java/
   scripts/  symap    symap.config   viewSymap

Externals: The ext/ directory contains the external programs MUMmer^1,2 (for sequence alignment) and MUSCLE⁷ (for Queries). The directory contains:

	README		mummer/		mummer4/	muscle/

1. Change into the symap_5 directory.

2. Edit symap.config and enter database and host information (see MySQL).

3. From the command line, type ./symap.

The first time you run SyMAP, it will create the database with information written to the terminal, e.g.

Creating database 'symapDemo' (jdbc:mysql://localhost/symapDemo?characterEncoding=utf8).

It will also check that the provided external programs (e.g. MUMmer) are executable; if it shows any problems, see Executables. For MacOS, you may also need MacOS externals.

The Project Manager window opens showing the three demo projects provided with the SyMAP tarball. Check Demo-Seq and Demo-Seq2.
A link Load All Projects will be displayed at the top of the right panel; select it to load the projects, which will take several minutes. If loading the Demo-Seq takes more than a few minutes, you may need to adjust the MySQL parameters, see TroubleShoot MySQL. When done, the Manager will look like the image shown on the right. In the Available Syntenies table, the cell for Demo_Seq2 and Demo_Seq will automatically be selected. Click the Selected Pair button to start the Alignment&Synteny. The All Pairs button can also be selected, as it will perform Alignment&Synteny on all pairs in the table.
The Alignment&Synteny takes less than 5 minutes on the MacOS 10.5 but could take up to 30 minutes on a slow machine. When done, the table will have a checkbox, signifying that the synteny is available for viewing.
Click Summary to view the summary shown on the right; there may be slight differences in the #Cluster hits because of different numbers of CPUs, MacOS vs Linux, etc (but the #Blocks come out the same). To view the other interfaces, see Demo Results.   Once the alignments are computed, the Parameters can be experimented without having to redo the alignments. Try using the Algorithm 2 (gene-centric) for Cluster Hits; bring up the Parameters window and select Algorithm 2 with its defaults. This gives a summary similar to this.

Load the Demo-Draft project. Under the Demo-Draft listing, you will see the parameter "Order against: demo_seq2". With this setting, the Demo-Draft contigs will be ordered using synteny to Demo-Seq2; this was set in the Project Parameters window.
Run the Alignment&Synteny, where the alignment should take less than 30 minutes with one CPU. When done, open the Summary for the pair, as shown on the right; as mentioned above, there may be slight difference in the number of anchors. See the first dot plot in Demo-draft. It is recommended that the Cluster Hits algorithm 1 be used for ordering sequence.

The ordering algorithm changes the order of the draft contigs in the database, but does not change the sequence files on disk. However, it writes the following files:

1. File of ordered contigs: It writes the order of the contigs along with whether they should be flipped to a file called /data/seq/demo_draft/ordered.csv.

2. Fasta files of ordered sequences: It creates sequence files from the ordered contigs that are flipped when appropriate, which are put into a new project with the suffix "_ordered", as shown in the image below. The chromosome names correspond to the order-against project (e.g. demo_seq2), and the third chromosome is 'chr0' which contains all draft sequences that were not placed.

As shown in the image on the left, a new project has been added. Click Demo_Draft-ordered and load it. Running the Alignment&Synteny between the _ordered project and Demo_seq2 will provide a more coherent display (see the second dot plot for Demo-draft). Using the project Parameters window, the Demo_Draft-ordered name can be shortened.

Self alignment

MySQL and parameters

Important Note: The default settings of MySQL are poorly suited for large-scale data storage. You will want to adjust the parameters innodb_buffer_pool_size and innodb_flush_log_at_trx_commit as described in Trouble Shoot MySQL.

Parameters for accessing the MySQL database should be set in the symap.config file in the main symap directory, as follows:

Example symap.config.

db_name             = symapDemo
db_server           = localhost
db_adminuser        = <adminid>
db_adminpasswd      = <password>
db_clientuser       =
db_clientpasswd     =

>./symap -c symapTmp.config

Runtime and Memory

The largest component of SyMAP execution time is in running MUMmer^1,2. The time and memory for MUMmer all depends on the size of the genomes. For example, to align rice (12 chromosomes, 370Mb) to maize (10 chromosomes, 2Gb) required 1 hour and 3 minutes using 8 CPUs with 2.3Ghz speed.

The memory usage of MUMmer is typically 5G per CPU, however it can be as high as 10G for very long or repetitive chromosomes. If MUMmer fails, it is often due to insufficient memory, see the MUMmer document, which explains how to determine the problem and ways around it. It also explains the CPU and Concat options. It also explains how to run MUMmer on a different machine and port the results to the symap_5/data/seq_results directory.

To create a new project via the SyMAP interface, press the Add Project button at the lower left. Enter the project name beside Name:. This is the directory name for the project. It should be a short and unique name containing only letters, numbers, and underscores. In the Project parameters, you will be able to add a more descriptive display name.
After saving the new project, it appears in the Projects list on the left, but it is still an empty shell. A directory will be made under the /data/seq, e.g. for the project added on the right, a directory will be created called /data/seq/foobar. Check its box and it will appear in the Selected section (right hand side).

SyMAP Parameters

See Project Parameters for details.

See Pair Parameters for details.

>Chr3  Oryza Sativa
GAATTCGAATTTGGGTAATGCTAATCAATACAGGTCAAAATCTATGTATTGAGTGGAATATACTGCAAAGTAATTACCTT
CTTCCAAAGGAAAGCATTCCTTCTCTCTTGTGGGACTAGCAGATGATCTCGCAGCCAAGACGTGACCACCCAAGGCTCAC
...

The first decision with whole-genome sequence is whether to used repeat masked sequences.

• Masking reduces alignment time and false-positive hits, but also runs a risk of concealing true hits due to inaccurate masking.
• SyMAP does not perform the repeat-masking so must be done with another program. However, you may obtain masked sequences from NCBI or Ensembl. NCBI provides soft-masked sequences where the scripts/ConvertNCBI script will convert it to hard-masked, and Ensembl provides both soft and hard-masked sequences (see Convert).
• Masking is not really necessary unless the genome is highly repetitive and those repeats are shared with other genomes being aligned. (Repeats cause particular trouble for self-alignments, see self-alignments in SyMAP).
• Occasionally, MUMmer fails aligning sequences (see MUMmer, which explains detection and solutions for known reasons for failure).

Another masking option which is available if you have gene annotation is to mask out everything but the annotated genes. You can enable the mask_all_but_genes option on the project's Parameter window; turn it on before doing the alignments.

Important points in naming sequences for SyMAP:

The last column (attributes) contains "keyword=value" pairs describing the annotation.

• For genes, all attributes are saved in the database for viewing. You can set which attributes keywords to save; open the project's Parameter window, look for parameter Anno keywords.
• For exons, the "parent" keyword=value is saved for viewing.

NCBI files: A Java script (scripts/ConvertNCBI.class) has been provided that converts NCBI genome FASTA files and gff annotation files into the format that works best with SyMAP. See the documentation for instructions.

Ensembl files: A Java script (scripts/ConvertEnsembl.class) has been provided that converts Ensembl genome FASTA files and gff3 annotation files into the format that works best with SyMAP. See the documentation for instructions.

If you are ordering the draft sequence against a closely related sequenced genome, see demo draft on how to proceed. It is strongly suggested you run the run the demo! In a nutshell, the steps are:

1. Load both sequences.
2. For the draft, bring up the Parameters window. Beside the Order against row is a drop-down of all loaded projects; select your whole genome project.
3. Run the Alignment&Synteny. At the end, you will see a new project with the suffix _ordered in the left panel. It will contains:
   • A sequence directory with a ".fa" FASTA file. Any scaffolds matching the Order against genome will be assigned the same chromosome name. All scaffolds aligning to an Order against chromosome will be appended together in order with 100 N's between each scaffold. Any extra sequences will be put in ">Chr0".
   • An annotation directory with a ".gff" file that specifies where the gaps are.
4. Load the _ordered project. Then run Alignment&Synteny between the whole genome project and the new _ordered project.

If the draft sequence is in too many sequence pieces, (1) it takes a long time for the MUMmer comparisons, (2) the display is very cluttered, and (3) the blocks display does not work right. Limit the number of sequence pieces by setting Minimum length in the project's Parameter window to only load the largest 150 sequences; there is a script called scripts/lenFasta.pl which will print out all the lengths; set the Minimum length to the 150th length. However, even 150 are a lot of blocks to view so you might want to start with the largest 50, merge them, then repeat.

To perform self-synteny, select the cell for the same project followed by Selected Pair. The All Pairs option does not include self-synteny.

By default, SyMAP uses the MUMmer 'NUCmer' program for self-alignments. Each chromosomes is compared to every other chromosome including itself.

The Alignment&Synteny Parameters window has an option to set Self Args, which is only used when comparing the chromosome sequence file to itself. Make sure that the Algorithm 1 option is selected.

The self-synteny of demo_seq shows a few tiny blocks. A better demonstration is to download Arabidopsis thaliana from NCBI, convert it with the NCBI convert script, and run the self synteny. It took 16 minutes with one processor on a Mac Mini (2018) with 64GB main memory. The dot plot is shown on the right (click on the image for a closeup view).

The "--maxmatch" parameter was the SyMAP default, but now it is up to the user as to whether they want to add it to Self Args (it can greatly increase the execution time). Reasoning for using "--maxmatch": MUMmer ordinarily seeds its alignments with unique matches, which eliminates the possibility of off-diagonal seeds in the alignment of a chromosome to itself. To overcome this problem, individual chromosome self-alignments can use the MUMmer parameter -maxmatch, which removes the uniqueness requirement at the cost of greatly increased noise. The extra noise is then filtered to a large extent by the default SyMAP filters, but the diagonal squares of the dot plot will still have more noise visible than the off-diagonal.

The Load and A&S methods have a popup progress window, as shown on the right. There is a Cancel button on the bottom that can be clicked to cancel the execution; it will remove the results from the database and exit. Occasionally, the Cancel will cause it to create an error, writing to the error.log or to the terminal. This is not a problem, though you may need to remove the results yourself. If MUMmer is running when you Cancel, make sure there is "Error: Failed command:" line to terminal for each MUMmer alignment that was running; if there is not, use the "top" linux command to view the running processes and stop any MUMmers still running.

General

   /data/seq/<project-name>/sequence
   /data/seq/<project-name>/annotation

1. Create these sub-directories under /data/seq and put your files there.

2. Create these sub-directories under /data/seq and use soft links to point to the file locations, e.g.

   cd data/seq
   mkdir foobar
   cd foobar
   ln -s <location of directory of sequence files> sequence
   ln -s <location of directory of annotation files> annotation
   

3. Use the project parameter window to enter the location of the sequences and optional annotation.

The file results of the alignment and synteny computations are is as follows:

   /data/seq_results/<project-name1>-to-<project-name2>/align
   /data/seq_results/<project-name1>-to-<project-name2>/final

The log files are in the /logs directory, see Running MUMmer for more details.
 

External programs, MUMmer and FPC

External programs

MUMmer with SyMAP details

1. It has the FPC demo files.
2. It has BLAT³ in the /ext directory.
3. It has the tar file doc.tar.gz of the documentation.
4. The AGCoL documentation applies to this release. See AGCoL System Guide.
5. This is the last release made from AGCoL, so the documentation will stay consistent.
6. It is also available from Github.

• Put symap_5.tar.gz in a permanent location and untar it.
• Replace the /data and symap.config from your previous SyMAP location to this new location.
• This approach is safest as it acquires all changes (e.g. scripts) except for changes to the demo files.

• Put symap_5.tar.gz in a temporary location and untar it.
• Move symap_5/java/jar/symap.jar to the java/jar location of your permanent SyMAP.
• Check to see if there are any /scripts or /ext changes that need to also be copied over.

The following table shows what versions require action by the user.

1. Always get the new java/jar/symap.jar.

2. The Alignment&Synteny will use existing MUMmer files if they have not been removed.

3. To update to the latest Gene# assignment, it depends on what version you have:

   v5.1.2 to v5.1.7	Execute ./symap -z, then select Reload Annotation; the synteny algorithm does NOT need to be re-run.
   pre-v5.1.2	Delete database, Load Project and run Alignment&Synteny. The database does not have to be deleted, but its cleaner to do so.

² Marcais, G., A.L. Delcher, A.M. Phillippy, R. Coston, S.L. Salzberg, A. Zimin (2018). MUMmer4: A fast and versatile genome alignment system, PLoS computational biology, 14(1): e1005944.

³ Kent, J. (2002) BLAT--the BLAST-like alignment tool, Genome Research 12:656-64.

⁴ Krzywinski, M., J. Schein, I. Birol, J. Connors, R. Gascoyne, D. Horsman, S. Jones, M. Marra (2009). Circos: An information aesthetic for comparative genomics. Genome Research doi:10.1101/gr.092759.109.

⁵ Soderlund, C., Nelson, W., Shoemaker, A., and Paterson, A.(2006). SyMAP: A system for discovering and viewing syntenic regions of FPC maps. Genome Res. 16:1159-1168.

⁶ Soderlund, C., Bomhoff, M., and Nelson, W. (2011). SyMAP: A turnkey synteny system with application to multiple large duplicated plant sequenced genomes. Nucleic Acids Res V39, issue 10, e68.

⁷ Edgar, R (2004). MUSCLE: a multiple sequence alignment method with reduced time and space complexity. BMC Bioinformatics 113.

⁸ Soderlund, C., S. Humphrey, A. Dunhum, and L. French (2000). Contigs built with fingerprints, markers and FPC V4.7. Genome Research 10:1772-1787.

⁹ Engler, F., J. Hatfield, W. Nelson, and C. Soderlund (2003). Locating sequence on FPC maps and selecting a minimal tiling path. Genome Research 13:2152:2163.