NCBI supplies FASTA formatted files for genome sequence and GFF3 formatted files for the annotation, where FASTA and GFF3 files are the input to SyMAP. However, using them directly can cause problems. The following provides a simple scheme to produce only the files necessary.

Contents

Download

1. Go to NCBI.
2. As shown in Fig 1:
   1. Select "Genome" from the pull-down at the top.
   2. Enter you genome name followed by "Search". You should see a page similar to Fig 2.
3. The following are two approaches to download the FASTA and GFF files:
   1. As shown in Fig 2:
      1. Use the genome link beside the "Download sequences in FASTA format for genome,..."
      2. Use the GFF link beside the "Download genome annotation in GFF...".
      3. Process as described in Convert A.
   2. As shown in Fig 2 and Fig 3:
      1. Go to the NCBI Datasets page (link at bottom of Fig 2). This will bring up a window as shown in the background of Fig 3.
      2. Select the assembly you want, then click on the Action dots, which brings up a menu; select Download. That brings up a window as shown in Fig 3.
      3. Select the file source, then genome FASTA and annotation GFF for download.
      4. Process as described in Convert B.

If NCBI only provides the Genbank format, try using bioconvert to convert to GFF3, followed by the SyMAP converter. Alternatively, try Galaxy.

Fig 1. Search the NCBI site.

Fig 2. Download A: download the genome and GFF files.

Fig 3. Download B: from NCBI Datasets, select the genome and GFF files.

ConvertNCBI optional flags

Beware: There are variations in the text associated with the FASTA ">" header lines. The rules used by this script are as follows:

1. If the ">" line contains "chromosome N", where N={number, X, Y or roman numeral}, than it is identified as such.
2. If the ">" line contains "scaffold" or "contig", than it is identified as a scaffold (if "chromosome" is also on the line, it is still treated as a scaffold).
3. All other ">" entries are considered "other".

   zgrep ">" [prefix]._genomic.fna.gz

Convert files from FASTA and GFF (download A)

1. Go to the symap_5/data/seq directory.
2. Make a subdirectory for your species and move the FASTA and GFF files into the directory.
3. From the seq directory, type the following at the command line to copy the ConvertNCBI script:

   cp ../../scripts/ConvertNCBI*.class .
   chmod 755 *.class
   

4. Execute

   java ConvertNCBI <species>

Example

> cd data/seq
> mkdir riceA
> cd riceA
> mv ~/Download/GCF_001433935.1_IRGSP-1.0_genomic.fna.gz .
> mv ~/Download/GCF_001433935.1_IRGSP-1.0_genomic.gff.gz .
> cd ..
> cp ../../scripts/ConvertNCBI*.class .
> chmod 755 *.class
> java ConvertNCBI riceA

data/seq
	ConvertNCBI.class
	ConvertNCBI$Gene.class
data/seq/riceA/
      GCF_001433935.1_IRGSP-1.0_genomic.fna.gz
      GCF_001433935.1_IRGSP-1.0_genomic.gff.gz
      annotation/
         anno.gff
         gap.gff
      sequence/
         genomic.fna

Convert files from ncbi_dataset.zip (download B)

Follow all steps from Download A, except for step 2, do the following:

2. Make a subdirectory for your species and move the ncbi_dataset.zip file to the species directory and unzip it.

Example

> cd data/seq
> mkdir riceB
> cd riceB
> mv ~/Download/ncbi_dataset.zip .
unzip ncbi_dataset.zip
Archive:  ncbi_dataset.zip
  inflating: README.md
  inflating: ncbi_dataset/data/assembly_data_report.jsonl
  inflating: ncbi_dataset/data/GCA_001433935.1/chr1.fna
  inflating: ncbi_dataset/data/GCA_001433935.1/chr2.fna
  ...
  inflating: ncbi_dataset/data/GCA_001433935.1/genomic.gff
  inflating: ncbi_dataset/data/GCA_001433935.1/sequence_report.jsonl
  inflating: ncbi_dataset/data/dataset_catalog.json

> cd ..
> cp ../../scripts/ConvertNCBI*.class .
> chmod 755 *.class
> java ConvertNCBI riceB

data/seq
	ConvertNCBI.class
	ConvertNCBI$Gene.class
data/seq/riceB/
	ncbi_dataset.zip
	ncbi_dataset/data/GCA_001433935.1/
		chr1.fna
		...
	annotation/
		anno.gff
		gap.gff
	sequence/
		genomic.fna

java ConvertNCBI rice -s

1. grp_prefix needs to be blank as there is no common prefix now.
2. min_size should be set to only load the largest scaffolds. To determine the value to use, run the lenFasta.pl script, e.g. from the seq directory and using rice as an example:

   cp ../../scripts/lenFasta.pl .
   perl lenFasta.pl riceA/sequence/genomic.fna
   

Read riceA/sequence/genomic.fna and print sorted lengths
Read 55 sequences
   N    Length  Seqid
   1  43270923  >C01 NC_029256.1
   2  36413819  >C03 NC_029258.1
   3  35937250  >C02 NC_029257.1
...
Values for min_len (assuming no duplicate lengths):
#Seqs  min_len
   10 27531856
   20    19457
   30    11447
   40    10311
   50     7140

Once you make your changes, execute:

   javac ConvertNCBI.java

• Reads the file ending in '.fna.gz' (or '.fna') and writes a new file called sequence/genomic.fna with the following changes:
  1. Sequences must have the word "chromosome" in their ">" header line in order to be copied (unless -l or -s flags).
  2. The header line is replaced with ">ChrN", where N comes from the header line "chromosome N", e.g. "chromosome 1", etc. For example,

     >NC_029256.1 Oryza sativa Japonica Group cultivar Nipponbare chromosome 1, IRGSP-1.0
     

     is replaced with:

     >Chr1  NC_029256.1
     

  3. Gaps of >30,000 are written to the annotation/gap.gff file (30,000 is hard-coded in ConvertNCBI script).
  
   
• Reads the file ending in 'gff.gz' (or .gff) and writes the file annotation/anno.gff. The gff3 format has 9 columns, where the first is the 'seqid', the third is the 'type' (e.g. feature 'gene'), the last column is a semicolon-delimited keyword=value attribute list. The input file is processed as follows:
  1. The type=gene with attribute gene-biotype=protein-coding are processed. The gene line is written to the anno.gff file with the following changes:
     1. The first column 'seqid' is replace with the 'ChrN' value assigned when reading the '.fna' file.
     2. A subset of the attributes are written, where the product keyword value is modified as follows:
        1. If there are multiple mRNA lines for a gene where the values are different, they are concatenated together.
        2. If there are multiple mRNA lines for a gene where the only difference is the variant, then only the variant difference is show, e.g.

           product=monocopper oxidase-like protein SKU5%2C transcript variant X2, X1, X3
           

  2. The first type=mRNA line for a gene is written to the anno.gff file followed by its type=exon, where they each are written with the new seqid and a subset of the attributes.
      
  3. If the "-p" flag is set, then the ID= from the first type=CDS for the gene is extracted, and the "cds-" removed. From the files I have tested, this name corresponds with the ">" name in the NCBI protein file. Likewise for the "-pa" flags, but all unique ID's from the gene's type=CDS records are made into a comma-delimited list; note, this can cause a long list.

Go to top