/doc/sphinx_doc/tuto.rst - Diff - NucleoMiner - Forge du Centre Blaise Pascal

Révision e5603c3f doc/sphinx_doc/tuto.rst

     Tutorial
     ========
     This tutorial describes steps allowing to perform quantitave analysis of
     nucleosomal epigenome. We assume that files are organised around a given
     hierarchie and that all command lines are launched from project's root.
     This tutorial describes steps allowing performing quantitative analysis of epigenetic marks on individual nucleosomes. We assume that files are organised according to a given hierarchy and that all command lines are launched from the project’s root directory.
     This tutorial is divided into t=wo main parts. First one consists in the python
     script `wf.py` that aligns and convert Illumina reads. Second one is the R
     script `main.r` that extracts information (nucleosome position and indicators)
     from the dataset.
     This tutorial is divided into two main parts. The first part covers the python script `wf.py` that aligns and converts short sequence reads. The second part covers the R scripts that extracts information (nucleosome position and indicators) from the dataset.
     Python and R Common Configuration File
     --------------------------------------
     First of all we define in one place some configuration variables that will be launch by python and R scripts. This file is **configurator.py**. The execution of this python script dumps variables into the **nucleo_miner_config.json** that will be launch by both kind of scriopts (R and puython).
     To do this launch at the root of your project the following command line:
     .. code:: bash
       python src/current/configurator.py
     Experimental Dataset, Working Directory and Configuration File
     --------------------------------------------------------------
     $$$ other python script to describe:
     - libcoverage.py
     - wf.py
     Working Directory Organisation
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     The working directory...
     $$$ TODO: Explain how and where retrieve the workdir
     Dataset and Configuration Variables
     -----------------------------------
     Retrieving Experimental Dataset
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     We want to compare nucleosomes of 3 yeast strains:
     The MNase-seq and MN-ChIP-seq raw data are available at ArrayExpress (http://www.ebi.ac.uk/arrayexpress/) under accession number E-MTAB-2671.
     - BY
     - RM
     - YJM
     $$$ TODO explain how organise Experimental Dataset into the `data` directory of the working directory.
     For each strain we perform Mnase-Seq and ChIP-Seq using the 5 following
     markers:
     - H3K4me1
     - H3K4me3
     - H3K9ac
     - H3K14ac
     - H4K12ac
     We want to compare nucleosomes of 2 yeast strains: BY and RM. For each strain we performed Mnase-Seq and ChIP-Seq using an antibody recognizing the H3K14ac epigenetic mark.
     In order to simplify the design of experiment, we considere Mnase as a marker.
     For each couple `(strain, marker)` we perform 3 replicates. So, theoritically
     we should have `3 * (1 + 5) * 3 = 54` samples. In practice we only obtain 2
     replicates for `(YJM, H3K4me1)`. Each one of the 53 samples is indentify by a
     uniq identifier. The file `CSV_SAMPLE_FILE` sums up this information.
     The dataset is composed of 55 files organised as follows:
     .. autodata:: configurator.CSV_SAMPLE_FILE
         :noindex:
     We use a convention to link sample and Illumina fastq outputs. Illumina output
     files of the sample `ID` will be stored in the directory
     `ILLUMINA_OUTPUTFILE_PREFIX` + `ID`. For example, sample 41 outputs will be
     stored in the directory `data/2012-09-05/FASTQ/Sample_Yvert_Bq41/`.
     .. autodata:: configurator.ILLUMINA_OUTPUTFILE_PREFIX
         :noindex:
     For BY (resp. RM and YJM) we use following reference genome
     `saccharomyces_cerevisiae_BY_S288c_chromosomes.fasta`
     (resp. `saccharomyces_cerevisiae_rm11-1a_1_supercontigs.fasta` and
     `saccharomyces_cerevisiae_YJM_789_screencontig.fasta`).
     The index `FASTA_REFERENCE_GENOME_FILES` stores this information.
       - 3 replicates for BY MNase Seq
         - sample 1 (5 fastq.gz files)
         - sample 2 (5 fastq.gz files)
         - sample 3 (4 fastq.gz files)
       - 3 replicates for RM MNase Seq
         - sample 4 (4 fastq.gz files)
         - sample 5 (4 fastq.gz files)
         - sample 6 (5 fastq.gz files)
       - 3 replicates for BY ChIP Seq H3K14ac
         - sample 36 (5 fastq.gz files)
         - sample 37 (5 fastq.gz files)
         - sample 53 (9 fastq.gz files)
       - 2 replicates for RM ChIP Seq H3K14ac
         - sample 38 (5 fastq.gz files)
         - sample 39 (4 fastq.gz files)
     .. autodata:: configurator.FASTA_REFERENCE_GENOME_FILES
         :noindex:
     Each chromosome/contig is identify in the fasta file by an obscure identifier.
     For example, BY chromosome I is identify by `gi|144228165|ref|NC_001133.7|` when
     TemplateFilter is waiting for an integer. So, we translate it. The index
     `FASTA_INDEXES` stores this translation.
     .. autodata:: configurator.FASTA_INDEXES
         :noindex:
     Python and R Common Configuration File
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     From a pragamatical point of view we discard some part of the genome (repeated
     sequence etc...). The list of the black listed area is explicitely detailled in
     `AREA_BLACK_LIST`.
     First of all we define in one place some configuration variables that will be launched by python and R scripts. These variables are contained in file `configurator.py`. The execution of this python script dumps variables into the `nucleominer_config.json` file that will then be used by both R and python scripts.
     .. autodata:: configurator.AREA_BLACK_LIST
         :noindex:
     To do this, go to the root directory of your project and run the following command:
     For BY-RM (resp. BY-YJM and RM-YJM) genome sequence alignment we use previously
     compute .c2c file `data/2012-03_primarydata/BY_RM_gxcomp.c2c` (resp.
     `BY_YJM_GComp_All.c2c` and `RM_YJM_gxcomp.c2c`). For more information about
     .c2c files, please read section 5 of the manual of `NucleoMiner`, the old
     version of `NucleoMiner2`
     (http://www.ens-lyon.fr/LBMC/gisv/NucleoMiner_Manual/manual.pdf).
     .. code:: bash
     .. autodata:: configurator.C2C_FILES
         :noindex:
       python src/current/configurator.py
     `nucleominer` uses specific directory to work in, these are described in
     `INDEX_DIR`, `ALIGN_DIR` and `LOG_DIR`.
     Finally, `nucleominer` use external ressources, the path to these resspources
     are describe in `BOWTIE_BUILD_BIN`, `BOWTIE2_BIN`, `SAMTOOLS_BIN`,
     `BEDTOOLS_BIN` and `TF_BIN` and `TF_TEMPLATES_FILE`.
     All paths, prefixes and indexes could be change in the
     `src/current/nucleominer_config.json` file.
     .. autodata:: wf.json_conf_file
         :noindex:
     Preprocessing Illumina Fastq Reads for Each Sample
     --------------------------------------------------
     This preprocessing step consists in the 4 main steps embed in the `wf.py` and
     described bellow. As a preamble, this script computes `samples` `samples_mnase`
     and `strains` that will be used along the 4 steps.
     This preprocessing step consists of 4 main steps embedded in the `wf.py` script. They are described bellow. As a preamble, this script computes `samples`, `samples_mnase` and `strains` that will be used along the 4 steps.
     .. autodata:: wf.samples
         :noindex:
-...
     Creating Bowtie Index from each Reference Genome
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     For each strain, we need to create bowtie index. Bowtie index of a strain is a
     tree view of the genemoe reference for this strain. It will be used by
     bowtie to align reads. This step is performed by the following part of the
     `wf.py` script:
     For each strain, we need to create bowtie index. Bowtie index of a strain is a tree view of the genome of this strain. It will be used by bowtie to align reads. This step is performed by the following part of the `wf.py` script:
     .. literalinclude:: ../../../snep/src/current/wf.py
        :start-after: # _STARTOF_ step_1
        :end-before: # _ENDOF_ step_1
        :language: python
     The following table sum up involved file sizes and process durations concerning
     this step.
     The following table summarizes the file sizes and process durations concerning this step.
     ======  ======================  ======================  ================
     strain  fasta genome file size  bowtie index file size  process duration
     ======  ======================  ======================  ================
     BY      12 Mo                          25 Mo                    11 s.
     RM      12 Mo                          24 Mo                    9 s.
     YJM     12 Mo                          25 Mo                    11 s.
     ======  ======================  ======================  ================
-...
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     Next, we launch bowtie to align reads to the reference genome. It produces a
     `.sam` file that we convert into a `.bed` file. Binaries for `bowtie`, `samtools`
     and `bedtools` are wrapped using python `subprocess` class. This step is
     performed by the followinw part of the `wf.py` script:
     `.sam` file that we convert into a `.bed` file. Binaries for `bowtie`, `samtools` and `bedtools` are wrapped using python `subprocess` class. This step is performed by the following part of the `wf.py` script:
     .. literalinclude:: ../../../snep/src/current/wf.py
        :start-after: # _STARTOF_ step_2
        :end-before: # _ENDOF_ step_2
        :language: python
     Convert Aligned Reads for TemplateFilter
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     TemplateFilter use particular input format for reads, so we convert `.bed` file.
     TemplateFilter expect reads as following: `chr coord strand #read` where:
     Convert Aligned Reads into TemplateFilter Format
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     TemplateFilter uses particular input formats for reads, so it is necessary to convert the `.bed` files. TemplateFilter expect reads as follows: `chr`, `coord`, `strand` and `#read` where:
     - chr is the number of the chromosome;
     - coord is the coordinate of the reads;
     - strand is `F` for forward and `R` for reverse;
     - #reads the number of reads for this position.
     - `chr` is the number of the chromosome;
     - `coord` is the coordinate of the reads;
     - `strand` is `F` for forward and `R` for reverse;
     - `#reads` the number of reads covering this position.
     Each entry is *tab*-separated.
     **WARNING** for reverse strand bowtie returns the position of left first
     nucleotid when TemplateFilter is waiting for right one. So this step takes it
     into account and add lenght of reads (in our case 50) to reverse reads
     coordinate.
     **WARNING** for reverse strands, bowtie returns the position of the first nucleotide on the left hand side, whereas TemplateFilter expects the first one on the right hand side.  This step takes this into account by adding the read length (in our case 50) to the reverse reads coordinates.
     This step is performed by the followinw part of the `wf.py` script:
     This step is performed by the following part of the `wf.py` script:
     .. literalinclude:: ../../../snep/src/current/wf.py
        :start-after: # _STARTOF_ step_3
        :end-before: # _ENDOF_ step_3
        :language: python
     The following table sum up number of reads, involved file sizes and process durations concerning
     the two last steps. In our case, aligment process have been multuthreaded over over 3 cores.
     The following table summarises the number of reads, the involved file sizes and process durations concerning the two last steps. In our case, alignment process have been multithreaded over 3 cores.
     ==  ==============  =========================  ======  ================  ==================  ================
     id  Illumina reads  aligned and filtred reads  ratio   `.bed` file size  TF input file size  process duration
-...
 13765584        10381903                   75,42%  931  Mo           59  Mo              352   s.
 15168268        11502855                   75,83%  1031 Mo           64  Mo              386   s.
 18850820        14024905                   74,40%  1254 Mo           69  Mo              482   s.
 15591124        12126623                   77,78%  1163 Mo           72  Mo              405   s.
 15659905        12475664                   79,67%  1194 Mo           71  Mo              416   s.
 14668641        10960565                   74,72%  1052 Mo           70  Mo              375   s.
 14339179        10454451                   72,91%  1049 Mo           51  Mo              363   s.
 18019895        13688774                   75,96%  1378 Mo           59  Mo              474   s.
 13746796        10810022                   78,64%  1084 Mo           54  Mo              360   s.
 15205065        11766016                   77,38%  990  Mo           54  Mo              381   s.
 17803097        13838883                   77,73%  1154 Mo           60  Mo              452   s.
 15434564        12307878                   79,74%  1032 Mo           57  Mo              394   s.
 16802587        12725665                   75,74%  1221 Mo           48  Mo              438   s.
 16058417        12513734                   77,93%  1192 Mo           63  Mo              422   s.
 16154482        13204331                   81,74%  1277 Mo           52  Mo              430   s.
 21013924        17102120                   81,38%  1646 Mo           59  Mo              555   s.
 17213114        14433357                   83,85%  1389 Mo           53  Mo              459   s.
 17360907        14733001                   84,86%  1203 Mo           55  Mo              450   s.
 18136816        15389581                   84,85%  1257 Mo           53  Mo              469   s.
 14763678        12173025                   82,45%  1140 Mo           56  Mo              393   s.
 15541709        12890345                   82,94%  1057 Mo           48  Mo              398   s.
 16433215        13094314                   79,68%  1241 Mo           57  Mo              433   s.
 17370850        14264136                   82,12%  1347 Mo           51  Mo              466   s.
 14613512        8654495                    59,22%  887  Mo           56  Mo              339   s.
 15248545        11367589                   74,55%  1166 Mo           67  Mo              405   s.
 14316809        10767926                   75,21%  1103 Mo           63  Mo              379   s.
 15178058        12265794                   80,81%  1030 Mo           66  Mo              390   s.
 14968579        11876186                   79,34%  1009 Mo           63  Mo              387   s.
 16912705        13550508                   80,12%  1143 Mo           70  Mo              442   s.
 16782154        12755111                   76,00%  1227 Mo           65  Mo              438   s.
 16741443        13168071                   78,66%  1260 Mo           71  Mo              442   s.
 13096171        10367041                   79,16%  992  Mo           62  Mo              350   s.
 17715118        14092985                   79,55%  1404 Mo           68  Mo              483   s.
 17288466        7402082                    42,82%  741  Mo           48  Mo              339   s.
 16116394        13178457                   81,77%  1101 Mo           63  Mo              420   s.
 14241106        10537228                   73,99%  880  Mo           57  Mo              348   s.
 13784738        10598464                   76,89%  1005 Mo           64  Mo              358   s.
 12438007        9620975                    77,35%  911  Mo           60  Mo              326   s.
 13853959        11031238                   79,63%  1045 Mo           64  Mo              365   s.
 12036162        6654780                    55,29%  684  Mo           46  Mo              268   s.
 13873129        10251074                   73,89%  1048 Mo           61  Mo              365   s.
 19817751        14904502                   75,21%  1520 Mo           72  Mo              528   s.
 13368959        10818619                   80,92%  912  Mo           63  Mo              350   s.
 7566467         6139001                    81,13%  520  Mo           44  Mo              201   s.
 32586928        21191363                   65,03%  1816 Mo           82  Mo              766   s.
 30733184        18791373                   61,14%  1801 Mo           89  Mo              721   s.
 41287616        30383875                   73,59%  2911 Mo           112 Mo              1065  s.
 40439965        31177914                   77,10%  2981 Mo           117 Mo              1070  s.
 40876476        33780065                   82,64%  3316 Mo           103 Mo              1165  s.
 52424414        47117107                   89,88%  3811 Mo           119 Mo              1477  s.
     ==  ==============  =========================  ======  ================  ==================  ================
     For some reasons (manipulation efficiency, e.g. PCR...), we remove samples 33, 45, 48 and 55.
     Run TemplateFilter on Mnase Samples
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     Finally, for each sample we perfome TemplateFilter analysis.
     Finally, for each sample we perform TemplateFilter analysis.
     **WARNING** TemplateFilter returns a list of nucleosomes. Each nucleosome is
     define by its center and its width. An odd width leads us to considere non
     interger lower and upper bound.
     define by its center and its width. An odd width leads us to consider non-
     integer lower and upper bound.
     **WARNING** TemplateFilter is not design to deal with replicate. So we choose to
     keep a maximum of nucleosome and filter in a second time using the benefit of
     replicate. To do that we set a low correlation threshold parameter (`0.5`) and a
     particularly high value of overlaping (`300%`).
     **WARNING** TemplateFilter is not designed to deal with replicates. So we recommend to keep a maximum of nucleosomes and filter the aberrant ones afterwards using the benefits of having replicates. To do this, we set a low correlation threshold parameter (0.5) and a particularly high value of overlap (300%).
     This step is performed by the followinw part of the `wf.py` script:
     This step is performed by the following part of the `wf.py` script:
     .. literalinclude:: ../../../snep/src/current/wf.py
        :start-after: # _STARTOF_ step_4
-...
 RM     88076       62 Mo          984 s.
 RM     90141       64 Mo          967 s.
 RM     87517       62 Mo          980 s.
 YJM    88945       64 Mo          566 s.
 YJM    88689       64 Mo          570 s.
 YJM    88128       63 Mo          565 s.
     ==  ======  ==========  =============  ================
-...
     ..
     ..
     .. $$$ other python script to describe:
     .. - libcoverage.py
     .. - wf.py
     ..
     ..
     ..
     ..
     ..
     ..
     .. In order to simplify the design of experiment, we consider Mnase as a marker.
     .. For each couple `(strain, marker)` we perform 3 replicates. So, theoritically
     .. we should have `3 * (1 + 5) * 3 = 54` samples. In practice we only obtain 2
     .. replicates for `(YJM, H3K4me1)`. Each one of the 53 samples is indentify by a
     .. uniq identifier. The file `CSV_SAMPLE_FILE` sums up this information.
     ..
     .. .. autodata:: configurator.CSV_SAMPLE_FILE
     ..     :noindex:
     ..
     .. We use a convention to link sample and Illumina fastq outputs. Illumina output
     .. files of the sample `ID` will be stored in the directory
     .. `ILLUMINA_OUTPUTFILE_PREFIX` + `ID`. For example, sample 41 outputs will be
     .. stored in the directory `data/2012-09-05/FASTQ/Sample_Yvert_Bq41/`.
     ..
     .. .. autodata:: configurator.ILLUMINA_OUTPUTFILE_PREFIX
     ..     :noindex:
     ..
     .. For BY (resp. RM and YJM) we use following reference genome
     .. `saccharomyces_cerevisiae_BY_S288c_chromosomes.fasta`
     .. (resp. `saccharomyces_cerevisiae_rm11-1a_1_supercontigs.fasta` and
     .. `saccharomyces_cerevisiae_YJM_789_screencontig.fasta`).
     .. The index `FASTA_REFERENCE_GENOME_FILES` stores this information.
     ..
     .. .. autodata:: configurator.FASTA_REFERENCE_GENOME_FILES
     ..     :noindex:
     ..
     .. Each chromosome/contig is identify in the fasta file by an obscure identifier.
     .. For example, BY chromosome I is identify by `gi|144228165|ref|NC_001133.7|` when
     .. TemplateFilter is waiting for an integer. So, we translate it. The index
     .. `FASTA_INDEXES` stores this translation.
     ..
     .. .. autodata:: configurator.FASTA_INDEXES
     ..     :noindex:
     ..
     .. From a pragamatical point of view we discard some part of the genome (repeated
     .. sequence etc...). The list of the black listed area is explicitely detailled in
     .. `AREA_BLACK_LIST`.
     ..
     .. .. autodata:: configurator.AREA_BLACK_LIST
     ..     :noindex:
     ..
     .. For BY-RM (resp. BY-YJM and RM-YJM) genome sequence alignment we use previously
     .. compute .c2c file `data/2012-03_primarydata/BY_RM_gxcomp.c2c` (resp.
     .. `BY_YJM_GComp_All.c2c` and `RM_YJM_gxcomp.c2c`). For more information about
     .. .c2c files, please read section 5 of the manual of `NucleoMiner`, the old
     .. version of `NucleoMiner2`
     .. (http://www.ens-lyon.fr/LBMC/gisv/NucleoMiner_Manual/manual.pdf).
     ..
     .. .. autodata:: configurator.C2C_FILES
     ..     :noindex:
     ..
     .. `nucleominer` uses specific directory to work in, these are described in
     .. `INDEX_DIR`, `ALIGN_DIR` and `LOG_DIR`.
     ..
     .. Finally, `nucleominer` use external ressources, the path to these resspources
     .. are describe in `BOWTIE_BUILD_BIN`, `BOWTIE2_BIN`, `SAMTOOLS_BIN`,
     .. `BEDTOOLS_BIN` and `TF_BIN` and `TF_TEMPLATES_FILE`.
     ..
     .. All paths, prefixes and indexes could be change in the
     .. `src/current/nucleominer_config.json` file.
     ..
     .. .. autodata:: wf.json_conf_file
     ..     :noindex:
     ..
     Inferring Nucleosome Position and Extracting Read Counts
     --------------------------------------------------------
     The second part of the tutorial uses `R` (http://http://www.r-project.org). It consists in a set of R scripts taht will be sourced in an R console launched at the root of your project. the R srcipts are:
     The second part of the tutorial uses R (http://http://www.r-project.org). It consists of a set of R scripts that will be sourced in an R from a console launched at the root of your project. These scripts are:
       - headers.R
       - extract_maps.R
       - compare_common_wp.R
       - translate_common_wp.R
       - split_samples.R
       - count_reads.R
       - get_size_factors
-...
     The Script headers.R
     ^^^^^^^^^^^^^^^^^^^^
     The script header.R is included in each other scripts. It is in charge of:
     The script headers.R is included in each other scripts. It is in charge of:
       - launching libraries used in thes scripts
       - launching libraries used in the scripts
       - launching configuration (design, strain, marker...)
       - computing and caching CURs
       - computing and caching CURs (caching means storing the information in the computer's memory)
     Note that you can customize the function “translate”. This function allows you to use the alignments between genomes when performing various tasks. You may be using NucleoMiner2 to analyse data of a single strain, or of several strains.
       - All the data corresponds to the same strain (e.g. treatment/control, or only few mutations): Then in step 1), the  regions to use are entire chromosomes. Instep 2) simply use the default translate function which is neutral.
       - The data come from two or more strains: In this case, edit a list of regions and customize the translate function which performs the correspondence between the different genomes. How we did it: a .c2c file is obtained with NucleoMiner 1.0 (refer to the Appendice "Generate .c2c Files"), then use it to produce the list of regions and customise “translate”.
     In your R console, run the following command line:
     .. code:: bash
       R CMD BATCH src/current/header.R
       source("src/current/headers.R")
     The Script extract_maps.R
     ^^^^^^^^^^^^^^^^^^^^^^^^^
     This script is in charge of extracting Maps for well positioned and fuzzy nucleosomes. First of all, this script computed intra and inter strain nucleosome maps for each CUR. This step is executed in parallel on many cores using the BoT library. Next, it collects results and produces well positioned, fuzzy and UNR maps.
     This script is in charge of extracting Maps for well-positioned and fuzzy nucleosomes. First of all, this script computes intra and inter-strain nucleosome maps for each CUR. This step is executed in parallel on many cores using the BoT library. Next, it collects results and produces well-positioned, fuzzy and UNR maps.
     The well-positioned map for BY is collected in the result directory and is called **BY_wp.tab**. It is composed of following columns:
     The well-positioned map for BY is collected in the result directory and is called `BY_wp.tab`. It is composed of following columns:
      - chr, the number of the chromosome
      - lower_bound, the lower bound of the nucleosome
      - upper_bound, the upper bound of the nucleosome
      - cur_index, index of the CUR
      - index_nuc, the index of the nucleosome in the CUR
      - wp, 1 if it is a well positioned nucleosome, 0 else
      - nb_reads, the number of reads that supports this nucleosome
      - nb_nucs, the number of TemplateFilter nucleosome across replicates (= the number of replicates if it is a well-positioned nucleosome)
      - llr_1, for a well-positioned nucleosome, it is the LLR1 between the first and the second TemplateFilter nucleosome.
      - llr_2, for a well-positioned nucleosome, it is the LLR1 between the second and the first TemplateFilter nucleosome.
      - wp_llr, for a well-positioned nucleosome, it is the LLR2 overall TemplateFilter nucleosomes.
      - wp_pval, for a well-positioned nucleosome, it is the p-value chi square test obtained with the LLR2 (**1-pchisq(2.LLR2, df=4)**)
      - dyad_shift, for a well-positioned nucleosome, it is shift between the two extreme TemplateFilter nucleosome dyad positions.
      - wp, 1 if it is a well positioned nucleosome, 0 otherwise
      - nb_reads, the number of reads that support this nucleosome
      - nb_nucs, the number of TemplateFilter nucleosome across replicates (= the number of replicates in which it is a well-positioned nucleosome)
      - llr_1, for a well-positioned nucleosome, it is the LLR1 (log-likelihood ratio) between the first and the second TemplateFilter nucleosome on the chain.
      - llr_2, for a well-positioned nucleosome, it is the LLR1 between the second and the third TemplateFilter nucleosome on the chain.
      - wp_llr, for a well-positioned nucleosome, it is the LLR2 that compares consistency of the positioning over all TemplateFilter nucleosomes.
      - wp_pval, for a well-positioned nucleosome, it is the p-value chi square test obtained with the LLR2 (`1-pchisq(2.LLR2, df=4)`)
      - dyad_shift, for a well-positioned nucleosome, it is the shift between the two extreme TemplateFilter nucleosome dyad positions.
     The fuzzy map for BY is collected in the result directory and is called **BY_fuzzy.tab**. It is composed of following columns:
     The fuzzy map for BY is collected in the result directory and is called `BY_fuzzy.tab`. It is composed of following columns:
      - chr, the number of the chromosome
      - lower_bound, the lower bound of the nucleosome
      - upper_bound, the upper bound of the nucleosome
      - cur_index, index of the CUR
     The common well-position map for BY and RM strains is collected in the result directory and is called **BY_RM_common_wp.tab**. It is composed of following columns:
     The map of common well-positioned nucleosomes aligned between the BY and RM strains is collected in the result directory and is called `BY_RM_common_wp.tab`. It is composed of following columns:
      - cur_index, the index of the CUR
      - index_nuc_BY, the index of the BY nucleosome in the CUR
      - index_nuc_RM,the index of the RM nucleosome in the CUR
      - llr_score, the LLR3 score between th eBy and RM nucleosomes
      - common_wp_pval,  the p-value chi square test obtained with the LLR3 (**1-pchisq(2.LLR3, df=2)**)
      - index_nuc_RM, the index of the RM nucleosome in the CUR
      - llr_score, , the LLR3 score that estimates conservation between the positions in BY and RM
      - common_wp_pval,  the p-value chi square test obtained from LLR3 (`1-pchisq(2.LLR3, df=2)`)
      - diff, the dyads shift between the positions in the two strains
     The common UNR map for BY and RM strains is collected in the result directory and is called **BY_RM_common_unr.tab**. It is composed of following columns:
     The common UNR map for BY and RM strains is collected in the result directory and is called `BY_RM_common_unr.tab`. It is composed of the following columns:
      - cur_index, the index of the CUR
      - index_nuc_BY, the index of the BY nucleosome in the CUR
      - index_nuc_RM,the index of the RM nucleosome in the CUR
     To execute this script, run the following command line in your R console:
     To execute this script, run the following command in your R console:
     .. code:: bash
       source("src/current/extract_maps.R")
     The Script compare_common_wp.R
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     This script is used to compare inter strain distances between common well-positioned nucleosomes.
     For example, it compute the file **BY_RM_common_wp_diff.tab** that contains dyad shifts between two well-positioned nucleosomes. It is composed of following columns:
      - cur_index, the index of the CUR
      - index_nuc_BY, the index of the BY nucleosome in the CUR
      - index_nuc_RM,the index of the RM nucleosome in the CUR
      - llr_score, the LLR3 score between th eBy and RM nucleosomes
      - common_wp_pval,  the p-value chi square test obtained with the LLR3 (**1-pchisq(2.LLR3, df=2)**)
      - diff, the dyad shifts between two well-positioned nucleosomes
     The Script translate_common_wp.R
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     It also translates well-positioned nucleosome maps from a strain to an other strain and stores it into a table.
     This script is used to translate common well-positioned nucleosome maps from a strain to another strain and stores it into a table.
     For example, the file **results/2014-04/RM_wp_tr_2_BY.tab** contains RM well-positioned nucleosome translated into the BY genome referential. It is composed of following columns:
     For example, the file `results/2014-04/RM_wp_tr_2_BY.tab` contains RM well-positioned nucleosome translated into the BY genome coordinates. It is composed of following columns:
      - strain_ref, the reference genome (in which positioned are defined)
      - begin, the translated lower bound of the nucleosome
      - end, the translated upper bound of the nucleosome
      - chr, the number of chromosome for the reference genome (in which positioned are defined)
      - chr, the number of chromosomes for the reference genome (in which positioned are defined)
      - length, the length of the nucleosome (could be negative)
      - cur_index, the index of the CUR
      - index_nuc, the index of the nucleosome in the CUR
     To execute this script, run the following command in your R console:
     .. code:: bash
     To execute this script, run the following command line in your R console:
       source("src/current/translate_common_wp.R")
     .. code:: bash
       R CMD BATCH src/current/compare_common_wp.R
     The Script split_samples.R
     ^^^^^^^^^^^^^^^^^^^^^^^^^^
     For memory space usage reasons, we split and compress TemplateFilter input files according to their corresponding  chromosome. for example, `sample_1_TF.tab` will be split into :
     Split and Compress Samples According CURs
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       - sample_1_chr_1_splited_sample.tab.gz
       - sample_1_chr_2_splited_sample.tab.gz
       - ...
       - sample_1_chr_17_splited_sample.tab.gz
     To execute this script, run the following command in your R console:
     .. code:: bash
       R CMD BATCH src/current/split_samples.R
       source("src/current/split_samples.R")
     Count Reads for Each Nucleosome
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     The Script count_reads.R
     ^^^^^^^^^^^^^^^^^^^^^^^^
     To associate a number of observations (read) to each nucleosome we run the script `count_reads.R`. It produces the files `BY_RM_H3K14ac_wp_and_nbreads.tab`, `BY_RM_H3K14ac_unr_and_nbreads.tab` `BY_RM_Mnase_Seq_wp_and_nbreads.tab` and `BY_RM_Mnase_Seq_unr_and_nbreads.tab`
     for H3K14ac common well-positioned nucleosomes, H3K14ac UNRs, Mnase common well-positioned nucleosomes and Mnase UNRs respectively.
     For example, the file `BY_RM_H3K14ac_unr_and_nbreads.tab` contains counted reads for well-positioned nucleosomes with the experimental condition ChIP H3K14ac. It is composed of the following columns:
       - chr_BY, the number of the chromosome for BY
       - lower_bound_BY, the lower bound of the nucleosome for BY
       - upper_bound_BY, the upper bound of the nucleosome  for BY
       - index_nuc_BY, the index of the BY nucleosome in the CUR for BY
       - chr_RM, the number of the chromosome for RM
       - lower_bound_RM, the lower bound of the nucleosome for RM
       - upper_bound_RM, the upper bound of the nucleosome  for RM
       - index_nuc_RM,the index of the RM nucleosome in the CUR for RM
       - cur_index, index of the CUR
       - BY_H3K14ac_36, the number of reads for the current nucleosome for the sample 36
       - BY_H3K14ac_37, #reads for sample 37
       - BY_H3K14ac_53, #reads for sample 53
       - RM_H3K14ac_38, #reads for sample 38
       - RM_H3K14ac_39, #reads for sample 39
     To execute this script, run the following command in your R console:
     .. code:: bash
       R CMD BATCH src/current/count_reads.R
       source("src/current/count_reads.R")
     The Script get_size_factors.R
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     This script uses the DESeq function `estimateSizeFactors` to compute the size factor of each sample. It corresponds to normalisation of read counts from sample to sample, as determined by DESeq. When a sample has n reads for a nucleosome or a UNR,
     the normalised count is n/f where f is the factor contained in this file.
     The script dumps computed size factors into the file `size_factors.tab`. This file has the form:
     ========= ======= ======= =======
     sample_id      wp     unr   wpunr
     ========= ======= ======= =======
 0.87396 0.88097 0.87584
 1.07890 1.07440 1.07760
 1.06400 1.05890 1.06250
 0.85782 0.87948 0.86305
 0.97577 0.96590 0.97307
 1.19630 1.18120 1.19190
 0.93318 0.92762 0.93166
 0.48315 0.48453 0.48350
 1.11240 1.11210 1.11230
 0.89897 0.89917 0.89903
 2.22650 2.22700 2.22660
     ========= ======= ======= =======
     sample_id are given in file samples.csv
     If you don't know which column to use, we recommend using wpunr.
     Get Size Factors Using DESeq
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     If you want the very detailed factors produced by DESeq, here are the information:
       - unr: factor computed from data of UNR regions. These regions are defined for every pairs of aligned genomes (e.g. BY_RM)
       - wp: same, but for well-positioned nucleosomes.
       - wpunr: both types of regions.
     To execute this script, run the following command in your R console:
     .. code:: bash
       R CMD BATCH src/current/get_size_factors.R
       source("src/current/get_size_factors.R")
     Performing DESeq Analysis
     The Script launch_deseq.R
     ^^^^^^^^^^^^^^^^^^^^^^^^^
     Finally, the script `launch_deseq.R` perform statistical analysis on each nucleosome using `DESeq`. It produces files:
       - results/current/BY_RM_H3K14ac_wp_snep.tab
       - results/current/BY_RM_H3K14ac_unr_snep.tab
       - results/current/BY_RM_H3K14ac_wpunr_snep.tab
       - results/current/BY_RM_H3K14ac_wp_mnase.tab
       - results/current/BY_RM_H3K14ac_unr_mnase.tab
       - results/current/BY_RM_H3K14ac_wpunr_mnase.tab
     These files are organised with the following columns (see file `BY_RM_H3K14ac_wp_snep.tab` for an example):
       - chr_BY, the number of the chromosome for BY
       - lower_bound_BY, the lower bound of the nucleosome for BY
       - upper_bound_BY, the upper bound of the nucleosome  for BY
       - index_nuc_BY, the index of the BY nucleosome in the CUR for BY
       - chr_RM, the number of the chromosome for RM
       - lower_bound_RM, the lower bound of the nucleosome for RM
       - upper_bound_RM, the upper bound of the nucleosome  for RM
       - index_nuc_RM,the index of the RM nucleosome in the CUR for RM
       - cur_index, index of the CUR
       - form
       - BY_Mnase_Seq_1, the number of reads for the current nucleosome for the sample 1
     Next columns concern indicators for each sample:
       - BY_Mnase_Seq_2, #reads for sample 2
       - BY_Mnase_Seq_3, #reads for sample 3
       - RM_Mnase_Seq_4, #reads for sample 4
       - RM_Mnase_Seq_5, #reads for sample 5
       - RM_Mnase_Seq_6, #reads for sample 6
       - BY_H3K14ac_36, #reads for sample 36
       - BY_H3K14ac_37, #reads for sample 37
       - BY_H3K14ac_53, #reads for sample 53
       - RM_H3K14ac_38, #reads for sample 38
       - RM_H3K14ac_39, #reads for sample 39
     The 5 last columns concern DESeq analysis:
       - manip[a_manip] strain[a_strain] manip[a_strain]:strain[a_strain], the manip (marker) effect, the strain effect and the snep effect. These are the coefficients of the fitted generalized linear model.
       - pvalsGLM, the pvalue resulting of the comparison of the GLM model considering or not the interaction term marker:strain. This is the statsitcial significance of the interaction term and therefore the statistical significance of the SNEP.
       - snep_index, a boolean set to TRUE if the pvalueGLM value is under the threshold computed with FDR function with a rate set to 0.0001.
     To execute this script, run the following command
     To execute this script, run the following command in your R console:
     .. code:: bash
       R CMD BATCH src/current/launch_deseq.R
       source("src/current/launch_deseq.R")
     Results
     -------
     Results: Number of SNEPs
     ------------------------
     Output Files Organisation
     ^^^^^^^^^^^^^^^^^^^^^^^^^
     Previous steps produce following 45 files.
     Each filename is under the form
     Here are the number of computed SNEPs for each forms.
     .. code:: bash
     ===== ======= ===== =======
      form strains #nucs H3K14ac
     ===== ======= ===== =======
        wp   BY-RM 30464    3549
       unr   BY-RM  9497    1559
     wpunr   BY-RM 39961    5240
     ===== ======= ===== =======
       results/current/[combi]_[marker]_[form]_snep.tab
     Where combi is in {BY_RM, BY_YJM, RM_YJM} for each strain combination, marker is
     in {H3K4me1, H3K4me3, H3K9ac, H3K14ac, H4K12ac} for each post translational
     histone modification and form is in {wp, fuzzy, wpfuzzy} considering well
     positioned nucleosomes, fuzzy nucleosomes or both for SNEP computation.
     chr_BY lower_bound_BY upper_bound_BY index_nuc_BY chr_RM lower_bound_RM upper_bound_RM index_nuc_RM roi_index form BY_Mnase_Seq_1 BY_Mnase_Seq_2 BY_Mnase_Seq_3 RM_Mnase_Seq_4 RM_Mnase_Seq_5 RM_Mnase_Seq_6 BY_H3K14ac_36
     BY_H3K14ac_37 BY_H3K14ac_53 RM_H3K14ac_38 RM_H3K14ac_39 pvalsGLM
     APPENDICE: Generate .c2c Files
     ------------------------------
     For each file, there is 1 line per nucleosome and each line is composed of many columns divided into 3 main topics:
       - nuc information
       - number opf reads for each sample
       - DESeq analysis results.
     The `.c2c` files is a simple table that describes how the genome sequence can be aligned. We generate it using NucleoMiner 1.0.
     For exemple for the file *BY_RM_H3K14ac_wp_snep.tab* informations are:
       - chr_BY, the BY chr involved
       - lower_bound_BY, the lower bound of the BY nuc
       - upper_bound_BY, the upper_bound of the BY nuc
       - index_nuc_BY, the index of the nuc in the entire list of BY nucs
       - chr_RM, lower_bound_RM, upper_bound_RM, index_nuc_RM
     	are the same information for the RM strain
       - roi_index, the index of the region of interrest involved.
     Next cols concern indicators for each sample. They are labeled [strain]_[marker]_[sample_id] and each value represents the number of reads for the current nuc for the sample *sample_id*.
     To install NucleoMiner 1.0 on your UNIX/LINUX computer you need first to install the Genetic Data analysis Library (GDL), which is a dynamic library of useful C functions derived from the GNU Scientific Library.
     The 5 final columns concern DESeq analysis:
       - manip[a_manip] strain[a_strain] manip[a_strain]:strain[a_strain], the manip (marker) effect, the strain effect and the snep effect.
       - pvalsGLM, the pvalue resulting of the comparison of the GLM model considering or the interaction term *marker:strain*
       - snep_index, a boolean set to TRUE if the *pvalueGLM* value is under the threshold computed with FDR function with a rate set to 0.01%.
     Installing the GDL library
     ^^^^^^^^^^^^^^^^^^^^^^^^^^
     It also produces the file that explicts size factor for each involved sample in differents strain combination and nucleosomal region type:
     Get the gdl-1.0.tar.gz archive on your computer (in the directory deps of your working directory). Copy it in a dedicated directory. Go into this directory using the cd command, and then unfold the archive by typing:
     TODO: include this file... /home/filleton/analyses/snepcatalog/data/2013-10-09/current/README.txt
     tar -xvzf gdl-1.0.tar.gz
     This creates a directory called gdl-1.0. You now need to go into this directory and compile the library, by typing:
     .. code:: bash
       results/current/size_factors.tab
       cd gdl-1.0
       ./configure
       make
     Now you need to install the library on your system. This needs root priviledges:
     .. code:: bash
       sudo make install
     Installing NucleoMiner
     ^^^^^^^^^^^^^^^^^^^^^^
     Get the nucleominer-1.0.tar.gz archive on your computer. Copy it in a dedicated directory. Go into this directory using the cd command, and then unfold the archive by typing:
     tar -xvzf nucleominer-1.0.tar.gz
     This creates a directory called nucleominer-1.0. You now need to go into this directory and compile the library, by typing:
     .. code:: bash
     Number of SNEPs
     ^^^^^^^^^^^^^^^
       cd nucleominer-1.0
       ./configure
       make
     Here are the number of computed for each forms.
     You can then use the binaries dircetly from this folder (best then is to add the path to this folder in your PATH environment variable). If you want to install nucleominer at the system's level (useful if mutiple users will need it) then type, with root priviledges:
     .. code:: bash
       [1] "wp"
              #nucs H3K4me1 H3K4me3 H3K9ac H3K14ac H4K12ac
       BY-RM  30234     520     798     83    3566      26
       BY-YJM 31298     303     619    102     103     128
       RM-YJM 29863     129     340     46    3177      18
       [1] "fuzzy"
              #nucs H3K4me1 H3K4me3 H3K9ac H3K14ac H4K12ac
       BY-RM  10748     294     308    101    1681      42
       BY-YJM 10669     122     176    124      93      87
       RM-YJM 11478      54     112     41    1389      20
       [1] "wpfuzzy"
              #nucs H3K4me1 H3K4me3 H3K9ac H3K14ac H4K12ac
       BY-RM  40982     770    1136    183    5404      73
       BY-YJM 41967     439     804    214     198     199
       RM-YJM 41341     184     468     87    4687      37
     TODO:
       - Print/study intra/inter strain LODs.
       - Check the normality of sample using Shapiro–Wilk (Hypothesis for computing LODs)
       sudo make install
     Generate .c2c Files
     ^^^^^^^^^^^^^^^^^^^
     To generate .c2c files you need to type the following command in a terminal:
     .. code:: bash
       mkdir dir_4_c2c
       NMgxcomp Data/BY_S288c/Sequence/Genome.fasta \
                Data/RM_11-1a/Sequence/Genome.fasta \
                dir_4_c2c/BY_RM 2>dir_4_c2c/BY_RM.log
     After execution, the directory dir_4_c2c will hold the .c2c files.

Formats disponibles : Unified diff

LBMC » NucleoMiner

Révision e5603c3f doc/sphinx_doc/tuto.rst