Data Repository tools and Data Upload module: Difference between pages

From MIPAV
(Difference between pages)
Jump to navigation Jump to search
m (1 revision imported)
 
MIPAV>Olga Vovk
mNo edit summary
 
Line 1: Line 1:
In order to help researches to upload and download data to the data repository, BRICS provides the Data Repository tool that includes the following  modules (or sub-tools):
The Data Upload module helps researches to upload their data to the data repository. The data should be uploaded in the form of a submission package (XML) that has a unique identifier - a submission ticket (XML).  
# [[#ImagingSubmission|Imaging data submission and validation]] module should be used for imaging data to create the image submission package.
# [[#DataValidation| Data Validation]] module verifies that data conforms to the required format and range values defined in the data dictionary. It also creates a data submission package ([http://en.wikipedia.org/wiki/XML XML]) and submission ticket ([http://en.wikipedia.org/wiki/XML XML]) that can be uploaded to the data repository via the [[#DataUpload| Data Upload]] module.
# [[#DataUpload| Data Upload]] module assists researches in uploading their data to the data repository (in the form of a submission package and submission ticket).
# [[#DataDownload| Data Download]] module assists researches in downloading data from the data repository.


== Steps in data preparation, validation, submission, and download ==
'''If you don't have a submission package ready''', use the [[Data Validation module]] to create it. The [[Data Validation module|module]] will also validate your data and make sure that they conform to the required format and range values defined in the data dictionary.
'''For [[Image submission plug-in| imaging data]]:'''
# Create and pre-validate the image submission package using the [[#ImagingSubmission|Imaging data submission and validation]] module.
# Validate the the image submission package using the [[#DataValidation| Data Validation]] module and create the data submission package/ticket using the same module.
# Use the data submission ticket from the previous step to upload the data submission package with the help of the [[#DataUpload| Data Upload]] module.
# Download the data using the means provided by the  [[#DataDownload| Data Download]] module.


'''For non-imaging data:'''
'''If you need to submit [[Image submission plug-in|imaging data]] to the repository''',
# Create and validate the data submission package using the [[#DataValidation| Data Validation]] module and create the submission ticket using the same module (creates automatically as soon as data pass validation).
# Use the data submission ticket from the previous step to upload the data submission package with the help of the [[#DataUpload| Data Upload]] module.
# Download the data using the means provided by the  [[#DataDownload| Data Download]] module.


== Data for upload ==
# Use the [[Image submission plug-in|Imaging data submission and validation module]] to create [[Image submission plug-in#Submission package|the image submission set]].  
<div id="ImagingData"><div>
# Use the [[Data Validation module]] to re-validate the data and create [[Data Validation module#Submission package|a submission ticket and submission package]].
=== Imaging data ===
Imaging data can be uploaded in the form of a brain image file and a corresponding [http://en.wikipedia.org/wiki/Comma-separated_values CSV] file that contains some additional patient/subject/visit information (not stored in the image header) as well as image related metadata. See also [[Image submission plug-in#Required image information|Required image information]].


The following information is required for all [[Image submission plug-in|image submissions]] and must be included in a CSV file:
Read more about [[Data Repository tools]].
# The patient/subject information including  - GUID, a patient/subject age, a study site name, and a visit date;
# The image information including - imaging study date and time, imaging file name, imaging file itself, imaging file format and modality, image QA/QC information.


The brain image file can be:  
'''See also:'''
* a single file in one of [[Supported Formats| supported formats]],
*[[Data Repository tools| Data Repository tools]]
* a [[Other formats supported by MIPAV#DicomFormat|DICOM multifile]] or some other multifile in one of the supported formats,
*[[Image submission plug-in|Imaging data submission and validation]] module
* a ZIP archive that contains an image dataset (e.g. [[Other formats supported by MIPAV#DicomFormat|DICOM dataset]] with images of multiple slices stored in separate image files - [[Image submission plug-in#Multifiles|a multifile]]).
*[[Data Upload module| Data Upload]] module
*[[Data Download module| Data Download]] module


For more information refer to: [[Image submission plug-in|Imaging data submission and validation module]].
== System requirements ==
The most recent version of [http://java.com/en/download/index.jsp Java Runtime Environment (JRE)] (6 or 7) is required in order to run the Data Upload module.


=== Clinical  data ===
== Module input and output ==
In order to upload clinical data to the data repository, the data should be submitted to the [[#DataValidation|Data Validation module]] as [[Data Validation module#CSVFiles|CSV]] files (in tab-delimited format). 
 
The [[#DataValidation|Data Validation module]] accepts [[Data Validation module#CSVFiles|CSV]] files from a researcher and validates the files' content against the values defined in the data dictionary. For  those data that pass validation, the [[#DataValidation| Data Validation module]] creates a submission package and submission ticket both in XML format.
 
The submission ticket is used by the [[#DataUpload| Data Upload module]] to upload the data (in the form of a corresponding submission package) to the data repository.
 
For more information about the structure of CSV files, data submission and validation, refer to [[Data Validation module]].
 
=== Genomics data ===
 
TBD.
 
<div id="ImagingSubmission"><div>
== Imaging data submission and validation module ==
The Imaging Data Submission and Validation module (also known as  MIPAV Image Submission and Validation tool or MIPAV [[Image submission plug-in]]) is designed to help researches to validate and submit their data. Data validation is a necessary step that must be done prior data submission in order to ensure the quality of uploaded data and to make data queryable. The module runs as a Java Web Start application locally on a user's computer (Java runtime environment is required).


'''Module input:'''
'''Module input:'''
# a brain image(s) in one of supported formats,
# A submission package and submission ticket (XML) from the [[Data Validation module]].
# a corresponding CSV file with metadata, see [[#ImagingData| Imaging data]].


'''Module output:'''
'''Module output:'''
# the brain image(s) in one of supported formats,
# Data submitted to the data repository.
# the CSV file confirmed against the data dictionary and ready for validation by the [[#DataValidation|Data Validation module]],
# [[Image submission plug-in#Output log|the Output log]] that lists all brain image files and CSV files added to the image submission package. It also displays the path(s) to the directory where the image package(s) is stored. 
 
Read [[Image submission plug-in| more]]...
 
<div id="DataValidation"><div>
 
== Data Validation module ==
The [[Data Validation module]] assists researchers with the submission of both imaging and non-imaging data into the repository. The Data Validation module verifies that data conforms to the required format and range values defined in the data dictionary (note that for the [[Image submission plug-in|imaging data]] this is going to be a second validation needed to ensure the quality of data). The [[Data Validation module]] validates the metadata associated with the data files identified by the user for submission against the data dictionary. If everything is OK, the Data Validation module creates a submission ticket and submission package. After that data are good for uploading. If errors are found, the module provides a detailed report of any data discrepancies, errors, and warnings. The module runs as a Java Web Start application locally on a user's computer (Java runtime environment is required).
 
'''Module input:'''
# CSV file with clinical data or [[#ImagingData|imaging metadata]].
 
'''Module output:'''
# a submission package and submission ticket (XML) ready for submission by the [[#DataUpload| Data Upload module]].
# [[Data Validation module#Error log|an error log]] with validation errors and warnings (if any).
 
Read [[Data Validation module|more]] ...


<div id="DataUpload"><div>
== Running the Data Upload module ==
== Data Upload module ==
[[File:DataUploadAgreement.png|200px|thumb|left|The data privacy user agreement]]
After the submission package has been created using either the [[Image submission plug-in|Imaging Data Submission and Validation module]] (for imaging data) or by the [[Data Validation module]] (for non-imaging data), data can be submitted to the data repository. The Data Upload module facilitates this process. The module runs as a Java Web Start application locally on a user's computer (Java runtime environment is required).
[[File:DataUploadUploadManagerWindow.png|200px|thumb|left|The Upload Manager window]]
The Data Upload module runs locally on your machine. In order to launch the module, navigate to the Data Repository > Upload Data page and click the Launch the Upload Tool link.
'''Note:''' the most recent version of [http://java.com/en/download/index.jsp Java Runtime Environment (JRE)] (6 or 7) is required in order to run the module. Make sure your computer has it installed.


'''Module input:'''
* Click Launch the Upload Tool. In the Opening uploadTool.jnlp window that appears, select Open with Java(TM) Web Start Launcher (default) and click OK. In the Java Runtime Environment window that appears next saying "Do you want to run this application?", click Run.
# a submission package and submission ticket (XML) from the [[#DataValidation|Data Validation module]].
* The EULE Agreement window appears displaying the data privacy user agreement. Read the agreement and click Accept if you agree.
* The Upload Manager window appears.
'''In the Upload Manager window,'''
## Use the drop-down Study Name menu to select the study name. Use the Refresh button to update the list of  studies.
## Navigate to the Submission Ticket (XML) box and use the Browse button to select the submission ticket file (XML).
## Navigate to the Dataset Name text box and type in a unique name for your dataset. The dataset name must be unique to the selected study. Make it a meaningful name - an easy one to search for.
## Press Start Submission Upload. Data upload begins and the progress bar appears next to the uploading file name.
## The submission package appears in the Upload Queue table. In the Upload Queue table you can watch the progress of your submission(s). The table will update as file(s) are being uploaded to the system. For successful upload(s), the Status (see the Status column) will be designated as "Completed".
* If you need to cancel your submission, press Cancel.
* To clear the completed submissions list, use the Clear Completed Submissions button located at the bottom of the Upload Manager window.
* To clear the cancelled submissions  list, use the Clear  Cancelled Submissions button located at the bottom of the Upload Manager window.
* To load pending submissions, use the Load Pending Submissions button located at the bottom of the Upload Manager window.


'''Module output:'''
=== Where to see uploaded data? ===
# Data submitted to the data repository.
After submitting the data, to make sure that your dataset appears under the study you selected:
# Navigate to the Data Repository > View Studies page.
# Find your study on the study table. Note 3 icons located in the Data Types column. If your study has any data submitted, at least one of the icons appears [[#DataTypes|in color]].
# Select the study and click on the study name to open the Study Overview page.
# On the study page, click on the "+" sign next to Dataset Submissions.
# The table that contains all submitted datasets opens. Make sure that your dataset is listed in this table.


Read [[Data Upload module| more]]...
<div id="DataTypes"><div>
'''The data types associated with a study are represented by three icons:'''
[[File:DataUploadStudy.png|350px|thumb|left|This study contains 2 types of data - clinical assessment data and imaging data. Genomics data are not presented in the study]]  
* Double helix - represents genomics data;
* Stethoscope - represents clinical assessment data;
* Head profile - represents imaging data.
   
If the icons next to the study name are highlighted in color, the study has datasets of the highlighted types.


<div id="DataDownload"><div>
=== Notes ===
== Data Download module ==
* You can only upload data to the studies you have the data upload permissions.
Data Download module helps users to select and download datasets from the data repository to their own systems. The module runs as a Java Web Start application locally on a user's computer (Java runtime environment are required).
* To make sure that the most recent list of studies is available for you, use the Refresh button to update the list of studies.
* The name assigned to the uploading dataset must be unique for the selected study.
* If you have any questions, please contact the operations team - TBD.


'''Module input:'''
== See also ==
# Data submitted to the repository by the [[#DataUpload|Data Upload module]]..
*[[Data Repository tools|Data Repository tools]]
*[[Data Validation module]]
*[[Image submission plug-in|Imaging data submission and validation module]]
*[[Data Download module]]


'''Module output:'''
# Data downloaded to your computer- TBD.


Read [[Data Download module|more]]...


[[Category:Help]]
[[Category:Help:Stub]]
[[Category:BRICS]]
[[Category:BRICS]]

Revision as of 13:04, 29 July 2013

The Data Upload module helps researches to upload their data to the data repository. The data should be uploaded in the form of a submission package (XML) that has a unique identifier - a submission ticket (XML).

If you don't have a submission package ready, use the Data Validation module to create it. The module will also validate your data and make sure that they conform to the required format and range values defined in the data dictionary.

If you need to submit imaging data to the repository,

  1. Use the Imaging data submission and validation module to create the image submission set.
  2. Use the Data Validation module to re-validate the data and create a submission ticket and submission package.

Read more about Data Repository tools.

See also:

System requirements

The most recent version of Java Runtime Environment (JRE) (6 or 7) is required in order to run the Data Upload module.

Module input and output

Module input:

  1. A submission package and submission ticket (XML) from the Data Validation module.

Module output:

  1. Data submitted to the data repository.

Running the Data Upload module

The data privacy user agreement
The Upload Manager window

The Data Upload module runs locally on your machine. In order to launch the module, navigate to the Data Repository > Upload Data page and click the Launch the Upload Tool link.

Note: the most recent version of Java Runtime Environment (JRE) (6 or 7) is required in order to run the module. Make sure your computer has it installed.

  • Click Launch the Upload Tool. In the Opening uploadTool.jnlp window that appears, select Open with Java(TM) Web Start Launcher (default) and click OK. In the Java Runtime Environment window that appears next saying "Do you want to run this application?", click Run.
  • The EULE Agreement window appears displaying the data privacy user agreement. Read the agreement and click Accept if you agree.
  • The Upload Manager window appears.

In the Upload Manager window,

    1. Use the drop-down Study Name menu to select the study name. Use the Refresh button to update the list of studies.
    2. Navigate to the Submission Ticket (XML) box and use the Browse button to select the submission ticket file (XML).
    3. Navigate to the Dataset Name text box and type in a unique name for your dataset. The dataset name must be unique to the selected study. Make it a meaningful name - an easy one to search for.
    4. Press Start Submission Upload. Data upload begins and the progress bar appears next to the uploading file name.
    5. The submission package appears in the Upload Queue table. In the Upload Queue table you can watch the progress of your submission(s). The table will update as file(s) are being uploaded to the system. For successful upload(s), the Status (see the Status column) will be designated as "Completed".
  • If you need to cancel your submission, press Cancel.
  • To clear the completed submissions list, use the Clear Completed Submissions button located at the bottom of the Upload Manager window.
  • To clear the cancelled submissions list, use the Clear Cancelled Submissions button located at the bottom of the Upload Manager window.
  • To load pending submissions, use the Load Pending Submissions button located at the bottom of the Upload Manager window.

Where to see uploaded data?

After submitting the data, to make sure that your dataset appears under the study you selected:

  1. Navigate to the Data Repository > View Studies page.
  2. Find your study on the study table. Note 3 icons located in the Data Types column. If your study has any data submitted, at least one of the icons appears in color.
  3. Select the study and click on the study name to open the Study Overview page.
  4. On the study page, click on the "+" sign next to Dataset Submissions.
  5. The table that contains all submitted datasets opens. Make sure that your dataset is listed in this table.

The data types associated with a study are represented by three icons:

This study contains 2 types of data - clinical assessment data and imaging data. Genomics data are not presented in the study
  • Double helix - represents genomics data;
  • Stethoscope - represents clinical assessment data;
  • Head profile - represents imaging data.

If the icons next to the study name are highlighted in color, the study has datasets of the highlighted types.

Notes

  • You can only upload data to the studies you have the data upload permissions.
  • To make sure that the most recent list of studies is available for you, use the Refresh button to update the list of studies.
  • The name assigned to the uploading dataset must be unique for the selected study.
  • If you have any questions, please contact the operations team - TBD.

See also

Retrieved from "https://mipav.cit.nih.gov/pubwiki/index.php?title=Data_Repository_tools&oldid=107"