THIS PAGE IS UNDER DEVELOPMENT

Deliverables & Milestones

The major deliverables are staged releases of the system.

The VDFS pipeline and science archive deliverables are designed around a series of incremental releases driven by the long term science requirements from VISTA using the WFCAM test bed as a prototype system. (Milestones are listed in Appendix D.4).

VDFS-v1 to v3 will operate on WFCAM data only

v4 onwards will operate on both VISTA and WFCAM data.

v3, released 6 months before VDFS-VISTA commissioning of v4 will contain the functionality needed for VISTA and v4 will effectively be a thoroughly debugged version of v3.

Various parts of the ESO deliverables sections of VDFS-v4 are delivered at various dates given in the Milestones section at Appendix D.4.

Scalability tests for v3 and v4 will be carried out by re-processing the complete WFCAM raw data archive (which should be 20 TB by mid 2005. i.e. equivalent to 6 weeks of VISTA data) because it is expected that the successive versions of the VDFS pipeline will increase the quality of the reduced frames and the parameters of the sources detected from them. 2004 observations processed with v1 of the pipeline, and 2005 observations processed with v2, will therefore be re-processed with v3 of the pipeline, planned to be completed by the end of 2005. This will be done in Q2-Q3 of 2006, in parallel with processing of the 2006 observations. This five-fold increase in the volume of data needing processing will itself be an excellent test of the scalability requirement for VISTA data. Similarly, the transfer to the Science archive at WFAU and archiving of these re-processed data in 2006 will test the scalability of data transfer and ingestion to volumes expected from VISTA. Although it might seem simpler and "cleaner" to overwrite the data from earlier versions of the pipeline, we recognise that this would be unacceptable to many users and that it will be necessary to hold both original and re-processed data. The resources required for the additional data storage are included in this bid (see Appendix B.4 for details).

Below we outline some of the major increments to science functionality that will be built up in each release, for both the Pipeline and Science Archive whose development have been aligned so that the major releases occur concurrently.

6.2 VDFS Pipeline (v4 includes ESO Deliverables)

The successful CDR for the v1 pipeline occurred in Oct 2002.

VDFS-Pipeline-v1(WFCAM) Delivered end Q4 2003; ready for WFCAM commissioning

pipeline content
· standard 2D processing, standard catalogue products and DQC at summit and in UK, with some tuning during commissioning and science verification
· further processing would include trialling PSF generation methods and investigating options 1,2 for PSF fitting work
· software for master catalogue generation and co-located list-driven photometry
· stacker and mosaicer, straightforward algorithms with ability to deal correctly with, scaling, confidence maps, spurion rejection, and accurate WCS alignment
· ability to do simple continuum subtraction for narrow band (eg. H2) work
· investigate alternative interpolation methods to assess which methods gives the best results from the point of view of deriving image parameters

VDFS-Pipeline-v2(WFCAM) released end Q4 2004

pipeline content
· full assessment and use of PSF fitting methods (options 1,2)
· development of single Sersic 1D/2D profile fits for galaxy modelling
· software for co-located photometry from externally derived lists
· in stacking/mosaicing optimal use of varying PSF information from both image detection and image resolution points of view
· software for more general purpose (ie. with adaptive kernel matching) difference imaging cf. continuum subtraction in Version 1.
· Low surface brightness(LSB) re-detections using fixed large detection filters having previously removed standard image detections including assessment of feasibility of LSB work in NIR mosaic data.

VDFS-Pipeline-v3(WFCAM) released end Q4 2005

pipleine content
· software for performing cross-photometry on external 2D datasets
· detection of transient phenomena, adaptive kernel matching difference imaging, including re-running image detection algorithms
· full iterative PSF modelling (option 3) for the further processing stage
· in stacking/mosaicing complete image restoration using all data contemporaneously
· await results from LSB detection work from v2. before attempting more general methods

VDFS-Pipleine-v4 released Q4 2006 (ESO deliverables in Q3) to operate on VISTA data
· as for v3 except handles the much greater VISTA data rates

VDFS-Pipeline-v5 released end Q2 2008 final release for VISTA after 18 months shakedown period
· incorporates changes from experience with running actual VISTA data, and completion of interfaces to the VO

Some of the more advanced pipeline functionalities will also be provided to WFAU where they will also operate, where appropriate, on the contents of the Science Archive.

6.3 VDFS Science Archive

The CDR for the v1 Science Archive occurred in April 2003 where the design was generally approved.

VDFS-v1(WFCAM) Delivered end Q4 2003; ready for WFCAM commissioning
Will allow access to the basic data products, will have a web-based interface, and documentation. It will include access to 2MASS, SSS, SDSS-DR1, and USNO-B catalogues as well as WFCAM data. It will allow the user to :
· Browse the database of calibrated observation frames and associated catalogues, and download any one of these
· Produce small images and associated multi-colour merged catalogues, from either the observation frame collection, or the accumulating UKIDSS surveys
· Produce mosaic images on demand up a width of 0.8 degrees
· Search UKIDSS surveys on a position rectangle or circle in Celestial or Galactic co-ordinates, and within a radius of a resolvable source name
· Allow joint position searches as above, and produce merged catalogues, with the combination of UKIDSS plus any of 2MASS, SSS, SDSS-DR1
· Make SQL-like queries in any sensible arithmetic combination of the catalogue parameters, for example colours
· Allow general queries on the combination of UKIDSS, 2MASS, SSS, SDSS-DR1

VDFS-v2(WFCAM) released end Q4 2004
Will be compliant with any relevant protocols and formats produced as part of the various Virtual Observatory initiatives, and will work especially closely with AstroGrid. It will allow the user to :
· Search open-time data as well as UKIDSS data
· Plot returned parameters in histograms
· Plot pairs of returned parameters as (x,y) graphs
· Generate on request images and catalogues across observation frame boundaries
· Generate mosaic and stacked images and source catalogues from any area, from user specified observation frames
· Choose a range of stacking and source extraction algorithms on request, with a menu of tunable parameters
Probably other simple user tools will be available, such as the SED tool developed for the AVO first-light demo with the GOODS dataset. v2 will also :
· Return catalogues and similar data in VOTable format, and images using binary data access protocols still under development
· Recast all the V1 services as web services for publication in the AstroGrid Registry

VDFS-v3(WFCAM) released end Q4 2005
· Develop (or install) a suite of advanced visualisation tools, e.g. pannable large area imaging, multi-dimensional catalogue plotting and rotation
· Develop (or install) a suite of data analysis tools, e.g. cluster analysis, principal component analysis, etc
· Implement both the above as server-based tools
· Develop a system to allow uploadable user-specified analysis algorithms
· Recast our web services as grid services, in order to allow queries and analysis using shared managed resources with other data centres

VDFS-v4 released Q4 2006 (ESO deliverables in Q3) to operate on VISTA data
· as for v3 except handles the much greater VISTA data rates

VDFS-v5 released end Q2 2008 final release for VISTA after 18 months shakedown period
· incorporates changes from experience with running actual VISTA data, and completion of interfaces to the VO