Proposal for 2010 Fast Track review

The response was straightforward and clear and added some valuable information; however, the proponents only partially answered the ISRP’s queries. The revised proposal did not deliver the information we were interested in obtaining regarding two issues: the CV (coefficient of variation) issue and the derivation (natural origin vs. hatchery) of the adults contributing to the redd count trends given in Figures 1-7. A list of the ISS streams x treatment and control was provided that indicated which streams were potential intensive and extensive monitoring sites for the coordinated anadromous salmon BiOp monitoring. However there was no explanation of the process or timeline for selection of which populations are actually going to be incorporated into the monitoring framework. The answer regarding the precision of the current estimates with reference to the 15% CV standard was not adequate. The ISRP was anticipating information on which locations actually met this standard using the current protocol, which could likely meet it with increased effort, and locations where the precision was not attainable. The precision of the existing data was not discussed. The ISS proponents can address the CV issue as an addendum in the upcoming RME Categorical Review. While the ISS study design was developed before the NOAA CV recommendation was made, the recommendation is an important step in helping to standardize methods and data collection within the region. With regard to comparing natural and hatchery contributions, Table(s) 3 and 4 summarized the types of adult (Table 3) and juvenile (Table 4) data being collected at each location, but did not actually provide numbers or an indication that the information was suitable for analysis using the anticipated ISS analysis framework. Similarly, Figures 1 through 7 provide estimates of redds in various treatment and control streams, but do not partition the observations by adult source (natural/wild, supplementation adult, conventional adult). Therefore, little could be learned from the tables and figures at this stage in terms of identifying differing performance among treatment and control groups. Consequently, any insights into the interpretation of the observations will need to wait until more thorough reports are prepared. The ISRP should review preliminary draft analyses of reports from the ISS project as they are developed in the next few years. The ISS study has a long history of iterative reviews with the ISRP that include reviews of potential changes to the design. Thus, reviewers were concerned that two treatment streams (Lolo and Newsome Creeks) were removed from the Phase III treatment portion of the study and because outplantings are still occurring in these streams, they are now classified as being in the Phase I and II parts of the study. The study design previously reviewed by the ISRP called for all Phase II outplanting treatments to end and for all treatment streams to enter the Phase III treatment as a test of the supplementation model. The proponents note that the power analysis of the Phase III groups for detection power conducted as part of the 2002 proposal review process (Lutch et al 2003) showed that up to 25% of the treatment streams could be removed from the analysis without effecting detection power. On this basis, it would seem that the original study objectives can be met; however, no biological justification was provided for the change in the treatment status of Lolo and Newsome Creeks. The action is also not consistent with the stated goal (p. 6) of the proposal: “The primary focus of this proposal is to maintain the ISS study design and take the program to completion, test the hypotheses, and make supplementation recommendations.” Finally, the proponents have removed the objective of evaluating the AHA model and HSRG assumptions from the proposal and a placeholder has been requested. If this objective is pursued in the future, the evaluation procedure should be reviewed before beginning.

The fast track element for Method #8 to install a second screw trap on Marsh Creek is sufficiently justified. For the remainder of the proposal, including the fast-track element to validate AHA modeling, a response is requested in the form of a revised narrative. The response should provide the following: 1. A table that outlines the ESU, MPG, Independent Populations, and streams in the Snake River system and identify which are potential high-precision and low-precision sites for RME. 2. A summary explanation of what process is underway (if any) to decide which component streams are part of the intensive and extensive sampling. 3. Greater detail of explanation for the precision/sampling intervals for intensive and extensive sites. 4. An overview-to-date of trends in ISS spring/summer Chinook abundance by location for all treatment and control (reference) streams. Include a summary table of the data collected for each of the sites by the ISS (or cooperators) since the last ISRP review. 5. Comparison of the precision and sampling intervals in the ISS streams with that desired by the BiOp RME and ISS statistical analysis. 6. Statements for the ISRP about any events or problems encountered since the last review that may compromise the analysis of the ISS. 7. Elaborate on the methods and approach to validate the assumptions and uncertainties in the AHA model In concluding, the ISRP suggests that the final report should be independently peer reviewed when in draft form, much like what was done with the 10-year retrospective for the comparative survival studies (ISAB/ISRP 2007-6). 1. Technical Justification, Program Significance and Consistency, and Project Relationships Justification for this project is well documented in the proposal and in the Council's current and past program language. The project has many links and relations—and large implications—for many other projects in the basin. The proposal clearly identifies relationships to the 2008 BiOp, TRTs recovery plan, Columbia Basin regional RM&E strategy, NPCC Fish and Wildlife Program, and WY-KAN-USH-MI WA-KISH-WIT. As the proposal states, the Idaho Supplementation Studies (ISS) study design was intensively scrutinized and updated prior to the last funding cycle. This proposal "represents the ongoing efforts of the cooperating agencies in the ISS program to take that study design to completion without change" which is a significant and necessary pledge. The objectives of the program are to evaluate the effects of supplementation on juvenile and adult Chinook abundance; evaluate changes in natural productivity after supplementation is stopped; evaluate various supplementation strategies; and develop supplementation recommendations. These objectives seem important, relevant and consistent with policies and with biological and policy needs. The key feature of this project is that it uses supplementation as a research treatment. Monitoring production and productivity responses for supplemented populations and unsupplemented reference populations could provide important insights into the effects of supplementation. Continued monitoring after supplementation ceases could contribute to a better understanding of the long-term effects of supplementation. At the completion of the project, status and trend monitoring currently provided by ISS will need to be incorporated into new or ongoing programs. The program has large potential significance to the region. It is supported by the region’s technical community as long as it reaches its objectives as planned and the results are unambiguous. If so, it should answer a number of long-standing and contentious issues about the impacts and efficacy of supplementation as a method of sustaining and providing long-term increases in depressed salmon populations' productivity. Data to allow such insights should start to come in over the next few years, thereby justifying this 20+ year investment. 2. Project History and Results The proposal provides a lengthy and good review of the project's 20-year history. The ISS study design was first proposed in 1990, and has been executed within this framework with challenges owing to the unavailability of fish for stocking during the mid-1990s and owing to policy/political decisions by the Nez Perce Tribe to begin a supplementation program on Johnson Creek, one of the control (reference) streams. Because of these difficulties, the ISRP urged a review of the experimental and analysis design in 2001 to determine whether the effort had the potential to produce data that could be analyzed and used to answer uncertainties about supplementation. The ISRP has reviewed updated study designs in 2005 and again in 2006 during the 2007/2009 project solicitation. Because of this history of review and the depth of detail in the design, it is probably not necessary to revisit these topics. However, the proposal is insufficient to fully glean this background from the proposal. The project history presentation also describes the extensive resources (funds) dedicated to the project for the past two decades. Treatment phase of this long-standing project ended in 2007, and the evaluation phase started in 2000. Thus limited data are available to represent the evaluation phase. However, virtually no results or data were presented in the project proposal other than the brief summary of Pahsimeroi Chinook genetic parentage studies. The lack of results, even at the minimum level of general trends and observations, was disappointing. The proponents should have provided more results on project performance to date. The proposal lists results as the development of protocols and the performance of annual data gathering (adult weirs, redd count and carcass surveys, emigrant traps, and snorkel estimates). While these steps were surely necessary to the project’s implementation and M&E, they do not constitute data or results—they are process. Only one peer-reviewed paper appears to have been produced through the project, and it is not focused the on primary objectives of project. There is a five-year-old masters thesis on parentage analysis in a study stream (Pahsimeroi) that has not been published in the open literature. This omission should be rectified, particularly given the reliance by the proposal proponents on the results of that study. Consequently, it should be validated through the peer review process. As written, the proposal suggests that data analysis will only begin after the last fish has been processed in 2014 or thereabout. It is reasonable (at least from the ISRP’s perspective) to expect that some preliminary analysis of data from the evaluation phase of this effort that began in 2008 start immediately. Finally, as a result of actions taken in response to the 2001 ISRP Provincial Review, the project has been modified to better provide relevant data. Unfortunately, the important work proposed in Genetic Evaluation of Salmon Supplementation in Idaho Rivers 200725000 was not funded. 3. Objectives, Work Elements, and Methods The proposal clearly states the objectives as research goals: 1) Assess the use of hatchery Chinook salmon to increase natural populations of spring and summer Chinook in the Salmon and Clearwater River drainages; and 2) Evaluate the genetic and ecological impacts of hatchery Chinook salmon on naturally reproducing Chinook populations. The proposal identifies the contribution that each task will make to program needs, though methods are not described by objective or work element Methods were not included in the proposal, though references were made to methods and analysis procedures that are well documented elsewhere, often as a result of discussion or reviews by the ISRP. A reviewer new to this project would likely find the proposal inadequate in these areas, if they were not familiar with the earlier documents and discussion. For example, a brief explanation and justification of the standardized index of adult escapement and natural production used to compare across treatment and control (reference?) streams should be provided in the proposal. Fortunately, the project appears to be remaining squarely on course with its schedule and methods as previously worked out with the ISRP through the extensive and intensive reviews done in 2001 and 2005. Two new objectives appear to be the basis for the Fast Track proposal review: 1) Method #8 would install a second screw trap on Marsh Creek in order to provide additional data on juvenile outmigration from the Marsh Creek drainage; and 2) Method #9 would "verify AHA and AHSWG model assumptions and predictions using ISS data." Installation of the screw trap at Marsh Creek appears reasonable and justified. The second Fast Track element, AHA model validation, is not currently justified and does not look time sensitive (i.e., fast track), and therefore more justification is needed. The other project work elements seem well established now and relatively routine. The proposed new task of using historical ISS juvenile release, survival, and adult return data to evaluate the AHA model by comparing actual adult returns to those predicted by AHA and AHSWG models is a useful component of the ongoing project; however, several issues are raised by this new task. First, ISS field activities are scheduled to end during this funding cycle, and appropriate plans will need to be developed if ISS infrastructure is to be maintained for intensive and extensive VSP status and trend monitoring (Appendix A). After 2012, ISS cooperators will no longer be evaluating adult returns to study streams (i.e., redd counts and carcass surveys), and juvenile migration evaluation will be complete after smolt trapping in 2014 (brood year 2012). In order to maintain current levels of intensive and extensive status and trend monitoring, ISS infrastructure and sampling duties will need to be incorporated into new or existing programs. Second, select personnel from the ISS cooperating agencies will need to be funded for a period of time after 2014 to complete data analysis, a completion report, peer reviewed publications, and ensure ISS findings are communicated to and incorporated into other regional supplementation programs. As the project nears completion and personnel are shifted within the project and agency, we encourage key personnel to remain engaged in the project through its completion, in order to best realize the project’s original objectives. Finally, AHA has been reviewed by the Puget Sound Technical Recovery Team (Review of the All-H analyzer model, March 18, 2005) and by the Recovery Implementation Science Team (April 9, 2009). There is no discussion of the past reviews of this model, how the ISS data will be used to address which assumptions and uncertainties, and how the analysis and assumption validation will be conducted. 4. M&E This is one of the project's strong suits, though not enough detail is provided in this specific proposal for a new reviewer to determine that (or possibly to support the project). The strength of the methods comes from the work done by University of Idaho (at the request of the ISRP) to identify a statistical procedure that could answer the supplementation questions posed by the project and by the proponents commitment as stated in this proposal to maintain the study design unchanged through the sunset date for the project. It is still not clear whether the ISS will yield data that can be analyzed to answer questions about the efficacy of supplementation. The primary challenge is the quality (precision) of adult abundance data that can be derived from redd count and carcass inspection in study streams that lack interrogation weirs. Even on streams that have weirs, estimating weir efficiency and adjusting data may be necessary. In the last review, the ISRP was critical of the carcass data that was collected in conjunction with redd counts, which would be used to assign total adult counts (based on redds) proportionately to wild, supplementation, and conventional hatchery (strays) production. For several sample streams the carcass information came from limited sampling and had no associated estimates of precision. The ISRP recommended that the redd counting and carcass inspection be more rigorous. The proposal cites that the goals and strategies for monitoring and evaluation of the status of Snake River Chinook salmon and steelhead identified in the fall 2009 RM&E workshop guide expansion of the ISS (Table 1, page 7). More information is needed for evaluation of whether the increased effort meets the RM&E goals. In particular, one goal is to "obtain high precision status and trend data for at least one population per adult life-history type per MPG (fish in, fish out monitoring).” One of the open questions is the selection of populations for this monitoring. The ISS proposal suggests that they may be collecting this information; however, the population is not yet selected. A succinct summary of the MPGs and independent populations established by the TRT, which have high precision data, and which are associated with the ISS needs to be included in the proposal. The high precision data type is not clear. The citation is to Crawford and Rumsey (2009) and reference is to data with a CV of 15% or less. CV (coefficient of variation) is not usually associated with precision of data, but with the variation associated with a state of nature. That is, salmon abundance across years has a CV, fall steelhead parr length has a CV. These are descriptions of the state of variation. They are not appropriate to determine confidence intervals. Crawford and Rumsey (2009) reference Carlile et al. (2008), which makes recommendations for coefficients of variation for estimates of total spawning escapement. The reference is to standard error of the estimate, not to variation in the population. More importantly, the statistical and biological basis for the recommendation in Carlile et al. (2008) has not been reviewed. The justification that the standard represents a realistic goal for planning because it corresponds to an acceptable risk (one year of one stock in six) of failing to label a stock of concern when warranted appears to be arbitrary. The observation that the standard has proven to be attainable for many escapement estimation studies does not mean that this is the appropriate data standard. Finally, the project proponents should continue to enlist the services of a statistician, such as they did with K. Steinhorst from the University of Idaho. It is hard to know from the information provided whether the final analysis after 2014 will be meaningful. As part of this review process, a solid reporting of the data already collected would help demonstrate project proponent’s ability to conduct the analysis. It would also be good for the ISRP/ISAB to review a draft report before the final report is released, similar to independent reviews of the Select Area Fisheries Evaluation (ISRP/IEAB 2007-3), Comparative Survival Studies 10-year retrospective (ISAB/ISRP 2007-6), and Captive Propagation projects (ISRP 2004-14). This is a very important (and expensive) long running project which heightens the need to make certain the program comes to fruition successfully.