MASTEP staff reviews reports provided by the BMP manufacturers and others, including verification studies. Field studies are compared with the TARP Tier 2 Protocol to determine if study design and quality assurance/quality control measures are sufficient to produce a valid data set. Laboratory studies are compared with the New Jersey DEP Total Suspended Solids Laboratory Testing Procedure.

Initially, all technologies are considered unrated with regards to existence of reliable performance data. Once information from verification studies is reviewed, a technology is rated as shown in table below. If a product claims to treat TSS, the TSS rating is shown. For all other products, the highest rating a product has received is shown.

It is important to note that a technology's category only reflects the availability of reliable studies. A rating of "1" does not imply that the vendor's performance claims are validated, or that the system performs better than one rated “2”, “3” or “4.”  Rather, it affirms that the BMP has been tested in a scientifically credible manner, based on MASTEP review of the study results. Technologies that are rated “2” or “3” may still meet the stormwater control needs at any given site.

Screening process used to rate each technology:

Field studies:
Field studies are preferred by many scientists and regulators because they are considered the best representation of “real world” conditions. Laboratory testing is conducted under idealized conditions that cannot predict or simulate all possible environmental variables encountered in field settings. Factors ranging from icing of system components to dead armadillos in stormdrains have been encountered in studies that MASTEP has reviewed. These may affect performance in ways not explored in laboratory testing. Laboratory studies typically test standardized “clean” sediment mixes, which may behave quite differently than sediments mixed in with grass clippings, leaves, Styrofoam cups and any variety of trash, debris, or pollutants. Similarly, actual meteorological events are likely to be much more variable in terms of duration, intensity, number of peaks, etc. than flows produced for laboratory tests. Field studies provide the ultimate answer to a fundamental question relevant to all laboratory studies: “When this system is installed and operating at a given location, will it perform as predicted?”  

The full TARP Tier II field protocol lists over 30 criteria on which a technology and relevant field studies are judged.  Some of highlights are:

Must have performance claim. This provides the basic performance hypothesis to test. Performance claims that specify the conditions under which pollution removal goals are met are preferred; simply to state that “80% TSS removal” will be achieved is not sufficient. Tests performed under optimal conditions – e.g. low flow rates, large sediment sizes, high pollutant concentrations may support such a claim, but they may not represent typical stormwater conditions. It is best when the claim states that performance goals will be met within a range or limit of flow rate and/or volume, particle size, influent pollutant concentration, and initial sediment loading.

Quality assurance plan, quality control data. This helps us determine whether the study was designed according to accepted scientific principles, whether and how well the test plan was followed.

Test site plan, description. Photographs, diagrams and text description of the test site and apparatus also help us determine how well accepted sampling protocols were followed.

Third party studies preferred. To maintain objectivity.

At least 50% of average annual rainfall at the test location must be sampled. This is to ensure that a large enough sample size was obtained and that the product was tested under a sufficient variety of conditions to be representative of all conditions that are likely to be encountered at a given installation. In addition, the capacity of a device to perform over time, as accumulating loads of sediment enter the system, is critical to its overall performance, as well as establishing maintenance schedules that will ensure consistent performance over time.

A minimum of 15 inches of precipitation must be monitored during the course of the study. Same reasons as the 50% annual rainfall criterion.

A minimum of 15 storms must be monitored. Same as the above two criteria.

Some sampling must be done under adverse weather conditions (e.g. winter, heavy rain events).   Runoff and contaminant transport is expected to be greater during these conditions. This criterion tests the ability of BMPs to handle such conditions.

Flow-weighted composite samples covering at least 70% of the total stormflow, including as much of the first 20% of the storm as possible. This approach is consistent with the scientific literature on the “first flush" – that period early in a storm when the greatest amount of pollutant transport is likely to occur.

Parameters to be monitored must be based on vendor’s claims. If sediment removal is claimed, measure both total suspended solids (TSS) and Suspended Sediment Concentration (SSC). If a manufacturer makes a claim of pollutant removal, it should be validated. Regarding sediment, the Massachusetts stormwater policy and the MASTEP evaluation system are primarily based on sediment removal – as are many of the TARP and NJ lab sampling criteria. An assumption is made that if sediment transport can be minimized, other pollutants (many of which adhere to sediment particles) will also be controlled. TSS is the traditional method for analyzing sediment. However, in recent years many in the scientific community have come to the conclusion that SSC is a more accurate and appropriate measurement method for stormwater. The TSS method selects a portion of a total sample for analysis rather than the whole sample – in the process creating a bias towards smaller sediments. This particularly has an impact on representativeness and accuracy of influent sediment concentrations, and may therefore affect reported sediment removal rates. The SSC method generally produces higher removal efficiencies than the TSS method. It is important to recognize that and to know what methods are used when reviewing performance results or comparing different studies. See A Comparison of Load Estimates Using Total Suspended Solids and Suspended-Sediment Concentration Data for more information on this topic.

Mean particle size should be < 100 microns; recommended particle size distribution is 55% sand, 40% silt, 5% clay. This is a mix that is considered to be typical of sediments found in stormwater.  Larger particles are easier for most systems to capture (filter systems are often an exception to this rule). 

Mean influent concentration 100-300 mg/l. Relevant for TSS/SSC only – not other parameters. 

Pollution removal efficiency calculation: Efficiency Ratio method is preferred, Summation Of Loads should also be computed where feasible. For a full discussion of calculation methods, see this ASCE/EPA technical memorandum which describes several methods. Of the two preferred by TARP, the ER method weights all storms equally regardless of the magnitude of a storm. For example a high concentration/high volume event has equal weight as a low concentration/low volume event. Conversely, the SOL method will emphasize large storms with heavy sediment loads. We have seen studies where one storm provided over 40% of an entire year’s sediment load, covering 10 or more storms. Although TARP states a preference for ER method, in Massachusetts the SOL may be more appropriate, because stormwater policy is volume-based, targeting a desired reduction in annual loading.

Where applicable, effect of bypass flow on process efficiency and system performance should be quantified. Depending on how treatment systems are built, some will divert high flows around a unit, in order to prevent scouring or re-suspension of sediments that have accumulated in a system. Scouring is caused by the turbulence and high velocities caused by high flows. Waters bypassing a system are not treated, therefore contributing higher pollutant loads to receiving waters. Without quantification of the effect of bypass on a unit, success in achieving overall pollution reduction goals cannot be measured.  We have seen studies that documented 90+% pollution removal for water passing through the system, but the system was so undersized that in some cases less than 1% of  a total storm load was treated – the remainder flowing by with no treatment.

Scour tests should be performed with initial 50% and 100% sediment loading. This can be done in a separate lab test. A system should be loaded with 50% (and 100% in a separate test) of the accumulated sediment load that should trigger maintenance action (i.e. cleaning). The system is then run at 125% of capacity, and the amount of sediment lost from the unit is then determined. This allows quantification of net performance of a system over several storm events, and helps in identifying appropriate maintenance schedules.  If systems are not maintained properly and/or designed properly to minimize this problem, they may simply re-circulate pollutants; sediments will collect in a unit, then may be scoured out at the next large storm, providing little or no net environmental benefit.

Laboratory studies
In the words of Jim Lenhart, chief technical officer for Contech Stormwater Solutions, “Laboratory testing enables one to control all variables, isolate one, and then test a hypothesis by comparing the model results and the device’s laboratory performance.” By using standardized methods and specific inputs, laboratory studies on a given system are repeatable, and comparison of different systems under similar conditions is possible. This is difficult if not impossible in field studies, which are subject to site specific and/or randomly occurring soil conditions,  pollutants, weather events and unpredictable conditions that nature and unplanned human activity happen to present to them. Laboratory studies are also usually less expensive than field studies, hence there tend to be more of them conducted.

The New Jersey DEP Total Suspended Solids Laboratory Testing Procedure includes criteria that are in many cases similar to field studies, adapted as appropriate for the lab. Because laboratory studies can control conditions better than can field studies, several criteria are more specific. Here are some highlights.

Must have performance claim. As with field studies, this provides the basic performance hypothesis to test. However, in some cases laboratory studies are conducted to establish a performance claim. Performance claims that specify the conditions under which pollution removal goals are met are preferred; simply to state that “80% TSS removal” will be achieved is not sufficient. Tests performed under optimal conditions – e.g. low flow rates, large sediment sizes, high pollutant concentrations may support such a claim, but they may not represent typical stormwater conditions. It is best when the claim states that performance goals will be met within a range or limit of flow rate and/or volume, particle size, influent pollutant concentration, and initial sediment loading.

Quality assurance plan, quality control data. This helps us determine whether the study was designed according to accepted scientific principles, whether and how well the test plan was followed.

Test site plan, description. Photographs, diagrams and text description of the test site and apparatus also help us determine how well accepted sampling protocols were followed.

Third party studies preferred. To maintain objectivity.

Model scale. Laboratory studies conducted on full scale units are preferred. It is impractical to expect that all sizes/models of a technology are tested. If models smaller than the smallest commercially available system are tested, a discussion of fluid dynamics (e.g. relationships between velocity, settling times, unit dimensions, particle sizes and how these are to be extrapolated to larger units) is desired.

Scour tests should be performed with initial 50% and 100% sediment loading. A system should be loaded with 50% (and 100% in a separate test) of the accumulated sediment load that should trigger maintenance action (i.e. cleaning). The system is then run at 125% of capacity, and the amount of sediment lost from the unit is then determined. This allows quantification of net performance of a system over several storm events, and helps in identifying appropriate maintenance schedules. If systems are not maintained properly and/or designed properly to minimize this problem, they may simply re-circulate pollutants; sediments will collect in a unit, then may be scoured out at the next large storm, providing little or no net environmental benefit.

Flow rates: number tested and at what % of design flow. This is one area where controlled laboratory conditions can be particularly useful in evaluating the performance of a system under a range of conditions. Most units work much better at low flows, relative to the peak flow, or design flow (the highest flow rate at which a desired treatment efficiency can be obtained). Tests that evaluate only at low flows will overestimate the effectiveness of a system.  At least 5 flow rates are preferred: at 25%, 50%, 75%, 100%, and 125% of the design flow. 

Influent pollutant concentration 100-300 mg/l. Relevant to TSS/SSC. This is considered to be a typical range for stormwater. Higher influent loads are considered easier to treat than is “cleaner” water.

Particle Size Distribution, average particle size: NJ recommends 55% sand, 40% silt, 5% clay, with an average size (d50) of < 100 microns. A variety of commercially produced sediment mixtures is available.  These are generally preferable to mixes created by a testing organization, because they are standardized, ensuring comparability among different tests. Both criteria (average size and distribution) are important. Some mixes have a narrow range - neither many small nor large sizes. Such mixes will not properly evaluate the “low range” capabilities of a system. 

Number of tests run: Ideally, at least 15 tests will be run. NJ recommends that the 15 involve 3 runs each at 25%, 50%, 75%, 100% and 125% operating rate. Within each rate, one run each at 100, 200, and 300 mg/l influent TSS.

Pollution removal efficiency calculation: Efficiency Ratio method is preferred, Summation Of Loads should also be computed where feasible. For a full discussion of calculation methods, see this ASCE/EPA technical memorandum which describes several methods. Of the two preferred by TARP, the ER method weights all storms equally regardless of the magnitude of a storm. For example a high concentration/high volume event has equal weight as a low concentration/low volume event. Conversely, the SOL method will emphasize large storms with heavy sediment loads. We have seen studies where one storm provided over 40% of an entire year’s sediment load, covering 10 or more storms. Although TARP states a preference for ER method, in Massachusetts the SOL may be more appropriate, because stormwater policy is volume-based, targeting a desired reduction in annual loading.

Where applicable, effect of bypass flow on process efficiency and system performance should be quantified. Depending on how treatment systems are built, some will divert high flows around a unit, in order to prevent scouring, or re-suspension of sediments that have accumulated in a system. Scouring is caused by the turbulence and high velocities caused by high flows. Waters bypassing a system are not treated, therefore contributing higher pollutant loads to receiving waters. Without quantification of the effect of bypass on a unit, success in achieving overall pollution reduction goals cannot be measured.  We have seen studies that documented 90+% pollution removal for water passing through the system, but the system was so undersized that in some cases less than 1% of  a total storm load was treated – the remainder flowing by with no treatment.

Calculating treatment efficiency. New Jersey uses a weighted calculation method (illustrated in the table below from the NJ laboratory guidelines paper) to assign a single efficiency score to stormwater treatment devices.

There are several reasons why a technology, or a study evaluating a technology, might be downgraded from a 1 to a 2 or a 3 rating. Understanding the reasons for a particular rating and comparing these with specific stormwater control needs of a particular location may help in selecting a solution that best fits local needs. There are many situations where a #2 or #3 rated technology may better serve a particular application than will a #1 rated technology.

There are 4 general categories that will get a performance study downgraded.  These are:

1. Small sample size. If only a few tests were run, or storms sampled, or a small  % of rainfall monitored, representativeness of the results are open to question. It becomes harder to predict with any assurance that the results obtained will occur in future storm events. Decisions made on this type of study are something of a gamble.
2. Narrow or inappropriate range of conditions. This commonly shows up in the flow rates, sediment sizes, and influent pollutant concentrations tested. It is manifested in two different ways. First, conditions may not have adequately “stressed” the system. If, for example, a laboratory test involves no flows greater than 50% of the system design flow, then the removal rates produced are likely to be optimistic. Second, the ranges tested may not match those that are found at the site where a BMP is to be installed. For example, sediment sizes tested in a study may be quite different from local soil conditions. On the other hand, if a narrow particle distribution contributes to a #2 or #3 rating for a study, but this is a range that happens to approximate what is found at the local site, it may be an appropriate performance predictor. Storm intensity is another factor that might vary greatly from one site to another, depending on precipitation patterns, watershed size and % imperviousness, or topography. In any case, knowledge of local environmental conditions will benefit decisions on selection of stormwater solutions.
3. Insufficient documentation. When a report omits or provides scant discussion of methods used, quality control results, or fails to include raw data. It becomes a matter of trust whether or not to accept the results as valid.
4. Problems with methodology, equipment, quality control, etc. that lead to missing or unreliable results. This is the one category that absolutely cannot be discounted, even if a study is otherwise well-designed and conducted.  The only redeeming value that studies with these problems may provide is a sound discussion of how the problems occurred and recommendations for avoiding them in future studies. In some cases (and with great caution), the study report may suggest reasonable approaches to drawing approximate conclusions by adjusting or extrapolating erroneous data sets. This should be done only as a qualitative exercise.  MASTEP does not recommend using unvalidated data for quantitative performance assessment.  

The database is capable of providing guidance on the applicability of various BMPs for different use applications. The end-user can utilize any number of site conditions, water quality objectives, cost, or performance-based factors to screen and evaluate compatibility of a BMP product with project needs and limitations. This database provides reports on vendor-provided information on cost, installation and maintenance requirements, and other product specifications. The database may be useful for comparisons of specific products with conditions and circumstances present in stormwater control projects. For additional information, the DEP-CZM Stormwater Technical Handbook describes performance-related and other issues involved in selecting a technology. These include:

Cost: price of system as well as installation and maintenance costs. See separate discussion of how
	costs are calculated
Maintenance requirements: frequency, type of maintenance required
Size of unit(s). Compare this to area available on the site to place a BMP
Setback requirements. Some products require minimum distances to:   septic systems surface water, private well, drinking water supply (ground or surface) steep slopes property line foundation
Depth to bedrock
Depth to water table
Soil types
Infiltration rates: some BMP types have recommended minimum and/or maximum rates
Watershed area drained
Percent impervious surface in drainage area
Precipitation types in project area: frequency, duration, intensity of storms
Particle sizes expected during storm events
Discharge rates for 1 yr, 2yr, 10 yr etc. storms in project area: compare these to flow and volume
	capacity of technology under consideration
Impacts/pollutants of concern, including need for recharge / storm volume attenuation.