CrimeSolutions.gov

Additional Resources:

Program Profile: Harlem (NY) Children's Zone – Promise Academy Charter Middle School

Evidence Rating: Effective - One study Effective - One study

Date: This profile was posted on June 13, 2011

Program Summary

This is a charter middle school that serves predominately low-income, minority students (grades 6–8) in Harlem who are usually 2 or 3 years behind grade level. The program is rated Effective. Students who were admitted to the school showed statistically significant improvements in English Language Arts scores, math scores, and attendance rates, compared with students who were not admitted. However, there were no significant differences between groups on matriculation.

This program’s rating is based on evidence that includes at least one high-quality randomized controlled trial.

Program Description

Program Goals/Target Population
The Harlem Children’s Zone (HCZ) is a 97-block area in the Harlem neighborhood of New York, NY, that combines "no excuses" charter schools with communitywide initiatives to address some of the main problems that underprivileged children face every day, such as inadequate schools, high-crime neighborhoods, and health issues. One important component of the HCZ Project is the Promise Academy Charter Middle School, which aims to provide students in grades 6–8 with a well-rounded, high-quality education.

The Promise Academy middle school serves predominately low-income, minority students in Harlem who are usually 2 or 3 years behind grade level. While the community programs offered by HCZ are available to anyone living near the area, the charter middle school is not open to all students. Because of the limited amount of space available, admission to the Promise Academy is only offered to students whose numbers are randomly selected during the middle school lottery. The school began in 2004 with 100 sixth graders, and by 2009, there were 700 middle school students in the system.

Program Components
The Promise Academy combines structural reforms with wraparound services to provide students with a comprehensive college preparatory program. The Promise Academy provides an extended school day and year, coordinated afterschool tutoring, and additional classes on Saturdays for anyone who needs extra help with homework. In the classroom, strong focus is placed on mastering basic skills, especially literacy and mathematics skills. Students have more than 2 hours of literacy instruction and more than 90 minutes of mathematics instruction every day. Students also explore coursework in the arts and sciences.

In addition, many students spend 10 or more hours a day in school. An average school day runs from 8 a.m. to 4 p.m., which is a 20 percent increase over a typical school day. Students are also given the opportunity to participate in afterschool programs, such as photography or music activities, that run from 4 p.m. to 6 p.m. The academic year is 210 days of school, which includes a mandatory 25-day summer program. Only 180 days of school are required by law.

The Promise Academy also runs a health clinic in the middle school building that provides students with free medical, dental, and mental-health services. Students are screened upon entering the school, and they receive regular check-ups through the Children’s Health Fund. Students also receive incentives for achievements (such as money or trips), nutritious and freshly made meals, bus fare, and other intangible benefits, such as support from committed staff members.

Additional Information
The Promise Academy Charter Middle School is part of the HCZ Project that provides a continuum of services focusing on the developmental needs of children in Harlem. In addition to the middle school programs, the Project includes elementary school programs that are available to students in grades K–5. However, the evaluation research examining the effectiveness of the elementary school programs has not yet reached the rigor of the research looking at the middle school programs.

Evaluation Outcomes

top border
Study 1
Math Scores
Dobbie and Fryer (2010) found that students who won the lottery and attended the Promise Academy middle school showed statistically significant improvements in math scores, compared with the control group students (i.e., youth who did not win the lottery) who did not attend.

English Language Arts (ELA) Scores
Students who attended the Promise Academy middle school showed statistically significant improvements in ELA scores, compared with control group students.

Absences
Students who attended the Promise Academy middle school showed a statistically significant reduction in absences in the first 180 days of school, compared with control group students.

Matriculation
There were no statistically significant differences between students in matriculation (which measured whether a student was “on-time” given their expected grade).
bottom border

Evaluation Methodology

top border
Study 1
Dobbie and Fryer (2010) examined the effects of being admitted to or attending the Promise Academy charter schools in the Harlem Children’s Zone (HCZ) on academic outcomes. In New York State, oversubscribed charter schools are required by law to allocate enrollment offers through a random lottery system. The study took advantage of this system by comparing the average outcomes of students who “won” the lottery and were offered admission to the Promise Academy to the average outcomes of students who “lost” the lottery and were not offered admission.

The sample included students from the middle school lotteries from 2005 through 2006 (data was not available from the 2004 middle school lottery). The treatment group included 169 lottery winners in grades 6–8. The lottery winners were 51 percent male, and 84 percent black and 14.5 percent Hispanic. Seventy-two percent qualified for free lunch. The control group included 289 grade 6–8 youth who did not win the lottery. The group was 43.3 percent male, and 82.6 percent black and 15.4 percent Hispanic. Sixty-five percent qualified for free lunch. There were no statistically significant differences between the groups.

The study used an intent-to-treat (ITT) analysis, which meant the treatment group included all students who were offered admission to Promise Academy, even if they did not attend. The ITT analysis therefore captured the effect of being offered admission, not of actually attending. The study also included treatment-on-treated (TOT) analysis, which looked at the average effect of enrollment in the Promise Academy on those students who won the lottery and chose to attend. The results for the TOT analysis are provided in the Other Information section of the profile.

Data was collected from files at HCZ as well as administrative data on student demographics and academic outcomes from the New York City Department of Education (NYCDOE). Academic outcomes in math and English Language Arts (ELA) were measured by the New York State tests that are conducted in the winters of third through eighth grade. The math portion of the exam includes questions on number sense and operations, algebra, geometry, measurement, and statistics. The section on ELA assesses students on information and understanding, literary response and expression, and critical analysis and evaluation. The study authors constructed measures of absenteeism and matriculation using NYCDOE data. Absenteeism was measured as the total number of absences a student accumulated during the first 180 days of the school year (after the first 180 days, NYCDOE no longer collects absence data from schools). Matriculation was used as an indicator for whether a student was “on-time” given their expected grade. Subgroup analysis was conducted by grade level.
bottom border

Cost

top border
In the 2008–09 school year, the Harlem Children’s Zone in the Harlem neighborhood of New York, NY, spent approximately $19,272 per pupil at the Promise Academy. This included in-school costs and afterschool programs. This can be compared to the median school district in New York State in 2006 that spent approximately $16,171 per pupil (Dobbie and Fryer 2010).
bottom border

Implementation Information

top border
For more information about Harlem Children’s Zone, please see the program’s website: https://hcz.org/
bottom border

Other Information (Including Subgroup Findings)

top border
Dobbie and Fryer (2010) looked at the effects of being admitted (lottery winners) to Promise Academy by grade level. They also examined the effects of enrolling in the charter school. Lottery winners were 11.9 percent more likely to be performing on grade level in math in the sixth grade, 16.3 percent more likely in the seventh grade, and 27 percent more likely in eighth grade, compared with lottery losers across the three grades. These differences were statistically significant. The treatment-on-treated (TOT) analysis, which looked at the effects of actually attending the charter middle school, showed a similar pattern. There was a convergence in test scores between students who attended the Promise Academy middle school and the average white student in NYC. In other words, Promise Academy students had nearly caught up to the average white student in New York City public schools in math test scores after 3 years of schooling. The TOT analysis found that lottery winners who enrolled in the charter middle school scored 0.338 standard deviations higher in sixth-grade math, 0.371 higher in seventh-grade math, and 0.745 higher in eighth-grade math, compared with the control group. The results were statistically significant for all 3 years.

As with math scores, lottery winners, losers, and the average African American student in NYC had similar ELA scores, which were roughly 0.65 standard deviations behind the average white student in the city. ITT analysis showed there were no significant differences in ELA scores between the lottery winners and losers until eighth grade. In the eighth grade, however, Promise Academy students had significantly higher scores than the control group, although the treatment effect was not as strong on ELA scores as on math scores (i.e., lottery winners scored 0.196 standard deviations higher in eighth grade). The TOT analysis found that scores for students enrolled in Promise Academy were 0.279 standard deviations higher in eighth
bottom border

Evidence-Base (Studies Reviewed)

top border
These sources were used in the development of the program profile:

Study 1
Dobbie, Will, and Roland G. Fryer Jr. 2010. Are High-Quality Schools Enough to Increase Achievement Among the Poor? Evidence from the Harlem Children’s Zone. Cambridge, Mass.: The Kennedy School, Harvard University.
http://scholar.harvard.edu/files/fryer/files/hcz_nov_2010.pdf
bottom border

Additional References

top border
These sources were used in the development of the program profile:

Hoxby, Caroline M., and Sonali Murarka. 2009. Charter Schools in New York City: Who Enrolls and How They Affect Their Students’ Achievement. Cambridge, Mass.: National Bureau of Economic Research Working Paper.
http://www.nber.org/papers/w14852

Hoxby, Caroline M., Jenny Lee Kang, and Sonali Murarka. 2009. Technical Report: How New York City Charter Schools Affect Achievement. Cambridge, Mass.: National Bureau of Economic Research.

Harlem Children’s Zone. N/d. Whatever It Takes: A White Paper on the Harlem Children’s Zone.
https://hcz.org/wp-content/uploads/2014/04/HCZ-White-Paper.pdf
bottom border

Related Practices

top border
Following are CrimeSolutions.gov-rated practices that are related to this program:

Targeted Truancy Interventions
These interventions are designed to increase attendance for elementary and secondary school students with chronic attendance problems. The practice is rated Effective for improving attendance.

Evidence Ratings for Outcomes:
Effective - More than one Meta-Analysis Education - Attendance/truancy



Mentoring
This practice provides at-risk youth with positive and consistent adult or older peer contact to promote healthy development and functioning by reducing risk factors. The practice is rated Effective in reducing delinquency outcomes; and Promising in reducing the use of alcohol and drugs; improving school attendance, grades, academic achievement test scores, social skills and peer relationships.

Evidence Ratings for Outcomes:
Effective - One Meta-Analysis Crime & Delinquency - Multiple crime/offense types
Promising - More than one Meta-Analysis Drugs & Substance Abuse - Multiple substances
Promising - One Meta-Analysis Education - Multiple education outcomes
Promising - One Meta-Analysis Mental Health & Behavioral Health - Psychological functioning



Dropout Prevention Programs
School- or community-based programs targeting frequently absent students or students at risk of dropping out of school. These programs are aimed at increasing school engagement, school attachment, and the academic performance of students, with the main objective of increasing graduation rates. The practice is rated Effective for reducing rates of school dropouts, and rated Promising for improving test scores/grades, graduation rates, and attendance.

Evidence Ratings for Outcomes:
Effective - More than one Meta-Analysis Education - Dropout
Promising - One Meta-Analysis Education - Academic achievement/school performance
Promising - One Meta-Analysis Education - Graduation
Promising - One Meta-Analysis Education - Attendance/truancy
bottom border


Program Snapshot

Age: 11 - 14

Gender: Both

Race/Ethnicity: Black, Hispanic

Geography: Urban

Setting (Delivery): School, High Crime Neighborhoods/Hot Spots

Program Type: Academic Skills Enhancement, Afterschool/Recreation, Classroom Curricula, Mentoring, School/Classroom Environment, Truancy Prevention

Current Program Status: Active

Listed by Other Directories: Top Tier Evidence Initiative

Program Director:
Harlem Children’s Zone
35 East 125th Street
New York NY 10035
Phone: 212.360.3255
Fax: 212.289.0661
Website
Email

Researcher:
Will Dobbie
Public Policy Doctoral Program, Harvard Kennedy School
79 John F. Kennedy Street
Cambridge MA 02138
Email