The 2014 New York City Neighborhood Survey (NYCNS) was a 20-minute internet survey of 1,104 New York City residents that ran in October and November of 2014. The survey was designed by Scott Minkoff and conducted by the survey research firm Qualtrics  (funding support from Barnard College).  The focus of the survey was on respondent evaluations of their neighborhood environment and city services as well as their use of public amenities.  Data was also collected on variables related to political preferences, wealth inequality, civic engagement, 311 usage, and other topics.  An important feature of the survey was the collection of geographic identifiers for respondents in the form of the nearest cross-streets to the respondent’s home. While providing a location was optional, 920 of the 1,104 respondents provided it.  

Survey Representativeness

Qualtrics uses a variety of methods to obtain its non-probability samples including recruiting people through social-media and marketing lists. Survey-takers are incentivized to complete surveys with cash, vouchers, airline miles, and other gifts.  Internet surveys can obtain large samples at a significantly lower cost than mail or telephone surveys.  However, internet surveys can lack the randomness and representativeness that more expensive survey methodologies have.   Despite the limitations, efforts were made to achieve as representative of a sample as is possible given the realities of internet survey research.  For example, quotas were instituted to ensure that at least 40% of the survey was non-white and 40% were parents.  To be certain that all of respondents live in one of New York City’s five boroughs, the survey began by asking them to provide their borough and ZIP Code without knowing that the survey was only for New York City residents.  Non-residents were removed from the survey. 

Table 1 compares the racial and income breakdown of the survey respondents with the racial and income breakdown of New York City (as of the 2010 Census).  As is evident, the survey is whiter and a bit wealthier than the city as a whole.   The 40% non-white quota that was used is lower than the actual percentage of New Yorkers that are non-white but above typical internet survey standards.  With respect to income, the survey is under-representative of those with the very lowest household incomes (primarily those in families making less than $20,000) and over-representative of those in the middle and upper ends of the spectrum (primarily families making between $50,000 and $150,000).  The most glaring difference is with the gender breakdown: nearly 60% of the survey’s respondents are female.  This is typical for recruited internet surveys like this one.  

Table 1. Sample Representativeness: Income and Race

The sample can also be examined for its geographic representativeness. There was slight under sampling in the Bronx, Brooklyn, and Queens and slight over sampling in Manhattan and Staten Island. Boroughs, however, are not a distinction of much governmental meaning in contemporary New York City. Though boroughs have developed some cultural identities, the reality is that all five boroughs are big enough to be highly diverse cities in and of themselves—Brooklyn and Queens both have well over 2 million residents (alone they would be the third and fourth most populated cities in the country respectively).  More important than the borough breakdown for this study is the neighborhood breakdown. The question is then whether neighborhoods (or parts of the city) are represented in the survey in proportion to their actual population breakdown.  ZIP Codes—a geographical unit that the Census Bureau tabulates data for—can be used to approximate neighborhood and the percent of the survey in a given ZIP code is correlated with the percent of the city population in a ZIP Code at 0.72 (p<.01). ZIP Codes with more of the city’s population have more respondents in the survey and ZIP Codes with less of the city population have fewer respondents in the survey so respondents are not concentrated in some unexpected part of the city.  Map 1 shows the actual tract-level population density of New York City with the survey respondents overlaid.  The map demonstrates that the spatial distribution of respondents conforms well to the actual spatial-distribution of the population.  A few areas including western Staten Island and some of the more business-oriented parts of Manhattan do appear to be over-represented in the survey.  And a few areas, including the more suburban parts of Queens and Brooklyn, tend to be under-represented in the survey.

Map 1. Sample Distribution & Tract-Level Population Density