ShiftSpace

Earlier this year we (David Nolen & Mushon Zer-Aviv of the ShiftSpace team together with Kevin Connor & Matthew Skomarovsky of LittleSis) have applied to the Knight News Challenge with NewsShift – watchdog journalism with a long tail, an application based on ShiftSpace and LittleSis.

Updating

Unfortunately we have been informed recently that our application has been rejected. We thank you all for supporting us and though it sure is disappointing we were very encouraged by the interest. Knight reports that nearly 2,400 grant applications were filed this year. Out of this huge list NewsShift has made it to the final 50 (!) but unfortunately was not chosen as one of the 7-12 applications that would be funded by Knight this year.

It might be the high price-tag we requested ($585,000) and it might be that the idea didn’t convince some of the jury members. However we do hope and believe that the main reason is that the other applications were just better and that we’ll see some innovative open source projects supported by Knight and advancing local journalism in the years to come. Knight will publish the results in June.

We’re still on it!

NewsShift is definitely still on our agenda, as we believe it is the right direction for ShiftSpace, the right direction for our partners LittleSis and the right direction for watchdog journalism. We intend to hold an intense several days hackathon this summer with hackers and designers from both the LittleSis and ShiftSpace teams coding away towards a LittleSis prototype – a § space that would mash the LittleSis dataset networking powerful Americans with the data on any page you browse.

We will keep you posted towards it, but in the meanwhile, we wanted to post our full application here, so you can see what we proposed to Knight and how do we see this progressing ideally.

NewsShift slides

Knight Application

Project Description

NewsShift is a web platform that adds a collaborative research layer to online news stories. This layer, accessible from the news page itself, offers readers powerful tools to communicate and develop the story with additional information and insight — facilitating collaborative watchdog journalism.

As news budgets get cut, local journalists have limited time and resources for investigative reporting. At the same time, readers are responding to news stories with valuable research and analysis using blogs, comments, and other social tools. NewsShift allows readers to share their findings and work constructively with journalists to add new depth to local news reporting.

NewsShift tools let readers augment a news story in simple ways: index important names and topics; link place names to maps; footnote quotes with informative URLs. NewsShift can also query web services for data related to the story, letting readers curate what’s important.

A reader can use NewsShift to expose a local conflict of interest. For example: a story on a waterfront development project mentions the mayor’s role in picking the real estate developer. Using NewsShift, a reader discovers that the developer is a top campaign donor to the mayor, and inserts a link to this information. The story’s writer notices this link, and the substantial investigative work that followed it, and writes a follow-up piece airing concerns about the mayor’s ties to the developer.

Similar networked researches happens frequently, but are often lost in the comments or spread thin over the web. NewsShift connects local reporters and readers more closely in the journalistic process, and harnesses their combined research power to connect the news to a broader network of information.

How will your project improve the way news and  information are delivered to geographic communities?

Commenting interfaces on news sites have limited the  potential of collaborative journalism to conversation only. NewsShift integrates and enhances these social spaces around shared geographies, allowing local readers to find each other, track what’s happening in their community, and develop stories using their combined knowledge and passion. As local journalists and bloggers join in and draw insight from NewsShift activity, valuable content from this social space can flow back into published news stories. NewsShift thus gathers news writers and readers with real life social bonds to share, interpret, and act upon the events that shape their community — helping reestablish local news as a core infrastructure for grassroots democracy.

How is your idea innovative? (new or different from what already exists)

NewsShift embraces a sustainable, open ecosystem where readership and authorship interact in powerful ways and news articles become living social documents. Unlike existing services, users of the open-source NewsShift can easily build new content tools using JavaScript, HTML & CSS — the lingua franca of the web. NewsShift combines automatic content parsing with active collaboration between publishers and readers to generate semantic indexes. Finally, whereas existing “walled-garden” web services own user  content, NewsShift embraces the only sustainable design: a decentralized one, allowing hosts of the NewsShift platform to easily share data with each other and users to maintain full control of their own content.

What experience do you or your organization have to  successfully develop this project?

NewsShift is a collaboration between two organizations: ShiftSpace, and LittleSis. The team will partner with local news sites, bloggers, and community groups to  design the software and get NewsShift off the ground in one or two local communities.

David Nolen & Mushon Zer-Aviv are leading ShiftSpace, an open-source project that started at NYU’s ITP and in the past two years have been developed in and supported by Eyebeam, an art and technology center in NY. ShiftSpace evolved from a web annotation tool to an agile social software platform that site owners can use in specific sites (like MoMA.org) and web-users can use over any site (using a browser extension). Looking beyond just user-generated content, ShiftSpace promotes user-generated interfaces.

Kevin Connor & Matthew Skomarovsky are the creators of LittleSis.org, a  collaborative research website that tracks the relationships between powerful people  and organizations. LittleSis integrates data from  government filings with research gathered by its online community of  watchdogs.

The team holds a wide range of professional skills: dynamic front-end, P2P back-end, browser plugins, social interface design, data scraping, investigative research, collaboration with big organizations, and development of open APIs. Having our legs both in academia (NYU, The New School) and in the industry we bring a wide perspective and a critical eye to the technical, social, and media aspects of this project.

What unmet need does your proposal answer?

To take full advantage of their online medium, local news sites need to become more than just hubs for professional reporting. News sites can and should also lead the conversations and shared investigations that build community knowledge. Web publishers and journalists want to seamlessly tie together diverse stories and media. Local news readers want to combine their time, skills, and knowledge to further discuss, investigate and contextualize the news. Data services need simple ways to put their data to work directly within a variety of local news stories and discussions. NewsShift introduces a new model for producing online local news that meets these needs, built around community, collaboration, and critical investigation.

What tasks/benchmarks need to be accomplished to develop your project and by when will you complete them?

Prototype – 9 weeks

• Proof-of-concept prototype using the ShiftSpace and LittleSis APIs
• Test the prototype internally

Prototype Assessment – 2 weeks

• Basic functionality and information architecture
• Mockup basic UI approach

Base Technology – 25 weeks

• Input/output API to query and update external web services
• Web embed script (branching off of ShiftSpace code)
• Webpage query tool

Little Sis Integration – 9 weeks

• Build an integration layer on top of the LittleSis API
• Design and develop the LittleSis widget for browsing and editing profiles

Alpha Release – 5 weeks

• Browser Plugin release (invitation only)
• Initial usability testing for widget
• Compile initial list of local individuals and organizations to be entered into the database

Post-Alpha / Pre-Beta – 10 weeks

• Optimization, Security, Cleanup
• Strategy for defining local partnership (alliance)

Beta Release – 5 weeks

• Allow user apply for invitation
• Explicit invitations (media partners, power users, local bloggers, etc.)
• Beta test alliance – multiple publications

Post-Beta / Pre-Launch – 10 weeks

• Optimization, Security, Cleanup
• Launch NewsShift.org as a documentation and support site
• Naming marathon with alliance and community – events, data sprints, etc.
• Coordinate parallel launch between alliance members
• Coordinate marketing / countdown towards launch

Launch – 5 weeks

• Parallel launch on multiple alliance sites
• Manage and coordinate media attention
• Manage community input and adoption
• Extinguish fires (critical software bugs)

Post-Launch – 20 weeks

• Second official deploy
• Overview and edit of documentation
• Reach out to potential new alliance members
• Reach out to new data services (DocumentCloud, MAPlight, OpenSecrets, EveryBlock, etc.)

Second Round – beyond the scope of this application

• Build new local alliances
• Support the integration of new data services

Total: 100 weeks

What will you have changed by the end of your project?

A suite of news-centered social research tools will be available at no cost to budget-strapped local news organizations and bloggers. Such software does not currently exist. Installation will be particularly easy for publishers that use WordPress or Drupal, and the overhead of deploying new tools as they become available on the NewsShift platform will be almost zero.

Local news sites that use the NewsShift platform will fundamentally change the relationship between writers and their more engaged and savvy readers. The readers will have more powerful means of responding to and developing news stories after they’re published. Reader contributions will be layered over and interact with the story itself instead of just dumped in a long list of comments at the bottom of the page. Reporters and bloggers will have research tools easily available for them both before and after publishing the story and will have more direct ways to build on each others’ work. Communities of local watchdogs will be able to gather and work around the news stories that affect their community.

Developers of web services that offer data relevant to local news will share a common platform for putting their data to use on the news pages themselves. For example, currently a website like LittleSis.org can gather information about notable names appearing in Oakland Local, but it can’t bring that information back to the pages of Oakland Local itself. However, if Oakland Local uses NewsShift then LittleSis can use a NewsShift “app” that hyperlinks names in a story to their profiles on LittleSis, and any other news site or blog running the NewsShift platform could also enjoy and contribute to the LittleSis name-indexing app.

How will you measure progress and ultimately success?

We will use a mix of simple metrics and qualitative assessments to gauge our progress.  Though the project poses novel challenges, ultimately its success will depend on partner and user adoption. During the first stage, we are aiming for critical mass within a locally-rooted community, so we will measure our success within that community. How many partner sites have adopted NewsShift? How many users have signed up? How many edits are created? How many edits are being shared on social networks like Twitter and Facebook?

We will also gauge the journalistic contributions of this community and develope these assessments with input and evaluation from partner sites. How is NewsShift advancing the journalistic work? Are journalists and readers interacting using the platform? How collaborative is the networked research across users of partner sites? Is the platform easing the research load on journalists and bloggers? Is it cross-pollinating research efforts that were previously siloed? Is it driving traffic between the sites? Is the spirit of the community collaborative and supportive?

After building critical mass within a specific community, we will be looking to replicate our success within new communities. Are we actively forming new local alliances? When we expand into new communities, are multiple partner sites adopting within that community? At this point, after working out the kinks in its launch community, is NewsShift easy to adopt and integrate? And finally are we being surprised by new uses of NewsShift that we could never imagine ourselves?

Do you see any risk in the development of your project?

There are several areas of risk we identified. Some of these can be controlled through software and others lie outside the realm of technical solution and are largely behavioral or social in nature.

Abuse
Clay Shirky once defined “social software” as “stuff that gets spammed”. All popular social software to date have indeed become spam vectors and NewShift will need to take this into consideration. Fortunately there are many algorithms and methods for dealing with spam. If spam becomes a problem a more robust moderation/reputation systems can be put into place. One novel approach might be to integrate with existing commenting interfaces (WordPress) as they have robust and user-friendly moderation already in place.

Misleading Information
Users might mistakenly or deliberately enter misleading data. Again a user friendly and robust reputation framework can help mitigate this problem. Another thing to consider is Wikipedia dynamics. Users who have invested in annotating a page tend to create a “homeostasis” of credibility as evidenced by the success of Wikipedia.

Low Participation
Probably the greatest risk that faces NewShift. Fortunately both LittleSis, ShiftSpace and the online publishers we’ve been partnering with have existing communities that would very likely happily join the NewShift community. But without a doubt the key to the participation is a friendly UX, smart and targeted marketing and the high availability of the team.

Screencasts, blog posts, social networks and microblogging are some of the ways to get the word out. NewsShift will make it easy to share links to modified pages making it simple to share activity via services like Twitter and Facebook. Through these we hope to encourage new users to participate in the NewsShift ecosystem.

What is your marketing plan? How will people learn about what you are doing?

To redefine the relationship between local journalists, news readers, and data providers, NewsShift must actively bring together and communicate the benefits to a number of stakeholders.

We’ll begin with marketing to local news sites beyond our primary organizational partners, personally reaching out to enterprising local news publishers and bloggers, journalism departments at schools, and spreading the word at conferences and through various networks focused on technology and journalism. Detailed mock-ups, demo videos, and use cases will communicate the advantages of a new collaborative and data-rich publishing model.

Early in the process we will contact developers of relevant social tools and web services to discuss how their features and content can enhance local online news through the NewsShift platform. We will develop and thoroughly document an example NewsShift “app” based on the LittleSis API that embeds LittleSis content into news pages. We will put out a call to other developers through the various open source, new media, and political transparency communities we’re involved in. A contest and commissions will additionally generate interest and help support early NewsShift apps.

As we deploy NewsShift on local news sites, our marketing focus will turn to building a community of active news readers and watchdogs. We can draw many initial users by aggressively introducing NewsShift to communities on our team’s existing sites, and will organize online and offline events for local writers, watchdogs, and info geeks in order to grow the NewsShift community. As usage increases NewsShift will help market itself: users will be able to easily share their modified news pages via RSS and social services like Facebook and Twitter.

Is this a one-time experiment or do you think it will continue after the grant?

LittleSis and ShiftSpace are both open source initiatives that have been running for several years now and have already been encouraged by various grants. Our interest in NewsShift comes from a long experience of what works and what doesn’t work in both our fields. NewsShift is the natural continuation of that and an ideal combination of collaborative data services and social web annotations.

The germ of the idea for the NewsShift platform was planted before our application to the Knight News Challenge because it was a shared goal between the two projects. LittleSis as a data service would benefit immensely from NewsShift. ShiftSpace is already extending its reach beyond the plugin environment and is continuously committed to lowering the costs and raising the accessibility of user generated interfaces.

We intend to bring the same enthusiasm and interest to NewsShift as we have been investing in LittleSis and ShiftSpace. We will continue our work after the grant and as you can see in the benchmarks section, intend to grow its scope beyond the grant period. If our goal of expanding adoption of the NewsShift software platform succeeds, the project will evolve in diverse environments without much direct involvement on our part.

NewsShift, like LittleSis and ShiftSpace, is exploratory in nature, but we intend to create (as we have already have) working software and to foster real communities. NewsShift is not a one-time experiment and we intend to leverage our experience to develop long lasting foundations for a thriving ongoing open source project well beyond the two years outlined here.

If it is to be self-sustainable, what is the plan for making that happen?

NewsShift bridges between publishers, data services, and readers. Much of the grant funds will go to lowering the costs and barriers for this multi-way collaboration.

It is not enough to release the code for an open source community to thrive. It is also not enough to launch a new web service for a collaborative community to grow. We have learned this from experience and this directly informs our approach in developing NewsShift.

Also, while we intend to be invested in NewsShift for years to come, it’s just as important that the project scale beyond our personal involvement. For this to happen we will focus on the following goals.

Iteration:
Get new things into the hands of enthusiastic users as soon as possible. Find out what works and what doesn’t quickly.

Documentation:
Make sure every feature and function is documented well and foster a culture of documentation in the community.

Hackability:
Simple installation. Allow people to easily experiment and realize their own ideas. Lower the barriers to doing so by developing simple and flexible APIs.

Community:
Choose the right (limited) set of communication tools and allow for emergent leadership. Lower the costs of collaboration by balancing individual autonomy and group coordination.

Incentivize contribution:
Learn from the experience of open source incentive management and be attentive to our partners and contributors interests and goals.  Support and help develop open source business models for installation and customization of the software (like ShiftSpace did on MoMA.org).

Open standards:
Standardize data formats to allow interoperability and prevent data lock-in.

Partnerships:
Identify both data services and new local news alliances to secure the adoption and development of the platform.

Possible future Data partners:
DocumentCloud, MAPlight, OpenSecrets.org, EveryBlock, Government Acronym Glossary, there are literally hundreds of data partners who are segragated within their URLs and could use a NewsShift …

Help us!

If you have tips for the application, if you have other funding venues you think we might want to look into, or if you want to help us make it with your code, ideas, or whatever, please contact the § mailing list or just comment here.

Much appreciated!